CloudQuery IAM crawling region limitations for us-east-1 query issues

Hey, for IAM crawling, is it necessary to query from us-east-1? That’s what it seemed like from some cloudquery.log errors that said that IAM roles/users couldn’t be iterated through except for a subset of regions (with us-east-1 being the only US one). I was trying to otherwise capture my us-west-2 environment. However, I noticed that the AWS CLI doesn’t need the region to be us-east-1, so I wasn’t sure if I was misinterpreting the cloudquery.log error message or if there is a restriction currently in place.

Hi @balanced-tick, can you please share your configuration YAML file?

The IAM table is a global table, so you’re right, it doesn’t need the region (technically it uses us-east-1). If you explicitly specify regions: in your configuration file, and also specify a table that does not support that region (e.g. regions: [us-west-2] and try to get the IAM tables), CloudQuery CLI will warn you since this could be a user error of trying to sync resources from a region they are not available in.

This is a bit different from the AWS CLI. I tried:

aws iam list-users --region us-west-2

and it seems to ignore that flag.

Yeah, that’s essentially what’s happening. I’m taking in an input region which could not be us-east-1, and trying to get IAM resources from that, and I believe it won’t get them if us-east-1 is not one of the regions. Here’s the format string I use for my source YAML:

kind: source
spec:
  # Source spec section
  name: aws
  path: cloudquery/aws
  registry: cloudquery
  version: "v25.4.0"
  tables: ["aws_apigateway*", "aws_autoscaling_groups*",
    "aws_autoscaling_plans*", "aws_cloudfront*", "aws_cloudwatch*",
    "aws_dynamodb*", "aws_ebs*", "aws_ec2*", "aws_eks_clusters*", "aws_elbv1*",
    "aws_elbv2*", "aws_iam*", "aws_kms*", "aws_lambda*", "aws_rds*",
    "aws_route53*", "aws_s3*", "aws_secretsmanager*"]
  skip_tables: ["aws_ec2_instance_types", "aws_ebs_encryption_by_defaults",
    "aws_ec2_image_block_public_access_states", "aws_ec2_regional_configs",
    "aws_ec2_serial_console_access_statuses", "aws_ec2_account_attributes",
    "aws_ec2_snapshot_block_public_access_states",
    "aws_iam_group_last_accessed_details",
    "aws_iam_policy_last_accessed_details",
    "aws_iam_role_last_accessed_details", "aws_lambda_runtimes"]
  destinations: ["postgresql"]
  spec:
    accounts:
      - id: "{self.session.account_id}"
        role_arn: "{self.session.role_arn}"
        role_session_name: "{self.session.role_session_name}"
        external_id: "{self.session.external_id}"
        regions:
          - "{self.session.region_name}"
    concurrency: 1000

:+1: So I think you can safely ignore that log message.

But isn’t the log message telling me that I’m not actually retrieving IAM resources?

Ah, sorry, I misunderstood. I thought you were syncing both regions. You would need to configure us-east-1 as a region to sync the IAM resources. A possible solution would be to create a separate configuration file only for IAM.

Yup, that’s what I ended up doing! Just wanted to make sure I wasn’t doing extra work for no reason, thanks!