Hey, for IAM crawling, is it necessary to query from us-east-1
? That’s what it seemed like from some cloudquery.log
errors that said that IAM roles/users couldn’t be iterated through except for a subset of regions (with us-east-1
being the only US one). I was trying to otherwise capture my us-west-2
environment. However, I noticed that the AWS CLI doesn’t need the region to be us-east-1
, so I wasn’t sure if I was misinterpreting the cloudquery.log
error message or if there is a restriction currently in place.
Hi @balanced-tick, can you please share your configuration YAML file?
The IAM table is a global table, so you’re right, it doesn’t need the region (technically it uses us-east-1
). If you explicitly specify regions:
in your configuration file, and also specify a table that does not support that region (e.g. regions: [us-west-2]
and try to get the IAM tables), CloudQuery CLI will warn you since this could be a user error of trying to sync resources from a region they are not available in.
This is a bit different from the AWS CLI. I tried:
aws iam list-users --region us-west-2
and it seems to ignore that flag.
Yeah, that’s essentially what’s happening. I’m taking in an input region which could not be us-east-1
, and trying to get IAM resources from that, and I believe it won’t get them if us-east-1
is not one of the regions. Here’s the format string I use for my source YAML:
kind: source
spec:
# Source spec section
name: aws
path: cloudquery/aws
registry: cloudquery
version: "v25.4.0"
tables: ["aws_apigateway*", "aws_autoscaling_groups*",
"aws_autoscaling_plans*", "aws_cloudfront*", "aws_cloudwatch*",
"aws_dynamodb*", "aws_ebs*", "aws_ec2*", "aws_eks_clusters*", "aws_elbv1*",
"aws_elbv2*", "aws_iam*", "aws_kms*", "aws_lambda*", "aws_rds*",
"aws_route53*", "aws_s3*", "aws_secretsmanager*"]
skip_tables: ["aws_ec2_instance_types", "aws_ebs_encryption_by_defaults",
"aws_ec2_image_block_public_access_states", "aws_ec2_regional_configs",
"aws_ec2_serial_console_access_statuses", "aws_ec2_account_attributes",
"aws_ec2_snapshot_block_public_access_states",
"aws_iam_group_last_accessed_details",
"aws_iam_policy_last_accessed_details",
"aws_iam_role_last_accessed_details", "aws_lambda_runtimes"]
destinations: ["postgresql"]
spec:
accounts:
- id: "{self.session.account_id}"
role_arn: "{self.session.role_arn}"
role_session_name: "{self.session.role_session_name}"
external_id: "{self.session.external_id}"
regions:
- "{self.session.region_name}"
concurrency: 1000
So I think you can safely ignore that log message.
But isn’t the log message telling me that I’m not actually retrieving IAM resources?
Ah, sorry, I misunderstood. I thought you were syncing both regions. You would need to configure us-east-1
as a region to sync the IAM resources. A possible solution would be to create a separate configuration file only for IAM.
Yup, that’s what I ended up doing! Just wanted to make sure I wasn’t doing extra work for no reason, thanks!