Rate limiting issues with CloudQuery IAM service access

CloudQuery Plugins
1

Hi all!

I’m facing some rate limiting issues while executing CloudQuery. The error I get is this one:

ERR table resolver finished with error error="operation error IAM: GenerateServiceLastAccessedDetails, exceeded maximum number of attempts, 10, https response error StatusCode: 400, RequestID: XXXXXXXXXXXXXXXXX, api error Throttling: Rate exceeded" client=XXXXXXXXXXXXX:us-east-1 module=aws-src table=aws_iam_role_last_accessed_details

Not sure how I can prevent this. I was adding more tables to the aws.yml file, so maybe the solution is to split the file into several ones and trigger CloudQuery several times, pointing to those “smaller” configuration files one after the other.

2

:wave: Hi Alex, there are a couple of things you can try to avoid the rate limiting:

  1. As you suggested, you can try splitting the resources into different configurations and have them run on different schedules.
  2. Try reducing the concurrency (see AWS Concurrency Documentation).
  3. Finally, you could try using the scheduler: shuffle option to balance the load across API calls (see Performance Tuning Documentation).

I hope this helps!

3

Thank you very much for the tips! I’ll try that and check if it works better.

4

One further option is to consider skipping some of the tables that may be slowing the sync down or triggering the rate limit but aren’t actually useful to you - see this link.

5

In this case, I’d like to get the info from all tables, but thanks anyway for the tip.