Error 429 Too Many Requests with CloudQuery Azure Storage configuration

CloudQuery Plugins
1

I have played around with the spec.concurrency and scheduler: "round-robin", but I still can’t pass this error 429:

ERR table resolver finished with error error="GET https://management.azure.com/subscriptions/xxx/resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/sa/blobServices/default/containers\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: TooManyRequests\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"TooManyRequests\",\n    \"message\": \"The request is being throttled as the limit has been reached for operation type - List_ObservationWindow_00:05:00. For more information, see - https://aka.ms/srpthrottlinglimits\"\n  }\n}\n--------------------------------------------------------------------------------\n" client=subscriptions/xxx module=azure-src table=azure_storage_containers

I read the mentioned document, Azure Subscription Service Limits. I do see the list limits on storage accounts: only 100 per 5 minutes.

Storage account management operations (list)  - 100 per 5 minutes

How can I adjust the source configuration to pass this issue?

2

Hi @calm-walrus,

I’m currently working on exposing the Azure Go SDK settings so one could configure the retries.

You can subscribe to this issue.

I’ll close that issue once the fix is done.

3

Yes, that’s the same issue, thanks.
@erez I saw this issue has been closed with a PR merged.

What’s the meaning?

Do I need to install a new version of cloudquery command line?

4

I’ll release a new Azure plugin version in a bit for you to try:

https://github.com/cloudquery/cloudquery/pull/14766 this one.

5

ic. Sure. I’ll wait for it.

6

@erez

I have switched to use the latest Azure azure@v10.1.0, but I still got a lot of 429 errors.

RESPONSE 429: 429 Too Many Requests
7

Hi @calm-walrus, have you tried changing the default retry options? See this link.

I would try a very high max_retry_delay_seconds for starters and also set max_retries and retry_delay_seconds to a higher value than the defaults.

8

let me try

retry_options:
  max_retries: 5
  try_timeout_seconds: 0
  retry_delay_seconds: 60
  max_retry_delay_seconds: 300