# Inconsistent sync results and rate limit issues in CloudQuery Azure configuration **URL:** \ **Category:** CloudQuery Plugins\ **Created:** [September 19, 2023, 11:42am UTC](https://community.cloudquery.io/t/inconsistent-sync-results-and-rate-limit-issues-in-cloudquery-azure-configuration/726 "2023-09-19T11:42:25Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![thorough-bear](https://avatars.discourse-cdn.com/v4/letter/t/f08c70/32.png) [@thorough-bear](https://community.cloudquery.io/u/thorough-bear)\ **Post date:** [September 19, 2023, 11:42am UTC](https://community.cloudquery.io/t/inconsistent-sync-results-and-rate-limit-issues-in-cloudquery-azure-configuration/726/1 "2023-09-19T11:42:25Z")
Hi, I’ve noticed that when an Azure sync experiences rate limit issues, I get inconsistent results. For example: 1. When tables are `azure_storage_accounts, azure_storage_containers`, there is no rate limit in the logs and the number of resources synced from `azure_storage_containers` is 866 with 0 errors, which is the expected result. I’ve done multiple syncs with this configuration and the number of resources stays the same. 2. When tables are `azure_storage_accounts, azure_storage_containers, azure_storage_file_shares, azure_storage_blob_services`, there is a rate limit in the logs and the number of resources synced from `azure_storage_containers` is 780 with 20 errors. I’ve done multiple syncs with this configuration and the number of resources is inconsistent, meaning that the number of resources is sometimes X and sometimes Y, but not the expected 866. I’m using Azure v9.3.0 if that matters. Has anyone else experienced this? Thanks! ---
**Author:** ![kemal](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/kemal/32/42_2.png) [@kemal](https://community.cloudquery.io/u/kemal)\ **Post date:** [September 19, 2023, 12:52pm UTC](https://community.cloudquery.io/t/inconsistent-sync-results-and-rate-limit-issues-in-cloudquery-azure-configuration/726/2 "2023-09-19T12:52:03Z")
That is interesting… Just opening an issue to track this. Here’s the link for tracking: [GitHub Issue #13944](https://github.com/cloudquery/cloudquery/issues/13944) Thanks for reporting. ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [September 20, 2023, 3:35pm UTC](https://community.cloudquery.io/t/inconsistent-sync-results-and-rate-limit-issues-in-cloudquery-azure-configuration/726/3 "2023-09-20T15:35:22Z")
@thorough-bear Reducing the `concurrency` value should help with this. But we can also investigate on our side to see if there’s a way to better schedule the queries so that they don’t hit Azure rate limits. ---
**Author:** ![thorough-bear](https://avatars.discourse-cdn.com/v4/letter/t/f08c70/32.png) [@thorough-bear](https://community.cloudquery.io/u/thorough-bear)\ **Post date:** [September 21, 2023, 8:36am UTC](https://community.cloudquery.io/t/inconsistent-sync-results-and-rate-limit-issues-in-cloudquery-azure-configuration/726/4 "2023-09-21T08:36:51Z")
@herman I’m using `concurrency=1`, but it doesn’t seem to help. ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [September 21, 2023, 8:38am UTC](https://community.cloudquery.io/t/inconsistent-sync-results-and-rate-limit-issues-in-cloudquery-azure-configuration/726/5 "2023-09-21T08:38:18Z")
Could you share your (redacted) config, @thorough-bear? ---
**Author:** ![thorough-bear](https://avatars.discourse-cdn.com/v4/letter/t/f08c70/32.png) [@thorough-bear](https://community.cloudquery.io/u/thorough-bear)\ **Post date:** [September 21, 2023, 9:24am UTC](https://community.cloudquery.io/t/inconsistent-sync-results-and-rate-limit-issues-in-cloudquery-azure-configuration/726/6 "2023-09-21T09:24:51Z")
```auto "kind: source\nspec:\n name: AZURE_135c8a6b-2a45-4225-9aee-87dde3de856b_09063151-829b-4cce-b6ba-8d89873620ed\n\ \ path: \"cloudquery/azure\"\n version: v9.3.0\n tables: ['azure_storage_accounts',\ \ 'azure_storage_containers']\n destinations: ['elasticsearch']\n skip_dependent_tables:\ \ True\n deterministic_cq_id: True\n spec:\n subscriptions: [4d9b301c-ceab-4682-ba37-88700e6f5d0b]\n\ \ cloud_name: AzurePublic\n concurrency: 1\n discovery_concurrency: 1" ``` Sorry for it looking so bad, it is loaded as part of a ConfigMap into a K8s job, so I had to print it. Here is the template before it is rendered: ```auto kind: source spec: name: {{ config.name }} path: "cloudquery/{{ config.source_name }}" version: {{ config.version }} tables: {{ config.tables }} destinations: {{ config.destinations }} skip_dependent_tables: {{ config.skip_dependent_tables }} deterministic_cq_id: {{ config.deterministic_cq_id }} spec: subscriptions: [{{ config.subscription_id }}] cloud_name: {{ config.cloud_name }} concurrency: {{ config.concurrency }} discovery_concurrency: {{ config.discovery_concurrency }} ``` ---
**Author:** ![herman](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/herman/32/27_2.png) [@herman](https://community.cloudquery.io/u/herman)\ **Post date:** [September 21, 2023, 9:29am UTC](https://community.cloudquery.io/t/inconsistent-sync-results-and-rate-limit-issues-in-cloudquery-azure-configuration/726/7 "2023-09-21T09:29:41Z")
And you’re saying this happens even with `concurrency` set to 1? That’s quite strange; we’ll have to investigate. 🤔 ---
**Author:** ![thorough-bear](https://avatars.discourse-cdn.com/v4/letter/t/f08c70/32.png) [@thorough-bear](https://community.cloudquery.io/u/thorough-bear)\ **Post date:** [September 21, 2023, 9:33am UTC](https://community.cloudquery.io/t/inconsistent-sync-results-and-rate-limit-issues-in-cloudquery-azure-configuration/726/8 "2023-09-21T09:33:45Z")
Yes, I set `concurrency=1` every time. As part of sending you the config, I had to run my program a couple of times, which initiated a couple of syncs. I noticed that: When I ran the tables `azure_storage_accounts` and `azure_storage_containers` twice one after another, I see in the logs that the second sync gets a lot of `429 TooManyRequests`, and the number of resources synced under `azure_storage_containers` is just 720 with 43 errors. The first sync has no errors, and the number of `azure_storage_containers` resources is 866 (as expected).