# CloudQuery Azure v9.3.0 error message indicates a potential timeout issue

**URL:** <https://community.cloudquery.io/t/cloudquery-azure-v9-3-0-error-message-indicates-a-potential-timeout-issue/670>\
**Category:** CloudQuery Plugins\
**Created:** [October 11, 2023, 10:10am UTC](https://community.cloudquery.io/t/cloudquery-azure-v9-3-0-error-message-indicates-a-potential-timeout-issue/670 "2023-10-11T10:10:43Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![thorough-bear](https://avatars.discourse-cdn.com/v4/letter/t/f08c70/32.png) [@thorough-bear](https://community.cloudquery.io/u/thorough-bear)\
**Post date:** [October 11, 2023, 10:10am UTC](https://community.cloudquery.io/t/cloudquery-azure-v9-3-0-error-message-indicates-a-potential-timeout-issue/670/1 "2023-10-11T10:10:43Z")

</div>

Another one…  
I’m using Azure v9.3.0.  
I’m seeing this log:

```auto
{"level":"error","module":"cli","client":"subscriptions/43e6231a-e00b-46fa-9d7b-aeb81c00b10d","error":"context deadline exceeded","message":"pre resource resolver failed","module":"azure-src","table":"azure_storage_tables","time":"2023-10-11T09:47:01Z"}

```

Does that message indicate a timeout?

- I’ve added a proxy server to handle 429 responses, which sleeps for 5 mins before retrying (it does it recursively until a successful response); only then it returns a response to the client.

---

<div class="post-metadata">

**Author:** ![marcel](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/marcel/32/48_2.png) [@marcel](https://community.cloudquery.io/u/marcel)\
**Post date:** [October 11, 2023, 1:27pm UTC](https://community.cloudquery.io/t/cloudquery-azure-v9-3-0-error-message-indicates-a-potential-timeout-issue/670/2 "2023-10-11T13:27:12Z")

</div>

Our tooling will raise that error if the server does not respond within an expected timeframe. This is different from a 429, which is a meaningful response from the server. [Context Deadline Exceeded](https://gosamples.dev/context-deadline-exceeded/)

---

<div class="post-metadata">

**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\
**Post date:** [October 11, 2023, 2:47pm UTC](https://community.cloudquery.io/t/cloudquery-azure-v9-3-0-error-message-indicates-a-potential-timeout-issue/670/3 "2023-10-11T14:47:08Z")

</div>

Hi @thorough-bear,

The Azure Go SDK already has retry logic, see [this link](https://github.com/Azure/azure-sdk-for-go/blob/ef4cd3ef8f9fc2deb4240daa0f822c3c9486e665/sdk/azcore/policy/policy.go#L83).

So you shouldn’t need to handle 429 yourself. We could consider exposing those settings via CloudQuery config if you submit a feature request for it [here](https://github.com/cloudquery/cloudquery/issues/new?assignees=&labels=enhancement&projects=&template=feature_request.yml&title=feat%3A+%3Ctitle%3E). Please state under which conditions you needed to handle the 429 error.

As for `context deadline exceeded`, that might also indicate a memory issue, so I’d try reducing concurrency. You can refer to the documentation [here](https://www.cloudquery.io/docs/reference/source-spec#concurrency).

Finally, please make sure you’re not using `az login` as it can cause performance issues, see [this link](https://www.cloudquery.io/docs/plugins/sources/azure/overview#authentication).

---

<div class="post-metadata">

**Author:** ![thorough-bear](https://avatars.discourse-cdn.com/v4/letter/t/f08c70/32.png) [@thorough-bear](https://community.cloudquery.io/u/thorough-bear)\
**Post date:** [October 11, 2023, 3:21pm UTC](https://community.cloudquery.io/t/cloudquery-azure-v9-3-0-error-message-indicates-a-potential-timeout-issue/670/4 "2023-10-11T15:21:55Z")

</div>

Hi @erez,

Thank you for the response!

The reason I added this handling of `429` is because I noticed an inconsistent behavior when throttling occurs. See this thread: [Discord Thread](https://discord.com/channels/872925471417962546/873606591335759872/1153657348842332210).

After adding this proxy as a workaround, I noticed that it did solve the inconsistency problem.

**BUT**

I then ran this solution on one huge Azure Subscription with lots of resources under storage, and I now get `context deadline exceeded` as I believe that there are simply too many resources and some requests sleep for too long. I changed the sleep time to 30 seconds, but some requests might get into a loop of sleep if they keep encountering `429`.

I’ll try reducing the concurrency and let you know. I’m not using `az login`; I pass the following environment variables to the CQ CLI:

```auto
AZURE_TENANT_ID
AZURE_CLIENT_ID
AZURE_CLIENT_SECRET
AZURE_SUBSCRIPTION_ID

```

---

<div class="post-metadata">

**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\
**Post date:** [October 11, 2023, 3:34pm UTC](https://community.cloudquery.io/t/cloudquery-azure-v9-3-0-error-message-indicates-a-potential-timeout-issue/670/5 "2023-10-11T15:34:28Z")

</div>

If you can monitor the resources of the machine that runs the sync, that would be good too to help us debug.

---

<div class="post-metadata">

**Author:** ![thorough-bear](https://avatars.discourse-cdn.com/v4/letter/t/f08c70/32.png) [@thorough-bear](https://community.cloudquery.io/u/thorough-bear)\
**Post date:** [October 12, 2023, 6:35pm UTC](https://community.cloudquery.io/t/cloudquery-azure-v9-3-0-error-message-indicates-a-potential-timeout-issue/670/6 "2023-10-12T18:35:01Z")

</div>

I wasn’t able to monitor the resources of the machine. Sorry about that.

So I did some research and found an interesting thing about the 429 problem mentioned in the other thread: According to the Azure Go Client SDK CHANGELOG, they mentioned that

```auto
Don't retry a request if the Retry-After delay is greater than the configured RetryOptions.MaxRetryDelay.

```

(found here - [Azure SDK for Go Releases](https://github.com/Azure/azure-sdk-for-go/releases/tag/sdk%2Fazcore%2Fv1.1.4)).

Using the proxy, I was able to find out that most of the headers returned from `azure_storage_*` of a 429 response have a `Retry-After` which is greater than the `Default RetryOptions.MaxRetryDelay`, which is set to 60, meaning that Azure won’t retry them.

Correct me if I’m wrong, but CQ uses the default value and that’s why it does not handle 429 responses well (at least not from `azure_storage_*`). This can explain the inconsistent behavior.

> If you can monitor the resources of the machine that runs the sync, that would be good too to help us debug.

Could you point me to how I can monitor this? I’m running on my local machine now, Linux if that makes a difference.

---

<div class="post-metadata">

**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\
**Post date:** [October 13, 2023, 9:31am UTC](https://community.cloudquery.io/t/cloudquery-azure-v9-3-0-error-message-indicates-a-potential-timeout-issue/670/7 "2023-10-13T09:31:48Z")

</div>

Hi @thorough-bear,

That makes sense about the Azure Go SDK. Thanks for digging into it. I assume they do that so it won’t sleep/hang for too long.

So I think the solution would be to expose those as options in the spec, or even have different defaults for different tables per [issue #9860](https://github.com/cloudquery/cloudquery/issues/9860) as depending on `Retry-After`, we might not retry some resources at all. I’ll take a look at that.

About monitoring, if you’re running locally, I would use a tool like [`htop`](https://htop.dev/) to get some basic visibility on the machine resources while running a sync.

Related issues we have open are [issue #11225](https://github.com/cloudquery/cloudquery/issues/11225) and [issue #9860](https://github.com/cloudquery/cloudquery/issues/9860).