Azure query locks client node with high memory usage in CloudQuery

CloudQuery Plugins
1

Hi there :wave:

I’m running a PoC for CloudQuery and successfully implemented queries for AWS and GCP, with full table scans across multiple accounts/projects. However, I have an issue with Azure where my client node (c5.4xlarge) locks up completely when I do the same for Azure; before it does, memory usage will spike up and kswapd0 will eat a CPU core which usually means something doesn’t play well with memory. If I can get CloudQuery to terminate, the VM might go back to being responsive, but I usually have to stop it from the AWS console (which takes some time and will eventually succeed).

I have tried both v9.3.7 and v9.3.8 with a straightforward configuration:

kind: source
spec:
  name: azure
  path: cloudquery/azure
  version: "v9.3.8"
  tables: ["*"]
  destinations: ["postgresql"]
  spec:
  backend_options:
    table_name: "cq_azure_state"
    connection: "@@plugins.postgresql.connection"

Is there a good way of debugging this further?

2

Hi @included-collie,

Are you syncing lots of subscriptions? If you are, I would suggest that you try and lower the discovery_concurrency value…

3

Yep, absolutely do - do you have a hint on a sane value when there are many subscriptions? I might want to lower concurrency as well …

4

How are you sourcing your credentials for Azure?

5

They’re exported to env.

6

Ok. Just wanted to make sure you aren’t using azure login as that spawns a process for each authentication token that is needed.

7

I’ve now started another run with a conservative discovery_concurrency: 50 and concurrency: 1000 which is looking much better.

8

Great! Let us know how that goes.

9

Thanks for the pointer to the concurrency settings…!