CloudQuery development mode token acquisition issue with missing AZURE_TENANT_ID

CloudQuery Plugins
1

Hi there!

I was trying to play with CloudQuery in the dev mode. I followed the steps mentioned in the development_environment doc. All steps till step no 3 are successful. All plugins have been built, the server is running, and the source and destination plugin configs are set.

However, step 4 fails saying that CloudQuery can’t acquire a token for Azure; the AZURE_TENANT_ID is missing. I have checked the environment variables, and both printenv and env give the right results. In fact, the normal non-dev mode works absolutely fine.

I am using WSL2 on Windows. The command I used is:

~/cloudquery-dev/cloudquery$ bin/cloudquery sync cloudquery-config

The cloudquery directory contains the bin, and the cloudquery-config is the directory for the two plugin files - azure.yml and neo4j.yml.

Any help would be appreciated!

2

Hi @summary-swift :wave:, would you also be able to include your cloudquery-config with any sensitive information redacted? Thanks.

AZURE_TENANT_ID missing
3

Sure!

azure.yml

kind: source
spec:
  # Source spec section
  name: "azure"
  path: "localhost:7777"
  version: "v9.3.4"
  registry: "grpc"
  destinations: ["neo4j"]
  tables: ["azure_compute_availability_sets", "azure_network_load_balancers", "azure_network_security_groups", "azure_network_virtual_networks", "azure_compute_virtual_machines", "azure_network_interfaces", "azure_compute_disks"]
  spec:
    # Optional parameters
    # subscriptions: []
    # cloud_name: ""
    # concurrency: 50000
    # discovery_concurrency: 400
    # skip_subscriptions: []
    # normalize_ids: false
    # oidc_token: ""

neo4j.yml

kind: destination
spec:
  name: "neo4j"
  registry: "github"
  path: "cloudquery/neo4j"
  version: "v4.0.8"
  spec:
    connection_string: "bolt://localhost:7687"
    username: "xxx"
    password: "xxx"
    # Optional parameters:
    # batch_size: 1000 # 1K entries
    # batch_size_bytes: 4194304 # 4 MiB
4

The plugin uses the SDK default credential chain, which attempts to get credentials via either the environment, workload identity, managed identity, or the CLI. Just to confirm, are you attempting to configure the credentials via environment variables?

5

Yes, environment variables.
It works fine with the non-dev method (essentially where the config doesn’t have the registry and the path is cloudquery/azure).

6

Would you be able to run the Azure server again but with the following command:

env | grep AZURE && go run main.go --serve

I would expect to see all the required environment variables set in the terminal session where you are running the source plugin. The source plugin logs would also be useful to see if you wouldn’t mind pasting the output of the above command here - with any sensitive info redacted. Thanks.

7

Ok, so it worked.

The server side wasn’t recognizing the env variables. I again exported them and ran the usual server command, and it worked!

Thanks again, and sorry for the trouble!