Rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error

General
1

This has been working fine for some time, not sure when exactly it started failing.
Using CQ on AWS infrastructure with a GCP service account; in this particular case, the source plugin/table is GCP’s IAM predefined roles. I am getting the following error constantly:

{"level":"info","grpc.code":"Unauthenticated","grpc.component":"client","grpc.error":"rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: Get "``http://169.254.169.254/latest/meta-data/iam/security-credentials/{obfuscated}\``": EOF","grpc.method":"ListRoles","grpc.method_type":"unary","grpc.request.deadline":"2026-07-07T11:57:01Z","grpc.service":"google.iam.admin.v1.IAM","grpc.start_time":"2026-07-07T11:56:01Z","grpc.time_ms":"4566.946","invocation_id":"{obfuscated}","message":"finished call","module":"gcp-source","peer.address":"@","protocol":"grpc","time":"2026-07-07T11:56:06Z","time":"2026-07-07T11:56:06Z"}

When doing the same call with curl from the same pod, it works fine:

GET /latest/meta-data/iam/security-credentials/{obfuscated} HTTP/1.1
Host: 169.254.169.254
User-Agent: curl/8.21.0
Accept: /

Request completely sent off
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Accept-Ranges: none
< Last-Modified: Tue, 07 Jul 2026 11:57:30 GMT
< Content-Length: 1582
< Date: Tue, 07 Jul 2026 11:58:55 GMT
< Server: EC2ws
< Connection: close
<
{
"Code" : "Success",
"LastUpdated" : "2026-07-07T11:57:18Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "{obfuscated}",
"SecretAccessKey" : "{obfuscated}",
"Token" : "{obfuscated}",
"Expiration" : "2026-07-07T18:07:44Z"
}

All source plugins and CLI have been upgraded to the latest version.
Thanks!
J.

2

Hi @juergen_k do you mind sharing the full sync configuration redacted from any personal information?

Also please share the sync logs if possible, specifically if the sync failed outright at the beginning, or did it finish, but with some errors.

Finally, could you explain how you’re configuring the GCP service account? Via a service account JSON env variable (or file)? Or a different set up?

Thanks,
Erez

3

@erez I cannot share the full logs because there’s too much to redact here - but maybe it’ll help when I explain what I see; otherwise I can probably provide them to you directly via E-Mail.
I’m wondering if this might be related to GCP APIs rate limiting us (e.g. for quota reasons) since we are querying data for ~4400 accounts. What I can say is that other tables work fine with the same configuration.

apiVersion: v1
data:
config-gcp.yaml: |
kind: source
spec:
name: "gcp"
path: "cloudquery/gcp"
version: "v23.0.0"
tables: ["${CLOUDQUERY_GCP_TABLE}"]
skip_tables: ["gcp_dataproc_regions","gcp_dataproc_autoscaling_policies"]
destinations: ["postgresql", "bigquery"]
backend_options:
table_name: "cq_gcp_state"
connection: "@@plugins.postgresql.connection"
kind: ConfigMap

We’re using a JSON env file:

apiVersion: v1
data:
service-account-gcp.json: |-
{
"type": "external_account", (...)

The logs start out clean, let me try to copy and paste the most notable entries:

(...)
{"level":"info","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"Using service account for GCP authentication","module":"gcp-source","service_account":"application-default-credentials","time":"2026-07-08T07:06:18Z","time":"2026-07-08T07:06:18Z"}
{"level":"info","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"No organization_ids or organization_filter specified - assuming all organizations","module":"gcp-source","time":"2026-07-08T07:06:18Z","time":"2026-07-08T07:06:18Z"}
{"level":"info","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"Listing organizations...","module":"gcp-source","time":"2026-07-08T07:06:18Z","time":"2026-07-08T07:06:18Z"}
{"level":"info","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"No project_ids, folder_ids, or project_filter specified - assuming all active projects","module":"gcp-source","time":"2026-07-08T07:06:18Z","time":"2026-07-08T07:06:18Z"}
{"level":"info","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"Found 4384 projects. First 100

(...)
{"level":"info","client":"project:{1}","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"top level table resolver started","module":"gcp-source","table":"gcp_iam_predefined_roles","time":"2026-07-08T07:07:17Z","time":"2026-07-08T07:07:17Z"}
{"level":"info","client":"project:{2}","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"top level table resolver started","module":"gcp-source","table":"gcp_iam_predefined_roles","time":"2026-07-08T07:07:17Z","time":"2026-07-08T07:07:17Z"}
(...)
{"level":"info","client":"project:{100}","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"top level table resolver started","module":"gcp-source","table":"gcp_iam_predefined_roles","time":"2026-07-08T07:07:17Z","time":"2026-07-08T07:07:17Z"}
{"level":"info","grpc.code":"Unauthenticated","grpc.component":"client","grpc.error":"rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: Get "``http://169.254.169.254/latest/meta-data/iam/security-credentials\``": EOF","grpc.method":"ListRoles","grpc.method_type":"unary","grpc.request.deadline":"2026-07-08T07:08:17Z","grpc.service":"google.iam.admin.v1.IAM","grpc.start_time":"2026-07-08T07:07:17Z","grpc.time_ms":"1047.618","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"finished call","module":"gcp-source","peer.address":"@","protocol":"grpc","time":"2026-07-08T07:07:18Z","time":"2026-07-08T07:07:18Z"}

and that’s where we start seeing tons of errors and the sync ultimately fails.

4

Hi @juergen_k based on the logs what seems to be the error is the credentials negotiation from AWS side

Unauthenticated desc = transport: per-RPC creds failed due to error: Get "``http://169.254.169.254/latest/meta-data/iam/security-credentials\``": EOF","grpc.method":"ListRoles","grpc.method_type":"unary","grpc.request.deadline"

Agree this is probably related to GCP plugin doing many concurrent requests.
To solve it we can try two options:

  1. Reduce the plugin concurrency (see example here GCP Source Plugin | CloudQuery ). I would try a low value to debug first, e.g. concurrency: 100 then raise until it’s stable.
  2. Make sure you use IMDSv2 and not IMDSv1 with AWS. IMDSv2 has better credentials caching.

Examples of the differences below:

IMDSv1

{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/providers/aws-provider",
  "subject_token_type": "urn:ietf:params:aws:token-type:aws4_request",
  "token_url": "https://sts.googleapis.com/v1/token",
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/my-sa@my-project.iam.gserviceaccount.com:generateAccessToken",
  "credential_source": {
    "environment_id": "aws1",
    "region_url": "http://169.254.169.254/latest/meta-data/placement/availability-zone",
    "url": "http://169.254.169.254/latest/meta-data/iam/security-credentials",
    "regional_cred_verification_url": "https://sts.{region}.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15"
  }
}

IMDSv2(additional imdsv2_session_token_url under credential_source)

{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/providers/aws-provider",
  "subject_token_type": "urn:ietf:params:aws:token-type:aws4_request",
  "token_url": "https://sts.googleapis.com/v1/token",
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/my-sa@my-project.iam.gserviceaccount.com:generateAccessToken",
  "credential_source": {
    "environment_id": "aws1",
    "region_url": "http://169.254.169.254/latest/meta-data/placement/availability-zone",
    "url": "http://169.254.169.254/latest/meta-data/iam/security-credentials",
    "regional_cred_verification_url": "https://sts.{region}.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15",
    "imdsv2_session_token_url": "http://169.254.169.254/latest/api/token"
  }
}

Please let me know if that helps

5

Reducing concurrency helps, so I’m going to validate outstanding quota requests with GCP.
For IMDSv2, the change made CQ lock up post this log line:

{"level":"info","grpc.code":"OK","grpc.component":"server","grpc.method":"Init","grpc.method_type":"unary","grpc.service":"cloudquery.plugin.v3.Plugin","grpc.start_time":"2026-07-08T12:02:24Z","grpc.time_ms":"194.648","message":"finished call","module":"postgresql-destination","peer.address":"@","protocol":"grpc","time":"2026-07-08T12:02:24Z","time":"2026-07-08T12:02:24Z"}
{"level":"info","grpc.component":"server","grpc.method":"Init","grpc.method_type":"unary","grpc.service":"cloudquery.plugin.v3.Plugin","grpc.start_time":"2026-07-08T12:02:24Z","grpc.time_ms":"0.009","message":"started call","module":"bigquery-destination","peer.address":"@","protocol":"grpc","time":"2026-07-08T12:02:24Z","time":"2026-07-08T12:02:24Z"}

I had it sit there for ~half an hour, with no CPU load until I reverted the change.

6

Thanks @juergen_k for the additional information.

Let me check the IMDSv2 angle maybe it’s not supported in your setup.

7

Hi @erez did you by chance find any clues wrt the IMDSv2 angle?

8

Hi @juergen_k sorry for the delay.

IMDSv2 should be supported.

It’s mostly a guess but it hanging could be related to some IMDS settings like the hop limit.

There are various ways to modify it depending on how the container is configured.