# Rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error **URL:** \ **Category:** General\ **Created:** [July 7, 2026, 12:11pm UTC](https://community.cloudquery.io/t/rpc-error-code-unauthenticated-desc-transport-per-rpc-creds-failed-due-to-error/1428 "2026-07-07T12:11:25Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![juergen\_k](https://avatars.discourse-cdn.com/v4/letter/j/f17d59/32.png) [@juergen\_k](https://community.cloudquery.io/u/juergen_k)\ **Post date:** [July 7, 2026, 12:11pm UTC](https://community.cloudquery.io/t/rpc-error-code-unauthenticated-desc-transport-per-rpc-creds-failed-due-to-error/1428/1 "2026-07-07T12:11:25Z")
This has been working fine for some time, not sure when exactly it started failing. Using CQ on AWS infrastructure with a GCP service account; in this particular case, the source plugin/table is GCP’s IAM predefined roles. I am getting the following error constantly: `{"level":"info","grpc.code":"Unauthenticated","grpc.component":"client","grpc.error":"rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: Get "``http://169.254.169.254/latest/meta-data/iam/security-credentials/{obfuscated}\``": EOF","grpc.method":"ListRoles","grpc.method_type":"unary","grpc.request.deadline":"2026-07-07T11:57:01Z","grpc.service":"google.iam.admin.v1.IAM","grpc.start_time":"2026-07-07T11:56:01Z","grpc.time_ms":"4566.946","invocation_id":"{obfuscated}","message":"finished call","module":"gcp-source","peer.address":"@","protocol":"grpc","time":"2026-07-07T11:56:06Z","time":"2026-07-07T11:56:06Z"}` When doing the same call with curl from the same pod, it works fine: ```auto GET /latest/meta-data/iam/security-credentials/{obfuscated} HTTP/1.1 Host: 169.254.169.254 User-Agent: curl/8.21.0 Accept: / Request completely sent off < HTTP/1.1 200 OK < Content-Type: text/plain < Accept-Ranges: none < Last-Modified: Tue, 07 Jul 2026 11:57:30 GMT < Content-Length: 1582 < Date: Tue, 07 Jul 2026 11:58:55 GMT < Server: EC2ws < Connection: close < { "Code" : "Success", "LastUpdated" : "2026-07-07T11:57:18Z", "Type" : "AWS-HMAC", "AccessKeyId" : "{obfuscated}", "SecretAccessKey" : "{obfuscated}", "Token" : "{obfuscated}", "Expiration" : "2026-07-07T18:07:44Z" } ``` All source plugins and CLI have been upgraded to the latest version. Thanks! J. ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [July 7, 2026, 4:54pm UTC](https://community.cloudquery.io/t/rpc-error-code-unauthenticated-desc-transport-per-rpc-creds-failed-due-to-error/1428/2 "2026-07-07T16:54:59Z")
Hi @juergen_k do you mind sharing the full sync configuration redacted from any personal information? Also please share the sync logs if possible, specifically if the sync failed outright at the beginning, or did it finish, but with some errors. Finally, could you explain how you’re configuring the GCP service account? Via a service account JSON env variable (or file)? Or a different set up? Thanks, Erez ---
**Author:** ![juergen\_k](https://avatars.discourse-cdn.com/v4/letter/j/f17d59/32.png) [@juergen\_k](https://community.cloudquery.io/u/juergen_k)\ **Post date:** [July 8, 2026, 7:18am UTC](https://community.cloudquery.io/t/rpc-error-code-unauthenticated-desc-transport-per-rpc-creds-failed-due-to-error/1428/3 "2026-07-08T07:18:06Z")
@erez I cannot share the full logs because there’s too much to redact here - but maybe it’ll help when I explain what I see; otherwise I can probably provide them to you directly via E-Mail. I’m wondering if this might be related to GCP APIs rate limiting us (e.g. for quota reasons) since we are querying data for ~4400 accounts. What I can say is that other tables work fine with the same configuration. `apiVersion: v1` `data:` `config-gcp.yaml: |` `kind: source` `spec:` `name: "gcp"` `path: "cloudquery/gcp"` `version: "v23.0.0"` `tables: ["${CLOUDQUERY_GCP_TABLE}"]` `skip_tables: ["gcp_dataproc_regions","gcp_dataproc_autoscaling_policies"]` `destinations: ["postgresql", "bigquery"]` `backend_options:` `table_name: "cq_gcp_state"` `connection: "@@plugins.postgresql.connection"` `kind: ConfigMap` We’re using a JSON env file: `apiVersion: v1` `data:` `service-account-gcp.json: |-` `{` `"type": "external_account", (...)` The logs start out clean, let me try to copy and paste the most notable entries: `(...)` `{"level":"info","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"Using service account for GCP authentication","module":"gcp-source","service_account":"application-default-credentials","time":"2026-07-08T07:06:18Z","time":"2026-07-08T07:06:18Z"}` `{"level":"info","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"No organization_ids or organization_filter specified - assuming all organizations","module":"gcp-source","time":"2026-07-08T07:06:18Z","time":"2026-07-08T07:06:18Z"}` `{"level":"info","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"Listing organizations...","module":"gcp-source","time":"2026-07-08T07:06:18Z","time":"2026-07-08T07:06:18Z"}` `{"level":"info","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"No project_ids, folder_ids, or project_filter specified - assuming all active projects","module":"gcp-source","time":"2026-07-08T07:06:18Z","time":"2026-07-08T07:06:18Z"}` `{"level":"info","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"Found 4384 projects. First 100 ` `(...)` `{"level":"info","client":"project:{1}","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"top level table resolver started","module":"gcp-source","table":"gcp_iam_predefined_roles","time":"2026-07-08T07:07:17Z","time":"2026-07-08T07:07:17Z"}` `{"level":"info","client":"project:{2}","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"top level table resolver started","module":"gcp-source","table":"gcp_iam_predefined_roles","time":"2026-07-08T07:07:17Z","time":"2026-07-08T07:07:17Z"}` `(...)` `{"level":"info","client":"project:{100}","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"top level table resolver started","module":"gcp-source","table":"gcp_iam_predefined_roles","time":"2026-07-08T07:07:17Z","time":"2026-07-08T07:07:17Z"}` `{"level":"info","grpc.code":"Unauthenticated","grpc.component":"client","grpc.error":"rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: Get "``http://169.254.169.254/latest/meta-data/iam/security-credentials\``": EOF","grpc.method":"ListRoles","grpc.method_type":"unary","grpc.request.deadline":"2026-07-08T07:08:17Z","grpc.service":"google.iam.admin.v1.IAM","grpc.start_time":"2026-07-08T07:07:17Z","grpc.time_ms":"1047.618","invocation_id":"6dc1a99c-8947-4911-baeb-0bd9d64dff34","message":"finished call","module":"gcp-source","peer.address":"@","protocol":"grpc","time":"2026-07-08T07:07:18Z","time":"2026-07-08T07:07:18Z"}` and that’s where we start seeing tons of errors and the sync ultimately fails. ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [July 8, 2026, 8:21am UTC](https://community.cloudquery.io/t/rpc-error-code-unauthenticated-desc-transport-per-rpc-creds-failed-due-to-error/1428/4 "2026-07-08T08:21:04Z")
Hi @juergen_k based on the logs what seems to be the error is the credentials negotiation from AWS side `Unauthenticated desc = transport: per-RPC creds failed due to error: Get "``http://169.254.169.254/latest/meta-data/iam/security-credentials\``": EOF","grpc.method":"ListRoles","grpc.method_type":"unary","grpc.request.deadline"` `Agree this is probably related to GCP plugin doing many concurrent requests.` `To solve it we can try two options:` 1. `Reduce the plugin concurrency (see example here ` [GCP Source Plugin | CloudQuery](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs#configuration-example) ). I would try a low value to debug first, e.g. `concurrency: 100` then raise until it’s stable. 2. Make sure you use `IMDSv2` and not `IMDSv1` with AWS. `IMDSv2` has better credentials caching. Examples of the differences below: `IMDSv1` ```json { "type": "external_account", "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/providers/aws-provider", "subject_token_type": "urn:ietf:params:aws:token-type:aws4_request", "token_url": "https://sts.googleapis.com/v1/token", "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/my-sa@my-project.iam.gserviceaccount.com:generateAccessToken", "credential_source": { "environment_id": "aws1", "region_url": "http://169.254.169.254/latest/meta-data/placement/availability-zone", "url": "http://169.254.169.254/latest/meta-data/iam/security-credentials", "regional_cred_verification_url": "https://sts.{region}.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15" } } ``` `IMDSv2`(additional `imdsv2_session_token_url` under `credential_source`) ```json { "type": "external_account", "audience": "//iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/aws-pool/providers/aws-provider", "subject_token_type": "urn:ietf:params:aws:token-type:aws4_request", "token_url": "https://sts.googleapis.com/v1/token", "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/my-sa@my-project.iam.gserviceaccount.com:generateAccessToken", "credential_source": { "environment_id": "aws1", "region_url": "http://169.254.169.254/latest/meta-data/placement/availability-zone", "url": "http://169.254.169.254/latest/meta-data/iam/security-credentials", "regional_cred_verification_url": "https://sts.{region}.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15", "imdsv2_session_token_url": "http://169.254.169.254/latest/api/token" } } ``` Please let me know if that helps ---
**Author:** ![juergen\_k](https://avatars.discourse-cdn.com/v4/letter/j/f17d59/32.png) [@juergen\_k](https://community.cloudquery.io/u/juergen_k)\ **Post date:** [July 8, 2026, 12:41pm UTC](https://community.cloudquery.io/t/rpc-error-code-unauthenticated-desc-transport-per-rpc-creds-failed-due-to-error/1428/5 "2026-07-08T12:41:44Z")
Reducing concurrency helps, so I’m going to validate outstanding quota requests with GCP. For IMDSv2, the change made CQ lock up post this log line: `{"level":"info","grpc.code":"OK","grpc.component":"server","grpc.method":"Init","grpc.method_type":"unary","grpc.service":"cloudquery.plugin.v3.Plugin","grpc.start_time":"2026-07-08T12:02:24Z","grpc.time_ms":"194.648","message":"finished call","module":"postgresql-destination","peer.address":"@","protocol":"grpc","time":"2026-07-08T12:02:24Z","time":"2026-07-08T12:02:24Z"}` `{"level":"info","grpc.component":"server","grpc.method":"Init","grpc.method_type":"unary","grpc.service":"cloudquery.plugin.v3.Plugin","grpc.start_time":"2026-07-08T12:02:24Z","grpc.time_ms":"0.009","message":"started call","module":"bigquery-destination","peer.address":"@","protocol":"grpc","time":"2026-07-08T12:02:24Z","time":"2026-07-08T12:02:24Z"}` I had it sit there for ~half an hour, with no CPU load until I reverted the change. ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [July 8, 2026, 12:50pm UTC](https://community.cloudquery.io/t/rpc-error-code-unauthenticated-desc-transport-per-rpc-creds-failed-due-to-error/1428/6 "2026-07-08T12:50:29Z")
Thanks @juergen_k for the additional information. Let me check the IMDSv2 angle maybe it’s not supported in your setup. ---
**Author:** ![juergen\_k](https://avatars.discourse-cdn.com/v4/letter/j/f17d59/32.png) [@juergen\_k](https://community.cloudquery.io/u/juergen_k)\ **Post date:** [July 15, 2026, 11:09am UTC](https://community.cloudquery.io/t/rpc-error-code-unauthenticated-desc-transport-per-rpc-creds-failed-due-to-error/1428/7 "2026-07-15T11:09:53Z")
Hi @erez did you by chance find any clues wrt the IMDSv2 angle? ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [July 15, 2026, 2:19pm UTC](https://community.cloudquery.io/t/rpc-error-code-unauthenticated-desc-transport-per-rpc-creds-failed-due-to-error/1428/8 "2026-07-15T14:19:12Z")
Hi @juergen_k sorry for the delay. IMDSv2 should be supported. It’s mostly a guess but it hanging could be related to some IMDS settings like the hop limit. > **[Configure the Instance Metadata Service options - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-options.html)** > > Configure instance metadata options to control the accessibility and behavior of the IMDS on an EC2 instance. There are various ways to modify it depending on how the container is configured.