Error with table gcp_iam_deny_policies due to quota exceeded issues

CloudQuery Plugins
1

Overall, most of my stuff is going great, but this table seems to error more than any other. Any tips on how to slow it down or optimize it?

{
  "level": "error",
  "module": "gcp-src",
  "client": "project:redact-infra-cost",
  "error": "rpc error: code = ResourceExhausted desc = Quota exceeded for quota metric 'Policy Read Requests' and limit 'Policy Read Requests per minute' of service 'iam.googleapis.com' for consumer 'project_number:redact'.\nerror details: name = ErrorInfo reason = RATE_LIMIT_EXCEEDED domain = googleapis.com metadata = map[consumer:projects/redact quota_limit:ReadPolicyRequestsPerMinutePerCredentialProject quota_limit_value:10 quota_location:global quota_metric:iam.googleapis.com/read_policies/per_credential service:iam.googleapis.com]\nerror details: name = Help desc = Request a higher quota limit. url = https://cloud.google.com/docs/quota#requesting_higher_quota",
  "message": "table resolver finished with error",
  "table": "gcp_iam_deny_policies",
  "time": "2024-03-01T00:00:17Z"
}

My config is:

---
kind: source
spec:
  # Source spec section
  name: "gcp"
  path: "cloudquery/gcp"
  registry: "cloudquery"
  version: "v11.0.2"
  destinations:
    - bq
  tables:
    - gcp_projects
    - gcp_iam*
  spec:
    project_filter: "NOT id:sys-*"

For context, without the project_filter, our GCP is pretty thicc:

➜  ~ gcloud projects list | wc -l
13732
2

I would try reducing the concurrency from the default of 50,000 to something like 500:

---
kind: source
spec:
  # Source spec section
  name: "gcp"
  path: "cloudquery/gcp"
  registry: "cloudquery"
  version: "v11.0.2"
  destinations:
    - bq
  tables:
    - gcp_projects
    - gcp_iam*
  spec:
    project_filter: "NOT id:sys-*"
    concurrency: 500

The concurrency is the value that defines the upper bound of tables being resolved in parallel, so by lowering that value you are less likely to hit the rate limit.

3

Ok, neat call! I’ll try that. I don’t care if it takes long to run; our SLA right now is daily.

4

Additionally, as outlined in the docs, I suggest to play with backoff_delay and backoff_retries settings (note that by default the backoff_retries is 0).

5

I might need to tweak that also

~ k -n cloudquery logs pod/cloudquery-28491840-q7cx9 | jq '.error' | sed 's/.*Quota exceeded for/Quota exceeded for/g' | sed 's/of service .*//g' | sort | uniq

still has 229 errors of varying

Quota exceeded for quota metric 'Policy Read Requests' and limit 'Policy Read Requests per minute'
Quota exceeded for quota metric 'Policy Read Requests' and limit 'Policy Read Requests per minute per user'