Errors configuring GCP plugin for CloudQuery on EC2 instance

Errors While Configuring GCP Source Plugin

Hey there! I’m trying to configure the GCP plugin to read data from my GCP organization. I’m using CloudQuery from an EC2 instance on AWS. This is my configuration so far:

spec:
  # Source spec section
  name: "rcp-gcp"
  path: "cloudquery/gcp"
  registry: "cloudquery"
  version: "v11.5.0"
  tables: ["gcp_storage_buckets"]
  destinations: ["postgresql"]
  spec:
    # GCP Spec
    organization_ids: ["XXXXXX"]
    target_principal: "XXXXXXXX.iam.gserviceaccount.com"
    scopes: "https://www.googleapis.com/auth/devstorage.read_only"
---
kind: destination
spec:
  name: "postgresql"
  path: "cloudquery/postgresql"
  registry: "cloudquery"
  version: "v7.0.1"

  spec:
    connection_string: ${PG_CONNECTION_STRING_GCP}
    pgx_log_level: error

I’ve created a service account and granted my user permissions to impersonate it. Then I’ve installed gcloud on my EC2 instance and logged in using my user, but I get this error:

Error: failed to sync v3 source rcp-gcp: rpc error: code = Internal desc = failed to init plugin: failed to initialize client: failed to create organizations client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information

Any idea of what may be wrong?

Hi, did you go through the auth section of the doc at https://hub.cloudquery.io/plugins/source/cloudquery/gcp/v11.5.0/docs#overview-authentication? I’m guessing you’re using the gcloud auth application-default login method?

Ups, I forgot the application-default.
I was using the standard gcloud auth login method.
I’m facing other issues now, but that worked.

Glad that worked! Let us know if you need anything else.

Sure! Just in case anyone faces the same issue, the Cloud Resource Manager API has to be enabled. I just wanted to let you know it worked fine.

It seems like viewer permissions are not enough, and you need browser permissions as well. We started with gathering info from buckets, but there was one specific permission missing.

Is there any place where you point out the list of permissions required per table? I didn’t see anything in the official documentation about all this.

Thank you for your support, @kemal!

I don’t think we have such a list; I will check.
Checked, looks like we currently don’t provide that information.

ok, thanks for the confirmation :ok_hand:

I’ve opened an internal issue to consider this in the future though :slightly_smiling_face: