Error in cloudquery config file project filter not recognized

CloudQuery Plugins
1

I’m trying to see what I did wrong here:

---
kind: source
spec:
  # Source spec section
  name: "gcp"
  path: "cloudquery/gcp"
  registry: "cloudquery"
  version: "v11.0.2"
  destinations:
    - bq
  tables:
    - gcp_projects
    - gcp_iam*
  project_filter: "NOT id:sys-*"

Errors with:

{"level":"error","module":"cli","error":"failed to load spec(s) from /config/config.yml. Error: failed to unmarshal file /config/config.yml: failed to decode spec: json: unknown field \"project_filter\"","time":"2024-02-28T00:10:37Z","message":"exiting with error"}

While the docs list it as valid here: https://hub.cloudquery.io/plugins/source/cloudquery/gcp/v11.0.2/docs#configuration-gcp-spec

2 
kind: source
spec:
  # Source spec section
  name: "gcp"
  path: "cloudquery/gcp"
  registry: "cloudquery"
  version: "v11.0.2"
  destinations:
    - bq
  tables:
    - gcp_projects
    - gcp_iam*
  plugin_spec:
    project_filter: "NOT id:sys-*"

Can you try :point_up: ? project_filter should be under the plugin spec not the top level one.

3


That fixed it, thanks for the quick win @erez
(I had to fix our alerts for this which took the longest)
Is there a plugin anywhere that I can use in CI to validate the sources/destinations as part of merge control?

✗ docker run -v $(pwd)/config:/config \
  ghcr.io/cloudquery/cloudquery:latest
CloudQuery CLI

Open source data integration at scale.

Find more information at:
        https://www.cloudquery.io

Usage:
  cloudquery [command]

Available Commands:
  addon       Addon commands
  login       Login to CloudQuery Hub.
  logout      Log out of CloudQuery Hub.
  migrate     Update schema of your destinations based on the latest changes in sources from your configuration
  plugin      Plugin commands
  switch      Switches between teams.
  sync        Sync resources from configured source plugins to destinations
  tables      Generate documentation for all supported tables of source plugins specified in the spec(s)

I don’t see anything immediately.

4

We have an open feature for it here.

You can run the following command:

migrate

It would act as a validation command, with the addition of creating/migrating schema. It will not run a sync.

5

The CI runner won’t (and shouldn’t) have the permissions to do that, so I’ll just watch that issue.