Adding cluster name metadata to k8s source data for gcs bigquery queries

CloudQuery Plugins
1

I’m testing out the k8s source. I’ve run it against ~4 different clusters (we have about 25 clusters). Is there a way to hydrate metadata like the cluster name into it? I send it to GCS, which uses transfer jobs to BigQuery, but I can’t distinguish the data from different clusters in my queries.

My config is:

---
kind: source
spec:
  name: k8s
  path: cloudquery/k8s
  registry: cloudquery
  version: "v5.2.6"
  tables: ["*"]
  destinations: ["gcs_k8s"]
---
kind: destination
spec:
  name: "gcs_k8s"
  path: "cloudquery/gcs"
  registry: "cloudquery"
  version: "v3.4.12"
  spec:
    bucket: "cloudquery"
    path: "k8s/${cluster_shortname}"
    format: "parquet"
    no_rotate: true
2

Hey :wave:.
One thing you could do is use the name as the cluster name and it will be available in a column called _cq_source_name.

3

Got a snippet I can steal?
I see what you mean, just not sure how to change it to your suggestion.

4 
---
kind: source
spec:
  name: your_cluster_name
  path: cloudquery/k8s
  registry: cloudquery
  version: "v5.2.6"
  tables: ["*"]
  destinations: ["gcs_k8s"]
---
kind: destination
spec:
  name: "gcs_k8s"
  path: "cloudquery/gcs"
  registry: "cloudquery"
  version: "v3.4.12"
  spec:
    bucket: "cloudquery"
    path: "k8s/${cluster_shortname}"
    format: "parquet"
    no_rotate: true
5

ah. that’s easy!
one sec
nvm lol

6

Thanks! That was super quick. Now that this is generally working, I’ll be able to write the design review next week and hopefully get it up and running for all 25 clusters. :tada:

7

Nice :partying_face: Keep us posted, curious to learn how it goes!