Upgrade module includes schema change will it work with GCS to BigQuery

CloudQuery Plugins
1

Quick question

We are currently shipping all data to GCS and using BigQuery data transfers to pull that into BigQuery. That’s what determines the schema, etc.

If I want to upgrade a module and it includes a data level change (schema, etc.), will that “just work” with GCS → BQ?

2

Hi @pretty-wolf,

I’m not that familiar with BigQuery data transfers. What do you mean by “upgrade a module”? Do you mean updating to a newer version of a CloudQuery source plugin?

3

We also have a BigQuery destination plugin you could use to ship directly to BigQuery.

4

Yeah, I’m wondering if GCS to BQ behind the scenes was a bad move.

5

I know the BigQuery plugin handles some schema changes (those that can be auto-applied without data loss).

6

Is there a document I can check to understand how migrations work for source upgrades?

7

Looks like BigQuery does some schema migration depending on the file type in GCS here.

Our own migration docs are in this link.

On our end, the BigQuery code that merges schemas is here.