Help with BigQuery append write mode and data duplication issues

CloudQuery Plugins
1

Hi! Is anyone using BigQuery as a destination? If so, how are you dealing with the “append” write mode support only? Drop the tables somehow before sync? Or just accept the ever-growing duplication and size??

2

Hi! Which source plugin are you using?

3

This one: CloudQuery BigQuery Plugin

4

I mean which source plugin do you use to sync data from? The BigQuery is the destination plugin.

5

I’m using CloudQuery PostgreSQL Plugin Documentation.

6

I see. Yeah, BigQuery is not great with overwrite mode and rate limits, so we don’t support it yet. There is a related ticket here - GitHub Issue #14536

7

I’m not experienced in Go, otherwise I would be able to help out. Also, not experienced in BQ, so that makes it even harder haha.
I don’t fully understand how CDC works either lol.