How does incremental syncing work with CloudQuery from aws to postgres

CloudQuery Plugins
1

Hello, how does incremental syncing work? I’m syncing from source:aws to destination:postgres.

2

Incremental syncing uses a state table in the destination to store some type of state that the table can use as a starting point or checkpoint for syncing.

3

I should have mentioned that I did set that up.
I see the table is created but it’s always empty.

Migrating table module=pg-dest table=cq_state_aws
Table exists, auto-migrating module=pg-dest table=cq_state_aws
Connected to state backend module=aws-src table_name=cq_state_aws
4

Ahh, incremental syncs are only supported on specific tables (usually high volume APIs)… For the AWS plugin, I think the only table that supports incremental sync is aws_cloudtrail_events.

But let me verify that.

5

Are there any tables that you would like to see support incremental sync?

6

So I see an issue was raised, here: https://github.com/cloudquery/cloudquery/issues/7286.

But essentially, we are syncing data from a bunch of accounts across different organizations. Some accounts may contain many types of certain resources versus others. I guess the idea of using incremental sync could help with load on the database.

7

If you are looking for ways of reducing load on the DB or just improve performance in general, I would be happy to set up a call to discuss your workload and what we have seen work at other large AWS deployments.

Sent you a DM.