Confusion about sync_state table creation in CloudQuery PostgreSQL integration

CloudQuery Plugins
1

Hi,

I’m trying out CloudQuery for the first time. So far, so good - I only tried the Shopify/PostgreSQL integration, since I have a handy test store. It works really well.

However, I’m a bit confused by sync_state, even after reading the docs twice.

kind: source
# Common source-plugin configuration
spec:
  name: shopify
  path: cloudquery/shopify
  version: "v3.1.1"
  tables: ["*"]
  destinations: ["postgresql"]
  # Shopify specific configuration
  spec:
    backend_options:
      table_name: "shopify_postgres_state"
      connection: "@@plugins.postgresql.connection"

My assumption was that this will create a table shopify_postgres_state in my PostgreSQL DB, but I don’t see it anywhere. So when I’m rerunning sync, it’s going through the same items again.

Did I mess something up, or simply misunderstand how it works?

2

Hmm, I could be wrong, but I think this is a mistake in our Shopify plugin documentation. The backend_options setting needs to move one level up.

Can you try with the following config?

kind: source
# Common source-plugin configuration
spec:
  name: shopify
  path: cloudquery/shopify
  version: "v3.1.1"
  tables: ["*"]
  destinations: ["postgresql"]
  backend_options:
    table_name: "shopify_postgres_state"
    connection: "@@plugins.postgresql.connection"
  # Shopify specific configuration
  spec:
3

Thanks. Will try that right now.
You are correct. The table was just created. Will need to wait until the sync, and will report back in 30 mins.
Excellent.
That was it.