CloudQuery Stripe plugin synchronizes only recent subscriptions needing time range adjustment

CloudQuery Plugins
1

Hey everyone,

I’m currently working with the Stripe plugin, attempting to synchronize all subscriptions stored in the stripe_subscriptions table of our PGSQL database.

However, during the synchronization process, we’re encountering an issue where only the most recent subscription is being scanned for some unknown reasons. I’m curious to find out if there’s a predefined time range set within the plugin for scanning purposes. Perhaps it’s configured to only scan items created within the last two weeks or something similar?

Additionally, I’m wondering if there’s a way to override or adjust this time range setting within the plugin to ensure that all subscriptions are properly synchronized. Any insights or suggestions would be greatly appreciated! :grinning:

2

@wired-chimp - Could you share a redacted copy of your config and any logs so we can better triage the issue?

3

Sure!

Here is the source file:

kind: source
spec:
  name: stripe-subscriptions
  path: cloudquery/stripe
  registry: "cloudquery"
  version: "v3.2.6"
  tables:
    - stripe_subscriptions
  destinations:
    - "postgresql"
  backend_options:
    table_name: "cq_state_stripe"
    connection: "@@plugins.postgresql.connection"
  # Stripe specific configuration
  spec:
    api_key: "${STRIPE_SK}"

Find here the destination one:

kind: destination
spec:
  name: "postgresql"
  path: "cloudquery/postgresql"
  version: "v8.0.0"
  write_mode: "append"
  migrate_mode: forced
  spec:
    connection_string: "${PG_CONNECTION_STRING}"

The logs are displaying: "Sync completed successfully. Resources: 1, Errors: 0, Warnings: 0, Time: 2s"

But the thing is that we’ve, of course, more than 1 subscription. We’ve also tried different write modes, but the results are still the same, just one subscription synced.

4

@wired-chimp - By setting

backend_options:
  table_name: "cq_state_stripe"
  connection: "@@plugins.postgresql.connection"

in your source configuration, you are enabling incremental syncing. This means that CloudQuery stores the state in the table named cq_state_stripe in your PostgreSQL database and only grabs new records. If you want to do a complete sync, you can either truncate the cq_state_stripe table or delete the backend_options from your config.

More information about incremental tables can be found here: CloudQuery Documentation on Managing Incremental Tables

5

wow! true, thanks a lot! Really appreciated your help :slightly_smiling_face:

6

Glad we could get it resolved quickly!
@wired-chimp - We recently became aware of an issue with regards to using write_mode: append with incremental tables. While we work to release a fix for it, I would suggest either not using incremental tables or using the default write_mode.
I can ping you when a fix is released (with the version number) sometime this week.

7

Ok, thanks for the information!