Stripe to s3 integration not updating state in sqlite

Hi team,

I developed a Stripe to S3 integration by using SQLite to save the state, and I am using the free version of this integration. However, it keeps writing the same 1 row of Stripe charge every day to S3 and doesn’t update the state. Can you please help me fix the below YAML files?

s3.yaml:

kind: destination
spec:
  name: "s3"
  path: "cloudquery/s3"
  version: "v4.10.0"
  write_mode: "append" 
  spec:
    bucket: "xxx-xxx-datalake"
    region: "eu-west-2"
    path: "raw_data/stripe_raw_data/{{TABLE}}/dt={{YEAR}}-{{MONTH}}-{{DAY}}/{{UUID}}.parquet"
    format: "parquet"
    athena: true

stripe.yaml:

kind: source
spec:
  name: "stripe"
  path: "cloudquery/stripe"
  version: "v2.1.11"
  registry: github
  tables: 
    - stripe_charges
  destinations: 
    - s3
  spec:
    api_key: ${STRIPE_TOKEN}
    rate_limit: 20
    incremental_sync: true
  backend_options:
    table_name: "cq_state_stripe"
    connection: "@@plugins.sqlite.connection"

sqlite.yaml:

kind: destination
spec:
  name: "sqlite"
  path: "cloudquery/sqlite"
  version: "v2.4.13"
  spec:
    connection_string: /cloudquery/state.sqlite

Thank you!

Are there records that are not being synced, or are you just seeing the same record always being synced?

In the initial run of the Glue batch job, it wrote all of the data. However, after that, in the following days, it writes only 1 same row of stripe charge every day instead of the charges data which hasn’t been synced yet.

OK, thanks for that information! Would you mind opening an issue on GitHub with the details of your setup so we can work to recreate the issue and fix it?

Open an issue on GitHub

I created the issue here: https://github.com/cloudquery/cloudquery/issues/18916
Thanks!