Understanding CloudQuery incremental plugin and cq_state_aws table functionality

CloudQuery Plugins
41 
kind: source
spec:
  # Source spec section
  name: aws_xxxxxx
  path: cloudquery/aws
  version: "v24.1.0"
  tables: ["aws_inspector2_findings"]
  destinations: ["postgresql"]
  backend_options:
    table_name: "cq_state_aws"
    connection: "@@plugins.postgresql.connection"


kind: destination
spec:
  name: postgresql
  path: "cloudquery/postgresql"
  version: "v7.0.0"
  write_mode: "overwrite-delete-stale"
  migrate_mode: "forced"
  spec:
    connection_string: "xxxxxxxx"

CLOUDQUERY CLI Version - 5.20.1

42

So the issue I linked is unrelated as the error you’re getting is specific to the state client (incremental syncs).

Please open a bug report via GitHub. It will guide you to provide the necessary information for us to reproduce the issue on our end.

43

I’ve created this issue: cloudquery/cloudquery#18039. Let me know if you need anything else.

44

Can you please share the OS version you’re using? e.g. Windows version, and state you’re using a bash terminal.
So this might be a Windows specific issue, but with incremental tables. I recommend testing on Linux/Windows if possible, or running inside a Docker container.

45

Hi @erez,

I did try with one of the new AWS CLI versions (5.20.0) to enable CloudQuery incremental sync for the Inspector table. However, I’m still facing an issue. The command

CLOUDQUERY_API_KEY=key cloudquery sync source.yml destination.yml

is an invalid command.

46

Hi @funny-whale, if you’re on Windows, you’d need to use cloudquery.exe.

47

I’m using this in a step function and it is a Linux environment.

48

How are you installing the CloudQuery CLI? Does running

./cloudquery

print anything?