# Understanding CloudQuery incremental plugin and cq\_state\_aws table functionality **URL:** \ **Category:** CloudQuery Plugins\ **Created:** [May 16, 2024, 11:45am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208 "2024-05-16T11:45:20Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![concise-oryx](https://avatars.discourse-cdn.com/v4/letter/c/67e7ee/32.png) [@concise-oryx](https://community.cloudquery.io/u/concise-oryx)\ **Post date:** [May 16, 2024, 11:45am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/1 "2024-05-16T11:45:20Z")
How does the incremental plugin actually work? What does the **cq\_state\_aws** table do? Does it store the data when we try to do an incremental sync? ---
**Author:** ![kemal](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/kemal/32/42_2.png) [@kemal](https://community.cloudquery.io/u/kemal)\ **Post date:** [May 16, 2024, 11:57am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/2 "2024-05-16T11:57:48Z")
Yes, it stores a state value for each incremental table. How it actually works is documented in [the CloudQuery documentation](https://docs.cloudquery.io/docs/advanced-topics/managing-incremental-tables). ---
**Author:** ![concise-oryx](https://avatars.discourse-cdn.com/v4/letter/c/67e7ee/32.png) [@concise-oryx](https://community.cloudquery.io/u/concise-oryx)\ **Post date:** [May 16, 2024, 12:06pm UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/3 "2024-05-16T12:06:43Z")
We are actually trying to have CloudQuery syncs based on the different environments. We have around 9 environments. I have tried setting up the incremental sync for the `aws_inspector2_findings` table, and the issue here is it is taking around 15 hours to complete the sync, and this environment has just 132 accounts. I also see that there is nothing stored in the `cq_aws_state` table. I have read the documentation but am not sure how the sync is going to happen. In my case, I am running the incremental sync for different environments with a 2-hour gap. How is the incremental sync going to happen in my case? ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [May 16, 2024, 12:34pm UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/4 "2024-05-16T12:34:35Z")
Hi @concise-oryx, can you share the configuration files you’re using for the sync? ---
**Author:** ![concise-oryx](https://avatars.discourse-cdn.com/v4/letter/c/67e7ee/32.png) [@concise-oryx](https://community.cloudquery.io/u/concise-oryx)\ **Post date:** [May 16, 2024, 12:49pm UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/5 "2024-05-16T12:49:38Z")
```auto ***source.yml*** kind: source spec: name: "AWS_ACCOUNT_ID" path: "cloudquery/aws" version: "23.1.0" tables: ["*"] destinations: ["postgresql"] backend_options: table_name: "cq_state_aws" connection: "@@plugins.postgresql.connection" spec: accounts: - id: "xxxxxxxxx" role_arn: "arn:aws:iam::xxxxxxxxx:role/test-role" ``` ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [May 16, 2024, 12:59pm UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/6 "2024-05-16T12:59:55Z")
OK, so I recommend at least skipping the tables in [CloudQuery AWS Configuration](https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs#configuration-skip-tables) when using `tables: ["*"]` as they contain a lot of static data and make the sync slow. ---
**Author:** ![kemal](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/kemal/32/42_2.png) [@kemal](https://community.cloudquery.io/u/kemal)\ **Post date:** [May 16, 2024, 1:39pm UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/7 "2024-05-16T13:39:38Z")
Are you using multiple `source` sections to fetch multiple AWS accounts? ~~If so, each one should have its own `table_name` backend option; otherwise, they will clash~~. I checked the code and looking at how it was implemented, they won’t clash for the AWS plugin. You need at least the `v24.1.0` version of the AWS plugin to utilize incremental table functionality for `aws_inspector2_findings`. `v23.x` or `24.0.0` doesn’t have that. ---
**Author:** ![concise-oryx](https://avatars.discourse-cdn.com/v4/letter/c/67e7ee/32.png) [@concise-oryx](https://community.cloudquery.io/u/concise-oryx)\ **Post date:** [May 17, 2024, 6:04am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/8 "2024-05-17T06:04:17Z")
I am already skipping around 80 AWS tables which are not needed for us and that which are recommended in the documentation. Okay… I will try it and let you know the output. Sorry… I didn’t get it… All the source plugins are AWS itself. I’m using this like… First, the sync is done for all the tables except the Inspector table, and later the incremental sync is done for that account. This process repeats for all the other accounts. ---
**Author:** ![kemal](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/kemal/32/42_2.png) [@kemal](https://community.cloudquery.io/u/kemal)\ **Post date:** [May 17, 2024, 8:54am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/9 "2024-05-17T08:54:13Z")
Interesting, you’re doing this to measure how long `aws_inspector2_findings` takes, to be able to get to the rest of the data early? ---
**Author:** ![concise-oryx](https://avatars.discourse-cdn.com/v4/letter/c/67e7ee/32.png) [@concise-oryx](https://community.cloudquery.io/u/concise-oryx)\ **Post date:** [May 17, 2024, 8:55am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/10 "2024-05-17T08:55:40Z")
The rest of the data sync can be done in 30 min or so, but when I have added the incremental sync, it is taking around 15 to 20 hours to complete the sync. I’ll first try with `24.1.0` and get back to you. Hi @kemal, @erez, The `aws_inspector2_findings` is a premium table for `v24.1.0`. How can we perform a CloudQuery sync as we are using an open source version? ---
**Author:** ![kemal](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/kemal/32/42_2.png) [@kemal](https://community.cloudquery.io/u/kemal)\ **Post date:** [May 17, 2024, 9:50am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/11 "2024-05-17T09:50:11Z")
You still get some amount of free rows per month, which resets every month, so it should be possible to get _some_ benefit without paying if you manage the `tables` option carefully. Also please refer to the blog post at [CloudQuery Official Free Plugins Moving to Paid](https://www.cloudquery.io/blog/cloudquery-official-free-plugins-moving-to-paid) about the announcement. You could even mix and match AWS plugin versions to further reduce your quota usage, I believe. ---
**Author:** ![concise-oryx](https://avatars.discourse-cdn.com/v4/letter/c/67e7ee/32.png) [@concise-oryx](https://community.cloudquery.io/u/concise-oryx)\ **Post date:** [May 20, 2024, 4:27am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/12 "2024-05-20T04:27:07Z")
I think this needs a CloudQuery login to use AWS Plugin version 24.1.0. Will it work even without providing a CloudQuery login? ---
**Author:** ![kemal](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/kemal/32/42_2.png) [@kemal](https://community.cloudquery.io/u/kemal)\ **Post date:** [May 20, 2024, 8:33am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/13 "2024-05-20T08:33:45Z")
Login is required in newer versions so we can track free usage quota for everybody. ---
**Author:** ![concise-oryx](https://avatars.discourse-cdn.com/v4/letter/c/67e7ee/32.png) [@concise-oryx](https://community.cloudquery.io/u/concise-oryx)\ **Post date:** [May 20, 2024, 8:58am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/14 "2024-05-20T08:58:31Z")
From which AWS plugin version onwards is the CloudQuery login needed? ---
**Author:** ![kemal](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/kemal/32/42_2.png) [@kemal](https://community.cloudquery.io/u/kemal)\ **Post date:** [May 20, 2024, 9:15am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/15 "2024-05-20T09:15:47Z")
Here’s the announcement about that: [Mandatory Login](https://www.cloudquery.io/blog/mandatory-login) ---
**Author:** ![concise-oryx](https://avatars.discourse-cdn.com/v4/letter/c/67e7ee/32.png) [@concise-oryx](https://community.cloudquery.io/u/concise-oryx)\ **Post date:** [May 20, 2024, 9:25am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/16 "2024-05-20T09:25:07Z")
This AWS Plugin _ **24.1.0** _ version was released on February 7th, and the announcement was made on January 17th. As a result, I can’t perform a sync without CloudQuery login. Is there any other solution that you can provide so that we can perform an incremental sync for the `inspector2_findings` table? ---
**Author:** ![kemal](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/kemal/32/42_2.png) [@kemal](https://community.cloudquery.io/u/kemal)\ **Post date:** [May 20, 2024, 9:57am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/17 "2024-05-20T09:57:20Z")
You can also create an API key and use that instead of the login. ---
**Author:** ![concise-oryx](https://avatars.discourse-cdn.com/v4/letter/c/67e7ee/32.png) [@concise-oryx](https://community.cloudquery.io/u/concise-oryx)\ **Post date:** [May 21, 2024, 4:27am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/18 "2024-05-21T04:27:43Z")
Can we run a CloudQuery sync (excluding Inspector tables) for some group of accounts, and after completing the sync, can we run the incremental sync for all these accounts only for Inspector? ---
**Author:** ![kemal](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/kemal/32/42_2.png) [@kemal](https://community.cloudquery.io/u/kemal)\ **Post date:** [May 21, 2024, 8:03am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/19 "2024-05-21T08:03:40Z")
You can. But if you’re using the `write_mode: overwrite-delete-stale`, it will delete all rows that weren’t synced in the same source `name`. So you’ll need different `name` fields in each source config. No issue if you’re using `write_mode: append` (or just `overwrite`). ---
**Author:** ![concise-oryx](https://avatars.discourse-cdn.com/v4/letter/c/67e7ee/32.png) [@concise-oryx](https://community.cloudquery.io/u/concise-oryx)\ **Post date:** [May 21, 2024, 10:02am UTC](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208/20 "2024-05-21T10:02:00Z")
For the sync that is excluding inspector, it is set to mode “overwrite-delete-stale”, and for the incremental sync to work, we need either “overwrite-delete-stale” or “overwrite”. How can I perform the syncs together? [Next page](https://community.cloudquery.io/t/understanding-cloudquery-incremental-plugin-and-cq-state-aws-table-functionality/208.md?page=2)