# CloudQuery sync time for AWS data in PG database on EKS **URL:** \ **Category:** CloudQuery Plugins\ **Created:** [October 20, 2023, 7:45am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647 "2023-10-20T07:45:54Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![unified-reptile](https://avatars.discourse-cdn.com/v4/letter/u/7feea3/32.png) [@unified-reptile](https://community.cloudquery.io/u/unified-reptile)\ **Post date:** [October 20, 2023, 7:45am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/1 "2023-10-20T07:45:54Z")
Hi, Can someone confirm how long CloudQuery takes to sync AWS data in a PG database? I am running it in EKS, and the cron has been running for 2 hours. ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [October 20, 2023, 10:07am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/2 "2023-10-20T10:07:50Z")
Hi 👋 Sorry, I thought I replied and forgot to send the message. ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [October 20, 2023, 10:08am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/3 "2023-10-20T10:08:05Z")
Duration of sync time ---
**Author:** ![unified-reptile](https://avatars.discourse-cdn.com/v4/letter/u/7feea3/32.png) [@unified-reptile](https://community.cloudquery.io/u/unified-reptile)\ **Post date:** [October 20, 2023, 10:08am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/4 "2023-10-20T10:08:26Z")
I am using `cli-v3.23.1` AWS plugin version `v22.15.0` PostgreSQL `v6.1.0` ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [October 20, 2023, 10:08am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/5 "2023-10-20T10:08:36Z")
The sync time depends on the amount of resources you have in your account, so it’s hard to predict in advance. ---
**Author:** ![unified-reptile](https://avatars.discourse-cdn.com/v4/letter/u/7feea3/32.png) [@unified-reptile](https://community.cloudquery.io/u/unified-reptile)\ **Post date:** [October 20, 2023, 10:08am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/6 "2023-10-20T10:08:55Z")
yes but it’s usually not this long ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [October 20, 2023, 10:08am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/7 "2023-10-20T10:08:56Z")
Can you share your configuration? Maybe we can optimize it. ---
**Author:** ![unified-reptile](https://avatars.discourse-cdn.com/v4/letter/u/7feea3/32.png) [@unified-reptile](https://community.cloudquery.io/u/unified-reptile)\ **Post date:** [October 20, 2023, 10:09am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/8 "2023-10-20T10:09:00Z")
```yaml its more than 4 hours kind: source spec: name: aws path: cloudquery/aws version: "v22.15.0" # latest version of aws plugin tables: ["*"] destinations: ["postgresql"] kind: destination spec: name: postgresql path: cloudquery/postgresql version: "v6.1.0" # latest version of postgresql plugin write_mode: "overwrite-delete-stale" spec: connection_string: ``` ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [October 20, 2023, 10:09am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/9 "2023-10-20T10:09:48Z")
Also see [Performance Tuning](https://www.cloudquery.io/docs/advanced-topics/performance-tuning) and [AWS Source Configuration](https://www.cloudquery.io/docs/plugins/sources/aws/configuration#skip-tables). And the source config? ---
**Author:** ![unified-reptile](https://avatars.discourse-cdn.com/v4/letter/u/7feea3/32.png) [@unified-reptile](https://community.cloudquery.io/u/unified-reptile)\ **Post date:** [October 20, 2023, 10:10am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/10 "2023-10-20T10:10:25Z")
```auto kind: source spec: name: aws path: cloudquery/aws version: "v22.15.0" # latest version of aws plugin tables: ["*"] destinations: ["postgresql"] ``` Also, we don’t want to skip any tables with the old version. It was working fine @erez. It’s just after the update I am seeing this. ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [October 20, 2023, 10:11am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/11 "2023-10-20T10:11:43Z")
So with each new version, we’re adding more tables and resources; that’s why newer versions might take longer. `tables: ["*"]` can be super slow as it syncs all tables, including static AWS data that doesn’t change often. If you’re using `tables: ["*"]`, it’s recommended to skip the tables in [this link](https://www.cloudquery.io/docs/plugins/sources/aws/configuration#skip-tables). Another option is to look at the diff from the previous version, then skip the tables that were added with the new version to get the same performance. But that will render the update less useful. ---
**Author:** ![unified-reptile](https://avatars.discourse-cdn.com/v4/letter/u/7feea3/32.png) [@unified-reptile](https://community.cloudquery.io/u/unified-reptile)\ **Post date:** [October 20, 2023, 10:24am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/12 "2023-10-20T10:24:02Z")
Yes, So any idea how long it takes with the new version? @erez New tables might be useful for us to get more data. @erez Can you also confirm if syncing a resource is incremental or if it starts syncing all from scratch? ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [October 20, 2023, 11:10am UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/13 "2023-10-20T11:10:35Z")
It’s hard to predict the duration as it’s also based on your connection speed and the machine you’re using to sync. Incremental syncs are supported for a few tables, and you’d have to enable it manually. See [CloudQuery Incremental Tables](https://www.cloudquery.io/docs/plugins/sources/aws/configuration#incremental-tables) for more information. ---
**Author:** ![unified-reptile](https://avatars.discourse-cdn.com/v4/letter/u/7feea3/32.png) [@unified-reptile](https://community.cloudquery.io/u/unified-reptile)\ **Post date:** [October 20, 2023, 12:06pm UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/14 "2023-10-20T12:06:24Z")
Earlier it used to be around 20 minutes, but now it’s more than 4 hours. Even the cron job terminates and creates a new pod. @erez, is there any new version I should use? Are you aware of any bugs in the versions I provided? ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [October 20, 2023, 12:07pm UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/15 "2023-10-20T12:07:45Z")
Can you share the previous version you were using? We’re not aware of any bugs. `v22.15.0` is a recent version. It’s doing more work now as we’ve added some tables that require quite a bit of time (listed in the doc I shared). We’ve also changed the default of `tables`. It used to be `*`, and we changed it as it’s not a good default for most people since sync time grows with new versions and it syncs data that most people don’t need. Which version are you upgrading from? Have you tried skipping the tables I shared? Most likely you were not syncing them before. ---
**Author:** ![unified-reptile](https://avatars.discourse-cdn.com/v4/letter/u/7feea3/32.png) [@unified-reptile](https://community.cloudquery.io/u/unified-reptile)\ **Post date:** [October 20, 2023, 12:11pm UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/16 "2023-10-20T12:11:56Z")
I removed the old version; it was 2.x. When I started the first sync with the new version, I was getting some migration errors. So, I deleted all tables and let CloudQuery create them again. ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [October 20, 2023, 12:13pm UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/17 "2023-10-20T12:13:32Z")
Do you mean the AWS plugin major version was `v2`? Usually, switching between major versions requires dropping some tables. Each major version has a breaking change that’s detailed in the changelog. ---
**Author:** ![unified-reptile](https://avatars.discourse-cdn.com/v4/letter/u/7feea3/32.png) [@unified-reptile](https://community.cloudquery.io/u/unified-reptile)\ **Post date:** [October 20, 2023, 12:17pm UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/18 "2023-10-20T12:17:27Z")
@erez I am seeing the below log: ```auto 2023-10-20T12:08:28Z INF table sync finished client=:us-east-1 errors=0 module=aws-src resources=0 table=aws_route53_hosted_zone_query_logging_configs 2023-10-20T12:08:28Z INF table sync finished client=::us-east-1 errors=0 module=aws-src resources=5052 table=aws_route53_hosted_zone_resource_record_sets 2023-10-20T12:08:28Z INF table sync finished client=::us-east-1 errors=0 module=aws-src resources=0 table=aws_route53_hosted_zone_traffic_policy_instances ``` After that, there are no logs in debug mode. It looks like it just got stuck. ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [October 20, 2023, 12:22pm UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/19 "2023-10-20T12:22:36Z")
That might also signal a memory consumption issue. Can you either try skipping some tables from [CloudQuery Documentation - Skip Tables](https://www.cloudquery.io/docs/plugins/sources/aws/configuration#skip-tables) or lowering concurrency settings per [CloudQuery Documentation - Tune Concurrency](https://www.cloudquery.io/docs/advanced-topics/performance-tuning#tune-concurrency)? Again, we highly recommend not to use ```auto tables: ["*"] ``` in a single sync, especially for AWS. What most users do is split the configuration into multiple sync jobs. ---
**Author:** ![unified-reptile](https://avatars.discourse-cdn.com/v4/letter/u/7feea3/32.png) [@unified-reptile](https://community.cloudquery.io/u/unified-reptile)\ **Post date:** [October 20, 2023, 12:28pm UTC](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647/20 "2023-10-20T12:28:44Z")
@erez, you mean defining tables in a different source file? [Next page](https://community.cloudquery.io/t/cloudquery-sync-time-for-aws-data-in-pg-database-on-eks/647.md?page=2)