# Issue with CloudQuery affecting multiple AWS RDS parameter tables **URL:** \ **Category:** CloudQuery Plugins\ **Created:** [April 17, 2024, 10:45pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245 "2024-04-17T22:45:21Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![honest-snipe](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@honest-snipe](https://community.cloudquery.io/u/honest-snipe)\ **Post date:** [April 17, 2024, 10:45pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/1 "2024-04-17T22:45:21Z")
This problem affects the `aws_rds_cluster_parameters` (123M), `aws_rds_cluster_parameter_group_parameters` (93M), and `aws_rds_db_parameter_group_db_parameters` (31M) tables. ---
**Author:** ![ben](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/ben/32/88_2.png) [@ben](https://community.cloudquery.io/u/ben)\ **Post date:** [April 17, 2024, 10:46pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/2 "2024-04-17T22:46:39Z")
Would you mind sharing your configurations? This will help us to debug the issue. ---
**Author:** ![honest-snipe](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@honest-snipe](https://community.cloudquery.io/u/honest-snipe)\ **Post date:** [April 17, 2024, 11:14pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/3 "2024-04-17T23:14:15Z")
It is a very simple and the version is probably out of date. ---
**Author:** ![ben](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/ben/32/88_2.png) [@ben](https://community.cloudquery.io/u/ben)\ **Post date:** [April 17, 2024, 11:19pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/4 "2024-04-17T23:19:09Z")
That config is helpful! Thank you. What is the query you used to grab the data you shared? ---
**Author:** ![honest-snipe](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@honest-snipe](https://community.cloudquery.io/u/honest-snipe)\ **Post date:** [April 17, 2024, 11:20pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/5 "2024-04-17T23:20:27Z")
Here are some interesting stats… But I am just chasing the ones \> 5M. ---
**Author:** ![ben](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/ben/32/88_2.png) [@ben](https://community.cloudquery.io/u/ben)\ **Post date:** [April 17, 2024, 11:21pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/6 "2024-04-17T23:21:36Z")
Can you share the config for the destination as well? ---
**Author:** ![honest-snipe](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@honest-snipe](https://community.cloudquery.io/u/honest-snipe)\ **Post date:** [April 17, 2024, 11:21pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/7 "2024-04-17T23:21:59Z")
The destination is just PostgreSQL. ---
**Author:** ![ben](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/ben/32/88_2.png) [@ben](https://community.cloudquery.io/u/ben)\ **Post date:** [April 17, 2024, 11:22pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/8 "2024-04-17T23:22:17Z")
Are you specifying `write_mode`? In your source config, you specify the name to be `{{.Name}}`. Is that value a deterministic value? Is it the same value each time you run a sync? ---
**Author:** ![honest-snipe](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@honest-snipe](https://community.cloudquery.io/u/honest-snipe)\ **Post date:** [April 17, 2024, 11:27pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/9 "2024-04-17T23:27:14Z")
Here is what I found… this section: ```yaml syncPolicy: syncOptions: - ApplyOutOfSyncOnly=true - CreateNamespace=true ``` Everything else seems standard = defaults. For the AWS plugin, we use defaults. We also developed 2 custom plugins, and for those, we have `write_mode: append` in the config file. ---
**Author:** ![ben](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/ben/32/88_2.png) [@ben](https://community.cloudquery.io/u/ben)\ **Post date:** [April 17, 2024, 11:29pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/10 "2024-04-17T23:29:48Z")
So the `append` write mode is designed to never delete any data. ---
**Author:** ![honest-snipe](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@honest-snipe](https://community.cloudquery.io/u/honest-snipe)\ **Post date:** [April 17, 2024, 11:30pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/11 "2024-04-17T23:30:31Z")
Unless the AWS plugin inherits this somehow. ---
**Author:** ![ben](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/ben/32/88_2.png) [@ben](https://community.cloudquery.io/u/ben)\ **Post date:** [April 17, 2024, 11:31pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/12 "2024-04-17T23:31:39Z")
How do you define the `{{.Name}}` variable that gets injected into the spec name? If that is unique every single run, then no data will ever be deleted. You can see more information about that here: [CloudQuery Documentation](https://docs.cloudquery.io/docs/advanced-topics/running-cloudquery-in-parallel). ---
**Author:** ![honest-snipe](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@honest-snipe](https://community.cloudquery.io/u/honest-snipe)\ **Post date:** [April 17, 2024, 11:32pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/13 "2024-04-17T23:32:50Z")
My reading of the code is that `write_mode` that we use in our custom plugin is not inherited by the standard AWS plugin. We dynamically create the AWS plugin name for each table and run each extractor as a cron (K8s). ---
**Author:** ![ben](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/ben/32/88_2.png) [@ben](https://community.cloudquery.io/u/ben)\ **Post date:** [April 17, 2024, 11:38pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/14 "2024-04-17T23:38:27Z")
When you say it is dynamically generated, does that mean that each time your cron job runs it is a different value? ---
**Author:** ![honest-snipe](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@honest-snipe](https://community.cloudquery.io/u/honest-snipe)\ **Post date:** [April 17, 2024, 11:39pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/15 "2024-04-17T23:39:11Z")
No, only at deployment time to set different `crontime` for each extractor so they do not swarm the AWS APIs. **Question:** Each time the extractor runs (based on AWS source plugin), will it repopulate the entire table? What is the default behavior (`trim` and `write`)? You are suspecting the data is appended, right? The sync time seems to indicate that the data is outdated (2023?). ---
**Author:** ![ben](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/ben/32/88_2.png) [@ben](https://community.cloudquery.io/u/ben)\ **Post date:** [April 18, 2024, 12:06am UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/16 "2024-04-18T00:06:44Z")
The behavior is that it will do an upsert and then a delete for any record where the `_cq_sync_time` is outdated and the source name is the same. Can you share the following: 1. The result of this query: 2. The schema of your `aws_rds_cluster_parameters` table. ---
**Author:** ![honest-snipe](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@honest-snipe](https://community.cloudquery.io/u/honest-snipe)\ **Post date:** [April 18, 2024, 12:26am UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/17 "2024-04-18T00:26:00Z")
It is taking a long, long time. I ran the following query: ```sql SELECT COUNT(*) FROM aws_rds_cluster_parameters WHERE _cq_sync_time < TO_DATE('2024-01-01', 'YYYY-MM-DD') LIMIT 10; ``` This was to prove that there are many records with an outdated timestamp. The primary key is defined as: ```sql PRIMARY KEY, BTREE (_cq_id); ``` ---
**Author:** ![ben](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/ben/32/88_2.png) [@ben](https://community.cloudquery.io/u/ben)\ **Post date:** [April 18, 2024, 2:40am UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/18 "2024-04-18T02:40:28Z")
Can you share a copy of the redacted logs? Specifically, I am looking for lines that contain `grpc.method`. ---
**Author:** ![honest-snipe](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@honest-snipe](https://community.cloudquery.io/u/honest-snipe)\ **Post date:** [April 18, 2024, 7:46pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/19 "2024-04-18T19:46:09Z")
also I have got the result of the query you requested: ```sql postgres=> select COUNT(DISTINCT "_cq_source_name") from aws_rds_cluster_parameters; count ------- 1 (1 row) ``` ---
**Author:** ![ben](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/ben/32/88_2.png) [@ben](https://community.cloudquery.io/u/ben)\ **Post date:** [April 18, 2024, 7:48pm UTC](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245/20 "2024-04-18T19:48:47Z")
OK! Thank you for that information! Are there any other logs? Because I am not seeing the events that correlate to the end of the sync. I would assume that the issues that you are facing have been resolved by now, as we are on v8 of the Postgres plugin and v26 of the AWS plugin. If you upgrade to the latest versions and still see issues persisting, we would be more than happy to dive in to better understand the issue and find a fix for it. [Next page](https://community.cloudquery.io/t/issue-with-cloudquery-affecting-multiple-aws-rds-parameter-tables/245.md?page=2)