CloudQuery table aws_route53_hosted_zone_resource_record_sets always appends records during sync

CloudQuery Plugins
1

Hi,

I have a question regarding the aws_route53_hosted_zone_resource_record_sets table. I’m using the AWS source with a PostgreSQL destination plugin.

No matter what write_mode I use, the records are being appended, and every sync is writing all records again. This table has a primary key _cq_id.

Has anyone had a similar experience? How can I solve this issue to overwrite the data and delete stale records?

2

Can you share the redacted logs and configs that you used to define the sync? So we can look into the issue.

3

Hi @mighty-hornet - Just checking in on this. If you are still having issues or if it got resolved.

4

Hi @ben,

So after updating to the next version, it is picking up the right primary key. We were using 22.19.0 in which the primary key was _cq_id and moved to 23.0.0 which has resource_arn as the primary key. This is solving our issue.

Thanks a lot!

5

I would suggest migrating to v24.x because that version we switched how we are doing Primary Keys. Rather than having composite PKs, we are using a deterministic hash of multiple columns as the PK. This means that we can fix PKs without having to release a breaking change.

For example, yesterday AWS announced cross-account sharing for Parameter Store resources, which would have required us to release a breaking change to update the Primary Key. But now we could do it as a minor update.