Error syncing CloudQuery with AWS AppSync GraphQL APIs column already exists

Also, failed to sync v3 source aws_xxxxxxxxxxxxx: write client returned error (insert): plugin returned error: failed to add column resolver_count_limit on table aws_appsync_graphql_apis:

ERROR: column "resolver_count_limit" of relation "aws_appsync_graphql_apis" already exists (SQLSTATE 42701)

What does this error mean?

  1. Does this error happen every time?
  2. What version of the CLI are you running?
  3. Can you please share the config that you were using when encountering this error? (Be sure to redact any sensitive info)
  4. Can you share the logs from this (can be found in cloudquery.log)?

I feel like I had a similar error, but I updated the AWS plugin.

@comic-firefly - To clarify, you had been getting this error, but now after upgrading, you are not getting it anymore?

It could also be the PG plugin that needs to be updated as well (if using that).

But I remember seeing the issue with columns and wondering why migration wasn’t working. If using the PostgREST plugin, you may need to force migration.

Interesting! Adding a column is considered a “safe” migration. If we can get more details on this issue, we will check it out.

So it happened right around this time: link
Not the same error in the thread, but I saw it around that time. Updating fixed it, if I remember correctly.

  1. It is the first time I have seen this error.
  2. Versions I am using:
    • Postgres - v7.3.3
    • AWS - v23.7.0
    • Azure - v11.4.3
    • CLI - v5.5.0

Also, for these versions, I am running into an issue where the syncing of tables is taking longer than expected. For the previous set of versions, it was good, and I was able to run it for around 1100 accounts in 2 and a half hours. But now it is taking around 10 hours. I have also skipped the aws_inspector_findings and aws_inspector2_findings tables.

Can I please know to which versions I should upgrade so that I can have a good migration?

Hi @concise-oryx,

Can you share the configuration spec you’re using to run the syncs? Each new plugin version adds new tables, so you might be syncing more data than before.

For the already exists (SQLSTATE 42701) error, are you running multiple syncs at the same time?

Can you please specify what the configuration spec means?

Yeah, I am running CloudQuery sync for multiple accounts at a time based on environments.

@erez, can you please help me out?

Hi @concise-oryx, can you open a bug report via this link that should instruct you on how to provide all the necessary information?

@erez GitHub Issue #17128

I’ll comment on the issue.

okay
@erez can you please check the issue?

@concise-oryx Let’s keep the discussion in the issue? It’s hard to track messages in multiple places.

Sure, I have updated the comment in the issues channel. Can you please take a look and let me know what is actually going wrong there?

Hi @concise-oryx,

To best assist you as we work through this issue, I am interested to learn how you and your organization are using CloudQuery. I would love to hop on a call to discuss. Feel free to schedule a slot on my calendar: https://calendar.app.google/tYWo4ox9GwwMAuUR8

Can you suggest the recommended tables to skip for AWS plugin 23.1.0? Also, could you provide the required PostgreSQL and CLI plugins for this AWS plugin version?

You can use the latest CLI and Postgres plugin versions; they are separate from the AWS plugin version.

Would be happy to discuss that on a call.