CloudQuery update issue with aws and postgres plugins migration error

CloudQuery Plugins
1

Hey there! I’m having some troubles after updating the versions of AWS and Postgres plugins. It seems like a couple of tables have changed, and that’s preventing me from running CloudQuery successfully.

I get the following error:

type: update, current: version:utf8:PK:NotNull, previous: version:utf8]] require migration. Migrate manually or consider using 'migrate_mode: forced'

module=cli

I’ve added the migrate_mode: forced to my Postgres config but I’m still getting the same error. I also tried the cloudquery migrate command, but I was not successful either. :frowning_with_open_mouth:

2

Hi :wave:

Sorry to hear you’re having trouble; can you share the config you’re using with any sensitive values redacted? Also, please share the output of

cloudquery --version
3 
CloudQuery Version 3.13.1

Source Plugin Config (AWS):
```yaml
kind: source
spec:
  ## Required. Name of the plugin to use.
  ## This should be unique if you have a number of AWS plugins.
  name: "aws"
  ## Optional. Where to search for the plugin. Default: "github". Options: "github", "local", "grpc"
  # registry: "github"

  ## Path for the plugin.
  ## If registry is "github", path should be "repo/name"
  ## If registry is "local", path is path to binary. If "grpc", then it should be the address of the plugin (usually useful in debug).
  path: "cloudquery/aws"

  ## Required. Must be a specific version starting with v, e.g. v1.2.3
  ## Checkout latest versions here: https://github.com/cloudquery/cloudquery/releases?q=plugins-source-aws&expanded=true
  version: "v22.18.0"

  ## Required. You can use ["*"] to sync all tables or specify specific tables. Please note that syncing all tables can be slow.
  ## See all tables: https://www.cloudquery.io/docs/plugins/sources/aws/tables
  tables: [
    "aws_eks_clusters",
    "aws_elbv2_load_balancers",
    "aws_cloudfront_distributions",
    "aws_cognito_identity_pools",
    "aws_ec2_instances",
    "aws_ec2_vpc_endpoints",
    "aws_ec2_images",
    "aws_amplify_apps",
    "aws_iam_users",
    "aws_iam_saml_identity_providers",
    "aws_backup_protected_resources",
    "aws_backup_jobs",
    "aws_backup_plans",
    "aws_acm_certificates",
    "aws_rds_clusters",
    "aws_waf_web_acls"
  ]

  ## Required. All destinations you want to sync data to.
  destinations: ["postgresql"]

  spec:
    ## Optional. Plugin specific configuration.
    ## By default, will use the current AWS credentials available (just like AWS CLI).
    ## See all available options here: https://github.com/cloudquery/cloudquery/blob/main/plugins/source/aws/docs/configuration.md
    regions:
      - XXXXX
      - XXXXXX
      - XXXXXXX
    aws_debug: false
    org:
      admin_account:
        local_profile: ""
      member_role_name: XXXXXXXXXXXXXXXXXXXXX

Destination Plugin Config (PostgreSQL):

kind: destination
spec:
  ## Required. Name of the plugin.
  ## This is an alias so it should be unique if you have a number of PostgreSQL destination plugins.
  name: "postgresql"
  ## Optional. Where to search for the plugin. Default: "github". Options: "github", "local", "grpc".
  # registry: "github"
  
  ## Path for the plugin.
  ## If registry is "github", path should be "repo/name".
  ## If registry is "local", path is path to binary. If "grpc", then it should be the address of the plugin (usually useful in debug).
  path: "cloudquery/postgresql"
  
  ## Required. Must be a specific version starting with v, e.g. v1.2.3
  ## Checkout latest versions here: https://github.com/cloudquery/cloudquery/releases?q=plugins-destination-postgresql&expanded=true
  version: "v7.0.1"
  
  ## Optional. Default: "overwrite-delete-stale". Available: "overwrite-delete-stale", "overwrite", "append".
  ## Not all modes are supported by all plugins, so make sure to check the plugin documentation for more details.
  write_mode: "overwrite-delete-stale" # overwrite-delete-stale, overwrite, append

  spec:
    ## Plugin-specific configuration for PostgreSQL.
    ## See all available options here: https://github.com/cloudquery/cloudquery/tree/main/plugins/destination/postgresql#postgresql-spec
    ## Required. Connection string to your PostgreSQL instance.
    ## In production, it is highly recommended to use environment variable expansion.
    ## connection_string: ${PG_CONNECTION_STRING}
    connection_string: ${PG_CONNECTION_STRING}
    pgx_log_level: error
    migrate_mode: forced
4

I think the first thing to try is probably to upgrade the CLI; there may have been some fixes around this since 3.13.1. You can see all the releases here: GitHub Releases and the changelog: CHANGELOG.md.

I think the other problem is that migrate_mode: forced is at the wrong level of indentation; it should be in the top-level spec, underneath write_mode. Here are the docs for this: CloudQuery Destination Spec.

5

Thanks for the answer.
I’ll update the CLI and also I’ll correctly set the migrate mode option.
It’s curious, though, that CloudQuery didn’t complain about the config file, so I guess it’s not doing any kind of validation on the format.

6

Yeah, I think it doesn’t error on unrecognized parameters on that level, but probably it should.

7

Seems like after updating the CloudQuery client and also specifying the migrate mode in the right part of the expect, it worked :slightly_smiling_face:. To be honest, I think the issue was with the version of CloudQuery, as I did try the migrate command and it didn’t work either.

Thanks for your support!