AWS Plugin Error

CloudQuery Plugins
,
1

I’ve recently been experiencing the following issue with an AWS batch:

"error":"failed to execute batch with pgerror: severity: ERROR
message: invalid byte sequence for encoding \"UTF8\": 0xe7 0xe3 0x6f

Looking into my PostgreSQL database, the enconding is already UTF-8 and the error message doesn’t show me wich table is causing the problem.

Does anyone know what’s going on?

2

Hey @Laura_Sorato ,

First of all, welcome to the CloudQuery community! :waving_hand:

In order for us to be able to help you, could you please share with us the sync configuration as well a the log file?

Thanks!

3

Hello, sorry for the delay in getting back to you.

Hera are the sync configuration and the error in the log file (I can’t share the entire file because it contains sensitive information).

  • Sync Configuration:
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudquery-config
data:
  aws: |
    kind: source
    spec:
      # Source spec section
      name: aws
      path: cloudquery/aws
      registry: cloudquery
      version: "v32.43.1"
      tables: ["*"]  
      skip_tables:
        - aws_accessanalyzer_analyzer_findings
        - aws_accessanalyzer_analyzer_findings_v2
        - aws_apigateway_rest_api_documentation_versions
        - aws_athena_work_group_query_executions
        - aws_cloudtrail_events
        - aws_cloudwatchlogs_log_streams
        - aws_codebuild_builds
        - aws_connect_evaluation_form_versions
        - aws_connect_view_versions
        - aws_costexplorer_cost_custom
        - aws_docdb_cluster_parameter_groups
        - aws_docdb_engine_versions
        - aws_ec2_instance_types
        - aws_ec2_vpc_endpoint_services
        - aws_ecr_repository_image_scan_findings
        - aws_elasticbeanstalk_platform_versions
        - aws_elasticache_engine_versions
        - aws_elasticache_parameter_groups
        - aws_elasticache_reserved_cache_nodes_offerings
        - aws_elasticache_service_updates
        - aws_elastictranscoder*
        - aws_emr_release_labels
        - aws_emr_supported_instance_types
        - aws_frauddetector_event_types
        - aws_frauddetector_model_versions
        - aws_glue_job_runs
        - aws_grafana_versions
        - aws_guardduty_detector_findings
        - aws_iam_group_last_accessed_details
        - aws_iam_policy_last_accessed_details
        - aws_iam_role_last_accessed_details
        - aws_iam_user_last_accessed_details
        - aws_inspector*
        - aws_inspector2*
        - aws_lex*
        - aws_macie2_findings
        - aws_neptune_cluster_parameter_groups
        - aws_neptune_db_parameter_groups
        - aws_rds_cluster_parameter_groups
        - aws_rds_cluster_parameter_group_parameters
        - aws_rds_db_parameter_groups
        - aws_rds_engine_versions
        - aws_rds_major_engine_versions
        - aws_securityhub_findings
        - aws_servicequotas_awsdefaultservicequotas
        - aws_servicequotas_quota_utilizations
        - aws_servicequotas_services
        - aws_snowball*
        - aws_ssm_command_invocations
        - aws_stepfunctions_map_run_executions
        - aws_stepfunctions_map_runs
        - aws_support_services
        - aws_support_trusted_advisor_check_results
      destinations: ["postgresql"]
      spec:
        org:
          admin_account:
            role_arn: ${AWS_ROLE}
          member_role_name: cloudquery-ro
        regions:
          - 'us-east-1'
          - 'ca-central-1'
          - 'sa-east-1'
          - 'us-west-2'
        use_paid_apis: false
    ---
    kind: destination
    spec:
      name: "postgresql"
      path: "cloudquery/postgresql"
      registry: "cloudquery"
      version: "v8.8.12"
      write_mode: "overwrite-delete-stale"
      spec:
        connection_string: ${CONNECTION_STR}
        batch_size: 2500
        batch_size_bytes: 4194304
  • Log File:
{"level":"error","err":"ERROR: invalid byte sequence for encoding \"UTF8\": 0xe7 0xe3 0x6f (SQLSTATE 22021)","invocation_id":"62345452-6bfa-4c34-aefg-987b470da0x6","message":"BatchClose","module":"pgx","pid":22866,"time":47.320705,"time":"2025-09-25T04:05:52Z","time":"2025-09-25T04:05:52Z"}
{"level":"error","duration":89.201799,"error":"failed to execute batch with pgerror: severity: ERROR, code: 22021, message: invalid byte sequence for encoding \"UTF8\": 0xe7 0xe3 0x6f, detail :, hint: , position: 0, internal_position: 0, internal_query: , where: unnamed portal parameter $13, schema_name: , table_name: , column_name: , data_type_name: , constraint_name: , file: mbutils.c, line: 1737, routine: report_invalid_encoding: ERROR: invalid byte sequence for encoding \"UTF8\": 0xe7 0xe3 0x6f (SQLSTATE 22021)","invocation_id":"62345452-6bfa-4c34-aefg-987b470da0x6","len":2222,"message":"failed to write batch","module":"pg-dest","time":"2025-09-25T04:05:52Z","time":"2025-09-25T04:05:52Z"}
Error: failed to sync v3 source aws: write client returned error (insert): plugin returned error: failed to execute batch with pgerror: severity: ERROR, code: 22021, message: invalid byte sequence for encoding "UTF8": 0xe7 0xe3 0x6f, detail :, hint: , position: 0, internal_position: 0, internal_query: , where: unnamed portal parameter $13, schema_name: , table_name: , column_name: , data_type_name: , constraint_name: , file: mbutils.c, line: 1737, routine: report_invalid_encoding: ERROR: invalid byte sequence for encoding "UTF8": 0xe7 0xe3 0x6f (SQLSTATE 22021)
{"level":"error","module":"cli","invocation_id":"62345452-6bfa-4c34-aefg-987b470da0x6","error":"failed to sync v3 source aws: write client returned error (insert): plugin returned error: failed to execute batch with pgerror: severity: ERROR, code: 22021, message: invalid byte sequence for encoding \"UTF8\": 0xe7 0xe3 0x6f, detail :, hint: , position: 0, internal_position: 0, internal_query: , where: unnamed portal parameter $13, schema_name: , table_name: , column_name: , data_type_name: , constraint_name: , file: mbutils.c, line: 1737, routine: report_invalid_encoding: ERROR: invalid byte sequence for encoding \"UTF8\": 0xe7 0xe3 0x6f (SQLSTATE 22021)","time":"2025-09-25T04:05:54Z","message":"exiting with error"}
4

@Laura_Sorato Can you identify which table is causing this? Just use the interval halving method to find out what’s causing this. It looks like there might be some string value from AWS that is not correctly encoded.

5

The logs don’t show which table might be causing this issue :confused:.

About 2 months ago this job was working normally, we tried some plugin updates to get it working again but it didn’t work, so we went back to the version that worked and the error still persisted.

6

Hi @Laura_Sorato , unfortunately the error isn’t showing the table causing this, and without knowing the table it’s going to be hard to help here. The issue seems to be an unusual response from the AWS API.

Like Michal suggested, could you run the sync again, but only with the first half the tables? If the error still happens, halve the tables again, or if not, run the sync the sync with the second half of the tables. Then keep repeating this procedure until there’s only one table in the sync and the error occurs, then we’ll know which table is causing this.

1 Like