AWS Pricing plugin sync command failed due to api call limits and concurrency settings

CloudQuery Plugins
1

About the AWS Pricing plugin, the sync command stopped in the middle, even though I set spec.concurrency to 5:

source plugin process failed with signal: killed

After doing some research, I found this document:

Using the AWS Price List Bulk API

It clearly shows that we need to set limits on the connections to the API call for the AWS Price List Bulk API.

What’s the best number I should define? Is the setting related to spec.concurrency? Any other parameters related?

kind: source
spec:
  name: "awspricing"
  path: "cloudquery/awspricing"
  version: "v3.0.10"
  tables: ["*"]
  destinations:
    - "postgresql"
  spec:
    # Optional parameters
    # region_codes: []
    # offer_codes: []
    concurrency: 10000 # what number do I need to change to, 5?

How can I understand the relationship between spec.concurrency and the API’s Token bucket size (10) and Refill rate per second (5)?

2

Hi @calm-walrus, let me take a quick look at this.

3

Enable debug and see some errors (frame too large)

2023-10-16T03:42:15Z INF Table exists, auto-migrating module=pg-dest table=awspricing_service_terms
2023-10-16T03:45:51Z INF End sync destinations=["postgresql (v5.0.1)"] module=cli source="aws-pricing (awspricing@v3.0.10)" sync_time=2023-10-16T03:42:05Z
2023-10-16T03:45:51Z INF Sending sync summary to analyticsv1.cloudquery.io:443 module=cli
2023-10-16T03:45:51Z WRN Failed to send sync summary error="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\"" module=cli
2023-10-16T03:45:51Z INF waiting for source plugin to terminate module=cli
4

@calm-walrus Do you mind opening a bug report via this link? That way, we’ll have all the debugging information in one place.

5

Raised: cloudquery/cloudquery#14552