Error on second incremental sync with CloudQuery invalid memory address or nil pointer

CloudQuery Plugins
41

Hey @mariano, yeah, it solved the issue :sweat_smile:. Thanks for your help!!

Hi @ben, Iā€™ve tried adding some code changes for the incremental sync and ran it. Now I can observe that it is not recording all the findings from the accounts.

Iā€™ve used the same configuration file with the latest AWS and PostgreSQL plugin versions, i.e., 27.7.0 and 8.2.5.

I have another doubt. If we do an incremental sync for a set of x accounts, do we need to have a different cq_state_aws table for every account?

42

Hey :wave:

You only need to set backend_options once for a given sync, no matter how many AWS accounts there are. It is strange that the same filter criteria works for one vulnerability but not for another. This filtering is done by the AWS SDK API, not by CloudQuery.

If you canā€™t share this data here, could you compare one vulnerability that gets picked up by your filter criteria and one that doesnā€™t? Is the classification exactly the same? For example, in the filter criteria options, thereā€™s something called severity and something else called vendor_severity. Maybe your console UI is filtering for both, but the sync is filtering just one of the two.

43

I think now it is collecting for every account that has findings (only high and critical because we used table_options). But now, a few of the accounts are taking longer than expected for incremental sync.

Hi @ben, I have a question. Can we directly move our current AWS and PostgreSQL plugin versions to the latest? Iā€™m currently on 23.1.0 for AWS and 7.1.2 for PostgreSQL.

Can you let me know what is the best way to update to the latest versions?

44

Hey :wave: Sorry, Iā€™m missing some context here. How is the sync being run? If youā€™re triggering it with the YAML files you shared before, the version of the plugins is stated in the YAML, e.g. from your last snippet:

kind: source
spec:
  name: aws_xxxxxx
  path: cloudquery/aws
  version: "v27.7.0"

The version is there. If you want to discover the latest versions of the plugins, this info is in the hub:

You should just be able to update the version in the YAML to the latest version in those pages, and re-trigger the syncs.

45

Hi @ben, @mariano,

Iā€™m encountering an error when attempting to change the AWS and PostgreSQL plugin versions from 23.1.0 and 7.1.2 to the latest versions directly. Hereā€™s the error message Iā€™m getting:

failed to sync v3 source aws_xxxxxxxxxxxx: write client returned
error
(insert): plugin returned
error
: failed to execute batch with pgerror: severity:
ERROR
, code: 40P01, message: deadlock detected, detail :Process 13778 waits for ShareLock on transaction 86584337; blocked by process 13775.
Process 13775 waits for ShareLock on transaction 86584338; blocked by process 13778., hint: See server log for query details., position: 0, internal_position: 0, internal_query: , where: while inserting index tuple (185,13) in relation "aws_health_events", schema_name: , table_name: , column_name: , data_type_name: , constraint_name: , file: deadlock.c, line: 1149, routine: DeadLockReport:
ERROR
: deadlock detected (SQLSTATE 40P01)

How can I perform this version upgrade without encountering these errors?