Using aws_inspector_findings and aws_inspector2_findings tables in CloudQuery is taking more time to sync for all the accounts in our database, and then failing for some accounts randomly. It is almost running for 7 to 8 hours and failing for some random accounts.
What should we do for these tables? One thing we have observed is that when we skip these two tables, the sync works fine and runs as expected in 2 to 2.5 hours.
How can we overcome this issue?
The two tables you have highlighted have the potential to have a lot of resources, so the fact that it takes so long is not that surprising. The best way to speed up syncing is to use the table_option. This allows you to specify which findings to sync based on the available filters from AWS. More details about this can be found here.
As for general performance, you can look into splitting your syncs into multiple smaller jobs by following the guidance found here.
Okay, we will try this and let you know. Thanks for the information.