Help needed with CloudQuery to prevent local object deletion for specific accounts

CloudQuery Plugins
1

Two quick questions. First, is there a way to tell CQ to not delete local objects for a particular account/list of accounts? We periodically want to suspend harvesting of select accounts during certain hours of the day, but when I ignore them, the resources in the database are purged on the next harvest. It would be nice if there was an option to tell CQ to persist local objects on the skipped accounts.

2

The best way to do this is to use separate configurations. The purging occurs based on the name field in the source. So if each config has a different value for name, it will only clean up data that it had previously synced.

More information about this can be found here: Running CloudQuery in Parallel

3

Hrm, that may be problematic on our end. See the next question :). I’ll have to give it some thought.

4

I have some questions regarding AWS syncing. Can someone help me clarify how to properly set up syncing between my local environment and S3?

Specifically, I would like to know:

  1. What are the best practices for using AWS CLI to sync files?
  2. Are there specific flags or options I should be aware of?
  3. How do I handle syncing large files or a large number of files efficiently?
  4. Can I automate this process, and if so, what tools or options would you recommend?

Any insights or examples would be greatly appreciated!

5

Second, we’ve got some custom tables that we’ve built to enrich CloudQuery. We’re using the _cq_source_name as our joining column. I noticed there’s no table indexing on this column across the Postgres tables. We could add it manually, but then we’re deviating from the core schema. Forced migrations would, of course, blow this away. Any thoughts here?

For some of these tables, we’re likely going to have over a million rows, so being able to do an indexed join would be preferable.

The problem with doing a custom configuration for the paused accounts is that when we want to do this, we’ll get a lot of duplicate resources.

6

Interesting, I will have to think about this and get back to you…

7

Understood, thanks Ben.