Migrating CloudQuery from EC2 to ECS using FARGATE_SPOT options

CloudQuery Plugins
1

Running CQ on ECS

Hey there! We are trying to migrate CloudQuery from EC2 to ECS to make it a little bit more scalable.

I was having a look at the configuration guide you provide on your website, and in the example task definition, I saw that you include

"requiresCompatibilities": ["FARGATE"]

I was intending to use FARGATE_SPOT, but I’m not sure if you require FARGATE because there may be some problems if the task is not finishing the process and has to be restarted.

2

Yes, that is correct. Running CQ on a spot instance can result in long-running syncs being interrupted. If you have a use case where you are alright with interruptions, then you can definitely try out Fargate Spot.

I would suggest using the overwrite-delete-stale write mode so that if a sync does get interrupted, as soon as the next sync completes, the state will be correct.

3

Ok, thanks for the info.
Regarding the write mode, we would like to change to append mode in order to be able to track how the configuration of the resources changes over time.
I guess for that model FARGATE instead of FARGTE_SPOT is recommended.
I guess there is no such mode that appends and at the same time deletes stale entries.

4

What you could theoretically do is maintain 2 different schemas in your DB, one that is in append mode and the other that is overwrite-delete-stale.

kind: source
spec:
  name: "aws"
  path: "cloudquery/aws"
  destination: ["pg-append", "pg-overwrite"]
  ...
  ...
---
kind: destination
spec:
  name: "pg-append"
  ...
  ...
---
kind: destination
spec:
  name: "pg-overwrite"
  ...
  ...

This would allow you to read the data once, but write in both locations.