Questions about AWS event-based sync in CloudQuery

CloudQuery Plugins
1

I have two questions about event-based sync with the AWS source plugin.

  1. I read the section on the AWS plugin documentation page but haven’t begun looking into the CloudFormation and Kinesis parts yet. Is the CloudQuery container routinely polling the Kinesis stream, or is Kinesis pushing data to CloudQuery?
  2. What impact does this have on _cq_sync_time? We key off of that for some TTL indexes in our database to only keep a few months of stale records.
2

Great questions:

  1. Currently, the CQ container is just polling the Kinesis Stream. In the future, we may add support for Enhanced Fan Out.
  2. Currently, _cq_sync_time is the time at which the process starts. Does that represent an issue for your team?
3

We use _cq_sync_time to see when a resource was last seen by CloudQuery. If the process is long-lived, then it becomes less useful for us as an approximation of the last-seen time. I guess we might have to rethink what we are doing with that.

@ben I was looking on the documentation for more information on pricing for event sync since it’s a premium feature, but the pricing page only mentions “Pay as You Go” and “Support” plans. Do you know where I could find this info?

4

Hi @dominant-worm,

Thank you for that feedback. I have opened an issue for changing the behavior around cq_sync_time for long-lived processes: #15690.

As for pricing, we haven’t released anything yet, but in the meantime, all premium features are free.

5

Appreciate you opening that issue! As for premium features, if you have any indication about the type of pricing model you’re thinking about, that would be helpful. We are in discussions to make use of event-based sync but don’t want to commit if it’s something we couldn’t afford for the whole org.

6

Hi @dominant-worm - Happy to walk you through our pricing structure and answer any questions you might have. Here’s a link to my calendar if you’d like to set up a quick call: https://calendly.com/josh-cloudquery

7

@bold-goshawk Awesome, thanks! I scheduled something for tomorrow.