Request for CloudQuery plugin to forward data via socket

Hey, I’ve been looking for various ELT and data piping solutions, and CloudQuery seems like a really good fit for many parts.

One thing we’d also need, though, is pushing some data. Is it possible to, for example, write a plugin for CloudQuery that listens to a socket and forwards data through CloudQuery as it gets it?

Hey :wave: The use case you’re considering is certainly possible with CloudQuery, but it’s slightly more involved.

As an example, our premium Postgres source plugin replicates Postgres tables via CDC, which is a similar use case.

You’ll have to manage the state of the sync via the “Incremental tables” feature: Managing Incremental Tables

And since the sync doesn’t really end, you have to implement the sync operation yourself, rather than using the existing scheduler’s Sync command, like the one in our example XKCD source plugin: XKCD Source Plugin

Note how, in this plugin, the incremental table state is flushed after the plugin finishes syncing:

Flush State in XKCD Plugin

In your case, it never finishes, so you must flush in between batches. Thus, you need to implement your own sync.

Okay, that’s probably doable! Thanks for the advice, I’ll probably suggest CloudQuery as one possible alternative to consider for this project. :+1:

@tight-hagfish - We would be interested in understanding more about the project you are working on and the other alternatives you are considering to see if there is anything we can do to help you out! If you are interested, feel free to find some time on my calendar to chat!

Schedule a chat