Hey, I’ve been looking for various ELT and data piping solutions, and CloudQuery seems like a really good fit for many parts.
One thing we’d also need, though, is pushing some data. Is it possible to, for example, write a plugin for CloudQuery that listens to a socket and forwards data through CloudQuery as it gets it?
Hey The use case you’re considering is certainly possible with CloudQuery, but it’s slightly more involved.
As an example, our premium Postgres source plugin replicates Postgres tables via CDC, which is a similar use case.
You’ll have to manage the state of the sync via the “Incremental tables” feature: Managing Incremental Tables
And since the sync doesn’t really end, you have to implement the sync operation yourself, rather than using the existing scheduler’s Sync command, like the one in our example XKCD source plugin: XKCD Source Plugin
Note how, in this plugin, the incremental table state is flushed after the plugin finishes syncing:
@tight-hagfish - We would be interested in understanding more about the project you are working on and the other alternatives you are considering to see if there is anything we can do to help you out! If you are interested, feel free to find some time on my calendar to chat!