Hi, your docs mention streaming to Google Pub/Sub. Is that actually possible? I don’t see a destination for it.
ben
November 11, 2023, 9:30pm
2
Can you provide a link to where you are seeing that?
the time the change was made, and the actual data that was changed.
The WAL records are written to disk on the master server,
and a separate process, known as the WAL sender, sends the WAL records to the replica server.
The replica server then applies the changes to its own copy of the data.
This process happens in real-time, so the replica server's copy of the data is always up-to-date with the master server's copy.
## How CloudQuery Uses Postgres Logical Replication to Sync Postgres to Any Other Destination
CloudQuery PostgreSQL source plugin provides the ability to sync data and changes in real time from a PostgreSQL database, directly to any
of the supported [destinations](/docs/plugins/destinations/overview), without the need for Kafka or any other intermediary. (You can still stream to Kafka or Google PubSub.)
This gives a much simpler setup to operate: just run one CloudQuery binary!
In the following section, we will go through how our CloudQuery PostgreSQL source plugin works internally. (For running and configuring, refer to the [docs](https://www.cloudquery.io/docs/quickstart).)
### Set `wal_level` to logical
To use logical replication, you will need to make sure you started your database with `wal_level`.
See [our documentation](https://hub.cloudquery.io/plugins/source/cloudquery/postgresql) on how to set `wal_level` to logical in various environments.
It says “You can still stream to Kafka or Google PubSub.”
ben
November 11, 2023, 9:34pm
4
Yeah, we do not (currently) support Google PubSub as a destination, but you can request that here if you want: Request a new destination plugin .
We will look into that confusing/ambiguous language. I have opened an issue to track that work: Issue #15229 .
Is there an option to publish to an HTTP endpoint?
ben
November 11, 2023, 9:49pm
6
There is not, but we have an open issue for one: #5107
I would suggest upvoting it as community engagement and feedback drive much of our prioritization.
I gave that a thumbs up. So, just to confirm, there’s no way for us to workaround this lack of pub/sub plugin? Any alternative suggestion?
ben
November 11, 2023, 10:15pm
8
I am not that familiar with what options there are within the GCP ecosystem with regards to data/event routing, but there are a few options if you need this functionality now:
You can write your own destination plugin. Docs for writing custom plugins can be found here .
We offer Paid Support plans where we can work with your team to implement features and plugins that are critical to your team. More information can be found here .
Others might weigh in as well on Monday if they have any ideas.