Login issues with CloudQuery after latest update returning 401 error

CloudQuery Plugins
1

Hi folks,

I recently noticed that with the latest update, there is a need for a login to run syncs. However, when I am trying to login via the browser pop-up screen or API key, none of those methods are working for me; it gives me a 401 error.

I might be missing something. Any suggestions around this issue?

2

Hi! Can you please share your config, CLI version, and error message?

3

cli version - 3.29.1

kind: destination
spec:
  name: "duckdb"
  path: "cloudquery/duckdb"
  version: "v4.2.10"
  write_mode: "overwrite-delete-stale"
  spec:
    connection_string: "${file:./data_sync/.cloudquery_env}"
    batch_size: 10000

Postgre → DuckDB
Sharing PostgreSQL’s config in 1 minute.

4

Hi @fun-griffon,

I think you’re missing the registry: cloudquery part in your configuration. Also, I recommend using the latest version v5.0.5. So the full config will be:

kind: destination
spec:
  name: "duckdb"
  path: "cloudquery/duckdb"
  version: "v5.0.5"
  registry: cloudquery
  write_mode: "overwrite-delete-stale"
  spec:
    connection_string: "${file:./data_sync/.cloudquery_env}"
    batch_size: 10000

If that doesn’t work, can you please share the cloudquery.log content and console output?

5

Hey, thanks, it worked! registry was missing.

Also, one follow-up question since the thread is open. I am syncing from Postgres to DuckDB, but currently, it is in non-CDC mode. I want to switch to CDC, which works as it should, but the problem is I cannot access the same DuckDB file once the CDC has started syncing, as the file gets locked (DuckDB’s architecture).

Is there a way to access the connection to DuckDB, or how should I go about this?

6

Not sure @fun-griffon, that’s an interesting case. I would open a feature request via GitHub. Maybe copying the DuckDB file will work during the sync?

7

Sure, I’ll open up an issue on this.
I am not sure if copying the file would work; I’ll try that and see how that works for my use case. But ideally, I was thinking if there could be a way to have multiple reader/writer threads to read and write at the same time, which will also serve the CDC purpose of syncing in real-time and reading at the same time.
Thanks, Erez, for your help. :slightly_smiling_face: