Another issue. CQ seems to be having issues when I provide a config.yml with 2 source plugins (AWS & K8s) and a destination. It only seems to process the first source in the file - any ideas? I can split the config easily enough, but it seems that should be supported.
Is the sync ending with an error?
Can you share the error (if you are getting one) and the config with any sensitive information redacted?
It was hitting some errors, but I’ve added a bunch of skip tables which seem to have cleared that.
Note, I’ve tried swapping the k8s and aws sections and I see the same behaviour - it just processes the first one.
Also, the exit code from the sync command is 0, suggesting no critical issues.
So if a source fails to complete a sync, CloudQuery will not go on to the next source as the state of the destination cannot be assured to be in the expected state.
Are there any errors in the logs?
Not if I run k8s first. We’ve got a bunch of errors in the AWS sync, but I think they’ve always been there.
And what version of the CQ binary are you using? You can get that by running
cloudquery --version
Can you share the exact (redacted) output of the cloudquery sync config.yml?
Interesting! Can you share the entirety of this line:
I0202 18:05:06.489447 1982 request.go:697] Waited for 1.005582183s due to client-side throttling, not priority and fairness, request: GET:https://<<eks endpoint>>/apis/policy/v1/namespaces/linkerd-viz/poddisruptionbudgets
That is the entirety of it - the redacted bit is just the EKS K8s API server endpoint.
I am trying to reproduce this issue on my end…
I am noticing that in your logs it only starts 2 plugins
2024-02-02T18:05:00Z INF Plugin server listening address=/tmp/cq-bPpuMgZabqcJEfiF.sock module=cli
2024-02-02T18:05:01Z INF Plugin server listening address=/tmp/cq-zgqPFoSOOPLDrRLU.sock module=cli
While it should be starting 3 plugins (2 sources and 1 destination).
This is what I get:
local-computer % CQ_DSN="postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable" cloudquery sync config-copy.yml --log-console
2024-02-02T18:19:42Z INF Loading spec(s) args=["config-copy.yml"] module=cli
2024-02-02T18:19:42Z INF Plugin server listening address=/var/folders/wj/t5q5kjds14lf_wfq8ldg_fbc0000gn/T/cq-EugMEnlefEIcrbMZ.sock module=cli
2024-02-02T18:19:43Z INF Plugin server listening address=/var/folders/wj/t5q5kjds14lf_wfq8ldg_fbc0000gn/T/cq-uKpIovalRgTtAJhB.sock module=cli
2024-02-02T18:19:44Z INF Plugin server listening address=/var/folders/wj/t5q5kjds14lf_wfq8ldg_fbc0000gn/T/cq-UhzSZtWzAdEIsomH.sock module=cli
Can you try running:
cloudquery migrate config.yml --log-console
I want to see if this is purely related to the sync or something else.
Think I’ve worked it out. I saw something similar before, but it failed in a different way. There are trailing spaces in the config after the first --- separator, which messes up the plugin loading. It is because this line (spec_reader.go#L79) is perhaps a bit too strict?
Yep - that was it. Thanks for the help. Have a great weekend!
Have a great weekend!