CloudQuery issue with Docker daemon connection in Kubernetes pod

CloudQuery Plugins
1

Hey, is it possible to use Artifactory when publishing and fetching a custom plugin for internal use?

I’m also getting this issue:

...
2024-02-22T19:36:08Z INF Loading spec(s) args=["/shared/cloudquery.yml"] module=cli
2024-02-22T19:36:08Z WRN when using the CloudQuery registry, it's recommended to log in via `cloudquery login`. Logging in allows for better rate limits and downloading of premium plugins module=cli
Error: failed to list Docker images: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2024-02-22T19:36:08Z ERR exiting with error error="failed to list Docker images: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?" module=cli

in the logs.

So, for context, this is running in a pod on a Kubernetes cluster.

2

Hi @comic-firefly,

So for the error you’re seeing, are you trying to reference a Docker-based plugin? If so, you might need a solution like Sysbox to run Docker in Kubernetes.

By Artifactory, do you mean JFrog Artifactory? We don’t support unauthenticated Docker registries at the moment; there’s a feature request for AWS ECR here. You can add a comment to it or upvote it (assuming Artifactory exposes a Docker registry API).

Another option is to publish to CloudQuery Docker Hub; that should allow keeping the plugin private if you’re not using a public registry.

There are instructions in the CloudQuery documentation under JavaScript/Python plugins.

3

<:sad:895405990764224512>

4

Adding support for more authenticated registries could make a good contribution.

5

I am actively working on supporting ECR. Would that work for your use case? If not, can you provide documentation on the Artifactory functionality you would like to see us support?

6

So Artifactory should function similarly to ECR, GHCR, and Docker Hub. You can docker login, push, and pull images, etc.

Would there ever be support for running the CloudQuery dashboard on-prem or in the customer’s cloud? Just basically purchase a license key or have a subscription for that.

Self-host.

7

Yeah, we work with many organizations that have needs to self-host and they use offline license keys.

8

Oh sweet. Maybe we should set up that meeting. Would that make creating and running custom plugins easier for us? My org would prefer that route.

9

Would love to chat! Calendly Meeting Link

10

Hey, I scheduled something for 3 PM EST next Wednesday. Is there a way to run the plugin separately as a Kubernetes cron job but run it as if I’m running locally? I’d like to use the features of the SDK (table creation, syncing).

So what I’m thinking is I use the following config and make sure I include the cloudquery.yaml config in the container, but the config will look like:

kind: source
spec:
  name: "my-custom-plugin"
  registry: "grpc"
  path: "localhost:7777"
  version: "v1.0.0"
  tables:
    ["*"]
  destinations:
    - "postgres"
---
kind: destination
spec:
  name: postgres
  path: cloudquery/postgres
  registry: cloudquery
  version: "v7.3.2"
  spec:
    connection_string: ...
11

I don’t think I fully understand the problem/use case… In your mind, what is the difference between a Kubernetes (k8s) CronJob and running locally?

12

So I’m running CloudQuery as a cron job where each job is associated with a config to source data from one AWS account and sync to a PostgreSQL database.

I created a custom plugin to also sync data from an internal app API to store in the same database. Since I can’t source from our Artifactory server (private Docker registry), I wanted to know if I can run the image as a Kubernetes cron job without telling CloudQuery to fetch the plugin container image.

Does that make sense? I mean I know I can run the image, but I still need to pass the config (using a config map or storing the config in the image) to specify the destination. Basically running the plugin but with the source config set up with grpc as the registry. But, this would run in a cron job.

Unless we can run a self-hosted version of CloudQuery (with a license).

13

So we just merged private Docker registry support into the CLI (have not released a new version yet, but will soon). Would that fix your issues?

If not, yes, you can startup the plugin as a gRPC server and not run a sync or migration, and it will be fine.

14

The only issue with private Docker registry support is that I’d have to run Docker in the container itself, correct?

15

I can try that route.

So, how does the CloudQuery registry work? Is it just pulling the plugin code (Go, Python, etc.), or do all the CloudQuery-provided plugins come with the CloudQuery image?

Btw, I can save these questions for the meeting.

16

The hub /registry is a centralized repository to enable discovery and downloading of plugins, transformations, and visualizations. You can think about the CloudQuery architecture very similarly to Terraform, where tf is a single binary that downloads the providers at runtime. You can build custom images that include the plugins that you need, but the default CloudQuery container only contains the CLI binary.

For more information, you can check the Dockerfile here: Dockerfile

17

We need a CloudQuery Enterprise, lol.
Yeah, we host TFE with all our custom modules. That makes sense.

18

Go plugins are able to be downloaded as a binary, so it doesn’t need Docker. We support plugins in Java, JavaScript, and Python, and all of those require Docker in order to provide a stable interface to interact with the plugins (over gRPC) without having to worry about underlying dependencies.

19

If you are controlling the container and dependencies, then as part of the cron job, you could just start up the Python plugin before you run sync.

20

Man, I almost just want to host CQ and re-write the plugin in Go, tbh. But I’ll mess with getting the Python plugin to run in the meantime. We can chat Wednesday about it.

Hey, so I was able to get this to work. I run a job with two containers, the custom plugin and CloudQuery. When I run the job, I see my custom tables successfully created in the PG destination database. However, I can’t get much information from the plugin logs in terms of why they are failing to sync data.

kubectl logs -n censusv3 -f censusv3-cloudquery-yellowpages-rrg9s
Defaulted container "cq-yellowpages" out of: cq-yellowpages, cloudquery, vault-provisioner (init)
{"address": "[::]:7777", "event": "Starting server", "level": "info", "timestamp": "2024-03-07T16:40:48Z"}
{"client_id": "yellowpages", "table": "yp_aws_accounts", "depth": 0, "event": "table resolver started", "level": "info", "timestamp": "2024-03-07T16:40:51Z"}
{"client_id": "yellowpages", "table": "yp_aws_accounts", "resources": 0, "depth": 0, "exc_info": true, "event": "table resolver finished with error", "level": "error", "timestamp": "2024-03-07T16:40:51Z"}
{"client_id": "yellowpages", "table": "yp_services", "depth": 0, "event": "table resolver started", "level": "info", "timestamp": "2024-03-07T16:40:51Z"}
{"client_id": "yellowpages", "table": "yp_teams", "depth": 0, "event": "table resolver started", "level": "info", "timestamp": "2024-03-07T16:40:51Z"}
{"client_id": "yellowpages", "table": "yp_services", "resources": 0, "depth": 0, "exc_info": true, "event": "table resolver finished with error", "level": "error", "timestamp": "2024-03-07T16:40:51Z"}
{"client_id": "yellowpages", "table": "yp_teams", "resources": 0, "depth": 0, "exc_info": true, "event": "table resolver finished with error", "level": "error", "timestamp": "2024-03-07T16:40:51Z"}

As you can see, we have an init container (vault) and the two main sidecar containers. The sync is initiated, but they are erroring, and I would like to capture print statements from my plugin to debug whether it’s a connection issue or something else.

Oh wait, I probably need to use the logger. Yeah, that didn’t work either. <:sad:895405990764224512>