Can plugins be published to AWS ECR with CloudQuery

CloudQuery Plugins
1

Can plugins be published to AWS ECR?
I am on

cloudquery version 5.18.1

and when I try to do a cloudquery plugin install from my own ECR, I get an error

Error: invalid cloudquery plugin path: xxx.dkr.ecr.us-west-2.amazonaws.com/company/plugin-name. format should be team/name

Curious if there’s any guide or documentation from others on the matter, if it’s possible.

2

We added support for private Docker repositories, and ECR was validated as part of this effort. Let me find the documentation…

3

Thanks for providing your configuration. Here’s the YAML config you shared:

kind: source
spec:
  name: plugin-name
  path: "xxx.dkr.ecr.us-west-2.amazonaws.com/company/plugin-name"
  version: "v0.0.1-alpha"
  destinations: ["postgresql"]
  spec:
    max_retries: 3
    scheduler: "shuffle"
  tables:
    - "*"

If you’re encountering an error related to authentication, it could also be linked to the format of your configuration. Could you provide the specific error message you are encountering? This might help in determining whether the issue is due to the authentication credentials or the format of your YAML file.

4

Can you remove the version and put it as part of the path.

kind: source
spec:
  name: plugin-name
  path: "xxx.dkr.ecr.us-west-2.amazonaws.com/company/plugin-name"
  destinations: ["postgresql"]
  spec:
    max_retries: 3
    scheduler: "shuffle"
  tables:
    - "*"

And then for actually running it, this is how we tested with ECR:

aws ecr get-login-password --region <REGION> | docker login --username AWS --password-stdin xxx.dkr.ecr.<REGION>.amazonaws.com;
cloudquery plugin install
5

Without version, the spec is invalid.

Error: failed to load spec(s) from ..
Error: failed to validate source plugin-name: version is required
6

Sorry, forgot, you need to specify registry: "docker"

kind: source
spec:
  name: plugin-name
  registry: "docker"
  path: "xxx.dkr.ecr.us-west-2.amazonaws.com/company/plugin-name:v0.0.1-alpha"
  destinations: ["postgresql"]
  spec:
    max_retries: 3
    scheduler: "shuffle"
  tables:
    - "*"
7

Gotcha!

Ok, thanks. I’m actually trying to build a container with all the plugins pre-installed. Not sure yet how Docker in Docker will work out… lol. Anyway, I’ll figure it out, thanks.

I figured out how to build a Docker image with an attached network path to the host Docker socket, but when I built my container, I noticed that my plugin was not in the .cq/plugins/source folder.

My Dockerfile is basically like this:

ARG TAG=latest
FROM ghcr.io/cloudquery/cloudquery:${TAG}

ARG DOCKER_HOST=tcp://socat:12345

ARG API_KEY
RUN apk add --no-cache aws-cli docker jq
WORKDIR /app
COPY base_config.yml base_config.yml

RUN docker ps
RUN CLOUDQUERY_API_KEY=${API_KEY} PG_CONNECTION_STRING=foobar ./cloudquery plugin install .

ENTRYPOINT [ "" ]

The base_config.yml has the custom module as I’ve described above. I would have expected that the cloudquery plugin install should add the source plugin, but it’s not there.

8

So Docker plugins rely on Docker running locally. When we run cloudquery plugin install, it is actually a wrapper around docker pull, so the images do not end up in .cq/plugins/source. The easiest way to deploy a Docker plugin in a containerized environment is to run the plugin as a sidecar and communicate via gRPC. We have an example of how to do this in our docs: Using Cloud Query Docker Registry Plugins Inside a Containerized Environment

Let me know if you have any other questions!

9

Afk now. Thanks for the info! If they don’t end up in the source folder, then is there a way to see if it worked? Besides, of course, testing it, I wanted to create a way to validate on build before any execution.

10

The best way to validate that it was successful is probably to run the cloudquery tables command as that interacts with the plugin but doesn’t execute it. CloudQuery Tables Documentation

Another option is to just run

docker images --format <IMAGE>

and then grep the response.