Custom module not installing with plugin install command

CloudQuery Plugins
1

I asked this in a thread, but figured I’d bubble up in main.

I’m trying to run a plugin install with my custom module, but the installer is only installing the one from the CQ registry. Is this a bug?

/app # tree .cq
.cq

0 directories, 0 files

/app # cat base_config.yml
# This is a stub file to help build the Docker container with all the plugins that we want.
kind: source
spec:
  name: plugin-name
  path: "xxx.dkr.ecr.us-west-2.amazonaws.com/company/plugin-name:0.0.1-alpha"
  registry: "docker"
  destinations: ["postgresql"]
  spec:
    max_retries: 3
    scheduler: "shuffle"
  tables:
    - "*"
---
kind: destination
spec:
  name: "postgresql"
  path: "cloudquery/postgresql"
  registry: "cloudquery"
  version: "v7.1.0"
  migrate_mode: "forced"
  write_mode: "overwrite-delete-stale"
  spec:
    connection_string: "${PG_CONNECTION_STRING}"
    pgx_log_level: "info"

/app # CLOUDQUERY_API_KEY=xxx PG_CONNECTION_STRING=xxx ./cloudquery plugin install .
Loading spec(s) from .
Downloading https://plugins.cloudquery.io/cq-cloud-releases/cloudquery/destination/postgresql/v7.1.0/linux_arm64
Downloading 100% |████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| (10/10 MB, 12 MB/s)

/app # tree .cq
.cq
└── plugins
└── destination
└── cloudquery
└── postgresql
└── v7.1.0
├── plugin
└── plugin.zip

5 directories, 2 files

2

So Docker plugins rely on Docker running locally. When we run cloudquery plugin install, it is actually a wrapper around docker pull, so the images do not end up in .cq/plugins/source. The easiest way to deploy a Docker plugin in a containerized environment is to run the plugin as a sidecar and communicate via gRPC. We have an example of how to do this in our docs: Using CloudQuery Docker Registry Plugins Inside a Containerized Environment.

Let me know if you have any other questions!

3

Maybe also worth mentioning that most official plugins are not Docker plugins. But if you write a custom plugin in Python, Java, or JavaScript, or use one of the few official ones that are written in one of those languages, this advice would apply.

4

You can verify that plugin install worked with any Docker plugin by running the following command:

docker images

You should see the image referenced by the spec in the images list.

5

So, would a GoLang plugin behave like the official ones and not need Docker?

6

Yes, a Go plugin would be like an official one and download the binary locally.

7

What is it about Go plugins that doesn’t require them to run as a gRPC server?

Or maybe another set of questions could be:

Why can’t Python plugins be released as a binary (like, what mechanism is behind the reasoning)?

Can Go plugins be released as an internal-only binary? (I do not see documentation other than publishing to the hub)

8

All plugins run as a gRPC server. Go plugins are built as a binary which is downloaded based on the local OS/Arch. Python, Java, and JavaScript plugins use Docker to support multiple OS/Archs.

It’s harder to build Python/JavaScript plugins to multiple OS/Archs, especially if you have native bindings. So, it’s harder to publish Python/JavaScript packages as a single binary (e.g., you’d have to package Python/Node.js with it too probably to ensure compatibility). Docker solves that.

You don’t have to publish to the Hub; you can configure

registry: local
path: <local-path-to-binary>

Then, if you download the binary manually before running the sync, it should work (e.g., via curl). You could even run the Python plugin directly via

registry: grpc
path: localhost:7777

assuming you have the plugin running via python main.py serve or something similar. Not sure what would work best for you.

You can also publish to the Hub as a private plugin; then it will be accessible only to your team.

9

Interesting, thanks for the detail.

So just spitballing here, not trying to reinvent what you have, but if I were to compile the Python code using something like Nuitka, then I can have it running locally and use the registry: local designation potentially.

10

Yes, that would be possible. If you would like to set up a call to talk through the ways to deploy a custom plugin in your organization, we would be happy to chat!