CloudQuery job not pulling locally built image in minikube

CloudQuery Plugins
1

Hi, I am running CloudQuery in a local Minikube cluster and created a source plugin in Python. I am having trouble making the CQ-Job pull the locally built image. Is there a way to define in the spec that the image of the container of the plugin should be fetched locally?

Update: I was able to fetch a local image of my plugin, but the…

2

Hi! We don’t have this support yet, and we only support remote Docker registry. Mind opening an issue here: GitHub - CloudQuery Issues?

One way that will be easy for development is just running the Python plugin locally and then connecting to it via registry: grpc. You can find more information in the documentation: CloudQuery Source Spec - Registry.

3

Hi. I actually was able to make the job fetch the Docker from the host. But now I am getting an error message:

{"level":"error","module":"cli","error":"context canceled","time":"2023-10-15T15:18:38Z","message":"failed to read log line from plugin"}
{"level":"error","module":"cli","error":"All attempts fail:\n#1: connection not ready\n#2: connection not ready\n#3: connection not ready\n#4: connection not ready\n#5: connection not ready\n#6: connection not ready\n#7: connection not ready\n#8: connection not ready\n#9: connection not ready\n#10: connection not ready\n#11: connection not ready\n#12: connection not ready\n#13: connection not ready\n#14: connection not ready\n#15: connection not ready\n#16: connection not ready\n#17: connection not ready\n#18: connection not ready\n#19: connection not ready\n#20: connection not ready\n#21: connection not ready\n#22: connection not ready\n#23: connection not ready\n#24: connection not ready\n#25: connection not ready\n#26: connection not ready\n#27: connection not ready\n#28: connection not ready\n#29: connection not ready\n#30: connection not ready","time":"2023-10-15T15:18:38Z","message":"exiting with error"}
4

I’m not sure right now, but from a quick glance, it seems like either it’s an issue on the plugin side or there is some issue in the connection between the CLI and the Docker plugin. Is Docker-in-Docker enabled?

5

I’ve enabled Docker-in-Docker via mounting docker.sock. I took the Square Python Plugin as a boilerplate.

Is there anything I need to define in the Kubernetes deployment to be able to run the plugin as a gRPC server? Should I use a Unix socket instead of TCP?