Hi, I’m trying to create a custom plugin in GoLang, but I’m having issues related to the OS type. The documentation provided is only for MacOS, and I couldn’t find any documentation related to Windows or Linux. Can anyone please share the documentation for Linux or Windows, or let me know if it’s even possible?
Hey 
Which part of these docs are you struggling with? Most of the instructions should be OS-agnostic.
Hi, @mariano. The document mentioned above contains macOS commands for creating a custom plugin, but I don’t have access to a macOS. Is it possible to create a custom plugin without using macOS?
It is certainly possible to create custom plugins on Linux/Windows. The instructions should be mostly identical. Can you give me an example of a step you couldn’t do in your OS?
I don’t have any documentation or related information to create a custom plugin in Windows/Linux for CloudQuery. I can’t seem to find the related documents. If you have any documents on how to create a custom plugin, could you please share them, like the above documentation for Mac OS?
I’m saying you should be able to use the same article.
okay I got it. I will get back to you if I face any further issues. Thank you.
Hi, @mariano,
This is for Mac OS:
Do you have any documentation or video for doing this in Linux or Windows?
Hey I think the only part of that video that is Mac-specific is the installation of the tools (cloudquery & scaffold). The rest of it is just coding on an IDE and then building, running, and testing in a terminal, which should be roughly the same in Linux or Windows.
If you want instructions for installing the tools in a different OS, they are here:
Linux Quickstart
Windows Quickstart
I reviewed these quickstart instructions, and while they tell you how to install the cloudquery CLI, they don’t really spell out where to get scaffold. You can download the latest scaffold version for your OS here:
Download Scaffold
[root@ip-10-129-6-17 environment]# cq-scaffold --verbose source CloudAdmin Cloudquery_Custom_Plugin
[root@ip-10-129-6-17 environment]# ls -l
total 38344
-rw-r--r--. 1 ec2-user ec2-user 569 Jun 25 03:09 README.md
-rwxr-xr-x. 1 root root 35581952 Jul 4 03:44 cloudquery
-rwxr-xr-x. 1 root root 3678208 Jun 25 14:33 cq-scaffold
[root@ip-10-129-6-17 environment]# cd cq-source-Cloudquery_Custom_Plugin
-bash: cd: cq-source-Cloudquery_Custom_Plugin: No such file or directory
[root@ip-10-129-6-17 environment]# go mod tidy
go: go.mod file not found in current directory or any parent directory; see 'go help modules'
[root@ip-10-129-6-17 environment]#
I am facing this issue.
Could you please help me out @mariano?
Could you try it without the --verbose flag? I tried to reproduce your results. If I try with verbose, I get an error, and if I try without it, it works well.
I tested with the latest version:
$ cq-scaffold --version ⏱ 10:21:45
scaffold version 3.0.20
But I also originally tested with a slightly older version I had installed (3.0.15), and I get the same results. Just as a sanity check that cq-scaffold is not broken.
Hi, @mariano
I am using the Kubernetes source plugin and Postgres destination plugin.
Now I am trying to sync the k8s_core_pods table, but I am getting the following error:
cloudquery sync k8s.yml postgres.yml --log-console --log-level error
2024-07-10T11:36:56Z ERR finished call grpc.code=Internal grpc.component=server grpc.error="rpc error: code = Internal desc = failed to init plugin: failed to initialize client: context \"context\" doesn't exist in kube configuration" grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-07-10T11:36:56Z grpc.time_ms=3.846 invocation-id=e26bed3c-fa69-4df2-acaf-c801ef8716eb module=cli peer.address=@ protocol=grpc
Error: failed to sync v3 source k8s_6169124300 failed to init source k8s_6169124300: rpc error: code = Internal desc = failed to init plugin: failed to initialize client: context "context" doesn't exist in kube configuration
2024-07-10T11:36:57Z ERR exiting with error error="failed to sync v3 source k8s_6169124300: failed to init source k8s_6169124300: rpc error: code = Internal desc = failed to init plugin: failed to initialize client: context \"context\" doesn't exist in kube configuration" invocation-id=e26bed3c-fa69-4df2-acaf-c801ef8716eb module=cli
Could you please help me out to resolve this error?
k8s.yml file:
kind: source
spec:
# Source spec section
name: k8s_6169124300
path: cloudquery/k8s
registry: cloudquery
version: "v6.1.4"
tables: ["k8s_core_pods"]
destinations: ["postgresql"]
# # Learn more about the configuration options at https://cql.ink/k8s_source
spec:
contexts: ["context"]
postgres.yml:
kind: destination
spec:
name: "postgresql"
path: "cloudquery/postgresql"
registry: "cloudquery"
version: "v8.2.5"
spec:
connection_string: "XXXXXX"
Hey @known-yeti
I’m assuming you have some level of access to this K8s deployment. If you can use kubectl, this will tell you what the available contexts are:
kubectl config get-contexts
If there is no context listed with the name context, then that’s the problem. You set this configuration in the spec:
contexts: ["context"]
Replace context there with the context listed on kubectl config get-contexts that you want to sync from.
There are other tools available other than kubectl to do the same. For example, k9s, or Lens.
Hi, @mariano
:~/environment/plugin (feat/kubernetes_images) $ kubectl config get-contexts
W0710 13:14:12.732346 22591 loader.go:221] Config not found: /home/ec2-user/.kube/config
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
This is the output of that command. Could you please help me out here?
The output of kubectl config get-contexts is saying that kubectl cannot answer your question because it hasn’t found any configuration to talk to your K8s cluster. This is outside of the scope of the help I can give you since it’s not related to CloudQuery, so I’d suggest Googling or using ChatGPT for that.
If you have further issues with the sync, I’m here to help.
In the Authentication section, it mentioned KUBECONFIG.
I tried with the default and used $ echo KUBECONFIG too.
It’s working fine over there, but it is not working when I try the sync command.
@mariano
From the error when you sync, it looks like you gave cloudquery correct access to your k8s cluster, because it was able to find out that your context doesn’t exist.
When you ran kubectl config get-contexts in your EC2 instance, kubectl wasn’t configured properly, because it couldn’t answer your question.
You need to set up kubectl properly first, then query the context names, and from there you can configure the spec as I explained before.
Thank you @mariano