Unable to fetch relevant information from meta client in cloudquery

CloudQuery Plugins
1

Hello Team,

I have just started with CloudQuery. It is a very nice tool created. Thanks for doing a good job.

I wrote a code to fetch the GitHub users’ keys by using the ListKeys GitHub API.

My question: from the meta client, I am not able to fetch the relevant information. I am not able to understand what I am missing in my code piece. Below is the basic code snippet, which itself is creating the problem.

c := meta.(*client.Client)
return fmt.Errorf("fetchKeys org: %s repo: %s", c.Org, *c.Repository.Name)

I ran it locally via go run main.go serve.

Logs:

ERR table resolver finished with panic error="invalid memory address or nil pointer dereference" client=org: module=github-source stack="runtime error: invalid memory address or nil pointer dereference\ngoroutine 50 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/cloudquery/plugin-sdk/v4/scheduler.(*syncClient).resolveTableDfs.func1.1()\n\t/Users/apple/Documents/go/pkg/mod/github.com/cloudquery/plugin-sdk/v4@v4.34.2/scheduler/scheduler_dfs.go:92 +0x66\npanic({0x8d23340?, 0x9a938e0?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x132\ngithub.com/cloudquery/cloudquery/plugins/source/github/resources/services/users.fetchKeys({0x0?, 0x0?}, {0x8f437c0?, 0xc000b1c6e0}, 0x866a525?, 0x8669933?)\n\t/Users/apple/Documents/Abhishek/cloudquery/plugins/source/github/resources/services/users/users.go:24 +0x6d\ngithub.com/cloudquery/plugin-sdk/v4/scheduler.(*syncClient).resolveTableDfs.func1()\n\t/Users/apple/Documents/go/pkg/mod/github.com/cloudquery/plugin-sdk/v4@v4.34.2/scheduler/scheduler_dfs.go:102 +0xc5\ncreated by github.com/cloudquery/plugin-sdk/v4/scheduler.(*syncClient).resolveTableDfs in goroutine 16\n\t/Users/apple/Documents/go/pkg/mod/github.com/cloudquery/plugin-sdk/v4@v4.34.2/scheduler/scheduler_dfs.go:89 +0x7e8\n" table=github_users_keys
2

Hi @rich-lion, can you show us what the code around services/users/users.go:24 looks like?

3

Pasted line is line 24

package users

import (
    "context"
    "fmt"

    "github.com/cloudquery/cloudquery/plugins/source/github/client"
    "github.com/cloudquery/plugin-sdk/v4/schema"
    "github.com/cloudquery/plugin-sdk/v4/transformers"
    "github.com/google/go-github/v59/github"
)

func Keys() *schema.Table {
    return &schema.Table{
        Name:      "github_users_keys",
        Resolver:  fetchKeys,
        Transform: client.TransformWithStruct(&github.User{}, transformers.WithPrimaryKeys("ID")),
        //Transform: client.TransformWithStruct(&github.User{}),
    }
}

func fetchKeys(ctx context.Context, meta schema.ClientMeta, _ *schema.Resource, res chan<- any) error {
    c := meta.(*client.Client)
    return fmt.Errorf("fetchKeys org: %s repo: %s", c.Org, *c.Repository.Name)
}

Other tables which are part of the official code are working fine.

Line #24:

return fmt.Errorf("fetchKeys org: %s repo: %s", c.Org, *c.Repository.Name)
4

Hi @rich-lion,

If you’re trying to access c.Repository.Name, you might need to define the repository multiplexer. Let me share some code with you one sec.

5 
Multiplex: client.OrgRepositoryMultiplex,
6

Yes :100:

Link to workflows.go

When the plugin initializes, it sets up the orgs and repositories, but the resource/table needs to define if it needs them in the fetch function. For example, installations only needs orgs.

Link to installations.go

By defining Multiplex: client.OrgRepositoryMultiplex, you’re telling the SDK to run the fetch for each repository in each org.

7

Thanks a lot. Great help.
I am banging my head for the last many hours.
NOW, I am able to sync the table from GitHub.

Thanks a lot once again. Awesome!!!

One more question. How do I print logs if I want to debug the issue like:

c.Logger().Printf("Username: %s", user.String())

var username string
user, _, err := c.Github.Users.Get(ctx, username)
if err != nil {
    c.Logger().Printf("fetchKeys error: %s", err)
}
c.Logger().Printf("Username: %s", user.String())

Where will this Username get printed?

I am not able to see it in cloudquery.log and also not on the server.

8

If you run locally via go run main.go serve, it will show in the plugin process’s stdout. If you compile as a binary and point CloudQuery to it using registry:local and path: </path/to/binary>, it will show in cloudquery.log.

9

I am running as go run main.go serve, but I am not able to see logs from c.Logger().Printf("Username: %s", user.String()) on the stdout.

Do I need to enable some log level or some option?

10

Oh, yes, Printf() prints to debug level Zerolog Logger Printf, so you will need to have that level enabled in the plugin with --log-level debug.

11

Great. Let me try. Thanks a lot.

Plugin invocation:

cloudquery sync config_local.yml --log-level debug

Server:

go run main.go serve

Not able to get the logs on the stdout.

12

You need to add --log-level to the plugin process, so it would be

go run main.go serve --log-level debug

Since you’re running the plugin as its own process, the CloudQuery CLI doesn’t control its output.

13

Thanks a lot. Makes sense.

14

@rich-lion Can you share the code? I also want to fetch the same.

15

Hi @rich-lion, are you planning to submit a PR with your changes? I’m asking since we have this issue open, and it makes a good first issue.

16

@erez Sure. I will refine the code and add test cases and then will raise.