Help needed understanding example code for CloudQuery source plugin development

CloudQuery Plugins
1

Regarding developing a source plugin: I am working from the example at CloudQuery Documentation and I want to make a source plugin that will collect the top viewed pages from the Wikipedia API. I’m having trouble understanding a part of the example code.

(More detail in thread:)

The example code is:

func Comics() *schema.Table {
    return &schema.Table{
        Name:     "xkcd_comics",
        Resolver: fetchComics,
        Transform: transformers.TransformWithStruct(&xkcd.Comic{}),
    }
}

What is the meaning of xkcd in &xkcd.Comic{}?

I assume I’ll change that to wikipedia, but I don’t see where else I may need to write wikipedia.

2

transformers (and Transform) is a helper of sorts that generates fields dynamically from a given struct.

3

Okay. I did write a struct.

4

You don’t need to use it, but it enables generating/updating the table columns automatically as the data struct is updated. Especially useful if you’re using a third-party client which provides the data model.

5

So if the data model changes, CloudQuery will handle database migrations, right?
My eventual aim is to write a Google Workspace plugin, and off hand I don’t know how stable that API is, so it seems worthwhile to learn about using a transformer.

6

Yes. As you upgrade a client dependency to a newer version, the new version of your plugin will have the updated columns.

You’re not limited to using transformers (or just transformers); of course, you can specify extra (or all) columns in the Columns field as well.

WithPrimaryKeys and WithSkipFields are the most used options for transformers. You might also want to look at WithUnwrap* options if the struct has some embedded data you’d like to “unwrap” as top-level columns.

These options provide some control on how the struct is transformed.

7

Okay, I can see how “unwrapping” could be useful.

8

Hope this helps. If you get stuck, let us know for more explanations :slightly_smiling_face:

9

That helped give general context on :slight_smile:

10

Our official open-source plugins live here if you want to see transformers in action. I suggest checking the smaller plugins first; large ones can be a bit too complex sometimes.

11

I’m still unclear about the xkcd in &xkcd.Comic{}.
Does xkcd refer to a file name? Something else?

12

xkcd is the package name for the xkcd API client (which I think is also bundled with that plugin, let me check).

13

Ohh, okay.

And I think I see the detail I missed:

so we will create our own struct inside a new xkcd package

Where would the xkcd package go?

Like, would this be the directory structure:

.
├── README.md
├── client
│   ├── client.go
│   └── spec.go
├── cq-source-xkcd
├── go.mod
├── go.sum
├── main.go
├── plugin
│   └── plugin.go
├── xkcd
│   └── xkcd.go
└── resources
    └── table.go

Or would there be an xkcd.go file in client?

14

it’s under internal/xkcd/
if you’re implementing a new client for this plugin only, best practice is to put it under internal/ so it’s not importable from “outside”