Need help adding a static value column in CloudQuery source plugin

CloudQuery Plugins
1

Hey there, for a new source plugin I want to add a column with a static value (same value for all records, for example the table name). Could you please advise what is the best option to achieve that? Do I need a resolver for that?

2

Hi @alert-wildcat, is this a plugin you’re building? If so, yes, you would define a column on a table with a custom resolver that always sets the same value.

3

Focus on this line in the code snippet in the link I just sent:

res <- comic

comic here is a struct that you can define any way you want. So you could add a field on it and give it any value you want.

Just make sure you use that struct when you define the table’s Transformer:

func Comics() *schema.Table {
    return &schema.Table{
        Name:     "xkcd_comics",
        Resolver: fetchComics,
        Transform: transformers.TransformWithStruct(&xkcd.Comic{}),
    }
}
4

Hi @erez,

Thanks for your reply. I am testing the build of a new plugin and wondering how to do that.

Hi @mariano,

Thanks a lot for your feedback. I will have a look on this.

5

Your plugin is essentially a server. If your plugin code compiles, you can build the binary and execute it with

./your_binary serve

This will give you a server listening at localhost:7777 by default (it will output this address on startup).

Then, you can put this address in your source configuration and run a sync. Make sure you have

registry: "grpc"

That’s how to test it.

Follow instructions here: CloudQuery Documentation

6

Hi @mariano,

Just to be sure that I get the point, in order to add a new column with a static value, do I have 2 options?

Option 1: I can use a “Column Resolver” as for the example ‘is_good’ column here (Column Resolver Documentation) and then set the static value in the Resolver function.

Option 2: Since the value is the same for all records/rows, I can set it in the Table Resolver during the data fetch/API call as in the code snippet you shared above:

res <- comic

Am I correct, or did I miss something? Which option do you recommend, please?

7

You’re correct in that you have those two options.

Typically, the struct you’re sending to res <- your_data is not one made by you, but from some external SDK. Adding a field to that struct then means creating a brand new struct that has all fields from the former, and then your extra field. For that reason, among other reasons, there’s the simpler option of adding a column resolver.

Use whichever option is simpler or looks clearer for your case.

8

Thanks a lot for your help. Just I am wondering if the option of adding a column resolver will be slower since it should be run for each record/row, right?

9

The slowness of a sync is determined by a number of factors, and this particular one is in the order of nanoseconds, whereas network latencies will be in hundreds of milliseconds. I’m not convinced either method is particularly slower, but if it is, it’s probably negligible compared to other factors. I wouldn’t optimise here.

10

Got it. Thanks again for your help!