Issue with timestamps when porting JavaScript CloudQuery plugin to Golang

CloudQuery Plugins
1

Hey team, I’m a JavaScript developer who has successfully written several plugins using the JS SDK. However, I’m currently trying to port them over to Golang since there aren’t enough examples and docs for JS for things like incremental tables.

That being said, I’m getting stuck on minor things since I’ve never developed in Go before. My latest issue is with timestamps. I have a column created and its value is 2023-02-24T18:51:07. In JavaScript, for the column, I just specify:

createColumn({
      name: 'created',
      type: new Timestamp(TimeUnit.SECOND),
      resolver: pathResolver('created'),
}),

But in Go, when I try to use:

{
    Name:     "created",
    Type:     arrow.FixedWidthTypes.Timestamp_s,
    Resolver: schema.PathResolver("created"),
},

It’s always null upon syncing. I’ve also tried all the other fixed-width type timestamp_* with no luck.

JavaScript works, Golang is null.

2

Hi @pro-mastiff,

We would love to hear more about your usage of the JS SDK, as it might encourage us to add incremental table support. :slightly_smiling_face:

As for the migration to Go, you might need

schema.PathResolver("Created")

as in Go PathResolver works on struct fields properties which are capital case.

3

I gave capital case a shot but no dice. FWIW, all the other columns I have are lowercase in the pathResolver because they’re lowercase in the JSON and they all work. I’m wondering if it’s because the timestamp doesn’t have a time zone on it. JavaScript is often more forgiving when it comes to that.

Might try a custom resolver to append a Z at the end to see if that allows it to be synced correctly.

4

Maybe you could share some code that should help debug the issue.

This is the relevant code for Go’s path resolver: resolvers.go. You can see it works on the struct fields.

Can you share the type of the struct with its fields? If you think the issue is time zone related, you could try using a string type and see if that solves it.

The conversion from the value received in the response happens here: timestamp.go. You can see some examples of supported values here: timestamp_test.go.

5

I’ve confirmed it’s because of the format of the timestamp string. Appending “Z” to the end does allow it to parse correctly. I’ve asked the maintainers of the API I’m consuming via my plugin to format the timestamp with the appropriate time zone.

time.ParseError {Layout: "2006-01-02T15:04:05Z07:00", Value: "2023-02-24T18:51:07", LayoutElem: "Z07:00", ValueElem: "", Message: ""}

For whatever reason, timezone is not required in JavaScript for parsing.

6

I think this line could be the reason.
Thanks for following up with the solution.