Request for example of incremental table implementation in Go

CloudQuery SDK
1

Heyo!

Does anyone possibly have an example of an incremental table implementation in Go?
Nevermind, found one :sweat_smile: example here

2

Hi :wave: Yes, there are a few, the Hacker News plugin being the best documented, meant to serve as an example: Hacker News Plugin

It’s also a bit more complicated than some of the others, so Stripe is also a good example :slightly_smiling_face:

Happy to help if you have questions though!

3

Ah awesome, thank you so much!
https://github.com/cloudquery/cloudquery/blob/main/plugins/source/hackernews/resources/services/items/items_fetch.go#L34

Is the expectation that there should only 1 cursor per table, or are we fine to create multiple cursors per table (e.g., 1 cursor per GitHub org per table in the GitHub plugin)?

4

No, this happens to be the case for the Hacker News plugin, but if the table makes use of multiplexing, you’d usually expect the key to be able to uniquely identify both the table and the client. For example, here’s how it’s done for the AWS CloudTrail Events table:

https://github.com/cloudquery/cloudquery/blob/0b74613eaed7183d9672fddde86f0d5753f05d60/plugins/source/aws/resources/services/cloudtrail/events.go#L53-L65