Arrow type for DateTime in CloudQuery

CloudQuery Plugins
1

What Arrow type should I use for a DateTime? Right now I’m using arrow.PrimitiveTypes.Date64, but that is clearly just the date.

2

We generally use timestamp pyarrow.timestamp.

3 
arrow.FixedWidthTypes.Timestamp_us worked. Thank you.

How do you guys choose to have a parent-child relation or just have a column resolver with the other API route?
4

If the relationship is 1:many, we use child tables. And generally, we prefer parent-child over column resolvers because this has better concurrency control in our SDK and gives end users more flexibility to skip the child tables if they don’t want it.

We prefer to keep tables as close as possible to the underlying APIs and leave further transformations to users. So we tend to use column resolvers only if they don’t trigger more API calls (maybe the data just needs to be massaged a bit to fit into the Arrow format, for example).

There are some historical tables that break these rules of thumb, but this is how I think about it now.

5

I need to use the S3 Destination Plugin, but I need to specify a KMS key for encryption. I noticed the following code snippet:

s, err = c.Client.StartStream(table, func(r io.Reader) error {
    _, err := c.uploader.Upload(ctx, &s3.PutObjectInput{
        Bucket: aws.String(c.spec.Bucket),
        Key:    aws.String(objKey),
        Body:   r,
    })
    return err
})

I would need to turn it into this:

s, err = c.Client.StartStream(table, func(r io.Reader) error {
    _, err := c.uploader.Upload(ctx, &s3.PutObjectInput{
        Bucket: aws.String(c.spec.Bucket),
        Key:    aws.String(objKey),
        Body:   r,
        
        SSEKMSKeyId:         aws.String("your-kms-key-id"), // replace with your KMS key ID
        ServerSideEncryption: aws.String("aws:kms"),
    })
    return err
})

Can I just fork the repo and create a merge request (MR)? I was planning on having it go through the spec, so I think I would need to complete testing as well. Is there any way to pass these via environment variables?

6

You are welcome to open a PR; it would most likely need to be a config option in the spec, yes. If you make it a config option, you can use environment variable expansion syntax to use an env variable in the config, e.g. ${SSE_KMS_KEY_ID}.