Hi all,
I’m having an issue with the Snowflake plugin that I need to upgrade to get the “overwrite-delete-stale” stale bit working. I’m running on ECS and CQ version 4.1.2. This issue occurs with any table.
Using v3.1.0 of the Snowflake plugin works as expected:
kind: destination
spec:
name: snowflake_prod
path: cloudquery/snowflake
version: "v3.1.0"
write_mode: "append"
spec:
connection_string: ${SNOWFLAKE_PROD_CONNECTION_STRING}
migrate_concurrency: 30
batch_size_bytes: 33554432
Here’s the log when using v3.1.0:
2023-12-05T10:25:28Z INF table sync finished client=xxxxxxxxx:eu-west-2 errors=0 module=aws-src resources=4 table=aws_rds_certificates
2023-12-05T10:25:28Z INF finished call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2023-12-05T10:25:26Z grpc.time_ms=2120.533 module=cli peer.address=@ protocol=grpc
2023-12-05T10:25:29Z INF batch written successfully duration=1440.186382 len=376 module=snowflake-src table=aws_rds_certificates
2023-12-05T10:25:29Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2023-12-05T10:25:26Z grpc.time_ms=3631.115 module=cli peer.address=@ protocol=grpc
2023-12-05T10:25:29Z INF started call grpc.code=OK grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2023-12-05T10:25:29Z grpc.time_ms=0.014 module=cli peer.address=@ protocol=grpc
2023-12-05T10:25:29Z INF finished call grpc.code=OK grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2023-12-05T10:25:29Z grpc.time_ms=16.911 module=cli peer.address=@ protocol=grpc
2023-12-05T10:25:29Z INF End sync destinations=["snowflake_prod (cloudquery@cloudquery/snowflake)"] module=cli source="aws_rds (cloudquery@cloudquery/aws)" sync_time=2023-12-05T10:25:24Z
However, using v3.3.3 of the Snowflake plugin fails:
kind: destination
spec:
name: snowflake_prod
path: cloudquery/snowflake
version: "v3.3.3"
registry: cloudquery
write_mode: "append"
spec:
connection_string: ${SNOWFLAKE_PROD_CONNECTION_STRING}
migrate_concurrency: 30
batch_size_bytes: 33554432
And here’s the log when using v3.3.3:
2023-12-05T10:29:50Z INF table sync finished client=xxxxxxxxx:eu-west-2 errors=0 module=aws-src resources=4 table=aws_rds_certificates
2023-12-05T10:29:50Z INF finished call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2023-12-05T10:29:47Z grpc.time_ms=2452.028 module=cli peer.address=@ protocol=grpc
2023-12-05T10:29:50Z ERR failed to write batch error="failed to put file into stage with last resource put 'file:///tmp/aws_rds_certificates.json.xxx' @cq_plugin_stage auto_compress=true: user: lookup userid 0: invalid argument" duration=509.323884 len=376 module=snowflake-src table=aws_rds_certificates
2023-12-05T10:29:50Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2023-12-05T10:29:47Z grpc.time_ms=2974.662 module=cli peer.address=@ protocol=grpc
2023-12-05T10:29:50Z INF started call grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2023-12-05T10:29:50Z grpc.time_ms=0.009 module=cli peer.address=@ protocol=grpc
2023-12-05T10:29:50Z INF finished call grpc.code=OK grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2023-12-05T10:29:50Z grpc.time_ms=21.721 module=cli peer.address=@ protocol=grpc
2023-12-05T10:29:50Z INF End sync destinations=["snowflake_prod (cloudquery@cloudquery/snowflake)"] module=cli source="aws_rds (cloudquery@cloudquery/aws)" sync_time=2023-12-05T10:29:46Z
Any help on this would be greatly appreciated!