I am soooo close to getting a SharePoint-to-Snowflake connection working. I’ve validated that SharePoint-to-File works correctly (99 line CSV). When I try SharePoint-to-Snowflake, the table gets created, but there are no rows! So clearly I’m connected, but try as I might, I cannot get the data.
Do you get any error? Or is it failing silently?
Silently failing.
But the table schema is correct.
can you run with --log-level debug? see if anything suspicious there?
Log looks okay to me:
2023-10-29T23:15:30Z INF End sync destinations=["snowflake (cloudquery<@&873653355623370752>/snowflake)"] module=cli source="sharepoint (v1.8.2)" sync_time=2023-10-29T23:15:25Z
2023-10-29T23:15:30Z INF Sending sync summary to analyticsv1.cloudquery.io:443 module=cli
2023-10-29T23:15:30Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write2 grpc.method_type=client_stream grpc.service=proto.Destination grpc.start_time=2023-10-29T16:15:27-07:00 grpc.time_ms=3142.941 module=cli peer.address=@ protocol=grpc
2023-10-29T23:15:30Z INF waiting for source plugin to terminate module=cli
2023-10-29T23:15:30Z INF source plugin process exited with exit status 1 module=cli
2023-10-29T23:15:30Z INF waiting for source plugin to terminate module=cli
2023-10-29T23:15:30Z INF source plugin process exited with exit status 1 module=cli
Is it with the --log-level debug flag?
No… here it is with:
2023-10-29T23:25:19Z INF finished call grpc.code=OK grpc.component=server grpc.method=Configure grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-10-29T16:25:19-07:00 grpc.time_ms=0.509 module=cli peer.address=@ protocol=grpc
2023-10-29T23:25:19Z INF started call grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-10-29T16:25:19-07:00 grpc.time_ms=0 module=cli peer.address=@ protocol=grpc
2023-10-29T23:25:20Z DBG Migrating table module=sf-dest table=sharepoint_bpm
2023-10-29T23:25:20Z DBG Table exists, auto-migrating module=sf-dest table=sharepoint_bpm
2023-10-29T23:25:21Z INF finished call grpc.code=OK grpc.component=server grpc.method=Migrate grpc.method_type=unary grpc.service=proto.Destination grpc.start_time=2023-10-29T16:25:19-07:00 grpc.time_ms=1792.462 module=cli peer.address=@ protocol=grpc
2023-10-29T23:25:21Z INF End migration destinations=["snowflake (cloudquery<@&873653355623370752>/snowflake)"] module=cli source="sharepoint (v1.8.2)" time_took=1.7935384
2023-10-29T23:25:21Z INF Start fetching resources destinations=["snowflake (cloudquery<@&873653355623370752>/snowflake)"] module=cli source="sharepoint (v1.8.2)"
2023-10-29T23:25:21Z INF started call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-10-29T16:25:21-07:00 grpc.time_ms=0 module=cli peer.address=@ protocol=grpc
2023-10-29T23:25:21Z INF started call grpc.component=server grpc.method=Write2 grpc.method_type=client_stream grpc.service=proto.Destination grpc.start_time=2023-10-29T16:25:21-07:00 grpc.time_ms=0 module=cli peer.address=@ protocol=grpc
2023-10-29T23:25:21Z DBG selecting columns from list cols=["ID","Incident","Owner/UserName","Reviewed","Status","Created","AuthorId","Modified","EditorId"] module=sharepoint-src table=sharepoint_bpm
2023-10-29T23:25:21Z INF sync finished duration=159.9249 errors=0 module=sharepoint-src panics=0 resources=99
2023-10-29T23:25:21Z INF finished call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-10-29T16:25:21-07:00 grpc.time_ms=159.924 module=cli peer.address=@ protocol=grpc
2023-10-29T23:25:21Z INF started call grpc.code=OK grpc.component=server grpc.method=GetMetrics grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-10-29T16:25:21-07:00 grpc.time_ms=0 module=cli peer.address=@ protocol=grpc
2023-10-29T23:25:21Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetMetrics grpc.method_type=unary grpc.service=cloudquery.source.v1.Source grpc.start_time=2023-10-29T16:25:21-07:00 grpc.time_ms=0 module=cli peer.address=@ protocol=grpc
2023-10-29T23:25:21Z DBG Migrating table module=sf-dest table=sharepoint_bpm
2023-10-29T23:25:21Z DBG Table exists, auto-migrating module=sf-dest table=sharepoint_bpm
2023-10-29T23:25:24Z INF batch written successfully duration=1756.8241 len=99 module=snowflake-src table=sharepoint_bpm
2023-10-29T23:25:24Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write2 grpc.method_type=client_stream grpc.service=proto.Destination grpc.start_time=2023-10-29T16:25:21-07:00 grpc.time_ms=3528.705 module=cli peer.address=@ protocol=grpc
2023-10-29T23:25:24Z INF End sync destinations=["snowflake (cloudquery<@&873653355623370752>/snowflake)"] module=cli source="sharepoint (v1.8.2)" sync_time=2023-10-29T23:25:18Z
2023-10-29T23:25:24Z INF Sending sync summary to analyticsv1.cloudquery.io:443 module=cli
2023-10-29T23:25:24Z DBG sending kill signal to destination plugin module=cli
2023-10-29T23:25:24Z INF waiting for source plugin to terminate module=cli
2023-10-29T23:25:24Z INF source plugin process exited with exit status 1 module=cli
2023-10-29T23:25:24Z DBG sending kill signal to destination plugin module=cli
2023-10-29T23:25:24Z INF waiting for source plugin to terminate module=cli
2023-10-29T23:25:24Z INF source plugin process exited with exit status 1 module=cli
I’ve tried all three write modes, cq-id-only… always zero rows in the table.
Strange indeed. Seems like a Snowflake connector then. Do you see any error on the Snowflake console?
Interesting to see how the plugin works. One thing: that file in the temp directory isn’t there now.
Yeah, I think it is deleted after the sync. And when you SELECT * it returns zero rows? I’ll open an issue then so we can try to run and reproduce. Will share a link here shortly.
Yes. Thank you!
This is the first time I’ve ever been motivated to seek help via Discord. CQ would be tremendously useful.
Thanks! Can you share the versions of CLI, source plugin, and destination plugin?
Would be interesting to have a flag that tells CQ not to clean up temp artifacts (file and stage).
Just installed all 3 today, so newest:
spec:
name: "sharepoint"
registry: "github"
path: "koltyakov/sharepoint"
version: "v1.8.2" # provide the latest stable version
kind: destination
spec:
name: "snowflake"
path: "cloudquery/snowflake"
registry: "cloudquery"
version: "v3.2.11"
cloudquery version 3.26.1
tyvm
@next-kiwi Hi! I just released Snowflake v3.3.0, which adds a leave_stage_files option. Set leave_stage_files: true in the spec, and you’ll be able to examine the files after a sync.
It should also show “Created stage file” messages in the logs to help locate the files.
Additionally, we’ve uploaded a new version that uses filepath.Base as well as OS-based escaping for the path. (see https://github.com/cloudquery/cloudquery/releases/tag/plugins-destination-snowflake-v3.3.3)
@concise-ladybug Could you check with the v3.3.3 if the problem still persists?
Thank you. I will try. Sadly, the source (SharePoint) has now stopped working with an UnauthorizedAccessException error, though nothing has changed.
However, I’ll hook up a different source to try the Snowflake destination update. One sec.
It worked! Very nice. I also see the temp file (but not the stage in Snowflake - I assume that was removed).
Any ideas for why the SharePoint source would all of a sudden start throwing UnauthorizedAccessException?
I would suggest asking the maintainer, as it’s a community plugin. See their repo for the issues section.
However, if you upgraded recently to the newest version available (v2.0.0) of that plugin, it contained some small bugs, so it would probably be a good idea to downgrade or wait for the new release.
@concise-ladybug Could you run with SharePoint v1.8.2? If that works for you, please open an issue at cq-source-sharepoint/issues.
Additionally, it would be nice to see that the initial issue that was reported for SharePoint 1.8.2 was resolved for that version, so we’d like to hear back from you if the sync ran successfully. This will require dropping the data from Snowflake first.