Introducing Automatic Sharding for Smarter, Faster CloudQuery Syncs! 🚀

Syncing large amounts of cloud data (especially from AWS) can be a time-consuming process. But with CloudQuery’s new Automatic Sharding feature, you can now split your syncs into multiple shards and run them in parallel, reducing sync time drastically! :bulb:

:mag: What’s Automatic Sharding?
Sharding allows CloudQuery to split the collection of tables into groups and sync them independently. Simply use the --shard option to define how many shards you’d like to use, and CloudQuery will handle the rest!

:briefcase: Real-World Example:
Setting up a parallel sync with GitHub Actions is now easier than ever. Just set up a matrix configuration to split your syncs into multiple jobs and run them concurrently. :zap:

:link: Check out the full post and documentation to see how you can leverage Automatic Sharding and make your syncs faster than ever.

2 Likes