Syncing large amounts of cloud data (especially from AWS) can be a time-consuming process. But with CloudQuery’s new Automatic Sharding feature, you can now split your syncs into multiple shards and run them in parallel, reducing sync time drastically!
What’s Automatic Sharding?
Sharding allows CloudQuery to split the collection of tables into groups and sync them independently. Simply use the --shard option to define how many shards you’d like to use, and CloudQuery will handle the rest!
Real-World Example:
Setting up a parallel sync with GitHub Actions is now easier than ever. Just set up a matrix configuration to split your syncs into multiple jobs and run them concurrently.
Check out the full post and documentation to see how you can leverage Automatic Sharding and make your syncs faster than ever.