# Running CloudQuery plugin recursively for large dataset retrieval **URL:** \ **Category:** CloudQuery Plugins\ **Created:** [May 20, 2024, 5:55pm UTC](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202 "2024-05-20T17:55:18Z")\ **Posts on this page:** 11\ **Page:** 2
**Author:** ![natural-beagle](https://avatars.discourse-cdn.com/v4/letter/n/c6cbf5/32.png) [@natural-beagle](https://community.cloudquery.io/u/natural-beagle)\ **Post date:** [May 21, 2024, 2:38pm UTC](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202/21 "2024-05-21T14:38:04Z")
Thanks so much, sir. Would you check this please? I am saving the result after fetching all pages, and am using Mongoose. Is this the right approach? @erez ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [May 21, 2024, 3:18pm UTC](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202/22 "2024-05-21T15:18:15Z")
Can you explain where you are trying to get data from and where you want to save it? Usually, sources and destinations are separate plugins. For example, if you write a custom source, you would do `stream.write` in the source, then configure a destination separately (e.g. [CloudQuery MongoDB Destination](https://hub.cloudquery.io/plugins/destination/cloudquery/mongodb/latest/docs)). Are you using `findOneAndUpdate` to avoid re-syncing the same data over and over again? ---
**Author:** ![natural-beagle](https://avatars.discourse-cdn.com/v4/letter/n/c6cbf5/32.png) [@natural-beagle](https://community.cloudquery.io/u/natural-beagle)\ **Post date:** [May 21, 2024, 3:20pm UTC](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202/23 "2024-05-21T15:20:14Z")
To save only one item in the collection, it saves the timestamp to get modified items. ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [May 21, 2024, 3:22pm UTC](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202/24 "2024-05-21T15:22:47Z")
I think that’s OK but not so much related to the CloudQuery SDK API. You need to make sure to do ```python stream.write ``` inside `fetchCVEList`. ---
**Author:** ![natural-beagle](https://avatars.discourse-cdn.com/v4/letter/n/c6cbf5/32.png) [@natural-beagle](https://community.cloudquery.io/u/natural-beagle)\ **Post date:** [May 21, 2024, 3:23pm UTC](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202/25 "2024-05-21T15:23:18Z")
Yes, I did. @erez Above is saving documents which have the same `id` value. Any solution, please? I want to update the document which is already inserted. ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [May 21, 2024, 3:37pm UTC](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202/26 "2024-05-21T15:37:00Z")
Can you share how you are running the plugin and the CloudQuery CLI? And the configuration spec you’re using? ---
**Author:** ![natural-beagle](https://avatars.discourse-cdn.com/v4/letter/n/c6cbf5/32.png) [@natural-beagle](https://community.cloudquery.io/u/natural-beagle)\ **Post date:** [May 21, 2024, 3:38pm UTC](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202/27 "2024-05-21T15:38:01Z")
```auto kind: source spec: name: 'sync' registry: 'grpc' path: '127.0.0.1:7777' version: 'v1.0.0' tables: ['*'] destinations: - 'mongodb' spec: connectionString: mongodb://127.0.0.1:27017 database: cloudquery-test --- kind: destination spec: name: mongodb path: cloudquery/mongodb registry: cloudquery version: 'v2.3.11' write_mode: "append" spec: connection_string: mongodb://127.0.0.1:27017 database: cloudquery-test ``` ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [May 21, 2024, 3:38pm UTC](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202/28 "2024-05-21T15:38:04Z")
Depending on the destination, if you define a primary key for the column based on `id`, the item will get updated instead of duplicated. ---
**Author:** ![natural-beagle](https://avatars.discourse-cdn.com/v4/letter/n/c6cbf5/32.png) [@natural-beagle](https://community.cloudquery.io/u/natural-beagle)\ **Post date:** [May 21, 2024, 3:38pm UTC](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202/29 "2024-05-21T15:38:39Z")
I set like this: ```auto [Your settings or configuration here] ``` But still duplicating. ---
**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\ **Post date:** [May 21, 2024, 3:39pm UTC](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202/30 "2024-05-21T15:39:44Z")
That’s because you have `write_mode: "append"`. You should remove that configuration to use the default `write_mode: "overwrite-delete-stale"`. More in [CloudQuery Documentation](https://docs.cloudquery.io/docs/reference/destination-spec#write_mode). ---
**Author:** ![natural-beagle](https://avatars.discourse-cdn.com/v4/letter/n/c6cbf5/32.png) [@natural-beagle](https://community.cloudquery.io/u/natural-beagle)\ **Post date:** [May 21, 2024, 3:45pm UTC](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202/31 "2024-05-21T15:45:27Z")
Can’t thank you enough. [Previous page](https://community.cloudquery.io/t/running-cloudquery-plugin-recursively-for-large-dataset-retrieval/202.md?page=1)