Running CloudQuery plugin recursively for large dataset retrieval

CloudQuery Plugins
21

Thanks so much, sir.
Would you check this please?
I am saving the result after fetching all pages, and am using Mongoose.
Is this the right approach?
@erez

22

Can you explain where you are trying to get data from and where you want to save it?

Usually, sources and destinations are separate plugins. For example, if you write a custom source, you would do stream.write in the source, then configure a destination separately (e.g. CloudQuery MongoDB Destination).

Are you using findOneAndUpdate to avoid re-syncing the same data over and over again?

23

To save only one item in the collection, it saves the timestamp to get modified items.

24

I think that’s OK but not so much related to the CloudQuery SDK API. You need to make sure to do

stream.write

inside fetchCVEList.

25

Yes, I did.
@erez
Above is saving documents which have the same id value.
Any solution, please?
I want to update the document which is already inserted.

26

Can you share how you are running the plugin and the CloudQuery CLI?
And the configuration spec you’re using?

27 
kind: source
spec:
  name: 'sync'
  registry: 'grpc'
  path: '127.0.0.1:7777'
  version: 'v1.0.0'
  tables: ['*']
  destinations:
    - 'mongodb'
  spec:
    connectionString: mongodb://127.0.0.1:27017
    database: cloudquery-test
---
kind: destination
spec:
  name: mongodb
  path: cloudquery/mongodb
  registry: cloudquery
  version: 'v2.3.11'
  write_mode: "append"
  spec:
    connection_string: mongodb://127.0.0.1:27017
    database: cloudquery-test
28

Depending on the destination, if you define a primary key for the column based on id, the item will get updated instead of duplicated.

29

I set like this:

[Your settings or configuration here]

But still duplicating.

30

That’s because you have write_mode: "append". You should remove that configuration to use the default write_mode: "overwrite-delete-stale".

More in CloudQuery Documentation.

31

Can’t thank you enough.