# Writing to DB in realtime with JS Plugin

**URL:** <https://community.cloudquery.io/t/writing-to-db-in-realtime-with-js-plugin/975>\
**Category:** CloudQuery Plugins\
**Created:** [December 7, 2024, 12:23am UTC](https://community.cloudquery.io/t/writing-to-db-in-realtime-with-js-plugin/975 "2024-12-07T00:23:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Duncan\_Mapes](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/duncan_mapes/32/85_2.png) [@Duncan\_Mapes](https://community.cloudquery.io/u/Duncan_Mapes)\
**Post date:** [December 7, 2024, 12:23am UTC](https://community.cloudquery.io/t/writing-to-db-in-realtime-with-js-plugin/975/1 "2024-12-07T00:23:37Z")

</div>

I have a custom plugin that pulling 10s of thousands of records, I am iterating API calls of 500 records each. After each 500, I am passing them to the stream.write function.

However, nothing is being written to the DB until after the entire table (destination is Postgres) has been processed. Is this the expected behavior? I have even tried to make the batch size very small to try and trigger it writing sooner. It seems it would be much more efficient to pass the data through so it doesn’t have to accumulate in memory.

Any comments would be appreciated.

---

<div class="post-metadata">

**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\
**Post date:** [December 9, 2024, 1:53pm UTC](https://community.cloudquery.io/t/writing-to-db-in-realtime-with-js-plugin/975/2 "2024-12-09T13:53:55Z")

</div>

Hi @Duncan_Mapes what you’re experiencing could be related to batch settings at the destination.  
There’s nothing on the JS SDK side that does buffering.  
So reducing batch size to 1 via `batch_size : 1` for the Postgres destination should cause rows to be committed immediately.  
The tradeoff here is between memory consumption and performance. It’s much faster to write to Postgres in batches. The default batch size was selected to work for most cases, you can experiment with different values to see what fits yours the best

---

<div class="post-metadata">

**Author:** ![Duncan\_Mapes](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/duncan_mapes/32/85_2.png) [@Duncan\_Mapes](https://community.cloudquery.io/u/Duncan_Mapes)\
**Post date:** [December 9, 2024, 9:44pm UTC](https://community.cloudquery.io/t/writing-to-db-in-realtime-with-js-plugin/975/3 "2024-12-09T21:44:31Z")

</div>

Hey @erez , thanks for replying,. I tried that, it doesn’t appear to write until the resolver is completed.

---

<div class="post-metadata">

**Author:** ![erez](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/erez/32/40_2.png) [@erez](https://community.cloudquery.io/u/erez)\
**Post date:** [December 10, 2024, 9:17am UTC](https://community.cloudquery.io/t/writing-to-db-in-realtime-with-js-plugin/975/4 "2024-12-10T09:17:07Z")

</div>

Hi @Duncan_Mapes can you open a bug report then on [GitHub - cloudquery/cloudquery: The open source high performance ELT framework powered by Apache Arrow](https://github.com/cloudquery/cloudquery)? If you can share a reproduction (via a GitHub repo would be best) that would be great

---

<div class="post-metadata">

**Author:** ![Duncan\_Mapes](https://sea1.discourse-cdn.com/flex001/user_avatar/community.cloudquery.io/duncan_mapes/32/85_2.png) [@Duncan\_Mapes](https://community.cloudquery.io/u/Duncan_Mapes)\
**Post date:** [June 26, 2025, 5:15pm UTC](https://community.cloudquery.io/t/writing-to-db-in-realtime-with-js-plugin/975/5 "2025-06-26T17:15:02Z")

</div>

done [bug: JS SDK - not properly supporting streaming · Issue #20910 · cloudquery/cloudquery · GitHub](https://github.com/cloudquery/cloudquery/issues/20910) six months later 🙂

might just switch to writing go though…not sure if this is a priority for you guys to fix in JS