When adding an array of 10 items in the channel directly, all items are processed in parallel (up to the child table concurrency setting) but if adding items one at a time, they are treated synchronously, waiting for all the child relations to complete before processing the next.
I’ve tried to buffer the items before adding them to the channel as a workaround but this is not great as the slowest processed item would make all of the other items in the batch wait.
Thank you for the quick response!
I’ve tested the shuffle-queue and it did work, it seems this strategy doesn’t have the same limits as the DFS one, does it?
I didn’t expect the behavior to be different when passing an array or an item to be honest so this surprised me and thought it could be a bug.
I think having some documentation / comment could be helpful for plugin developers.
I’ve now updated my code to iterate manually on the results and getting way faster syncs for tables that have relations with the DFS scheduler.
Agree this can be considered a bug. We could fix it by making the channel a buffered (let’s say size of 20), but then if someone passes multiple arrays (e.g. each array size of 100) that can increase memory usage.
The shuffle-queue scheduler uses a worker queue pattern so each child table does not resolve resources immediately, but instead the resources are inserted into a queue (which is fast), to be later be picked up for resolving by a free worker.
This pattern allows distributing the work more efficiently
I’ll add that to our docs