Describe the bug
RoundRobin repartitioning currently does not distribute the input tasks evenly over the output channels, causing the work to be not distributed evenly.
To Reproduce
When loading the data in memory in the TPC-H benchmark, this can be seen in the number of batches in MemoryExec (which uses RoundRobin partitioning).
MemoryExec: partitions=32, partition_sizes=[32, 32, 32, 32, 32, 32, 32, 32, 26, 26, 26, 25, 25, 25, 25, 25, 25, 25, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16], metrics=[]
It has a bias for the first output partitions/channels.
Expected behavior
Batches should be distributed more evenly over output channels.
Additional context
No response
Describe the bug
RoundRobin repartitioning currently does not distribute the input tasks evenly over the output channels, causing the work to be not distributed evenly.
To Reproduce
When loading the data in memory in the TPC-H benchmark, this can be seen in the number of batches in
MemoryExec(which uses RoundRobin partitioning).MemoryExec: partitions=32, partition_sizes=[32, 32, 32, 32, 32, 32, 32, 32, 26, 26, 26, 25, 25, 25, 25, 25, 25, 25, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16], metrics=[]It has a bias for the first output partitions/channels.
Expected behavior
Batches should be distributed more evenly over output channels.
Additional context
No response