Currently, the behavior seems to be to assign a thread per node, and then try and collapse if there's a forward edge between nodes and operator chaining is enabled.
To reduce overheads of communication between threads, and enable horizontal scalability even with lesser number of threads, we can explore how we could shard the data and run the entire pipeline on each thread, on a specific shard of the data
Currently, the behavior seems to be to assign a thread per node, and then try and collapse if there's a forward edge between nodes and operator chaining is enabled.
To reduce overheads of communication between threads, and enable horizontal scalability even with lesser number of threads, we can explore how we could shard the data and run the entire pipeline on each thread, on a specific shard of the data