Fork-Union for parallelization?

[varchasgopalaswamy]

I came across this project which seems quite interesting. The author claims substantial speed-ups compared to using Rayon for parallelization. Does this seem like something worth investigating?

[akern40]

There are some really impressive numbers there! Thank you for sharing. I would welcome some experimental code, e.g., an experimental branch, that tried to look at how ForkUnion could improve the performance of our parallel iterators. I don't have the time right now to dedicate to it, but if you're interested I'd be happy to review and discuss!

[nilgoyette]

I read their readme and I can't find "Microsoft" or "Windows" anywhere. Did I miss it? I understand that this feature would be locked behind a feature gate, but if the most used OS is not supported, I don't think it's worth putting much effort on this project. At least, not with the current throughput we have.

Impressive numbers indeed. I'm curious to test on my non-ndarray projects!

[varchasgopalaswamy]

I can't say I've ever programmed on Windows, so I didn't think to check (and wouldn't know where to start even if I did!) I see this issue ashvardanian/ForkUnion#26 suggesting that Windows support was added.

@akern40 I don't have time myself right now, but I am adding rust based acceleration to my work projects so I could in the future. Frankly, ndarray+Rayon is already giving me ~ order of magnitude speedup over my previous code with very little pain, so I'm already quite happy!

[akern40]

Well I'm glad to hear that the codebase is delivering! Let's leave this issue open as a marker that this might be worth returning to?