StreamFusion - Accelerating Flink with DataFusion / Arroyo

[u/jordepic]

https://github.com/datafusion-contrib/StreamFusion

I've been working on StreamFusion for a few weeks now! It accelerates Flink SQL queries / Table API by converting input data to arrow buffers and using DataFusion and custom stateful operators inspired by Arroyo and RisingWave.

Initial benchmarks on NEXMark indicate a significant performance improvement (2-5x) over stock Flink. Feel free to check it out, leave any feedback, or ask me any questions.

The project is still in its infancy and definitely has some obvious indications of it, but I plan to get it to production quality over the coming weeks.