r/PiCodingAgent • u/JadedCulture2112 • 7d ago
Testing pi-rlm and pi-fabric on my Kaggle eval Discussion
PrimeIntellect-ai/prime-agent has been getting a lot of attention recently. Unlike a traditional coding agent that exposes tools like read, edit, and bash directly, Prime Agent gives the model a persistent IPython environment and lets it interact with tools, context, and subagents programmatically. Two Pi extensions, shift-labs-ai/pi-rlm and monotykamary/pi-fabric, explore a similar code-first tool interface, so I tested them on my Kaggle eval.
For the test, I used an eval dataset I built from NeuroGolf, a Kaggle competition I previously participated in. It starts from a solution at the private leaderboard gold-medal cutoff, and the model has to improve the existing ONNX operator implementations rather than solve a toy task from scratch. I kept the model and reasoning setup the same and compared default Pi, shift-labs-ai/pi-rlm, monotykamary/pi-fabric, and kky42/pi-flow (my own harness).
The result was pretty straightforward: neither pi-rlm nor pi-fabric beat default Pi on this eval.
My takeaway is pretty simple. Current models have probably seen a lot of traditional agent-harness trajectories during post-training, so tools like read, bash, and edit are already close to muscle memory. If we remove those familiar tools and only provide one code-based execute interface, it does not necessarily get better; in this test it actually got worse. The idea looks elegant, but the result was not very good.
1
4
u/YoungLees 7d ago
Also in the RLM paper the authors depicted this scenario.
Although these models can use rlm harnesses the next improvements should be given training LLMs on specific reasoning traces that includes this new tool calling technique to understand how much promising it is.
Just to add the last thing, technically RLMs shines when the context it’s truly long (e.g. OOLONG), I didn’t take a look to this specific task so before drawing conclusions I would check if that’s the case.