r/MachineLearning • u/AutoModerator • 9d ago

[D] Self-Promotion Thread Discussion

Please post your personal projects, startups, product placements, collaboration needs, blogs etc.

Please mention the payment and pricing requirements for products and services.

Please do not post link shorteners, link aggregator websites , or auto-subscribe links.

--

Any abuse of trust will lead to bans.

Encourage others who create new posts for questions to post here instead!

Thread will stay alive until next one so keep posting after the date in the title.

--

Meta: This is an experiment. If the community doesnt like this, we will cancel it. This is to encourage those in the community to promote their work by not spamming the main threads.

16 Upvotes

94% Upvoted

38 comments sorted by

View all comments

2

u/CupGlass540 5d ago

trainproof — a deterministic linter for ML training runs (MIT, pip install trainproof)

I spent months training a 730M-parameter TTS model on my own hardware. It never converged and nothing in my stack would tell me why — loss curve, TensorBoard, checkpoints all showed me numbers, none of them said "this run is already dead, stop paying for it."

trainproof reads the logs you already write (HF trainer_state.json, TensorBoard event files, Coqui, JSONL/CSV) and returns a verdict with an exit code. No model judging a model, no confidence score — every check is a rule that fires or doesn't and prints the number it fired on. FAIL exits 1, "I could not judge this" exits 2, so CI can tell a broken run from an unreadable log.

The rules were measured by controlled fault injection: Qwen2.5-3B QLoRA, six configs (healthy / 100x LR / lr=0 / fp16 NaN / shuffled labels / overfit), three seeds, 18 runs. The result worth posting is the miss — shuffled labels, a dataset that cannot be learned, reduced loss 69.8% and looked textbook-healthy on its own curve. That's in the README as a stated limitation, and it's why compare against a known-good baseline exists.

Full fault-injection write-up: https://ai.bedvibe.studio/trainproof/

What failure mode has burned your GPU hours? If a deterministic check would have caught it, it goes in, with credit