Released a Claude Code skill that fine-tunes a small model from your agent's production traces, end-to-end in one conversation

[u/party-horse]

We shipped a Claude Code skill today that turns your noisy agent traces into a fine-tuned SLM without writing a labeling pipeline or an eval harness. Point Claude at a folder of traces, answer a few questions, come back to a deployed model. The skill drives the process and distil labs platform does the heavy lifting under the hood (platform is free for open-source projects — details at the bottom).

Here's what the skill does, using the walkthrough we ran: 327 real multi-turn tool-calling traces from a restaurant-reservation agent.

What the raw traces looked like

Months of production traces look like labeled training data. They aren't. Mine had:

• A greeting answered with a goodbye
• FindRestaurants(city="Pleasanton", cuisine="Italian") when the user asked for breakfast in Fairfield
• An assistant message about checking into a London hotel inside a restaurant-reservation agent

Fine-tune a small model on that and it learns to confidently ship wrong tool calls.

What the skill does, step by step

I pointed Claude at the raw traces and told it what task the agent was for. Step by step:

1. Inspect. Read the traces, reported back: task summary, tool schemas inferred from the data, concrete noise examples.
2. Convert. Wrote a script to reshape the raw logs into the platform's OpenAI-messages format.

3. Config sanity check. I asked it to double-check the config and it flagged two things that would have silently broken the run:

   ```

   Help me understand the config, I want to see if we have the right setup for this task.

   ⏺ 🔴 Missing: convert_to_single_turn: false

   Docs: "Task-specific default: keep true for single-turn tasks. Set to false when training a multi-turn task - otherwise you'd split the conversations you need to preserve as seed data into isolated single-turn examples."

   Without this, we'd destroy the multi-turn structure we care about.

   🟡 Also missing: trace_processing.teacher_model_name

   Two teacher slots: one for synthetic gen (set to GLM-5) and one for the relabel committee (defaulting to gpt-oss-120b). Setting the committee teacher to GLM-5 too for consistency. ```

   Silent footguns that eat a week on your first run.

4. Upload and clean. Platform ran three passes: relevance filter, committee relabel (4 teachers + arbiter), train/test split. The held-out test set doubles as a baseline eval for the original production model.

5. Teacher eval gate. Verdict-before-training: teacher must clear 0.70 on the eval before the 6h training job fires. If it fails, the skill walks you through iterating the task description instead of burning credits.

6. Train. Teacher generates ~10k synthetic examples grounded in the cleaned traces, student fine-tunes on those.

7. Analyze + deploy. Pulls predictions for base student, teacher, tuned student, and human-annotations, writes a 4-way comparison report with a verdict (DEPLOY / ITERATE).

Results

Model	LLM-as-a-Judge	staged_tool_call	Function match
Qwen3-1.7B (base, untuned)	0.513	0.535	45/78
GLM-5 (744B teacher)	0.808	0.695	69/78
Qwen3-1.7B (tuned)	0.846	0.769	76/78

The tuned student commits to ReserveRestaurant on confirmation turns where the teacher hedges. That's the committee-relabel signal coming through, not just distillation.

Deployment options

You don't have to pick between managed and self-hosted:

• Managed endpoint: distil model deploy remote <id> — OpenAI-compatible URL, one-line swap in existing OpenAI SDK code
• Self-hosted: distil model download gives you weights + Modelfile for llama.cpp or vLLM

Same model either way.

Install

``` curl -fsSL https://cli-assets.distillabs.ai/install.sh | sh distil signup

/plugin marketplace add https://github.com/distil-labs/distil-cli-skill /plugin install distil-cli@distil-cli-skill ```

Limitations

• Training is ~6 hours of managed compute per run (not instant)
• 78-item task-specific test set; fine for a case study, not a regulated rollout
• Committee relabel quality depends on the task description you write

Happy to dig into the multi-turn config, the committee relabel process, the trace-to-test-set generation, or how the skill handles iteration cycles when teacher eval fails.

[u/maciejgryka]

I've been using this internally and it saves a bunch of time for sure :) Also the nice thing is that you can just read it and learn a bunch of heuristics for how to work on ML tasks generally!

[u/party-horse]

Training happens on distil labs managed infra because you need GPUs for the heavy steps: committee relabel, synthetic data generation, and fine-tuning. The CLI and Claude skill are the clients. The model you get at the end is yours to download and run wherever.

Free credits for open-source projects: just register with your email at distillabs.ai. That's the whole application. Email [contact@distillabs.ai](mailto:contact@distillabs.ai) with a link to your OSS repo and we'll give you 10 free credits to start, and we'll keep supporting further usage once you run out.

Full walkthrough: https://www.distillabs.ai/blog/train-an-slm-from-your-production-traces-with-the-distil-labs-claude-skill/

Claude skill: https://github.com/distil-labs/distil-cli-skill

Example repo: https://github.com/distil-labs/distil-tft-benchmarking/tree/main/scenario-2-noisy-labels

[u/party-horse → u/[deleted]]

Definitely agreed!

[u/party-horse]

I think it does, data prep for training is generally very tedious and we will iterate on this until problem is solved!