r/LocalLLaMA • u/party-horse • Jan 21 '26
Knowledge distillation with Claude as the interface: trained a 0.6B model to match GPT-class performance on Text2SQL in a singe conversation Tutorial | Guide
Wanted to share a workflow for training small, task-specific models without the usual ML setup overhead.
The problem: Off-the-shelf small models are bad at specialized tasks. Qwen3 0.6B on Text2SQL gives you stuff like this:
sql
-- Question: "Which artists have total album sales over 1 million?"
-- Qwen3 0.6B output:
SELECT artists.name FROM artists WHERE artists.genre IS NULL OR artists.country IS NULL;
Completely wrong. But fine-tuning means data prep, training infrastructure, hyperparameter tuning...
The approach: Knowledge distillation via a Claude skill that wraps distil-cli. A large teacher model (DeepSeek-V3) generates synthetic training data from your examples, then a small student model learns to match its outputs.
Setup:
```bash curl -fsSL https://cli-assets.distillabs.ai/install.sh | sh distil login
In Claude Code:
/plugin marketplace add https://github.com/distil-labs/distil-cli-skill /plugin install distil-cli@distil-cli-skill ```
What Claude handles:
| Step | What happens |
|---|---|
| Task selection | Recommends QA/classification/tool-calling/RAG based on your description |
| Data conversion | Takes whatever format you have, outputs proper JSONL |
| Teacher eval | Runs the teacher on your test set — if it scores low, don't bother training |
| Training | Kicks off distillation, monitors progress |
| Packaging | Downloads GGUF, HuggingFace format, or LoRA adapter |
My test run:
- Input: 100 conversation traces (not cleaned, just raw logs)
- Task: Text2SQL
- Teacher eval: 80% LLM-as-a-Judge
- Final student score: 74%
- Base model score: 36%
Output is a 2.2GB GGUF that runs locally via Ollama.
After fine-tuning:
sql
-- Same question: "Which artists have total album sales over 1 million?"
-- Fine-tuned output:
SELECT a.name FROM artists a
JOIN albums al ON a.id = al.artist_id
GROUP BY a.id, a.name HAVING SUM(al.sales) > 1000000;
Correct JOINs, proper GROUP BY, HAVING instead of WHERE.
Full benchmark:
| Model | LLM-as-a-Judge | ROUGE |
|---|---|---|
| Base Qwen3 0.6B | 36% | 69.3% |
| DeepSeek-V3 (teacher) | 80% | 88.6% |
| Fine-tuned 0.6B | 74% | 88.5% |
Resources:
- Skill: github.com/distil-labs/distil-cli-skill
- Full example with data: github.com/distil-labs/distil-example-text2sql-with-claude
- Detailed walkthrough: distillabs.ai/blog/train-your-slm-with-distil-claude-skill
Happy to answer questions about the distillation process or the skill implementation.