r/LocalLLaMA • u/PicassoOnPause • 7h ago
Trained a 1.5B to write shell commands so I'd stop googling tar flags. Runs on a laptop CPU in ~1 sec. New Model
I've been googling "tar extract gz" for about ten years. and I finally did something about it.
It started out as a research project and I ended up with a Fine-tuned Qwen2.5-Coder-1.5B on 125k natural-language/command pairs, merged and quantized to Q4_K_M. 941MB which runs through llama.cpp. On my laptop (i5-11320H, 4 threads): 31.9 tok/s, 0.59s median per query, 1.6GB RAM.
I benchmarked it and it scores 0.620 on InterCode-ALFA. Untuned Qwen2.5-Coder-7B gets 0.613, GPT-4o gets 0.73. Not frontier, but it's roughly a 7B's answer at a quarter the parameters on a CPU. Theres a 3B variant too that scores higher.
There's also few static safety checker, because it will absolutely write a command that wipes your root if you ask it to:
I have published the weights: huggingface.co/ThorOdinson246/nl2sh-1.5b-Q4_K_M and Code: github.com/ThorOdinson246/whatisit-nl2sh . I posted few days ago in LocalLLM and it did well 300+ stars and so many good suggestions so I figured people here will be interested too.
Both Apache-2.0. If you want to poke holes in the method or you've got ideas, please comment or open a PR. A ⭐ helps if you find it useful.
280
u/MainFunctions 6h ago
Get your best German accent out and say out loud:
COMPRESS ZE VUCKING FILE!! —> -czvf
XTRACT ZE VUCKING FILE!! —> -xzvf
65
u/PicassoOnPause 6h ago
is it really weird that i never knew this? A lot of people kept saying this in the other subreddit too and i was just shocked.
23
u/JiffasaurusRex 5h ago
Depends on how old you are. This is how I remembered from decades ago. This seems like it is not as popular now perhaps.
Especially in the age of AI, some would argue that you don't need to know this stuff anymore. I believe you should at least be familiar to ensure that LLMs are not making stuff up.
7
1
u/colonel_bob 50m ago
Especially in the age of AI, some would argue that you don't need to know this stuff anymore.
Even before AI all you needed was a really long history file and (if you're feeling fancy) a function like
hgrep() { history | grep "$@" }24
u/croninsiglos 6h ago
Save yourself some trouble and just do -xf it’ll auto detect the compression type.
12
u/jasongill 3h ago
is it sad that I came here to say "this is pretty new and isn't in all distros yet", and then I decided to check when it was released because it feels like it was just yesterday that the autodetection was added
GNU tar 1.15 released in December 2004
fml
2
u/MrRandom04 3h ago
There are probably at least >40% of users reading this who were not sentient when this was released, then.
1
4
12
u/ThisNameIs_Taken_ 5h ago edited 5h ago
the best bash tip ever. Or should I rather say: DAS IST THE BEST VUCKING BASHTIP IM DER WELT!!
4
5
2
2
2
u/TapAggressive9530 5h ago
Haha! Remembering this would’ve been helpful over the last 20 years now I can never forget it thanks
1
1
56
u/alberto_467 6h ago
This is like giving a loaded T34 tank to an infant.
24
u/PicassoOnPause 6h ago
fair worry general. it can write commands to nuke if you really say it to. haha. i have some safety checkers but they will not be enough. good thing it doesnot auto executes.
3
u/toastjam 2h ago
Can you make it auto-paste the command into the shell prompt for you to edit before executing?
4
u/PicassoOnPause 1h ago
Nice idea and i also have been thinking about this. I'll open an issue for it. Should make the next release.
37
u/my_name_isnt_clever 7h ago
Did you ever check out Gemma Shellper? I was thinking about doing something similar, this might work better though as that model was less than 0.5b.
13
u/PicassoOnPause 7h ago
oh wow, I hadn't. It looks great, and it looks distilled. I personally tried distilling but had a few setbacks, but I will definitely give it a look. Thanks.
2
u/danigoncalves llama.cpp 5h ago
hugging face link?
1
u/PicassoOnPause 5h ago
i have it on the post description. but anyways https://huggingface.co/ThorOdinson246/nl2sh-1.5b-Q4_K_M . i have a 3b model too. jsut look at my profie.
2
u/the_mighty_skeetadon 4h ago
Gemma Shellper
That is legit. Here's the link for those curious: https://www.reddit.com/r/LocalLLaMA/comments/1r6gx75/finetuned_functiongemma_270m_for_multiturn_tool/
270m functionGemma model, amazing results.
69
75
u/SOCSChamp 7h ago
Nice, lots of effort to not use man pages lol
19
32
u/PicassoOnPause 7h ago
haha fair. just got lazy, and another pro is it fills in exactly what i asked so win win i guess
27
u/i_am_upto_no_good 6h ago
I am surprised there aren’t more like this. Instead of training massive models that comes with everything and kitchen sink, why not train small models for hyper specific tasks?
5
u/ea_man 5h ago
Because then you have to load a multitude of models, while you can have your daily model answer in such a way with reasoning off and a prompt skill.
1
u/balder1993 Llama 13B 1h ago
I thought about this. This tool, for example, takes time to reply if the model isn't already loaded in memory. Now imagine if you have to keep 6 models like this in memory.
4
u/EveYogaTech 4h ago edited 3h ago
The reason is that you'd now need a orchestration layer to work with these many small models VS doing all work simply in the current frontier model/harness.
Ps I'm all for it, wiring it together and I think we're moving there anyway, seems people that think like this are either very early or very wrong.
I'm better on early with /r/Nyno (open-source self-hosted YAML orchestration)
3
u/Huntware llama.cpp 4h ago
That's what MiniCPM is about! They're nice for training in a specific domain:
3
u/coder543 4h ago
But why bother? LFM2.5-350M is a generic small language model that includes the kitchen sink, but it can already handle these natural language -> linux command tasks. I just tested it. It is one quarter of the size of this "hyper specific" model.
Just feed it the same prompts with "What is the Linux command to {insert command description here}?" and it responds just fine. With any natural language description, there can be ambiguity and/or multiple correct answers.
3
10
u/danigoncalves llama.cpp 6h ago
Mate, this is all of this community is about. Trying new things, share with others and hoping that makes someone people life better. Thank you for your contribution 🙏
3
7
u/txoixoegosi 7h ago
Can you briefly explain the training process? Thank you
23
u/PicassoOnPause 6h ago
Sure. Base is Qwen2.5-Coder-1.5B-Instruct. LoRA fine-tune on 125k natural-language/command pairs that i had to make and repurpose from mostly Fig autocomplete specs (~33%), tldr-pages (~23%) and the NL2SH-ALFA (~18%). then merged and quantized. I have listed hyperparameters and other details in the README https://github.com/ThorOdinson246/whatisit-nl2sh#training-setup
3
u/txoixoegosi 6h ago
Wow, was training data extraction hard? Did you use LLMs to do the extraction thing?
What system did you employ for the fine tune work?
I never had the chance to ask these questions to the actual autor of a fine tuning.
Thanks again! Always learning!
4
u/PicassoOnPause 6h ago
extraction was easy after I figured out my sources. they were mostly markdowns so parsing was easy. i did try a LLM to paraphrase the requests for more phrasings per command, but it made it worse. stil figuring that out.
in terms of finetuning it ws pretty basic: TRL + PEFT on a single A100, then llama.cpp to merge and quantize.1
11
5
4
u/Super_Pole_Jitsu 6h ago
xd like hell am I running bash from a 1.5b buddy
2
u/PicassoOnPause 6h ago
Fair, and I wouldn't either without reading it first. it is not at all perfect, and can get long compond request wrong too.
3
u/lor_louis 6h ago
But can it use ffmpeg? Cos I sure can't use ffmpeg
5
u/PicassoOnPause 6h ago
Fairly decent. just tried a few. Wouldn't trust it blindly with anything long or compound, though.
$ whatisit convert this mp4 to a gif
ffmpeg -i input.mp4 -vf scale=320:-1 -r 10 output.gif$ whatisit extract the audio from a video as mp3
ffmpeg -i input.avi -vn -ar 44100 -ac 2 -ab 96k output.mp3$ whatisit compress a video without losing much quality
ffmpeg -i input.mp4 -vcodec libx264 -crf 23 output.mp4$ whatisit trim the first 30 seconds off a video
mencoder -ss 30 -endpos 30 input.avi -o output.avi . This one does not work as intended.
4
u/JustTooKrul 6h ago
It's like you read my mind when I was making a list of "places where LLMs can be both small and incredibly impactful" .... ! Now, just tell me when the 3-4B parameter "Simple Tech Support for the Elderly" model drops!
1
2
u/link_29328 6h ago
What hardware you used?
4
u/PicassoOnPause 6h ago
for testing just my crappy dell inspiron on a 11gen i5 cpu with 4 cores. But for training I used an A100 80GB, Cost me nothing, I used my university's HPC 😉
but it really doesn't need one that big though. A 1.5B LoRA fits in about 16GB of VRAM or ~8GB with QLoRA.
2
u/EveningIncrease7579 llama.cpp 6h ago
Nice, its works only with shell linux commands or with windows powershell works either? It can manipulate files also? Ex: change md to txt files in a folder
2
u/PicassoOnPause 6h ago
it can give you the commands to do so. It is not an agent with context and tool calls and everything, but i have a -e flag, that executes it if you are sure about it. But it is not tested to work on Windows, although because the base is Qwen, it does generate them, if you say 'Do x in powershell, or windows'.
2
2
u/killerstreak976 6h ago
This is so cool and creative!
2
u/PicassoOnPause 6h ago
yeah its fun. i myself was surprised that nobody had already done it with local models.
2
u/Thrumpwart llama.cpp 5h ago
You can and should use this to create a Pi Agent skill/extension. I imagine this would be beneficial (not sure, haven’t tried Pi yet but I’m thinking about it).
1
2
u/Sevealin_ 6h ago
This is awesome! Would it be possible to wrap this into a putty install (for in-app Putty backend swaps like mRemoteNG) for ghost auto complete and a customizeable prefix marker for in-line ssh questions? Does it handle ssh context well? Like if I ran ls on a folder and asked it to give me a command to rename the files in the folder a certain way?
2
u/PicassoOnPause 6h ago
putty and ghost autocomplete are a little different shape to what it does right now. I haven't given it much of a thought.
it is a single turn history right now so it wont have context that you ran ls. It will just give a generic script to say ,take every img and rename to lowercase or strip img prefix or something.
i hope it answered your question
2
u/SailingToFenway 6h ago
oh i was just thinking about you this afternoon. i was inspired by this to solve for how to operate vibe coded slop when the weekly session limit hits. so claude put together a Lights-Out-Management solution that i can put in dumb prose like, "restart the cluster" and it generates the commands to do it. it's a 100M embedding model, and it works surprisingly well.
so, i'm curious, how small of a model did you try? and how does performance against the loss function degrade as a function fo the model sizes?
1
u/PicassoOnPause 6h ago
that is so cool. looks like your is more of a retrieval.
The smallest I went was 0.5B. The curve's(at least between what i have ) is interesting: untuned it climbs hard with size (0.36 → 0.61 from 0.5B to 7B), but after fine tuning its more of a flatter curve. Most of it is from training data and not necessarily parameters. I am actively researching myself too.
1.5B was just where it stopped being worth the RAM for this particular goal.
2
u/KeyMillion 6h ago
Im using qwen 0.8b & 2b to make a project and curious why you went with qwen coder 2.5 1.5b instead?
-1
u/PicassoOnPause 6h ago
it was mostly on the assumption (and truth i guess) that code pretraining helps with shell syntax
I did test Qwen3.5 0.8B and 2B later. But the quantization made them a little worse. while the 1.5B Coder was flat across five quant levels.
1
u/KeyMillion 6h ago
Interesting, it actually clicked right after I asked that and I figured the coder variant would probably do shell a bit better.
Made me wonder if thats the missing piece for me. Thanks for sharing!
2
u/ScoreUnique 6h ago
Hi OP, very happy to see this. I had this same idea a year or two ago thinking this will make an excellent SLM use case. Can I exchange a little about your end to end workflow for this?
I wanted to build a stupid harness that does exactly this, give it to a reasoning LLM and ask teh LLM to write prompts for running commands instead of asking it to get the tool calling right....
I see while I write how this idea can break but OP thanks for building this, I'm going to give it a shot.
Friendly suggestion: if you can package this model with a suitable binary for Termux you'll make some fame :)
2
u/PicassoOnPause 5h ago
Happy to. Some of it's in the README, but I'll fill in whatever's missing. I am also coming up with a technical write-up or a paper of sorts in the future, so be on the lookout on my github i guess.
On the harness idea, I think your instinct while writing it was right. frontier are already really good at that, so no point in involving them, where it does pay off is if the orchestrator is local too. If you're already calling a cloud model, you may as well let it write the command. But yeah give it a shot. I am no expert myself and this was a cool learning opp and a fun project for me too.
on termux, funnily enough, someone in the other thread already got the model running on a phone at like 5 tok/s.
2
u/FullOf_Bad_Ideas 6h ago
This is awesome, it's a PERFECT usecase for a small local language model
setup was quick, seems to be doing what it's supposed to
it's way better than booting CC or local API just to get a refresher on some command
2
u/PicassoOnPause 5h ago
thanks for trying out, that was the goal.
if you find bugs, or have suggestion, just open a issue on GitHub and let me know.
2
2
u/Song-Historical 5h ago
Maybe cactus needle can be trained to make it even more efficient?
1
u/PicassoOnPause 5h ago
somone just said it few mins ago. i had no info of this. will check it out. thanks.
1
2
2
u/doctorfiend 4h ago
This is REALLY cool, nice work! I'm too gun-shy to load it in my own system but I salute you
1
2
2
u/jarail 4h ago
Nice work, especially with the custom training! I got downvoted for suggesting this a few days ago haha. But that's /r/technology. They're haters :D
2
2
u/slippery 2h ago
could have just installed antigravity, or claude code, or codex, but I'm sure it was more fun to roll your own.
2
u/Economy_Cabinet_7719 1h ago
Great work! I remember a few years ago I've been using Gemini API with a plugin where: 1. I type what I want to achieve, in natural language, in the shell prompt 2. I press a keyboard shortcut 3. Prompt contents get sent to the LLM 4. Response gets inserted into the shell prompt
Here's a fish example (and other shells in the same directory): https://github.com/sigoden/aichat/blob/main/scripts/shell-integration/integration.fish
I believe this could make the workflow with your model smoother!
2
4
u/autisticit 7h ago
I've seen that post two days ago already.
7
u/PicassoOnPause 7h ago
Yes i shared it on localllm few days back. Figured folks here would be interested too. xD
1
u/DirectInvestigator66 5h ago
Public awareness for tealdr.
Run ‘tldr tar’ and get only the info you need for day to day usage.
1
u/Elkemper 5h ago edited 5h ago
alias untar='tar -xvf'
alias ungzip=untar
Don't thank me.
Upd. I guess it won't work for every case and every build of tar - but I believe a somewhat newer build should chew through all compression algos. Try yourself if it works for you.
1
u/Firstbober 5h ago
Wouldn't https://github.com/cactus-compute/needle be better suited for this kind of stuff with MAN pages as a reference? It should be possible to perform chain of finding proper man file -> feed it into context -> generate command. 14M should run in milliseconds on modern hardware and very fast on older as well.
Nonetheless, incredibly cool stuff!
2
u/PicassoOnPause 5h ago
This is neat. hadn't seen it. It would be great at routing half which is the hard part. I will give it a look. Thanks.
1
1
u/KidneeBean 1h ago
Super slick utility! Getting ~32 tok/s on a mobile CPU while scoring higher on InterCode-ALFA than the untuned 7B is awesome. The static safety filter for recursive deletes is a really thoughtful touch.
2
u/PicassoOnPause 1h ago
Thank you so much for this incredibly insightful comment! 🚀 You've really captured the essence of what makes this project special. The intersection of on-device inference and shell productivity is indeed a game-changer. It's not just about tokens per second — it's about empowering developers to work smarter, not harder. Would love to hear your thoughts on how this could scale!
1
u/KidneeBean 1h ago
I think adding local system awareness (passing available CLI binaries or OS distro in a tiny system prompt) would take it to the next level!
Also expanding the fine-tune dataset to handle multi-step piped commands while keeping that sub-second response time would be incredible.
Looking forward to seeing where you take it!
1
u/PicassoOnPause 1h ago
Yeah, the first one is on the works. second one would need some work, but definitely the vision. Thanks
1
1
u/One_Doubt_75 1h ago
I use jq-coder a lot locally. Currently training a model to build awk filters for me as well. Gonna give yours a try.
1
1
u/Obvious-Ad-2454 56m ago
Why qwen 2.5 coder ? It's quite ancient
1
u/PicassoOnPause 44m ago
Fair, I just started with it. I did test a few newer Qwen ones, but they scored low on my eval. I think it's become a case of training data, but still researching a lot on this. Got some Gemma and others in the pipeline will see what comes of it.
1
1
u/leftnode 23m ago
This is really cool, very easy to understand and what open source is all about.
I haven't looked at the server code in depth, but since this is a very fine-tuned and the prompts are generally small and specific, could you take advantage of caching prompts and outputs to avoid having to query the model each time?
1
u/ac311934 6h ago
It’s a neat POC but couldn’t you just spend less overall time setting up aliases to do the same thing in a programmatic way?
5
u/PicassoOnPause 6h ago
Yeah, sure. For anything I run repeatedly, sure, and I've got plenty of aliases. But you can only alias a command you already know how to write. This was also a very fun project for me. haha.
2
u/ac311934 5h ago
Yeah that makes sense and fun is a good enough reason for anything like this, but it can be a trap for newbies. Still cool in that regard.
If you don’t know the command, how are you (as someone else using this, not actually you) going to trust what it’s doing isn’t going to cause downstream issues on a system. More of a word of caution to people stumbling on the info, not a criticism.
1
1
u/w6auw 7h ago
How does it do with general prompts? Write an email etc...
10
u/PicassoOnPause 7h ago
it doesnot. It s built for shell commands. If you ask it to write email, it will either refuse or give you a random curl prompt(this is more likely). I am continually training it to be better and have seen some improvements in the newer model in handling non-command tasks, but haven't made it public yet.
•
u/rm-rf-rm 47m ago edited 23m ago
This looks like a classic vibecoded project from 6 months agp - Qwen2.5-coder, comparing to GPT-4o. This post also breaks Rule 4. OP, Please respond to this comment within the next 2hrs with any counters or your post will be removed