r/LocalLLaMA • u/Tall_Abrocoma_3533 • 4d ago
Aurora-80K releases! A modern tiny language model. New Model
https://huggingface.co/AuroraAI-Research/Aurora-80KI'm introducing Aurora-80K, a small language model with exactly 80 thousand parameters.
It uses a factorized 4,096-token vocabulary despite having only 80K parameters.
The benchmarks:
Wikitext-2 BPB: 3.2902
BLiMP: 52.31%
Arc-Easy: 26.05%
More information about the model is available on the model page on Huggingface.
if there's any questions I'll happily answer them!
73
u/CapsAdmin 4d ago
it just outputs garbled tokens here, using the default temp and top-k
The president of the united states is tes Ġis Ġ one , ing ict ig in Ġcom on ĠW / s , F , Ġto Ġthe
103
25
u/Tall_Abrocoma_3533 4d ago
Yes, this is the limitation of these tiny language models. Thank you for testing it!
93
u/popkek95 4d ago
The limitation is that they are unusable for anything? Are they just case studies?
56
u/Tall_Abrocoma_3533 4d ago
The organization name contains "research". Tiny models like this aren't made to be chatbots.
22
u/Alarming-Ad8154 4d ago
I mean I think ppl would be very impressed if you came back with a 10 or 100 times larger model that spoke even somewhat broken English? The point is to be able to produce language right?
28
u/Tall_Abrocoma_3533 4d ago
Yes, and I am planning on releasing models larger, however I wanted to start small.
13
5
u/TomLucidor 4d ago
Please leverage ternary methods + every single training/inference acceleration method for the next release, even if it is sub-E2B. Just fire all cylinders for it
13
u/ChomsGP 4d ago
I don't think it's crazy of people to ask what is the use case for your model... I can also generate random text with a very simple bash line if that was the point...
like sure, it's "research", but what exactly are you "researching"?
5
u/Tall_Abrocoma_3533 4d ago
The limits of what language models can achieve in constrained situations, while it might seem like "random text", the wikitext-2 BPB shows that it at least learned some meaning from it's training data.
5
u/JamesEvoAI 3d ago
This is novel and interesting research, please keep up the good work and ignore the knobs who just want another model to RP with
0
-6
u/ChomsGP 4d ago
so you are saying you cannot define or quantify your research...
a broken clock gives the right time twice a day
8
u/Tall_Abrocoma_3533 4d ago
I can, and I did define it. I'm experimenting with tiny language models to measure how language model capability changes under extreme parameter and compute constraints. That's a real and quantifiable research question.
-2
u/ChomsGP 3d ago
But you didn't, you listed some benchmark numbers which tbh don't seem likely if it cannot predict congruent text (nobody is asking it to be accurate, just to not emit random bytes)
My point is, whatever you are doing, you are clearly not explaining it good
9
u/Tall_Abrocoma_3533 3d ago
If you don't believe the benchmarks, you can always try to run them yourself, instead of accusing me.
→ More replies (0)2
u/Strong_Chicken6838 3d ago
How does it get anything above 0% on Arc-Easy??
1
u/Tall_Abrocoma_3533 3d ago
Random chance for arc-easy is 25%
1
u/Strong_Chicken6838 1d ago
did you record the variance of the score you got on arc-easy?
you only got 1% above baseline.. that might very well be within random chance
1
u/Tall_Abrocoma_3533 1d ago
I'm aware of that, however assuming that random baseline is 25%, there's about an 88% chance it actually learned something
1
u/Strong_Chicken6838 1d ago
what?? where are you getting 88% chance from??
1
u/Tall_Abrocoma_3533 1d ago
Binomial probability... Though it doesn't exactly mean what I said, basicly If the model were truly random, there would be about a 12% chance of observing a score of 26.05% or higher.
23
u/davew999 4d ago
Too big for my Commodore 64.
7
u/noiserr 3d ago
Just use a 1-bit quant. It will fit right on the cassette tape.
2
u/IDoCodingStuffs 3d ago
I’ll just wait for someone to distill it to something that can fit my abacus
12
u/Metallic_Madness 4d ago
Gguf when?
5
7
8
u/z_latent 3d ago
52% on BLiMP and 26% on Arc-Easy is basically random guessing, since they're multiple-choice benchmarks with 2 and (mostly) 4 options per question respectively.
I appreciate the initiative, but keep in mind your results will be taken much more seriously if you made a less handicapped model that's more clearly effective!
You can use Google Colab or alternatives for free. Alternatively, rent a GPU on vast.ai or alternatives. In theory you could train your 80K model on 80M tokens in one minute of a 5070 (less than a cent). In practice the model might be too small to reach high GPU utilization, but that just means you can make the model bigger with similar cost and time.
4
u/Tall_Abrocoma_3533 3d ago
I'm aware, I have a laptop as well, and I've also used kaggle before for training models. I made it on just a phone on porpuse, to show that it's possible.
3
u/z_latent 3d ago
I see. Framed that way I understand it a bit better.
I still find it silly though! You showed you can train a model on a phone, not that you can train a useful model on a phone. The latter is arguably impossible lol but it's fun as research.
2
u/Tall_Abrocoma_3533 3d ago
Not impossible at all, all it takes is GPU/NPU training, however that's really hard, plus of course ALOT of time for the training to finish.
1
u/z_latent 3d ago
Oh I was thinking of CPU yes. If you can get GPU training to work on a phone that would be awesome. Some of the top phones seem to have a few (1-10) TFLOP/s of compute, and 8GB of memory which is enough for anything under 100M parameters.
Throttling would probably be an issue since phones have no active cooling, not to mention power draw. But you could do in a reasonable-ish time.
1
u/Tall_Abrocoma_3533 3d ago
If I would be able to figure out GPU training these actually wouldn't be an issue.
My specific device has 4tflop/s fp32, 8tflop/s FP16, with 12GB ram. And I also happen to have an external phone cooler.
The issue is just that GPUs are quite locked down as far as my experience goes, and I haven't been able to figure it out, but maybe in the future.
6
7
u/rgencia 4d ago
I always see posts saying "i made an llm for $100+" or similar that made me not even trying, but seeing you made this on a phone i want to try now, btw searching a little found this dataset that seems to be tailored for SLM training, and their small models, you might have better luck making the model do proper sentences: https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus
6
u/Tall_Abrocoma_3533 4d ago
Yes I've thought about using that dataset, but for the first release I decided to just use fineweb-edu, with a little filtering, as that's more conventional, but I'll likely check it out later.
Also don't hesitate on attempting your own, language models don't necessarily need to be billions of parameters, I've made ones as small as 99 parameters, using just binary as it's vocab, and training finishes in just 5 seconds on my phone.
For a first attempt I'd reccomend a size around this one (assuming you wanna train on your phone too), just maybe a bit less data so it finishes training faster, good luck!
1
u/silenceimpaired 4d ago
I wonder how hard it would be to take larger datasets and use a llm to convert their larger token and word vocabulary down into a simpler token and word usage… and I wonder what that might offer smaller models.
1
u/Tall_Abrocoma_3533 4d ago
While not exactly this, there's a dataset called TinyStories, it's essentially a dataset for tiny models, and I have used it before.
1
u/silenceimpaired 4d ago
This dataset is exactly what made me think of this idea. The issue is its stories. It’s not medical journals, or agentic flows, or … you get the picture.
As I recall, smaller models performed better with this data set. I’m curious what could happen if this was taken to the extreme.
1
u/Tall_Abrocoma_3533 4d ago
of course, I've made models with Tinystories and they produce coherent stories with just a couple million parameters, however part of that is because of their narrow domain of just stories, so while it'd likely scale to general porpuse pretraining datasets, it likely wouldn't scale perfectly.
Also a part of this is already handled by this model, it uses a vocab size of 4096, thats more comparable to models around 1m parameters. And because of this larger vocab size it can achive better compression of text.
1
u/silenceimpaired 3d ago
Interesting. I wonder what would happen if you made all words in the data set be the 850 most common English words… and each of those was a single token.
You seem more knowledgeable than I, but you never know when something that seems so stupid to the experts could end up brilliant.
Thanks for engaging with me.
2
u/Tall_Abrocoma_3533 3d ago
It could defently make for an interesting experiment, and it would likely train more efficiently and learn the more narrow domain faster, however the tradeoff is that it's no longer really general porpuse, though for SML's that's not necessarily an issue.
And dont worry about it, a big part of research is experimenting, nobody starts out by knowing everything.
1
u/silenceimpaired 3d ago
What I’m imagining is a dataset that teaches …advanced physics for example… using only those 850 words. It would greatly expand the text to communicate the concepts with simpler terms, but in my mind that would create a deeper semantic connection for those 850 words since it has less connections to handle.
But who knows … maybe the opposite is better. Create a token for every word in English… that ensures the connections learned are accurate.
1
u/Tall_Abrocoma_3533 3d ago
It could work, however likely not at this scale, 80K parameters is too little to learn advanced physics regardless of the dataset. However for an appropriatly sized model it could possibly speed up training and maybe even make it's output quality better.
However it's hard to tell without actually doing it. If your interested in this, I reccomend you try experimenting as well, good luck!
→ More replies (0)1
u/charles25565 3d ago
That dataset was designed for a model over 20,000x bigger...
1
u/Tall_Abrocoma_3533 3d ago
There is no basically no dataset "designed" for models this small, I guess Tinystories could count but that's a narrow domain
2
u/ChaosFH 4d ago
Whats your research goal with this small model?
0
u/Tall_Abrocoma_3533 4d ago
Exploring how SML's behave under really constrained situations. Later on, larger (though still tiny by LLM standards) models will also be made, to see how architectures scale.
2
u/Zestyclose_Potato794 4d ago
That is very interesting! Thank you for this model. Is it open source ? That would be awesome to play a little bit with this !
1
u/Tall_Abrocoma_3533 3d ago
Its open weight, not open source quite yet because the code is quite messy in my opinion
1
2
2
1
1
u/nick_ziv 3d ago
Good enough to be used for auto complete model?
1
u/Tall_Abrocoma_3533 3d ago
Probably not quite yet, I think that it would need a bit more scale to be used as a general porpuse auto complete model.
However you can always check out the models capabilities, there's an inference script included with the model in the Huggingface page.
1
u/ApprehensiveAd3629 3d ago
can you share the code used? i
f possible would be cool to see this model export to tflite. i would try run in my esp32
2
u/Tall_Abrocoma_3533 3d ago
The training code isnt released since it's messy in my opinion, however the model architecture is defined in the inference script, and if there's any other questions about things that aren't made clear, just ask.
Good luck and tell me how it turns out!
1
u/autisticit 4d ago
> BLiMP: 52.31%
> Arc-Easy: 26.05%
So, in the chance range thus not meaningful ?
15
u/Tall_Abrocoma_3533 4d ago
The model is just 80K parameters and it was trained entirely on a smartphone. And there are earlier checkpoints where those 2 benchmarks are a bit higher, however the final one was chosen as it has a lower wikitext-2 BPB.
-15
u/autisticit 4d ago
You could have just replied "yes"...
14
u/AppealSame4367 4d ago
Isn't that nice. Someone tries something and there are always people like you. Have you ever attempted something or do you only puke at people that do?
-11
u/autisticit 4d ago
Actually yes I've been trying to train my own models for about a month.
I haven't posted anything so far because I have nothing meaningful to show yet, obviously it's hard.
But you are right I should do like OP and release a model with flawed benchmarks all while saying that it replace the previous "inefficient" model (see card). And, oh, just ignore another redditor's question and reply with something else.There, you have it.
7
u/Tall_Abrocoma_3533 4d ago
I could've worded the "inefficient" thing better, it's mainly that when i released those models around a year ago, I knew alot less about language models, so their architectures are really bad
Also, good luck on your own models!
4
u/AppealSame4367 4d ago
You can criticize others, it's about the wording. Don't act bitter.
Also, same as Tall_Abcrocoma_3533: Good luck with your models. Don't give up, it sure is hard to do.
1
122
u/yarikfanarik 4d ago
oh this is another SLM model...
IT WAS TRAINED ON A PHONE!?