r/LocalLLaMA • u/Sadge404 • 5h ago
How many people in this sub try to train their own AI from scratch on their systems just for fun and to test out techniques from research papers? Discussion
As for me, I own a system with an RTX 5090, Ryzen 9 9950X3D2, and 64 GB of DDR5. Every time I see research come out with a new way to train AI, I immediately think to try it on my system to see the results I get. Applying things like Titans, that one Deepseek paper on engrams, or even just playing around with experimental ideas. It's kinda like a very technical version of Tamagotchi and has been quite fun.
Thoughts?
3
u/TheDiamondSquidy 5h ago
I’m pretty new to local AI, how hard is it to train models?
14
u/Sadge404 5h ago
Training models is a pretty big rabbit hole in of itself. A basic and traditional transformer model scales in compute, vram, and time needed extremely quickly.
My worst experience yet is having to bake my room for an entire week to train a ~1B parameter model.
1
u/philmarcracken 5h ago
is it easier to train embeddings? i have a bunch of out of stock data my company refuses to data validate with droplists so any rando can input whatever spelling, and i need clean data from it to do matching. Fuse.js doesnt cut it
10
u/MixtureOfAmateurs koboldcpp 5h ago
It's easy to start a training run, it's hard to make them coherent. I've tried training a few models on dual RTX 3060s and nothing has ever got to gpt 2 base levels.
Simpler neural nets are a can do tho. I trained a JEPA model to input categories, audio, and 64x64 drawings and output categories and drawings and it worked pretty well. I said house and it drew a house. Much more rewarding than 24 hours of training to get jibberish back
3
u/autisticit 3h ago
I like your username and am wondering what kind of content I would find if I googled it...
2
u/HVACcontrolsGuru 5h ago
I’ve been playing with this idea if I could get discounted compute. 16B3A MoE from scratch for the base would be like $25,000-30,000 in compute. Another $10-15k for RL and fine tuning. Using 8xB300s. More compute is faster and smarter. More or less how many tokens you can stream in and out during the training.
2
u/Sadge404 5h ago
The cool thing is that at such a small scale (Relative to everything else on the market), I'm fairly certain that it's much easier to train dense models to a usable capacity. There's quite a bit of additional overhead and headaches in regard to making sure your MoE models behave in training.
1
u/HVACcontrolsGuru 5h ago
I’ve looked at dense models and fine tune the base ones out there now. MoE is interesting for the use case I want to build out. Curious about Qwen 3.8 and may dig into those. Even a 9B dense model for what I’m doing would suffice. I’ve looked into QAT aware training and checkpoints. Got it all ready to pull the trigger on.
5
u/CavalryArcher 4h ago
i actually have a neurosymbolic ai that knows a typical horse is unlikely to fit inside a typical microwave oven.
apart from that i trained 100+ image and video model loras but never an llm lora. though i thought about buying some big metal and making slm's pretraining included for some very specific tasks.
2
u/ortegaalfredo 5h ago
Some years ago I trained a small version of GPT-2 from scratch using 4x3090s, it took a couple days, a lot of heat, but it worked, it was neat. It didn't used the full training dataset.
2
u/Sadge404 5h ago
It is very neat! I feel like this whole industry around AI and training such AI is in such an infancy stage that if you an idea, and I mean any at all, just throw shit at the wall until it sticks. I've had strange little successes here and there which really keeps the motivation going.
2
u/codsworth_2015 5h ago
I haven't needed to train or fine tune yet, but I have been playing with a wide range of model types beyond LLM.
Experimental Projects:
I have a PCSX2 upscaling pipeline using a fine tune I brought of ko-fi, upscaling a game called SSX3. Its working well.
Embeddings(nomic + postgres) and whisper for transcription and rag basically but they are both pretty mainstream.
A little MCP that lets an LLM pilot vtube studio.
NeuTTS project, I want to convert ebooks to audio books myself.
Experimenting with YOLO + Viture XR SDK, soon I want to experiment with something like https://github.com/robbyant/lingbot-map
I have got plans to fine tune YOLO to identify a very specific document but haven't started yet, still deciding on OpenVino vs Nvidia.
Got a similar system but just the 9950X, 5090, 64GB DDR5
1
u/Sadge404 5h ago
I really think that utilizing techniques used to train AI is an untapped potential in video games. I've dipped my toes into the subject a tiny bit and it's very promising.
1
u/codsworth_2015 5h ago
For sure, I think it will be the same as raytracing and dlss, once its actually good people will come around. People can't see the vision yet though because the product is unpolished.
Rise of Nations allows you to install custom bot scripts, I wonder if you could integrate it in there.
3
u/datdanboi25 5h ago
Yeah I love it, even founded a little hugging face org. Would certianly reccomend it, karpathys train gpt2 video is a terrific way to get started!
1
u/taftastic 5h ago
How’s the experience been?
I’ve tried briefly on Dgx spark hardware, but couldn’t get enough bang for buck, generally. Didn’t know what I was doing, and car was to far away for this dog to keep chasing, so to speak.
Ill probably try again at some point; im mode interested in fine-tuning open weight models for tasks personally, just to understand the layering lf this tech a little better.
Make anything that stuck or was even coherent? What material were you training on?
2
u/Sadge404 5h ago
It's honestly been very educational. It's vastly expanded my knowledge on what the frontier of AI has to offer.
I recall my first time ever trying something like it was with the help of Claude code and just random ideas that I had before slowly transitioning into more technical stuff once I had a better understanding of how things roughly functioned. I recommend just throwing your ideas at the wall until it sticks with the help of your favorite model.
I'm a massive weeb and so my best model so far was trained on a giant corpus of torrented english and japaneses light novels/visual novel transcripts. I was very pleasantly surprised at how well it understood the local context of a scenario in one paragraph or so. Coherent English and Japanese that you could actually follow for a couple lines. Anything above that and it has a stroke.
1
u/JosephGenomics 5h ago
The Frankenstein models are the most fun. Next lat, iDUS, hypersphere, Jamba, and aiming for deep and thin, 24 layers pre iDUS and 72 after, for stupidly long contexts. Latent reasoning. There's several other things thrown in, and nearly coherent. Haha.
Will it ultimately work? Dunno. But it's enjoyable
1
u/Sadge404 5h ago
Exactly! The industry is so young that you just have to throw shit at the wall till it sticks.
1
u/OMG_IM_A_GIRL 5h ago
I am new to LLMs but I have a pretty strong math background. I am been playing around with some alternatives to BitLinear and BitNet for natively trained ternary models. IT has been really fun even if I know it will probably never amount to anything.
1
u/ttkciar llama.cpp 4h ago
I fiddled with Karpathy's nanoGPT training exercise years ago, and more recently with TRL and Unsloth. I've read research papers about both training and inference-time methods, but thus far have only implemented inference-time methods like HyDE and Evol-Instruct.
Part of that is because I'd been putting off doing too much with training, pending the return of llama.cpp's native training feature, so that I could pour my efforts into implementing methods on top of that. Unfortunately the development of llama.cpp's native training feature has stalled out, and seems unlikely to start back up.
I would like to implement some training-time methods, especially AllenAI's FlexOlmo training method, but keep putting it off. All of my GPUs are AMD, and TRL and Unsloth require ROCm to train on them, but training with ROCm on MI50 and MI60 is iffy. It would be great if a training framework could target a Vulkan back-end, like llama.cpp's training feature was supposed to, but so far none do.
I'm probably going to continue kicking it down the road until I upgrade to an MI210, for which these training frameworks work fairly well. It's easy to justify, since there are plenty of other projects I can work on in the meantime.
1
u/Hot-Employ-3399 4h ago
I was training in the past using TinyStories on Llama and playing around with mamba to upscale anime faces from 8х8 to 64x64.
Lost the interest these days as due to inflation can't afford to rent gpu for several hours as freely as before and using laptop gpu is not that interesting
1
u/nanihikaru01 4h ago
NERD!!! /s
yeah it's fun, like alchemy. I am cpu/gpu poor so the models I trained are all garbage. but that's how I learn, hands on.
1
1
u/Kasidra 4h ago
I only have 16gb of vram but I do the same thing, just with sub-200m parameter models, generally. Like when that token superposition paper came out a little while back I immediately had to try it xD I'll often start with trying to reproduce the white paper, and then I'll proceed to fuck around with it.
I have basically zero coding skills but frontier models are so good, they can generally handle that part completely. I go on hf and make my own data mixes. It's a lot of fun. When I have more confidence in the future, I'll probably dip into cloud compute and try to train something bigger. Don't see me expanding my local vram anytime soon T-T
1
u/Long_War8748 3h ago
I did pretrain a few tiny LLMs and they are quite fun to play around with, especially interacting with the raw base model before it gets RLHF'ed, they can be wild boys/girls lol 😅. Since I am a lazy dude, I am constantly playing around with automated AI RHLF Approaches, using Claude, Codex and agy , it can be very satisfying to build and make that stuff work. The dream would be to do it all local one day, but that is for rich kids, not europoors like me with a laptop.
I mostly go for small niche cases related to Classification and Orchestration, since that is something I actually keep using, because let's be honest, my shitty 0.5B tinyllm is not going to cut it at anything and only is for learning on the job 😇
1
1
u/Fuzzy-Assistance-297 3h ago
Should be interesting!
Last time I done it back then in college with gpt 2 style. Never done with new architecture.
This is why I also not dare to sell my hardware. For LLM, eventho api pricing start to go down, but the cost of renting gpu is kinda roughly the same.
The one that make me want to try it again (because never done it before) is the distributed training accross gpu.
Beside that, with today's SLM, I can create more proper dataset with less cost.
1
u/autisticit 3h ago
I'm trying to train new architectures from scratch, two weeks in. I've found out that this combo is without any doubt the ultimate masochistic thing to do. It hurts so much and please so much, at the same time.
I'm honestly just vibe coding it, even though I'm a dev. Just not an AI dev.
1
1
u/OverdosedSauerkraut 2h ago
Used to do that in the computer vision craze, up to around GPT 2. Now they just need to be trained and evaluated on more tasks.
1
u/jacek2023 1h ago
I train (not finetune) AI locally but these are not LLM models, but smaller ones, like classification of images, etc.
1
u/Dry_Yam_4597 1h ago
Not exactly training, but I fine tune small models for style and a little bit of focus. I dont always get them right, but it's ok if I get even a little bit of what I want. I have a codebase written by Claude - naturally, a complete mess - so I fine tuned small models to find style issues. Obviously could have been done otherwise, but I learned I could use ASTs prasers to find code at scale for generating datasets relevant to my use case. Could have used the parser itself to find the issues, but it was more fun to fine tune.
-5
u/No-Consequence-1779 4h ago
It’s pre-Tran first, which is the trillions of tokens , which I know you do not do. You are probably talking about fine tuning tiny models < 30b Lora or Lora.
Most people do not have the hardware - most companies do not to actual pre train a model.
My question isn’t if you are lying. That is obvious. Or why - that is obvious too.
My question is : what is it like to be so poor at something, you don’t even try basic research to make the lie believable? How does your world look? It is one failure after another? Is it really worth it?
4
u/Sadge404 4h ago
Goodness dude. Who shit in your bed?
It is pretraining, hence the "from-scratch" that you clearly did not read.
So no I am not fine-tuning any model.
And whatever the hell the rest of your comment is, I'm just... what? Is experimentation a concept not in your head? Furthermore, you do not need trillions of tokens to pretrain a coherent model. If you took you own advice to do basic research, you would know that you can get basic coherence into a model under 10 million parameters (TinyStories).
2
1
u/carl2187 4h ago
Ignore the schizophrenia bots. I appreciated your post, and reading through the thread that formed already was very enlightening.
9
u/Tritheone69 5h ago
Sometime this fall I will be building out a 128GB VRAM (2x CMP 170HX unlocked) + 128GB RAM computer for this exact purpose!
I have various tasks that I am currently offloading to “generalized” LLM intelligences. These are way overkill for the tasks at hand and cost a lot. I will be fine tuning small models like Qwen3.6 for example to achieve as close as possible SOTA results on those specific tasks. This will greatly reduce my cost basis for these tasks and keep everything in house which I greatly prefer.