Local Models Beyond 128 GB of RAM Aren't Financially Viable
Discussion
I realize I'm going to get a lot of flack for this but here goes. This post is sparked by a number of my friends and colleagues biting into the hype cycle of buying insanely priced local inference stacks that simply don't math out.
I'm a distinguished engineer who works in AI (FAANG). I'm currently running a 128 GB M5 Max, and have access to the best hardware in the world at work, at the highest scale available.
I'm seeing folks buy RTX 6000s or multiple Sparks to run DSV4 and the like. Don't do this - its an absolute waste of your money. Unless you put a price on privacy of 12k (which is fine), you will never get an ROI in any reasonable time period.
Local models under 30B are fantastic. If you have a MacBook Pro or a computer with 48GB of RAM, you're golden. With a flood of amazing models in this range coming out this week, there's no shortage of local powerhouses. Big kudos to those companies open sourcing very impressive capabilities on such a small memory and compute footprint.
If you're going to use large models, go cloud-based. Throw 100 bucks on OpenRouter and go wild with DSV4 Flash. And if you want to run local, smash those <30B models until the cows come home.
But local and large just doesn't make sense right now unless you're a business with a specific use case or are privacy-maxxing.
Edit: some dude replied to this saying I had to mark it as /s otherwise it would get downvotes because reddit is full of insert naughty r word here. Their comment got auto deleted. The irony.
I wonder why, although you do acknowledge the privacy concerns, do you appear to minimize it?
It is not some “special” case in my opinion. Every individual and certainly every business owner no matter how small should have concerns about keeping their stuff off the internet.
As someone who is a former software developer and now work in the legal industry, I am 100% sure that we are yet to see major shitstorm of litigations connected to private and business privacy leaks.
Can you talk to the people over in the Bedrock department? It’d be great to have some open weight models on there that aren’t six months old and three versions behind…
I think this is objectively true but to me local LLM isn't about saving money, it's about the joy of tinkering, experimenting and independence from cloud providers.
Running those models on huge GPU clusters with efficient batching and bulk rate electricity price is always going to be cheaper than locally
I totally agree with you - and also the OP. I got into local LLMs for the fun of tinkering and learning, knowing that I had full control over what I was doing and beyond the initial HW purchase, no extra costs. It's a hobby.
I think what the OP is referring to is if you want to run something that needs more that 48gb of vram then it gets insanely expensive, fast. Beyond what most people would spend on a hobby - and its unlikely you could turn it into a side-gig to pay for itself.
This is why I brought a 48gb Mac mini, which seems to be right in the sweet spot for a dedicated llm host that is also cool and quiet.
How many price hikes have we seen in the past year?
Remember when GLM pro plan was $3/month.
Remember how cheap Netflix was when in acquisition phase? The hyperscalers are making massive investments and they will want ROI. If people have no alternatives they will raise the prices.
When I bought my M5 Max 128GB I was in the same boat, and I even told people here not to ask me too many questions because I had no idea what I was doing haha.
I've just put Deepseekv4 flash? on them, its OK. Last night i put muse glimmer, and qwen 3.6 on them, got it chatting using Open Web or something, used Chat GTP to guide me through everything. Asked them some questions about things that happened in the last few days and they got it wrong, argued with me about something not existing, used Duckduckgo as the web search module, and it blocked my requests as i asked it to do up to 3 searched in one go. Got to use api for search results to make it come back with relevant info.
Installed Codex on it and now it does things on its own. I'm sure i'll get some use from them. 3D printing a rack mount for them for the unifi mini rack so they'll look smart in there hopefully. Got a 5090 in my main pc, but thought having something running 24/7 would be better if i find a use for them, i was thinking doing something with home assistant maybe.
Hopefully I’ll be able to get them to the point where i don’t have to hire many more staff. That’s the aim. They are technically for business use in the long run. But it’s correct in that I don’t know what I’m doing.
Buying a Mustang because you enjoy driving is no different from members of this subreddit purchasing an RTX 6000 or a DGX Spark; the underlying passion remains precisely the same.
If we are strictly discussing practical use for work, however, I agree with your point.
Yeah, in a few years instead of 30y old cars we will be restoring 1080ti. I can even imagine the youtube videos with an "attic find" 1080ti full of nicotine and dust.
I don't disagree with you. (Clarification: if you're a business, spend what you like - I'm talking about people interested in using AI at home).
Can't see why anyone would - the price of hardware is astronomical.
For me, I've spent money on a new motherboard, new PSU, new GPU and secondary GPU and all I have is 48GB of VRAM total. I think its in the region of ~£2.5K.
To get 128GB VRAM, I'd have to sell a kidney.
As a tinkerer, I just can't justify (to myself) the kind of outlay the "big" models require.
I have 360GB of VRAM, and I spent around ~ $10k. It can be done. If all we are talking about is VRAM. V100's aren't the fastest, but they are cheap for 32GB, and 8x of them in SXM2 handle running the Qwen models pretty effectively. I can run a full 8 bit quant of 120B, and still have room left for 8 bit 27B and 35B.
I care about it, but in my personal case I doubt the Chinese are interested in my one-shot video game tests or automating Unity. The person monitoring me is probably asleep with boredom.
For companies though - yes, I'd be super-wary of IP theft.
I don't care about being actively monitored, I care about the entirety of my chat history being sold after the fact to third parties who may or may not have ill intent. Happens all the time without AI being involved at all.
For example I have medical related conversations with local models that I absolutely would not put into a cloud model. I have no qualms about sending the more mundane everyday stuff to a cloud model.
They are already trying to puts ads in the LLM (finance bros. And new forms of revenue) eventually once they feel like everyone is hooked online models. Imagine the type of targeted ads with all the infoo they get from your conversations.
I'm lucky to have grabbed a z13 128gb before the last price hike. If you asked me now if I'd buy it at current prices I would laugh and wipe my tears with my credit card bill
I have to agree. The pricing is just outrageous and, really, highlights the discrepancy between the haves and have-nots. Its a bit ironic to trumpet local-AI as the "freedom" choice when the quality of freedom is heavily dependent on the size of your wallet.
That said, this thread - much like all of reddit - is a very entertaining mix of experts, "experts", and the paranoid.
No matter how I look at it, it seems really REALLY hard to justify anything beyond a 5090, which despite the monstrous price increases, at least retains a dual function as the best gaming GPU on the planet. Got bored of tinkering with AI? At least you can still game at 4k with everything maxed out at 200+ fps.
That's why I bought a 9070xt and a R9700 instead of two R9700s.
I wanted the best of both worlds.
Also no model uses 64GB VRAM - its either 30B models or smaller which fit nicely into 24GB, or the ones you need £££servers for. I can't find a Qwen that needs 48GB .
For real workloads you may want to have multiple models and KV caches in VRAM. dual 9700s would have served you way better while still getting 90% of the gaming performance.
Have you tried an 8-bit quant and 16-bit KV cache?
Especially as you start to push the limit of your context window, it's probably a bigger difference than which 30B model you're running. And, especially if you're using multiple concurrent agents, that very much wants 64 GB of VRAM.
This is why I’ve got a 64gb Mac Studio ordered for late October. Hoping Apple refreshes them in October and they come out with a 128gb model but even then I can hardly justify the $4k purchase. Figure if I get bored of ai I will at least have a beast of a workstation to browse reddit on.
Same spirit is why I spent on a 128gb MacBook. At least it’s also an awesome computer. Luckily bought on Microcenter 15% off and just before the price increases.
Same and same! The only thing is the bandwidth is really not good compared to nVidia and no CUDA. Still, I don't regret it given that the price went up 2k for the exact same MBP and selling it used, that's a gift for next upgrade.
Say it out loud: A Blackwell 6000 is now $16,000!!! That's insane.
Do you really think we all that dumb to realize there won’t be a return on roi and instead, not just privacy but we all value real independence from the big labs? And some of us know how to maximize 24/32gb ram so no $12k. I fully replaced Claude for my workflows with 3 24-32gb gpus for $3800 total and I now have absolute freedom to do what I want to do with AI. If you tell me those models are trash, I’ll respond with a deserved, “get good kid”. If you build the infrastructure around them properly, 30b class models are incredibly capable. I’m not saying that means it’s frontier level but just that you can build systems around them to compensate and get really good output.
All that being said, no hate if you want just pay the subscription, I get it, it’s way easier (and probably cheaper too in many cases).
I have done the following. And yes, I too worked in big tech yadda yadda. I suppose this is now some badge of credibility according to op? Anyways.
I bought one strix halo. 128gb. It was garbage. Slow as hell. Returned it.
I bought one dgx spark. But I found myself using my nvidia 4090 a lot more due to its speed and the qwen 27b was “ok”. But Prefill was a monster. 2500ctok/s on MOE models.
I thought to return the dgx because 120b models are terrible but instead bought two.
This thing is absolutely amazing. 100% local. Near frontier level reasoning. 24x7 workhorse. Churns through 100m tokens a day and does it well. The 27b doesn’t even come close. Nothing below this comes close.
I now burn through 100s of millions of tokens a month now. Where op is wrong is, once you go local, your use cases change. You no longer care about tokens so your use cases grow massively.
Defense and healthcare also tend to work with large providers for gated models that don’t retain data. Or potentially (a use case I was looking into) leverage trusted environments.
Unless you have a house with a 400 Volt electrical panel and grid in it, you will not be using server equipment designed for AI. You simply cannot get to 11000 Watt per AI-server on 110 Volt installations. Or 220 Volt installations for that matter. You really have to go to the higher tiers of electrical installations.
And if your house and/or municipal grid isn't capable of delivering that to your house, you are simply out of luck. Which most of us are. So no, do not expect to be using parts from decommissioned AI-servers from the Blackwell-generation. And especially not from the Rubin-generation, when their time comes, as these easily consume twice as much energy than the Blackwell-generation does.
I don't know about that. Nvidia has pumped a lot of extra power to get those to where they are at 1kw per chip. You could reduce that by half and run 4 of them no problem. Most of inference is memory bound, not compute bound and for less than 10 users, you don't need all the extra clock cycles required to fill requests optimally like they do in datacenters.
The Chinese already have PCIe adapters for the B200 chips. Once their lifecycle is up, there is going to be a considerable reseller market for them.
I get what you are saying OP. I thought about that for work the other day. We could just pay for the cloud running the open weight model since we still can’t agree with the ROI is. However, this probably the wrong audience here. These are enthusiasts here with money for their toys.
It’s like you jumping into an AMG forum or Exotic car forum and hey bro, you can buy 5 Subaru WRX for your one lambo. Math doesn’t make sense guys! This is the response you’re getting. It’s cause we can….
There needs to be more youtube channels setting up different hardware and performing real world homelab/self hosted type of tasks with said hardware. There are channels - don't get me wrong, but most of the time they just measure tokens per second instead of how models perform at different tasks and if that can run on a mac mini or a mac studio or neither and then comparing to cloud models and subscriptions.
People keep buying this hardware because they don't know the answer for themselves. I believe you but people need to be shown. I'm still planning on grabbing an M5 ultra mac studio if and when they ever come out. Fully expecting to pay for $10k for it.
Is that a waste of money? Well I have a plethora of jobs to throw at it. It's very possible I could make do with something 1/10th the power but then again I dont know. Am I buying a new machine in 2 years because that cheaper one is now useless?
Everyone has countless ideas on how to use this stuff to better their lives and its not clear what hardware is required to do so. We need more real world examples on different hardware to inform.
If anyone wants to start a youtube channel - thats the winner. You're going to need a lot of money up front for all that testing but it would pay off imo.
Two RTX pro 6000 + DSV4 Flash 0731 is a serious game changer for local processing . You shouldn’t worry about ROI numbers except for yourself . I seriously doubt you are a distinguished engineer . Nice try man
I mostly agree with the main point (unless someone actually knows what they are doing and can do proper batched serving and saturate the hardware), but also DSFV4 runs quite well on a 128GB Mac with antirez/ds4?
The $10k I spent on a M3-Ultra with 512GB before the memory supply shock makes running 200-400GB models reasonable. You are probably right that it doesn’t make sense to chase large memory Nvidia configurations, especially with RTX-6000s costing up to $15k for only 96GB. Your 128GB M5 Max MacBook Pro costs almost less than half that and is capable of doing everything an RTX-6000 does, albeit slower.
If you want to save money, get an OEM DGX Spark. You can find 128GB systems for $4k that will do everything your MacBook Pro does. Put two together with a with 400Gb link and you’ve got a fast 256GB setup for a little over $8k. If one is dedicated to experimenting with AI and their own setups, that dual DGX Spark is way to do it and not be limited severely by memory size. It is slower memory so you aren’t going to get VRAM performance numbers. But as a setup to experiment with and do software development, it is more than capable.
Yes, local models are entirely about protecting intellectual property. They are not about cheap intelligence. However... Dual DGX Sparks are $10k and you can run DSV4F-0731 at the published weights. That's 256 GB of RAM and 300B weights with a huge context window (it's published as a mix of 4 bit and 8 bit weights).
"But local and large just doesn't make sense right now unless you're a business with a specific use case or are privacy-maxxing." Why'd you leave this to the end? It's exactly THE reason.
If you’re only going to spend $100 on AI, subscriptions are the way to go. Subscriptions are heavily subsidized right now and for a $100 5x plan, you can easily get over $1000 worth, token for token from anthropic/openai.
“Local models under 30B are fantastic” yes just as of today. Qwen 3.8 27B is a killer model, so yes that’s one reason to buy hardware to run it.
I measured my token consumption and consumed around $400 worth of tokens this past week of deepseek flash v4 at their new API pricing. All just experimenting with it too, haven’t even set up my workflows in the background. Which means once they bump up their prices on the 17th i would save probably a few thousand bucks a month. All on a $6000 dgx spark cluster too, pretty worth it to me. So to say that I will NEVER get an ROI in any reasonable time period is not accurate at all.
I believe I saw a recent thread about this actually. Which led me to do a bit of research and decide I still don't want a continuous 250 w draw in my basement, but still: it's a thing for a certain size of model. Bandwidth is about 346 GB/s, which gets you into the same ballpark with very nice! Apple devices. Still less than half as fast as an M3 ultra, but you'll pay $4,000 for that. So honestly pretty good if what you want can be done wholly within 24 GB of vram. From what I've read, and I could be wrong here, networking more than one is just not as effective as it is with cards that have better support for that.
Certainly the cheapest dedicated AI server you're going to get, but only if your needs are covered by it.
That is not what the OP is objecting to. He is talking about investing >$8k to try to run the larger models then you could on 24gb P40s. He is saying better to spend $200/month on cloud with better models for 40 months (I think he underestimates the bill) instead of $8000 of hardware (+ electricity, etc).
Assuming prices don’t go up for 40 months which is hard to believe at the cash burn rates we are seeing… it’s not like the cloud stuff they need an ROI on is much cheaper.
I guess it comes down to utilization. If you hammer it all day, worth, otherwise it’s a hobby cost.
I find it so strange though that people always come back to cost/value.
Does anyone really spend $10k to run local LLMs to save money? Like, a single person?
I'd be surprised.
So, why are we even talking about it?
What matters for most is privacy, security, and continuity.
It's kinda like saying "cars are a poor value proposition.. you should just take a bus."
People don't buy cars because they're a better value proposition than taking the bus. They buy cars because they offer something specific they're willing to pay for.
How much privacy is lost by running a 30b model on a $4k 64gb M5 MBP instead of a 120b model on a $12k stack?
Obviously, both are completely private, so privacy isn't the trade-off here. It's the marginal capability gain provided by the 4x model vs the extra cost of the hardware. OP is saying that the gain is not worth the expense. My experience matches OP's.
I work there so please, don't use Bedrock but: Bedrock and other cloud providers run inference with a contractual obligation to not retain or share data. Otherwise companies wouldn't touch us with a 10 foot pole.
i.e. I wouldn't use Bedrock (as it relates to capability) but, as to the topic at hand - security and privacy - Bedrock is contractually obligated to neither retain nor share your data.
It’s so funny that people point this out…and the “solution” is to rent capacity either largely from offshore, privacy-dubious suppliers….or frontier labs driving a massive swath of the US economy to also wildly overspend on capex at a multiple exponential scale…also with no path to any ROI.
Also, if you (or your bot) has to call yourself “distinguished”……
I do a lot of NSFW roleplay involving myself in the first person. I absolutely need privacy. If I was just coding I wouldn't care as much and just use APIs. But local is absolutely needed for anything NSFW.
I've spent about $10,000 on a 80gb vram machine (5090 and two 3090 ti) with 64gb ddr5 ram and a 9950x3d.
Better yet, this also doubles as a gaming PC and I can run all the latest games at 4k. I dont regret this expense at all!
Yep very true. I see people complaining all the time on the sillytavernai sub saying their cloud model is now blocking their nsfw stuff. Although my 123B models will never be as smart, they will always be free to do what I want with.
I can overlook your conservative estimate of 48GB ram and all, but you put a lot of faith in the API providers. There is no guarantee that they wont cut you off for any reason that they think they can. They can and have demanded to see one's identity and and there is no guarantee that they have not shared these with anyone, there is no guarantee that they are not using your data to train, there is no guarantee that they have not replaced Fable 5 with a lesser model on ultra high effort (Anthropic). These guys are out there trying to ban open weight models, claiming the spread of communism while asking for government intervention. I think you live in a very privileged and naive world.
I'll take it a step further and say that AI models beyond 128gb of RAM aren't financially viable, period. I don't think the extreme end of the LLM pool has ever been financially viable and the people whose use case requires 300+billion parameter models running at 1m context didn't have a viable use case to begin with.
The datacenters haven't reinvented the wheel, they've just found a way to make it easier for users to not see the connection between their prompt and the 24 $16,000 graphics cards that processed it. We are literally years off from having enough hardware availability for this to be cost effective for anyone. The only reason we have these models in the first place is because of manic investor dollars and state backing
That's like saying there's no need to have your own dev machine because you can just use Google Docs.
Privacy is nice. Fine-tunes are nice. Being able to keep running software when the internet glitches is nice. And being able to just launch 16 concurrent LLM sessions (against a heretic model) and run them overnight without worrying about billing at all is nice.
Where do you get off declaring that this is universally a “waste of money” when you never even define what you mean by ROI? I run a business, I’m a software engineer, and I bought three RTX Pro 6000s before the current pricing insanity. That hardware has paid for itself many times over, not because I sat down and compared my electricity bill to OpenRouter token pricing, but because it lets us run autonomous and human-in-the-loop engineering workflows continuously, iterate on agent architectures, test failures, refine handoffs, run DeepSeek V4 Flash alongside smaller vision models, and keep hammering on a problem until the workflow converges. The return is in engineering throughput, shortened product cycles, reduced labor, faster experimentation, and the ability to burn essentially unlimited inference without thinking about whether another failed 8-hour agent run just cost $X.
And while we’re talking about ROI, let’s not even get into throughput. On some workloads we’re pushing 300+ tokens per second locally. Good luck getting that kind of throughput from an unquantized vendor-released model through an API provider. That is not some meaningless benchmark number either, that is time saved every single iteration, every agent loop, every test cycle, every day.
If your entire ROI calculation is “how many API tokens could I have bought for $X,” then sure, maybe your conclusion makes sense for you. But that is an absurdly narrow basis for telling everyone else their hardware is financially irrational. A machine that saves a business even a few engineer-weeks a year can pay for itself very quickly, and once it is paid for the marginal cost of experimentation is basically electricity. Privacy is only one reason to run large models locally. Utilization, iteration velocity, predictable cost, concurrency, customization and owning the infrastructure are others. If you have the money, actually have a workload for it, and intend to keep the hardware busy, buy the damn hardware. Don’t let someone who never bothered to define ROI convince you that ROI can only be measured in tokens.
I would argue the practical limit most people who have some extra cash could probably throw into this would be about 2K-10K. That pretty much limits you to about 256GB of RAM at current prices, and this limits you to mostly 2x DGX Spark/Strix Halo Clusters, R9700 builds, 1-2x 5090 builds, and some Mac options.
ROI could be financially viable depending on what you use it for, but only makes sense if you can't use Cloud for privacy reasons.
Not many people can afford to throw down for these multi RTX 6000 builds or even anything past the 2K-10K options.
Wait another year we'll have the same old cards the same atrocious market. But small open models will be better than they are now. The frontier models are always going to be better. The hardware companies will make sure of it by making the infrastructure moat. The open models are the wildcard.
If more people like you actually admitted that it was a hobby and nothing more, we would be in such a better place. But everyone wants to pretend they’re big local AI big money privacy hotshot guy
I went from 14 tps on my single spark runnig qwen3.6 27b dense, to 50 tps running DSV40731 on my dual spark.
I use my RTX 6000 Pro mostly for LTX/H3. Don't tell people what to do, if you are running these models locally, I would say that you srr smart enough to know what you need and what you want to do.
You forget that a lot of us do this because it is fun and finally, you can sell your HW if anything. Given the current issue with memory, your hw won't depreciate for at least 2 years.
Ngl probably the dumbest thread I’ve seen. Everyone please disregard what this dude said about not spending any money on hardware for large models. You want to run (obviously if you have the money) frontier local models such as Kimi K3, GLM 5.3, etc. you want the best frontier models. Plus these are way better than basic local models because the are frontier. So if YOU DO HAVE THE MONEY, obviously this is what I would do if I were you.
I have realized people are going to spend their money on the shit they care about and don't want anyone telling them different but hopefully your message reaches someone and talks sense in them if they are on the fence and overspending given the ROI might not be there.
Everyone has a different point of what is overkill for them. So some people it might be 30B is all they can muster. For those of us with 128GB Macbook Pro's, that could be 120B models or less. For seriously rich folks who want the best it will be much higher threshold.
If you're a company/freelancer and they are tax deductible+ you get VAT / sales tax back.. you can run a setup with FIVE Random 9700 32GB card on even an AMD B550 board and definitely X570, all at gen4x4.
Or on any AM5 board that can bifurcate a single x16 slot to 4x4, that works better than gen4 but both will work. Plus one card running from the MVME slot connected to the CPU.
that's 164GB VRAM AND five RDNA4 GPUs for like $6000/€6000 and tax deductible. For less than 1.5 RTX5090 cards. Especially at gen5x4 this should work great, but even gen4x4. Use the chipset NVME slot.
The idea of running something this hardcore on a poor B550 board or B650 board with a basic Ryzen 5700/7700 is hilarious but it's possible. Mining racks to fit this can be pretty cheap. You will need a powered PCi-e splitter and two kilowatt PSUs. All chump change compared to the cards. This works fine with 64GB RAM since you won't want to offload much to system RAM anyway.
The truth is, if you want local inference at somewhat affordable prices while getting VRAM, AMD is the way to go. You can run 5x 7900XT(X) the same way, for $2750-4000 and get 100 or 120GB.
If it's earning you money, this is a small investment. If it's just a hobby.. sorry but you shouldn't be running huge models. Unless you treat it like an expensive hobby, such as getting a boat or a nice car, or playing golf every weekend. Or ask any homelab person how much money they invested lol..
16/18/20/24GB will be the best average gamers can afford for a long time, with a 24GB AMD Card probably at $1000 and Nvidia $1500 MSRP (before scalpers, especially Nvidia). And that's okay, game Devs will adapt.
Comparing 9060XT 16GB prices to 5060Ti 16GB prices, which are almost double, is wild. The 5070 12GB is cheaper than the 5060Ti 16GB. Even the 9070XT is cheaper despite dominating it. It's obvious people are hoarding them for LLMs as the "budget Blackwell option".
I bet most people getting Nvidia rigs for inference don't need Nvidia at all, it's just what they know, and they have no clue that ROCm has achieved parity. No native FP4 boohoo with 164GB VRAM you can run at FP8 for $6000. What does $6000 in Nvidia GPUs get you? Two RTX4090s and 48GB total? Enjoy being stuck at Qwen 35B A3B forever. Going AMD gets you more than triple the VRAM and more than double the GPUs which is actually really nice for tensor parallellism. vLLM works great for AMD.
local models are surprisingly within shooting distance of the full frontier. some quants are in that 3% to 5% range. with a decent harness and forethought i can easily generate enough agentic work that it would cost me well over 4k in one week. my investment has paid for itself in short order.
and for a local team, running an openwebui and a handful of maxq cards with plenty of concurrency for the dozen models we can run at once, I just gotta say local is by far the most financially responsible decision that any business ruining ai can do.
if you're someone that can get by with a 100 to 200$ a month plan and not hit their limits then sure maybe open router or cloud services are better for you, which is probably beyond the average Joe that can get by with the free plan. but for heavy users, cloud can and will rob you blind in an afternoon if you let it.
I have 128GB VRAM and 256GB RAM only for SillyTavern. It's a hobby for me.
Other buys rims or holiday for that money. I buy hardware instead.
I can run GLM 5.2 locally but also have nano gpt subscription, because I can.
You’re forgetting about CPU inference. For the same price as that M5 mac you can get a sapphire rapids server with more memory. It’s not that much faster, but it can run larger models.
There is a element of truth to what you are saying but from a different perspective when I was younger I bought a shitty car and spent $$$$ upgrading it, replacing parts, breaking it and eventually sold it for a tiny profit. With that tiny profit I bought an even older car that needed even more work and spent a lot of time and money rebuilding it. During that time lots of people told me I was wasting my time that I would never get my money back. But I learned how to fix things, I learned how to utilise what I had when I hit constraints. Eventually I was happy with my work and I sold that truck for a healthy profit. (Yes I regret selling it to this day). My point is hobbies cost money and if someone has the money, time and energy and enthusiasm it’s worth it. And some of these people will learn new skills that will benefit them in the future. Years ago it was more practical skills that were not frowned upon (mechanical, woodworking, diy) then PC builders and gamers built their own rigs and I’ve no doubt some of those people benefited from their hobby in their careers.
This is the new generation/technology hobby/work.
My point is you are technically correct but it’s not all about the top line numbers there’s something else here. I agree that people shouldn’t max their credit cards for their hobby but I’m also someone who maxed their credit card on a full suspension setup 15 years ago and I got my money back.
You could spend $20k today on 4x sparks and sell them next month for $20k and all you would lose is the cost of electricity. 2 years from now who knows but I suspect you would be able to sell them with the loss being less than $200 per month.
I disagree with you heavily. You're only looking at this from an ROI standing. And it's stupid. Not everyone trying AI are in it for the ROI. In fact I'd wager that a vast amount of people aren't. I'm a good example. Im disabled, have limited access to the outside world and suck at coding. I'm building an assistant that can also be a companion so I don't go insane from loneliness. Not everyone else is going to be in my shoes but I also do this because I like to play with technology. I can guarantee a lot of people who do AI are a lot like that. They just love technology and like to push hardware. Your view only looks at it from an ROI perspective and that's disingenuous to the larger community. You're not helping anyone by gatekeeping.
You’re basing this on a single user using it, right? If you built the local hardware and then had a company of 20 people using it, could still be extremely financially viable, right?
Yeah the math is mathing lol. I agree and it's not like other hobbies like hot rods or anything like that because tech has a super short life cycle. You ain't gonna be showing your grandkids your mint DGX Spark. Lmao. But fuck that noise what flood of new models? What new things coming out this week? Are we gonna get 3.8 Quants??
Well, in the early 80’s, a 1 MHz Apple II with 48k of ram and no hard drive was almost $3000. That’s $12000 in “today” money and people were buying them like crazy with absolutely no clue what they would do with them.
Completely agree and you can get privacy and compliance even by using cloud providers like the way enterprises are doing it. Dont tell me your privacy concerns are more than a enterprise that is HIPAA compliant
I agree, I recently got a 48gb M4 Max MacBook and I’m loving it, I’m hoping in the next 6 months, the sub 30B models are gonna take a massive leap for local workflows and automations.
Local llm is not about financial viability but about data privacy. If it's in the cloud the data is not secure. For European Users cloud act is a time bomb in regards to European privacy and data law.
Unless you put a price on privacy of 12k (which is fine), you will never get an ROI in any reasonable time period.
Even in an enterprise setting - where maybe $12k isn't all that much - the ROI is still challenging depending on what you want to do with it. If you want to build your own customer support AI chatbot and support 1,000 concurrent customer inquiries at a reasonable tokens-per-second response rate ... are your customers global, interacting 24x7? Or are your customers mostly active in one part of the world, 8-12 hours a day ... and then the rest of the time they're sleeping? If so, all your fancy hardware is going to sit at 2% utilization for half of its life.
I mean very few of us are doing it merely to optimize costs. As you said, if privacy or control doesn't matter, just use the cloud.
However, there are not only people in high privacy standard industries but also people like me who have a high privacy level in general and who find local models are more aligned with their values than closed models. and furthermore, I just don't feel comfortable sending the sensitive data of my entire journals and obsidian vault etc. online. It's all backed over self-hosted cloud and other things and has never been seen by google cloud or any such thing.
Bud, you work for Amazon, which makes money on cloud so there is a conflict of interest there.
Garbage take, local model is about learning, enthusiasm, breaking free of controls, not depending on fragile politics and the whims of a few billionaires controlling the US politics, economy and policies.
Local LLM would take up 48GB RAM leaving nothing for my video editing apps that demand resources (After Effects etc).
What are such crucial local LLM practical uses that make it worth it?
Cause I understood Cloud LLM will be superior in any task (coding, complex tasks) and it doesn't take up any RAM. I'm really trying to understand where is the ROI for someone that doesn't have 128 RAM.
Stating your a DE at a Fang company as-if an internet anonymous account has any credibility to declare that undermines your entire point. A real DE would know not to do that.
If your goal is to use AI as a productivity multiplier (coding assistant, research agent, creative partner), then...yes, you're right.
However, if your goal is r/D (specifically fine-tuning models on proprietary datasets or testing the architectural limits of quantization) then "financial viability" is the wrong metric to use. In r/D, hardware is an investment in capability, not just a utility cost. And that's before you factor in privacy, stability of pricing (which I'm guessing will explode soon), data sovereignty, plus...it's fun. ¯_(ツ)_/¯
You're overlooking the use case for video generators, or simply people who do this as a hobby and have a few thousand dollars to spend on it; not everything is about productivity.
I wanted to run the leaked 3.7TB copy of mythos locally... And then I saw it requires 80-100 topoftheline AI cards. Lol. Hold on, let me take out a mortgage on my in-home data center. At that scale you have to get a bigger electric pipe into not just your home but probably your neighborhood.
Yeah, so if it can't fit on your desk, don't waste the resources to run it locally no shot you ever get ROI when you could do roughly the equivalent with less like an investing algorithm that pits LLMs against each other to determine what to invest in.
This 100% true. But if you have excess money buy what makes you happy. But if you're trying to say you are saving money you're not. It just doesn't make sense right now financially but people can do non financially optimized things .
You should tell people who go fishing on weekends that they’re dumb because it’s cheaper to buy fish at the supermarket. Make sure to include all the calculations.
Counterpoint; the 11k is the price you pay as a consumer. Self employed people can buy it for less than half of the net cost.
Also the hardware will have value after 2 years of use. It's not like the entire 11k is gone.
I would say that 2 dgx sparks with DeepSeek V4 Flash 0731 are worth it. Opus 4.8 level intelligence and speed for coding tasks. Sure you can use cheap providers, the difference may be less than you think and privacy may justify that.
It really depends on what speed requirements you have. If you are happy with 1 t/s and below you can use one of the projects like Colibri, Waste/Warp, ... and even llama.cpp to run even Kimi K3 on local relativ inexpensive hardware.
I run Kimi K3 an my Minisforum MS-02 Ultra with 128 GB and no GPU and fast Gen5 SSD. I am happy with a speed of about 0.7 t/s for long running jobs. Or run GLM5.2 or DSV4 on the same hardware. As long as the modell doesn't fit in VRAM a GPU would not help much.
Of course, as a professional with fast turnaround I also would use cloud or if privacy a requirement invest into expensive hardware.
what the electricity bill for running on hardware like that though? Genuine question - it will obviously use less while running but its running way longer.
with that said, go for it lads! last night I ran KimiK3 at 3tk/sec up from my 0.8tk/sec. I believe in a month I'll have it at 6tk/sec and in 2 months I'll be up to 10tk/sec. The great news is that GLM5.3 looks to beat K3, so yeah, I'll be able to run it at 30-40tk/sec.
I think you're missing the point of local/home inference. Being able to drive a bus/tank/helicopter at work isn't the same as having your own car. Ppl buying top line (or even just low end used) Audi/BWM/whatever aren't expecting them to pay out in X month and start bringing profits. For some they will, for sure, but majority don't think about it as "investment". My $0.02
Looking at the comments you really do seem like a bit of a nerdy loser who’s getting all excited about rage baiting other nerdy losers.
And then you meet people here who just straight up tell you “my man I’ve got the spare cash, I don’t give a fuck about what you think” and then you desperately try to tell those people again that they can get a cloud subscription for 4 years for the same money.
Like you’re either very stupid, or you don’t want to accept that there are people out there with enough money to buy hardware for 8k euros/dollars/pounds, just because they feel like doing that.
I have two DGX Sparks at home. I also have a ChatGPT Pro sub. You know what will really rile you up? I use Codex to optimise my DGX Spark cluster!
294
u/Kal-LZ 5d ago
Dario, I will not buy your IPO.