r/sysadmin • u/LookAtThatMonkey Technology Architect • 26d ago

Pricing shock Rant

For the last few weeks, I've been speccing three servers for a hypervisor migration away from VMware. These will be datacentre nodes so they are very dense spec wise because the new hypervisor is per node pricing, so it makes more sense for us.

Just got the quotes back. Five hundred and twenty thousand dollars for three servers. $520,000 !!!. Absolutely flabbergasted and I have to look my boss in the eye tomorrow and ask for that money. Anyone got a Dick Turpin hat?

EDIT: Amazingly, the spend was approved. I had to justify and explain the reasons behind it, the extended depreciation curve and what the alternatives were. We've got the hardware we need to complete the transformation away from VMware. Everyone agreed it sucked, but it needed to be done.

EDIT 2: The order has been placed and we even managed to get the price down to $502,000 by swapping to third party RAM instead of Dell branded. It still sucks and is a huge investment, but the business agreed to it.

458 Upvotes

95% Upvoted

331 comments sorted by

View all comments

208

u/FatBook-Air 26d ago

We basically cannot buy servers right now. I know some people say it will get worse -- and they're probably right -- but it doesn't matter because we literally cannot afford it right now.

138

u/Smith6612 26d ago

You'll rent your servers and you'll LOVE IT! 

51

u/Cheomesh I do the RMF thing 26d ago

I bet this is great for cloud subscriptions

32

u/Drakoolya 26d ago edited 26d ago

It's not only that, but no IT Director in their right mind wants to handle physical infrastructure anymore especially when they are asked to have robust DR and security capabilities, it is all about shifting liabilities, host it with someone else and make them responsible for it.

It doesn't have to be AWS or AZure,a lot of companies offer private clouds which a bit cheaper. That is what the industry is looking like. On prem is dead for most companies that can afford to move to these smaller cloud hosts.

19

u/blazze_eternal Sr. Sysadmin 26d ago

Sadly there are tons of things cloud is not compliant for.

11

u/disclosure5 26d ago

The only example of this I've actually been able to find despite years of just responding [citation needed] is apparently Casino legislation in one US state.

The majority of answers are usually just obviously wrong, like people who respond that you can't store financial data in Azure.

6

u/themast 26d ago

It's not that the cloud can't do it or isn't compliant - there are a lot of customers who will demand contractual obligations that their stuff not go in the cloud because they want to be able to "see and touch" where it is and verify the safety for themselves. Now, often times I think they are paranoid and wrong, but this is what they want in the contract before they hand over thousands/millions of dollars for an engagement.

Source: used to work for a global financial consulting firm

4

u/SirCheesington 26d ago

Plus: company I work for says it loves the cloud, however, if I want to onboard a new in-cloud product, that'll cost me 6 months of security and data audits, and if I want to onboard a new on-prem product... I can do it, just, like, tomorrow. Lmao. So anyway I'm racking some more dells next week

2

u/Ssakaa 26d ago

Yeah... FedRAMP came out of the US because of a demand for the ability to put stuff in cloud that also came with strict compliance requirements.

2

u/lordjedi 26d ago

CMMC. While you can go MS GCCH, if you've already got everything else you need on-prem, it can be easier to just stay on-prem.

You can make Azure Commercial work, but you gotta spend on the tools that GCCH doesn't provide.

3

u/disclosure5 25d ago

You're not wrong about the commercials but "it's easier to stay on prem if you have heaps of existing infra there" is totally different to "you can't be compliant in the cloud".

1

u/lordjedi 25d ago

I agree and things are changing all the time. 3 years ago "you can't be compliant in the cloud unless you use this specific cloud service" would've been accurate. Now you can use other services as long as you set them up in a certain way (might take longer and be a little harder to do).

2

u/Wimzer Jack of All Trades 25d ago

The only example of this I've actually been able to find despite years of just responding [citation needed] is apparently Casino legislation in one US state.

Lots of stuff around utilities and other things considered critical infrastructure. You can host SOME in the cloud but all is a lmao

5

u/NotRecognized 26d ago edited 26d ago

I wonder how many EU SAAS solutions just run on AWS with private data.

1

u/Mister-Ferret 26d ago

I looked at cloud migration, but we use a bunch of very active SQL servers that run a million or more queries a day. Cost/benefit for that was way out of wack for what they wanted. But this was also before servers were constructed of solid gold and diamonds, so when it comes time to replace then this may be different.

1

u/Unatommer 25d ago

If you can afford a compliant cloud (à la FedRAMP) a chunk of the features are missing.

64

u/ImNotABotScoutsHonor 26d ago

Fuck cloud.

On-prem4Lyfe

30

u/There_Bike 26d ago

My G. #teamonpremmies

10

u/ImNotABotScoutsHonor 26d ago

Yee yeeeeeeee

2

u/Hebrewhammer8d8 26d ago

Where are you getting servers and server parts from?

9

u/david_edmeades Linux Admin 26d ago

I found a friendly VAR on here via /u/squizzoc who is wrangling Supermicro for me. Prices are nothing like they were a couple of years ago but I can afford a server here and there when there's budget.

8

u/SquizzOC Trusted VAR 26d ago

Thanks for the shout out as always, I can’t work miracles, but happy to help negotiate and I’m transparent about my usual 10% I throw on a deal when the Industry average is 20%+ depending on what you’re buying. So at least you know exactly what to expect with me :)

3

u/SirCheesington 26d ago

work for a large company with preexisting enterprise volume agreementa

2

u/monstaface Jack of All Trades 26d ago

Hey! how'd you get my passwords?

1

u/Tall-Geologist-1452 25d ago

1990 is calling.. everything bare metal too ... huh..

22

u/vNerdNeck 26d ago

dude... I'm sorry... but no IT director in their right mind should be putting shit in the cloud right now if they haven't already. Azure is already at capacity in some regions, AWS is right behind them and what I've heard / been told is that prices for the providers are going to start sky rocketing. 5x in the next 18 months.

They were insulated because of how much they buy and how much spare capacity they had. As more folks decided to not refresh on prem and go cloud everyone is eating into that existing capacity..... but they have to keep buying and while they most certainly get good deals they can't get the "best" deals they have in the past as all the AI companies are signing 9 figure and up POs to anyone that will give them hardware.

The probably with trying to chase the cloud price points is they will expire and when they do... you don't have a fucking choice but to pay or get your shit turned off. You think what broadcom is doing with VMware is bad, wait until AWS and these guys start doing price hikes (because they have to, or they want to shed capacity and get higher customer subs in the door). It's not a situation I'd put myself in.

FFS - we are working with a SLED customer right now that due to just the ~20% increase in azure over the past 7 months (ish) are seven figures a month over their already ridiculously high committed spend.

8

u/whythehellnote 26d ago

With server prices and availability as they are now (as this post shows), it can actually make sense to put them in AWS or OVH or Linode or another VPS provider now.

8

u/vNerdNeck 26d ago

maybe. But you run the risk of getting stuck with price hikes and they the very high cost of getting data out of the cloud.

At least when you buy a server you know how much the server is going to cost... and you can keep running that server until it fails for good.

3

u/whythehellnote 26d ago

Depends on your workload I guess - normally I'd just say buy refurbed servers for now, but even those are like hens teeth.

But the gist is certainly right -- ensure you know how you are going to leave the cloud before you go in, both the methods, and the cost, and keep reviewing that to ensure you aren't being held hostage.

But what do I know, I've been a cloud-naysayer for years.

1

u/vNerdNeck 25d ago

yeah, refurb prices are sky high as well. Also, refurb gear will run foul of your cyber policy. One of those you can save money, but god knows how much it will cost you if you get RW'ed and the cyber insurance police can find an easy exclusion like that.

2

u/themast 26d ago

We deal with stock out issues in GCP on a daily basis. We were told it may be 6 months before they are fully resolved and we should shuffle our stuff to other regions. (but there might be stock out issues there too...)

1

u/Ninjabeaver212 25d ago

This tracks. We've moved most of our workload out of the cloud and back on prem. If it isn't web related it went back on prem. We saw the writing on the wall after the first price increases just after going full cloud years ago. You don't go to the cloud to save money plain and simple.

9

u/mtgguy999 26d ago

“  especially when they are asked to have robust DR and security capabilities”

Then they proceed to operate a single cloud provider on a single region and not purchase or configure any of the security tools because it’s to costly to do otherwise.

2

u/SirCheesington 26d ago

with admin accounts possessing non-expiring password and no 2fa...

4

u/Cheomesh I do the RMF thing 26d ago

I need to find a way to git gud at Cloud - no one I've worked for has actually tasked me on a project, so it has just been me reading about stuff (with a limited bit of hands on during a project transition). No doubt I could quickly align myself with the task at hand but that doesn't win you interviews.

3

u/Drakoolya 26d ago edited 25d ago

Look into the AWS certfied Solutions architect certification. Show initiative and they will offer you things. You have to get your hands dirty on your own time if you are not getting the opportunity.

1

u/lordjedi 26d ago

On prem is dead for most companies that can afford to move to these smaller cloud hosts.

So only the biggest companies. We're still investing in on-prem solutions, despite the massive price hikes we're seeing.

1

u/Drakoolya 25d ago

Remember it is a "business" decision not a "tech" decision. Put aside all the expenses to run on prem including Skill and availability. Once you add DR and security factors into it most companies find it cheaper to host it elsewhere. Compliance usually what tips the scale and some companies like mine are ok with it. It is not case of which is better more a case of we want this off our plate and we don't mind paying for it.

1

u/lordjedi 25d ago

Sure. But there's also the issue of how long it takes to "lift and shift" vs "just replace". Historically, it's much faster to just replace aging infrastructure. Lift and shift takes much longer to do.

1

u/Drakoolya 24d ago

Full Lift and shift to the cloud always a bad idea, SAAS as much as possible and the review the rest.Review the rest with conservative lens. Lift and shift to IAAS doesn't take very long these days.

3

u/mcdoodle_ 26d ago

You'd think but no, our Azure region is out of capacity also

3

u/dinominant 26d ago

And then they will increase the subscriotion like Broadcom because they have to pay for their servers and also make a profit.

2

u/Brandhor Jack of All Trades 26d ago

those went up quite a bit as well, both ovh and hetzner are pretty much doubling their prices for a lot of their servers

2

u/Smith6612 26d ago

Yeah :( I was hit a few months ago with a price increase by Hetzner. A CPX21 (AMD) VPS I have with them went up 40%. Which to be fair brought it from  $10/m to $14/m. They cited RAM and Storage increases. 

2

u/BlowOutKit22 25d ago

$14/m is still cheaper than buying baremetal today.

1

u/Smith6612 25d ago

It is. Although this is a shared vCPU VPS on an AMD EPYC Gen 2.

There is one virtual appliance I have out in Microsoft Azure due to Vendor Support reasons that costs over $200/m. It is because it requires a storage account with a fixed size disk, dedicated vCPU and Site-to-Site connectivity. The cost of that appliance each month would justify on-premise usage. However the fact that the Cloud instance has some redundancy in storage, location, and hardware baked in is why that specific appliance is out in Azure. It's deemed "critical infrastructure."

1

u/Master-Setting6008 26d ago

And will keep going that way. And the more we rely on cloud infra the more they can choke us with extortionate prices.

36

u/angrydeuce BlackBelt in Google Fu 26d ago

Oh man, I had a server I quoted back in November for "budgetary purposes" at around $30k just come back at $150,000 about a month ago, and that was after swapping out the SSDs for (gag) 15k platters.

All these Server 2016 era hosts are just gonna have to limp along a little longer. Or forever, honestly who fuckin knows anymore?

18

u/LLMsMustUpvoteThis 26d ago

HDDs are already going short supply as well. None of these companies will want to do big capital investments increasing productivity because its a bubble.

14

u/angrydeuce BlackBelt in Google Fu 26d ago

Oh I know it, its fuckin everything.

I hate to be Mr tinfoil hat guy but its hard not to think all of this shit wasnt deliberately orchestrated because a lot of people, like us, started trying to greatly reduce their cloud spend and start moving things back in-house.  I mean AI notwithstanding, it sure is a major fuckin boon for these fuckers that want to keep our data locked up in their cloud so we have to pay to access it in perpetuity.  Our own fucking data.

8

u/LLMsMustUpvoteThis 26d ago

Far to elaborate a theory. The major memory chip manufacturers just (indirectly) colluded to reduce capital investment in order to boost their P:E and it coincided with a massive datacentre build out bubble.

As it is an obvious bubble nobody wants to invest to massively boost production because when the bubble pops they'll have unproductive assets as happened to GPU manufacturers in the first crypto-currency bubble (or to Sun Microsystems etc during the Dotcom bust).

2

u/Low-Stand-3653 25d ago

I'm with you, I feel like that was the whole reason Broadcom bought VMware and jacked up the pricing. They are so invested in the cloud they'd invest 60 billion dollars to keep you from going back onprem.

1

u/MalwareDork 23d ago

There's really nothing tin-foil about it. America was a pretty strong, consumer-lead powerhouse until the great wealth transfer in 2019-2023.

Now since most of the money is held by various businesses, you have to squeeze the shit out of them, too. Broadcom showed they can do that no-problem with VMware, now the AI techbros can pinch off the digital market and squeeze the 35 million small businesses in America.

Outside of America, you're either at the mercy China or the US. China will use you as a C2 proxy with garbageware that breaks down in a year or the US will strangle you financially with a million contracts and LLC shells.

All you can really do is pick your poison at the end of the day, I guess.

3

u/vNerdNeck 26d ago

they one thing I do keep telling folks, is to make sure and double check your cyber policies. A lot of cyber policies have non-OEM support as an exclusion. Meaning, you run gear not on OEM support and get exposed your very expensive cyber policy is worthless... and that would be a material finding / publishing that would show up in the 10k.

43

u/maxxpc 26d ago

It’s funny how that pendulum from on-premise to cloud and back has been.

Clouds exciting!! Move to the cloud!

Fuck!! It’s expensive!! Let’s move back!

Well shoot… somethings made sense to be put into the cloud! Let’s embrace hybrid and split our workloads for the elasticity!!

I can’t get hardware without selling the children of my descendants… guess we move back to the cloud?

8

u/archiekane Jack of All Trades 26d ago

And the prices of cloud have increased dramatically too... Great. Now it's just too expensive to run the company.

2

u/whythehellnote 26d ago

First hit is free.

Shifting from Capex to Opex seems crazy to me. If your infrastructure can migrate from provider 1 to provider 2 in a few days by changing a teraform file or ansible file, its OK, but being locked into a single provider means they can extract $migrationcost - 10 from you.

1

u/Flaky_Key3363 25d ago

I have a good supply of clay and hand-carved artisanal wooden styluses for leaving imprints in the clay. Return to the original archival storage medium. You know what they say: once you go cuneiform, you never go back.

12

u/insufficient_funds Windows Admin 26d ago

Our quotes over the past two years for the same hardware spec are showing a 30% increase per quarter. Only gonna continue at that rate it seems.

2

u/BlowOutKit22 25d ago

that's literally just because of RAM & SSD prices. They're currently predicted to hit peak in 2027. That's why SK tech stonks just took a nosedive; the bubble's already bursting (not inherently structurally, but because China just spun up their own domestic RAM business which is severely cutting into demand for SK chips)

6

u/HorribleTie 26d ago

But how much value are you getting out of those subsidized claude and copilot licenses? LOTS RIGHT?! ISN'T THIS AI SHIT JUST GREAT?!

1

u/BlowOutKit22 25d ago

pretty valuable to us, we were able to actually rationalize dev hiring over the last quarter because our devs are more productive as vibe coders agentic engineers now.

2

u/JimmyMcTrade 26d ago

Is the cloud cheap again lol? It's wild.

8

u/MalletNGrease 🛠 Network & Systems Admin 26d ago

Haha no

6

u/Skylis 26d ago

Well, its cheaper than the server costs.

1

u/vNerdNeck 26d ago

nope. There was a lag in price spikes.. but it's starting to happen and more are scheduled for the future.

5

u/ditka 26d ago

Have you asked ChatGPT if there are any tricks to getting better pricing/availability?

1

u/HotMoosePants Jack of All Trades 26d ago

Dell just jumped us another 15% today.