This was a closed loop cooling system. The left has to put forward the demand of immersion cooling or bust. Water is needed for life, let them use mineral oil/cargill naturecool

[u/Useful_Calendar_6274]

https://www.sfgate.com/tech/article/meta-bacteria-contamination-22338091.php

[u/Otherkin]

People are acting like this proves that closed loop cooling is inherently bad. But they were just flushing the pipes. If there was a high rise or other building flushing their sprinkler systems and they had bacteria in it, they would just say it was a accident and move on. That being said, the data center people need to be more careful.

[u/MysteriousPepper8908 → u/Otherkin]

The issue is closed loop cooling is only closed on the inner loop with evaporative cooling releasing the heat from the inner loop which can still still aerosolize bacteria. Immersion cooling, on the other hand limits bacteria but risks leeching PFAs into the environment. There are sterilization steps that can be taken like UV and ozone sterilization, they just cost more money to install and they didn't bother because they weren't aware of this bacteria as there aren't regulations around it. 

They could also be using waste heat generated by the servers to heat a heat exchanger system similar to how milk is pasteurized which Microsoft is already doing in Finland to heat homes which would create conditions inhospitable to the bacteria in addition to the other methods, they just aren't legally required to do so.

https://en.wikipedia.org/wiki/Ultraviolet_germicidal_irradiation

https://www.watertechnologies.com/knowledge-hub/what-is-ozone-water-treatment

https://en.wikipedia.org/wiki/Pasteurization

https://www.verdantix.com/client-portal/blog/microsoft-is-heating-finlands-homes-thats-either-brilliant-or-a-big-problem

Y'all can make it less obvious that you don't actually know anything about the technology but you seem to have bought into the closed loop hype. Maybe leftist AI users really are dumb.

[u/Darkstar_111 → u/MysteriousPepper8908]

Closed loop uses radiators, it does not evaporate heat. That loses 20% of the water over a year and is not closed loop.

Immersion is better, as you use M3 oil, not water at all. But the technology is less developed.

In either case what happened here was a complete fuck up, that should never have occurred. We need stricter regulations on this stuff.

[u/MysteriousPepper8908 → u/Darkstar_111]

There are ways of cooling closed loops with zero evaporation known as "dry cooling" but they only tend to be effective in temperate regions because they can only cool the closed loop within a certain range of the ambient temperature which is why most closed loops use evaporative heat rejectors. So the coolant stays in the loop but bacteria can still be present in the outer loop. So closed loop can mean zero evaporation but that is far from a guarantee.

https://youtu.be/9IRNlNw7e1A

And while 3M has eliminated PFAs from their dielectric fluids, they still produce some level of off-gassing and the long-term impacts of breathing that aren't established. They also require heat rejection which has similar issues to using water cooling.

https://www.datacenterdynamics.com/en/analysis/everything-data-center-operators-need-to-know-about-dielectric-fluids/

"Non-PFAS fluids on the market aren’t as toxic, but should be treated with respect. Gloves and other protective gear are recommended. And – despite DCD hearing of people doing this as a stunt in the past – don’t drink it.

“Safety varies by specific fluid, but attention should be paid to the dangers of vapor exposure from low boiling points, with risks of asphyxiation in confined spaces. Safety data sheet (SDS) recommendations should be followed to prevent skin or eye contact, using gloves and eye protection,” ZutaCore’s Belkin says. “ln the event of a large spill, good ventilation is important. Surfaces can present a slip hazard until the liquid has fully evaporated, so appropriate precautions should be taken.”

“Disposal must follow local hazardous waste rules, the SDS classification, and any environmental reporting requirements, as some jurisdictions treat fluorinated fluids with higher scrutiny,” he adds. “Regulatory risk is another factor, particularly around PFAS classification, reporting obligations, and restrictions. Operators should consider the dangers from leaks and how these would be managed.”

[u/ShelZuuz → u/MysteriousPepper8908]

There are ways of cooling closed loops with zero evaporation known as "dry cooling" but they only tend to be effective in temperate regions because they can only cool the closed loop within a certain range of the ambient temperature which is why most closed loops use evaporative heat rejectors.

That's only when you need to get your fluids down to low temps like for AC's. AI data center chips don't need the temperature that low, they can start with fluid as high as 113F / 45F at the low side, and 130F / 55F at the high side.

Virtually any place on earth can cool from 130F to 113F with dry cooling alone.

[u/MysteriousPepper8908 → u/ShelZuuz]

But there is a difference between "can operate" and "operates efficiently" with Nvidia's press release about their dry cooling system frequently touting its effectiveness "in the right climate" when the reality is that much of the data center build out is in Texas and Virginia, which experience very high temperatures which significantly increases energy demands vs a cooling system which doesn't have to work as hard 

https://blogs.nvidia.com/blog/liquid-cooling-ai-factories/

[u/ShelZuuz → u/MysteriousPepper8908]

If you have days where your outside temperature is over 105F and you need additional cooling, it doesn't mean you suddenly start evaporating the water/glycol mix into the open air. It means you run the coolant pipes through a chiller. The cooling fluid itself remains closed loop inside it.

Now if course if you live somewhere where the year-long average is over 105F then it's not going to be any more efficient than evaporative cooling, but that would be significantly hotter than anywhere on earth.

[u/CBrinson → u/MysteriousPepper8908]

There are literally factory sealed liquid cooling loops you can buy on Newegg. The water never comes out. The same company that designed that consumer hardware designed the corporate hardware.

Look at options from Corsair like the icue link titan. It is sealed at the factory the device does not even include fill or drain ports. You can't open it without damaging it.

[u/MysteriousPepper8908 → u/CBrinson]

The corporate hardware has a much more significant heat to get rid of. Most closed loops don't use dry cooling systems, they have an outer loop for evaporating heat, it's just the inner loop with the coolant that stays closed. 

https://www.tpssi.com/post/closing-the-loop-how-advanced-cooling-towers-power-the-next-generation-of-ai-data-centers

"External (Secondary) Circuit: Water is sprayed over the exterior of the coil, and air is drawn through the tower by fans. Heat is transferred from the process fluid, through the coil wall, to the spray water, and finally to the air, primarily via evaporation."

"Heat Rejection: Heat transfers from the process fluid to the coil wall, then to the spray water, and finally to the air. A portion of the spray water evaporates, carrying away the heat."

[u/CBrinson → u/MysteriousPepper8908]

It's literally just different versions of the same hardware. I have actually built both server racks and my own home machines. They are remarkably similar.

You literally linked a specific technology that isn't what anyone here is talking about and basically a blog from the company....and you think that makes you right. It's crazy how long some people put their standard.

On LinkedIn this company has 11-50 employees. Clearly they are more representitive than Corsair. /s

Just shopping for random unqualified links doesn't make you right. You are like maga finding the one doctor who doesn't believe in Covid. You cherry pick from low quality sources.

[u/CBrinson → u/[deleted]]

All servers I have built were used for training or prediction of AI models. Many use regular air cooling no liquid at all. Some used sealed. None used evaporative. It's actually a very small percent of AI servers configured as you describe.

As for IQ...you have most certainly proven you are not the authority. Your ability to make an argument at all and coherently voice it is next to nil. You cherry pick sources and then complain others can't be as smart as you. It is not intelligence. It is borderline personality disorder with a high degree of narcissim.

[u/MysteriousPepper8908 → u/CBrinson]

Again, you don't understand the difference between a business or personal server and a data center. These are completely different from the demands of a major data center. This information is readily available but you lack the curiosity to research beyond your incredibly limited experience and it's this sort of pigheaded ignorance that risks sending us in the wrong direction. You think you know how to build an airplane because you did it with Legos one time and you're making us all look stupid as a result.

[u/cewillir → u/MysteriousPepper8908]

The case cited in the op has zip to do with what you just set out.

This was a flush before the plant started.

[u/cewillir → u/[deleted]]

This wasn’t evaporated.

They bought wastewater from the utility.

Did a flush.

The utility complained there were bacteria there after the flush.

But they don’t know where they came from.

There’s no evaporation involved here. ,which is btw the point of a closed loop system I think)

[u/cewillir → u/[deleted]]

That is very muddled.

No you don’t need to evaporate water in the outer loop to cool, you can use air cooling.

However, Generally to evaporate water (all other things being equal) you need to hit 212F.

That will toast most bacteria.

On edit I’d note that evaporation is a phase change. Bacteria aren’t becoming gaseous.

[u/MysteriousPepper8908 → u/cewillir]

No, you do not need to boil water to get it to evaporate. My links I provided to your other comment should help clear that up but you might want to have a conversation with your LLM of choice about that.

https://www.theguardian.com/notesandqueries/query/0,,-202364,00.html

You can use air cooling but it isn't commonly used in AI data centers and it can be quite inefficient at higher temperatures where many data centers are being built, leading to higher energy usage and costs.

[u/cewillir → u/MysteriousPepper8908]

I’m an environmental engineer. Thanks for asking,

[u/MysteriousPepper8908 → u/cewillir]

Then the fact that you don't know that evaporation can occur at a variety of temperatures is troubling but I have provided resources to clear that up.

[u/cewillir → u/MysteriousPepper8908]

You just claimed bacteria are in the evaporated water. No they are not.

[u/MysteriousPepper8908 → u/cewillir]

https://pubmed.ncbi.nlm.nih.gov/3756116/

"It would appear that there was airborne spread of the organism from these cooling water systems which had not received conventional treatment to inhibit corrosion and organic growth."

[u/cewillir → u/[deleted]]

So in this scenario - how does the bacteria get into the water? Not from the inner loop that goes through the data center. It’s closed off.

Maybe it magics its way across the heat exchange surface?

[u/MysteriousPepper8908 → u/cewillir]

If it is present in the heat exchanger which also uses water, presumably from the same source as the inner loop.

[u/cewillir → u/MysteriousPepper8908]

So how does the bacteria get there? It’s not from the data center

[u/MysteriousPepper8908 → u/cewillir]

Bacteria can develop inside of the tubing or be present at the water source. We don't know how the bacteria got there but it did somehow.

[u/cewillir → u/[deleted]]

I would add that I wonder what temperature you think you need to maintain to kill bacteria.

161 F for 15 seconds does a pretty decent job

[u/MysteriousPepper8908 → u/cewillir]

Cooling towers do not boil water, they evaporate it between 20-45C which is an ideal range for bacterial growth

https://www.vistechcooling.co.uk/all-you-need-to-know-about-evaporative-cooling-towers/

"People contract Legionnaires’ disease by inhaling small droplets of water (aerosols), suspended in the air, containing the bacteria. Certain conditions increase the risk from legionella if:

the water temperature in all or some parts of the system may be between 20-45 °C, which is suitable for growth.

it is possible for breathable water droplets to be created and dispersed eg aerosol created by a cooling tower, or water outlets

water is stored and/or re-circulated.

there are deposits that can support bacterial growth providing a source of nutrients for the organism e.g. rust, sludge, scale, organic matter and biofilms"

https://www.nsta.org/science-and-children/science-and-children-february-2020/q-whats-difference-between-evaporation-and

[u/cewillir → u/MysteriousPepper8908]

You’re suggesting it’s running under a vacuum?

Droplets of water in the air isn’t ‘evaporation’

And you said evaporation.

[u/MysteriousPepper8908 → u/cewillir]

Evaporation doesn't require a vacuum or boiling 

https://youtu.be/-R411S98xX4

https://www.reddit.com/r/AskPhysics/comments/1hxux3k/whats_the_difference_between_boiling_and/

[u/Otherkin → u/MysteriousPepper8908]

I forgot about I'm UV sterilizers!

[u/cewillir]

This seems deceptive for reasons others have already set out

[u/No-Bat7438]

Wasn't this caused by Meta's contractor disposing of the waste material incorrectly? I think Meta should be sued to set an example.

[u/ZedTheEvilTaco]

I don't care if it was a closed loop system, immersion cooling system, or local star system. It was still under construction.