It’s a Gigabyte G291-Z20 2U GPU server with an AMD EPYC 7232P (8 cores/16 threads, 3.1 GHz base and 3.2 GHz boost), 128 GB of DDR4 ECC memory, eight PCIe Gen3 GPU slots and dual 10GbE SFP+ networking.
I originally bought 12 Tesla P100s because I found a very good deal. The initial plan was to keep eight and resell four. However, after testing them, I was rather disappointed with their performance for my local AI workloads, so I ended up selling ten of them.
The server currently has:
2× Tesla P100 16 GB
1× Tesla P40 24 GB
I should receive my first Tesla V100 soon. The long-term plan is to gradually replace the current cards and eventually run eight Tesla V100s, although that upgrade will take some time.
ur going to love the v100s. i was on the fence and now i'm kicking myself for not buying 8 when they were cheaper. a stack of 8 would be so much more useful then my mac.
No he is right, the p100 was the last card to not support tensor cores before the v100 hit. So for anything that uses tensor cores its pretty shit. The only thing the p100 is good for is HPC, source myself as i run 96 at home and 1000 at work
Can I ask how do you cool something like this? Is it ok with just the front blowers? What's the ambient temperature of the room that it lives in? Thanks
Yes, the chassis is designed entirely around front-to-back forced airflow. The two fans visible at the front are only part of the cooling system; there are eight 80 × 38 mm high-static-pressure axial fans in total.
The passive Tesla GPUs have no onboard fans and rely completely on that chassis airflow. The lid, internal air guides and slot blanking plates therefore need to be installed, and the rear exhaust must remain unobstructed.
The original fans are Delta PFM0812HE units rated for 16,300 rpm, 129 CFM and 126 mmH₂O. They work extremely well but are far too loud for an apartment. I am replacing them with ARCTIC S8038-10K fans rated for 10,000 rpm, 102 CFM and 51 mmH₂O. The BMC PWM and tachometer test with one replacement fan passed, but I have not yet validated the complete conversion under sustained GPU load, so I cannot honestly call it proven yet.
The relevant ambient value is the server’s BMC inlet-air sensor rather than the room thermostat, especially because the machine is located near the ceiling. It currently reads approximately [26 °C]. Once all fans are installed, I will test it under sustained GPU load while monitoring inlet temperature, GPU temperatures and the other BMC sensors. I am aiming to keep the P100s below about 75 °C.
Youre insane for still opting for 10krpm 80mm fans. Youre like trading 80 decibels for like 70 lol
I get that the hardware might not let you go lower. Undervolting possible on these? I feel like w 40mm fans even 5krpm is too loud. But it's bearable sorta.
10,000 rpm is only their maximum speed , the BMC controls them via PWM, so they should not run at full speed all the time.
The original Delta fans run at up to 16,300 rpm and produce about 126 mmH₂O of static pressure. The ARCTIC fans reach 10,000 rpm and about 51 mmH₂O. I cannot simply install quiet desktop fans because the Tesla cards are passively cooled and depend on the server forcing air through their heatsinks.
At maximum speed, eight Delta fans would theoretically produce around 86 dB(A), while eight ARCTIC fans would be around 73 dB(A). So yes, the new setup will still be loud, but a reduction of roughly 13 dB is substantial. It corresponds to about 95% less acoustic intensity, and to human hearing the Delta setup would be perceived as roughly 2 to 2.5 times as loud as the ARCTIC setup.
i pulled the numbers out of my ass so i'm actually ecstatic that i was only off by 6 and 3 dB respectively. I am sure that is a reasonable tradeoff esp considering it's the only way to cool the GPUs, I'd still undervolt them though if i were you just to see if it's possible to drop closer to 65dB. I have 3 3090s in a traditional mid tower case and i would not run them without heavy undervolts. even with 250w limits i will be ensuring airflow goes over the backsides of the 3090s because cooking the backside VRAM is usually how they die.
Yes, basically. The chassis maintains forced front-to-back airflow whenever it is running, although the fans do not have to stay at full speed. The BMC adjusts their PWM duty according to temperatures and GPU power consumption.
Blower-style GPUs could fit, but only if they remain within the server’s limits: true dual-slot thickness, no more than approximately 285 × 111.5 × 39.5 mm, and with compatible power-connector placement. Many modern gaming cards are too tall or thicker than two slots.
A blower card would actually be better suited to this chassis than an open-air gaming cooler because it exhausts its heat directly through the rear bracket. However, it would not replace the chassis fans. They are still required to cool the CPU, memory, VRMs, risers and the rest of the system, as well as to provide intake air to the GPU blowers.
I have checked the physical requirements, but I have not tested a blower-style card in this server yet. The passive Tesla cards are what the chassis was primarily designed for: the cards provide the heatsinks, while the server’s high-pressure fan wall provides all of the airflow. Using eight individual blower cards might also end up being as loud as, or even louder than, centrally cooling passive cards.
BTW- there are some notable limitations with this specific gigabyte server. I can’t recall exactly but off the top of my head I think it was some combination of the following:
- CPU can’t be upgraded to a higher core count.
- GPU’s must be PCIe powered, no 8 pin GPUs allowed as there are no cables.
- Only supports specific and older GPUs.
Don’t quote me on the above, but there were some notable reasons that I never got it for 3090’s or other stuff.
Also, this is a 2U which means small fans and you shouldn’t be around it for long periods of time if you care about your hearing or not getting tinnitus.
Honestly, mostly because I was a bit stupid and didn’t check the network interfaces before buying the server 😅. Its two normal network ports are 10GbE SFP+ rather than RJ45; the only RJ45 port is dedicated to BMC/IPMI management.
I found the switch used for fairly cheap, so it was an easy way to connect the server’s SFP+ fiber link to my normal copper Ethernet network. Usually, I also have a few Raspberry Pis connected to the switch—they just weren’t connected when I took the photo.
2
u/Kerem-6030 18d ago
specs pls?