AMD EPYC 9555P + SuperMicro H13SSL-NT Home Lab Server
Table of Contents
In early 2025, I built a home server using the newly released AMD EPYC Zen 5 platform. My goal was to have a powerful, multi-tenant research server in my home office, capable of handling memory and IO-intensive workloads, including databases, blockchain nodes, and virtualization. This page gives a detailed look at how the build and setup went. I will continue to update it with benchmarks and observations as time goes on.
If you have thoughts, comments, suggestions on this article that you'd like to share with me, send an email to website at this domain.
Figure 1: ChatGPT amalgamation of photos of the assembly process.
1. Parts List
Here I'll write a bit about the parts I've chosen and why I picked them, and I'll talk about my thoughts on them after buying and installing them in later sections.
New parts:
- Motherboard: SuperMicro H13SSL-NT Rev 2.0
- CPU: AMD EPYC 9555P (64 cores)
- RAM: 12x Micron 64GB DDR5-5600 RDIMM 2Rx4 CL46 (768GB)
- Storage: Micron SSD 9400 PRO U.3 15.36TB PCIe Gen4x4
- CPU Cooler: SilverStone SST-XED120S-WS
- RAM Cooler: 2x Corsair Vengeance Airflow
- PSU: Seasonic PRIME TX-1300 ATX30 Modular 1300W
Reused parts:
- Case: Lian Li Lancool 216
- GPU: Old GPU (for initial setup only)
- Boot SSD: 1TB Sabrent M2 disk
- HDDs: 2x 16TB Seagate HDDs
When I started my investigation I wasn't sure if I wanted to get HEDC (High End Desktop Computer) or server grade parts. I wanted the machine to be performant but not at the expense of stablility. Going the HEDC route I could get up to 6400MT/s, but with non-JEDEC voltages, eg AMD EXPO and Intel XMP. There are various vendors selilng packs of 4 or 8 sticks of RAM designed specifically for these overclocking profiles already vetted for specific motherboards so this was attractive, for example https://www.kingston.com/en/memory/gaming/fury-renegade-ddr5-pro-rdimm. I read reports that they do run quite hot when overclocked. They are cheaper than server parts of the same or faster speeds because in reality they are just 4800MT/s parts that have been overclocked. The Zen 4 AMD Threadripper has 4 memory channels and the Pro has 8. For server grade memory the only that I could find that was available was DDR-5600, but this is running at standard JEDEC voltages and timings so it is likely to be more stable and run at a lower temperature. With an EPYC processor I would have 12 memory channels instead of 4 or 8. I decided to go down the server parts route; since this system to be heavily multitenant, I reasoned that memory bandwidth may be more imporant than a small speed increase, and also because I don't want to deal with potential instability from overclocking. The AMD EPYC 9555P was chosen for its exceptional core count and performance-to-price ratio within budget, and future upgradeability to DDR5-6400 memory when it's cheaper in the future.
Figure 2: The 9555P proessor in bulk packaging.
Figure 3: 768GB of DDR-5600 RDIMMs
At the beginning of my research into parts I read the Phoronix articles by Michael Larabel (https://www.phoronix.com/review/supermicro-h13ssln-epyc-turin https://www.phoronix.com/review/corsair-ddr5-ram-cooling-server) about his experience setting up an EPYC 9005 series system and it was invaluable for helping me select the motherboard. SuperMicro is a solid enterprise brand, and the H13SSL-NT has supports all 12 channels of memory and fits in an ATX case. I went in knowing that enterprise motherboards are kind of a different beast to the consumer boards I was used to, with a little bit of trepidation after reading reports of fan issues, firmware, case incompatibility; all of which I had to deal with later.
Figure 4: The H13SSL-NT mounted in the Lian Li case.
The Micron SSD provides ample high-speed enterprise grade storage and I wanted something simple and reliable on the software side compared to making a large logical volume over a bunch of consumer SSDs. The "enterprise grade" features include: over 10x the endurance, measured by TBW, of prosumer SSDs like the Samsung 990 Pro, and power loss capacitors to flush pending writes.
Figure 5: The Micron 9400 Pro mounted on a StarTech U.3 to PCIE adapter.
The SilverStone air cooler was chosen specifically because it provides effective yet quiet cooling, and also because at the time I couldn't find a vendor in Switzerland for the XE360-SP5 water cooler from the same brand. I may still switch to the XE360-SP5 or some other water cooler in future for space savings in the case and airflow improvement. The RAM coolers were also suggested in the Phoronix articles.
Figure 6: The SilverStone SST-XED120S-WS prepared for install, showing the included thermal pad.
I haven't owend a Seasonic PSU before but it is known to be a rock solid enterprise brand, and I wanted ample headroom in case I add GPUs to this machine.
I was researching large workstation cases with noise dampening properties and so was deciding between the Fractal Design Define 7 XL or the quiet! Dark Base Pro 900, but we had an extra Lian Li Lancool 216 lying around. From what I read about it, it's not particularly quiet but has decent airflow for cooling, so I decided to use it.
2. Assembly
Figure 7: The SP5 socket opened up with protective cover removed.
2.1. CPU Heatsink
Assembly of this build was fairly easy except for the heatsink (SilverStone SST-XED120S-WS). The six bolts on the included SP5 bracket that attach to the mount points on the motherboard are tensioned with very strong springs. The manual says to tighten the middle bolts first, then an alternating pattern on the four corners. When you sit the bracket on the mount points, the middle bolts will reach the mount points with just a moderate amount of force, but even after tightening them completely, the tension on the springs of the corner bolts cause them to sit above the mount points and not bite into the thread when pressed in with a moderate amount of force. Annoyingly I had only a small T20 screwdriver so I could not apply enough force to and turn at the same time because the entire screwdriver fit in the screw bolt cutout in the heatsink. Without the heatsink fully seated, the machine would boot only to POST screen before resetting after trying to enter setup, probably due to thermal monitoring. I was ready to give up for the day, but fortunately there was a local hardware store open, despite it being a public holiday, and I was able to get a longer T20 screwdriver. The right tool can make all the difference, and with that I was able to apply the concerning amount of force required to depress the springs enough to fasten it all the way.
Figure 8: Fastening the heatsink.
2.2. RAM coolers
The Corsair Vengeance Airflow coolers snap onto the RDIMM sockets with plastic arms. The coolers themselves are quite wide and the size of the CPU heatsink means they're not able to cover all the RDIMMs and the lower one also stick out over the first PCIE slot, which is not ideal.
Figure 9: The bottom cooler actually sticks out over the first PCIe slot. I was still able to fit one of the Sabrent SSDs in a Sabrent M2 to PCIe adapter.
2.3. Case
Overall I am quite impressed by the cable management features of the Lian Li case for keeping cables out of the way of the airflow. The case itself is quite bare bones which appeals to me in a certain way. It has two different motherboard positions that they call water and air cooling modes. The water cooling mode leaves more space at the top for a radiator. I mounted the motherboard in water cooling mode, even though I'm currently using air cooling mode; in case I change the cooler later I didn't want to deal with disassembling everything to move the motherboard a few centimeters to make room for the radiator.
Figure 10: Lian Li Lancool 216
An interesting detail is that the H13SSL-NT motherboard does not have a standard ATX header for the LEDs and buttons on the front panel. SuperMicro makes their own cases too, so they probably intend for you to use theirs. The motherboard manual has the pinout so it was easy to adapt the power button, reset button, and power LED to the case's ATX connector using some jumper cables.
Figure 11: Makeshift SuperMicro to standard ATX cable adapter.
3. Firmware
3.1. IPMI
I didn't have a VGA adapter to be able to use the onboard graphics so I put an old graphics card in, but nothing was coming up on the screen right away when I booted. In order to try to figure out what was going on, I decided to learn about IPMI, since there is a special IPMI LAN port on the back IO panel. The H13 has an IPMI BMC which by default tries to get an address over DHCP, and it starts a webserver you can log into as well. While I was figuring out how to set up a DHCP server on my laptop, the EPYC machine finished booting to POST and displayed it on screen - I think it was just doing an initial memory check for the first boot. The H13 board still takes a long time to POST, but not as long as the first boot. Probably, being a server board, it's not designed to be restarted very often. Once at the POST screen I could enter setup and change the IPMI server to use a fixed IP, then I didn't have to bother with DHCP. IPMI gives you a web and CLI interface for managing the system firmware and settings, which was useful for subsequent steps.
I used the web interface to update the BIOS and IPMI firmwares at this time.
Tip: IPMI login defaults: ADMIN (user), password located on motherboard sticker.
Figure 12: IPMI web console showing completed BIOS update.
3.2. Fan Speed Issues
Initially, the server fans triggered constant low speed alarms because the default RPM thresholds on the Supermicro board were too high for my large, low-RPM case fans. When the fan low speed alarm triggers, all fans are set to full, which was causing an oscillation between a low whirr and a vaccuum cleaner every 15 seconds. This was obviously super annoying so it was one of the first things I had to fix. I'm more used to the super fine-grained control that's available for fan speed settings, for instance on Gigabyte Aurus motherboard it actually lets you draw a response curve for temp to fan speed, for each fan, so I was a little disappointed by the fan controls provided by the IPMI firmware on the H13SSL-NT. It only provides 4 preset profiles "Standard", "Full", "Optimal", and "HeavyIO", and only "Full" allows manual control. I think because SuperMicro mostly sells to datacenters, they don't really care about controlling for noise. There's a long thread here about SuperMicro IPMI fan control which I found very helpful for understanding how to fix the situation: https://www.truenas.com/community/threads/script-to-control-fan-speed-in-response-to-hard-drive-temperatures.41294/
Setting the profile to "standard", which is an adaptive profile, and then adjusting thresholds (orignially 420) via IPMI commands fixed the problem quickly by allowing the case fans to settle at about 360RPM when the machine is idle.
ipmitool -I lanplus -H <ip> -U ADMIN -P <password> sensor thresh FANA lcr 100 ipmitool -I lanplus -H <ip> -U ADMIN -P <password> sensor thresh FANB lcr 100
The Corsair RAM fans only have 3 pin cables, so have no PWM speed control, and due to their small size they were a bit noisy initially running at almost 4000 RPM, but conveniently the packaging included resistor cables. This significantled reduced the noise, but it is still a bit noisy for my liking so I will probably experiment with different resistor values in future.
$ sudo ipmitool -I open sensor list CPU Temp | 46.000 | degrees C | ok | na | 5.000 | na | na | 95.000 | na Inlet Temp | na | | na | na | na | na | na | na | na System Temp | 34.000 | degrees C | ok | na | 5.000 | na | na | 85.000 | na Peripheral Temp | 46.000 | degrees C | ok | na | 5.000 | na | na | 85.000 | na NIC Temp | 61.000 | degrees C | ok | na | 5.000 | na | na | 100.000 | na CPU_VRM0 Temp | 48.000 | degrees C | ok | na | 5.000 | na | na | 100.000 | na CPU_VRM1 Temp | 41.000 | degrees C | ok | na | 5.000 | na | na | 100.000 | na SOC_VRM Temp | 52.000 | degrees C | ok | na | 5.000 | na | na | 100.000 | na VDDIO_VRM Temp | 42.000 | degrees C | ok | na | 5.000 | na | na | 100.000 | na DIMMA~F Temp | 41.000 | degrees C | ok | na | 5.000 | na | na | 85.000 | na DIMMG~L Temp | 40.000 | degrees C | ok | na | 5.000 | na | na | 85.000 | na MB 12V | 11.915 | Volts | ok | 10.571 | 10.739 | na | na | 12.839 | 13.091 MB 5VCC | 5.058 | Volts | ok | 4.594 | 4.636 | na | na | 5.564 | 5.606 MB 3.3VCC | 3.303 | Volts | ok | 3.022 | 3.069 | na | na | 3.654 | 3.678 VBAT | 0x4 | discrete | 0x0400| na | na | na | na | na | na MB 5VSB | 5.058 | Volts | ok | 4.594 | 4.636 | na | na | 5.564 | 5.606 MB 3.3VSB | 3.258 | Volts | ok | 2.966 | 3.019 | na | na | 3.602 | 3.629 MB 1.8VSB | 1.850 | Volts | ok | 1.553 | 1.642 | na | na | 2.058 | 2.077 BMC 2.5V | 2.471 | Volts | ok | 2.392 | 2.412 | na | na | 2.608 | 2.627 BMC 1.8V | 1.781 | Volts | ok | 1.728 | 1.741 | na | na | 1.899 | 1.912 BMC 1.2V | 1.190 | Volts | ok | 1.151 | 1.161 | na | na | 1.259 | 1.269 BMC 1.0V | 0.990 | Volts | ok | 0.961 | 0.971 | na | na | 1.049 | 1.059 CPU_VDDCR0 | 0.764 | Volts | ok | 0.515 | 0.530 | na | na | 1.576 | 1.654 CPU_SOC | 0.858 | Volts | ok | 0.577 | 0.608 | na | na | 1.357 | 1.420 CPU_VDDCR1 | 1.123 | Volts | ok | 0.515 | 0.530 | na | na | 1.576 | 1.654 CPU_VDDIO | 1.092 | Volts | ok | 0.749 | 0.780 | na | na | 1.342 | 1.404 FAN1 | 3080.000 | RPM | ok | na | 420.000 | na | na | na | na FAN2 | na | | na | na | na | na | na | na | na FAN3 | 3080.000 | RPM | ok | na | 420.000 | na | na | na | na FAN4 | 560.000 | RPM | ok | na | 140.000 | na | na | na | na FANA | 700.000 | RPM | ok | na | 140.000 | na | na | na | na FANB | 700.000 | RPM | ok | na | 140.000 | na | na | na | na M2_SSD1 Temp | na | | na | na | na | na | na | na | na M2_SSD2 Temp | na | | na | na | na | na | na | na | na Chassis Intru | 0x0 | discrete | 0x0000| na | na | na | na | na | na
3.3. Broadcom network controller
After fixing the fan issues and installing Ubuntu 24.10, I noticed that Ubuntu was super slow to boot. Looking at the journalctl logs I could see that there were errors related to the BCM57416 network controllers. It would show "bnxten.rdma.0: Worker [3041] processing SEQNUM=11295 is taking a long time" after 5 minutes of doing nothing before continuing to boot, along with other errors including a lot of stack traces. I read about other users having trouble with issues like this and they often reported that updating the firmware on these controllers helped, like in this thread: https://forum.proxmox.com/threads/broadcom-nics-down-after-pve-8-2-kernel-6-8.146185/
The original firmware versions installed were:
Active Package version : 226.1.107.1 Package version on NVM : 226.1.107.1 Firmware version : 226.0.145.0
I found the firmware for the BCM57416 on the Broadcom website (not the easiest to search).
Installing latest firmware with niccli fixed all the errors and made startup reasonably quick.
$ nmccli install -recover -force <image>
The image needed to be installed to both controllers. After a reboot, nmccli showed the following information:
Description : Broadcom NetXtreme E-Series Dual-port 10GBASE-T Ethernet OCP 3.0 Adapter Firmware Name : CHIMP_FW Firmware Version : 230.0.168.0 RoCE Firmware Version : 230.0.168.0 HWRM Interface Spec : 1.10.3 Kong mailbox channel : Disabled Active Package Version : 230.1.123.0 Package Version on NVM : 230.1.123.0 Active NVM config version : 227.0.2 NVM config version : 227.0.2
3.4. OS
3.4.1. Filesystem
TODO
3.5. Software
TODO
3.6. Benchmarks
TODO