Supermicro X10/X11 Idiosyncrasies

Supermicro X10/X11 Idiosyncrasies

January 5, 2020
Hardware
Supermicro

We’d standardized on the Supermicro X10SRH-CLN4F a set of deployments a while back, the last of our X10 deployments, and these boards are still kicking around with enough idiosyncrasies that it’s worth documenting. This is an uncomfortable generation of boards from Supermicro that support a lot of interesting new features, but not as easily as X11 or H11. For example you have to do most things via DOS on X10, but on X11+ and H11+ almost all hardware operations are handled via built in EFI shell. One thing we’d used these for is connecting to SAS expanders to address many drives without needing a dedicated card. As these have limited PCIe lanes, and the X10SRH-CLN4F doesn’t actually have an x16 slot, using the on-board controller made the most sense. As we’re using ZFS on Linux we want to be able to individually address the drives, the best choice for the SAS controller is to function as an HBA, which it can do via “Initiator Target” mode firmware. These boards have been work horses for a while for us, however the future appears to be very bright with ARM and EPYC adoptions… X11 may be the last of our Intel deployments.

BIOS Updates #

Be warned about using the web based IPMI to flash BIOS updates, I’ve personally had it brick systems likely due to how their browser interactions (e.g js) are meant to function. Personally the safest path has always been to flash the BIOS using a DOS boot utility, which sadly means you have to garner physical access to the machine.

BIOS Boot #

The BIOS will boot quite a bit faster if you modify all of the subsystems under Advanced > PCIe/PCI/PnP Configuration to boot as “EFI”. This will prevent optionroms from being loaded.

Shell based IPMI Tools #

I advised against using the web based IPMI for BIOS updates, the real power comes from ipmitool in conjunction with the terrifying little out of band controller. You need to:

  • leave the SoL settings default in the BIOS
  • set up users on the IPMI system
  • realize that you should never allow access to the IPMI system from the Internet
  • add some settings to your bootloader, for example here’s grub:
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --work=8 --parity=no --stop=1"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS1,115200n8"

Now you can use ipmitool to manage the system from the command line over ssh, this command will drop you into a Serial-over-Lan tty:

ipmitool -H 172.16.0.9 -U agd -I lanplus sol activate

You can also power on and off the system with chassis commands:

ipmitool -H 172.16.0.9 -U agd -I lanplus chassis power on

ipmitool -H 172.16.0.9 -U agd -I lanplus chassis power off

This combined with a proper VPN allows for you to implement LUKS boot encryption on all your machines with very long random character passwords as you can just paste them in over SoL.

Flashing Initiator Target Mode #

You’ll need to launch the SAS configuration utility (ctrl+c at boot), this allows you to examine the adapters in the system (only one if you’re just using the on-board). You need to write down the SAS address for each card, specifically the last 9 digits. This is a required input during the flashing process.

You will need to get a copy of the most recent firmware from Supermicro. The contents (specifically the IT/UEFI folder) need to be unpacked onto a FAT32 filesystem and be connected to the machine.

Invoke the boot menu (f11) and select “UEFI: Built-in EFI Shell”, if you don’t see this then you need to make sure it is available in the boot options of the BIOS setup.

Once in the EFI shell you need to navigate to where you’ve unpacked the firmware update files (likely a USB stick, which is also likely located at fs0):

Shell> fs0:

fs0:\> SMC3008T.NSH

The process begins and goes on for a little while. Eventually it prompts you for an address, which is the last 9 digits of what you examined earlier.

Adding NVME #

The X10SRH-CLN4F doesn’t support any NVME via M.2 or other connectors, where the X11 and H11 almost all have this option. Having small NVME drives for our boot/root spaces and having all of the “important” files on ZFS is generally how our deployments look. To get NVME into these systems we’ve used the AOC-SLG3-2M2 addon card. The only difficulty is that these cards require “PCIe Bifurcation” which means you really have to worry about where your manufacture has put bridges in the board. There is no public documentation about where the bridges are for these systems, instead you have to contact Supermicro support directly and they will reply from the internal knowledge base, heres the common Intel boards I’ve been working with to date (AMD boards seem to work automatically):

X10SRH-CLN4F, CPU root port of PCIe slot
Slot 4 => IOU1 (IIO1 PCIe Port 3A), B0:D3:F0
Slot 5 => IOU0 (IIO1 PCIe Port 2A), B0:D2:F0
Slot 6 => IOU0 (IIO1 PCIe Port 2C), B0:D2:F2
Slot 7 => IOU1 (IIO1 PCIe Port 3C), B0:D3:F2

X11SPM-TPF:
        Slot 4 -> IOU 0 Br1, B16:D0:F0
        Slot 6 -> IOU 1 Br2, B64:D0:F0
        Slot 7 -> IOU 2 Br3, BB2:D2:F0

X11SPH-NCTPF:

IIO0(IIO PCIe Br1):
P0PE1A[B16:D00:F00] -> OcuLink, JNVME1 [B17:D00:F00] P0PE1B[B16:D01:F00] -> OcuLink, JNVME2 [B18:D00:F00] P0PE1C[B16:D02:F00] -> LSISAS3008 [B19:D00:F00]

IIO1(IIO PCIe Br2):
P0PE2A[B64:D00:F00] -> PCH PCIE UPLINK
P0PE2A[B64:D02:F00] -> PCIE SLOT3 [B68:D00:F00]

IIO2(IIO PCIe Br3):
P0PE3A[BB2:D00:F00] -> slot5/6 [BB3:D00:F00] P0PE3C[BB2:D02:F00] -> slot5/6 [BB4:D00:F00]

For the X10SRH-CLN4F the only slot I’ve gotten working is “Slot 5” or “Slot 6”, which requires you to set “IOU0 (IIO1 PCIe Port 2)” to “x4x4x4x4”. This is located in the BIOS under Advanced > Chipset Configuration > North Bridge > IIO Configuration > IIO1 Configuration. Presumably other slots can work if you examine the table above, but “Slot 5” or “Slot 6” was the only ones that have worked consistently across these boards.