I wanted a way to add more NVMe drives to my server, and this seemed like a good idea.
I deployed two of these. Perfect by design. It’s a 24-lane PCI switch from IDT/Renasas, part number 89HPES24T6G2, plus a Marvel 88S9235 2-port SATA controller on the back. It is an 8-lane physical and logical connection at up to PCIe 2.0 speeds. Each M.2 slot supports any M-key SSD as long as they are both SATA or both NVMe. In NVMe mode, that’s a 4-lane PCIe slot. In total, it uses 16 of the 24 lanes, meaning the PCIe switch is never overloaded.
Oracle Virtual Machine Server 3.4.6-2105 with Linux kernel 4.1.12-124.21.1.el6uek.x86_64 automatically supports this switch, somewhat. When this card was serving up two NVMe drives, I could see them, but as soon as I wrote to either of them, it went into I/O aborts, timeouts, and hung until powered off. Two of these cards, 4 total NVMe drives. Not a bad device issue. I’m using OVM Server 3.4.6.2. Pulling the NVMe drives loose, and plugging them into single-drive risers, they work fine, so it’s specifically related to PCIe switching. There is an ASM card on the market that does the same thing, and maybe it works better. I have not tried it. There is a high-point raid card that some people have had success with.
Doing research, it seems there is still code work going on in Linux and FreeBSD for these kinds of errors. perhaps, some newer release would make this work, but that does not help for an enterprise Linux distribution today. There may be fixes in kernel 4.1.10 or 4.1.11.
Synology has specific hardware and specific OS builds to solve this problem, and some of it may be throttling the performance of each lane somewhat in the kernel, handling TRIM commands, handling reset commands, etc.
Maybe in OVM 4 and RHEL 8 we’ll see something stable, but today, I just can’t use these in a standalone Linux system.
That said, this is my limitation. The hardware is great. The heat sink assembly is well thought out. The screw kit is perfect, and it comes with both low and full profile backplates. It looks clean, well designed, and just overall a great piece of hardware. Also, the price is excellent for what it is. It’s very competitive. If you have a supported Synology storage array, or are on bleeding edge Linux, or are using SATA M-Key drives, then this could be a great option for you.
┌────────────────────────────────────────────────────────────────────┐
│ • MobaXterm Professional v11.1 • │
│ (X server, SSH client and network tools) │
│ │
│ ➤ Your computer drives are accessible through the /drives path │
│ ➤ Your DISPLAY is set to 192.168.1.100:0.0 │
│ ➤ When using SSH, your remote DISPLAY is automatically forwarded │
│ ➤ Each command status is specified by a special symbol (✔ or ✘) │
│ │
│ Registered to Josh Davis (1 user) │
└────────────────────────────────────────────────────────────────────┘
[2020-03-07 14:19.34] ~
[jdavis.JoshD6860] ➤ ssh vhost0 -l root
Warning: Permanently added 'vhost0' (RSA) to the list of known hosts.
root@vhost0's password:
X11 forwarding request failed on channel 0
Last login: Sat Mar 7 13:40:51 2020
Warning: making manual modifications in the management domain
might cause inconsistencies between Oracle VM Manager and the server.
[root@vhost0 ~]# cat /etc/redhat-release
Oracle VM server release 3.4.6
[root@vhost0 ~]# rpm -qa | grep kexec-tools
kexec-tools-ovs-2.0.16-100.3.el6.x86_64
[root@vhost0 ~]# vi /etc/default/grub
[root@vhost0 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.1.12-124.21.1.el6uek.x86_64
Found initrd image: /boot/initramfs-4.1.12-124.21.1.el6uek.x86_64.img
done
[root@vhost0 ~]# vgs
[root@vhost0 ~]# ll /dev/mapper
total 0
crw-rw---- 1 root root 10, 236 Mar 7 14:18 control
[root@vhost0 ~]# chkconfig kdump on
[root@vhost0 ~]# service kdump restart
Stopping kdump: [ OK ]
Starting kdump: [FAILED]
[root@vhost0 ~]# service kdump restart
Stopping kdump: [ OK ]
Starting kdump: [FAILED]
[root@vhost0 ~]# cat /etc/kdump
kdump-adv-conf/ kdump.conf
[root@vhost0 ~]# cat /etc/kdump.conf
# Configures where to put the kdump /proc/vmcore files
#
# This file contains a series of commands to perform (in order) when a
# kernel crash has happened and the kdump kernel has been loaded. Directives in
# this file are only applicable to the kdump initramfs, and have no effect if
# the root filesystem is mounted and the normal init scripts are processed
#
# Currently only one dump target and path may be configured at once
# if the configured dump target fails, the default action will be preformed
# the default action may be configured with the default directive below. If the
# configured dump target succedes
#
# Basics commands supported are:
# path - Append path to the filesystem device which you are
# dumping to. Ignored for raw device dumps.
# If unset, will default to /var/crash.
#
# core_collector
