Ubuntu LTS UEFI NVME Mirror Boot

This is super touchy, and here is what I did to make it happy and stable.

This does not address if UEFI decides to write to one of these mirrors.  Someone else has a systemd unit to assemble with resync.

In the past, I used someone else’s bypass script, but this was cleaner, and works in 18 and 20 LTS.

 

### Filesystem / Mirror for EFI / UEFI booting:
mdadm raid 1, metadata 1.0
vfat filesystem for /boot/EFI

### Proper GRUB package
apt-get purge grub\*
apt-get install grub-efi
apt-get autoremove

### Settings that seem to not stick
dpkg-reconfigure -p low grub-efi-amd64
Update NVRAM variables to automatically boot into Debian? NO
echo "grub-pc grub2/update_nvram boolean false" | debconf-set-selections
echo "grub-pc grub-efi/install_devices multiselect /dev/md0" | debconf-set-selections

### Grub config
update-grub
grub-install --no-nvram /dev/md0

### UEFI boot list (variables)
[root@tsm2: /root]
/bin/bash# efibootmgr -?
efibootmgr: invalid option -- '?'
efibootmgr version 17
usage: efibootmgr [options]
-a | --active sets bootnum active
-A | --inactive sets bootnum inactive
-b | --bootnum XXXX modify BootXXXX (hex)
-B | --delete-bootnum delete bootnum
-c | --create create new variable bootnum and add to bootorder
-C | --create-only create new variable bootnum and do not add to bootorder
-D | --remove-dups remove duplicate values from BootOrder
-d | --disk disk (defaults to /dev/sda) containing loader
-r | --driver Operate on Driver variables, not Boot Variables.
-e | --edd [1|3|-1] force EDD 1.0 or 3.0 creation variables, or guess
-E | --device num EDD 1.0 device number (defaults to 0x80)
-g | --gpt force disk with invalid PMBR to be treated as GPT
-i | --iface name create a netboot entry for the named interface
-l | --loader name (defaults to "\EFI\ubuntu\grub.efi")
-L | --label label Boot manager display label (defaults to "Linux")
-m | --mirror-below-4G t|f mirror memory below 4GB
-M | --mirror-above-4G X percentage memory to mirror above 4GB
-n | --bootnext XXXX set BootNext to XXXX (hex)
-N | --delete-bootnext delete BootNext
-o | --bootorder XXXX,YYYY,ZZZZ,... explicitly set BootOrder (hex)
-O | --delete-bootorder delete BootOrder
-p | --part part partition containing loader (defaults to 1 on partitioned devices)
-q | --quiet be quiet
-t | --timeout seconds set boot manager timeout waiting for user input.
-T | --delete-timeout delete Timeout.
-u | --unicode | --UCS-2 handle extra args as UCS-2 (default is ASCII)
-v | --verbose print additional information
-V | --version return version and exit
-w | --write-signature write unique sig to MBR if needed
-y | --sysprep Operate on SysPrep variables, not Boot Variables.
-@ | --append-binary-args file append extra args from file (use "-" for stdin)
-h | --help show help/usage

[root@tsm2: /root]
/bin/bash# efibootmgr -v
BootCurrent: 0019
Timeout: 5 seconds
BootOrder: 0005,0006,0007,000C,0019,0018
Boot0000 Startup Menu FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)....ISPH
Boot0001 System Information FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0002 Bios Setup FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0003 3rd Party Option ROM Management FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0004 System Diagnostics FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0005* nvme0_grub HD(1,GPT,e41eb9e0-6606-411a-bb83-bed7577f29b3,0x800,0x8e800)/File(\EFI\ubuntu\grub.efi)
Boot0006* nvme1_grub HD(1,GPT,aa23256a-95c6-4148-b56c-c8861fc7966a,0x800,0x8e800)/File(\EFI\ubuntu\grub.efi)
Boot0007* nvme2_grub HD(1,GPT,1f7f7f5b-2a89-4d87-a617-6ccaf15078dd,0x800,0x8e800)/File(\EFI\ubuntu\grub.efi)
Boot0008 Boot Menu FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0009* Kingston DataTraveler 3.0 408D5CE57214E331293064F6 BBS(USB,USB1,0x900)/PciRoot(0x0)/Pci(0x1d,0x0)......ISPH
Boot000B Network Boot FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot000C* nvme3_grub HD(1,GPT,cb8bc8b4-affc-4765-97c2-72af0c615d44,0x800,0x8e800)/File(\EFI\ubuntu\grub.efi)
Boot000E* IPV6 Network - Aquantia AQtion 10Gbit Network Adapter PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(88c9b3bfa1e9,0)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.....ISPH
Boot0010* IBA GE Slot 00C8 v1550 BBS(Network,Network1,0x0)/PciRoot(0x0)/Pci(0x19,0x0)......ISPH
Boot0011 USB: PciRoot(0x0)/Pci(0x1d,0x0)N.....YM....R,Y.....ISPH
Boot0012 HP Recovery FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0013* hp PLDS DVDRW DU8AESH PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0,0,0)N.....YM....R,Y.....ISPH
Boot0014* hp PLDS DVDRW DU8AESH BBS(CDROM,CDROM1,0x400)/PciRoot(0x0)/Pci(0x1f,0x2)......ISPH
Boot0018* ubuntu HD(1,GPT,e41eb9e0-6606-411a-bb83-bed7577f29b3,0x800,0x8e800)/File(\EFI\ubuntu\shimx64.efi)....ISPH
Boot0019* ubuntu HD(1,GPT,e41eb9e0-6606-411a-bb83-bed7577f29b3,0x800,0x8e800)/File(\EFI\grub\shimx64.efi)....ISPH
Boot001A* IPV4 Network - Aquantia AQtion 10Gbit Network Adapter PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(88c9b3bfa1e9,0)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.....ISPH

[root@tsm2: /root]
/bin/bash# efibootmgr -B -b 0005
/bin/bash# efibootmgr -B -b 0006
/bin/bash# efibootmgr -B -b 0007
/bin/bash# efibootmgr -B -b 0009
/bin/bash# efibootmgr -B -b 000c
/bin/bash# efibootmgr -B -b 0018
BootCurrent: 0019
Timeout: 5 seconds
BootOrder: 0019
Boot0000 Startup Menu
Boot0001 System Information
Boot0002 Bios Setup
Boot0003 3rd Party Option ROM Management
Boot0004 System Diagnostics
Boot0008 Boot Menu
Boot000B Network Boot
Boot000E* IPV6 Network - Aquantia AQtion 10Gbit Network Adapter
Boot0010* IBA GE Slot 00C8 v1550
Boot0011 USB:
Boot0012 HP Recovery
Boot0013* hp PLDS DVDRW DU8AESH
Boot0014* hp PLDS DVDRW DU8AESH
Boot0019* ubuntu
Boot001A* IPV4 Network - Aquantia AQtion 10Gbit Network Adapter

[root@tsm2: /root]
/bin/bash# efibootmgr -v
BootCurrent: 0019
Timeout: 5 seconds
BootOrder: 0019
Boot0000 Startup Menu FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)....ISPH
Boot0001 System Information FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0002 Bios Setup FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0003 3rd Party Option ROM Management FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0004 System Diagnostics FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0008 Boot Menu FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot000B Network Boot FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot000E* IPV6 Network - Aquantia AQtion 10Gbit Network Adapter PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(88c9b3bfa1e9,0)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.....ISPH
Boot0010* IBA GE Slot 00C8 v1550 BBS(Network,Network1,0x0)/PciRoot(0x0)/Pci(0x19,0x0)......ISPH
Boot0011 USB: PciRoot(0x0)/Pci(0x1d,0x0)N.....YM....R,Y.....ISPH
Boot0012 HP Recovery FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0013* hp PLDS DVDRW DU8AESH PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0,0,0)N.....YM....R,Y.....ISPH
Boot0014* hp PLDS DVDRW DU8AESH BBS(CDROM,CDROM1,0x400)/PciRoot(0x0)/Pci(0x1f,0x2)......ISPH
Boot0019* ubuntu HD(1,GPT,e41eb9e0-6606-411a-bb83-bed7577f29b3,0x800,0x8e800)/File(\EFI\grub\shimx64.efi)....ISPH
Boot001A* IPV4 Network - Aquantia AQtion 10Gbit Network Adapter PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(88c9b3bfa1e9,0)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.....ISPH

[root@tsm2: /root]
/bin/bash# efibootmgr -c -d /dev/nvme0n1 -L nvme0_grub -l '\EFI\grub\shimx64.efi'
/bin/bash# efibootmgr -c -d /dev/nvme1n1 -L nvme1_grub -l '\EFI\grub\shimx64.efi'
/bin/bash# efibootmgr -c -d /dev/nvme2n1 -L nvme2_grub -l '\EFI\grub\shimx64.efi'
/bin/bash# efibootmgr -c -d /dev/nvme3n1 -L nvme3_grub -l '\EFI\grub\shimx64.efi'
/bin/bash# efibootmgr -o 0005,0006,0007,0009,00019
BootCurrent: 0019
Timeout: 5 seconds
BootOrder: 0005,0006,0007,0009,0019
Boot0000 Startup Menu
Boot0001 System Information
Boot0002 Bios Setup
Boot0003 3rd Party Option ROM Management
Boot0004 System Diagnostics
Boot0005* nvme0_grub
Boot0006* nvme1_grub
Boot0007* nvme2_grub
Boot0008 Boot Menu
Boot0009* nvme3_grub
Boot000B Network Boot
Boot000E* IPV6 Network - Aquantia AQtion 10Gbit Network Adapter
Boot0010* IBA GE Slot 00C8 v1550
Boot0011 USB:
Boot0012 HP Recovery
Boot0013* hp PLDS DVDRW DU8AESH
Boot0014* hp PLDS DVDRW DU8AESH
Boot0019* ubuntu
Boot001A* IPV4 Network - Aquantia AQtion 10Gbit Network Adapter

[root@tsm2: /root]
/bin/bash# efibootmgr -v
BootCurrent: 0019
Timeout: 5 seconds
BootOrder: 0005,0006,0007,0009,0019
Boot0000 Startup Menu FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)....ISPH
Boot0001 System Information FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0002 Bios Setup FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0003 3rd Party Option ROM Management FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0004 System Diagnostics FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0005* nvme0_grub HD(1,GPT,e41eb9e0-6606-411a-bb83-bed7577f29b3,0x800,0x8e800)/File(\EFI\grub\shimx64.efi)
Boot0006* nvme1_grub HD(1,GPT,aa23256a-95c6-4148-b56c-c8861fc7966a,0x800,0x8e800)/File(\EFI\grub\shimx64.efi)
Boot0007* nvme2_grub HD(1,GPT,1f7f7f5b-2a89-4d87-a617-6ccaf15078dd,0x800,0x8e800)/File(\EFI\grub\shimx64.efi)
Boot0008 Boot Menu FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0009* nvme3_grub HD(1,GPT,cb8bc8b4-affc-4765-97c2-72af0c615d44,0x800,0x8e800)/File(\EFI\grub\shimx64.efi)
Boot000B Network Boot FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot000E* IPV6 Network - Aquantia AQtion 10Gbit Network Adapter PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(88c9b3bfa1e9,0)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.....ISPH
Boot0010* IBA GE Slot 00C8 v1550 BBS(Network,Network1,0x0)/PciRoot(0x0)/Pci(0x19,0x0)......ISPH
Boot0011 USB: PciRoot(0x0)/Pci(0x1d,0x0)N.....YM....R,Y.....ISPH
Boot0012 HP Recovery FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0013* hp PLDS DVDRW DU8AESH PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0,0,0)N.....YM....R,Y.....ISPH
Boot0014* hp PLDS DVDRW DU8AESH BBS(CDROM,CDROM1,0x400)/PciRoot(0x0)/Pci(0x1f,0x2)......ISPH
Boot0019* ubuntu HD(1,GPT,e41eb9e0-6606-411a-bb83-bed7577f29b3,0x800,0x8e800)/File(\EFI\grub\shimx64.efi)....ISPH
Boot001A* IPV4 Network - Aquantia AQtion 10Gbit Network Adapter PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/MAC(88c9b3bfa1e9,0)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.....ISPH

 

[root@tsm2: /root]
/bin/bash# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal

[root@tsm2: /root]
/bin/bash# uname -a
Linux tsm2 5.4.0-97-generic #110-Ubuntu SMP Thu Jan 13 18:22:13 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux


lilo slow boot map

Happily switched my boot back to SATA mirrors, and was able to reenable LILO COMPACT mode.

This means instead of 13,000 reads per boot file, it’s more like 50. Not only is booting a few seconds faster, more importantly updating the liko boot map after installing a new kernel takes 10 seconds instead of 5 minutes.


HOWTO: AIX support for R/W filesystem on USBMS

JFS2 Unsupported
Putting JFS2 on non-LVM block devices has been working for a long time. I​ wrote up how to put JFS2 on a ramdisk back at AIX 4.3.3. I lost the techdoc from back then, but IBM has a newer re-write dated 2008 here: http://www-01.ibm.com/support/docview.wss?uid=isg3T1010722

JFS2 requires the underlying system to tell it if something goes away, or for it to stay there as long as the filesystem is mounted. LVM does this for disk, and the ramdisk drivers do this as well (mostly because if the ramdisk fails, likely the system has failed). The key there is that with JFS2, the ramdisk pages are pinned.

I wrote up including performance on USB 1 and USB2 ports in January of 2010 HOWTO: JFS2 on USB device on AIX 5.3.11.1. Everything seems fine, and dandy, even mount on boot, except it’s not supported by AIX Development.

JFS2 Problems
The problem for USB Mass Storage Devices is that the device can just go away unexpectedly. If a disk goes into deep sleep, or resets because of a loose connection, the JFS drivers do not get notified. So, they take writes, and JFS2 saves them up until it’s time to flush. It goes to flush, and the I/O channel is gone. Sometimes, this is just loss of everything in cache. Sometimes, config methods hang until reboot. Other times, the system crashes.

​Because of that, we still cannot put LVM on a USB Mass Storage Device. This would take changes to notification of device availability, perhaps changes to the sync daemon, etc. Who knows, but there’s not been enough push from paying customers to make it a priority for AIX Development. Until that happens, don’t expect formal support for JFS2 on these devices.

UDF is the solution
AIX development supports read/write and even booting from USB Mass Storage Devices, but only with UDFS. The purpose is for writing a mksysb (system boot) image, or tar/cpio files, etc, and exists because of the RDX USB Internal Dock sold with some systems.
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_61/com.ibm.aix.files/usbms_fileref.htm

​Boot support is provided as well: REF: ​http://www-01.ibm.com/support/docview.wss?uid=isg1IZ66737

More info on RDX USB Internal Dock. https://www.ibm.com/support/knowledgecenter/POWER7/p7hdt/fc1103.htm

However, RDX is just a hot-swap USB to SATA drive bay. Any current USB drive (USB3 is preferred due to performance), should work fine.

HOWTO: Create, Read, and Write UDF on AIX

Create a bootable filesystem
mksysb -eXpi /dev/usbms0

Create an empty filesystem
udfcreate -d /dev/usbms0

Create a UDF 2.01 filesystem
udfcreate -f3 -d/dev/usbms0

NOTE: UDF 2.01 supports a real-time filesystem. It’s still UDF, so don’t try to put a database, or a million files on there.

Access read/write
mount -vudfs /dev/usbms0 /USBDRIVE

NOTE: The mksysb is a SPOT, plus a mksysb image, so adding files to the UDF will not make the restore huge.

USB Adapters on AIX
Add-in USB3 XHCI adapter for POWER8 is:
* CCIN 58F9 – PCIE2 4-port USB3 adapter
* FC EC45 and FRU 00E2932 for Low Profile
* FC EC46 and FRU 00E2934 for full height.
* driver is 4c1041821410b204 internal or 4c10418214109e04 PCIe

Add-in USB2 EHCI adapter for POWER7 is:
* CCIN 57D1 – PCI-E 4-port USB2 adapter
* driver is 33103500 integrated or 3310e000 PCIe
* FC 2728 or FRU 46K7394

Add-in USB2 EHCI adapter for POWER6/POWER5 is:
* CCIN 28EF – PCI 2-port USB2 adapter
* FC 2738 or FRU 80P2994
* Belkin F5U219 – exact same card without the sticker.
* driver is 99172604 internal or 99172704 PCI

Original USB1 OHCI /UHCI adapter for POWER5 and earlier was
* driver 22106474 on blades or c1110358 PCI
* This device is not really available anymore.