High Level VIO/Client build

This is off the cuff, and is not a technical walkthrough. This is enough for you to teach yourself assuming you have a system to hack on.

IBM’s POWER8 docs are missing almost everything. I don’t understand how they can call them docs at all. They want you to use some really picky tools that are cumbersome and not flexible in all the right ways.

The IBM POWER7 docs are close, but are missing the SR-IOV info. Your best bet is to skim though this, and stop when you find the bits you want (concepts, config):

The high level jist of building a VIO environment is as follows:

  • Configure to HMC
  • Clear managed system profile data
  • Build a couple VIO servers:
    • 6GB RAM, 3 virtual procs, 0.3 virtual CPUs, 255 CPU weight
    • At least one storage and one network adapter
    • You can use SR-IOV to share an ethernet adapter from firmware if needed
    • One virtual ethernet trunk for each separate physical network.  Assign VLANs here
    • One virtual ethernet non-trunk for each VLAN you want an IP address on (ideal, but you can also hang IPs and VLANs directly from AIX)
    • One virtual SCSI server adapter for each client LPAR that will need virtual CDROM, Virtual Tape, or legacy Virtual SCSI disk (higher CPU load).
    • One virtual fibre adapter for each client port (usually two per client on each VIO server, but can be anywhere from 18)
  • Upload the VIO base media into the HMC media repository
  • Install the VIO server from the HMC
  • SSH into the HMC, and use vtmenu to rebuild the VIO networking
    • Remove all en, et, ent, hba devices, then cfgmgr
    • mkvdev -lnagg for any etherchannel bonded pairs needed for the Shared Ethernet Adapter(s)
    • mkvdev -sea  to build any shared ethernet adapters (ethernet bridge from virtual switch to physical port)
    • mkvdev -lnagg for any etherchannel bonded pairs needed for local IP communication
    • mkvdev -vlan for any additional VLANs hanging directly off an SEA rather than through a virtual ethernet client adapter
    • mktcpip to configure your primary interface, gateway, etc
    • Add any extra IP addresses.
  • Build your Client LPARs
    • Memory, CPU, RAM as desired
    • Virtual ethernet just picks the switch and VLAN that you need.  If this does not exist on any VIO trunk adapters, then you need to fix that.
    • Virtual SCSI client adapter
      • this needs the VIO server partition ID, and the VIO server slot number added to it for the firmware connection.
      • The VIO server virtual SCSI adapter needs the same mapping back to the client LPAR id and slot.
      • There may be some GUI improvements to add this all for you, but it’s been decades of garbage for so long that I just do it all manually.
    • Virtual Fibre adapter – This maps back and forth to the VIO server virtual fibre similar to how VSCSI did.
  • SSH into the VIO server
    • make virtual optical devices attached to the “vhost” (virtual SCSI” if needed
    • Use vfcmap to map the “vfchost” adapters to real “fcs” ports.  This requires them to be NPIV capable (8gbit or newer), logged into an NPIV capable switch (lsnports).
  • Zone any LUNs
    • lsnportlogin can give you the WWNs for the clients, or you can get it from the client profile data manually
    • You can use OpenFirmware’s “ioinfo” to light up a port to force it to log in to the switch.
    • If the LPAR is down, you can use “chnportlogin” from the HMC to log in all ports for that client.
    • You can also zone directly to the VIO server, and “mkvdev” to map them as vscsi disks (higher CPU load on VIO server, and kind of a pain in the rump).
    • Note that LPM requires any VSCSI LUNs to be mapped to all VIO servers in advance.
    • Note that LPM requires any NPIV LUNs to be mapped to the secondary WWNs in advance
  • SSH into the VIO server
    • Make sure lsmap and lsmap -npiv show whatever mapping is required
    • Make sure loadopt has mounted any ISO images as virtual CDROMs if needed
    • You can also just mask an alt_disk_install LUN from a source host.
    • You can also use NIM to do a network install
  • Activate the LPAR profile.
    • If you did not open a vterm from SSH into the HMC, then you can do it from the activate GUI.
    • You can use SMS to pick your boot device
    • Install or boot as desired
    • Reconfigure your network as normal
      • smitty tcpip or “chdev -l en0” and “chdev -l inet0” with appropriate flags
      • Tune everything as desired.
      • If it was a Linux install, then that has its own config options.

SR-IOV can be used instead of Shared Ethernet above. 

It allows you to share a single PCI NIC or single ethernet port between LPARs.  It uses less CPU on the VIO server, and has lower latency for your LPARs.  It’s sort of the Next Generation of network virtualization, though there are some restrictions in its use.  It’s best to review all of the info, and decide up front, but is worth your time to do so.  If you want to use an SEA on SR-IOV, you still only have one VIO server per port, but you can have different ports on different VIO servers.  When sharing among all clients and VIO server without SEA, understand that the percentage capacity is a minimum guaranteed, not a cap.  Leave it low unless you have some critical workload that needs to crowd out anyone else. Some of the best URLs today when I look up “SR-IOV vNIC vio howto” are as follows:

CLI and Automation

If you want to build a whole bunch of VIO clients and servers at once, it may be worth the effort to do it from the HMC CLI.  It gets really complicated, but once you have it set up, you can adjust and rebuild things quickly.  This also lets you manually specify WWNs for your LPARs in case there are collisions, or if you are rebuilding and need to keep the same numbers.

The VIO server can be installed with alt_disk_copy, or from NIM, or from physical CD, or from the HMC.  The CLI version is called “installios” and you MUST specify the MAC address of the boot adapter for it to work properly. Without CLI options, installios will prompt you for all of the info.


AIX and PowerHA levels

Research shows these dates for AIX:
It’s generally 26 weeks, plus or minus, from the initial YYWW date. Once a TLSP APARs releases, the YYWW code is be updated.

  • 7300-01-01-2246 2022-12-02 (Next 2023Q2)
  • 7200-05-05-2246 2022-12-02 (Next 2023Q2)
  • 7100-05-10-2220 2022-09-09 (Next 2023Q1)
  • 6100-09-12-1846 2018-11-16 (EoL CSP)

My AIX selection process would be:

  • AIX from 2022 week 46 is what I have in my repo.  Another TL should be coming out 2023 Q1.  None of my customers run this, but you want this for POWER10.  NIM should be latest of all versions as well.
  • AIX from 2022 week 46 is what I have in my repo.  Another TL should be coming out 2023 Q1.  You probably want this for POWER7 and up.
  • AIX from 2022 week 20 is what I have in my repo.  I think the CSP is 2023Q1.  Supports AIX 5.2 and 5.3 WPARs.  Not much other reason to use this now other than some specific apps that are OK with OpenSSL, OpenSSH, and Java updates, but not kernel updates.
  • AIX is where I stopped tracking.  No real need for AIX 6 anywhere.  Either you’re stuck on 5.3, or you came up to 7.1 (or ideally 7.2). was needed for application compatibility on POWER9.
  • For anything POWER6 or older should really upgrade to p710 to p740 or s81x/s82x as replacements (cost).  POWER8 is EoS 2024-10-31.  POWER7 is EoS 2019.
  • AIX + U866665 on POWER8 is end-stage.  AIX 5.3 was EoS in 2012, but some people still run it now.  Power8 is EoS 2024-10-31.  Power7 was 2019.
  • AIX 5.3 PTF U866665.bff (bos.mp64. enables POWER8. AIX must be Must be patched before moving to p8. p8 must be 840 firmware or later. VIO must be or later.  Migration is by LPM, NIM, or mksysb.  equires active extended support agreement AIX p8 systems on file to download.
  • AIX older – You should not be running anything older.  AIX 5.1 was all CHRP, and AIX 4.x was all PCI.  AIX 3.x was MicroChannel.  AIX 2.1 had some PS/2 systems.  Outside of a museum, on an isolated network (or no network because CYLONS!), just have this recycled.

My PowerHA (HA/CMP) selection process would be:

  • 7.2.7 Base is what I last grabbed.  OK for AIX 7.1.5, 7.2.5, 7.3.0 and later patch levels.
  • 7.1.3 SP09 was the end-stage for this.  OK for AIX,,, and later patch levels.  No AIX 5, and no AIX 7.3.
  • 6.1 SP15 was end-stage for this, and supported AIX 5.3.9,, 7.1.0, and later patch levels.  No AIX 5.1, 7.2, or 7.3.

Code sources:

UPDATE 2023-03-07:

  • Refreshed all of the info above to current.  If you’re on AIX 5.2, HA/CMP 5.x, or VIO 1.x, that’s really disappointing.
  • VIO should be  Always go current whenever possible.  If not, is your target.  If you have any VIO 1.x, upgrade.  Period.
  • System firmware, adapter microcode, disk microcode, tape microcode, and library firmware should all be latest available.
  • Storage array firmware should be latest LTS patch level, excluding any .0 versions.  I still don’t trust Data Reduction Pools.
  • ADSM/ITSM/TSM/Spectrum Protect/Storage Protect – These should typically be the final version supporting your OS/App combo.  I’m sweet on 8.1.17, though there are still major issues with deduplication pools when your database is over 3TB.  Containers and extents cannot be purged.  I have not seen SP 9.1, but I assume it will be extremely similar to SP 8.1.18 other than some minor rebadging.  Not sure.  I got dropped from the beta program because I didn’t have cycles to test their new code, and they didn’t have any interest in steering features.  They pick what they want to pick, and you’ll like it. 
  • IBM is phasing out Spectrum Protect Plus (Catalogic DPX), but might still be keeping Catalogic ECX (Copy Data Management).  In Storage Defender, IBM has picked up Cohesity DataProtect because of the cloud / DRaaS bits.  These all integrate with DS8000/Flash Systems for data immutability / vaulting / ransomware protection, and they want you to buy Rapid7 for the AI/Logic behind it.  I know regular ISP’s operations center anomaly detection us unusable due to its lack of adaptability/logic.  It just says everything is alerted every week when you run weekly fulls, etc.

I don’t really track IBMi OS (OS400), zOS, zVM, etc.  Storage should be rebranding this year though, but still no NFS/CIFS hardware.  At best, IBM sells a GPFS cluster with Ceph and with some StorWize FS7200s.