Replacing TSM / ISP server

I ran into an issue where a primary TSM 7.1.8 server was broken, and it was easier to just move all of the clients over to the secondary 8.1.5 server. These used the new TLS encryption, and I kept running into issues.

    —————

First, I physically shut down the old server, and updated the new server to use the IP as an alias by editing /etc/network/interfaces. (Ubuntu 16 LTS)

    —————

Various errors included:
ANR8599W The connection with host address:host port failed due to an untrusted server certificate. An attempt to reconnect and establish certificate trust might follow.

ANR2284S The server master encryption key has changed. Passwords protected with the previous master encryption key are not available.

I had to make the server trust itself again with:[LINK]
dsmcert -add -server TSM -file /home/tsminst1/cert256.arm

    —————

This error:
ANR0456W Session rejected for server DT – the server name at 192.168.1.99, 1500 does not match.

I removed the server:
REMOVE SERVER DT

    —————

These errors:
ANR1651E Server information for DT is not available.
ANR4377E Session failure, target server DT is not defined on the source server.
ANR3151E Configuration refresh failed with configuration manager DT.

I disabled replication config
REMOVE REPLNODE *
Q REPLSERVER
REMOVE REPLSERVER {GUID}
DEL SUBSCRIPTION DEFAULT_PROFILE

    —————

These errors:
ANS1695E The certificate is not valid.
ANS1592E Failed to initialize SSL protocol.
ANS8023E Unable to establish session with server.
ANR3335W Unable to distribute certificate to for session .
ANR8599W The connection with 192.168.1.2:40250 failed due to an untrusted server certificate. An attempt to reconnect and establish certificate trust might follow.

From the clients, I had to remove /opt/tivoli/tsm/client/ba/bin/dsmcert.* and C:\Program Files\Tivoli\TSM\baclient\dsmcert.* on the clients per http://andrewjtobiason.com/index.php/2018/08/16/resolving-ssl-errors/

However, I also had to fix dsmcert / dsmcert.exe which I clobbered in the process.

    —————

And on the server, allow keys to be swapped again:[LINK]
UPD NODE * SESSIONSECURITY=TRANSITIONAL
UPD ADMIN * SESSIONSECURITY=TRANSITIONAL

Swap keys on client:
dsmadmc

    —————

NOTE: I also removed /etc/adsm/* during troubleshooting, but that was not needed. That just lead to me having to re-enter passwords again. Simply deleting the cert database corrected the problem on other clients.

    —————

I tried to order this in dependency order. I was sort of all over the place when I did it, and might have missed something. I just could not exactly get all of the right info from any one documentation source.


AIXPCM vs SDDPCM

AIX geeks, converting from SDDPCM to AIXPCM, when you uninstall the drivers, you also need to uninstall host attachment. On 2% of our migrations, mksysb/alt clones would fail to find the boot disk (554).

devices.fcp.disk.ibm.mpio.rte devices.sddpcm.*

Note that from SVC 7.6.1, AIX 6.1.8, AIX 7.1.3, and AIX 7.2, you MAY switch to AIXPCM. On POWER9 and later, you MUST switch to AIXPCM.

The rm script is from storage development, but the manage_disk_drivers command is from AIX dev. Either is okay, but tge AIX one does not require making a PMR.

Best reference:

https://www.ibm.com/developerworks/community/blogs/cgaix/entry/One_Path_Control_Module_to_Rule_Them_All


IBM PCIe3 Ethernet Decoder

Some modern Ethernet adapter decoder wheel information.

PCIe2 4-Port (10GbE SFP+ & 1GbE RJ45) Adapter
PCIe2 4-Port Adapter (10GbE SFP+) (e4148a1614109304)
PCIe2 4-Port Adapter (1GbE RJ45) (e4148a1614109404)
FRU Number.................. 00E2715
Customer Card ID Number..... 2CC3
Part Number................. 00E2719
Feature Code/Marketing ID... EN0S or EN0T
Make/Model: Broadcom NetXtreme II 10Gb
Code Name: Shiner

PCIe3 10GbE RoCE Converged Host Bus Adapter (b31507101410eb04)
RoCE Converged Network Adapter
2-Port 10GbE RoCE SR PCIe 3.0
Part Number................. 00RX871
FRU Number.................. 00RX875
Feature Code/Marketing ID... EC2M / EC2N / EC37 / EC38 / EL3X / EL40
Customer Card ID Number..... 57BE
Make/Model: Mellanox ConnectX-3 Pro / Gen 3.0 x8 10GbE
Code Name: Core

PCIe3 4-Port 10GbE SR Adapter (df1020e21410e304)
PCIe3 4-port 10GbE SR Adapter, NIC PF
Part Number................. 00ND462 or 00ND464 or 00ND468 or 00RX863
Feature Code/Marketing ID... EN15 or EN16
Customer Card ID Number..... 2CE3
Make/Model: Emulex OneConnect NIC (Lancer) (rev 30)

I believe the 4-port 1gbit PCI cards are Intel.

Emulex, Brocade, and Broadcom are all owned by the same company as of 2016. Currently known as “Broadcom, Inc.”, the parent was formerly Avago, formerly Agilent, and formerly the semiconductor division of HP.


On misunderstanding

  • Everyone joins echo chambers full of people who share the same delusions.
  • For most people, a shared delusion is fact.

  • Asking questions instead of accepting on trust/faith is threatening.

  • Not sharing the same context can be gravely offensive.

  • Rational thought is a facade tinted by emotions.

  • Traumatic experiences prevent rational thought.

  • Invalid rationalizations do not mean the feelings are not real.

  • Feeling strongly does not make thoughts real.

  • The inner workings of problem solving are scary.

  • Assumptions are not guaranteed to be correct.

  • Experts are not guaranteed to be accurate, but they do have a lot of experience.

  • Being upset does not mean you have been wronged.

  • Being calm does not mean you have not committed offense.

  • You can offend without doing anything wrong.

  • Angry interactions are very likely to offend and be wrong actions.

  • Frustration looks like anger and hostility.

  • Your experiences are guaranteed to affect your views, judgements, and decisions.

  • Your prejudices are affecting your judgement.

The gravest offense is not sharing someone else’s delusion. Understanding a delusion is hard work, but can give compassion.

For me, that means sharing information, identifying facts vs opinions, asking questions to see how the pieces fit together, and avoiding some types of inflexibility and pre-judgement.

That’s threatening and incompatible with some people. It triggers a lot of emotions for a lot of people, including me. We’re all just human.

This is not about “a person”, nor an event, or any single thing. This was just brainstorming about “misunderstanding”, so at best, it’s about me and a few thousand other people, groups, etc.

So, consider what emotions this brings out. Spend some time thinking about why.


AIX types of ethernet interfaces

AIX shows a lot of different info in different places.  This is because AIX predates the time when everyone had RJ45 ethernet ports.

HBA represents a high-function PCI adapter that contains multiple protocols, and which can sometimes be configured to provide ENT devices.  Primary candidates are “Integrated Virtual Ethernet” on POWER5 and POWER6 servers, as well as ROCE adapters, which are “RDMA Over Converged Ethernet”, with RDMA being “Remote Direct Memory Addressing” or “Access”.  Basically, Infiniband adapters which can use ethernet at the link layer.

ENT represents the “physical port”, though that is not always the case.  I’ll explain more later.  There is one one of these for every Ethernet port visible to the operating system.

EN represents the “ETHERNET II” protocol device for IP communication.  This is the standard today, also known as “DIX Etehrnet”, named after DEC, Intel, Xerox.  This is where you will normally put your IP address.  There is one of these for every ENT device.

ET represents an IEEE 802.3 protocol device.  This would have been used in the days of Novel Networking, or with SNA protocol.  Almost no-one uses this anymore, but I’m sure there’s an AIX 3.2.5U2 microchannel server running with this somewhere in the bottom of an old government facility, with coaxial cables and barrell terminators.  Really, I don’t know why this still is needed on anything produced in the last 20 years.  There is one of these for every ENT device.

INET is for config options that affect the entire TCP/IP stack, such as persistent routes, the hostname, and whether you are bypassing ODM for config of your network (rare).  There is only one of these per system, and it is always inet0 unless someone gets cheeky.

There are other ways to get IP devices, such as IP over Fibre Channel, IP over Infiniband, IP over ATM, over FDDI, over serial or parallel, etc.  These are less common, so I’m not going into them here.

Generally, you may have a stack like this:

ent0    physical ethernet port
ent1    physical ethernet port
ent2    Etherchannel (Static, or LACP bond created out of both of the above)
ent3    Virtual Ethernet (Connects to a virtual, firmware-only switch)
ent4    Shared Ethernet (VIO server only, a software bridge between a virtual physical)
ent5    VLAN (an additional VLAN port configured off of any of the above)
en0     IP interface – unused because we give ENT0 as a backing device to ENT3
en1     IP interface – also unused for the same reason
en2     IP Interface – also unused, because this is the backing device for the SEA
en3     IP interface – Also unused because this is a backing device for the SEA.
en4     IP interface hanging off of ENT4 – this can be skipped, and a virtual ethernet used
en5     IP interface hanging off of ENT5 – this can be skipped, and a virtual ethernet used

Each device has its own type of parameters.  You can use “lsattr -El $device”, “netstat -in”, and “entstat -d $device” to get details of this.  Note that entstat wants to be on the top device, not the bottom device.  Start with where the IP address is assigned, and it will show the subdevices, virtual connections, etc.


High Level VIO/Client build

This is off the cuff, and is not a technical walkthrough. This is enough for you to teach yourself assuming you have a system to hack on.

IBM’s POWER8 docs are missing almost everything. I don’t understand how they can call them docs at all. They want you to use some really picky tools that are cumbersome and not flexible in all the right ways.

The IBM POWER7 docs are close, but are missing the SR-IOV info. Your best bet is to skim though this, and stop when you find the bits you want (concepts, config):

The high level jist of building a VIO environment is as follows:

  • Configure to HMC
  • Clear managed system profile data
  • Build a couple VIO servers:
    • 6GB RAM, 3 virtual procs, 0.3 virtual CPUs, 255 CPU weight
    • At least one storage and one network adapter
    • You can use SR-IOV to share an ethernet adapter from firmware if needed
    • One virtual ethernet trunk for each separate physical network.  Assign VLANs here
    • One virtual ethernet non-trunk for each VLAN you want an IP address on (ideal, but you can also hang IPs and VLANs directly from AIX)
    • One virtual SCSI server adapter for each client LPAR that will need virtual CDROM, Virtual Tape, or legacy Virtual SCSI disk (higher CPU load).
    • One virtual fibre adapter for each client port (usually two per client on each VIO server, but can be anywhere from 18)
  • Upload the VIO base media into the HMC media repository
  • Install the VIO server from the HMC
  • SSH into the HMC, and use vtmenu to rebuild the VIO networking
    • Remove all en, et, ent, hba devices, then cfgmgr
    • mkvdev -lnagg for any etherchannel bonded pairs needed for the Shared Ethernet Adapter(s)
    • mkvdev -sea  to build any shared ethernet adapters (ethernet bridge from virtual switch to physical port)
    • mkvdev -lnagg for any etherchannel bonded pairs needed for local IP communication
    • mkvdev -vlan for any additional VLANs hanging directly off an SEA rather than through a virtual ethernet client adapter
    • mktcpip to configure your primary interface, gateway, etc
    • Add any extra IP addresses.
  • Build your Client LPARs
    • Memory, CPU, RAM as desired
    • Virtual ethernet just picks the switch and VLAN that you need.  If this does not exist on any VIO trunk adapters, then you need to fix that.
    • Virtual SCSI client adapter
      • this needs the VIO server partition ID, and the VIO server slot number added to it for the firmware connection.
      • The VIO server virtual SCSI adapter needs the same mapping back to the client LPAR id and slot.
      • There may be some GUI improvements to add this all for you, but it’s been decades of garbage for so long that I just do it all manually.
    • Virtual Fibre adapter – This maps back and forth to the VIO server virtual fibre similar to how VSCSI did.
  • SSH into the VIO server
    • make virtual optical devices attached to the “vhost” (virtual SCSI” if needed
    • Use vfcmap to map the “vfchost” adapters to real “fcs” ports.  This requires them to be NPIV capable (8gbit or newer), logged into an NPIV capable switch (lsnports).
  • Zone any LUNs
    • lsnportlogin can give you the WWNs for the clients, or you can get it from the client profile data manually
    • You can use OpenFirmware’s “ioinfo” to light up a port to force it to log in to the switch.
    • If the LPAR is down, you can use “chnportlogin” from the HMC to log in all ports for that client.
    • You can also zone directly to the VIO server, and “mkvdev” to map them as vscsi disks (higher CPU load on VIO server, and kind of a pain in the rump).
    • Note that LPM requires any VSCSI LUNs to be mapped to all VIO servers in advance.
    • Note that LPM requires any NPIV LUNs to be mapped to the secondary WWNs in advance
  • SSH into the VIO server
    • Make sure lsmap and lsmap -npiv show whatever mapping is required
    • Make sure loadopt has mounted any ISO images as virtual CDROMs if needed
    • You can also just mask an alt_disk_install LUN from a source host.
    • You can also use NIM to do a network install
  • Activate the LPAR profile.
    • If you did not open a vterm from SSH into the HMC, then you can do it from the activate GUI.
    • You can use SMS to pick your boot device
    • Install or boot as desired
    • Reconfigure your network as normal
      • smitty tcpip or “chdev -l en0” and “chdev -l inet0” with appropriate flags
      • Tune everything as desired.
      • If it was a Linux install, then that has its own config options.

SR-IOV can be used instead of Shared Ethernet above. 

It allows you to share a single PCI NIC or single ethernet port between LPARs.  It uses less CPU on the VIO server, and has lower latency for your LPARs.  It’s sort of the Next Generation of network virtualization, though there are some restrictions in its use.  It’s best to review all of the info, and decide up front, but is worth your time to do so.  If you want to use an SEA on SR-IOV, you still only have one VIO server per port, but you can have different ports on different VIO servers.  When sharing among all clients and VIO server without SEA, understand that the percentage capacity is a minimum guaranteed, not a cap.  Leave it low unless you have some critical workload that needs to crowd out anyone else. Some of the best URLs today when I look up “SR-IOV vNIC vio howto” are as follows:

CLI and Automation

If you want to build a whole bunch of VIO clients and servers at once, it may be worth the effort to do it from the HMC CLI.  It gets really complicated, but once you have it set up, you can adjust and rebuild things quickly.  This also lets you manually specify WWNs for your LPARs in case there are collisions, or if you are rebuilding and need to keep the same numbers.

The VIO server can be installed with alt_disk_copy, or from NIM, or from physical CD, or from the HMC.  The CLI version is called “installios” and you MUST specify the MAC address of the boot adapter for it to work properly. Without CLI options, installios will prompt you for all of the info.

 


AIX 7.2.3.1 breaks GSKit 8.0.50.89

AIX 7.2.3 breaks GSKit8, up through GP29 (8.0.50.89).

This affects TSP/Spectrum Protect, Content Manager, Tivoli Directory Server, Websphere, DB2, Informix, IBM HTTP Server, etc.

Before reboot, everything works still, which implies the change is in the kernel.

We found it on TSM, and AIX 7200-03-01-1838, and Spectrum Protect server 8.1.6.0.

Application crash and DBX follow below.

ANR7800I DSMSERV generated at 12:17:13 on Sep 11 2018.
IBM Spectrum Protect for AIX
Version 8, Release 1, Level 6.000
Licensed Materials - Property of IBM
(C) Copyright IBM Corporation 1990, 2018.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.

ANR7801I Subsystem process ID is 10944920.
ANR0900I Processing options file /home/tsminst1/dsmserv.opt.
ANR7811I Using instance directory /home/tsminst1.
Illegal instruction(coredump)

# dbx /opt/tivoli/tsm/server/bin/dsmserv core.10944896.28165312
Type 'help' for help.
[using memory image in core.10944896.28165312]
reading symbolic information ...warning: no source compiled with -g

Illegal instruction (illegal opcode) in . at 0x0 ($t1)
warning: Unable to access address 0x0 from core

(dbx) where
.() at 0x0
gsk_src_create__FPPvPv(??, ??) at 0x9000000015b6d88
__ct__8GSKMutexFv(??) at 0x9000000018d664c
__ct__20GSKPasswordEncryptorFv(??) at 0x9000000018cb248
__ct__7gsk_envFv(??) at 0x900000000aaa6b0
GskEnvironmentOpen__FPPvb(??, ??) at 0x900000000ab14c4
gsk_environment_open(??) at 0x900000000ab277c
IPRA.$CheckGSKVersion() at 0x100eecf68
tlsInit() at 0x100eecd70
main(??, ??) at 0x10000112c

(dbx) th
thread state-k wchan state-u k-tid mode held scope function

$t1 run running 41877977 k no sys
$t2 run blocked 21234465 u no sys _cond_wait_global
$t3 run running 24380103 u no sys waitpid


AIX ramdisks

Long ago (think 1999ish), I wrote a techdoc on how to put JFS on a ramdisk on AIX. We called them FAXES, because we would fax them to people, and this was before FAQ was a common acronym. At some point, I put it into the TechDoc system when that came out, because there was a push to use the system.

I lost the original text, but the techdoc lived on. It was rewritten after I left big blue. You can see their better version here:
http://www-01.ibm.com/support/docview.wss?uid=isg3T1010722

I don’t want to copy their doc, because they can be testy about such things. Heck, they can be testy when they plagiarize my docs. The key reference is syntax, which I’ll summarize here. You can also just look up the manpages on mkramdisk, mkfs,

Make a pinned-memory ramdisk: mkramdisk $bytes
The default uses pinend RAM, which is required for JFS or JFS2.

Make an un-pinned ramdisk: mkramdisk -u $bytes
This is okay for raw devices, maybe UDFS, but not for JFS. There are latency/access requirements on JFS, but at least mkfs knows to throw an error here if you try to skip it.

When you run mkfs on /dev/ramdisk0 as JFS, it’s normal, except you mount -o nointegrity.

When you run mkfs on /dev/ramdisk0 as JFS2, use -o log=INLINE on the format, and the mount.

You can, of course, format UDF as well: udfcreate -f3 -d/dev/ramdisk0 ; mount -vudfs /dev/ramdisk0 /RAMDISK

You could probably run a mksysb to the ramdisk. I don’t know if it would be raw, or if it would be UDFS. That might be useful for high speed testing, but of course, the ramdisk evaporates on reboot. You could dd the ramdisk out to some other media.