Tag Archives: isp

TSM/ISP Recovering from “SKIP UPGRADING THIS INSTANCE”

Posted by xaminmo

Recovering from “SKIP UPGRADING THIS INSTANCE”

REFS:
https://www.ibm.com/support/pages/manually-upgrading-ibm-spectrum-protect-server-instances
https://www.ibm.com/support/pages/anr0187e-failure-during-server-startup
http://issen007.blogspot.com/2017/05/manual-upgrade-ibm-spectrum-protect-71x.html

###################################################
### Stop the instance completely
su - tsminst1

### This may not work if your environment or links are bad.
db2 list db directory

### If db2sysc is still running
ps | grep db2 
db2stop
db2stop force
db2 terminate ### Kill off db2bp fragments

### kill everything else other
ps | grep db2

### Remove IPC
ipcrm -a


###################################################
### Clean up remainders
su - root
/opt/tivoli/tsm/db2/instance/db2ilist
/opt/tivoli/tsm/db2/instance/db2idrop tsminst1

### Verify nothing left
/opt/tivoli/tsm/db2/instance/db2ilist


###################################################
### Redefine the instance
su - root
#/opt/tivoli/tsm/db2/instance/db2icrt -a server -u tsminst1 tsminst1
/opt/tivoli/tsm/db2/instance/db2icrt -u tsminst1 tsminst1

DBI1446I The db2icrt command is running.
DB2 installation is being initialized.
Total number of tasks to be performed: 4
Total estimated time for all tasks to be performed: 309 second(s)

Task #1 start
Description: Setting default global profile registry variables
Estimated time 1 second(s)
Task #1 end

Task #2 start
Description: Initializing instance list
Estimated time 5 second(s)
Task #2 end

Task #3 start
Description: Configuring DB2 instances
Estimated time 300 second(s)
Task #3 end

Task #4 start
Description: Updating global profile registry
Estimated time 3 second(s)
Task #4 end

The execution completed successfully.
For more information see the DB2 installation log at "/tmp/db2icrt.log.21176".
DBI1070I Program db2icrt completed successfully.


###################################################
### Set up Db2 environment variables
# NOTE: userprofile and db2profile get reset after db2icrt
su - tsminst1
/opt/tivoli/tsm/db2/adm/db2set -i tsminst1 "DB2_SKIPINSERTED=ON"
/opt/tivoli/tsm/db2/adm/db2set -i tsminst1 "DB2_KEEPTABLELOCK=ON"
/opt/tivoli/tsm/db2/adm/db2set -i tsminst1 "DB2_EVALUNCOMMITTED=ON"
/opt/tivoli/tsm/db2/adm/db2set -i tsminst1 "DB2_SKIPDELETED=ON"
/opt/tivoli/tsm/db2/adm/db2set -i tsminst1 "DB2CODEPAGE=819"
/opt/tivoli/tsm/db2/adm/db2set -i tsminst1 "DB2_PARALLEL_IO=*"

cat <<EOF >>${HOME}/sqllib/userprofile
export LD_LIBRARY_PATH=${HOME}/sqllib/lib64/gskit:${HOME}/sqllib/lib32:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/tivoli/tsm/server/bin/dbbkapi:/opt/ibm/lib:/opt/ibm/lib64:/usr/lib64:${HOME}/sqllib/lib64
export PATH=$PATH:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/tivoli/tsm/server/bin64
export PATH=$PATH:/opt/tivoli/tsm/server/bin:/usr/tivoli/tsm/server/bin64:/usr/tivoli/tsm/server/bin
export PATH=$PATH:/opt/tivoli/tsm/client/ba/bin64:/opt/tivoli/tsm/client/ba/bin:/usr/tivoli/tsm/client/ba/bin64
export PATH=$PATH:/usr/tivoli/tsm/client/ba/bin:/usr/tivoli/tsm/client/api/bin64:/usr/tivoli/tsm/client/api/bin
export PATH=$PATH:/opt/tivoli/tsm/client/api/bin64:/opt/tivoli/tsm/client/api/bin:/opt/tivoli/tsm/db2/bin
export PATH=$PATH:${HOME}/sqllib/bin:${HOME}/sqllib/adm:${HOME}/sqllib/misc

DSMI_CONFIG=${HOME}/tsmdbmgr.opt
DSMI_DIR=/opt/tivoli/tsm/server/bin/dbbkapi
DSMI_LOG=${HOME}
export DSMI_CONFIG DSMI_DIR DSMI_LOG 
EOF

cat <<EOF >>${HOME}/.profile
. ${HOME}/sqllib/db2profile
. ${HOME}/sqllib/userprofile

alias ll='ls -laF --color=auto'
set -o vi
EOF

. ./.profile


###################################################
### Catalog the DB to make sure it is okay.
db2start

### Find the TSMDB1 instances).
DBDIR=$(find /home /sp /tsm -name sqldbdir -exec strings {} \; 2>/dev/null | grep inst | cut -c 2-99 | sort | uniq)
echo $DBDIR

### Register the instance(s).
for i in $DBDIR ; do db2 catalog db TSMDB1 on $i ; done
# SQL6028N Catalog database failed because database "tsminst1" was not found in the local database directory.

### List the instances.
db2 list db directory


###################################################
### Upgrade the DB2 system catalog tables
#db2 upgrade db tsminst1
#SQL1013N The database alias name or database name "TSMINST1" could not be found. SQLSTATE=42705
db2 upgrade db TSMDB1
SQL1103W The UPGRADE DATABASE command was completed successful.

### Stop DB2 to make sure it flushes everything.
db2stop
SQAL1064N DB2STOP processing was successful.


###################################################
### Upgrade the TSM database schema
/opt/tivoli/tsm/server/bin/dsmserv upgradedb
ANR7800I DSMSERV generated at 18:03:03 on Nov 19 2021.

IBM Spectrum Protect for AIX
Version 8, Release 1, Level 13.000

Licensed Materials - Property of IBM

(C) Copyright IBM Corporation 1990, 2021.
All rights reserves.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.

ANR7801I Subsystem process ID is 64684398.
ANR7811I Using instance directory /tsm/tsminst1/
ANR3339I Default Label in key database is TSM Server SelfSigned SHA Key.
ANR4726I The ICC support module has been loaded.
ANR0990I Server restart-recovery in progress.

###################################################
### Make sure the server accepts workload

# start the server normally (rc script, systemd, or inittab line run from an at-job).

### Run these from dsmadmc
REG LIC FILE=*ee.lic
enable ses all

 


Spectrum Protect (TSM) Operations Center on Ubuntu LTS

Posted by xaminmo

Per IBM, the Spectrum Protect server is supported on Ubuntu LTS 14, 16, 18, and 20 (aka 2014.04, 2016.04, etc.) 

https://www.ibm.com/support/pages/overview-ibm-spectrum-protect-supported-operating-systems

However, Operations Center (web GUI) is not supported on Ubuntu, only RHEL and SLES.

https://www.ibm.com/support/pages/ibm-spectrum-protect-operations-center-software-and-hardware-requirements

./install.sh -c
Validating package prerequisites...
=====> IBM Installation Manager> Update> Prerequisites
Validation results:
* [ERROR] IBM Spectrum Protect Operations Center 8.1.12000.20210326_0723 contains validation errors.
1. ERROR: The operating system on which you are installing the product is not supported. For more information, see http://www.ibm.com/support/docview.wss?uid=swg21243309.

Enter the number of the error or warning message above to view more details.

To skipp the OS and platform checks, and convert the ERROR into WARNING:

./install.sh -c -vmargs "-DBYPASS_TSM_REQ_CHECKS=true"
Validation results:
* [WARNING] IBM Spectrum Protect Operations Center 8.1.12000.20210326_0723 contains validation warning.
1. WARNING: The operating system on which you are installing the product is not supported. For more information, see http://www.ibm.com/support/docview.wss?uid=swg21243309.

Enter the number of the error or warning message above to view more details.

I recommend ONLY install/update Operations Center with this, and then exit and go back in normally to make sure the other filesets validate okay.


Replacing TSM / ISP server

Posted by xaminmo

I ran into an issue where a primary TSM 7.1.8 server was broken, and it was easier to just move all of the clients over to the secondary 8.1.5 server. These used the new TLS encryption, and I kept running into issues.

    —————

First, I physically shut down the old server, and updated the new server to use the IP as an alias by editing /etc/network/interfaces. (Ubuntu 16 LTS)

    —————

Various errors included:
ANR8599W The connection with host address:host port failed due to an untrusted server certificate. An attempt to reconnect and establish certificate trust might follow.

ANR2284S The server master encryption key has changed. Passwords protected with the previous master encryption key are not available.

I had to make the server trust itself again with:[LINK]
dsmcert -add -server TSM -file /home/tsminst1/cert256.arm

    —————

This error:
ANR0456W Session rejected for server DT – the server name at 192.168.1.99, 1500 does not match.

I removed the server:
REMOVE SERVER DT

    —————

These errors:
ANR1651E Server information for DT is not available.
ANR4377E Session failure, target server DT is not defined on the source server.
ANR3151E Configuration refresh failed with configuration manager DT.

I disabled replication config
REMOVE REPLNODE *
Q REPLSERVER
REMOVE REPLSERVER {GUID}
DEL SUBSCRIPTION DEFAULT_PROFILE

    —————

These errors:
ANS1695E The certificate is not valid.
ANS1592E Failed to initialize SSL protocol.
ANS8023E Unable to establish session with server.
ANR3335W Unable to distribute certificate to for session .
ANR8599W The connection with 192.168.1.2:40250 failed due to an untrusted server certificate. An attempt to reconnect and establish certificate trust might follow.

From the clients, I had to remove /opt/tivoli/tsm/client/ba/bin/dsmcert.* and C:\Program Files\Tivoli\TSM\baclient\dsmcert.* on the clients per http://andrewjtobiason.com/index.php/2018/08/16/resolving-ssl-errors/

However, I also had to fix dsmcert / dsmcert.exe which I clobbered in the process.

    —————

And on the server, allow keys to be swapped again:[LINK]
UPD NODE * SESSIONSECURITY=TRANSITIONAL
UPD ADMIN * SESSIONSECURITY=TRANSITIONAL

Swap keys on client:
dsmadmc

    —————

NOTE: I also removed /etc/adsm/* during troubleshooting, but that was not needed. That just lead to me having to re-enter passwords again. Simply deleting the cert database corrected the problem on other clients.

    —————

I tried to order this in dependency order. I was sort of all over the place when I did it, and might have missed something. I just could not exactly get all of the right info from any one documentation source.


AIX 7.2.3.1 breaks GSKit 8.0.50.89

Posted by xaminmo

AIX 7.2.3 breaks GSKit8, up through GP29 (8.0.50.89).

This affects TSP/Spectrum Protect, Content Manager, Tivoli Directory Server, Websphere, DB2, Informix, IBM HTTP Server, etc.

Before reboot, everything works still, which implies the change is in the kernel.

We found it on TSM, and AIX 7200-03-01-1838, and Spectrum Protect server 8.1.6.0.

Application crash and DBX follow below.

ANR7800I DSMSERV generated at 12:17:13 on Sep 11 2018.
IBM Spectrum Protect for AIX
Version 8, Release 1, Level 6.000
Licensed Materials - Property of IBM
(C) Copyright IBM Corporation 1990, 2018.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.

ANR7801I Subsystem process ID is 10944920.
ANR0900I Processing options file /home/tsminst1/dsmserv.opt.
ANR7811I Using instance directory /home/tsminst1.
Illegal instruction(coredump)

# dbx /opt/tivoli/tsm/server/bin/dsmserv core.10944896.28165312
Type 'help' for help.
[using memory image in core.10944896.28165312]
reading symbolic information ...warning: no source compiled with -g

Illegal instruction (illegal opcode) in . at 0x0 ($t1)
warning: Unable to access address 0x0 from core

(dbx) where
.() at 0x0
gsk_src_create__FPPvPv(??, ??) at 0x9000000015b6d88
__ct__8GSKMutexFv(??) at 0x9000000018d664c
__ct__20GSKPasswordEncryptorFv(??) at 0x9000000018cb248
__ct__7gsk_envFv(??) at 0x900000000aaa6b0
GskEnvironmentOpen__FPPvb(??, ??) at 0x900000000ab14c4
gsk_environment_open(??) at 0x900000000ab277c
IPRA.$CheckGSKVersion() at 0x100eecf68
tlsInit() at 0x100eecd70
main(??, ??) at 0x10000112c

(dbx) th
thread state-k wchan state-u k-tid mode held scope function
$t1 run running 41877977 k no sys
$t2 run blocked 21234465 u no sys _cond_wait_global
$t3 run running 24380103 u no sys waitpid


dsmserv fails to start if LDAP is inaccessible

Posted by xaminmo

IBM, and the white books, say this is working as designed.
• If LDAP dies, dsmserv stays up without it.
• If LDAP dies, and dsmserv restarts, it refuses to come up.
• ANR3103E https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.5/srv.msgs/b_msgs_server.pdf
• Workaround is remove LDAPURL from dsmserv.opt, or wait for LDAP to become accessible.

On a multi-homed server, any links serving LDAP should be fault tolerant.

Any server using LDAP should have fault tolerant LDAP servers.

Here’s where to vote on getting the start-up limitation changed.
http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=121985


Spectrum Protect / TSM systemd autostart

Posted by xaminmo


cat < <'EOF' >/etc/systemd/system/db2fmcd.service
[Unit]
Description=DB2V111

[Service]
ExecStart=/opt/tivoli/tsm/db2/bin/db2fmcd
Restart=always
KillMode=process
KillSignal=SIGHUP

[Install]
WantedBy=default.target
EOF
systemctl enable db2fmcd.service
systemctl start db2fmcd.service

cp -p /opt/tivoli/tsm/server/bin/dsmserv.rc /etc/init.d/tsminst1
cat < <'EOF' >>/etc/systemd/system/tsminst1.service
[Unit]
Description=tsminst1
Requires=db2fmcd.service

[Service]
Type=forking
ExecStart=/etc/init.d/tsminst1 start
ExecReload=/etc/init.d/tsminst1 reload
ExecStop=/etc/init.d/tsminst1 stop
StandardOutput=journal

[Install]
WantedBy=multi-user.target
EOF
systemctl enable tsminst1.service
systemctl start tsminst1.service

ln -s /opt/tivoli/tsm/client/ba/bin/rc.dsmcad /etc/init.d/dsmcad
cat < <'EOF' >>/etc/systemd/system/dsmcad.service
[Unit]
Description=dsmcad

[Service]
Type=forking
ExecStart=/etc/init.d/dsmcad start
ExecReload=/etc/init.d/dsmcad reload
ExecStop=/etc/init.d/dsmcad stop
StandardOutput=journal

[Install]
WantedBy=multi-user.target
EOF
systemctl enable dsmcad.service
systemctl start dsmcad.service


Spectrum Protect – container vulnerability

Posted by xaminmo

We ran into an issue where a level-zero operator became root, and cleaned up some TSM dedupe-pool containers so he’d stop getting full filesystem alerts.

Things exposed:

How does someone that green get full, unmonitored root access?
* They told false information about timestaps during defense
* Their senior tech lead was content to advise they not move or delete files without contacting the app owner.
* Imagine if this had been a customer facing database server!

In ISP/TSM, once extents are marked damaged, a new backup of that extent will replace it.
* Good TDP4VT CTL files and other incrementals will send missing files.
* TDP for VMWare full backups fail if the control file backup is damaged.
* Damaged extents do not mark files as damaged or missing.

Replicate Node will back-propagate damaged files.
* Damaged extents do not mark files as damaged or missing.

Also, in case you missed that:
* Damaged extents do not mark files as damaged or missing.

For real, IBM says:
* Damaged extents do not mark files as damaged or missing.
* “That might cause a whole bunch of duplicates to be ingested and processed.”

IBM’s option is to use REPAIR STGPOOL.
* Requires a prior PROTECT STGPOOL (similar to BACKUP STGPOOL and RESTORE STGPOOL).
* PROTECT STGPOOL can go to a container copy on tape, a container copy on FILE, or a container primary on the replica target server.
* PROTECT STGPOOL cannot go to a cloud pool
* STGRULE TIERING only processes files, not PROTECT extents.
* PROTECT STGPOOL cannot go to a cloud pool that way either.
* There is NO WAY to use cloud storage pool to protect a container pool from damage.

EXCEPTION: Damaged extents can be replaced by REPLICATE NODE into a pool.
* You can DISABLE SES, and reverse the replication config.
* Replicate node that way will perform a FULL READ of the source pool.

There is a Request For Enhancement from November, 2017 for TYPE=CLOUD POOLTYPE=COPY.
* That would be a major code effort, but would solve this major hole.
* That has not gotten a blink from product engineering.
* Not even an “under review”, nor “No Way”, nor “maybe sometime”.

Alternatives for PROTECT into CLOUD might be:
* Don’t use cloud. Double the amount of local disk space, and replicate to another datacenter.
* Use NFS (We would need to build a beefy VM, and configure KRB5 at both ends, so we could do NFSv4 encrypted).
* Use CIFS (the host is on AIX, which does not support CIFS v3. Linux conversion up front before we had bulk data was given a big NO.)
* Use azfusefs (Again, it’s not Linux)

Anyway, maybe in 2019 this can be resolved, but this is the sort of thing that really REALLY was poorly documented, and did not get the time and resources to be tested in advance. This is the sort of thing that angers everyone at every level.

REFERENCE: hard,intr,nfsvers=4,tcp,rsize=1048576,wsize=1048576,bg,noatime


ANR3114E LDAP error 81. Failure to connect to the LDAP server

Posted by xaminmo

This used to be on IBM’s website, but it disappeared.  It is referenced all over the net, and needed to still exist. I only found it in the wayback machine, so I’m adding another copy to the internet.

2013 SOURCE: www-01.ibm.com/support/docview.wss?uid=swg21656339

Problem(Abstract)

When the SET LDAPUSER command is used, the connection can fail with:

ANR3114E LDAP error 81 (Can’t contact LDAP server)

Cause

The user common name (CN) in the SET LDAPUSER command contains a space or the ldapurl option is incorrectly specified.

Diagnosing the problem

Collect a trace of the Tivoli Storage Manager Server using the following trace classes:
session verbdetail ldap ldapcache unicode

More information about tracing the server can be found here: Enabling a trace for the server or storage agent

The following errors are reported within the trace:

11:02:04.127 [44][output.c][7531][PutConsoleMsg]:ANR2017I Administrator ADMIN issued command: SET LDAPPASSWORD ?***? ~
11:02:04.171 [44][ldapintr.c][548][ldapInit]:Entry: ldapUserNew =      CN=tsm user,OU=TSM,DC=ds,DC=example,DC=com
11:02:04.173 [44][ldapintr.c][5851][LdapHandleErrorEx]:Entry: LdapOpenSession(ldapintr.c:2340) ldapFunc = ldap_start_tls_s_np, ldapRc = 81, ld = 0000000001B0CAB0
11:02:04.174 [44][ldapintr.c][5867][LdapHandleErrorEx]:ldap_start_tls_s_np returned LDAP code 81(Can't contact LDAP server), LDAP Server message ((null)), and possible GSKIT SSL/TLS error 0(Success)
11:02:04.174 [44][output.c][7531][PutConsoleMsg]:ANR3114E LDAP error 81 (Can't contact LDAP server) occurred during ldap_start_tls_s_np.~
11:02:04.174 [44][ldapintr.c][6079][LdapHandleErrorEx]:Exit: rc = 2339, LdapOpenSession(ldapintr.c:2340), ldapFunc = ldap_start_tls_s_np, ldapRc = 81, ld = 0000000001B0CAB0
11:02:04.174 [44][ldapintr.c][1580][ldapCloseSession]:Entry: sessP = 0000000009B99CD0
11:02:04.175 [44][ldapintr.c][3159][LdapFreeSess]:Entry: sessP = 0000000009B99CD0
11:02:04.175 [44][ldapintr.c][2449][LdapOpenSession]:Exit: rc = 2339, ldapHandleP = 000000000AFDE740, bindDn =                              (CN=tsm user,OU=TSM,DC=ds,DC=example,DC=com)
11:02:04.175 [44][output.c][7531][PutConsoleMsg]:ANR3103E Failure occurred while initializing LDAP directory services.~
11:02:04.175 [44][ldapintr.c][856][ldapInit]:Exit: rc = 2339
11:02:04.175 [44][output.c][7531][PutConsoleMsg]:ANR2732E Unable to communicate with the external LDAP directory server.~

Resolving the problem

  • In the trace provided, the common name (CN) contains a space. (CN=tsm user,OU=TSM,DC=ds,DC=example,DC=com)

    Remove the space in the common name when using the SET LDAPUSER command. For example:

    SET LDAPUSER “CN=tsmuser,OU=TSM,DC=ds,DC=example,DC=com”

  • Use an LDAP connection utility such as ldp.exe to ensure the ldapurl option is correct and the LDAP server is accepting connections

    <ldapurl> port 636, check the box for SSL

    Verify there are no errors in the output


AIX and PowerHA levels

Posted by xaminmo

Research shows these dates for AIX:
https://www.ibm.com/support/pages/aix-support-lifecycle-information
It’s generally 26 weeks, plus or minus, from the initial YYWW date. Once a TLSP APARs releases, the YYWW code is be updated.

  • 7300-01-01-2246 2022-12-02 (Next 2023Q2)
  • 7200-05-05-2246 2022-12-02 (Next 2023Q2)
  • 7100-05-10-2220 2022-09-09 (Next 2023Q1)
  • 6100-09-12-1846 2018-11-16 (EoL CSP)

My AIX selection process would be:

  • AIX 7.3.1.1 from 2022 week 46 is what I have in my repo.  Another TL should be coming out 2023 Q1.  None of my customers run this, but you want this for POWER10.  NIM should be latest of all versions as well.
  • AIX 7.2.5.5 from 2022 week 46 is what I have in my repo.  Another TL should be coming out 2023 Q1.  You probably want this for POWER7 and up.
  • AIX 7.1.5.10 from 2022 week 20 is what I have in my repo.  I think the CSP is 2023Q1.  Supports AIX 5.2 and 5.3 WPARs.  Not much other reason to use this now other than some specific apps that are OK with OpenSSL, OpenSSH, and Java updates, but not kernel updates.
  • AIX 6.1.9.12 is where I stopped tracking.  No real need for AIX 6 anywhere.  Either you’re stuck on 5.3, or you came up to 7.1 (or ideally 7.2).  6.1.9.9 was needed for application compatibility on POWER9.
  • For anything POWER6 or older should really upgrade to p710 to p740 or s81x/s82x as replacements (cost).  POWER8 is EoS 2024-10-31.  POWER7 is EoS 2019.
  • AIX 5.3.12.9 + U866665 on POWER8 is end-stage.  AIX 5.3 was EoS in 2012, but some people still run it now.  Power8 is EoS 2024-10-31.  Power7 was 2019.
  • AIX 5.3 PTF U866665.bff (bos.mp64.5.3.12.10.U) enables POWER8. AIX must be 5.3.12.9. Must be patched before moving to p8. p8 must be 840 firmware or later. VIO must be 2.2.4.10 or later.  Migration is by LPM, NIM, or mksysb.  equires active extended support agreement AIX p8 systems on file to download.
  • AIX older – You should not be running anything older.  AIX 5.1 was all CHRP, and AIX 4.x was all PCI.  AIX 3.x was MicroChannel.  AIX 2.1 had some PS/2 systems.  Outside of a museum, on an isolated network (or no network because CYLONS!), just have this recycled.

My PowerHA (HA/CMP) selection process would be:
https://www.ibm.com/support/pages/powerha-aix-version-compatibility-matrix 

  • 7.2.7 Base is what I last grabbed.  OK for AIX 7.1.5, 7.2.5, 7.3.0 and later patch levels.
  • 7.1.3 SP09 was the end-stage for this.  OK for AIX 6.1.9.11, 7.1.3.9, 7.2.0.6, and later patch levels.  No AIX 5, and no AIX 7.3.
  • 6.1 SP15 was end-stage for this, and supported AIX 5.3.9, 6.1.2.1, 7.1.0, and later patch levels.  No AIX 5.1, 7.2, or 7.3.

Code sources:

UPDATE 2023-03-07:

  • Refreshed all of the info above to current.  If you’re on AIX 5.2, HA/CMP 5.x, or VIO 1.x, that’s really disappointing.
  • VIO should be 3.1.4.10.  Always go current whenever possible.  If not, 2.2.6.65 is your target.  If you have any VIO 1.x, upgrade.  Period.
  • System firmware, adapter microcode, disk microcode, tape microcode, and library firmware should all be latest available.
  • Storage array firmware should be latest LTS patch level, excluding any .0 versions.  I still don’t trust Data Reduction Pools.
  • ADSM/ITSM/TSM/Spectrum Protect/Storage Protect – These should typically be the final version supporting your OS/App combo.  I’m sweet on 8.1.17, though there are still major issues with deduplication pools when your database is over 3TB.  Containers and extents cannot be purged.  I have not seen SP 9.1, but I assume it will be extremely similar to SP 8.1.18 other than some minor rebadging.  Not sure.  I got dropped from the beta program because I didn’t have cycles to test their new code, and they didn’t have any interest in steering features.  They pick what they want to pick, and you’ll like it. 
  • IBM is phasing out Spectrum Protect Plus (Catalogic DPX), but might still be keeping Catalogic ECX (Copy Data Management).  In Storage Defender, IBM has picked up Cohesity DataProtect because of the cloud / DRaaS bits.  These all integrate with DS8000/Flash Systems for data immutability / vaulting / ransomware protection, and they want you to buy Rapid7 for the AI/Logic behind it.  I know regular ISP’s operations center anomaly detection us unusable due to its lack of adaptability/logic.  It just says everything is alerted every week when you run weekly fulls, etc.

I don’t really track IBMi OS (OS400), zOS, zVM, etc.  Storage should be rebranding this year though, but still no NFS/CIFS hardware.  At best, IBM sells a GPFS cluster with Ceph and with some StorWize FS7200s.