Spectrum Protect: Failed to prepare update packages.

OpCenter GUI Failed to prepare update packages, and the button is greyed out.
We have never been able to use OpCenter client updates.

We’ve gone through the steps in the 8.1.007 release notes
https://www.ibm.com/support/pages/ibm-spectrum-protect-version-817000-fix-pack-readme-files
* Command routing works from hub to spoke and spoke to hub.
* Every server has a FILE class, and half have a directory class pool.
* Every server has an HLA and LLA set.
* Most of our clients to use cad with passwordaccc=generate

We’ve gone through the dependencies in opcenter help, which is mostly the same.

We have tried deleting the DEPLOY nodes and letting them re-replicate.

Protect: TSM1>q act begind=-1 endd=today msgno=3759
02/07/23 08:00:31 ANR3759E An error occurred during the replication of client update packages from node IBM_DEPLOY_CLIENT_UNX to the monitored server, TSM2. The return code is 18. (PROCESS: 1612)
02/07/23 08:00:31 ANR3759E An error occurred during the replication of client update packages from node IBM_DEPLOY_CLIENT_WIN to the monitored server, TSM2. The return code is 18. (PROCESS: 1613)

Reset the remote clients that are failing

TSM2: DEL FI IBM_DEPLOY_CLIENT_UNX *
TSM2: DEL FI IBM_DEPLOY_CLIENT_WIN *

TSM2: remove replnode IBM_DEPLOY_CLIENT_UNX server=TSM1
TSM2: remove replnode IBM_DEPLOY_CLIENT_WIN server=TSM1

TSM2: remove node IBM_DEPLOY_CLIENT_UNX
TSM2: remove node IBM_DEPLOY_CLIENT_WIN

Reset deploypkgmgr – not really needed

SET DEPLOYPKGMGR off
SET DEPLOYREPOSITORY /sp/software/octemp/downloads/
SET DEPLOYMAXPKGS 4
SET DEPLOYPKGMGR on

HELPS – The underlying hidden command that does the sync.

refresh pkg clean=no startnow=yes

Once this processes, the button turned blue.

Asked IBM to address this because if there are multiple spokes, failure of one spoke should not block the others.


Replacing TSM / ISP server

I ran into an issue where a primary TSM 7.1.8 server was broken, and it was easier to just move all of the clients over to the secondary 8.1.5 server. These used the new TLS encryption, and I kept running into issues.

    —————

First, I physically shut down the old server, and updated the new server to use the IP as an alias by editing /etc/network/interfaces. (Ubuntu 16 LTS)

    —————

Various errors included:
ANR8599W The connection with host address:host port failed due to an untrusted server certificate. An attempt to reconnect and establish certificate trust might follow.

ANR2284S The server master encryption key has changed. Passwords protected with the previous master encryption key are not available.

I had to make the server trust itself again with:[LINK]
dsmcert -add -server TSM -file /home/tsminst1/cert256.arm

    —————

This error:
ANR0456W Session rejected for server DT – the server name at 192.168.1.99, 1500 does not match.

I removed the server:
REMOVE SERVER DT

    —————

These errors:
ANR1651E Server information for DT is not available.
ANR4377E Session failure, target server DT is not defined on the source server.
ANR3151E Configuration refresh failed with configuration manager DT.

I disabled replication config
REMOVE REPLNODE *
Q REPLSERVER
REMOVE REPLSERVER {GUID}
DEL SUBSCRIPTION DEFAULT_PROFILE

    —————

These errors:
ANS1695E The certificate is not valid.
ANS1592E Failed to initialize SSL protocol.
ANS8023E Unable to establish session with server.
ANR3335W Unable to distribute certificate to for session .
ANR8599W The connection with 192.168.1.2:40250 failed due to an untrusted server certificate. An attempt to reconnect and establish certificate trust might follow.

From the clients, I had to remove /opt/tivoli/tsm/client/ba/bin/dsmcert.* and C:\Program Files\Tivoli\TSM\baclient\dsmcert.* on the clients per http://andrewjtobiason.com/index.php/2018/08/16/resolving-ssl-errors/

However, I also had to fix dsmcert / dsmcert.exe which I clobbered in the process.

    —————

And on the server, allow keys to be swapped again:[LINK]
UPD NODE * SESSIONSECURITY=TRANSITIONAL
UPD ADMIN * SESSIONSECURITY=TRANSITIONAL

Swap keys on client:
dsmadmc

    —————

NOTE: I also removed /etc/adsm/* during troubleshooting, but that was not needed. That just lead to me having to re-enter passwords again. Simply deleting the cert database corrected the problem on other clients.

    —————

I tried to order this in dependency order. I was sort of all over the place when I did it, and might have missed something. I just could not exactly get all of the right info from any one documentation source.