Recovering from “SKIP UPGRADING THIS INSTANCE”

REFS:
https://www.ibm.com/support/pages/manually-upgrading-ibm-spectrum-protect-server-instances
https://www.ibm.com/support/pages/anr0187e-failure-during-server-startup
http://issen007.blogspot.com/2017/05/manual-upgrade-ibm-spectrum-protect-71x.html

###################################################
### Stop the instance completely
su - tsminst1

### This may not work if your environment or links are bad.
db2 list db directory

### If db2sysc is still running
ps | grep db2 
db2stop
db2stop force
db2 terminate ### Kill off db2bp fragments

### kill everything else other
ps | grep db2

### Remove IPC
ipcrm -a


###################################################
### Clean up remainders
su - root
/opt/tivoli/tsm/db2/instance/db2ilist
/opt/tivoli/tsm/db2/instance/db2idrop tsminst1

### Verify nothing left
/opt/tivoli/tsm/db2/instance/db2ilist


###################################################
### Redefine the instance
su - root
#/opt/tivoli/tsm/db2/instance/db2icrt -a server -u tsminst1 tsminst1
/opt/tivoli/tsm/db2/instance/db2icrt -u tsminst1 tsminst1

DBI1446I The db2icrt command is running.
DB2 installation is being initialized.
Total number of tasks to be performed: 4
Total estimated time for all tasks to be performed: 309 second(s)

Task #1 start
Description: Setting default global profile registry variables
Estimated time 1 second(s)
Task #1 end

Task #2 start
Description: Initializing instance list
Estimated time 5 second(s)
Task #2 end

Task #3 start
Description: Configuring DB2 instances
Estimated time 300 second(s)
Task #3 end

Task #4 start
Description: Updating global profile registry
Estimated time 3 second(s)
Task #4 end

The execution completed successfully.
For more information see the DB2 installation log at "/tmp/db2icrt.log.21176".
DBI1070I Program db2icrt completed successfully.


###################################################
### Set up Db2 environment variables
# NOTE: userprofile and db2profile get reset after db2icrt
su - tsminst1
/opt/tivoli/tsm/db2/adm/db2set -i tsminst1 "DB2_SKIPINSERTED=ON"
/opt/tivoli/tsm/db2/adm/db2set -i tsminst1 "DB2_KEEPTABLELOCK=ON"
/opt/tivoli/tsm/db2/adm/db2set -i tsminst1 "DB2_EVALUNCOMMITTED=ON"
/opt/tivoli/tsm/db2/adm/db2set -i tsminst1 "DB2_SKIPDELETED=ON"
/opt/tivoli/tsm/db2/adm/db2set -i tsminst1 "DB2CODEPAGE=819"
/opt/tivoli/tsm/db2/adm/db2set -i tsminst1 "DB2_PARALLEL_IO=*"

cat <<EOF >>${HOME}/sqllib/userprofile
export LD_LIBRARY_PATH=${HOME}/sqllib/lib64/gskit:${HOME}/sqllib/lib32:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/tivoli/tsm/server/bin/dbbkapi:/opt/ibm/lib:/opt/ibm/lib64:/usr/lib64:${HOME}/sqllib/lib64
export PATH=$PATH:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/tivoli/tsm/server/bin64
export PATH=$PATH:/opt/tivoli/tsm/server/bin:/usr/tivoli/tsm/server/bin64:/usr/tivoli/tsm/server/bin
export PATH=$PATH:/opt/tivoli/tsm/client/ba/bin64:/opt/tivoli/tsm/client/ba/bin:/usr/tivoli/tsm/client/ba/bin64
export PATH=$PATH:/usr/tivoli/tsm/client/ba/bin:/usr/tivoli/tsm/client/api/bin64:/usr/tivoli/tsm/client/api/bin
export PATH=$PATH:/opt/tivoli/tsm/client/api/bin64:/opt/tivoli/tsm/client/api/bin:/opt/tivoli/tsm/db2/bin
export PATH=$PATH:${HOME}/sqllib/bin:${HOME}/sqllib/adm:${HOME}/sqllib/misc

DSMI_CONFIG=${HOME}/tsmdbmgr.opt
DSMI_DIR=/opt/tivoli/tsm/server/bin/dbbkapi
DSMI_LOG=${HOME}
export DSMI_CONFIG DSMI_DIR DSMI_LOG 
EOF

cat <<EOF >>${HOME}/.profile
. ${HOME}/sqllib/db2profile
. ${HOME}/sqllib/userprofile

alias ll='ls -laF --color=auto'
set -o vi
EOF

. ./.profile


###################################################
### Catalog the DB to make sure it is okay.
db2start

### Find the TSMDB1 instances).
DBDIR=$(find /home /sp /tsm -name sqldbdir -exec strings {} \; 2>/dev/null | grep inst | cut -c 2-99 | sort | uniq)
echo $DBDIR

### Register the instance(s).
for i in $DBDIR ; do db2 catalog db TSMDB1 on $i ; done
# SQL6028N Catalog database failed because database "tsminst1" was not found in the local database directory.

### List the instances.
db2 list db directory


###################################################
### Upgrade the DB2 system catalog tables
#db2 upgrade db tsminst1
#SQL1013N The database alias name or database name "TSMINST1" could not be found. SQLSTATE=42705
db2 upgrade db TSMDB1
SQL1103W The UPGRADE DATABASE command was completed successful.

### Stop DB2 to make sure it flushes everything.
db2stop
SQAL1064N DB2STOP processing was successful.


###################################################
### Upgrade the TSM database schema
/opt/tivoli/tsm/server/bin/dsmserv upgradedb
ANR7800I DSMSERV generated at 18:03:03 on Nov 19 2021.

IBM Spectrum Protect for AIX
Version 8, Release 1, Level 13.000

Licensed Materials - Property of IBM

(C) Copyright IBM Corporation 1990, 2021.
All rights reserves.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.

ANR7801I Subsystem process ID is 64684398.
ANR7811I Using instance directory /tsm/tsminst1/
ANR3339I Default Label in key database is TSM Server SelfSigned SHA Key.
ANR4726I The ICC support module has been loaded.
ANR0990I Server restart-recovery in progress.

###################################################
### Make sure the server accepts workload

# start the server normally (rc script, systemd, or inittab line run from an at-job).

### Run these from dsmadmc
REG LIC FILE=*ee.lic
enable ses all