Upgrading from VMware vRealize Suite Lifecycle Manager 8.10 to 8.12

I recently upgraded my vRSLCM (Now VMware Aria Suite Lifecycle Manager) instance from 8.10 to 8.12. I ran into some major issues. The upgrade kept failing. I wanted to share how I found the root issue and resolved it.

1. Snapshot before you begin (no matter what!).

2. Always use the ISO Repository Archive mount option to do your vRSLCM upgrade. Do not do it from Online (At least this hasnt worked for me with this upgrade with success). The online method for me kept having issues with properly downloading the Linux-PAM package for some reason. https://customerconnect.vmware.com/downloads/details?downloadGroup=SUITELIFECYCLE812&productId=938&rPId=104277

3. Debugging the upgrade is your friend. https://kb.vmware.com/s/article/89606

I found during debugging that the /opt/vmware/var/log/vami/updatecli.log log file documented what really was happening. Turns out my RPM DB was corrupted on vRSLCM. I had to do the following to clear this DB and then was able to get the upgrade to complete.

Example Error from my uppdatecli.log file. I have underlined the real issue: 24/04/2023 15:30:00 [INFO] Update status: Done pre-install scripts

24/04/2023 15:30:00 [INFO] Update status: Running installation tests

24/04/2023 15:30:00 [INFO] Running /opt/vmware/var/lib/vami/update/data/job/5/test_command

error: rpmdb: BDB0113 Thread/process 45741/139742890768192 failed: BDB1507 Thread died in Berkeley DB library

error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

error: cannot open Packages index using db5 - (-30973)

error: cannot open Packages database in /var/lib/rpm

warning: /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/Linux-PAM-1.3.0-1.ph3.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 66fd4949: NOKEY

error: rpmdb: BDB0113 Thread/process 45741/139742890768192 failed: BDB1507 Thread died in Berkeley DB library

error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

error: cannot open Packages index using db5 - (-30973)

error: cannot open Packages database in /var/lib/rpm

24/04/2023 15:30:00 [ERROR] Failed with exit code 256

24/04/2023 15:30:00 [INFO] Update status: Running post-install scripts

24/04/2023 15:30:00 [INFO] Running /opt/vmware/var/lib/vami/update/data/job/5/post_install '8.10.0.6' '8.12.0.7' 2

Updating vami-sfcb.service - Removing vmtoolsd service dependency

already service is having restart policy

Failed with status of 2 while installing version 8.12.0.7

VM version is still 8.10.0.6

++ basename /usr/lib/bootstrap/postupdate

+ readonly name=postupdate

+ name=postupdate

+ readonly bootstrap_dir=/etc/bootstrap/postupdate.d

+ bootstrap_dir=/etc/bootstrap/postupdate.d

+ readonly bootstrap_log=/var/log/bootstrap/postupdate.log

+ bootstrap_log=/var/log/bootstrap/postupdate.log

+ exec

+ log 'Main bootstrap postupdate started'

++ date '+%Y-%m-%d %H:%M:%S'

+ echo '2023-04-24 15:30:00 Main bootstrap postupdate started'

+ for script in "${bootstrap_dir}"/*

+ echo

+ '[' '!' -e /etc/bootstrap/postupdate.d/00-00-rpm-status-check ']'

+ '[' '!' -x /etc/bootstrap/postupdate.d/00-00-rpm-status-check ']'

+ log '/etc/bootstrap/postupdate.d/00-00-rpm-status-check starting...'

++ date '+%Y-%m-%d %H:%M:%S'

+ echo '2023-04-24 15:30:00 /etc/bootstrap/postupdate.d/00-00-rpm-status-check starting...'

+ /etc/bootstrap/postupdate.d/00-00-rpm-status-check 8.10.0.6 8.12.0.7 2

+ res=1

+ echo 'Script /etc/bootstrap/postupdate.d/00-00-rpm-status-check failed, error status 1'

+ exit 1

24/04/2023 15:30:00 [ERROR] Failed with exit code 256

24/04/2023 15:30:00 [INFO] Update status: Running VMware tools reconfiguration

24/04/2023 15:30:00 [INFO] Running /opt/vmware/share/vami/vami_reconfigure_tools

vmware-toolbox-cmd is /bin/vmware-toolbox-cmd

vmtoolsd wrapper not required on this VM with systemd.

24/04/2023 15:30:01 [INFO] Update status: Done VMware tools reconfiguration

24/04/2023 15:30:01 [INFO] Update status: Error while running installation tests and post-install scripts

24/04/2023 15:30:01 [ERROR] Failure: updatecli exiting abnormally

24/04/2023 15:30:01 [INFO] Install Finished

To resolve this DB issue, I followed this Redhat (Redhat?!, I know right) bug report steps to resolve it https://bugzilla.redhat.com/show_bug.cgi?id=1270528

SSH Steps I took to clean the RPM DB:

1. cd /var/lib/rpm

2. rm -f __db*

3. rpm -qa 1> /dev/null

4. rpm -v --rebuilddb

5. tdnf clean

Steps to take after doing the RPM DB Clean up:

Simply run the upgrade again against the ISO and you should see your upgrade run successfully.