Upgrading from VMware vRealize Suite Lifecycle Manager 8.10 to 8.12
I recently upgraded my vRSLCM (Now VMware Aria Suite Lifecycle Manager) instance from 8.10 to 8.12. I ran into some major issues. The upgrade kept failing. I wanted to share how I found the root issue and resolved it.
Snapshot before you begin (no matter what!).
Always use the ISO Repository Archive mount option to do your vRSLCM upgrade. Do not do it from Online (At least this hasnt worked for me with this upgrade with success). The online method for me kept having issues with properly downloading the Linux-PAM package for some reason. https://customerconnect.vmware.com/downloads/details?downloadGroup=SUITELIFECYCLE812&productId=938&rPId=104277
Debugging the upgrade is your friend. https://kb.vmware.com/s/article/89606
I found during debugging that the /opt/vmware/var/log/vami/updatecli.log log file documented what really was happening. Turns out my RPM DB was corrupted on vRSLCM. I had to do the following to clear this DB and then was able to get the upgrade to complete.
Example Error from my uppdatecli.log file. I have underlined the real issue: 24/04/2023 15:30:00 [INFO] Update status: Done pre-install scripts
24/04/2023 15:30:00 [INFO] Update status: Running installation tests
24/04/2023 15:30:00 [INFO] Running /opt/vmware/var/lib/vami/update/data/job/5/test_command
error: rpmdb: BDB0113 Thread/process 45741/139742890768192 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 - (-30973)
error: cannot open Packages database in /var/lib/rpm
warning: /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/Linux-PAM-1.3.0-1.ph3.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 66fd4949: NOKEY
error: rpmdb: BDB0113 Thread/process 45741/139742890768192 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 - (-30973)
error: cannot open Packages database in /var/lib/rpm
24/04/2023 15:30:00 [ERROR] Failed with exit code 256
24/04/2023 15:30:00 [INFO] Update status: Running post-install scripts
24/04/2023 15:30:00 [INFO] Running /opt/vmware/var/lib/vami/update/data/job/5/post_install '8.10.0.6' '8.12.0.7' 2
Updating vami-sfcb.service - Removing vmtoolsd service dependency
already service is having restart policy
Failed with status of 2 while installing version 8.12.0.7
VM version is still 8.10.0.6
++ basename /usr/lib/bootstrap/postupdate
+ readonly name=postupdate
+ name=postupdate
+ readonly bootstrap_dir=/etc/bootstrap/postupdate.d
+ bootstrap_dir=/etc/bootstrap/postupdate.d
+ readonly bootstrap_log=/var/log/bootstrap/postupdate.log
+ bootstrap_log=/var/log/bootstrap/postupdate.log
+ exec
+ log 'Main bootstrap postupdate started'
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2023-04-24 15:30:00 Main bootstrap postupdate started'
+ for script in "${bootstrap_dir}"/*
+ echo
+ '[' '!' -e /etc/bootstrap/postupdate.d/00-00-rpm-status-check ']'
+ '[' '!' -x /etc/bootstrap/postupdate.d/00-00-rpm-status-check ']'
+ log '/etc/bootstrap/postupdate.d/00-00-rpm-status-check starting...'
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2023-04-24 15:30:00 /etc/bootstrap/postupdate.d/00-00-rpm-status-check starting...'
+ /etc/bootstrap/postupdate.d/00-00-rpm-status-check 8.10.0.6 8.12.0.7 2
+ res=1
+ echo 'Script /etc/bootstrap/postupdate.d/00-00-rpm-status-check failed, error status 1'
+ exit 1
24/04/2023 15:30:00 [ERROR] Failed with exit code 256
24/04/2023 15:30:00 [INFO] Update status: Running VMware tools reconfiguration
24/04/2023 15:30:00 [INFO] Running /opt/vmware/share/vami/vami_reconfigure_tools
vmware-toolbox-cmd is /bin/vmware-toolbox-cmd
vmtoolsd wrapper not required on this VM with systemd.
24/04/2023 15:30:01 [INFO] Update status: Done VMware tools reconfiguration
24/04/2023 15:30:01 [INFO] Update status: Error while running installation tests and post-install scripts
24/04/2023 15:30:01 [ERROR] Failure: updatecli exiting abnormally
24/04/2023 15:30:01 [INFO] Install Finished
To resolve this DB issue, I followed this Redhat (Redhat?!, I know right) bug report steps to resolve it https://bugzilla.redhat.com/show_bug.cgi?id=1270528
SSH Steps I took to clean the RPM DB:
cd /var/lib/rpm
rm -f __db*
rpm -qa 1> /dev/null
rpm -v --rebuilddb
tdnf clean
Steps to take after doing the RPM DB Clean up:
Simply run the upgrade again against the ISO and you should see your upgrade run successfully.
Comments