Intro
Virtualization made a significant change in the IT (Information Technology)
industry. This technology helped many organizations to use server resources
efficiently. Even though cloud technology is emerging some companies are not
ready to move their workloads to the cloud due to data sensitivity and business
obstacles. So the best cost-effective option is to use virtualization technology which helps both purposes saving IT capex costs and having a secure environment.
As organizations hosting their critical database workloads, we need to ensure
virtualization platforms are fully secure. Oracle Linux Virtualization Manager
has these security certificates that need to be renewed every
year.
In this article, I will elaborate on how we can renew these certificates from
the OLVM engine.
Use below mentioned metalink note for the OLVM renewal process :
OLVM: How to renew OLVM Hosts Certificate in OLVM Environment/Infrastructure (Doc ID 2885203.1)
Validate the current certificate expiration time :
Execute the openssl command to validate the certificate expiration time.
Validate engine certificate :
Note: First we need to renew the engine certificate and then we can renew the KVM certificates.
[root@sofe-olvm-01 ~]# openssl x509 -in /etc/pki/ovirt-engine/certs/engine.cer -noout -dates
notBefore=Jul 26 13:31:25 2022 GMT
notAfter=Aug 29 13:31:25 2023 GMT
[root@sofe-olvm-01 ~]#
Validate KVM certificate :
Execute openssl command to validate the KVM certificate
As per below mentioned outputs KVM certificates are expiring on Oct 11th, 2023. These certificates are valid only for 1 year.
Note: We need to renew these certificates before these expiry dates, Once the certificates expired there is a chance of KVM's abnormal behavior. Because OLVM recognizes these KVM using the certificates.
[root@kvm01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates
notBefore=Sep 7 20:05:13 2022 GMT
notAfter=Oct 11 20:05:13 2023 GMT
[root@kvm01 ~]#
[root@kvm02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates
notBefore=Sep 7 20:41:43 2022 GMT
notAfter=Oct 11 20:41:43 2023 GMT
[root@kvm02 ~]#
Review Engine Certificates
First, we need to renew the engine certificate using the below-mentioned steps. For this renewal, we do not need to take any downtime.
As a best practice let's backup the engine.
engine-backup --scope=all --mode=backup --file=/root/backup_renew_certificate_30Aug2023.bck --log=/root/backup_renew_certificate.log
[root@engine-olvm-01 ~]# engine-backup --scope=all --mode=backup --file=/root/backup_renew_certificate_30Aug2023.bck --log=/root/backup_renew_certificate.log
Start of engine-backup with mode 'backup'
scope: all
archive file: /root/backup_renew_certificate_30Aug2023.bck
log file: /root/backup_renew_certificate.log
Backing up:
Notifying engine
- Files
- Engine database 'engine'
- DWH database 'ovirt_engine_history'
- Grafana database '/var/lib/grafana/grafana.db'
Packing into file '/root/backup_renew_certificate_30Aug2023.bck'
Notifying engine
Done.
[root@engine-olvm-01 ~]#
Validate all the certificates in the engine
It's better to validate all the certificates before the renewal, This gives you a clear idea about what we need to renew
Login to the engine server and navigate to "/etc/pki/ovirt-engine/certs/" folder and execute for cert in *.cer; do echo "Certificate: $cert"; openssl x509 -in "$cert" -noout -dates; echo; do a script to get all the certificates.
cd /etc/pki/ovirt-engine/certs/
for cert in *.cer; do echo "Certificate: $cert"; openssl x509 -in "$cert" -noout -dates; echo; done
[root@engine-olvm-01 certs]# for cert in *.cer; do echo "Certificate: $cert"; openssl x509 -in "$cert" -noout -dates; echo; done
Certificate: apache.cer
notBefore=Jul 26 13:31:26 2022 GMT
notAfter=Aug 29 13:31:26 2023 GMT
Certificate: engine.cer
notBefore=Jul 26 13:31:25 2022 GMT
notAfter=Aug 29 13:31:25 2023 GMT
Certificate: jboss.cer
notBefore=Jul 26 13:31:26 2022 GMT
notAfter=Aug 29 13:31:26 2023 GMT
Certificate: KVM120.sofe.ca.cer
notBefore=Sep 7 20:05:13 2022 GMT
notAfter=Oct 11 20:05:13 2023 GMT
Certificate: KVM120.sofe.ca-ssh.cer
notBefore=Sep 7 20:06:34 2022 GMT
notAfter=Oct 11 20:06:34 2023 GMT
Certificate: KVM121.sofe.ca.cer
notBefore=Sep 7 20:41:43 2022 GMT
notAfter=Oct 11 20:41:43 2023 GMT
Certificate: KVM121.sofe.ca-ssh.cer
notBefore=Sep 7 20:43:08 2022 GMT
notAfter=Oct 11 20:43:08 2023 GMT
Certificate: ovirt-provider-ovn.cer
notBefore=Jul 26 13:31:27 2022 GMT
notAfter=Aug 29 13:31:27 2023 GMT
Certificate: ovn-ndb.cer
notBefore=Jul 26 13:31:26 2022 GMT
notAfter=Aug 29 13:31:26 2023 GMT
Certificate: ovn-sdb.cer
notBefore=Jul 26 13:31:27 2022 GMT
notAfter=Aug 29 13:31:27 2023 GMT
Certificate: reports.cer
notBefore=Jul 26 13:31:26 2022 GMT
notAfter=Aug 29 13:31:26 2023 GMT
Certificate: vmconsole-proxy-helper.cer
notBefore=Jul 26 13:31:42 2022 GMT
notAfter=Aug 29 13:31:42 2023 GMT
Certificate: vmconsole-proxy-host.cer
notBefore=Jul 26 13:31:43 2022 GMT
notAfter=Aug 29 13:31:43 2023 GMT
Certificate: vmconsole-proxy-user.cer
notBefore=Jul 26 13:31:43 2022 GMT
notAfter=Aug 29 13:31:43 2023 GMT
Certificate: websocket-proxy.cer
notBefore=Jul 26 13:31:26 2022 GMT
notAfter=Aug 29 13:31:26 2023 GMT
[root@engine-olvm-01 certs]#
How to renew a Certificate
To renew a certificate we need to execute the "engine-setup --offline" command, This generates all the new certificates.
[root@sofe-olvm-01 ~]# engine-setup --offline
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf
Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20230830141846-ehsd5w.log
Version: otopi-1.9.5 (otopi-1.9.5-1.el8)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment setup (late)
[ INFO ] Stage: Environment customization
--== PRODUCT OPTIONS ==--
[ INFO ] ovirt-provider-ovn already installed, skipping.
--== PACKAGES ==--
--== NETWORK CONFIGURATION ==--
Setup can automatically configure the firewall on this system.
Note: automatic configuration of the firewall may overwrite current settings.
Do you want Setup to configure the firewall? (Yes, No) [Yes]: No
--== DATABASE CONFIGURATION ==--
The detected DWH database size is 138.05131435394287 MB.
Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete.
If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost.
Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: No
[WARNING] Are you sure you do not want to backup the DWH database?
A positive reply makes sense only if you do not need the data in DWH, or have some other, external means to restore it to a working state.
Are you sure you do not want to backup the DWH database?(Yes, No) [No]:
The DWH Database will be backed up prior to upgrade.
Perform full vacuum on the oVirt engine history
database ovirt_engine_history@localhost?
This operation may take a while depending on this setup health and the
configuration of the db vacuum process.
See https://www.postgresql.org/docs/12/sql-vacuum.html
(Yes, No) [No]:
--== OVIRT ENGINE CONFIGURATION ==--
Perform full vacuum on the engine database engine@localhost?
This operation may take a while depending on this setup health and the
configuration of the db vacuum process.
See https://www.postgresql.org/docs/12/sql-vacuum.html
(Yes, No) [No]:
--== STORAGE CONFIGURATION ==--
--== PKI CONFIGURATION ==--
One or more of the certificates should be renewed, because they expire soon, or include an invalid expiry date, or they were created with validity period longer than 398 days, or do not include the subjectAltName extension, which can cause them to be rejected by recent browsers and up to date hosts.
See https://www.ovirt.org/develop/release-management/features/infra/pki-renew/ for more details.
Renew certificates? (Yes, No) [No]: Yes
--== APACHE CONFIGURATION ==--
--== SYSTEM CONFIGURATION ==--
--== MISC CONFIGURATION ==--
--== END OF CONFIGURATION ==--
[ INFO ] Stage: Setup validation
During execution engine service will be stopped (OK, Cancel) [OK]:
[ INFO ] Cleaning stale zombie tasks and commands
--== CONFIGURATION PREVIEW ==--
Default SAN wipe after delete : False
Host FQDN : oracle-olvm-01.sofe.ca
Firewall manager : firewalld
Update Firewall : False
Set up Cinderlib integration : False
Engine database host : localhost
Engine database port : 5432
Engine database secured connection : False
Engine database host name validation : False
Engine database name : engine
Engine database user name : engine
Engine installation : True
PKI organization : sofe.ca
Renew PKI : True
Set up ovirt-provider-ovn : True
Grafana integration : True
Grafana database user name : ovirt_engine_history_grafana
Configure WebSocket Proxy : True
DWH installation : True
DWH database host : localhost
DWH database port : 5432
DWH database secured connection : False
DWH database host name validation : False
DWH database name : ovirt_engine_history
DWH database user name : ovirt_engine_history
Backup DWH database : True
Configure VMConsole Proxy : True
Please confirm installation settings (OK, Cancel) [OK]:
[ INFO ] Cleaning async tasks and compensations
[ INFO ] Unlocking existing entities
[ INFO ] Checking the Engine database consistency
[ INFO ] Stage: Transaction setup
[ INFO ] Stopping engine service
[ INFO ] Stopping ovirt-fence-kdump-listener service
[ INFO ] Stopping dwh service
[ INFO ] Stopping vmconsole-proxy service
[ INFO ] Stopping websocket-proxy service
[ INFO ] Stopping service: grafana-server
[ INFO ] Stage: Misc configuration (early)
[ INFO ] Stage: Package installation
[ INFO ] Stage: Misc configuration
[ INFO ] Upgrading CA
[ INFO ] Renewing engine certificate
[ INFO ] Renewing jboss certificate
[ INFO ] Renewing websocket-proxy certificate
[ INFO ] Renewing apache certificate
[ INFO ] Renewing reports certificate
[ INFO ] Updating OVN SSL configuration
[ INFO ] Updating OVN timeout configuration
[ INFO ] Backing up database localhost:ovirt_engine_history to '/var/lib/ovirt-engine-dwh/backups/dwh-20230830142301.tv3pj2k_.dump'.
[ INFO ] Creating/refreshing DWH database schema
[ INFO ] Configuring WebSocket Proxy
[ INFO ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20230830142309.uai8smw2.dump'.
[ INFO ] Creating/refreshing Engine database schema
[ INFO ] Creating/refreshing Engine 'internal' domain database schema
Unregistering existing client registration info.
[ INFO ] Install selinux module /usr/share/ovirt-engine/selinux/ansible-runner-service.cil
[ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
[ INFO ] Stage: Transaction commit
[ INFO ] Stage: Closing up
[ INFO ] Starting service: grafana-server
[ INFO ] Starting engine service
[ INFO ] Starting dwh service
[ INFO ] Starting Grafana service
[ INFO ] Restarting ovirt-vmconsole proxy service
--== SUMMARY ==--
[ INFO ] Restarting httpd
Web access is enabled at:
http://oracle-olvm-01.sofe.ca:80/ovirt-engine
https://oracle-olvm-01.sofe.ca:443/ovirt-engine
Internal CA DD:55:2B:EF:9C:D7:93:F9:A1:AF:9B:64:54:02:0A:FF:88:6F:5B:BD
SSH fingerprint: SHA256:XPSQG8LoQaL2r65lTUeNkzdChjFZUXaowO6kNUTKZaQ
Web access for grafana is enabled at:
https://oracle-olvm-01.sofe.ca/ovirt-engine-grafana/
Please run the following command on the engine machine oracle-olvm-01.sofe.ca, for SSO to work:
systemctl restart ovirt-engine
--== END OF SUMMARY ==--
[ INFO ] Stage: Clean up
Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20230830141846-ehsd5w.log
[ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20230830142400-setup.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ INFO ] Execution of setup completed successfully
[root@engine-olvm-01 ~]#
Post certificate renewal
After executing the engine renewal command, Make sure to restart the overt-engine so this will help to login to the OLVM console page.
[root@engine-olvm-01 ~]# systemctl restart ovirt-engine.service
[root@engine-olvm-01 ~]# systemctl status ovirt-engine.service
● ovirt-engine.service - oVirt Engine
Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2023-08-30 14:25:24 EDT; 6s ago
Main PID: 3289277 (ovirt-engine.py)
Tasks: 104 (limit: 100229)
Memory: 685.7M
CGroup: /system.slice/ovirt-engine.service
├─3289277 /usr/libexec/platform-python /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify start
└─3289464 ovirt-engine --add-modules java.se -server -XX:+TieredCompilation -Xms3930M -Xmx3930M -Xss1M -Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcIn>
Aug 30 14:25:24 engine-olvm-01.sofe.ca systemd[1]: Starting oVirt Engine...
Aug 30 14:25:24 engine-olvm-01.sofe.ca systemd[1]: Started oVirt Engine.
Aug 30 14:25:24 engine-olvm-01.sofe.ca ovirt-engine.py[3289277]: 2023-08-30 14:25:24,999-0400 ovirt-engine: INFO _detectJBossVersion:186 Detecting JBoss version. Running: /usr/lib/jvm/jre-11-openjdk-11.0.16>
Aug 30 14:25:25 engine-olvm-01.sofe.ca ovirt-engine.py[3289277]: 2023-08-30 14:25:25,838-0400 ovirt-engine: INFO _detectJBossVersion:206 Return code: 0, | stdout: '['WildFly Full 23.0.2.Final (WildFly Core>
lines 1-14/14 (END)
Conclusion
Any platform security is a major concern, As system and database architects we need to make sure these major SSL certificate renewals are happening at the right time. Renewing the certificate every year gives you a fully secure compliant virtualized platform and reduces the changes to OLVM engine vulnerabilities.
No comments:
Post a Comment