Thursday, August 18, 2022

Virtualized ODA upgrade 18.8.0.0 to 19.8.0.0 - Journey to 12.1.2.12 - 19.13.0.0 - Part 3

 

ODA upgrade 18.8.0.0 to 19.8.0.0



                        


Hope previous blog posts were useful to address ODA upgrade issues for the virtualization platforms. 

19.13 will be the last upgrade for the virtualized platform, After that oracle going to discontinue OVM and they are moving to KVM-based virtualization. This is going to be a game changer for ODA for performance.

In this article, I will cover the 19.8.0.0 upgrade steps. I will write another article to cover the issues we encountered during the 18.8.0.0 - 19.8.0.0. This is the major jump for ODA. This is going to be more challenging than other upgrades Mainly because previously we had to tackle mainly grid upgrades. Here we need to upgrade the operating system (OS) from oel6 to oel7. So we need to make sure the OS is stable before moving forward with any other component.

Two Major upgrades are 

  1. OS upgrade OEL 6 to OEL 7.
  2. Grid update from 18c to 19c.

Journey to move virtualized ODA to 19.8.0.0. We found this below-mentioned plan from looking at the oracle documents link.

https://docs.oracle.com/en/engineered-systems/oracle-database-appliance/19.8/cmtxn/patching-oda.html

Section 1: Gather ODA server information

Find the ODA hardware version.




######### Find ODA version:
[root@ecl-oda-DOM0-0 ~]# ipmitool sunoem cli 'show System' | grep model
        model = ODA X5-2
        component_model = ORACLE SERVER X5-2
[root@ecl-oda-DOM0-0 ~]#

[root@ecl-oda-DOM0-1 ~]# ps -ef | grep pmon
root      5577  4433  0 13:57 pts/2    00:00:00 grep pmon
[root@ecl-oda-DOM0-1 ~]# ipmitool sunoem cli 'show System' | grep model
        model = ODA X5-2
        component_model = ORACLE SERVER X5-2
[root@ecl-oda-DOM0-1 ~]



Find the server version:


[root@ecl-odabase-0 delshare]# oakcli show server

        Power State              : On
        Open Problems            : 0
        Model                    : ODA X5-2
        Type                     : Rack Mount
        Part Number              : 33765810+1+1
        Serial Number            : 1535NMF006
        Primary OS               : Not Available
        ILOM Address             : 10.11.99.202
        ILOM MAC Address         : 00:10:E0:8D:5E:AC
        Description              : Oracle Database Appliance X5-2 1535NMF006
        Locator Light            : Off
        Actual Power Consumption : 288 watts
        Ambient Temperature      : 15.750 degree C
        Open Problems Report     : System is healthy



Find hardware version:


[root@ecl-odabase-0 delshare]# oakcli show env_hw
VM-ODA_BASE ODA X5-2
[root@ecl-odabase-0 delshare]#



Check for running VMs and repositories :

If the environment is virtualized you will need to note down the VM names and repository names before patching. These repositories are ACFS mount points, which stores allocated to store the virtual machines and virtual disks.



=======================================
Check the running VMS and respository
=======================================


[root@ecl-odabase-0 PATCH]# oakcli show vm

          NAME                                  NODENUM         MEMORY          VCPU            STATE           REPOSITORY

        kali_server                             0               4196M              2            ONLINE          kali_test
        qualyssrv                               0               4196M              2            ONLINE          qualys


[root@ecl-odabase-0 PATCH]#




[root@ecl-odabase-0 PATCH]# oakcli show repo

          NAME                          TYPE            NODENUM  FREE SPACE     STATE           SIZE


          kali_test                     shared          0            94.78%     ONLINE          512000.0M

          kali_test                     shared          1            94.78%     ONLINE          512000.0M

          odarepo1                      local           0               N/A     N/A             N/A

          odarepo2                      local           1               N/A     N/A             N/A

          qualys                        shared          0            98.44%     ONLINE          204800.0M

          qualys                        shared          1            98.44%     ONLINE          204800.0M

          vmdata                        shared          0            99.99%     ONLINE          4068352.0M

          vmdata                        shared          1               N/A     UNKNOWN         N/A

          vmsdev                        shared          0            99.99%     ONLINE          1509376.0M

          vmsdev                        shared          1               N/A     UNKNOWN         N/A




Find the current ODA version


[root@ecl-odabase-0 ~]# oakcli show version -detail
Reading the metadata. It takes a while...
System Version  Component Name            Installed Version         Supported Version
--------------  ---------------           ------------------        -----------------
18.8.0.0.0
                Controller_INT            4.650.00-7176             Up-to-date
                Controller_EXT            13.00.00.00               Up-to-date
                Expander                  001E                      Up-to-date
                SSD_SHARED {
                [ c1d20,c1d21,c1d22,      A29A                      Up-to-date
                c1d23,c1d44,c1d45,c1
                d46,c1d47 ]
                [ c1d16,c1d17,c1d18,      A29A                      Up-to-date
                c1d19,c1d40,c1d41,c1
                d42,c1d43 ]
                             }
                HDD_LOCAL                 A7E0                      Up-to-date
                HDD_SHARED {
                [ c1d24,c1d25,c1d26,      A3A0                      Up-to-date
                c1d27,c1d29,c1d30,c1
                d31,c1d32,c1d33,c1d3
                4,c1d35,c1d36,c1d37,
                c1d38,c1d39 ]
                [ c1d0,c1d1,c1d2,c1d      PD51                      Up-to-date
                3,c1d4,c1d5,c1d6,c1d
                7,c1d8,c1d9,c1d10,c1
                d11,c1d12,c1d13,c1d1
                4,c1d15,c1d28 ]
                             }
                ILOM                      4.0.4.52 r132805          Up-to-date
                BIOS                      30300200                  Up-to-date
                IPMI                      1.8.15.0                  Up-to-date
                HMP                       2.4.5.0.1                 Up-to-date
                OAK                       18.8.0.0.0                Up-to-date
                OL                        6.10                      Up-to-date
                OVM                       3.4.4                     Up-to-date
                GI_HOME                   18.8.0.0.191015           Up-to-date
                DB_HOME                   12.1.0.2.191015           Up-to-date
[root@ecl-odabase-0 ~]#


Section 2: Pre-patching preparation steps

2.1 Space check

First of all, we need to ensure we have enough space on /, /u01, and /opt file systems. At least 20 GB should be available. If not, we can do some cleaning or extend the LVM partitions. 

 Usage of below mentioned mount points :

  •  Unpacking is happening in /opt mount point 
  •  Grid and DB patching and upgrades are happening in /u01 mount. 
  •  OS upgrade is happening in / mount point. 

 Note: After unpacking the root “/” mount point we need 8GB free on both nodes. /u01 we need 15GB free on both nodes


Space Check


############# space check 

[root@ecl-odabase-0 orapkgs]# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda2       55G   42G  9.7G  82% /


[root@ecl-odabase-0 ~]# df -h / /u01 /opt
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda2       55G   42G  9.9G  81% /
/dev/xvdb1       92G   70G   18G  81% /u01
/dev/xvda2       55G   42G  9.9G  81% /
[root@ecl-odabase-0 ~]#


[root@ecl-odabase-1 ~]# df -h / /u01 /opt
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda2       55G   44G  8.3G  84% /
/dev/xvdb1       92G   71G   17G  82% /u01
/dev/xvda2       55G   44G  8.3G  84% /
[root@ecl-odabase-1 ~]#


2.2 Unpacking patches and upgrading repository

In virtualized environments, copy the patch bundle to ODA_Base on both nodes. Use the below mentioned commands to unpack 19.8.0.0 patches to /opt mount point.



oakcli unpack -package /db_backup/19.8.0.0/p31481822_198000_Linux-x86-64_1of3.zip
oakcli unpack -package /db_backup/19.8.0.0/p31481822_198000_Linux-x86-64_2of3.zip
oakcli unpack -package /db_backup/19.8.0.0/p31481822_198000_Linux-x86-64_3of3.zip

Expected output:



-- node01

[root@ecl-odabase-0 19.8.0.0]# oakcli unpack -package /db_backup/19.8.0.0/p31481822_198000_Linux-x86-64_1of3.zip
Unpacking will take some time,  Please wait...
Successfully unpacked the files to repository.

[root@ecl-odabase-0 19.8.0.0]# oakcli unpack -package /db_backup/19.8.0.0/p31481822_198000_Linux-x86-64_2of3.zip
Unpacking will take some time,  Please wait...
Successfully unpacked the files to repository.
[root@ecl-odabase-0 19.8.0.0]#

[root@ecl-odabase-0 19.8.0.0]# oakcli unpack -package /db_backup/19.8.0.0/p31481822_198000_Linux-x86-64_3of3.zip
Unpacking will take some time,  Please wait...
Successfully unpacked the files to repository.


-- node02

[root@ecl-odabase-1 oak]# oakcli unpack -package /db_backup/19.8.0.0/p31481822_198000_Linux-x86-64_1of3.zip
Unpacking will take some time,  Please wait...
Successfully unpacked the files to repository.

[root@ecl-odabase-1 oak]# oakcli unpack -package /db_backup/19.8.0.0/p31481822_198000_Linux-x86-64_2of3.zip
Unpacking will take some time,  Please wait...
Successfully unpacked the files to repository.

[root@ecl-odabase-1 ~]# oakcli unpack -package /db_backup/19.8.0.0/p31481822_198000_Linux-x86-64_3of3.zip
Unpacking will take some time,  Please wait...
Successfully unpacked the files to repository.
You have new mail in /var/spool/mail/root



Update the repository with the latest patches.


Once the unpacking is successful update the repository with latest patches.

oakcli update -patch  19.8.0.0.0 --verify

[root@ecl-odabase-0 oak]# oakcli update -patch  19.8.0.0.0 --verify
INFO: 2022-05-17 15:16:30: Reading the metadata file now...
                Component Name            Installed Version         Proposed Patch Version
                ---------------           ------------------        -----------------
                Controller_INT            4.650.00-7176             Up-to-date
                Controller_EXT            13.00.00.00               16.00.08.00
                Expander                  001E                      Up-to-date
                SSD_SHARED {
                [ c1d20,c1d21,c1d22,      A29A                      Up-to-date
                c1d23,c1d44,c1d45,c1
                d46,c1d47 ]
                [ c1d16,c1d17,c1d18,      A29A                      Up-to-date
                c1d19,c1d40,c1d41,c1
                d42,c1d43 ]
                             }
                HDD_LOCAL                 A7E0                      Up-to-date
                HDD_SHARED {
                [ c1d24,c1d25,c1d26,      A3A0                      Up-to-date
                c1d27,c1d29,c1d30,c1
                d31,c1d32,c1d33,c1d3
                4,c1d35,c1d36,c1d37,
                c1d38,c1d39 ]
                [ c1d0,c1d1,c1d2,c1d      PD51                      Up-to-date
                3,c1d4,c1d5,c1d6,c1d
                7,c1d8,c1d9,c1d10,c1
                d11,c1d12,c1d13,c1d1
                4,c1d15,c1d28 ]
                             }
                ILOM                      4.0.4.52 r132805          Up-to-date
                BIOS                      30300200                  Up-to-date
                IPMI                      1.8.15.0                  Up-to-date
                HMP                       2.4.5.0.1                 Up-to-date
                OAK                       18.8.0.0.0                19.8.0.0.0
                OL                        6.10                      7.8
                OVM                       3.4.4                     Up-to-date
                GI_HOME                   18.8.0.0.191015           19.8.0.0.200714
                DB_HOME                   12.1.0.2.191015           12.1.0.2.200714
                AHF                       213400                    202100
[root@ecl-odabase-0 oak]#



-- node02

[root@ecl-odabase-1 oak]# oakcli update -patch  19.8.0.0 --verify
INFO: 2022-05-17 15:20:48: Reading the metadata file now...
                Component Name            Installed Version         Proposed Patch Version
                ---------------           ------------------        -----------------
                Controller_INT            4.650.00-7176             Up-to-date
                Controller_EXT            13.00.00.00               16.00.08.00
                Expander                  001E                      Up-to-date
                SSD_SHARED {
                [ c1d20,c1d21,c1d22,      A29A                      Up-to-date
                c1d23,c1d44,c1d45,c1
                d46,c1d47 ]
                [ c1d16,c1d17,c1d18,      A29A                      Up-to-date
                c1d19,c1d40,c1d41,c1
                d42,c1d43 ]
                             }
                HDD_LOCAL                 A7E0                      Up-to-date
                HDD_SHARED {
                [ c1d0,c1d1,c1d2,c1d      A3A0                      Up-to-date
                3,c1d5,c1d6,c1d7,c1d
                8,c1d9,c1d10,c1d11,c
                1d12,c1d13,c1d14,c1d
                15 ]
                [ c1d4,c1d24,c1d25,c      PD51                      Up-to-date
                1d26,c1d27,c1d28,c1d
                29,c1d30,c1d31,c1d32
                ,c1d33,c1d34,c1d35,c
                1d36,c1d37,c1d38,c1d
                39 ]
                             }
                ILOM                      4.0.4.52 r132805          Up-to-date
                BIOS                      30300200                  Up-to-date
                IPMI                      1.8.15.0                  Up-to-date
                HMP                       2.4.5.0.1                 Up-to-date
                OAK                       18.8.0.0.0                19.8.0.0.0
                OL                        6.10                      7.8
                OVM                       3.4.4                     Up-to-date
                GI_HOME                   18.8.0.0.191015           19.8.0.0.200714
                DB_HOME                   12.1.0.2.191015           12.1.0.2.200714
                AHF                       213400                    202100
[root@ecl-odabase-1 oak]#


2.3 Health check report

I would recommend performing a health check on the system before proceeding with patching. For health checks reports can be obtained by "oakcli validate -d ", There are many other options for validating the command. I normally do this when planning for Exadata patching. 

 Execute Validate command, check the health status



################## validate the ODA environment 

https://docs.oracle.com/cd/E68623_01/doc.121/e68637/GUID-0B61366B-720E-4DBC-A805-20435EB5E9CF.htm#:~:text=oakcli%20validate%20%2Dl,sensor%0A%20%20%20%20%20%20%20%20%20data%20readings

oakcli validate -l

         Checkname -- Description
         =========    ===========
         *SystemComponents -- Validate system components based on ilom sensor data
         readings
         *OSDiskStorage -- Validate OS disks and filesystem information
         *SharedStorage -- Validate Shared storage and multipathing information
         DiskCalibration -- Check disk performance with orion
         *NetworkComponents -- Validate public and private network components
         *StorageTopology -- Validate external JBOD connectivity
         asr -- Validate asr components based on asr config file and ilom sensor
         data readings

oakcli validate -a -- Validate all the components

===================================================================================================
======================================== OS Upgrade

Pre-Patching There are two main pre-patching steps are there for 19.8.0.0 

  •  Main Pre-patching to validate os and components 
  •  OS Pre-patching - Need to execute during downtime

Main Pre-patching to validate os and components


  
  ################# Pre-Patching steps OS

  oakcli update -patch 19.8.0.0.0 --pre-patchreport --local            - First
  
  
  

Note: Patching only can run after everything looks clean on pre-check reports.

Please find the Sample pre-patching report, This report has failed due to unallocated ASM disks to DATA and RECO.

Note: 1st attempt failed with ASM disk failure.

  
  ############ Log 


#### Node01
[root@ecl-odabase-0 oak]# oakcli update -patch 19.8.0.0.0 --pre-patchreport --local
INFO: Running OS prepatch checks ...

INFO: Validate supported versions
SUCCESS: Validated minimum supported versions.
INFO: Validate patching tag
SUCCESS: Validated patching tag: 19.8.0.0.0
INFO: Is patch location available
SUCCESS: Patch location is available.
INFO: Space checks for OS upgrade
ERROR: Minimum 7 GB free space required under "/root"
INFO: Validate custom rpms installed
ERROR: The custom RPM checks may not have run
ERROR: Some of the OS prepatch tasks went wrong
INFO: Running Storage prepatch checks ...
INFO: Validate patching tag
SUCCESS: Validated patching tag: 19.8.0.0.0
INFO: Patch location validation
SUCCESS: Verified patch location.
INFO: Patch tag validation
SUCCESS: Verified patching tag.
INFO: Verify ASM disks status
ERROR: One or more ASM disk are not online. Rolling storage update cannot proceed.
ERROR: Some of the Storage prepatch tasks went wrong
ERROR: Pre patch report ended with errors
You have new mail in /var/spool/mail/root

##### 

Solution: Fix the ASM issue


  
  ############# ASM disk work around.

SQL> alter diskgroup /*+ _OAK_AsmCookie */ reco add disk '/dev/mapper/HDD_E1_S08_1931561940p2' name HDD_E1_S08_1931561940p2 force;
  
  
2nd attempt: After allocating the ASM disk to DATA and RECO disk groups. 

Note: Make sure you have enough space on the root “/” mount point.

  
  [root@ecl-odabase-0 ~]# oakcli update -patch 19.8.0.0.0 --pre-patchreport --local
INFO: Running OS prepatch checks ...

INFO: Validate supported versions
SUCCESS: Validated minimum supported versions.
INFO: Validate patching tag
SUCCESS: Validated patching tag: 19.8.0.0.0
INFO: Is patch location available
SUCCESS: Patch location is available.
INFO: Space checks for OS upgrade
SUCCESS: Validated space checks.
INFO: Install OS upgrade software
INFO: This action might take a while if packages need to be uncompressed
SUCCESS: Extracted OS upgrade patches into /root/oda-upgrade.
Do not remove this directory until OS upgrade completes.
INFO: Verify OS upgrade by running preupgrade checks
INFO: This operation will take long, please wait
SUCCESS: Results stored in: '/root/preupgrade-results/preupg_results-220520152633.tar.gz' .
Read complete report file '/root/preupgrade/result.html' before attempting OS upgrade.
INFO: Validate custom rpms installed
WARNING: Found additional RPMs installed
Warning: user is responsible for updating custom RPM files
SUCCESS: All OS prepatch tasks ran correctly
INFO: Running Storage prepatch checks ...
INFO: Validate patching tag
SUCCESS: Validated patching tag: 19.8.0.0.0
INFO: Patch location validation
SUCCESS: Verified patch location.
INFO: Patch tag validation
SUCCESS: Verified patching tag.
INFO: Verify ASM disks status
SUCCESS: ASM disk are online.
SUCCESS: All Storage prepatch tasks ran correctly

SUCCESS: Pre patch report ended successfully
You have new mail in /var/spool/mail/root
  
  
-– downtime

Patching


OS Pre-patching



################ Note 
This will stop the  local node, run the oakcli update -patch with the pre-osupgrade option to stop the Virtual Machines, shared repositories, Oracle Clusterware, and databases:

oakcli update -patch 19.8.0.0.0 --pre-osupgrade --local              - Second 

3rd party rpm list pre-patching will create a directory in “/root/oda-upgrade” this has information about the 3rd party rpm list.



########## Validate 3rd party rpms 

cd /root/oda-upgrade

[root@ecl-odabase-0 oda-upgrade]# cat rpms-added-from-ThirdParty
perl-Expect
perl-IO-Stty
perl-Net-OpenSSH
perl-RPC-XML
socat
[root@ecl-odabase-0 oda-upgrade]#

Sample output for OS Pre-Patching on node 01.

OS pre-patching needs to be executed on both nodes in a sequential manner.

Note: On the local node, run the oakcli update -patch with the pre-osupgrade option to stop the Virtual Machines, shared repositories, Oracle Clusterware, and databases.


############# OS-Pre-Patch

[root@ecl-odabase-0 ~]# oakcli update -patch 19.8.0.0.0 --pre-osupgrade --local
INFO: Trying to ping dom0 agent
INFO: dom0 agent is live
INFO: dom0 agent is not updated.
INFO: Updating dom0 agent, this will take a while...
SUCCESS: dom0 agent updated
INFO: Stopping Vms and Repos on both nodes. This may take a while...
SUCCESS: Repos and vms stopped
INFO: Disabling start repo
Start repo operation is now DISABLED on node 0
Start repo operation is now DISABLED on node 1
SUCCESS: Start repo disabled
INFO: Stopping CRS in local node
INFO: 2022-05-20 16:44:47: Clusterware is running on local node
INFO: 2022-05-20 16:44:47: Attempting to stop clusterware and its resources locally
SUCCESS: 2022-05-20 16:46:32: Successfully stopped the clusterware on local node
SUCCESS: CRS stopped in local node
SUCCESS: Loop devices check passed
INFO: Checking network script file (ifcfg-eth0)
INFO: Network script file (ifcfg-eth0) check finished
INFO: Updating ovsvf-config RPM on DOM0
SUCCESS: Upgrade DOM0 ovsvf-config RPM succeeded
SUCCESS: Pre-OSupgrade finished successfully
You have new mail in /var/spool/mail/root

Patching OS

 Make sure to stop all the database and other application services before starting the work and backup the ODA_BASE in case you need to restore the backup to 18.8. 

  •  Stop database services and other application services. 
  •  Backup ODA_BASE

  1. stop the oem agent 
  2. stop vm and repo 
  3. stop cluster on patching node.


####################### stop emagent - both nodes 
/u01/agent13.3/agent_13.3.0.0.0/bin/emctl stop agent 



####################### Stop Servers 
####################### VM stop commands 

oakcli show vm - note down the running vm's
oakcli stop vm  dns_10_7 -node 1 -- example 

oakcli stop vm  dns_10_7



###################### Repo stop commands 
[root@ecl-odabase-0 ~]# oakcli stop repo vmstor1 -node 0
[root@ecl-odabase-0 ~]# oakcli stop repo vmstor1 -node 1

Note : OS patching can be executed using option --local. Make sure to run on one node at a time as this is a risk involved in the upgrade.



Note : OS Path needs to be complete on both nodes , move the server patching.

########### Before OS upgrade 
1. stop vm 
2. stop repo 
3. stop cluster : /u01/app/18.0.0.0/grid/bin/crsctl stop crs -f

###### 30 min - 45 min 
script /root/19.8_UPGRADE/output_os_upgrade_07june2022.txt
oakcli update -patch 19.8.0.0.0 --osupgrade --local


Once completes the os-upgrade run the post-upgrade as the last step to validate all the components.



### post os patch 
[root@hostname ~]# /opt/oracle/oak/bin/oakcli update -patch 19.8.0.0.0 --post-osupgrade --local

screen -ls
script /root/19.8_UPGRADE/output_os_post_07june2022.txt
oakcli update -patch 19.8.0.0.0 --post-osupgrade --local

Sample output for post-osupgrade



############## post upgrade 

### node01

[root@ecl-odabase-0 ~]# script /root/19.8_UPGRADE/output_os_post_07june2022.txt
Script started, file is /root/19.8_UPGRADE/output_os_post_07june2022.txt
[root@ecl-odabase-0 ~]# oakcli update -patch 19.8.0.0.0 --post-osupgrade --local
INFO: Validating OS version
SUCCESS: OS has been upgraded to 4.14.35-1902.303.4.1.el7uek successfully.
INFO: Validating OAK version
SUCCESS: ODA software has been upgraded to 19.8.0.0.0 successfully.
INFO: Validating GI version
SUCCESS: GI software is running at correct version: 18.8.0.0.191015(30112122,30581079).
INFO: Starting CRS
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
WARNING: There are additional RPMs installed on the system.
WARNING: Please upgrade these RPMs.
INFO: Starting NFS service
SUCCESS: NFS service has been started successfully.
INFO: Removing ODA OS patching software
SUCCESS: ODA OS patching software has been removed
INFO: Updating system inventory
SUCCESS: System inventory updated
SUCCESS: Patching post-osupgrade command finished successfully
[root@ecl-odabase-0 ~]#


Validate after os upgrade


########### Verification

oakcli update -patch  19.8.0.0.0 --verify


Expected output


-- node01
[root@ecl-odabase-0 oda-upgrade]# oakcli update -patch  19.8.0.0 --verify
INFO: 2022-06-07 15:44:55: Reading the metadata file now...
                Component Name            Installed Version         Proposed Patch Version
                ---------------           ------------------        -----------------
                Controller_INT            4.650.00-7176             Up-to-date
                Controller_EXT            13.00.00.00               16.00.08.00
                Expander                  001E                      Up-to-date
                SSD_SHARED {
                [ c1d20,c1d21,c1d22,      A29A                      Up-to-date
                c1d23,c1d44,c1d45,c1
                d46,c1d47 ]
                [ c1d16,c1d17,c1d18,      A29A                      Up-to-date
                c1d19,c1d40,c1d41,c1
                d42,c1d43 ]
                             }
                HDD_LOCAL                 A7E0                      Up-to-date
                HDD_SHARED {
                [ c1d24,c1d25,c1d26,      A3A0                      Up-to-date
                c1d27,c1d29,c1d30,c1
                d31,c1d32,c1d33,c1d3
                4,c1d35,c1d36,c1d37,
                c1d38,c1d39 ]
                [ c1d0,c1d1,c1d2,c1d      PD51                      Up-to-date
                3,c1d4,c1d5,c1d6,c1d
                7,c1d8,c1d9,c1d10,c1
                d11,c1d12,c1d13,c1d1
                4,c1d15,c1d28 ]
                             }
                ILOM                      4.0.4.52 r132805          Up-to-date
                BIOS                      30300200                  Up-to-date
                IPMI                      1.8.15.0                  Up-to-date
                HMP                       2.4.5.0.1                 Up-to-date
                OAK                       19.8.0.0.0                Up-to-date
                OL                        7.8                       Up-to-date
                OVM                       3.4.4                     Up-to-date
                GI_HOME                   18.8.0.0.191015           19.8.0.0.200714
                DB_HOME                   12.1.0.2.191015           12.1.0.2.200714
                AHF                       213400                    202100
[root@ecl-odabase-0 oda-upgrade]#

--node02
[root@ecl-odabase-1 ~]# oakcli update -patch  19.8.0.0 --verify
INFO: 2022-06-07 16:07:19: Reading the metadata file now...
                Component Name            Installed Version         Proposed Patch Version
                ---------------           ------------------        -----------------
                Controller_INT            4.650.00-7176             Up-to-date
                Controller_EXT            13.00.00.00               16.00.08.00
                Expander                  001E                      Up-to-date
                SSD_SHARED {
                [ c1d20,c1d21,c1d22,      A29A                      Up-to-date
                c1d23,c1d44,c1d45,c1
                d46,c1d47 ]
                [ c1d16,c1d17,c1d18,      A29A                      Up-to-date
                c1d19,c1d40,c1d41,c1
                d42,c1d43 ]
                             }
                HDD_LOCAL                 A7E0                      Up-to-date
                HDD_SHARED {
                [ c1d0,c1d1,c1d2,c1d      A3A0                      Up-to-date
                3,c1d5,c1d6,c1d7,c1d
                8,c1d9,c1d10,c1d11,c
                1d12,c1d13,c1d14,c1d
                15 ]
                [ c1d4,c1d24,c1d25,c      PD51                      Up-to-date
                1d26,c1d27,c1d28,c1d
                29,c1d30,c1d31,c1d32
                ,c1d33,c1d34,c1d35,c
                1d36,c1d37,c1d38,c1d
                39 ]
                             }
                ILOM                      4.0.4.52 r132805          Up-to-date
                BIOS                      30300200                  Up-to-date
                IPMI                      1.8.15.0                  Up-to-date
                HMP                       2.4.5.0.1                 Up-to-date
                OAK                       19.8.0.0.0                Up-to-date
                OL                        7.8                       Up-to-date
                OVM                       3.4.4                     Up-to-date
                GI_HOME                   18.8.0.0.191015           19.8.0.0.200714
                DB_HOME                   12.1.0.2.191015           12.1.0.2.200714
                AHF                       213400                    202100
[root@ecl-odabase-1 ~]#

Patching Server

This includes server patching includes upgrading the grid to 19c.

Note: /u01 needs more than 12GB, Safe side we need 25GB


######################################################################
############################ Server Patching 
Note: /u01 needs more than 12GB , Safe side we need 25GB
Sample : /opt/oracle/oak/bin/oakcli update -patch 19.8.0.0.0 --server

screen -ls
script /root/19.8_UPGRADE/output_server_08june2022.txt
oakcli update -patch 19.8.0.0.0 --server

Patching Storage


screen 
screen -ls
script /root/19.8_UPGRADE/script_Storage_10June2022.txt - record all the steps  

oakcli update -patch  19.8.0.0.0 --storage

Conclusion

Moving to 18.8.0.0 - 19.8.0.0 is a really critical upgrade Because this upgrade includes the os component. Make sure you are a consistent backup to recover in case of failure. As a best practice create a proactive SR with oracle. Also, validate OS precheck before proceding further on the upgrade. Virtualized ODA is very stable after moving to 19.8.0.0.

Next article I will cover the 19.13.0.0 upgrade. 

Tuesday, August 16, 2022

OLVM - (Oracle Linux Virtualization Manager) - Part 1/5 (Installation and Configuration)





Intro

Virtualization made a significant change in the IT (Information Technology) industry.  This technology helped many organizations to use server resources efficiently. Server virtualization revolutionized data center management and helped organizations cut the fat out of the IT infrastructure. This was a game changer for data center investments and optimization. Nowadays IT engineers are managing software-driven data centers. 

Virtualization requires the use of a hypervisor, hypervisor abstracts operating systems and applications from their underlying hardware. This enables the host hardware to operate multiple VMs independently of each other and share abstract resources among those VMs.

Virtualization there are two major types.

  1. Full-virtualization 
  2. Para-virtualization

Even though the cloud exists still organizations invest in virtualization technology. Because there are many benefits. A few pros and cons are listed below.

Pros of virtualization

  1. Reduced IT costs, This cost reduction comes from different levels.  
  2. I.   Capital expenditure savings:

    Virtualization enables reducing the cost by reducing the hardware footprint in the data center by requiring fewer hardware servers and related resources to achieve the same level of computing performance, availability, and Scalability. 

    II.  Operational expenditure savings:

    Once servers are virtualized, IT staff can greatly reduce the ongoing administration and management of manual intervention time-consuming processes like server installation.

    III. Data center and energy savings:

           As we shrink company hardware and server footprint, we are lowering the data center energy consumption and square footage.

    2. Efficient resource utilization.

       Most companies spend a lot of capital settings up their systems and server but eventually use only a fraction of it effectively. Virtualization enables to host of multiple VMS on a single hypervisor. This will save the hardware cost and attain a high-efficiency level.

    3. Scalability and Availability.

            Virtualization platforms offer many advanced features that are not found on physical servers. Which increases the uptime and availability. Although vendor feature names may be different, all provide features like 

    • Live migration
    • Storage migration
    • fault tolerance
    • high availability.
    • distribute resource scheduling.
These features elevated the ability to recover from unplanned outages. 

     4. Ease the cloud migration.

       If there is a plan to move to the cloud, virtualization makes this easy. Because cloud and virtualization use the same technology stack.

Cons of virtualization

  1. The upfront costs are hefty.
  2. Not all the hardware or software can be virtualized
  3. It's easily carried away with adding servers. 

Last decade there is a significant improvement in open source virtualization technology.  Oracle also change the view of the virtualization path by moving from OVM to KVM (Kernel Based Virtual Machine). KVM is actually an open-source virtualization technology that turns Linux into a hypervisorAlso, Oracle came up with a management engine to manage multiple hypervisors called OLVM (Oracle Linux Virtualization Manager) this is a similar product to VMware vcenter.

OLVM Installation and Configuration.

In this article, I will cover OLVM (Oracle Linux Virtualization Manager) installation and other administrative tasks like adding a KVM host to OLVM etc.. This environment I have hosted in a VirtualBox environment. (Nested virtualization). I will explain how to set up the nested virtualized environment in VBox.

I have mentioned the oracle documentation link for OLVM-engine 4.4 installation and configuration.

Installation and Configuration (oracle.com)

Required Packages 

First, we need to enable the oracle Linux 8 repository. We need to enable an internet connection to the KVM server or need to create a sample repository.


dnf install oracle-ovirt-release-el8
Sample output

[root@kvm02 ~]# dnf install oracle-ovirt-release-el8
Last metadata expiration check: 0:02:04 ago on Fri 22 Jul 2022 12:08:59 AM EDT.
Dependencies resolved.
===============================================================================================================================================================================================================
 Package                                                    Architecture                             Version                                         Repository                                           Size
===============================================================================================================================================================================================================
Installing:
 oracle-ovirt-release-el8                                   x86_64                                   1.0-1.0.3.el8                                   ol8_baseos_latest                                    23 k

Transaction Summary
===============================================================================================================================================================================================================
Install  1 Package

Total download size: 23 k
Installed size: 35 k
Is this ok [y/N]: y
Downloading Packages:
oracle-ovirt-release-el8-1.0-1.0.3.el8.x86_64.rpm                                                                                                                              9.0 kB/s |  23 kB     00:02
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                          9.0 kB/s |  23 kB     00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                       1/1
  Running scriptlet: oracle-ovirt-release-el8-1.0-1.0.3.el8.x86_64                                                                                                                                         1/1
  Installing       : oracle-ovirt-release-el8-1.0-1.0.3.el8.x86_64                                                                                                                                         1/1
  Running scriptlet: oracle-ovirt-release-el8-1.0-1.0.3.el8.x86_64                                                                                                                                         1/1
Enabling OLVM 4.4 required Yum Channels...
Enabling ol8_baseos_latest Yum Channel...
Enabling ol8_UEKR6 Yum Channel...
Enabling ol8_gluster_appstream Yum Channel...
Enabling ol8_kvm_appstream Yum Channel...
Disabling yum module virt:ol
Enabling yum module virt:kvm_utils2
Enabling module pki-deps
Enabling module postgresql:13
Required oVirt 4.4 Yum Channels enabled.
Done.

  Verifying        : oracle-ovirt-release-el8-1.0-1.0.3.el8.x86_64                                                                                                                                         1/1

Installed:
  oracle-ovirt-release-el8-1.0-1.0.3.el8.x86_64

Complete!
[root@kvm02 ~]#

List repository

This will enable the new oel 8 repository required for KVM installation.


[root@kvm02 ~]# dnf repolist
repo id                                                                      repo name
ol8_UEKR6                                                                    Latest Unbreakable Enterprise Kernel Release 6 for Oracle Linux 8 (x86_64)
ol8_appstream                                                                Oracle Linux 8 Application Stream (x86_64)
ol8_baseos_latest                                                            Oracle Linux 8 BaseOS Latest (x86_64)
ol8_gluster_appstream                                                        Oracle Linux 8 Gluster Appstream (x86_64)
ol8_kvm_appstream                                                            Oracle Linux 8 KVM Application Stream (x86_64)
ovirt-4.4                                                                    Latest oVirt 4.4 Release (x86_64)
ovirt-4.4-extra                                                              oVirt 4.4 Extra (x86_64)
[root@kvm02 ~]#


Install Engine

Execute the "installed ovirt-engine" mentioned commands to install the engine-related packages.


dnf install ovirt-engine

Configure Engine


Make sure the hostname is configured with FQDN (Full qualified domain name) eg:kvm02.local.com

To configure the engine setup need to execute the engine-setup command. If need to accept all the default values execute engine-setup --accept-defaults to automatically accept all questions that have default answers.

Sample output


[root@kvm02 ~]# engine-setup
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf
          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20220725103208-vp6a4t.log
          Version: otopi-1.9.5 (otopi-1.9.5-1.el8)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup (late)
[ INFO  ] Stage: Environment customization

          --== PRODUCT OPTIONS ==--

          Configure Cinderlib integration (Currently in tech preview) (Yes, No) [No]:
          Configure Engine on this host (Yes, No) [Yes]:

          Configuring ovirt-provider-ovn also sets the Default cluster's default network provider to ovirt-provider-ovn.
          Non-Default clusters may be configured with an OVN after installation.
          Configure ovirt-provider-ovn (Yes, No) [Yes]:
          Configure WebSocket Proxy on this host (Yes, No) [Yes]:

          * Please note * : Data Warehouse is required for the engine.
          If you choose to not configure it on this host, you have to configure
          it on a remote host, and then configure the engine on this host so
          that it can access the database of the remote Data Warehouse host.
          Configure Data Warehouse on this host (Yes, No) [Yes]:
          Configure VM Console Proxy on this host (Yes, No) [Yes]:
          Configure Grafana on this host (Yes, No) [Yes]:

          --== PACKAGES ==--

[ INFO  ] Checking for product updates...
[ INFO  ] DNF Package grafana-postgres available, but not installed.
[ INFO  ] No product updates found

          --== NETWORK CONFIGURATION ==--

          Host fully qualified DNS name of this server [kvm02.local.com]:
[WARNING] Failed to resolve kvm02.local.com using DNS, it can be resolved only locally

          Setup can automatically configure the firewall on this system.
          Note: automatic configuration of the firewall may overwrite current settings.
          Do you want Setup to configure the firewall? (Yes, No) [Yes]:
[ INFO  ] firewalld will be configured as firewall manager.

          --== DATABASE CONFIGURATION ==--

          Where is the DWH database located? (Local, Remote) [Local]:

          Setup can configure the local postgresql server automatically for the DWH to run. This may conflict with existing applications.
          Would you like Setup to automatically configure postgresql and create DWH database, or prefer to perform that manually? (Automatic, Manual) [Automatic]:
          Where is the Engine database located? (Local, Remote) [Local]:

          Setup can configure the local postgresql server automatically for the engine to run. This may conflict with existing applications.
          Would you like Setup to automatically configure postgresql and create Engine database, or prefer to perform that manually? (Automatic, Manual) [Automatic]:

          --== OVIRT ENGINE CONFIGURATION ==--

          Engine admin password:
          Confirm engine admin password:
[WARNING] Password is weak: The password is shorter than 8 characters
          Use weak password? (Yes, No) [No]: Yes
          Application mode (Virt, Gluster, Both) [Both]:
          Use default credentials (admin@internal) for ovirt-provider-ovn (Yes, No) [Yes]:

          --== STORAGE CONFIGURATION ==--

          Default SAN wipe after delete (Yes, No) [No]:

          --== PKI CONFIGURATION ==--

          Organization name for certificate [local.com]:

          --== APACHE CONFIGURATION ==--

          Setup can configure the default page of the web server to present the application home page. This may conflict with existing applications.
          Do you wish to set the application as the default page of the web server? (Yes, No) [Yes]:

          Setup can configure apache to use SSL using a certificate issued from the internal CA.
          Do you wish Setup to configure that, or prefer to perform that manually? (Automatic, Manual) [Automatic]:

          --== SYSTEM CONFIGURATION ==--


          --== MISC CONFIGURATION ==--

          Please choose Data Warehouse sampling scale:
          (1) Basic
          (2) Full
          (1, 2)[1]:
          Use Engine admin password as initial Grafana admin password (Yes, No) [Yes]:

          --== END OF CONFIGURATION ==--

[ INFO  ] Stage: Setup validation
[WARNING] Less than 16384MB of memory is available

          --== CONFIGURATION PREVIEW ==--

          Application mode                        : both
          Default SAN wipe after delete           : False
          Host FQDN                               : kvm02.local.com
          Firewall manager                        : firewalld
          Update Firewall                         : True
          Set up Cinderlib integration            : False
          Configure local Engine database         : True
          Set application as default page         : True
          Configure Apache SSL                    : True
          Engine database host                    : localhost
          Engine database port                    : 5432
          Engine database secured connection      : False
          Engine database host name validation    : False
          Engine database name                    : engine
          Engine database user name               : engine
          Engine installation                     : True
          PKI organization                        : local.com
          Set up ovirt-provider-ovn               : True
          Grafana integration                     : True
          Grafana database user name              : ovirt_engine_history_grafana
          Configure WebSocket Proxy               : True
          DWH installation                        : True
          DWH database host                       : localhost
          DWH database port                       : 5432
          DWH database secured connection         : False
          DWH database host name validation       : False
          DWH database name                       : ovirt_engine_history
          Configure local DWH database            : True
          Configure VMConsole Proxy               : True

          Please confirm installation settings (OK, Cancel) [OK]:
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stopping engine service
[ INFO  ] Stopping ovirt-fence-kdump-listener service
[ INFO  ] Stopping dwh service
[ INFO  ] Stopping vmconsole-proxy service
[ INFO  ] Stopping websocket-proxy service
[ INFO  ] Stage: Misc configuration (early)
[ INFO  ] Stage: Package installation
[ INFO  ] DNF Downloading 4 files, 16083.99KB
[ INFO  ] DNF Downloaded ovirt-vmconsole-1.0.9-3.el8.noarch.rpm
[ INFO  ] DNF Downloaded selinux-policy-3.14.3-80.0.4.el8_5.2.noarch.rpm
[ INFO  ] DNF Downloaded selinux-policy-targeted-3.14.3-80.0.4.el8_5.2.noarch.rpm
[ INFO  ] DNF Downloaded ovirt-vmconsole-proxy-1.0.9-3.el8.noarch.rpm
[ INFO  ] DNF Upgraded: selinux-policy-3.14.3-80.0.4.el8_5.2.noarch
[ INFO  ] DNF Upgraded: selinux-policy-targeted-3.14.3-80.0.4.el8_5.2.noarch
[ INFO  ] DNF Upgraded: ovirt-vmconsole-1.0.9-3.el8.noarch
[ INFO  ] DNF Upgraded: ovirt-vmconsole-proxy-1.0.9-3.el8.noarch
[ INFO  ] DNF Unknown: ovirt-vmconsole-proxy-1.0.9-2.el8.noarch
[ INFO  ] DNF Unknown: ovirt-vmconsole-1.0.9-2.el8.noarch
[ INFO  ] DNF Unknown: selinux-policy-targeted-3.14.3-80.0.1.el8.noarch
[ INFO  ] DNF Unknown: selinux-policy-3.14.3-80.0.1.el8.noarch
[ INFO  ] DNF Verify: selinux-policy-3.14.3-80.0.4.el8_5.2.noarch 1/8
[ INFO  ] DNF Verify: selinux-policy-3.14.3-80.0.1.el8.noarch 2/8
[ INFO  ] DNF Verify: selinux-policy-targeted-3.14.3-80.0.4.el8_5.2.noarch 3/8
[ INFO  ] DNF Verify: selinux-policy-targeted-3.14.3-80.0.1.el8.noarch 4/8
[ INFO  ] DNF Verify: ovirt-vmconsole-1.0.9-3.el8.noarch 5/8
[ INFO  ] DNF Verify: ovirt-vmconsole-1.0.9-2.el8.noarch 6/8
[ INFO  ] DNF Verify: ovirt-vmconsole-proxy-1.0.9-3.el8.noarch 7/8
[ INFO  ] DNF Verify: ovirt-vmconsole-proxy-1.0.9-2.el8.noarch 8/8
[ INFO  ] Stage: Misc configuration
[ INFO  ] Upgrading CA
[ INFO  ] Initializing PostgreSQL
[ INFO  ] Creating PostgreSQL 'engine' database
[ INFO  ] Configuring PostgreSQL
[ INFO  ] Creating PostgreSQL 'ovirt_engine_history' database
[ INFO  ] Configuring PostgreSQL
[ INFO  ] Creating CA: /etc/pki/ovirt-engine/ca.pem
[ INFO  ] Creating CA: /etc/pki/ovirt-engine/qemu-ca.pem
[ INFO  ] Updating OVN SSL configuration
[ INFO  ] Updating OVN timeout configuration
[ INFO  ] Creating/refreshing DWH database schema
[ INFO  ] Setting up ovirt-vmconsole proxy helper PKI artifacts
[ INFO  ] Setting up ovirt-vmconsole SSH PKI artifacts
[ INFO  ] Configuring WebSocket Proxy
[ INFO  ] Creating/refreshing Engine database schema
[ INFO  ] Creating a user for Grafana
[ INFO  ] Creating/refreshing Engine 'internal' domain database schema
[ INFO  ] Creating default mac pool range
[ INFO  ] Adding default OVN provider to database
[ INFO  ] Adding OVN provider secret to database
[ INFO  ] Setting a password for internal user admin
[ INFO  ] Install selinux module /usr/share/ovirt-engine/selinux/ansible-runner-service.cil
[ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
[ INFO  ] Starting engine service
[ INFO  ] Starting dwh service
[ INFO  ] Starting Grafana service
[ INFO  ] Restarting ovirt-vmconsole proxy service

          --== SUMMARY ==--

[ INFO  ] Restarting httpd
          Please use the user 'admin@internal' and password specified in order to login
          Web access is enabled at:
              http://kvm02.local.com:80/ovirt-engine
              https://kvm02.local.com:443/ovirt-engine
          Internal CA CF:C8:A2:E0:42:FE:5F:19:55:B3:E2:9F:A9:7F:4C:DC:49:8D:C7:CB
          SSH fingerprint: SHA256:aRJ1E8zUzNaYsXG2tCfvV4EGTMxw/mMzrKQQt2ZFZKE
[WARNING] Less than 16384MB of memory is available
          Web access for grafana is enabled at:
              https://kvm02.local.com/ovirt-engine-grafana/
          Please run the following command on the engine machine kvm02.local.com, for SSO to work:
          systemctl restart ovirt-engine

          --== END OF SUMMARY ==--

[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20220725103208-vp6a4t.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20220725103526-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ INFO  ] Execution of setup completed successfully
[root@kvm02 ~]#


Config Firewall

Enable the below-mentioned firewall rule to access the olvm url.


firewall-cmd --zone=public --permanent --add-service=https
firewall-cmd --zone=public --permanent --add-service=http
firewall-cmd --reload


Access the URL


OLVM URL can be found in the engine setup summary section, This figure shows the main URL of the virtual manager.


Login


Once you click the administration portal, you can log in to the login page. The username for the administration portal is admin.






Dashboard


This figure shows the main dashboard, the dashboard has the main tab to navigate the data-center, cluster, host, VM (virtual machines), and Storage domains.




Conclusion


Virtualization is the best solution to utilize resources efficiently and reduce the data center footprint. Last decade virtual manager and virtualization technology have improved drastically on open source platforms. It's not easy to manage 100 to 300 KVM separately by login into each URL. OLVM (Oracle Linux virtualization manager) will act as a single point of contact to manage all the DC (data center), KVM, and VM (Virtual Machine) and storage domains. The best practice should be to host this machine out of the KVM server. As many developers working on this OLVM, this will be more mature with many more features. Also, virtualization is the backbone of the cloud platform which enabled the software-driven data center concept. 

Next article I will cover the Storage domain configuration and VM creation.


Unified Auditing Housekeeping

  Intro  Data is the new currency. It is one of the most valuable organizational assets, however, if that data is not well...