Tuesday, April 1, 2025

Protecting Your Encryption Keys: Lessons from the Oracle Cloud Security Breach (OKV - Part 1 )

 








Intro

Recent reports have surfaced about a significant security breach involving Oracle Cloud. A threat actor, identified as "rose87168," claims to have exfiltrated over six million records, potentially affecting more than 140,000 tenants. The compromised data reportedly includes encrypted Single Sign-On (SSO) and Lightweight Directory Access Protocol (LDAP) passwords, Java KeyStores (JKS) files, and Enterprise Manager Java Platform Security (JPS) keys. The attacker is allegedly demanding payments from affected organizations to prevent further exposure.

This incident underscores the critical need for robust encryption key management and security best practices. While Oracle Cloud offers built-in security measures, organizations must proactively safeguard their sensitive credentials and encryption keys. This is where Oracle Key Vault (OKV) plays a crucial role in enhancing data security and mitigating potential threats.

It's not advisable to store your Transparent Data Encryption (TDE) keys locally, as disk corruption could lead to the loss of these critical keys. Managing keys manually across multiple Oracle and MySQL databases with TDE encryption can be challenging and error-prone. That’s why I always encourage customers to back up their keys to an external repository as part of disaster recovery planning. Oracle Key Vault (OKV) serves a crucial role in this context by providing a centralized repository to protect and manage keys. This product is highly mature in terms of security and offers robust protection for sensitive data.

How Oracle Key Vault Can Help

Oracle Key Vault (OKV) is designed to provide centralized and secure storage for encryption keys, certificates, and credentials, reducing the risk of unauthorized access. In light of this breach, organizations leveraging OKV can benefit in the following ways:

  1. Secure Key Management—OKV provides a dedicated environment for securely storing and managing encryption keys, preventing exposure even if other credentials are compromised.

  2. Separation of Duties – By decoupling encryption key storage from application environments, OKV minimizes the risk of key exposure in case of a breach.

  3. End-to-End Encryption – Ensuring that sensitive data, including authentication credentials and cryptographic keys, remains encrypted both in transit and at rest.

  4. Auditing & Compliance – OKV provides robust auditing capabilities, helping organizations track key access and usage to meet regulatory compliance requirements.

  5. Automated Key Rotation – Regular key rotation can help mitigate risks associated with long-term key exposure, making it harder for attackers to exploit stolen credentials.


In this blog, I will walk you through the installation of the Oracle Key Vault (OKV) appliance in an on-premises environment. For this demonstration, I’ll be using VirtualBox to set up and test the deployment.

Initial Step 

The first step is to download the Oracle Key Vault (OKV) ISO image from Oracle e-Delivery.

  1. Visit Oracle e-Delivery.

  2. In the search bar, type Oracle Key Vault and locate the latest version (21.10).

  3. Download the ISO file for installation.




                                                         Figure 1 : Download OKV


Create a VM using the mentioned system requirements.

  • CPU: Minimum: x86-64 16 cores. Recommended: 24-48 cores with cryptographic acceleration support (Intel AESNI).
  • Memory: Minimum 16 GB of RAM. Recommended: 32–64 GB.
  • Disk: Both BIOS and UEFI boot mode. For a system with a boot disk size greater than 2 TB, Oracle Key Vault supports booting in UEFI mode only.

Installation 

After downloading the OKV ISO, boot your system from the ISO image. Once the installation menu appears, simply press Enter to start a fresh installation of Oracle Key Vault.

                                             
                                                    Figure 2: Installation   


Note: The installation will fail if the recommended values are not provided. For example, insufficient disk space can cause the installation to fail, as shown below.



                                                 Figure 3: Installation failure.

After pressing Enter, the installation process will begin by creating the Logical Volume Manager (LVM) and then proceed with the database installation.


                                            Figure 4 : Installation: LVM creation


During the installation, you will be prompted to provide the ISO file again. Reattach the ISO to proceed with the database installation and Oracle REST Data Services (ORDS), which powers the web-based interface built on APEX.



                                                       Figure 5: ISO file prompt 01 


                      

                                                    Figure 6 : ISO file prompt 02 


Setup Network


The next step in the installation process is configuring the network for Oracle Key Vault (OKV). Since this is a test setup, I am selecting Classic Mode for the network configuration.

  
                                                         Figure 7 : Setup network

Choose the appropriate network interface and configure the IP address for the Oracle Key Vault (OKV) appliance.


                                                      Figure 8 : Setup network - 2


                                                     Figure 9 : Setup network - 3


After applying the Database Release Update (RU), the installation proceeds with deploying the Oracle Key Vault (OKV) application within Oracle REST Data Services (ORDS).



                                                    Figure 9: Apply DB RU



                                                       Figure 10: Beginning application installation.

This figure illustrates the final step of the application installation process.


                                              Figure 10: Completing the application installation.

Now we can access take OKV URL using IP or the hostname. 



                                                           Figure 11 : OKV Loging page.

Set up roles and users for OKV.

There are 3 different roles in OKV. 
  • Admin

  • System Administrator

  • Audit Manager

Alternatively, you can use the same account for all these roles. For this configuration, I will use a single account.



                                                    Figure 12: Setup users.

Once everything is configured, this is how the dashboard will appear when you log in to OKV.



                                              Figure 13 : Dashboard view.


Next article I will show you how you can migrate TDE keys to OKV. 

Conclusion

In an era where AI-driven security threats are becoming increasingly sophisticated, robust encryption key management is more critical than ever. Oracle Key Vault (OKV) provides a secure, centralized solution for managing encryption keys, reducing the risks associated with manual handling and local storage. By integrating OKV into their security strategy, organizations can enhance data protection, ensure compliance, and mitigate potential threats. As cybersecurity challenges evolve, proactive key management remains a fundamental pillar of a strong defense strategy.

Friday, March 21, 2025

Ensuring Database Integrity During Migration: A Production DBA's Challenge with Datapatch Execution (19c PDB : datapatch issue :ORA-01110 )

 





Intro

In today’s fast-paced data-driven world, data is more valuable than gold, and protecting databases has become a critical priority. As a production DBA, one of my key responsibilities is ensuring that the software stays up to date with the latest patches, as Oracle releases these patches every quarter.

Recently, I faced a challenge after migrating a database from ODA to Exadata, upgrading from Oracle version 19.13 to 19.24. During the process, the datapatch execution failed on a pluggable database (PDB) due to a missing PDB temporary file.

Before running datapatch, I performed a sanity check to confirm that everything was in good condition, and the results were clean.

SANITY CHECK

Sanity check results : 



[oracle@exanode01 OPatch]$ ./datapatch -sanity_checks
SQL Patching sanity checks version 19.24.0.0.0 on Wed 23 Oct 2024 04:20:17 PM UTC
Copyright (c) 2021, 2024, Oracle.  All rights reserved.

Log file for this invocation: /u02/app/oracle/cfgtoollogs/sqlpatch/sanity_checks_20241023_162017_119323/sanity_checks_20241023_162017_119323.log

Running checks
Use of uninitialized value in concatenation (.) or string at /u02/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 645.
Use of uninitialized value in concatenation (.) or string at /u02/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 645.
Use of uninitialized value in concatenation (.) or string at /u02/app/oracle/product/19.0.0.0/dbhome_1/sqlpatch/sqlpatch_sanity_checks.pm line 645.

JSON report generated in /u02/app/oracle/cfgtoollogs/sqlpatch/sanity_checks_20241023_162017_119323/sqlpatch_sanity_checks_summary.json file
Checks completed. Printing report:

Check: Database component status - OK
Check: PDB Violations - OK
Check: Invalid System Objects - OK
Check: Tablespace Status - OK
Check: Backup jobs - OK
Check: Temp file exists - OK
Check: Temp file online - OK
Check: Data Pump running - OK
Check: Container status - OK
Check: Oracle Database Keystore - OK
Check: Dictionary statistics gathering - WARNING
  Patching the database while dictionary statistics are being collected may lead to performance issues.
  Dictionary statistics are currently being collected in the database.
  Consider patching the database when dictionary statistics are not being collected. To check if such operations are currently running, run this query:
    SELECT sofar, totalwork, units, start_time, time_remaining, message
    FROM sys.v$session_longops
    WHERE opname = 'Gather Dictionary Schema Statistics'
    AND time_remaining > 0;
  PDB$SEED:
    |     LATEST      |        OPERATION        |
    |-----------------+-------------------------|
    | 17-JUL-22 09:16 | gather_dictionary_stats |
    |-----------------+-------------------------|
Check: Scheduled Jobs - WARNING
  Execution of scheduler jobs while database patching is running may lead to failures and/or performance issues.
  There are jobs currently running or scheduled to be executed during next hour.
  Consider patching the database when jobs are not running and will not be scheduled to run during patching.
  To check for jobs that are running or scheduled to run:
    SELECT owner as schema_name, job_name, state, next_run_date
    FROM sys.all_scheduler_jobs
    WHERE state = 'RUNNING'
    UNION
      SELECT owner as schema_name, job_name, state, next_run_date
      FROM sys.all_scheduler_jobs
      WHERE state = 'SCHEDULED'
      and cast(next_run_date as date)  sysdate
      and cast(next_run_date as date)  sysdate + 1/24;
  CDB$ROOT:
    |         JOB_NAME         |            NEXT_RUN_DATE            | SCHEMA_NAME |   STATE   |
    |--------------------------+-------------------------------------+-------------+-----------|
    | CLEANUP_ONLINE_IND_BUILD | 23-OCT-24 04.48.21.428584 PM +00:00 |     SYS     | SCHEDULED |
    |--------------------------+-------------------------------------+-------------+-----------|
    |    CLEANUP_ONLINE_PMO    | 23-OCT-24 04.49.01.459102 PM +00:00 |     SYS     | SCHEDULED |
    |--------------------------+-------------------------------------+-------------+-----------|
    |   CLEANUP_TAB_IOT_PMO    | 23-OCT-24 04.48.31.422134 PM +00:00 |     SYS     | SCHEDULED |
    |--------------------------+-------------------------------------+-------------+-----------|
  EGDEV_PDB1:
    |         JOB_NAME         |            NEXT_RUN_DATE            | SCHEMA_NAME |   STATE   |
    |--------------------------+-------------------------------------+-------------+-----------|
    | CLEANUP_ONLINE_IND_BUILD | 23-OCT-24 05.11.03.390152 PM +00:00 |     SYS     | SCHEDULED |
    |--------------------------+-------------------------------------+-------------+-----------|
    |    CLEANUP_ONLINE_PMO    | 23-OCT-24 05.11.43.403227 PM +00:00 |     SYS     | SCHEDULED |
    |--------------------------+-------------------------------------+-------------+-----------|
    |   CLEANUP_TAB_IOT_PMO    | 23-OCT-24 05.11.13.329389 PM +00:00 |     SYS     | SCHEDULED |
    |--------------------------+-------------------------------------+-------------+-----------|
  EGQA_PDB1:
    |         JOB_NAME         |            NEXT_RUN_DATE            | SCHEMA_NAME |   STATE   |
    |--------------------------+-------------------------------------+-------------+-----------|
    | CLEANUP_ONLINE_IND_BUILD | 23-OCT-24 05.11.03.441575 PM +00:00 |     SYS     | SCHEDULED |
    |--------------------------+-------------------------------------+-------------+-----------|
    |    CLEANUP_ONLINE_PMO    | 23-OCT-24 05.11.43.187682 PM +00:00 |     SYS     | SCHEDULED |
    |--------------------------+-------------------------------------+-------------+-----------|
    |   CLEANUP_TAB_IOT_PMO    | 23-OCT-24 05.11.13.443853 PM +00:00 |     SYS     | SCHEDULED |
    |--------------------------+-------------------------------------+-------------+-----------|
Check: GoldenGate triggers - OK
Check: Logminer DDL triggers - OK
Check: Check sys public grants - OK
Check: Statistics gathering running - OK
Check: Optim dictionary upgrade parameter - OK
Check: Symlinks on oracle home path - OK
Check: Central Inventory - OK
Check: Queryable Inventory dba directories - OK
Check: Queryable Inventory locks - OK
Check: Queryable Inventory package - OK
Check: Queryable Inventory external table - OK
Check: Imperva processes - OK
Check: Guardium processes - OK
Check: Locale - OK

Refer to MOS Note 2680521.1 and debug log
/u02/app/oracle/cfgtoollogs/sqlpatch/sanity_checks_20241023_162017_119323/sanity_checks_debug_20241023_162017_119323.log

SQL Patching sanity checks completed on Wed 23 Oct 2024 04:21:18 PM UTC
[oracle@exanode01  OPatch]$


Everything looked clean, so I executed datapatch -version to update the registry information. However, it was hanging on for a long time. Upon checking the database alert log, I noticed complaints about the temp files under PDB$SEED.

I decided to kill the session since the temp files hadn't been moved. Now, the issue is that PDB$SEED has moved to the MOUNT state.


Datapatch output :



[oracle@exanode01 OPatch]$ pwd/u02/app/oracle/product/19.0.0.0/dbhome_2/OPatch

[oracle@exanode01 OPatch]$ ./datapatch -verboseSQL Patching tool version 19.24.0.0.0 Production on Tue Dec 10 22:15:39 2024
Copyright (c) 2012, 2024, Oracle.  All rights reserved.

Log file for this invocation: /u02/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_380836_2024_12_10_22_15_39/sqlpatch_invocation.log

Connecting to database...OK
Gathering database info...Unable to acquire sqlpatch global lock in EXCLUSIVE mode
because another datapatch session is currently running.
Waiting for that session to complete before continuing...



^CError: prereq checks failed!
DBD::Oracle::st execute failed: ORA-01013: user requested cancel of current operation
ORA-06512: at "SYS.DBMS_LOCK", line 92
ORA-06512: at "SYS.DBMS_LOCK", line 108
ORA-06512: at line 9 (DBD ERROR: OCIStmtExecute) [for Statement "DECLARE
           lockmode INTEGER;
         BEGIN
           IF ? = 'SHARED' THEN
             lockmode := dbms_lock.s_mode;
           ELSE
             lockmode := dbms_lock.x_mode;
           END IF;
           ? := dbms_lock.request(?, lockmode, dbms_lock.maxwait, false);
         END;" with ParamValues: :p1='EXCLUSIVE', :p2='1', :p3='10737418441073741844167'] at /u02/app/oracle/product/19.0.0.0/dbhome_2/sqlpatch/sqlpatch.pm line 2134.


Please refer to MOS Note 1609718.1 and/or the invocation log
/u02/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_380836_2024_12_10_22_15_39/sqlpatch_invocation.log
for information on how to resolve the above errors.

SQL Patching tool complete on Tue Dec 10 22:26:39 2024

We migrated this database using the RMAN backup and restore method. However, RMAN does not create temp files during the backup and restore process. As a result, PDB$SEED temp files were missing, which prevented the database from completing its operations in PDB$SEED.


Alert log error :



PDB$SEED(2):Pluggable database PDB$SEED closing
PDB$SEED(2):Increasing priority of 2 RS
PDB$SEED(2):JIT: pid 141609 requesting stop
PDB$SEED(2):Closing sequence subsystem (601820723513).
PDB$SEED(2):Buffer Cache flush started: 2
PDB$SEED(2):Buffer Cache flush finished: 2
2024-10-23T16:25:13.153882+00:00
Errors in file /u02/app/oracle/diag/rdbms/egdev_exa/DEV1/trace/DEV1_dbw0_59940.trc:
ORA-01157: cannot identify/lock data file 1026 - see DBWR trace file
ORA-01110: data file 1026: '/u02/app/oracle/oradata/DEV/DEV/datafile/temp012024-05-06_05-30-49-610-AM.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
2024-10-23T16:25:13.154139+00:00
Errors in file /u02/app/oracle/diag/rdbms/dev_exa/DEV1/trace/DEV1_dbw0_59940.trc:
ORA-01186: file 1026 failed verification tests
ORA-01157: cannot identify/lock data file 1026 - see DBWR trace file
ORA-01110: data file 1026: '/u02/app/oracle/oradata/DEV/DEV/datafile/temp012024-05-06_05-30-49-610-AM.dbf'
2024-10-23T16:25:13.154193+00:00
File 1026 not verified due to error ORA-01157

Error from datapath log :



[2024-10-23 16:25:12] sql_error_handler called: [2024-10-23 16:25:12] 1 ADD_TO_QUEUE PDB$SEED: DBD::Oracle::st bind_param failed: ORA-01157: cannot identify/lock data file 1026 - see DBWR trace file
ORA-01110: data file 1026: '/u02/app/oracle/oradata/EGDEV/EGDEV/datafile/temp012024-05-06_05-30-49-610-AM.dbf' (DBD ERROR: OCILobCreateTemporary) [for Statement "DECLARE
             rec dba_registry_sqlpatch%ROWTYPE;
           BEGIN
             rec.patch_id := :patch_id;
             rec.patch_uid := :patch_uid;
             rec.patch_type := :patch_type;
             rec.action := :action;
             rec.description := SUBSTR(:description, '1', '100');
             rec.flags := :flags;
             rec.patch_descriptor := :patch_descriptor;
             rec.patch_directory := :patch_directory;
             rec.source_version := :source_version;
             rec.source_build_description := :source_build_description;
             rec.source_build_timestamp :=
               TO_TIMESTAMP(:source_build_timestamp, 'YYMMDDHH24MISS');
             rec.target_version := :target_version;
             rec.target_build_description := :target_build_description;
             rec.target_build_timestamp :=
               TO_TIMESTAMP(:target_build_timestamp, 'YYMMDDHH24MISS');
             sys.dbms_sqlpatch.set_patch_metadata(rec);


Resolving PDB$SEED in MOUNT State to Create Tempfiles.

After encountering an issue where PDB$SEED moved to the MOUNT state, preventing access to create the necessary tempfiles, I applied the following method to regain access to PDB$SEED and proceed with tempfile creation.


SQL> alter session set container=PDB$SEED;
SQL> alter session set "_oracle_script"=TRUE;
SQL> alter pluggable database pdb$seed close immediate instances=all;
SQL> alter pluggable database pdb$seed OPEN READ ONLY;



Recreating Tempfiles in PDB$SEED

To resolve the issue with missing temp files in PDB$SEED, follow these steps:

  1. Create New Tempfiles: Start by creating new tempfiles to ensure that the database functions as expected.
  2. Handle Old Tempfiles: For the old tempfiles that were not created, you can take them offline and drop them once they're no longer needed.

This approach ensures the smooth creation of temp files and the proper cleanup of any old or unnecessary ones.



SQL> alter database tempfile '/u02/app/oracle/oradata/DEV/DEV/datafile/temp012024-05-06_05-30-49-610-AM.dbf' offline;

Database altered.

SQL>  alter database tempfile '/u02/app/oracle/oradata/DEV/DEV/datafile/temp012024-05-06_05-30-49-610-AM.dbf' drop including datafiles;

Database altered.

SQL>


Once all the necessary changes have been made and everything is in order, you can proceed to rerun the datapatch -version command to confirm that the patching process has been completed successfully.


[oracle@exadevdb02-tsgb71 OPatch]$ ./datapatch -verbose
SQL Patching tool version 19.24.0.0.0 Production on Wed Oct 23 16:53:31 2024
Copyright (c) 2012, 2024, Oracle.  All rights reserved.

Log file for this invocation: /u02/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_258820_2024_10_23_16_53_31/sqlpatch_invocation.log

Connecting to database...OK
Gathering database info...

done

Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)

Bootstrapping registry and package to current versions...done

Determining current state...done

Current state of interim SQL patches:
Interim patch 34086870 (OJVM RELEASE UPDATE: 19.16.0.0.220719 (34086870)):
  Binary registry: Not installed
  PDB CDB$ROOT: Applied successfully on 17-JUL-22 09.18.13.470863 AM
  PDB DEV_PDB1: Applied successfully on 17-JUL-22 09.18.24.040293 AM
  PDB QA_PDB1: Applied successfully on 17-JUL-22 09.18.24.040293 AM
  PDB PDB$SEED: Applied successfully on 17-JUL-22 09.18.24.040293 AM
Interim patch 36414915 (OJVM RELEASE UPDATE: 19.24.0.0.240716 (36414915)):
  Binary registry: Installed
  PDB CDB$ROOT: Not installed
  PDB DEV_PDB1: Not installed
  PDB QA_PDB1: Not installed
  PDB PDB$SEED: Not installed

Current state of release update SQL patches:
  Binary registry:
    19.24.0.0.0 Release_Update 240627235157: Installed
  PDB CDB$ROOT:
    Applied 19.16.0.0.0 Release_Update 220703022223 successfully on 17-JUL-22 09.18.13.460370 AM
  PDB DEV_PDB1:
    Applied 19.16.0.0.0 Release_Update 220703022223 successfully on 17-JUL-22 09.18.24.036238 AM
  PDB QA_PDB1:
    Applied 19.16.0.0.0 Release_Update 220703022223 successfully on 17-JUL-22 09.18.24.036238 AM
  PDB PDB$SEED:
    Applied 19.16.0.0.0 Release_Update 220703022223 successfully on 17-JUL-22 09.18.24.036238 AM

Adding patches to installation queue and performing prereq checks...done
Installation queue:
  For the following PDBs: CDB$ROOT PDB$SEED DEV_PDB1 QA_PDB1
    The following interim patches will be rolled back:
      34086870 (OJVM RELEASE UPDATE: 19.16.0.0.220719 (34086870))
    Patch 36582781 (Database Release Update : 19.24.0.0.240716 (36582781)):
      Apply from 19.16.0.0.0 Release_Update 220703022223 to 19.24.0.0.0 Release_Update 240627235157
    The following interim patches will be applied:
      36414915 (OJVM RELEASE UPDATE: 19.24.0.0.240716 (36414915))

Installing patches...

Patch installation complete.  Total patches installed: 12

Validating logfiles...done
Patch 34086870 rollback (pdb CDB$ROOT): SUCCESS
  logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/34086870/24803071/34086870_rollback_DEV_CDBROOT_2024Oct23_16_54_23.log (no errors)
Patch 36582781 apply (pdb CDB$ROOT): SUCCESS
  logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/36582781/25751445/36582781_apply_DEV_CDBROOT_2024Oct23_16_54_24.log (no errors)
Patch 36414915 apply (pdb CDB$ROOT): SUCCESS
  logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/36414915/25700138/36414915_apply_DEV_CDBROOT_2024Oct23_16_54_24.log (no errors)
Patch 34086870 rollback (pdb PDB$SEED): SUCCESS
  logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/34086870/24803071/34086870_rollback_DEV_PDBSEED_2024Oct23_16_58_32.log (no errors)
Patch 36582781 apply (pdb PDB$SEED): SUCCESS
  logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/36582781/25751445/36582781_apply_DEV_PDBSEED_2024Oct23_16_58_59.log (no errors)
Patch 36414915 apply (pdb PDB$SEED): SUCCESS
  logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/36414915/25700138/36414915_apply_DEV_PDBSEED_2024Oct23_16_58_59.log (no errors)
Patch 34086870 rollback (pdb DEV_PDB1): SUCCESS
  logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/34086870/24803071/34086870_rollback_DEV_DEV_PDB1_2024Oct23_16_58_32.log (no errors)
Patch 36582781 apply (pdb DEV_PDB1): SUCCESS
  logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/36582781/25751445/36582781_apply_DEV_DEV_PDB1_2024Oct23_16_58_34.log (no errors)
Patch 36414915 apply (pdb DEV_PDB1): SUCCESS
  logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/36414915/25700138/36414915_apply_DEV_DEV_PDB1_2024Oct23_16_58_33.log (no errors)
Patch 34086870 rollback (pdb QA_PDB1): SUCCESS
  logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/34086870/24803071/34086870_rollback_DEV_QA_PDB1_2024Oct23_16_58_31.log (no errors)
Patch 36582781 apply (pdb QA_PDB1): SUCCESS
  logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/36582781/25751445/36582781_apply_DEV_QA_PDB1_2024Oct23_16_58_33.log (no errors)
Patch 36414915 apply (pdb QA_PDB1): SUCCESS
  logfile: /u02/app/oracle/cfgtoollogs/sqlpatch/36414915/25700138/36414915_apply_DEV_QA_PDB1_2024Oct23_16_58_33.log (no errors)
SQL Patching tool complete on Wed Oct 23 17:04:31 2024

Conclusion

In conclusion, migrating a database and applying patches is a complex process that requires careful attention to detail. Although the sanity check showed no immediate issues, the missing PDB temporary file caused a failure during the datapath execution. This experience reinforced the importance of verifying all database components especially temp files before patching. By addressing the missing temp files and following the proper steps to recreate and clean them, I was able to resolve the issue and successfully complete the patching process. As a production DBA, these challenges are part of the job, and being proactive and methodical ensures smooth database operations and minimal downtime.

Friday, January 31, 2025

EXACS: Creating an 11g Database Home

 



Intro

Oracle Exadata Cloud Service (ExaCS) historically supported Oracle Database 11g, but with recent updates, Oracle has phased out support for older database versions like 11g in ExaCS. While existing Oracle Database 11g instances may continue to run, it is no longer possible to provision new Oracle 11g databases on Exadata Cloud Service through the Oracle Cloud Infrastructure (OCI) console. Oracle encourages users to upgrade to more recent supported versions, such as Oracle Database 19c or 23ai, to leverage enhanced features, security, and performance improvements.

Some customers continue to run Oracle 11g because they have not been able to schedule downtime due to mission-critical business applications. Additionally, Oracle 11.2.0.4 R2 is considered one of the most stable environments that Oracle has built. However, these customers face challenges in finding an efficient migration method, as their databases have grown to terabyte sizes.

Customers are still stuck on Oracle 11g for several key reasons:

Mission-Critical Applications: Many businesses run critical applications that require constant uptime, making it difficult to schedule downtime for upgrades or migrations.

Stable Environment: Oracle 11.2.0.4 R2 is considered a highly stable release. Some businesses prefer to stick with this known stability rather than risk issues with newer versions.

Cost and Complexity: Upgrading Oracle databases, especially from 11g, can be expensive and complex. It may involve re-architecting applications, testing, and retraining staff, which some companies may not have the resources to handle.

Large Databases: Over time, databases have grown to terabyte sizes, which makes migration and upgrade processes much more challenging in terms of time, performance, and storage.

Custom Solutions and Legacy Code: Many organizations have customized solutions that are tightly integrated with Oracle 11g, and updating these systems can require significant rework, leading to compatibility issues with newer versions.

Business Prioritization: In some cases, businesses prioritize operational stability over modernization, choosing to delay upgrades to focus on other initiatives.

In this article, I will show you how you can create the 11g database in exacs using dbaascli.

Note: You can create this only if you have upgrade support for 11g. Oracle ExaCS does not support running 11g in the ExaCS environment.

List db images in EXACS



[root@exadevdb-01 ~]# dbaascli cswlib showImages
DBAAS CLI version 24.3.1.0.0
Executing command cswlib showImages
Job id: 10706bde-2e70-42fe-a371-7cb9bd572f7c
Session log: /var/opt/oracle/log/cswLib/showImages/dbaastools_2024-09-06_05-54-57-PM_59941.log
Log file location: /var/opt/oracle/log/cswLib/showImages/dbaastools_2024-09-06_05-54-57-PM_59941.log

############ List of Available database Artifacts  #############

1.IMAGE_TAG=11.2.0.4.230418
  VERSION=11.2.0.4.230418
  DESCRIPTION=11.2 APR 2023 DB Image
2.IMAGE_TAG=12.2.0.1.230718
  VERSION=12.2.0.1.230718
  DESCRIPTION=12.2 JUL 2023 DB Image
3.IMAGE_TAG=18.23.0.0.0
  VERSION=18.23.0.0.0
  DESCRIPTION=18c JUL 2023 DB Image
4.IMAGE_TAG=19.22.0.0.0
  VERSION=19.22.0.0.0
  DESCRIPTION=19c JAN 2024 DB Image
5.IMAGE_TAG=12.1.0.2.231017
  VERSION=12.1.0.2.231017
  DESCRIPTION=12.1 OCT 2023 DB Image
6.IMAGE_TAG=23.4.0.24.05
  VERSION=23.4.0.24.05
  DESCRIPTION=23ai DB image 23.4.0.24.05
7.IMAGE_TAG=18.22.0.0.0
  VERSION=18.22.0.0.0
  DESCRIPTION=18c APR 2023 DB Image
8.IMAGE_TAG=11.2.0.4.231017
  VERSION=11.2.0.4.231017
  DESCRIPTION=11.2 OCT 2023 DB Image
9.IMAGE_TAG=12.1.0.2.230418
  VERSION=12.1.0.2.230418
  DESCRIPTION=12.1 APR 2023 DB Image
10.IMAGE_TAG=19.23.0.0.0
  VERSION=19.23.0.0.0
  DESCRIPTION=19c APR 2024 DB Image
11.IMAGE_TAG=19.20.0.0.0
  VERSION=19.20.0.0.0
  DESCRIPTION=19c JUL 2023 DB Image
12.IMAGE_TAG=23.5.0.24.07
  VERSION=23.5.0.24.07
  DESCRIPTION=23ai DB image 23.5.0.24.07
13.IMAGE_TAG=12.2.0.1.230418
  VERSION=12.2.0.1.230418
  DESCRIPTION=12.2 APR 2023 DB Image
14.IMAGE_TAG=11.2.0.4.230718
  VERSION=11.2.0.4.230718
  DESCRIPTION=11.2 JUL 2023 DB Image
15.IMAGE_TAG=12.1.0.2.230718
  VERSION=12.1.0.2.230718
  DESCRIPTION=12.1 JUL 2023 DB Image
16.IMAGE_TAG=19.24.0.0.0
  VERSION=19.24.0.0.0
  DESCRIPTION=19c JUL 2024 DB Image
17.IMAGE_TAG=12.2.0.1.231017
  VERSION=12.2.0.1.231017
  DESCRIPTION=12.2 OCT 2023 DB Image
18.IMAGE_TAG=18.24.0.0.0
  VERSION=18.24.0.0.0
  DESCRIPTION=18c OCT 2023 DB Image
19.IMAGE_TAG=19.21.0.0.0
  VERSION=19.21.0.0.0
  DESCRIPTION=19c OCT 2023 DB Image
Images can be downloaded using their image tags. For details, see help using 'dbaascli cswlib download --help'.

dbaascli execution completed
[root@exadevdb-01 ~]#



Download the the 11204 Binary OCT2023 version.


dbaascli cswlib download --version 11204 --bp OCT2023 --bp_update no
Sample output


[root@exadevdb-01 ~]# dbaascli cswlib download --version 11204 --bp OCT2023 --bp_update no
DBAAS CLI version 24.3.2.0.0
Executing command cswlib download --version 11204 --bp OCT2023 --bp_update no
Job id: 8dab6933-9505-40a9-a398-c1ba46eae6e3
Session log: /var/opt/oracle/log/cswLib/download/dbaastools_2024-09-19_02-00-06-PM_166429.log
Loading PILOT...
Session ID of the current execution is: 40
Log file location: /var/opt/oracle/log/cswLib/download/pilot_2024-09-19_02-00-10-PM_166741
-----------------
Running initialize job
Completed initialize job
-----------------
Running validate_url_reachability job
Completed validate_url_reachability job
-----------------
Running validate_image_tag_existence job
Completed validate_image_tag_existence job
-----------------
Running validate_free_space job
Completed validate_free_space job
-----------------
Running validate_file_permissions job
Completed validate_file_permissions job
-----------------
Running download_image job
Image location=/var/opt/oracle/dbaas_acfs/dbnid/11.2.0.4.231017
Download succeeded
Completed download_image job
-----------------
Running decrypt job
Completed decrypt job
-----------------
Running post_download_update job
Completed post_download_update job
-----------------
Running verify_signature job
Skipping verifySignature check because the catalog contains signed property as false
Completed verify_signature job
-----------------
Running validate_sha256sum job
Skipping Sha256Sum check because the catalog does not contain sha256sum
Skipping Sha256Sum check because the catalog does not contain sha256sum
Completed validate_sha256sum job
-----------------
Running update_file_permissions job
Completed update_file_permissions job
{"11.2.0.4.231017":{"file_list":[{"file":"/var/opt/oracle/dbaas_acfs/dbnid/11.2.0.4.231017/db11204_bits_EXA.zip"},{"file":"/var/opt/oracle/dbaas_acfs/dbnid/11.2.0.4.231017/exadbf_11204.tar.gz"}]}}

dbaascli execution completed
[root@exadevdb-01 ~]#


List downloaded db images

After downloading the image you can list local database binary images using cswLib listlocal.


[opc@exadevdb-01 ~]$ sudo dbaascli cswLib listLocal
DBAAS CLI version 24.3.1.0.0
Executing command cswLib listLocal
Job id: 0405333f-04d5-4043-97f6-0fa0a9eca51b
Session log: /var/opt/oracle/log/cswLib/listLocal/dbaastools_2024-09-06_06-15-17-PM_154824.log
Log file location: /var/opt/oracle/log/cswLib/listLocal/dbaastools_2024-09-06_06-15-17-PM_154824.log

############ List of Available Database Images  #############

1.IMAGE_TAG=11.2.0.4.231017
  IMAGE_SIZE=3GB
  VERSION=11.2.0.4.231017
  DESCRIPTION=11.2 OCT 2023 DB Image
2.IMAGE_TAG=19.24.0.0.0
  IMAGE_SIZE=5GB
  VERSION=19.24.0.0.0
  DESCRIPTION=19c JUL 2024 DB Image

dbaascli execution completed
[opc@exadevdb-01 ~]$

Create a new database home using 11g binary

The next step is to create a database home 

[root@exadevdb-01 ~]# dbaascli dbhome create --version 11.2.0.4
DBAAS CLI version 24.3.2.0.0
Executing command dbhome create --version 11.2.0.4
Job id: 615dc2e8-af82-47c1-8612-526082b6e9bf
Session log: /var/opt/oracle/log/dbHome/create/dbaastools_2024-10-10_07-30-24-PM_99121.log
Loading PILOT...
Session ID of the current execution is: 34
Log file location: /var/opt/oracle/log/dbHome/create/pilot_2024-10-10_07-30-27-PM_99403
-----------------
Running Plugin_initialization job
Acquiring native write lock: _u02_app_oracle_product_11.2.0_dbhome_1
Acquiring native write lock: OraHome2
Completed Plugin_initialization job
-----------------
Running OH_image_validate job
Completed OH_image_validate job
-----------------
Running DB_home_version_check job
Completed DB_home_version_check job
-----------------
Running DB_home_environment_version_check job
Completed DB_home_environment_version_check job
-----------------
Running GI_version_check job
Completed GI_version_check job
-----------------
Running Validate_backup_location job
Completed Validate_backup_location job
-----------------
Running Cluster_nodes_check job
Completed Cluster_nodes_check job
-----------------
Running Validate_users_group job
Completed Validate_users_group job
-----------------
Running OH_image_download job
Completed OH_image_download job
-----------------
Running OH_image_version_check job
Completed OH_image_version_check job
-----------------
Running Disk_space_check job
Completed Disk_space_check job
Acquiring write lock: provisioning
-----------------
Running Pre_OH_creation_lock_manager job
Completed Pre_OH_creation_lock_manager job
-----------------
Running OH_pre_existence_check job
Completed OH_pre_existence_check job
-----------------
Running Local_node_oh_image_unzip job
Completed Local_node_oh_image_unzip job
-----------------
Running OH_creation_cvu_prechecks job
Completed OH_creation_cvu_prechecks job
-----------------
Running Local_node_oh_clone job

Completed Local_node_oh_clone job
-----------------
Running Update_dbnid_bits job
Skipping. Job is detected as not applicable.
-----------------
Running Local_node_oh_backup_creation job
Completed Local_node_oh_backup_creation job
-----------------
Running Remote_nodes_software_copy job

Completed Remote_nodes_software_copy job
-----------------
Running Remote_nodes_attach_home job
Completed Remote_nodes_attach_home job
-----------------
Running Remote_nodes_post_install_steps job
Skipping. Job is detected as not applicable.
-----------------
Running Inventory_nodes_list_update job
Completed Inventory_nodes_list_update job
-----------------
Running Root_script_run job
Completed Root_script_run job
-----------------
Running Post_OH_creation_lock_manager job
Completed Post_OH_creation_lock_manager job
Releasing lock: provisioning
-----------------
Running release_lock job
Releasing native lock: OraHome2
Releasing native lock: _u02_app_oracle_product_11.2.0_dbhome_1
Completed release_lock job
-----------------
Running Generate_dbhome_metadata job
Completed Generate_dbhome_metadata job
-----------------
Running Generate_dbhome_system_details job
Acquiring native write lock: global_dbsystem_details_generation
Releasing native lock: global_dbsystem_details_generation
Completed Generate_dbhome_system_details job
-----------------
Running Plugin_cleanup job

Completed Plugin_cleanup job
---------- START OF PLUGIN RESULT ----------
{"ORACLE_HOME_NAME":"OraHome2","ORACLE_HOME":"/u02/app/oracle/product/11.2.0/dbhome_1"}
---------- END OF PLUGIN RESULT ----------

dbaascli execution completed

Conclusion


While Oracle Exadata Cloud Service (ExaCS) no longer officially supports provisioning new Oracle Database 11g instances through the OCI console, it is still possible to create and manage these databases using tools like dbaascli. This approach provides a practical solution for customers who rely on the stability and performance of Oracle 11g for mission-critical applications but face challenges in scheduling downtime or migrating to newer database versions. 

However, it’s essential to recognize the limitations of continuing to use an unsupported version, such as missing out on critical security updates, performance enhancements, and new features offered by modern versions like Oracle Database 19c or 23c AI. Organizations are encouraged to develop a long-term migration strategy to align with Oracle's support roadmap and ensure the sustainability of their database operations.

EXACS: Resolving PDB Console Issue

 





Intro 


The roles of DBAs and architects are continually evolving, becoming more technical and requiring broader expertise. For Oracle DBAs, it's no longer just about managing databases; there's a growing need to understand and work with the OCI (Oracle Cloud Infrastructure) GUI as well. This is because many OCI services depend on the database being properly visible in the OCI console to function correctly.

Recently, I faced an issue after migrating a database from on-premises to the cloud in an Exadata Cloud@Customer (ExaCS) deployment. The pluggable database (PDB) did not appear in the OCI console.

Steps I followed the below-mentioned steps to migration.

  • Created a dummy database directly from the OCI console.
  • Deleted this dummy database at the backend.
  • Used RMAN to perform a duplicate operation.

  • After the RMAN duplicate, the PDB (pluggable database) was correctly visible under the CDB (container database). However, the status in the OCI console is still displayed as "failed."

    This highlights the importance of understanding the OCI console and ensuring proper configuration post-migration to avoid operational disruptions.


    As per Figure 1, the OCI console shows pdb state from console shows as "failed".

                                                        Figure 1 : PDB state from OCI console.

    Troubleshooting steps 


    If a Pluggable Database (PDB) resource appears in a Failed status in the Cloud Console but remains available in the VM, refer to the following Oracle Meta Link note for details on this known issue:

    • Meta Link Note: Pluggable Database (PDB) Resource Shows Failed Status in Cloud Console While It Is Available in VM (Doc ID 2855062.1).

    Step 1: Validate the CON_UID

    Log in to the database and run the following query to verify the Container UID (CON_UID)

    SELECT CON_UID, NAME FROM V$PDBS;

    
    
    SQL> l
      1* select  CON_UID, NAME from V$PDBS
    SQL> /
    
       CON_UID NAME
    ---------- --------------------------------------------------------------------------------------------------------------------------------
    2368539052 PDB$SEED
    1479666877 EGDEV_PDB1
    2175707955 EGQA_PDB1
    
    SQL>
    
    

    Step 2: Confirm the PDB Details

    
    
    dbaascli pdb getDetails --dbname EGPRD --pdbUID 2297085946
    
    

    Step 3: Handle Orphaned PDB Records

  • If the above command returns a valid result, the PDB entry still exists in the backend.
  • In such a case, you may need to delete the orphaned database entry manually.

  • Step 4: Delete Manually 


    To manually delete the PDB using dbaascli, run the following command:

    
    dbaascli pdb delete --dbName  --pdbUID 2297085946
    

    Important Note:

    🚨 Before deleting any PDB, always verify the CON_UID returned from the database.

    • If the CON_UID from V$PDBS matches the one being checked, you are dealing with a live PDB.
    • Do NOT delete the database in this case, as it is currently active.

    If the CON_UID in the Cloud Console is different from the one in the database, you can safely delete the PDB from the console.

    Once deleted, wait 15-20 minutes, and the correct PDB will be reflected in the Cloud Console.




                                                      Figure 2: This is the same figure highlighting the con_uid.

    Friday, November 29, 2024

    Exacs database creation using dbaascli

     







    Intro

    OCI (Oracle Cloud Infrastructure) provides robust automation capabilities for routine maintenance tasks such as patching, creating database homes, and server OS updates. Oracle further simplifies these tasks by handling the staging process, reducing the administrative burden.

    Additionally, Oracle has introduced several tools to streamline maintenance and management tasks, including:

  • dbaascli
  • exacli
  • dbcli
  • exadbcpatchmulti
  • dbaascli is a command-line utility for managing Oracle Database Cloud Services (DBaaS) instances. Mainly for Oracle Exadata cloud services. It provides several features to administer Oracle databases running on Oracle Cloud Infrastructure (OCI). This tool allows users to perform various administrative tasks like backup and recovery, patching, scaling, and checking the status of the database environment.

    Key Features of dbaascli:

    1. Backup and Restore:

      • Manage and schedule backups.
      • Perform on-demand backups.
      • Restore the database from backups.
    2. Patching:

      • Apply database patches.
      • Check for available patches.
      • Rollback patches if necessary.
    3. Database Management:

      • Start and stop databases.
      • Perform database health checks.
      • Manage Data Guard configurations (for high availability and disaster recovery).
    4. Storage Management:

      • Resize the database storage as required.
    5. Diagnostic Tools:

      • Collect diagnostic information to troubleshoot issues.
      • Generate diagnostic logs for support cases.
    As a DBA, it's essential to start utilizing these orchestration tools to streamline and simplify your tasks. These tools make database management more efficient by automating key processes. With dbaascli, for example, DBAs no longer need to manually download and copy binary files to servers—dbaascli will automatically download the latest binaries, making the entire process faster and smoother.

    In this article, I will demonstrate how to create a database using dbaascli.

    Note: Creating a non-CDB database is not supported via the console. The only way to create a non-CDB database is through the backend.

    Step 1: Identify the Database Home


    First, identify and note down the Database Home associated with the version you want to use for creating the database.

    For this example, since I want to create a database using the 19c home, I will record the path for the 19c Database Home.

    
    
    [root@exadevdb-node01 ~]# dbaascli dbhome info
    DBAAS CLI version 24.3.2.0.0
    Executing command dbhome info
    [INFO] [DBAAS-14011] - The usage of this command is deprecated.
       ACTION: It is recommended to use 'dbaascli dbHome getDetails or dbaascli system getDBHomes' for this operation.
    Enter a homename or just press enter if you want details of all homes
    
    1.HOME_NAME=OraHome1
      HOME_LOC=/u02/app/oracle/product/19.0.0.0/dbhome_1
      VERSION=19.24.0.0
      PATCH_LEVEL=19.24.0.0.0
      DBs installed=
       OH Backup=NOT Configured
    
    2.HOME_NAME=OraHome3
      HOME_LOC=/u02/app/oracle/product/11.2.0/dbhome_1
      VERSION=11.2.0.4
      PATCH_LEVEL=11.2.0.4.231017
      DBs installed=TEST
       Agent DB IDs=ec3f0396-d79d-450f-a9b3-aaae88b37c56
     OH Backup=NOT Configured
    
    [root@exadevdb-node01 ~]#
    
    
    

    To retrieve the DB home information, you can use the latest dbaascli command:

    
    dbaascli system getDBHomes
    
    This output will be a Jason output
    
    
    [root@exadevdb-node01 ~]# dbaascli system getDBHomes
    DBAAS CLI version 24.3.2.0.0
    Executing command system getDBHomes
    Job id: 1530c019-99c4-467d-9b2f-be3a2b5d3cd0
    Session log: /var/opt/oracle/log/system/getDBHomes/dbaastools_2024-09-30_03-45-24-PM_342051.log
    {
      "OraHome1" : {
        "homePath" : "/u02/app/oracle/product/19.0.0.0/dbhome_1",
        "homeName" : "OraHome1",
        "version" : "19.24.0.0.0",
        "createTime" : 1724441733874,
        "updateTime" : 1724683497105,
        "unifiedAuditEnabled" : false,
        "ohNodeLevelDetails" : {
          "exadevdb-node02" : {
            "nodeName" : "exadevdb-node02",
            "version" : "19.24.0.0.0",
            "patches" : [ "34697081", "36538667", "36414915", "36587798", "36582781" ]
          },
          "exadevdb-node01" : {
            "nodeName" : "exadevdb-node01",
            "version" : "19.24.0.0.0",
            "patches" : [ "34697081", "36538667", "36414915", "36587798", "36582781" ]
          }
        },
        "messages" : [ ]
      },
      "OraHome2" : {
        "homePath" : "/u02/app/oracle/product/19.0.0.0/dbhome_2",
        "homeName" : "OraHome2",
        "version" : "19.24.0.0.0",
        "createTime" : 1724680153103,
        "updateTime" : 1724692594136,
        "unifiedAuditEnabled" : false,
        "ohNodeLevelDetails" : {
          "exadevdb-node02" : {
            "nodeName" : "exadevdb-node02",
            "version" : "19.24.0.0.0",
            "patches" : [ "34697081", "36538667", "36414915", "36587798", "36582781" ]
          },
          "exadevdb-node01" : {
            "nodeName" : "exadevdb-node01",
            "version" : "19.24.0.0.0",
            "patches" : [ "34697081", "36538667", "36414915", "36587798", "36582781" ]
          }
        },
        "messages" : [ ]
      },
      "OraHome3" : {
        "homePath" : "/u02/app/oracle/product/11.2.0/dbhome_1",
        "homeName" : "OraHome3",
        "version" : "11.2.0.4.231017",
        "createTime" : 1724683333257,
        "updateTime" : 1727708629760,
        "unifiedAuditEnabled" : false,
        "ohNodeLevelDetails" : {
          "exadevdb-node02" : {
            "nodeName" : "exadevdb-node02",
            "version" : "11.2.0.4.231017",
            "patches" : [ "25139545", "33613829", "33991024", "35638387", "22291453", "30432076", "31335037", "32327201", "32558369", "34006614", "34533061", "34698179", "35099667", "35220732", "35239280", "35313335", "35685663", "32224895", "22366322", "31228670", "21289564", "35574089" ]
          },
          "exadevdb-node01" : {
            "nodeName" : "exadevdb-node01",
            "version" : "11.2.0.4.231017",
            "patches" : [ "25139545", "33613829", "33991024", "35638387", "22291453", "30432076", "31335037", "32327201", "32558369", "34006614", "34533061", "34698179", "35099667", "35220732", "35239280", "35313335", "35685663", "32224895", "22366322", "31228670", "21289564", "35574089" ]
          }
        },
        "messages" : [ ]
      },
      "OraHome4" : {
        "homePath" : "/u02/app/oracle/product/19.0.0.0/dbhome_3",
        "homeName" : "OraHome4",
        "version" : "19.24.0.0.0",
        "createTime" : 1724695480172,
        "updateTime" : 1727709224645,
        "unifiedAuditEnabled" : false,
        "ohNodeLevelDetails" : {
          "exadevdb-node02" : {
            "nodeName" : "exadevdb-node02",
            "version" : "19.24.0.0.0",
            "patches" : [ "34697081", "36538667", "36414915", "36587798", "36582781" ]
          },
          "exadevdb-node01" : {
            "nodeName" : "exadevdb-node01",
            "version" : "19.24.0.0.0",
            "patches" : [ "34697081", "36538667", "36414915", "36587798", "36582781" ]
          }
        },
        "messages" : [ ]
      }
    }
    
    dbaascli execution completed
    [root@exadevdb-node01 ~]#
    
    

    Create a database using dbaascli commands


    Here is a sample command to create the database. For detailed information, refer to the Oracle documentation at:  Using dbaascli for Exadata Database Service

    Since we are creating a non-container database, the createAsCDB parameter is set to false.

    
    
    /bin/dbaascli database create \
    --dbname SYNPOC \
    --dbUniqueName SYNPOC_EXA \
    --dbSID SYNPOC1 \
    --createAsCDB false \
    --dbCharset AL32UTF8 \
    --dbNCharset AL16UTF16 \
    --sgaSizeInMB 4096 \
    --pgaSizeInMB 1024 \
    --oracleHome /u02/app/oracle/product/19.0.0.0/dbhome_1 \
    --oracleHomeName OraHome1 \
    --tdeConfigMethod FILE \
    --executePrereqs no
    
    

    Sample log output

    
    
    [root@exadevdb-node01 ~]# /bin/dbaascli database create \
    > --dbname SYNPOC \
    > --dbUniqueName SYNPOC_EXA \
    > --dbSID SYNPOC \
    > --createAsCDB false \
    > --dbCharset AL32UTF8 \
    > --dbNCharset AL16UTF16 \
    > --sgaSizeInMB 4096 \
    > --pgaSizeInMB 1024 \
    > --oracleHome /u02/app/oracle/product/19.0.0.0/dbhome_1 \
    > --oracleHomeName OraHome1 \
    > --tdeConfigMethod FILE \
    > --executePrereqs no
    DBAAS CLI version 24.3.2.0.0
    Executing command database create --dbname SYNPOC --dbUniqueName SYNPOC_EXA --dbSID SYNPOC1 --createAsCDB false --dbCharset AL32UTF8 --dbNCharset AL16UTF16 --sgaSizeInMB 4096 --pgaSizeInMB 1024 --oracleHome /u02/app/oracle/product/19.0.0.0/dbhome_1 --oracleHomeName OraHome1 --tdeConfigMethod FILE --executePrereqs no
    Job id: 88f650c6-0467-4b3b-bcda-3f38eaa0ea6e
    Session log: /var/opt/oracle/log/SYNPOC/database/create/dbaastools_2024-09-30_03-51-46-PM_368426.log
    Enter SYS_PASSWORD:
    
    Enter SYS_PASSWORD (reconfirmation):
    
    Enter TDE_PASSWORD:
    
    Enter TDE_PASSWORD (reconfirmation):
    
    Loading PILOT...
    Enter SYS_PASSWORD
    ********************
    Enter SYS_PASSWORD (reconfirmation):
    **********************
    Enter TDE_PASSWORD
    ***********************
    Enter TDE_PASSWORD (reconfirmation):
    ********************
    Session ID of the current execution is: 1490
    Log file location: /var/opt/oracle/log/SYNPOC/database/create/pilot_2024-09-30_03-53-13-PM_374149
    -----------------
    Running Plugin_initialization job
    Completed Plugin_initialization job
    Acquiring read lock: _u02_app_oracle_product_19.0.0.0_dbhome_1
    -----------------
    Running Default_value_initialization job
    Completed Default_value_initialization job
    -----------------
    Running Validate_input_params job
    Completed Validate_input_params job
    -----------------
    Running Validate_cpu_availability job
    Completed Validate_cpu_availability job
    -----------------
    Running Validate_asm_availability job
    Completed Validate_asm_availability job
    -----------------
    Running Validate_disk_space_availability job
    Completed Validate_disk_space_availability job
    -----------------
    Running Validate_huge_pages_availability job
    Completed Validate_huge_pages_availability job
    -----------------
    Running Validate_hostname_domain job
    Completed Validate_hostname_domain job
    -----------------
    Running Validate_crs_state job
    Completed Validate_crs_state job
    -----------------
    Running Install_db_cloud_backup_module job
    Skipping. Job is detected as not applicable.
    -----------------
    Running Perform_dbca_prechecks job
    Completed Perform_dbca_prechecks job
    -----------------
    Running Validate_backup_report job
    Skipping. Job is detected as not applicable.
    Acquiring write lock: create_db_cloud
    -----------------
    Running Setup_acfs_volumes job
    Completed Setup_acfs_volumes job
    -----------------
    Running Setup_db_folders job
    Completed Setup_db_folders job
    -----------------
    
    Running DB_creation job
    Completed DB_creation job
    Releasing lock: create_db_cloud
    -----------------
    Running Generate_db_metadata job
    Completed Generate_db_metadata job
    -----------------
    Running Create_db_from_backup job
    Skipping. Job is detected as not applicable.
    Completed Create_db_from_backup job
    -----------------
    Running Load_db_details job
    Completed Load_db_details job
    -----------------
    Running Populate_creg job
    Completed Populate_creg job
    -----------------
    Running Register_ocids job
    Skipping. Job is detected as not applicable.
    -----------------
    Running Run_datapatch job
    Skipping. Job is detected as not applicable.
    -----------------
    Running Create_users_tablespace job
    Skipping. Job is detected as not applicable.
    -----------------
    Running Configure_pdb_service job
    Skipping. Job is detected as not applicable.
    -----------------
    Running Set_pdb_admin_user_profile job
    Skipping. Job is detected as not applicable.
    -----------------
    Running Lock_pdb_admin_user job
    Skipping. Job is detected as not applicable.
    -----------------
    Running Configure_flashback job
    Completed Configure_flashback job
    -----------------
    Running Update_cloud_service_recommended_config_parameters job
    Completed Update_cloud_service_recommended_config_parameters job
    -----------------
    Running Update_distributed_lock_timeout job
    Completed Update_distributed_lock_timeout job
    -----------------
    Running Configure_archiving job
    Completed Configure_archiving job
    -----------------
    Running Configure_huge_pages job
    Completed Configure_huge_pages job
    -----------------
    Running Set_credentials job
    Completed Set_credentials job
    -----------------
    Running Update_dba_directories job
    Completed Update_dba_directories job
    -----------------
    Running Set_cluster_interconnects job
    Completed Set_cluster_interconnects job
    -----------------
    Running Create_db_secure_profile job
    Completed Create_db_secure_profile job
    -----------------
    Running Set_utc_timezone job
    Completed Set_utc_timezone job
    -----------------
    Running Run_dst_post_installs job
    Completed Run_dst_post_installs job
    -----------------
    Running Enable_auditing job
    Completed Enable_auditing job
    -----------------
    Running Apply_security_measures job
    Completed Apply_security_measures job
    -----------------
    Running Set_listener_init_params job
    Completed Set_listener_init_params job
    -----------------
    Running Update_db_wallet job
    Completed Update_db_wallet job
    -----------------
    Running Add_oratab_entry job
    Completed Add_oratab_entry job
    -----------------
    Running Setup_dbaastools_schema job
    Skipping. Job is detected as not applicable.
    -----------------
    Running Purge_rmf_configuration job
    Completed Purge_rmf_configuration job
    -----------------
    Running Configure_sqlnet_ora job
    Completed Configure_sqlnet_ora job
    -----------------
    Running Configure_tnsnames_ora job
    Completed Configure_tnsnames_ora job
    -----------------
    Running Enable_fips job
    Completed Enable_fips job
    -----------------
    Running DTRS_DB_Configure_Job job
    Completed DTRS_DB_Configure_Job job
    -----------------
    Running Restart_database job
    Completed Restart_database job
    -----------------
    Running Create_db_login_environment_file job
    Completed Create_db_login_environment_file job
    -----------------
    Running Generate_dbsystem_details job
    Acquiring native write lock: global_dbsystem_details_generation
    Releasing native lock: global_dbsystem_details_generation
    Completed Generate_dbsystem_details job
    -----------------
    Running Cleanup job
    Completed Cleanup job
    Releasing lock: _u02_app_oracle_product_19.0.0.0_dbhome_1
    
    dbaascli execution completed
    [root@exadevdb-node01~]# 
    
    

    Conclusion

    To sum up, dbaascli is an effective command line tool that helps in performing various tasks involved in the lifecycle of a database in the context of the Oracle cloud. It makes the execution of tasks like the creation of databases, the administration of homes and even the fetching of information effortless. With dbaascli, the administrators have a way to automate most of the usual activities and cut down on errors and increase efficiency thereby making the management of databases smooth and effective.

    Wednesday, August 21, 2024

    Oracle world 2024 - AI

     




    Intro 

    The world is transitioning from the data era to the age of artificial intelligence. Many organizations are leveraging AI features to gain a competitive edge and expand their market share. Oracle 23c AI introduces a wide array of AI-driven features to support this shift. With Oracle CloudWorld just around the corner, the event will be a key platform to showcase the power and potential of AI in transforming businesses.

    Oracle CloudWorld 2024 is set to take place from September 9 to 12 in Las Vegas at the Caesars Forum Convention Center. This major event will bring together Oracle customers, partners, and industry experts to explore the latest innovations in cloud technology, AI, databases, and more.

    Oracle 23ai : True Cache

     





    Intro

    In today’s fast-paced digital landscape, speed is paramount. Users expect instant responses, and applications must seamlessly handle high traffic and demanding workloads. Oracle databases, often the backbone of critical systems, bear the weight of these performance demands. Fortunately, Oracle Database has a powerful solution: True Cache, an advanced in-memory caching technology designed to optimize database performance and enhance application efficiency.

    What is Oracle True Cache? 

    True Cache is a fast storage layer that temporarily holds frequently accessed data from an Oracle database to speed up read operations. It's like a quick-access memory that helps retrieve data faster without repeatedly querying the main database.

    Simple terms: True Cache is an in-memory, read-only caching layer positioned in front of an Oracle database.


    General Architecture 


                                                         Figure 1: True Cache Architecture


    The benefits of True Cache in Oracle 23c AI include:

    Faster Data Access: By storing frequently accessed data in memory, True Cache reduces the time needed to retrieve data from the database, leading to quicker response times for applications

    Improved Performance: With data readily available in memory, the overall performance of the database engine is enhanced, allowing for more efficient processing of queries and transactions.

    Reduced Database Load: True Cache minimizes the number of direct queries to the database, thereby reducing the workload on the database engine and freeing up resources for other tasks.

    Enhanced User Experience: Applications that rely on quick data retrieval will see significant improvements, providing a smoother and more responsive user experience.

    Scalability: As demand for data grows, True Cache can help maintain performance levels by efficiently managing and accessing frequently used data.

    If you need more detail, refer to Oracle's main blog for true cache.

    https://blogs.oracle.com/database/post/introducing-oracle-true-cache

    In this article, I will elaborate on how you can set up Oracle's true cache instance.

    Pre-requisites for True cache testing

    • Create two DBCS instances with 23ai. (Both instances should be in the same subnet).
    • Make sure connective is there for two VMs. (default ports 22 and 1521 need to be enabled for the subnet ).
    • Make sure to add the IP address of both nodes to the host file.

    Below is my machine hostname :

    Primary :

    Cache instance :




    Prepare instance 

    Using the DBCA utility, I create a BLOB file, which I then transfer to the True Cache instance. Upon inspecting the contents of the BLOB file, you'll find both the password file and the wallet stored within it.
      $ORACLE_HOME/bin/dbca -configureDatabase -prepareTrueCacheConfigFile \
     -sourceDB P23AI \
     -trueCacheBlobLocation /home/oracle/scripts/ \
     -silent 
      
    

    Expected output

    
      
    [oracle@dbsdpl24 admin]$ $ORACLE_HOME/bin/dbca -configureDatabase -prepareTrueCacheConfigFile \
    -sourceDB P23AI \
    -trueCacheBlobLocation /home/oracle/scripts/ \
    -silent
    
    Enter password for the TDE wallet:
    
    Enter password for the TDE wallet:
    Session ID of the current execution is: 1
    Log file location: /u01/app/oracle/cfgtoollogs/dbca/P23AI_jt7_yyz/trace.log_2024-08-20_04-35-42PM_53669
    -----------------
    Running Initialization job
    Completed Initialization job
    33% complete
    -----------------
    Running Validate_dataguard job
    Skipping. Job is detected as not applicable.
    40% complete
    -----------------
    Running Validate_db_version job
    Completed Validate_db_version job
    47% complete
    -----------------
    Running Validate_tde_credentials job
    Completed Validate_tde_credentials job
    53% complete
    -----------------
    Running Validate_true_cache_instance job
    Completed Validate_true_cache_instance job
    60% complete
    -----------------
    Running Validate_archive_log_mode job
    Completed Validate_archive_log_mode job
    67% complete
    -----------------
    Running Prepare_blob job
    Completed Prepare_blob job
    100% complete
    ---------- PLUGIN NOTES ----------
    Successfully created blob file: /home/oracle/scripts/blob_2024-08-20_04-35-42PM.tar.gz
    ---------- END OF PLUGIN NOTES ----------
    Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/P23AI_jt7_yyz/P23AI_jt7_yyz0.log" for further details.
    [oracle@dbsdpl24 admin]$
      
      
    Note: Make sure to transfer: the blob_2024-08-20_04-35-42PM.tar.gz file to the true cache instance. 


    Configure Listener on the primary instance 

    Add the required entries to the listener configuration and then restart the listener.
    
    REGISTRATION_INVITED_NODES_Listener = (10.0.1.6,10.0.1.240)
    VALID_NODE_CHECKING_REGISTRATION_LISTENER=OFF
    

    Drop DBCS-created database

    Using DBCA to delete a database on a True Cache instance.

    
    $ORACLE_HOME/bin/dbca -deleteDatabase -sourceDB TC23AI -forceArchiveLogDeletion -silent
    

    Configure True Cache

    The next step is to create true cache instance. With DBCA, we can also create a True Cache instance.

    $ORACLE_HOME/bin/dbca -createTrueCache \
    -gdbName TC23AI \
    -sid TC23AI \
    -sourceDBConnectionString dbsdpl24.sub08101943421.cnvcn01.oraclevcn.com:1521/P23AI_jt7_yyz.sub08101943421.cnvcn01.oraclevcn.com \
    -trueCacheBlobFromSourceDB /home/oracle/scripts/blob_2024-08-20_04-35-42PM.tar.gz \
    -sgaTargetInMB 14848 \
    -pgaAggregateTargetInMB 3712 \
    -silent -listeners LISTENER
    

    Expected output

    
    
    [oracle@dbsdpl53 scripts]$ $ORACLE_HOME/bin/dbca -createTrueCache \
    -gdbName TC23AI \
    -sid TC23AI \
    -sourceDBConnectionString dbsdpl24.sub08101943421.cnvcn01.oraclevcn.com:1521/P23AI_jt7_yyz.sub08101943421.cnvcn01.oraclevcn.com \
    -trueCacheBlobFromSourceDB /home/oracle/scripts/blob_2024-08-20_04-35-42PM.tar.gz \
    -sgaTargetInMB 14848 \
    -pgaAggregateTargetInMB 3712 \
    -silent -listeners LISTENER
    Enter Remote DB SYS user password:
    
    Enter password for the source database TDE wallet:
    
    [FATAL] [DBT-10317] Specified SID Name (TC23AI) already exists.
       ACTION: Specify a different SID Name that does not already exist.
    [oracle@dbsdpl53 scripts]$ vi /etc/oratab
    [oracle@dbsdpl53 scripts]$ $ORACLE_HOME/bin/dbca -createTrueCache -gdbName TC23AI -sid TC23AI -sourceDBConnectionString dbsdpl24.sub08101943421.cnvcn01.oraclevc                                                n.com:1521/P23AI_jt7_yyz.sub08101943421.cnvcn01.oraclevcn.com -trueCacheBlobFromSourceDB /home/oracle/scripts/blob_2024-08-20_04-35-42PM.tar.gz -sgaTargetInMB 1                                                4848 -pgaAggregateTargetInMB 3712 -silent -listeners LISTENER
    Enter Remote DB SYS user password:
    
    Enter password for the source database TDE wallet:
    
    [WARNING] [DBT-10331] Specified SID Name (TC23AI) may have a potential conflict with an already existing database on the system.
       CAUSE: The specified SID Name without the trailing numeric characters (TC23AI) may have a potential conflict with an already existing database on the system.
       ACTION: Specify a different SID Name that does not conflict with existing databases on the system.
    Session ID of the current execution is: 5
    Log file location: /u01/app/oracle/cfgtoollogs/dbca/TC23AI/trace.log_2024-08-20_05-28-57PM_98530
    -----------------
    Running Extract_password_file_from_blob_file job
    Completed Extract_password_file_from_blob_file job
    25% complete
    -----------------
    Running Create_static_listener job
    Skipping. Job is detected as not applicable.
    38% complete
    -----------------
    Running Register_listener job
    Completed Register_listener job
    50% complete
    -----------------
    Running Extract_tde_wallet_from_blob_file job
    Completed Extract_tde_wallet_from_blob_file job
    54% complete
    -----------------
    Running Setup_required_directories job
    Skipping. Job is detected as not applicable.
    57% complete
    -----------------
    Running Create_pfile job
    Completed Create_pfile job
    61% complete
    -----------------
    Running Start_nomount_instance job
    Completed Start_nomount_instance job
    64% complete
    -----------------
    Running Create_TDE_wallet job
    Completed Create_TDE_wallet job
    68% complete
    -----------------
    Running Create_truecache_instance job
    
    Completed Create_truecache_instance job
    71% complete
    -----------------
    Running Add_oratab_entry job
    Completed Add_oratab_entry job
    75% complete
    -----------------
    Running Reopen_wallet job
    Completed Reopen_wallet job
    100% complete
    ---------- PLUGIN NOTES ----------
    Successfully created True Cache.
    In order to complete the operation,configure the True Cache database application services from the primary database.
    ---------- END OF PLUGIN NOTES ----------
    Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/TC23AI/TC23AI1.log" for further details.
    
    

    After the creation True cache instance status

    No data file is needed for this, as everything is managed in-memory. Similar to Active Data Guard, archives will push all frequently accessed data.
    
    
    select open_mode,database_role from v$database;
    
    OPEN_MODE            DATABASE_ROLE
    -------------------- ----------------
    READ ONLY WITH APPLY TRUE CACHE
    
    -- MRP 
    
    SQL>  select INST_ID,PROCESS,STATUS,THREAD#,SEQUENCE#,BLOCK# from GV$MANAGED_STANDBY;
    
       INST_ID PROCESS   STATUS                  THREAD#  SEQUENCE#     BLOCK#
    ---------- --------- -------------------- ---------- ---------- ----------
             1 DGRD      ALLOCATED                     0          0          0
             1 DGRD      ALLOCATED                     0          0          0
             1 ARCH      CONNECTED                     0          0          0
             1 ARCH      CONNECTED                     0          0          0
             1 ARCH      CONNECTED                     0          0          0
             1 ARCH      CONNECTED                     0          0          0
             1 RFS       IDLE                          1          0          0
             1 RFS       RECEIVING                     1          5    1335415
             1 RFS       IDLE                          0          0          0
             1 MRP0      APPLYING_LOG                  1          5    1335416
             1 DGRD      ALLOCATED                     0          0          0
    
       INST_ID PROCESS   STATUS                  THREAD#  SEQUENCE#     BLOCK#
    ---------- --------- -------------------- ---------- ---------- ----------
             1 RMI       IDLE                          0          0          0
             1 RMI       IDLE                          0          0          0
    
    13 rows selected.
    
    

    Conclusion

    In a fast-paced world where every millisecond counts, caching is essential, and Oracle's True Cache is revolutionizing database performance. By accelerating data access and enhancing reliability and efficiency, True Cache empowers businesses to stay competitive and deliver exceptional user experiences. If you're aiming to boost your database performance, True Cache could be the game-changing solution you've been looking for.

    Protecting Your Encryption Keys: Lessons from the Oracle Cloud Security Breach (OKV - Part 1 )

      Intro Recent reports have surfaced about a significant security breach involving Oracle Cloud. A threat actor, identified as "rose871...