Wednesday, October 8, 2025

My Focus at Oracle AI World 2025 – AI, Security, and the Road Ahead

 



Intro

Navigating the Future of AI and Security at Oracle AI World 2025

As we step into the age of intelligent enterprises, AI and data security have become inseparable pillars of innovation and resilience. This year’s Oracle AI World 2025, taking place October 13–16 in Las Vegas, marks a defining shift in Oracle’s vision ,  from cloud transformation to AI-driven reinvention.

CloudWorld has been reimagined as Oracle AI World, reflecting Oracle’s commitment to embedding AI across every layer of its ecosystem from applications to infrastructure to data platforms. The event will highlight Oracle’s latest advancements in agentic AI, AI infrastructure, and autonomous systems, reshaping how businesses operate, secure, and scale in the era of intelligent automation.

My Focus: AI and Security

For me, understanding AI and agentic AI is essential. However, my current focus is on the intersection of AI and security. With AI, it’s increasingly difficult to predict which options or vulnerabilities could be exposed, and data has become the lifeblood of AI. Businesses are driven by data, and without strong protection, even a hard-earned reputation can be compromised within hours.

Scripts and agentic AI are already having a profound impact, and it’s challenging to imagine the scale of threats we may face in the future regarding hacking and data theft. The stakes for securing data have never been higher.

Sessions I Recommend at Oracle AI World 2025

If you’re attending Oracle AI World, here are some sessions I highly recommend based on your area of interest:

1st Choice – For Those Focused on Security

  • Be a Data Security Superhero with Oracle Data Safe [HOL2820]

  • How Oracle Ensures Security and Compliance with Oracle Cloud Infrastructure [THR3119]

  • Build a Secure Landing Zone on Oracle Cloud Infrastructure [LRN2727]

2nd Choice – For Those Managing Large-Scale Workloads and Migrations

  • Deploying Mission-Critical: What’s New with Oracle Database 23ai, Exadata, Cloud [LRN1139]

  • Exadata Mastery: Maximize Performance, Reduce Risk, and Save Costs [LRN2980]

3rd Choice – For Those Exploring Virtualization Alternatives

With VMware licensing costs rising, many small and medium-sized businesses are rethinking their virtualization strategy. It’s the perfect time to evaluate Oracle Linux Virtualization Manager (OLVM) as a cost-effective, enterprise-grade alternative.

  • Migrating to Oracle Virtualization Using Oracle Cloud Migrations Service [LRN3085]

  • Oracle Virtualization: The Cost-Effective, High-Performance Alternative [LRN1119]


Final Thoughts

AI brings enormous opportunity , but also a new dimension of risk. As AI systems become more autonomous and interconnected, security must evolve at the same pace. At Oracle AI World 2025, I look forward to exchanging ideas with peers, customers, and Oracle experts on how to build intelligent, secure, and resilient data-driven enterprises.

Tuesday, October 7, 2025

Join Us at Oracle AI World 2025 – Optimizing and Securing Enterprise Databases

 






Join Us at Oracle AI World 2025 – Optimizing and Securing Enterprise Databases

I’m excited to announce that I will be attending Oracle AI World 2025, taking place October 13–16 in Las Vegas, alongside the Eclipsys and DSP teams.

This year marks a significant milestone for Oracle’s flagship event: CloudWorld has been reimagined as Oracle AI World. AI is no longer a supporting element, it is now the foundation for Oracle’s strategy across applications, infrastructure, and data platforms.

Recent developments highlight Oracle’s leadership in the enterprise AI space:

  • Participation in the $500 billion “Stargate” AI infrastructure initiative with OpenAI and SoftBank, providing unprecedented compute capacity for AI workloads.

  • A $30 billion-per-year agreement with OpenAI to host large-scale model training clusters powered by tens of thousands of NVIDIA GPUs.

  • Support for Google Cloud’s Gemini AI models via Vertex AI, giving customers access to advanced multimodal AI tools directly in Oracle Cloud Infrastructure (OCI).

Oracle is also advancing agentic AI capabilities, enabling AI systems to autonomously plan, decide, and act. The Globally Distributed Exadata on Exascale service ensures mission-critical systems and agentic AI workloads remain resilient and high-performing across regions. Additionally, AI Agent Studio empowers customers and partners to build, extend, and orchestrate AI agents within Fusion Applications, creating autonomous workflows tailored to real business needs. Collectively, these innovations position Oracle AI World as a central hub for the enterprise AI era.

Why Database Optimization and Security Matter

Databases remain the backbone of modern enterprises. Optimizing them for performance, scalability, and security is more important than ever. With tools such as Oracle 23ai, Autonomous Database, and Oracle Linux Virtualization Manager (OLVM), organizations can achieve more intelligent, secure, and cost-efficient database operations across both cloud and on-premises environments.

At Eclipsys and DSP, we specialize in helping organizations:

  • Enhance database performance with Oracle Cloud and AI-driven optimization

  • Strengthen security using Oracle Data Safe, Audit Vault, and AI-powered threat detection

  • Modernize workloads leveraging Oracle 23ai, Autonomous Database, and containerized architectures

  • Simplify infrastructure management with OLVM for efficient VM provisioning, clustering, and high availability

  • Deliver end-to-end managed services to ensure reliability, compliance, and operational excellence

Let’s Connect at Oracle AI World

If you are attending Oracle AI World and are looking to optimize and secure your Oracle databases, implement OLVM, or leverage AI-driven Oracle services, we would be delighted to connect. We’ll share practical strategies, real-world use cases, and insights from across the Oracle ecosystem.

See you in Las Vegas, October 13–16, 2025 at Oracle AI World!

Tuesday, August 19, 2025

Optimizer Regression After Upgrade? Fix Slow Queries Using DBMS_SQLDIAG.CREATE_SQL_PATCH

 







Intro


We are in the era of AI, where increasing processing power is critical to managing rapidly growing and complex workloads. As enterprise systems continue to scale, database performance challenges have become increasingly common, especially under high-throughput operations. 

From my experience, Oracle Database stands out as the most flexible and robust platform for identifying, analyzing, and resolving performance issues, thanks to its advanced tooling and diagnostic capabilities.

Recently, we encountered a performance bottleneck in an Exadata Cloud@Customer (ExaCS) environment, where the database was processing a high volume of insert operations originating from a Kafka data stream.

In this case, the application had been running on Oracle 11g for more than eight years without major issues. However, after migrating to a newer database version, we faced multiple performance hiccups. While upgrades are usually straightforward, the optimizer behavior changes introduced in the newer version made performance tuning trickier and more time-consuming. Handling execution plan shifts and adapting SQL performance required careful analysis and fine-tuning to stabilize the environment.

It’s not easy to introduce changes directly on a long-running production database, especially one supporting critical business operations. In this example, most queries performed well under the 19c optimizer, but we encountered one major query that experienced significant slowness and failed to complete within the expected time window. Troubleshooting required deep investigation, plan comparisons, and precise tuning to restore performance without disrupting stable workloads.

In this blog, I will demonstrate how we resolved performance issues for specific queries without making any changes to the application code. The DBMS_SQLDIAG.CREATE_SQL_PATCH procedure provides an effective way to apply optimizer hints and fix problematic SQL statements without modifying the original SQL text.

For more details, you can also refer to Oracle’s documentation:
How to Create a SQL Patch to Add Hints to Application SQL Statements (Doc ID 1931944.1).

In our case, the application had been running on Oracle 11g for more than eight years.
After upgrading to Oracle 19c, most queries performed better with the new optimizer, but a few critical queries started experiencing severe performance issues.

Introducing changes directly into a long-running production database is not easy.
For one problematic query, switching the optimizer behavior back to 11.2.0.4 resolved the issue, and we achieved this without modifying the application code — using a SQL Patch.

Sample SQL Patch Creation.

Here’s an example of how to create a SQL Patch using the DBMS_SQLDIAG.CREATE_SQL_PATCH procedure:


DECLARE
  v_patch_name VARCHAR2(30);
BEGIN
  v_patch_name := DBMS_SQLDIAG.CREATE_SQL_PATCH (
    sql_id       => 'your_sql_id',
    hint_text    => '/*+ USE_HASH_JOIN */',
    name         => 'patch_use_hash',
    description  => 'Forcing hash join to improve performance'
  );
END;
/
  


Key Parameters


Parameter    Description
sql_id           The SQL_ID from AWR/SQL Monitor of the problematic query
hint_text      The optimizer hints you want to inject
name               A custom name for your SQL patch
description  A note to describe the purpose


Example: Forcing 11.2.0.4 Optimizer for a Specific SQL.

In our case, we applied the 11.2.0.4 optimizer behavior to a specific SQL by using the following:


VARIABLE x VARCHAR2(100);
EXEC :x := DBMS_SQLDIAG.CREATE_SQL_PATCH(
  sql_id      => 'grnbsvudp26j3',
  hint_text   => 'optimizer_features_enable(''11.2.0.4'')',
  name        => 'SQL_Patch_grnbsvudp26j3'
);
    


How to Validate If the SQL Is Using the New Optimizer

You can confirm that the SQL Patch has been applied using:

SELECT * FROM table(DBMS_XPLAN.DISPLAY_CURSOR(FORMAT=>'OUTLINE BASIC NOTE')); 

Sample Execution Plan Output

SQL> perf_dplan_cursor_outline.sql
Enter value for SQL_ID : grnbsvudp26j3
Enter Child Number (Default is 0) :

+------------------------------------------------------------------------+
| Report   : Execution Plan for SQL_ID in Cursor Cache                   |
| Instance : EXADB1                                                     |
| SQL_ID   : grnbsvudp26j3                         |
+------------------------------------------------------------------------+

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
EXPLAINED SQL STATEMENT:
------------------------

Plan hash value: 2610349306

------------------------------------------------------------------------
| Id  | Operation                            | Name                    |
------------------------------------------------------------------------
|   0 | INSERT STATEMENT                     |                         |
|   1 |  LOAD TABLE CONVENTIONAL             | TEST_ASSET_ALERT_HISTORY |
|   2 |   CONCATENATION                      |                         |
|   3 |    NESTED LOOPS                      |                         |
|   4 |     NESTED LOOPS                     |                         |
|   5 |      TABLE ACCESS BY INDEX ROWID     | TEST_ASSET_WORK          |
|   6 |       INDEX RANGE SCAN               | IDX_JAW_JID_CID_JEID    |
|   7 |      PARTITION RANGE ITERATOR        |                         |
|   8 |       INDEX RANGE SCAN               | IDX_DGH_CAID_ET_FID_TT  |
|   9 |     TABLE ACCESS BY LOCAL INDEX ROWID| TEST_HISTORY        |
|  10 |    NESTED LOOPS                      |                         |
|  11 |     NESTED LOOPS                     |                         |
|  12 |      TABLE ACCESS BY INDEX ROWID     | TEST_ASSET_WORK          |
|  13 |       INDEX RANGE SCAN               | IDX_JAW_JID_CID_JEID    |
|  14 |      PARTITION RANGE ITERATOR        |                         |
|  15 |       INDEX RANGE SCAN               | IDX_DGH_CAID_ET_FID_TT  |
|  16 |     TABLE ACCESS BY LOCAL INDEX ROWID| TEST_HISTORY        |
------------------------------------------------------------------------

Outline Data
-------------

  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.2.0.4')
      DB_VERSION('19.1.0')
      ALL_ROWS
      OUTLINE_LEAF(@"SEL$58A6D7F6")
      MERGE(@"SEL$1" >"SEL$2")
      OUTLINE_LEAF(@"INS$1")
      OUTLINE_LEAF(@"SEL$58A6D7F6_1")
      USE_CONCAT(@"SEL$58A6D7F6" 8 OR_PREDICATES(8) PREDICATE_REORDERS((16 4)
              (8 6) (9 7) (10 8) (11 9) (12 10) (13 11) (14 12) (15 13) (4 14) (6 15)
              (7 16)))
      OUTLINE_LEAF(@"SEL$58A6D7F6_2")
      OUTLINE(@"SEL$2")
      OUTLINE(@"SEL$1")
      FULL(@"INS$1" "TEST_ASSET_ALERT_HISTORY"@"INS$1")
INDEX_RS_ASC(@"SEL$58A6D7F6_1" "A"@"SEL$1" ("TEST_ASSET_WORK"."JOB_ID"
"JOB_ASSET_WORK"."COMPANY_ID" "JOB_ASSET_WORK"."JOB_EVENT_ID")) INDEX(@"SEL$58A6D7F6_1" "DH"@"SEL$1" ("TEST_HISTORY"."COMPANY_ASSET_ID" "TEST_HISTORY"."EVENT_TIME"
"TEST_HISTORY"."FACILITY_ID" "TEST_HISTORY"."TRANSACTION_TYPE"))
INDEX_RS_ASC(@"SEL$58A6D7F6_2" "A"@"SEL$58A6D7F6_2" ("TEST_ASSET_WORK"."JOB_ID" "JOB_ASSET_WORK"."COMPANY_ID"
"TEST_ASSET_WORK"."JOB_EVENT_ID"))
INDEX(@"SEL$58A6D7F6_2" "DH"@"SEL$58A6D7F6_2" ("TEST_HISTORY"."COMPANY_ASSET_ID" "TEST_HISTORY"."EVENT_TIME"
"TEST_HISTORY"."FACILITY_ID" "TEST_HISTORY"."TRANSACTION_TYPE"))
LEADING(@"SEL$58A6D7F6_1" "A"@"SEL$1" "DH"@"SEL$1") LEADING(@"SEL$58A6D7F6_2" "A"@"SEL$58A6D7F6_2" "DH"@"SEL$58A6D7F6_2") USE_NL(@"SEL$58A6D7F6_1" "DH"@"SEL$1") NLJ_BATCHING(@"SEL$58A6D7F6_1" "DH"@"SEL$1") USE_NL(@"SEL$58A6D7F6_2" "DH"@"SEL$58A6D7F6_2") NLJ_BATCHING(@"SEL$58A6D7F6_2" "DH"@"SEL$58A6D7F6_2") END_OUTLINE_DATA */ Note ----- - SQL patch "SQL_Patch_grnbsvudp26j3" used for this statement 87 rows selected. SQL>
Key Highlights from the Execution Plan: Optimizer hint applied successfully:

Summary

  • Problem: After upgrading to 19c, a few queries slowed down due to optimizer changes.

  • Solution: Created a SQL Patch to apply legacy optimizer features.

  • Advantage: No changes were made to the application code.

  • Result: Query performance was restored to expected levels.


Conclusion

Handling performance issues in long-running Oracle databases, especially after major version upgrades, can be challenging. Optimizer behavior often changes between versions, and while most queries benefit from the enhancements, certain critical queries may experience unexpected regressions.

In such scenarios, DBMS_SQLDIAG.CREATE_SQL_PATCH provides a powerful, non-intrusive solution to fix problematic queries without making any application changes. By injecting optimizer hints or enforcing a specific optimizer version, we can stabilize performance quickly and ensure business continuity.

This approach not only reduces risk but also saves significant time during troubleshooting, particularly in production environments where application changes may involve lengthy testing cycles. SQL patches act as a bridge between performance optimization and application stability, making them an essential tool for every Oracle DBA.

Thursday, August 7, 2025

Performance issue : Handling SQL Version Count Issues with High-Volume Kafka Inserts on ExaCS

 






Intro 

We are in the era of AI, where increasing processing power is crucial for handling growing and complex workloads. As enterprise systems continue to scale, database performance challenges become increasingly common, especially under high-throughput operations. In my experience, no other database platform matches Oracle’s flexibility and tooling when it comes to identifying and resolving such performance issues.

Recently, we encountered a performance bottleneck in an Exadata Cloud@Customer (ExaCS) environment, where the database was handling a high volume of insert operations coming from a Kafka stream.

In this article, I’ll walk through the technical details of the SQL version count issue we faced and the solution we implemented to stabilize performance.

The Issue: Excessive SQL Versioning and Hard Parses

The ExaCS database was receiving a continuous stream of dynamically structured INSERT statements from Kafka. The column structure of these inserts varied significantly; some contained 100 columns, while others had up to 150. This variation was ongoing and unpredictable.

Due to these structural differences, Oracle’s optimizer treated each statement as a unique SQL. As a result, the database began to experience:
  • Excessive hard parses

  • High CPU utilization

  • Shared pool contention and pressure

Even though we were running Oracle 19.24, which includes enhancements to SQL plan management and version count handling, the optimizer still created new cursor versions for each structurally distinct INSERT, which led to rapid cursor growth and degraded overall performance.


                                    
                                        Figure 1: AWR report for Oracle SQLID version count


Temporary Workaround: Manual Flushing of High Version Count Cursors

As an immediate mitigation step, we identified the SQLs with high version counts and manually flushed them from the shared pool using their memory IDs. This helped temporarily relieve pressure on CPU and memory by:

  • Reducing shared pool bloat

  • Freeing up memory consumed by excessive cursor versions

  • Preventing further hard parsing on the same overloaded SQL

However, it's important to note that this is only a temporary workaround. The relief is short-lived, as the issue resurfaces once new INSERT statements with varying structures continue streaming in from Kafka.

    To clarify, this issue has not been resolved in Oracle 19.24, despite the version including several recent patches and updates. Here’s the output from the environment confirming the exact patch level:
    
    [oracle@exaprd01-node01 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
    34697081;NOT SHIPPING LIBAUTH_SDK_IAM.SO IN 23 SHIPHOME INSTALL
    36538667;JDK BUNDLE PATCH 19.0.0.0.240716
    36414915;OJVM RELEASE UPDATE: 19.24.0.0.240716 (36414915)
    36587798;OCW RELEASE UPDATE 19.24.0.0.0 (36587798)
    36582781;Database Release Update : 19.24.0.0.240716 (36582781)
    OPatch succeeded.
    [oracle@exaprd01-node01 ~]$
    


    To monitor and identify SQL statements with high version counts typically those contributing to shared pool pressure, you can use the following query:
    
    SELECT version_count, sql_id, sql_text FROM   v$sqlarea WHERE  version_count >  512;
    
    For any SQLs with unusually high version counts, manual flushing can be performed as a short-term mitigation step using the following commands:

    select inst_id,ADDRESS, HASH_VALUE from V$SQLAREA where SQL_ID like '&sqlid';
    exec sys.DBMS_SHARED_POOL.PURGE ('-ADDRESS-, -HASH-VALUE-', 'C');
    

    Note: Use manual flushing with caution, especially in production environments, as it may impact performance for frequently executed queries.

    Permanent Solution

    To address the high version count issue more permanently, Oracle provides specific guidance in the following My Oracle Support note:

    High Version Counts For SQL Statements (>1024) Post Upgrade To 12.2 and Above Causing Database Slow Performance Doc ID 2431353.1

    As per the recommendation, you should add the following initialization parameter _cursor_obsolete_threshold = <recommended_value>

    This parameter helps control the number of obsolete child cursors and can significantly reduce version count growth, improving shared pool performance and overall database stability.

    If you’re running in a RAC environment, apply this change and restart the database instances in a rolling fashion to avoid downtime.

    
    alter system set “_cursor_obsolete_threshold”=1024 scope=spfile;
    

    Conclusion

    This issue showed us how dynamic SQL from systems like Kafka can create serious performance problems, even in powerful environments like Exadata Cloud@Customer. Because each INSERT had a slightly different structure, Oracle treated them as new statements, leading to too many cursor versions, high CPU usage, and shared pool pressure.

    Even though we were on Oracle 19.24, the problem still occurred. The key was identifying the root cause and taking action, monitoring version counts, applying a temporary fix, and then implementing a permanent solution using the _cursor_obsolete_threshold parameter.

    In short, managing SQL behavior and understanding how Oracle handles different workloads is critical for keeping your systems running smoothly, especially in today’s fast-moving, high-volume environments.

    Friday, August 1, 2025

    OLVM : Expanding Fiber Channel (FC) Support in OLVM Using a Data Domain Storage System

     




    Intro

    We are living in a data-driven era where AI is advancing rapidly, placing even greater demands on processing power and virtualized environments. While cloud adoption continues to grow—largely fueled by virtualization—many organizations still rely heavily on their on-premises virtual infrastructure alongside cloud technologies.

    Oracle Linux Virtualization Manager (OLVM) is quickly emerging as a strong alternative, addressing key gaps left by other platforms like VMware. With Broadcom tightening its licensing policies, many organizations are now considering a move away from VMware. For those looking to transition, OLVM stands out as a reliable and cost-effective option, backed by Oracle’s 24/7 enterprise-grade support.

    In this article, I’ll walk you through how to expand Fiber Channel (FC) support in OLVM using a Data Domain storage system.

    Document Reference:

    OLVM: Expanding the Size of a Storage Domain (FC/iSCSI) (Doc ID 2881013.1)

    Steps to increase the FC data domain :

    • Increase the Storage LUN at the SAN level.
    • On all KVM hypervisors where the storage is mounted, execute the following command:/usr/bin/rescan-scsi-bus.sh
    • Increase the size of the Data Domain from the OLVM storage side.


    This is a sample output from executing /usr/bin/rescan-scsi-bus.sh.

    Please ensure any issues are resolved before proceeding with the disk size increase from the OLVM side.

    Sample output :


    
    [root@kvm01 ~]# /usr/bin/rescan-scsi-bus.sh
    Scanning SCSI subsystem for new devices
    Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
     Scanning for device 0 0 0 0 ...
    OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00
          Vendor: Generic- Model: SD/MMC CRW       Rev: 1.00
          Type:   Direct-Access                    ANSI SCSI revision: 06
    .Scanning host 1 for  all SCSI target IDs, all LUNs
     Scanning for device 1 0 0 0 ...
    OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 00
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 0 1 ...
    OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 01
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 0 2 ...
    OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 02
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 0 3 ...
    OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 03
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 0 5 ...
    OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 05
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 0 9 ...
    OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 09
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 0 10 ...
    OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 10
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 0 11 ...
    OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 11
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 1 0 ...
    OLD: Host: scsi1 Channel: 00 Id: 01 Lun: 00
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 1 1 ...
    OLD: Host: scsi1 Channel: 00 Id: 01 Lun: 01
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 1 2 ...
    OLD: Host: scsi1 Channel: 00 Id: 01 Lun: 02
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 1 3 ...
    OLD: Host: scsi1 Channel: 00 Id: 01 Lun: 03
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 1 5 ...
    OLD: Host: scsi1 Channel: 00 Id: 01 Lun: 05
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 1 9 ...
    OLD: Host: scsi1 Channel: 00 Id: 01 Lun: 09
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 1 10 ...
    OLD: Host: scsi1 Channel: 00 Id: 01 Lun: 10
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 1 0 1 11 ...
    OLD: Host: scsi1 Channel: 00 Id: 01 Lun: 11
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
    Scanning host 2 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
     Scanning for device 2 0 0 0 ...
    OLD: Host: scsi2 Channel: 00 Id: 00 Lun: 00
          Vendor: HPE      Model: Smart Adapter    Rev: 3.53
          Type:   Enclosure                        ANSI SCSI revision: 05
     Scanning for device 2 1 0 0 ... 0 ...
    OLD: Host: scsi2 Channel: 01 Id: 00 Lun: 00
          Vendor: HPE      Model: LOGICAL VOLUME   Rev: 3.53
          Type:   Direct-Access                    ANSI SCSI revision: 05
     Scanning for device 2 2 0 0 ... 0 ...
    OLD: Host: scsi2 Channel: 02 Id: 00 Lun: 00
          Vendor: HPE      Model: P408i-a SR Gen10 Rev: 3.53
          Type:   RAID                             ANSI SCSI revision: 05
    Scanning host 3 for  all SCSI target IDs, all LUNs
     Scanning for device 3 0 0 0 ...
    OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00
          Vendor: DGC      Model: VRAID            Rev: 0532
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 3 0 0 1 ...
    OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 01
          Vendor: DGC      Model: VRAID            Rev: 0532
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 3 0 0 2 ...
    OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 02
          Vendor: DGC      Model: VRAID            Rev: 0532
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 3 0 1 0 ...
    OLD: Host: scsi3 Channel: 00 Id: 01 Lun: 00
          Vendor: DGC      Model: VRAID            Rev: 0532
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 3 0 1 1 ...
    OLD: Host: scsi3 Channel: 00 Id: 01 Lun: 01
          Vendor: DGC      Model: VRAID            Rev: 0532
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 3 0 1 2 ...
    OLD: Host: scsi3 Channel: 00 Id: 01 Lun: 02
          Vendor: DGC      Model: VRAID            Rev: 0532
          Type:   Direct-Access                    ANSI SCSI revision: 04
    Scanning host 4 for  all SCSI target IDs, all LUNs
     Scanning for device 4 0 0 0 ...
    OLD: Host: scsi4 Channel: 00 Id: 00 Lun: 00
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 0 1 ...
    OLD: Host: scsi4 Channel: 00 Id: 00 Lun: 01
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 0 2 ...
    OLD: Host: scsi4 Channel: 00 Id: 00 Lun: 02
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 0 3 ...
    OLD: Host: scsi4 Channel: 00 Id: 00 Lun: 03
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 0 5 ...
    OLD: Host: scsi4 Channel: 00 Id: 00 Lun: 05
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 0 9 ...
    OLD: Host: scsi4 Channel: 00 Id: 00 Lun: 09
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 0 10 ...
    OLD: Host: scsi4 Channel: 00 Id: 00 Lun: 10
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 0 11 ...
    OLD: Host: scsi4 Channel: 00 Id: 00 Lun: 11
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 1 0 ...
    OLD: Host: scsi4 Channel: 00 Id: 01 Lun: 00
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 1 1 ...
    OLD: Host: scsi4 Channel: 00 Id: 01 Lun: 01
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 1 2 ...
    OLD: Host: scsi4 Channel: 00 Id: 01 Lun: 02
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 1 3 ...
    OLD: Host: scsi4 Channel: 00 Id: 01 Lun: 03
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 1 5 ...
    OLD: Host: scsi4 Channel: 00 Id: 01 Lun: 05
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 1 9 ...
    OLD: Host: scsi4 Channel: 00 Id: 01 Lun: 09
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 1 10 ...
    OLD: Host: scsi4 Channel: 00 Id: 01 Lun: 10
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
     Scanning for device 4 0 1 11 ...
    OLD: Host: scsi4 Channel: 00 Id: 01 Lun: 11
          Vendor: DGC      Model: VRAID            Rev: 0430
          Type:   Direct-Access                    ANSI SCSI revision: 04
    Scanning host 5 for  all SCSI target IDs, all LUNs
    0 new or changed device(s) found.
    0 remapped or resized device(s) found.
    0 device(s) removed.
    
    


    Once the rescan is completed on all KVM hosts, the next step is to increase the Data Domain from the storage side.

    Keep in mind that even if you extend the LUN at the SAN level, the change will not automatically reflect in the Data Domain.

    From OLVM : Navigate to Storage >  Storage Domain > (Select respective data domain) > Select Manage domain as higlighted below.

                                                           
                                                        Figure 1 : Select Manage Domain

    The Manage Windows interface will display the updated size, allowing you to increase it as needed.


                                                   Figure 2 : Manage window

    Now, select the desired size and click OK to apply the increase.



                                                   Figure 3: Increase in size.

    Now Data Domain will reflect the new size.


    Figure: Data domain after an increase in size. 

    Conclusion 

    In today's evolving IT landscape, organizations are under increasing pressure to modernize infrastructure while maintaining flexibility, control, and cost-efficiency. As AI and data workloads grow, the need for robust, scalable virtualization solutions becomes even more critical.

    With Broadcom’s licensing changes pushing many to reconsider their reliance on VMware, OLVM offers a compelling path forward. It not only fills the functionality gaps but also provides enterprise-grade reliability, backed by Oracle’s 24/7 support.

    Whether you're planning a full migration or building out a hybrid environment, OLVM is well-positioned to meet the demands of modern workloads. In this article, we explored how to extend Fiber Channel (FC) capabilities in OLVM using Data Domain, helping organizations take a step forward in building a resilient and future-ready virtual infrastructure.


    Thursday, June 26, 2025

    Streamlining TDE Key Management with Oracle Key Vault and Secure Endpoint Deployment

     






    Intro

    Streamlining TDE Key Management with Oracle Key Vault and Secure Endpoint Deployment

    One of the core pillars of database security is robust encryption key management. Yet many organizations still store Transparent Data Encryption (TDE) keys locally, a practice that introduces significant risk because keys can be lost through disk corruption, system failure, or accidental mismanagement. Managing those keys manually across multiple Oracle and MySQL databases only compounds the challenge and increases the potential breach surface.

    Oracle Key Vault (OKV) offers a modern alternative by centralizing the storage and lifecycle management of encryption keys, certificates, and secrets in a secure, policy‑driven environment that communicates with each database over KMIP. Using OKV translates to reduced development time, improved security posture, and compliance with data‑protection standards. While there are associated costs for using the service, the benefits of enhanced security and streamlined operations often outweigh the expenses.

    To accelerate adoption at scale, OKV exposes a RESTful API that lets you automate installation and registration of endpoint software, dramatically shortening the time required to bring hundreds of database servers under centralized key management. For added protection, you can restrict the API to trusted IP ranges during rollout and disable the reset functionality once onboarding is complete, further narrowing the attack surface.

    The remainder of this post walks through a practical, script‑driven workflow that installs the endpoint client, registers it with OKV, and provisions a default wallet, laying the foundation for secure, compliant, and easily auditable encryption across your Oracle estate.

    In this article, I will elaborate on how we can perform the EP installation via RESTful API calls.

    Configure the OKV RESTful service


    First, create a directory  /u01/app/oracle/okvapi to download the okvresetcli package. Additionally, create a wallet directory that will be used to download the OKV wallet files.

    Proceed to download the REST API package using the curl command.

    
    mkdir -p /u01/app/oracle/okvapi
    
    [oracle@ora-01 okvapi]$ curl -O -k https://10.0.0.180:5695/okvrestclipackage.zip
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 4134k  100 4134k    0     0  96.1M      0 --:--:-- --:--:-- --:--:-- 96.1M
    


    Once the download is complete, extract the package. This will create the required directory structure and files, including the configuration file that needs to be updated with OKV server details.


    
    [oracle@ora-01 okvapi]$ ls -lrth
    total 4.1M
    -rw-r--r--. 1 oracle oinstall 4.1M Jun 13 14:36 okvrestclipackage.zip
    [oracle@ora-01 okvapi]$ unzip okvrestclipackage.zip
    Archive:  okvrestclipackage.zip
       creating: lib/
       creating: bin/
      inflating: bin/okv
      inflating: bin/okv.bat
       creating: conf/
      inflating: conf/okvrestcli_logging.properties
      inflating: conf/okvrestcli.ini
      inflating: lib/okvrestcli.jar
    [oracle@ora-01 okvapi]$ 


    Next, update the okvrestcli.ini file located in the conf directory with the appropriate OKV server details. Additionally, ensure that the client_wallet parameter is included and correctly set.

    
    
    [oracle@ora-01 conf]$ cat okvrestcli.ini
    #Provide absolute path for log_property, okv_client_config properties
    [Default]
    log_property=/u01/app/oracle/okvapi/conf/okvrestcli_logging.properties
    server=10.0.0.180
    okv_client_config=./conf/okvclient.ora
    user=admin
    client_wallet=/u01/app/oracle/okvapi/wallet
    
    


    Create the wallet for the OKV RESTful API

    If the client_wallet directory is already specified in the configuration file, the next step is to download the wallet from the Oracle Key Vault (OKV) server. This wallet is essential for secure communication between the client and OKV.

    Before download, make sure to export the init location and execute "/u01/app/oracle/okvapi/bin/okv admin client-wallet add --client-wallet /u01/app/oracle/okvapi/wallet --wallet-user admin"

    Step 1: Set the Configuration File Path

    
    export JAVA_HOME=/u01/app/oracle/product/19.0.0/dbhome_1/jdk
    export OKV_RESTCLI_CONFIG=/u01/app/oracle/okvapi/conf/okvrestcli.ini
    
    

    Step 2: Initialize the Client Wallet


    Run the following command to create the client wallet:
    
    
    /u01/app/oracle/okvapi/bin/okv admin client-wallet add \
      --client-wallet /u01/app/oracle/okvapi/wallet \
      --wallet-user admin
    


    You will be prompted to enter the password. Upon success, you should see:

    
    export JAVA_HOME=/u01/app/oracle/product/19.0.0/dbhome_1/jdk
    export OKV_RESTCLI_CONFIG=/u01/app/oracle/okvapi/conf/okvrestcli.ini
    [oracle@ora-01 okvapi]$ /u01/app/oracle/okvapi/bin/okv admin client-wallet add --client-wallet /u01/app/oracle/okvapi/wallet --wallet-user admin
    Password:
    {
      "result" : "Success"
    }
    [oracle@ora-01 okvapi]$
    
    

    Important Note:

    This command will fail if the RESTful service is not enabled in OKV. By default, the REST service is disabled to maintain security.

    Example of failure message:

    
    [oracle@dbsdpl55 bin]$ /u01/app/oracle/okvapi/bin/okv admin client-wallet add --client-wallet /u01/app/oracle/okvapi/wallet --wallet-user admin
    Password:
    {
      "result" : "Failure",
      "message" : "REST service is disabled"
    }
    [oracle@dbsdpl55 bin]$
    


    Solution

    For testing purposes, you may temporarily enable the REST service for all IP addresses. However, in a production environment, it is strongly recommended to enable the service only for specific, trusted targets. Once the installation and configuration are complete, you can disable the REST service again to maintain a secure setup.





    Verifying the OKV RESTful API Installation

    Use the following commands to verify the REST CLI and retrieve server details:

    
    [oracle@ora-01 okvapi]$ $OKV_HOME/bin/okv
    {
      "restCLIVersion" : "21.10.0.0.0"
    }
    
    [oracle@ora-01 okvapi]$  $OKV_HOME/bin/okv server info get
    {
      "result" : "Success",
      "value" : {
        "caCertificateExpirationDate" : "2028-05-19 18:59:37",
        "cpuCores" : "2",
        "deploymentType" : "Cluster",
        "diskInGB" : "3695",
        "fraInGB" : "160",
        "installDate" : "2025-05-20 18:59:54",
        "memoryInKB" : "16365128",
        "platformCertificatesExpirationDate" : "2027-03-12 02:48:20",
        "serverCertificateExpirationDate" : "2026-05-20 19:03:31",
        "serverTime" : "2025-06-13 14:41:53",
        "version" : "21.10.0.0.0"
      }
    }
    
    This confirms that the OKV RESTful API is correctly installed and communicating with the server.

    Configuring Endpoint (EP) via REST API


    To configure OKV using RESTful calls, follow the steps below:

    Prerequisites

    Before provisioning the endpoint, ensure the following directory structure exists under the wallet_root:
    
    mkdir -p /u01/app/oracle/admin/ORCL/wallet_root/tde        # Location for TDE keys
    mkdir -p /u01/app/oracle/admin/ORCL/wallet_root/okv        # Location for OKV endpoint
    mkdir -p /u01/app/oracle/admin/ORCL/wallet_root/tde_seps   # Location for External SSO Key
    

    Step-by-Step Configuration

    1. Create the Endpoint

    
    /u01/app/oracle/okvapi/bin/okv admin endpoint create --endpoint EP_ORCL --description "$HOSTNAME, $(hostname -i)" --type ORACLE_DB --platform LINUX64 -- Create end point 
    
    

    Sample Output:
    
    export OKV_RESTCLI_CONFIG=/u01/app/oracle/okvapi/conf/okvrestcli.ini
    export JAVA_HOME=/u01/app/oracle/product/19c/dbhome_1/jdk
    
    [oracle@ora-01 wallet_root]$ /u01/app/oracle/okvapi/bin/okv admin endpoint create --endpoint EP_ORCL --description "$HOSTNAME, $(hostname -i)" --type ORACLE_DB --platform LINUX64
    {
      "result" : "Success",
      "value" : {
        "status" : "PENDING",
        "locatorID" : "CBB0383C-CC14-4EA4-BF75-01981C522F74"
      }
    }
    [oracle@ora-01 wallet_root]$
    

    After creating the endpoint, verify its status in the OKV console; it should display as "REGISTERED". To complete the endpoint setup, you must proceed with the provisioning step.


    2. Set the Default Wallet for the Endpoint

    
    /u01/app/oracle/okvapi/bin/okv manage-access wallet set-default \
      --wallet WLT_ORCL \
      --endpoint EP_ORCL
    

    3. Provision the Endpoint

    
    /u01/app/oracle/okvapi/bin/okv admin endpoint provision \
      --endpoint EP_ORCL \
      --location /u01/app/oracle/admin/ORCL/wallet_root/okv \
      --auto-login FALSE
    

    Post-Configuration

    Once the endpoint is created, you can verify its status in the OKV console. The endpoint status should change to "ENROLLED" after successful provisioning. This completes the REST-based configuration of an Oracle Database endpoint with OKV.




    Conclusion

    Configuring Oracle Key Vault (OKV) using RESTful API provides a secure, automated, and centralized approach to managing encryption keys for Oracle databases. By following the steps outlined for setting up the configuration, creating the required directories, defining the endpoint, associating the default wallet, and provisioning the endpoint, you establish a robust foundation for TDE and other security integrations.

    Always ensure the RESTful service is enabled securely, especially in production environments, by restricting access to trusted IPs. Once the configuration is verified and functional, you can disable the REST service to further harden your setup.

    This approach not only simplifies key lifecycle management but also enhances compliance and operational security across your Oracle infrastructure.

    Friday, June 20, 2025

    Oracle Key Vault (OKV) Endpoint (EP) Health Check: Best Practices & Troubleshooting Tips

     






    Intro 

    Oracle Key Vault (OKV) plays a vital role in the AI era, where data security, encryption, and regulatory compliance are more critical than ever. OKV communicates securely with its targets using KMIP (Key Management Interoperability Protocol) over TLS. In simple terms, without a properly configured OKV Endpoint (EP), OKV cannot communicate with the target system.

    Installing the Endpoint is essential because it provides the secure link between the database and OKV.
     



                                                 Figure 1 : OKV communication with target

    Before installing the OKV Endpoint agent, there are several prerequisites to consider. One of the most important is the correct folder structure. All required directories must be created directly under the wallet_root location.

    Folder structure required for OKV installation. 

    
    mkdir -p /u01/app/oracle/admin/TCDB/wallet_root/tde - Location for TDE keys
    mkdir -p /u01/app/oracle/admin/TCDB/wallet_root/okv - Location for OKV End point
    mkdir -p /u01/app/oracle/admin/TCDB/wallet_root/tde_seps - Location for External SSO Key    


    After extracting the OKV Endpoint binary, it's important to run the provided health check script. This script validates the endpoint configuration, ensuring everything is set up correctly. If there are any issues especially with the folder structure the health check will flag them.

    The OKV health check scripts are located in the bin directory of the extracted endpoint package. Below is a sample taken from an Oracle DBCS (Database Cloud Service) instance:
    
    [oracle@dbsdpl55 bin]$ pwd
    /opt/oracle/dcs/commonstore/wallets/PWHSE01_3c9_yyz/okv/bin
    [oracle@dbsdpl55 bin]$ ls -lrth
    total 8.4M
    -rw-r--r-- 1 oracle oinstall    0 May 23 11:11 okv.log.0
    -rwxr-x--- 1 oracle oinstall 5.3K May 23 15:03 root.sh
    -rwxr-x--- 1 oracle oinstall 8.0K May 23 15:03 okvutil
    -rwxr-x--- 1 oracle oinstall  29K May 23 15:03 okv_ssh_ep_lookup_authorized_keys
    -rwxr-x--- 1 oracle oinstall 8.4M May 23 15:03 okveps.x64
    -rwxr-x--- 1 oracle oinstall  20K May 23 15:03 ep_healthcheck.sh
    [oracle@dbsdpl55 bin]$
    
    

    Executing OKV Endpoint Health Check.


    Once the OKV Endpoint binary is extracted and the environment is set up, you should run the health check script to validate the configuration. Below is a sample output from a DBCS environment:

    
     
     [oracle@dbsdpl55 bin]$ ./ep_healthcheck.sh
    Verifying the environment variables...
    PASSED.
    
    
    Verifying if there are multiple HSM libraries...
    PASSED.
    
    
    WALLET_ROOT=/opt/oracle/dcs/commonstore/wallets/PWHSE01_3c9_yyz is set. Verifying whether okv,tde and tde_seps folders exist under /opt/oracle/dcs/commonstore/wallets/PWHSE01_3c9_yyz...
    PASSED.
    
    
    Verifying whether the environment variables set for gen0 process...
    PASSED.
    
    
    Verifying whether Oracle Key Vault client configuration is set correctly...
    PASSED.
    
    
    Verifying whether the SSL wallet location is set correctly...
    PASSED.
    
    Enter endpoint Password (for an auto login endpoint, just press Enter):
    Verifying whether okvutil fetches data successfully...
    
    Connected to 10.0.0.180:5696.
    7F7407FF-651F-412B-834C-A616BC97C509    Template        Default template for PWHSE01_EP
    PASSED.
    
    
    
    Fetching Persistent cache details...
    WARNING: Persistent Cache file doesn't exist. If the 'EXPIRE PKCS11 PERSISTENT CACHE ON DATABASE SHUTDOWN' parameter is configured for this endpoint then persistent cache will not be accessible. Please note that this parameter won't appear in the okvclient.ora file and can be modified only from OKV webconsole.
    DONE.
    
    Verifying whether PKCS Library is set correctly...
    PASSED.
    
    
    Please verify /opt/oracle/dcs/commonstore/wallets/PWHSE01_3c9_yyz/okv/log/ep_healthcheck_PWHSE01_2025.05.23-11.25.01.log for more details.
    
    [oracle@dbsdpl55 bin]$
    
     
     

    Conclusion

    In today's AI-driven landscape, where safeguarding sensitive data is paramount, Oracle Key Vault (OKV) offers a robust and centralized approach to managing encryption keys and secrets. A correctly configured OKV Endpoint (EP) is not just a technical requirement—it is the foundation for secure communication between the database and OKV. By ensuring the proper setup of the EP, including the necessary folder structures and successful health checks, organizations can enforce strong encryption practices, maintain compliance, and confidently protect their data assets in both on-premises and cloud environments.

    My Focus at Oracle AI World 2025 – AI, Security, and the Road Ahead

      Intro Navigating the Future of AI and Security at Oracle AI World 2025 As we step into the age of intelligent enterprises, AI and data sec...