Troubleshoot ODA Repo Issue

 

Intro 

If you found previous blog posts helpful for addressing ODA upgrade issues on virtualization platforms, the following article will ease your journey toward the 19.8 upgrade.

Article on Upgrades and Issues:

• ODA upgrade 12.1.2.12 to 18.3.0.0 -- Journey to 19.9.0.0 - Part 1
• Virtualized ODA upgrade 18.3.0.0 to 18.8.0.0 - Journey to 12.1.2.12 - 19.13.0.0 - Part 2
• Virtualized ODA upgrade 18.8.0.0 to 19.8.0.0 - Journey to 12.1.2.12 - 19.13.0.0 - Part 3
• ODA upgrade issues for - 12.1.2.6 to 12.1.2.10 

ODA virtualized upgrades are always challenging, with unexpected issues often arising. One consistent problem I've noticed in upgrades up to version 18.8.0.0 is the repository disappearing after a node reboot.

12.1.2.6 -> 12.1.2.10 
12.1.2.10 -> 12.1.2.12 
12.1.2.12 -> 18.3.0.0 
18.3.0.0 -> 18.8.0.0 
18.8.0.0 -> 19.8.0.0 
19.8.0.0 -> 19.13.0.0

In this article, I will elaborate on how you can overcome this repo issue. First, check the cluster VIP status.

crsctl status res -t | grep -A 2 -i havip

If the ha-vips are down you can start ha-vip using below mentioned command

-- If the havip are down , Tried the start havip from cluster command 
crsctl start res ora.havip_3.havip -unsupported

Verify the repos

-- Verify the repos
srvctl config havip
srvctl status havip

Expected output

[root@dbsldb0 ~]# srvctl config havip
HAVIP exists: /havip_0/192.168.16.10, network number 0
Description: HAVIP_NODE1
Home Node: dbsldb0
HAVIP is enabled.
HAVIP is individually enabled on nodes:
HAVIP is individually disabled on nodes:
HAVIP exists: /havip_1/192.168.16.11, network number 0
Description: HAVIP_NODE2
Home Node: dbsdldb1
HAVIP is enabled.
HAVIP is individually enabled on nodes:
HAVIP is individually disabled on nodes:
HAVIP exists: /havip_3/192.168.18.21, network number 3
Description: HAVIP_NODE1
Home Node: dbsldb0
HAVIP is enabled.
HAVIP is individually enabled on nodes:
HAVIP is individually disabled on nodes:
HAVIP exists: /havip_4/192.168.19.21, network number 4
Description: HAVIP_NODE2
Home Node: dbsdldb1
HAVIP is enabled.
HAVIP is individually enabled on nodes:
HAVIP is individually disabled on nodes:

Check the HA VIP status to clearly understand its condition and which node is currently running.

[root@dbsldb0 ~]# srvctl status havip
HAVIP ora.havip_0.havip is enabled
HAVIP ora.havip_0.havip is not running
HAVIP ora.havip_1.havip is enabled
HAVIP ora.havip_1.havip is not running
HAVIP ora.havip_3.havip is enabled
HAVIP ora.havip_3.havip is running on nodes dbsldb0
HAVIP ora.havip_4.havip is enabled
HAVIP ora.havip_4.havip is running on nodes dbsdldb1
[root@piodadb0 ~]#

The next step is to check the dependencies

below-mentioned commands help us to find the associate dependencies.

crsctl status res -t | grep -A 2 -i havip
[root@dbsldb0 ~]# crsctl status res ora.havip_3.havip -p | grep -i havip
NAME=ora.havip_3.havip
TYPE=ora.havip.type
HAVIP_DESCRIPTION=HAVIP_NODE1
START_DEPENDENCIES=hard(ora.net3.network,uniform:type:ora.havip_3.export.type) attraction(ora.data.vmstor1.acfs) dispersion:active(type:ora.havip.type) pullup(ora.net3.network) pullup:always(type:ora.havip_3.export.type)
STOP_DEPENDENCIES=hard(intermediate:ora.net3.network,uniform:intermediate:type:ora.havip_3.export.type)
[root@dbsldb0 ~]#

Also, you can use the same commands with -p display as a list, "crsctl status res ora.havip_3.havip -p"

[root@dbsldb0 ~]# crsctl status res ora.havip_3.havip -p
NAME=ora.havip_3.havip
TYPE=ora.havip.type
ACL=owner:root:rwx,pgrp:root:r-x,other::r--,group:oinstall:r-x,user:grid:r-x
ACTIONS=
ACTION_SCRIPT=
ACTION_TIMEOUT=60
ACTIVE_PLACEMENT=1
AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX%
AUTO_START=always
CARDINALITY=1
CHECK_INTERVAL=30
CHECK_TIMEOUT=60
CLEAN_TIMEOUT=60
DEGREE=1
DELETE_TIMEOUT=60
DESCRIPTION=Oracle HA VIP resource
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
GEN_USR_ORA_STATIC_VIP=
GEN_USR_ORA_VIP=
HAVIP_DESCRIPTION=HAVIP_NODE1
HOSTING_MEMBERS=piodadb0
INSTANCE_FAILOVER=1
INTERMEDIATE_TIMEOUT=0
LOAD=1
LOGGING_LEVEL=1
MODIFY_TIMEOUT=60
NETWORK_COUNT=
NLS_LANG=
OFFLINE_CHECK_INTERVAL=0
OPC_NAT_ADDRESS=
PLACEMENT=favored
RELOCATE_BY_DEPENDENCY=1
RESTART_ATTEMPTS=0
SCRIPT_TIMEOUT=60
SERVER_CATEGORY=ora.hub.category
SERVER_POOLS=*
START_CONCURRENCY=0
START_DEPENDENCIES=hard(ora.net3.network,uniform:type:ora.havip_3.export.type) attraction(ora.data.vmstor1.acfs) dispersion:active(type:ora.havip.type) pullup(ora.net3.network) pullup:always(type:ora.havip_3.export.type)
START_TIMEOUT=120
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=hard(intermediate:ora.net3.network,uniform:intermediate:type:ora.havip_3.export.type)
STOP_TIMEOUT=0
TYPE_VERSION=1.1
UPTIME_THRESHOLD=1h
USER_WORKLOAD=no
USE_STICKINESS=0
USR_ORA_ENV=
USR_ORA_VIP=192.168.18.21
[root@dbsldb0 ~]#

if the repositories are missing, you can manually add them using the commands provided below.

-- IF the exports are missing you need to add the exportfs manually using below mention commands.
- HAVIP4
/u01/app/12.1.0.2/grid/bin/srvctl add exportfs -id havip_4 -path /u01/app/sharedrepo/vmstor1 -name vmstor1_1 -options rw,no_root_squash,no_wdelay
/u01/app/12.1.0.2/grid/bin/srvctl start exportfs -id havip_4
/u01/app/12.1.0.2/grid/bin/srvctl start havip -id havip_4

- HAVIP3
/u01/app/12.1.0.2/grid/bin/srvctl add exportfs -id havip_3 -path /u01/app/sharedrepo/vmstor1 -name vmstor1_0 -options rw,no_root_squash,no_wdelay
/u01/app/12.1.0.2/grid/bin/srvctl start exportfs -id havip_3
/u01/app/12.1.0.2/grid/bin/srvctl start havip -id havip_3

Conclusion

The upgrade from 12.1.2.6 to 19.13.0.0 is not a smooth ride, and you may encounter various issues along the way. I always encourage DBAs to proactively create a Service Request (SR). Repository issues are especially common—I’ve encountered them multiple times, even after a reboot. However, the steps mentioned above are the key to resolving these repository issues.