Patching in 19C RAC Environment
Download: Patch 34762026: GI RELEASE UPDATE 19.18.0.0.0 (REL-JAN230131)
###############
1. Perform pre-checks
###############
ps -ef | grep pmon
ps -ef | grep tns
ps -ef | grep d.bin
ps -ef | grep mrp
/oracle/grid/19c/bin/kfod op=patches #Check applied patches
/oracle/grid/19c/bin/kfod op=patchlvl #Check Current Patch Level
/oracle/grid/19c/bin/crsctl query crs activeversion -f #Check active version
/oracle/grid/19c/bin/crsctl query crs softwarepatch
/oracle/grid/19c/bin/crsctl stat res -t
From Grid Home
export ORACLE_HOME=/oracle/grid/19c
export PATH=$ORACLE_HOME/bin:$PATH
export PATH=$ORACLE_HOME/OPatch:$PATH
opatch lsinv
opatch lspatches
opatch lsinv | grep -i applied
opatch lspatches | sort -nr
From DB Home
srvctl status database -d PROD
select name,open_mode,database_role from v$database;
export ORACLE_HOME=/oracle/db/19c
export PATH=/oracle/db/19c/bin:$PATH
export PATH=/oracle/db/19c/OPatch:$PATH
opatch lsinv
opatch lspatches
opatch lsinv | grep -i applied
On ORACLE_HOME validate the database component
select name,open_mode,database_role from V$database;
show parameter cluster
set lines 200 pages 500
col owner for a15
col OBJECT_NAME for a35
select owner,object_name,object_type,status from dba_objects where status='INVALID';
col COMP_NAME for a40
select COMP_ID,COMP_NAME,VERSION,STATUS from dba_registry;
col ACTION_TIME for a30
col action for a30
col version for a10
col BUNDLE_SERIES for a30
col COMMENTS for a47
select ACTION_TIME,ACTION,VERSION,BUNDLE_SERIES,COMMENTS from dba_registry_history;
###############
2. Perform Conflict Checks on each nodes before patching
###############
GRID Home
export ORACLE_HOME=/oracle/grid/19c
export PATH=$ORACLE_HOME/bin:$PATH
export PATH=$ORACLE_HOME/OPatch:$PATH
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32904851
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32916816
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32915586
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32918050
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32585572
DB Home
export ORACLE_HOME=/oracle/db/19c
export PATH=$ORACLE_HOME/bin:$PATH
export PATH=$ORACLE_HOME/OPatch:$PATH
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32904851
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32916816
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle//32900083/32876380
###############
3. Run OPatch SystemSpace Check
###############
GRID HOME
Check if enough free space is available on the ORACLE_HOME filesystem for the patches to be applied as given below:
For Grid Infrastructure Home, as home user:
Create file /tmp/patch_list_gihome.txt with the following content:
% cat /tmp/patch_list_gihome.txt
/home/oracle/32900083/32895426/32904851
/home/oracle/32900083/32895426/32916816
/home/oracle/32900083/32895426/32915586
/home/oracle/32900083/32895426/32918050
/home/oracle/32900083/32895426/32585572
Run the opatch command to check if enough free space is available in the Grid Infrastructure Home:
$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt
############
DB HOME
############
Create file /tmp/patch_list_dbhome.txt with the following content:
cat /tmp/patch_list_dbhome.txt
/home/oracle/32900083/32895426/32904851
/home/oracle/32900083/32895426/32916816
/home/oracle/32900083/32876380
Run opatch command to check if enough free space is available in the Database Home:
$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_dbhome.txt
######################################
5. Stop Oracle HOME in rolling manner
######################################
/oracle/db/19c/bin/srvctl stop home -o /oracle/db/19c -s RACNODE1_JUL21_PSU_08Sept.txt -n racnode1
If there are any application specific listeners please stop it.
#######################
6. Unlock GI Stack (AS root user)
#######################
/oracle/grid//19c/crs/install/rootcrs.sh -prepatch
#########################
7. Apply Patches on GI Home in Rolling manner from grid user
#########################
export ORACLE_HOME=/oracle/grid/19c
export PATH=$ORACLE_HOME/bin:$PATH
export PATH=$ORACLE_HOME/OPatch:$PATH
/oracle/grid/19c/OPatch/opatch apply -oh /oracle/grid/19c -local -silent /home/oracle/34762026/34863894
/oracle/grid/19c/OPatch/opatch apply -oh /oracle/grid/19c -local -silent /home/oracle/34762026/34768569
/oracle/grid/19c/OPatch/opatch apply -oh /oracle/grid/19c -local -silent /home/oracle/34762026/34768559
/oracle/grid/19c/OPatch/opatch apply -oh /oracle/grid/19c -local -silent /home/oracle/34762026/34765931
/oracle/grid/19c/OPatch/opatch apply -oh /oracle/grid/19c -local -silent /home/oracle/34762026/33575402
###############
8. Verification
###############
$ORACLE_HOME/OPatch/opatch lsinv | grep -i applied
$ORACLE_HOME/OPatch/opatch lspatches | sort -nr
#################################
9. Run Pre-script for DB Component from oracle user
#################################
/home/oracle/34762026/34765931/custom/scripts/prepatch.sh -dbhome /oracle/db/19c
/home/oracle/34762026/34768559/custom/scripts/prepatch.sh -dbhome /oracle/db/19c
--- Due to patch conflict, If require need to perform rollback of patch ####
##############################
10. Apply Patches on RDBMS Home from oracle user
##############################
/oracle/db/19c/OPatch/opatch apply -oh /oracle/db/19c -local -silent /home/oracle/34762026/34863894
/oracle/db/19c/OPatch/opatch apply -oh /oracle/db/19c -local -silent /home/oracle/34762026/34768569
/oracle/db/19c/OPatch/opatch apply -oh /oracle/db/19c -local -silent /home/oracle/34762026/34768559
/oracle/db/19c/OPatch/opatch apply -oh /oracle/db/19c -local -silent /home/oracle/34762026/34765931
/oracle/db/19c/OPatch/opatch apply -oh /oracle/db/19c -local -silent /home/oracle/34762026/33575402
###############
11. Verification
###############
$ORACLE_HOME/OPatch/opatch lsinv | grep -i applied
$ORACLE_HOME/OPatch/opatch lspatches | sort -nr
#################################
12. Run Post-script for DB Component from Oracle USER
#################################
/home/oracle/34762026/34768559/custom/scripts/postpatch.sh -dbhome /oracle/db/19c
##########################################
13. Post patch rootadd_rdbms script execution from root user:
##########################################
/oracle/grid/19c/rdbms/install/rootadd_rdbms.sh
/oracle/grid/19c/crs/install/rootcrs.sh -postpatch
Issue:
CRS-1705: Found 0 configured voting files but 1 voting files are required, terminating to ensure data integrity; details at (:CSSNM00065:) in /home/oracle/app/oracle/diag/crs/racnode1/crs/trace/ocssd.trc
CRS-2883: Resource 'ora.cssd' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-41053: checking Oracle Grid Infrastructure for file permission issues
CRS-4000: Command Start failed, or completed with errors.
2023/05/05 11:06:05 CLSRSC-117: Failed to start Oracle Clusterware stack from the Grid Infrastructure home /oracle/grid/19c
Died at /oracle/grid/19c/crs/install/crspatch.pm line 1899.
The command '/oracle/grid/19c/perl/bin/perl -I/oracle/grid/19c/perl/lib -I/oracle/grid/19c/crs/install -I/oracle/grid/19c/xag /oracle/grid/19c/crs/install/rootcrs.pl -postpatch' execution failed
###############
14. Start DB Homes from oracle user
###############
/oracle/db/19c/bin/srvctl start home -o /oracle/db/19c -s RACNODE1_JUL21_PSU_08Sept.txt -n racnode1
If there are any application specific listeners please start it.
###########################
15. Perform on NODE2
###########################
Preform above Steps from 5 to 14 on second node.
################
16. Verify below for Cluster status post patching all nodes
######
ps -ef | grep pmon
ps -ef | grep tns
ps -ef | grep d.bin
ps -ef | grep mrp
/oracle/grid/19c/bin/kfod op=patches
/oracle/grid/19c/bin/kfod op=patchlvl
/oracle/grid/19c/bin/crsctl query crs activeversion -f
/oracle/grid/19c/bin/crsctl query crs softwarepatch
/oracle/grid/19c/bin/crsctl stat res -t
export ORACLE_HOME=/oracle/grid/19c
export PATH=/oracle/grid/19c/bin:$PATH
export PATH=/oracle/grid/19c/OPatch:$PATH
opatch lsinv
opatch lspatches
opatch lsinv | grep -i applied
########
17. Perform Post Patch Task
###########
Post patching task only on prod after completion of patching on all nodes:
cd /oracle/db/19c/OPatch
./datapatch -verbose
##############################
18. Validate the result
##############################
select name,open_mode,database_role from V$database;
show parameter cluster
set lines 200 pages 500
col owner for a15
col OBJECT_NAME for a35
select owner,object_name,object_type,status from dba_objects where status='INVALID';
col COMP_NAME for a40
select COMP_ID,COMP_NAME,VERSION,STATUS from dba_registry;
col ACTION_TIME for a30
col action for a30
col version for a10
col BUNDLE_SERIES for a30
col COMMENTS for a47
select ACTION_TIME,ACTION,VERSION,BUNDLE_SERIES,COMMENTS from dba_registry_history;
#############################################
