Patching in 19C RAC Environment

Download: Patch 34762026: GI RELEASE UPDATE 19.18.0.0.0 (REL-JAN230131)

 

###############

1.  Perform pre-checks 

###############

ps -ef | grep pmon

ps -ef | grep tns

ps -ef | grep d.bin

ps -ef | grep mrp

/oracle/grid/19c/bin/kfod op=patches #Check applied patches

/oracle/grid/19c/bin/kfod op=patchlvl  #Check Current Patch Level

/oracle/grid/19c/bin/crsctl query crs activeversion -f #Check active version

/oracle/grid/19c/bin/crsctl query crs softwarepatch

/oracle/grid/19c/bin/crsctl stat res -t

From Grid Home

export ORACLE_HOME=/oracle/grid/19c

export PATH=$ORACLE_HOME/bin:$PATH

export PATH=$ORACLE_HOME/OPatch:$PATH

opatch lsinv

opatch lspatches  

opatch lsinv | grep -i applied

opatch lspatches | sort -nr

From DB Home

srvctl status database -d PROD

select name,open_mode,database_role from v$database;

export ORACLE_HOME=/oracle/db/19c

export PATH=/oracle/db/19c/bin:$PATH

export PATH=/oracle/db/19c/OPatch:$PATH

opatch lsinv

opatch lspatches  

opatch lsinv | grep -i applied

On ORACLE_HOME validate the database component 

select name,open_mode,database_role from V$database;

show parameter cluster

set lines 200 pages 500

col owner for a15

col OBJECT_NAME for a35

select owner,object_name,object_type,status from dba_objects where status='INVALID';

col COMP_NAME for a40

select COMP_ID,COMP_NAME,VERSION,STATUS from dba_registry;

col ACTION_TIME for a30

col action for a30

col version for a10

col BUNDLE_SERIES for a30

col COMMENTS for a47

select ACTION_TIME,ACTION,VERSION,BUNDLE_SERIES,COMMENTS from dba_registry_history;

###############

2. Perform Conflict Checks on each nodes before patching

###############

GRID Home

export ORACLE_HOME=/oracle/grid/19c

export PATH=$ORACLE_HOME/bin:$PATH

export PATH=$ORACLE_HOME/OPatch:$PATH

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32904851

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32916816

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32915586

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32918050

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32585572

DB Home

export ORACLE_HOME=/oracle/db/19c

export PATH=$ORACLE_HOME/bin:$PATH

export PATH=$ORACLE_HOME/OPatch:$PATH

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32904851

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/32900083/32895426/32916816

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle//32900083/32876380

###############

3. Run OPatch SystemSpace Check

###############

GRID HOME

Check if enough free space is available on the ORACLE_HOME filesystem for the patches to be applied as given below:

For Grid Infrastructure Home, as home user:

Create file /tmp/patch_list_gihome.txt with the following content:

% cat /tmp/patch_list_gihome.txt

 /home/oracle/32900083/32895426/32904851

 /home/oracle/32900083/32895426/32916816

 /home/oracle/32900083/32895426/32915586

 /home/oracle/32900083/32895426/32918050

 /home/oracle/32900083/32895426/32585572

Run the opatch command to check if enough free space is available in the Grid Infrastructure Home:

$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt

############

DB HOME

############

Create file /tmp/patch_list_dbhome.txt with the following content:

cat /tmp/patch_list_dbhome.txt

 /home/oracle/32900083/32895426/32904851

 /home/oracle/32900083/32895426/32916816

 /home/oracle/32900083/32876380

Run opatch command to check if enough free space is available in the Database Home:

$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_dbhome.txt

######################################

5. Stop Oracle HOME in rolling manner 

######################################

/oracle/db/19c/bin/srvctl stop home -o /oracle/db/19c -s RACNODE1_JUL21_PSU_08Sept.txt -n racnode1

If there are any application specific listeners please stop it.

#######################

6. Unlock GI Stack (AS root user)

#######################

/oracle/grid//19c/crs/install/rootcrs.sh -prepatch

#########################

7.  Apply Patches on GI Home in Rolling manner from grid user

#########################

export ORACLE_HOME=/oracle/grid/19c

export PATH=$ORACLE_HOME/bin:$PATH

export PATH=$ORACLE_HOME/OPatch:$PATH

/oracle/grid/19c/OPatch/opatch apply -oh /oracle/grid/19c -local -silent /home/oracle/34762026/34863894

/oracle/grid/19c/OPatch/opatch apply -oh /oracle/grid/19c -local -silent /home/oracle/34762026/34768569

/oracle/grid/19c/OPatch/opatch apply -oh /oracle/grid/19c -local -silent /home/oracle/34762026/34768559

/oracle/grid/19c/OPatch/opatch apply -oh /oracle/grid/19c -local -silent /home/oracle/34762026/34765931

/oracle/grid/19c/OPatch/opatch apply -oh /oracle/grid/19c -local -silent /home/oracle/34762026/33575402

###############

8. Verification

###############

$ORACLE_HOME/OPatch/opatch lsinv | grep -i applied

$ORACLE_HOME/OPatch/opatch lspatches | sort -nr

#################################

9. Run Pre-script for DB Component from oracle user

#################################

/home/oracle/34762026/34765931/custom/scripts/prepatch.sh -dbhome /oracle/db/19c

/home/oracle/34762026/34768559/custom/scripts/prepatch.sh -dbhome /oracle/db/19c

--- Due to patch conflict, If require need to perform rollback of patch ####

##############################

10. Apply Patches on RDBMS Home from oracle user

##############################

/oracle/db/19c/OPatch/opatch apply -oh /oracle/db/19c -local -silent /home/oracle/34762026/34863894

/oracle/db/19c/OPatch/opatch apply -oh /oracle/db/19c -local -silent /home/oracle/34762026/34768569

/oracle/db/19c/OPatch/opatch apply -oh /oracle/db/19c -local -silent /home/oracle/34762026/34768559

/oracle/db/19c/OPatch/opatch apply -oh /oracle/db/19c -local -silent /home/oracle/34762026/34765931

/oracle/db/19c/OPatch/opatch apply -oh /oracle/db/19c -local -silent /home/oracle/34762026/33575402

###############

11. Verification

###############

$ORACLE_HOME/OPatch/opatch lsinv | grep -i applied

$ORACLE_HOME/OPatch/opatch lspatches | sort -nr

#################################

12. Run Post-script for DB Component from Oracle USER

#################################

/home/oracle/34762026/34768559/custom/scripts/postpatch.sh -dbhome /oracle/db/19c

##########################################

13. Post patch rootadd_rdbms script execution from root user:

##########################################

/oracle/grid/19c/rdbms/install/rootadd_rdbms.sh

/oracle/grid/19c/crs/install/rootcrs.sh -postpatch

Issue:

CRS-1705: Found 0 configured voting files but 1 voting files are required, terminating to ensure data integrity; details at (:CSSNM00065:) in /home/oracle/app/oracle/diag/crs/racnode1/crs/trace/ocssd.trc

CRS-2883: Resource 'ora.cssd' failed during Clusterware stack start.

CRS-4406: Oracle High Availability Services synchronous start failed.

CRS-41053: checking Oracle Grid Infrastructure for file permission issues

CRS-4000: Command Start failed, or completed with errors.

2023/05/05 11:06:05 CLSRSC-117: Failed to start Oracle Clusterware stack from the Grid Infrastructure home /oracle/grid/19c

Died at /oracle/grid/19c/crs/install/crspatch.pm line 1899.

The command '/oracle/grid/19c/perl/bin/perl -I/oracle/grid/19c/perl/lib -I/oracle/grid/19c/crs/install -I/oracle/grid/19c/xag /oracle/grid/19c/crs/install/rootcrs.pl -postpatch' execution failed

###############

14. Start DB Homes from oracle user

###############

/oracle/db/19c/bin/srvctl start home -o /oracle/db/19c -s RACNODE1_JUL21_PSU_08Sept.txt  -n racnode1 

If there are any application specific listeners please start it.

###########################

15. Perform on NODE2

###########################

Preform above Steps from 5 to 14 on second node.

################

16. Verify below for Cluster status post patching all nodes 

######

ps -ef | grep pmon

ps -ef | grep  tns

ps -ef | grep d.bin

ps -ef | grep mrp

/oracle/grid/19c/bin/kfod op=patches

/oracle/grid/19c/bin/kfod op=patchlvl

/oracle/grid/19c/bin/crsctl query crs activeversion -f

/oracle/grid/19c/bin/crsctl query crs softwarepatch

/oracle/grid/19c/bin/crsctl stat res -t

export ORACLE_HOME=/oracle/grid/19c

export PATH=/oracle/grid/19c/bin:$PATH

export PATH=/oracle/grid/19c/OPatch:$PATH

opatch lsinv

opatch lspatches  

opatch lsinv | grep -i applied

########

17. Perform Post Patch Task 

###########

Post patching task only on prod after completion of patching on all nodes:

cd /oracle/db/19c/OPatch

./datapatch -verbose

##############################

18. Validate the result

##############################

select name,open_mode,database_role from V$database;

show parameter cluster

set lines 200 pages 500

col owner for a15

col OBJECT_NAME for a35

select owner,object_name,object_type,status from dba_objects where status='INVALID';

col COMP_NAME for a40

select COMP_ID,COMP_NAME,VERSION,STATUS from dba_registry;

col ACTION_TIME for a30

col action for a30

col version for a10

col BUNDLE_SERIES for a30

col COMMENTS for a47

select ACTION_TIME,ACTION,VERSION,BUNDLE_SERIES,COMMENTS from dba_registry_history;

#############################################