CSS is not coming up after server reboot.

ps -ef|grep d.bin

grid 7935 1 1 04:54 ? 00:00:00 /u002/app/12.2.0/grid/bin/ohasd.bin reboot
grid 10242 1 3 04:55 ? 00:00:00 /u002/app/12.2.0/grid/bin/oraagent.bin
grid 10264 1 1 04:55 ? 00:00:00 /u002/app/12.2.0/grid/bin/evmd.bin
grid 10271 1 3 04:55 ? 00:00:00 /u002/app/12.2.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid 10751 10264 0 04:55 ? 00:00:00 /u002/app/12.2.0/grid/bin/evmlogger.bin -o /u002/app/12.2.0/grid/log/[HOSTNAME]/evmd/evmlogger.info -l /u002/app/12.2.0/grid/log/[HOSTNAME]/evmd/evmlogger.log

ASM will not come up due to CSS is down

sqlplus “/as sysasm”
SQL*Plus: Release 12.2.0.1.0 Production on Wed Feb 20 04:56:00 2019
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
ORA-29701: unable to connect to Cluster Synchronization Service
SQL> exit

Now check the stat of all and its look good

crsctl stat res -p

NAME=ora.asm

TYPE=ora.asm.type
ACL=owner:grid:rwx,pgrp:oinstall:r-x,other::r–
ACTIONS=
ACTION_SCRIPT=
ACTION_TIMEOUT=60
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ASM_DISKSTRING=/dev/oracle/,AFD:
AUTO_START=restore
CHECK_INTERVAL=1
CHECK_TIMEOUT=30
CLEAN_TIMEOUT=60
CSS_CRITICAL=no
DEGREE=1
DELETE_TIMEOUT=60
DESCRIPTION=Oracle ASM resource
ENABLED=1
GEN_USR_ORA_INST_NAME=+ASM
IGNORE_TARGET_ON_FAILURE=no
INSTANCE_FAILOVER=1
INTERMEDIATE_TIMEOUT=0
LOAD=1
LOGGING_LEVEL=1
MODIFY_TIMEOUT=60
NLS_LANG=
OFFLINE_CHECK_INTERVAL=0
OS_CRASH_THRESHOLD=0
OS_CRASH_UPTIME=0
PRESENCE=standard
PWFILE=+OCRVOTE/orapwasm
REGISTERED_TYPE=srvctl
RESOURCE_GROUP=
RESTART_ATTEMPTS=5
RESTART_DELAY=0
SCRIPT_TIMEOUT=60
SERVER_CATEGORY=
SPFILE=+OCRVOTE/ASM/ASMPARAMETERFILE/registry.253.962081145
START_CONCURRENCY=0
START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)
START_TIMEOUT=900
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=hard(ora.cssd)
STOP_TIMEOUT=600
TARGET_DEFAULT=default
TYPE_VERSION=1.2
UPTIME_THRESHOLD=1d
USER_WORKLOAD=no
USR_ORA_ENV=
USR_ORA_INST_NAME=+ASM
USR_ORA_OPEN_MODE=mount
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
WORKLOAD_CPU=0
WORKLOAD_CPU_CAP=0
WORKLOAD_MEMORY_MAX=0
WORKLOAD_MEMORY_TARGET=0

ASM and all disk group, offline

crsctl stat res -t

Name Target State Server State details

Local Resources

ora.DATA3.dg
ONLINE OFFLINE testd01 STABLE
ora.FRA3.dg
ONLINE OFFLINE testd01 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE testd01 STABLE
ora.OCRVOTE.dg
OFFLINE OFFLINE testd01 STABLE
ora.asm
ONLINE OFFLINE testd01 Instance Shutdown,ST
ABLE
ora.ons

OFFLINE OFFLINE testd01 STABLE

Cluster Resources

ora.cssd
1 ONLINE OFFLINE STABLE

trying to start manually but it did’nt work and no luck

crsctl start resource ora.cssd

CRS-4995: The command ‘Start resource’ is invalid in crsctl. Use srvctl for this command.

(testd01-grid-+ASM) /home/grid

here trying start manully only ASM and CONFIRMED AFD driver issue

srvctl start asm

PRCR-1079 : Failed to start resource ora.asm
CRS-2674: Start of ‘ora.driver.afd’ on ‘testd01’ failed

asmcmd afd_lsdsk

ASMCMD-9520: AFD is not ‘Loaded’

Stop HAS

crsctl stop has

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘testd01’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘testd01’
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘testd01’ succeeded
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘testd01’
CRS-2677: Stop of ‘ora.evmd’ on ‘testd01’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘testd01’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
(testd01-grid-+ASM) /u002/app/grid/diag/asm/+asm/+ASM

tried below commnd from Grid user

:->/u002/app/12.2.0/grid/bin/asmcmd afd_configure
ASMCMD-9522: Insufficient permission to execute the command. Require privileged user

need to run as root with grid ENV SETUP.. Before running please discuss with TEAM and Sreejith

[root@testd01 ~]

# /u002/app/12.2.0/grid/bin/asmcmd afd_configure
ASMCMD-00001: ORACLE_HOME environment variable not set

# ./asmcmd afd_configure

Dont try to run below command always…Overview of what has to be done

[root@testd01 ~]

# export ORACLE_HOME=/u002/app/12.2.0/grid

[root@testd01 ~]

# export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH

[root@testd01 ~]

# export ORACLE_SID=+ASM

[root@testd01 ~]

# $ORACLE_HOME/bin/asmcmd afd_configure
AFD-627: AFD distribution files found.
AFD-632: Existing AFD installation detected.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies – this may take some time.
AFD-9154: Loading ‘oracleafd.ko’ driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device ‘/dev/oracleafd/admin’.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies – this may take some time.

Acutal output shluld be like this

Load AFD:

/u01/app/12.1.0/grid/bin/asmcmd afd_configure

Connected to an idle instance.
AFD-627: AFD distribution files found.
AFD-632: Existing AFD installation detected.
AFD-634: Removing previous AFD installation.
AFD-635: Previous AFD components successfully removed.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies – this may take some time.
AFD-9154: Loading ‘oracleafd.ko’ driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device ‘/dev/oracleafd/admin’.
AFD-638: AFD installation correctness verified.

Now started the HAS
:->ps -ef|grep d.bin
grid 1562 1516 0 05:41 pts/0 00:00:00 grep –color=auto d.bin
(testd01-grid-+ASM) /home/grid
:->crsctl start has
CRS-4123: Oracle High Availability Services has been started.
(testd01-grid-+ASM) /home/grid
:->crsctl check has
CRS-4638: Oracle High Availability Services is online
(testd01-grid-+ASM) /home/grid
:->sqlplus “/as sysasm

^C
(testd01-grid-+ASM) /home/grid
:->ps -ef|grep d.bin
grid 1726 1 2 05:42 ? 00:00:00 /u002/app/12.2.0/grid/bin/ohasd.bin reboot
grid 1814 1 0 05:42 ? 00:00:00 /u002/app/12.2.0/grid/bin/orarootagent.bin
grid 1896 1 0 05:42 ? 00:00:00 /u002/app/12.2.0/grid/bin/oraagent.bin
grid 1918 1 0 05:42 ? 00:00:00 /u002/app/12.2.0/grid/bin/evmd.bin
grid 1927 1 4 05:42 ? 00:00:01 /u002/app/12.2.0/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
grid 1977 1918 0 05:42 ? 00:00:00 /u002/app/12.2.0/grid/bin/evmlogger.bin -o /u002/app/12.2.0/grid/log/[HOSTNAME]/evmd/evmlogger.info -l /u002/app/12.2.0/grid/log/[HOSTNAME]/evmd/evmlogger.log
grid 1998 1 0 05:42 ? 00:00:00 /u002/app/12.2.0/grid/bin/cssdagent
grid 2011 1 0 05:42 ? 00:00:00 /u002/app/12.2.0/grid/bin/ocssd.bin
grid 2253 1516 0 05:42 pts/0 00:00:00 grep –color=auto d.bin
(testd01-grid-+ASM) /home/grid
:->sqlplus “/as sysasm”
SQL*Plus: Release 12.2.0.1.0 Production on Wed Feb 20 05:42:49 2019
Copyright (c) 1982, 2016, Oracle. All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 – 64bit Production

:->asmcmd
ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 1048576 4194304 2447858 0 2447858 0 N DATA3/
MOUNTED EXTERN N 512 512 4096 1048576 2097152 2091129 0 2091129 0 N FRA3/
MOUNTED EXTERN N 512 512 4096 1048576 40960 40897 0 40897 0 N OCRVOTE/
ASMCMD> exit
(testd01-grid-+ASM) /home/grid
:->sqlplus “/as sysasm”
SQL*Plus: Release 12.2.0.1.0 Production on Wed Feb 20 05:47:39 2019
Copyright (c) 1982, 2016, Oracle. All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 – 64bit Production
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 – 64bit Production
(testd01-grid-+ASM) /home/grid
:->./crsctl stat res -t
-bash: ./crsctl: No such file or directory
(testd01-grid-+ASM) /home/grid

:->crsctl stat res -t

Name Target State Server State details

Local Resources

ora.DATA3.dg
ONLINE ONLINE testd01 STABLE
ora.FRA3.dg
ONLINE ONLINE testd01 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE testd01 STABLE
ora.OCRVOTE.dg
ONLINE ONLINE testd01 STABLE
ora.asm
ONLINE ONLINE testd01 Started,STABLE

Note: Once got issue/error if feeling difficult to fix without delay please call team members and share info So It will help us to fix issue within downtime window
After Os patch we should ensure ASM and its service up or not.