DRAFT - Oracle GI 12c - Apply - Patch 24412235: GRID INFRASTRUCTURE PATCH SET UPDATE 12.1.0.2.161018 (OCT2016)
------- DRAFT
I am very happy to share my patching process on the testing system, as preparation for production patching.
I have the following architecture:
Two node Oracle RAC Cluster 12c without any PSU - just 12.1.0.2.0 :)
----- full details will be added soon in the next posts.
Start with the patching
- Response file creation
- Apply Patch
Prior to the actual start of the patching, it is very important to check target Patch level and current patch level for possible conflicts by using Oracle Support site/metalink or via opatch tool
I did Metalink checks prior to the actual start and all checks passed successfully.
Also, an important step is to download the latest OPatch tool from Metalink, which will further help you to resolve a lot of issues or conflicts during the patching.
As per the below command, I am going to apply patch in rolling mode. If that is not specified - nonrolling opatchauto will be started in rolling mode by default.
In my case, I am going to resolve a few issues and help you understand how.
As per the below command, I am going to apply patch in rolling mode. If that is not specified - nonrolling opatchauto will be started in rolling mode by default.
In my case, I am going to resolve a few issues and help you understand how.
- Patching
-- Patch should be downloaded from metalink and uploaded to the target server node from where patching process will be started.
-- change directory to the location of the patch
[root@dbabg-rac1 cvutrc]# cd /home/grid/24412235
-- apply patch
Here, it is important to specify that opatchauto commands in all cases are mandatory to be executed as root user!
[root@dbabg-rac1 24412235]# /u01/app/12.1.0/grid/OPatch/opatchauto apply /home/grid/24412235 -oh /u01/app/12.1.0/grid -ocmrf /home/grid/ocm.rsp
OPatchauto session is initiated at Wed Jan 11 06:32:30 2017
System initialization log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2017-01-11_06-32-34AM.log.
Session log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2017-01-11_06-32-51AM.log
WARNING: the option -ocmrf is deprecated and no longer needed. OPatch no longer checks for OCM configuration. It will be removed in a future release.
The id for this session is 6YYX
Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.1.0/grid
Patch applicablity verified successfully on home /u01/app/12.1.0/grid
Verifying patch inventory on home /u01/app/12.1.0/grid
Patch inventory verified successfully on home /u01/app/12.1.0/grid
Bringing down CRS service on home /u01/app/12.1.0/grid
Prepatch operation log file location: /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_dbabg-rac1_2017-01-11_06-35-52A M.log
CRS service brought down successfully on home /u01/app/12.1.0/grid
Start applying binary patch on home /u01/app/12.1.0/grid
Binary patch applied successfully on home /u01/app/12.1.0/grid
Starting CRS service on home /u01/app/12.1.0/grid
Postpatch operation log file location: /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_dbabg-rac1_2017-01-11_06-59-20 AM.log
Failed to start CRS service on home /u01/app/12.1.0/grid
Execution of [GIStartupAction] patch action failed, check log for more details. Failures:
Patch Target : dbabg-rac1->/u01/app/12.1.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/12.1.0/grid, host: dbabg-rac1.
Command failed: /u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/OPatch/auto/dbtm p/bootstrap_dbabg-rac1/patchwork/crs/install /u01/app/12.1.0/grid/OPatch/auto/dbtmp/bootstrap_dbabg-rac1/patchwork/crs/ install/rootcrs.pl -postpatch
Command failure output:
Using configuration parameter file: /u01/app/12.1.0/grid/OPatch/auto/dbtmp/bootstrap_dbabg-rac1/patchwork/crs/install/crs config_params
2017/01/11 06:59:30 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2017/01/11 07:00:18 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'dbabg-rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dbabg-rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.evmd' on 'dbabg-rac1'
CRS-2676: Start of 'ora.mdnsd' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.evmd' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dbabg-rac1'
CRS-2676: Start of 'ora.gpnpd' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'dbabg-rac1'
CRS-2676: Start of 'ora.gipcd' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbabg-rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'dbabg-rac1'
CRS-2676: Start of 'ora.diskmon' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'dbabg-rac1'
CRS-2676: Start of 'ora.ctssd' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'dbabg-rac1'
CRS-2676: Start of 'ora.asm' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'dbabg-rac1'
CRS-2676: Start of 'ora.storage' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'dbabg-rac1'
CRS-2676: Start of 'ora.crf' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'dbabg-rac1'
CRS-2676: Start of 'ora.crsd' on 'dbabg-rac1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2672: Attempting to start 'ora.ons' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.hdbabg-rac2.vip' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.cvu' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.dbabg-rac1.vip' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'dbabg-rac1'
CRS-6017: Processing resource auto-start for servers: dbabg-rac1
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.oc4j' on 'dbabg-rac1'
CRS-2676: Start of 'ora.cvu' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.ons' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.dbabg-rac1.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'dbabg-rac1'
CRS-2676: Start of 'ora.scan2.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'dbabg-rac1'
CRS-2676: Start of 'ora.scan3.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'dbabg-rac1'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.oc4j' on 'dbabg-rac1' succeeded
CRS-5017: The resource action "ora.hdbabg-rac2.vip start" encountered the following error:
CRS-5005: IP Address: 192.168.0.104 is already in use in the network
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/dbabg-rac1/crs/trace/crsd_orarootagent_root.trc".
CRS-2674: Start of 'ora.hdbabg-rac2.vip' on 'dbabg-rac1' failed
CRS-2679: Attempting to clean 'ora.hdbabg-rac2.vip' on 'dbabg-rac1'
CRS-5017: The resource action "ora.scan1.vip start" encountered the following error:
CRS-5005: IP Address: 192.168.0.105 is already in use in the network
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/dbabg-rac1/crs/trace/crsd_orarootagent_root.trc".
CRS-2681: Clean of 'ora.hdbabg-rac2.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.hdbabg-rac2.vip' on 'dbabg-rac1'
CRS-2674: Start of 'ora.scan1.vip' on 'dbabg-rac1' failed
CRS-2679: Attempting to clean 'ora.scan1.vip' on 'dbabg-rac1'
CRS-2681: Clean of 'ora.scan1.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'dbabg-rac1'
CRS-5017: The resource action "ora.hdbabg-rac2.vip start" encountered the following error:
CRS-5005: IP Address: 192.168.0.104 is already in use in the network
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/dbabg-rac1/crs/trace/crsd_orarootagent_root.trc".
CRS-2674: Start of 'ora.hdbabg-rac2.vip' on 'dbabg-rac1' failed
CRS-2679: Attempting to clean 'ora.hdbabg-rac2.vip' on 'dbabg-rac1'
CRS-5017: The resource action "ora.scan1.vip start" encountered the following error:
CRS-5005: IP Address: 192.168.0.105 is already in use in the network
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/dbabg-rac1/crs/trace/crsd_orarootagent_root.trc".
CRS-2681: Clean of 'ora.hdbabg-rac2.vip' on 'dbabg-rac1' succeeded
CRS-2674: Start of 'ora.scan1.vip' on 'dbabg-rac1' failed
CRS-2679: Attempting to clean 'ora.scan1.vip' on 'dbabg-rac1'
CRS-2681: Clean of 'ora.scan1.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'dbabg-rac1'
CRS-5017: The resource action "ora.scan1.vip start" encountered the following error:
CRS-5005: IP Address: 192.168.0.105 is already in use in the network
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/dbabg-rac1/crs/trace/crsd_orarootagent_root.trc".
CRS-2674: Start of 'ora.scan1.vip' on 'dbabg-rac1' failed
CRS-2679: Attempting to clean 'ora.scan1.vip' on 'dbabg-rac1'
CRS-2681: Clean of 'ora.scan1.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'dbabg-rac1'
CRS-5017: The resource action "ora.scan1.vip start" encountered the following error:
CRS-5005: IP Address: 192.168.0.105 is already in use in the network
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/dbabg-rac1/crs/trace/crsd_orarootagent_root.trc".
CRS-2674: Start of 'ora.scan1.vip' on 'dbabg-rac1' failed
CRS-2679: Attempting to clean 'ora.scan1.vip' on 'dbabg-rac1'
CRS-2681: Clean of 'ora.scan1.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'dbabg-rac1'
CRS-5017: The resource action "ora.mgmtdb start" encountered the following error:
ORA-00205: error in identifying control file, check alert log for more info
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/dbabg-rac1/crs/trace/crsd_oraagent_grid.trc".
CRS-2674: Start of 'ora.mgmtdb' on 'dbabg-rac1' failed
CRS-2679: Attempting to clean 'ora.mgmtdb' on 'dbabg-rac1'
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
2017/01/11 07:13:38 CLSRSC-117: Failed to start Oracle Clusterware stack
After fixing the issue you can run opatchauto resume with session id "6YYX"
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Wed Jan 11 07:13:41 2017
Time taken to complete the session 41 minutes, 11 seconds
opatchauto failed with error code 42
[root@dbabg-rac1 24412235]#
---- checking inventory
[grid@dbabg-rac1 OPatch]$ ./opatch lsinv
Oracle Interim Patch Installer version 12.2.0.1.8
Copyright (c) 2017, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/12.1.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/12.1.0/grid/oraInst.loc
OPatch version : 12.2.0.1.8
OUI version : 12.1.0.2.0
Log file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2017-01-11_09-04-56AM_1.log
Lsinventory Output file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2017-01-11_09-04-56AM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: dbabg-rac1.localdomain
ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1):
Oracle Grid Infrastructure 12c 12.1.0.2.0
There are 1 products installed in this Oracle Home.
Interim patches (4) :
Patch 24007012 : applied on Wed Jan 11 06:59:13 EET 2017
Unique Patch ID: 20511653
Patch description: "ACFS Patch Set Update : 12.1.0.2.161018 (24007012)"
Created on 30 Aug 2016, 00:17:17 hrs PST8PDT
Bugs fixed:
19060056, 21369858, 19355146, 19690653, 19526234, 20952520, 19475588
21231953, 18510745, 19149476, 19013966, 22222222, 19859183, 21519796
19469788, 21555038, 18877486, 20438706, 19610022, 19335268, 21058846
22472364, 22319558, 19183802, 19452723, 21124596, 19051391, 20010980
19526283, 22198405, 22689989, 19279106, 19078259, 19644505, 21077218
21233961, 19184398, 18321597, 19811784, 18185024, 22174238, 20098064
19353057, 19502657, 22661787, 19434261, 18951113, 20140148, 18900953
19053182, 22975177, 18955907, 18934139, 19134464, 19450090, 19517835
22241945, 20923224, 19446719, 20402287, 19201087, 21208140, 19649858
21076821, 19428756, 19001684, 19610001, 18899600, 20314907, 19509898
19528981, 18957085, 19127216, 23557970, 19901825, 22810422, 18915417
19270227, 22087780, 22198176, 19919907, 19593769, 23112252, 19557156
19195735, 23252830, 22846612, 20681968, 18851012, 19844362
Patch 24006101 : applied on Wed Jan 11 06:50:26 EET 2017
Unique Patch ID: 20683584
Patch description: "Database Patch Set Update : 12.1.0.2.161018 (24006101)"
Created on 1 Oct 2016, 12:33:50 hrs PST8PDT
Sub-patch 23054246; "Database Patch Set Update : 12.1.0.2.160719 (23054246)"
Sub-patch 22291127; "Database Patch Set Update : 12.1.0.2.160419 (22291127)"
Sub-patch 21948354; "Database Patch Set Update : 12.1.0.2.160119 (21948354)"
Sub-patch 21359755; "Database Patch Set Update : 12.1.0.2.5 (21359755)"
Sub-patch 20831110; "Database Patch Set Update : 12.1.0.2.4 (20831110)"
Sub-patch 20299023; "Database Patch Set Update : 12.1.0.2.3 (20299023)"
Sub-patch 19769480; "Database Patch Set Update : 12.1.0.2.2 (19769480)"
Bugs fixed:
21099555, 22175564, 19141838, 22083366, 20842388, 20117253, 19865345
19791273, 19243521, 20951038, 19908836, 21281532, 19238590, 24577566
21184223, 19134173, 20387265, 19149990, 21263635, 18886413, 17551063
22160989, 22507210, 19703301, 19366375, 19001390, 18202441, 20267166
19358317, 19706965, 18549238, 19068970, 18797519, 22148226, 20825533
19649152, 19670108, 18940497, 18948177, 18964978, 19035573, 19176326
20413820, 19176223, 20904530, 20134339, 19074147, 20868862, 18411216
21322887, 22507234, 20425790, 18966843, 21329301, 19333670, 19468991
20124446, 19883092, 19658708, 19591608, 19402853, 20618595, 21787056
22380919, 21266085, 17835294, 19721304, 19791377, 19068610, 22178855
22173980, 20746251, 20048359, 20898391, 19185876, 20281121, 20907061
6599380, 19577410, 22092979, 20603378, 19001359, 19490948, 21387964
20832516, 17532734, 19309466, 19081128, 20627866, 20844426, 21188532
18791688, 21442094, 20890311, 20596234, 18973548, 21296029, 19303936
19461270, 21479753, 20936905, 20235511, 21220620, 18964939, 19430401
22296366, 21153266, 19409212, 22657942, 20657441, 19879746, 19684504
20528052, 19024808, 20977794, 18799993, 20466322, 18740837, 19662635
20228093, 19065556, 20212067, 19524384, 17722075, 20446883, 18952989
16870214, 19928926, 19835133, 21629064, 20466628, 24386767, 19931709
19730508, 18819908, 23124895, 19188927, 20074391, 20356733, 14643995
19547370, 19065677, 21960504, 21225209, 20397490, 18967382, 19174430
21241829, 19536415, 19171086, 22465352, 22168163, 19335438, 20447445
18856999, 20471920, 19869255, 21620471, 18990693, 17890099, 18990023
20101006, 21300341, 20848335, 21744290, 20897759, 21668627, 19304354
19052488, 20794034, 23260854, 18681056, 20952966, 19896336, 18618122
20328248, 20440930, 18456643, 19699191, 19201867, 22865673, 18743542
20798891, 20347562, 22551446, 19777862, 19687159, 21373076, 19174942
20424899, 21899588, 18899974, 20598042, 19032777, 19058490, 22815955
19399918, 19434529, 21273804, 19018447, 22757364, 18851894, 19284031
19022470, 18043064, 20173897, 22062026, 20475845, 17274537, 19440586
16887946, 22374754, 17319928, 20708701, 17655240, 16439813, 19805359
19155797, 20859910, 19393542, 22024071, 17210525, 21847223, 19189525
21649497, 19075256, 22762046, 22075064, 19280225, 18845653, 20560611
19248799, 21756699, 18988834, 20245930, 18921743, 18799063, 20373598
20476175, 19571367, 20925795, 19018206, 20509482, 20711718, 20588502
18849537, 19183343, 21917884, 19189317, 19644859, 19390567, 19279273
20669434, 16863642, 22528741, 19619732, 18607546, 20348653, 19315691
19676905, 20165574, 17867700, 20558005, 20734332, 19532017, 20922010
19450314, 22353346, 20361671, 20009833, 22366558, 20294666, 18191823
19307662, 19371175, 19195895, 20043616, 19154375, 18914624, 20139391
21291274, 19174521, 19520602, 19382851, 21875360, 19676012, 19326908
20217801, 20093776, 21097043, 21246723, 21665897, 19143550, 20428621
19627012, 14283239, 19518079, 18610915, 18674024, 18306996, 19524158
19915271, 20122715, 20284155, 20017509, 19363645, 19597439, 21239530
19888853, 20880215, 21756677, 19534363, 19354335, 19044962, 19639483
22353199, 22243719, 22916353, 20378086, 21756661, 21260431, 22923409
20877664, 19028800, 20879889, 19723336, 19077215, 21421886, 19604659
19308965, 19048007, 18288842, 19689979, 21526048, 19180770, 19197175
19902195, 20318889, 19013183, 19012119, 20464614, 19067244, 21632821
19512341, 19841800, 20331945, 19587324, 24316947, 19578350, 19637186
18674047, 19054077, 20898997, 19708632, 21091431, 19289642, 20869721
19258504, 17365043, 19468347, 21373473, 16359751, 19439759, 19769480
19272708, 19978542, 20402832, 19329654, 19873610, 23229229, 21517440
13542050, 19291380, 21915719, 19076343, 19561643, 19990037, 19487147
18909599, 20831538, 18250893, 19016730, 16619249, 18354830, 21188584
19989009, 17414008, 20688221, 20704450, 20441797, 19157754, 18885870
21785691, 21450666, 18893947, 18705806, 22223463, 16923858, 18417036
20919320, 20474192, 22046677, 19385656, 19501299, 20920911, 20899461
21387128, 21315084, 18122373, 20581111, 19606174, 18436647, 19023822
19178851, 19124589, 19597583, 18499088, 19050649
Patch 23854735 : applied on Wed Jan 11 06:42:32 EET 2017
Unique Patch ID: 20511532
Patch description: "OCW Patch Set Update : 12.1.0.2.161018 (23854735)"
Created on 29 Sep 2016, 23:50:00 hrs PST8PDT
Bugs fixed:
21339083, 18589889, 20768643, 19061429, 19133945, 19341538, 19521169
20011424, 19361757, 19187207, 19302350, 19130141, 19530755, 19699720
19620918, 19168690, 18899171, 19244316, 19653795, 18634372, 19471722
19536337, 19184188, 19131709, 20160490, 20995001, 21131037, 19380733
20985167, 18943696, 20552947, 19591984, 18135723, 18461246, 20014326
19524857, 18890943, 19154753, 19499021, 19046190, 20923930, 20028666
19051385, 19050688, 19682695, 19831611, 21607972, 18922918, 21087799
18956780, 19683886, 23757020, 19026993, 20110156, 21269977, 18261648
18362971, 19558324, 22651583, 20185476, 18882642, 19478119, 19414274
17447588, 19262534, 20752167, 19147513, 22475834, 21242118, 19514085
20758997, 19659885, 19319904, 18703978, 23079844, 19700256, 19703246
20467806, 19192901, 21527118, 19479503, 19147751, 18901356, 21232394
18964974, 22162062, 18835366, 19013789, 19207286, 18950232, 22583325
19148791, 19556820, 16505171, 22261732, 22135419, 19449737, 21878613
21646169, 19187515, 19230771, 20795241, 19453778, 19853036, 18843572
18520351, 21430567, 19185148, 19232454, 21220597, 18541110, 18834955
19204743, 20175174, 19140891, 18747335, 20676340, 19270660, 19325701
20877003, 19222693, 19662663, 19684131, 19010177, 19272663, 18910576
18907170, 19426384, 9372680, 20737462, 19700294, 20289828, 19331454
22568500, 20278216, 20827053, 20119857, 19703199, 22456393, 20588164
18955644, 20991879, 18508710, 19908346, 19589221, 18952857, 19537762
21152052, 16286734, 20669114, 18762843, 18945249, 19184799, 19205086
19351363, 20091753, 19031737, 19079087, 19148367, 18968981, 19150517
20231741, 19217019, 18730096, 18975620, 19067804, 21259359, 19054979
19209951, 19318983, 18747382, 19154673, 18752378, 22905714, 19150088
19013444, 19234177, 18999857, 19075747, 19367276, 19612597, 19874047
19677813, 23049292, 19427050, 18910443, 20033787, 19148982, 19777496
20730170, 18523468, 19277814, 20562133, 19071526, 21420753, 18850051
20433932, 20061168, 19370739, 20822591, 19609388, 18776786, 20136892
18989446, 19148793, 19585454, 18317489, 19955755, 19059053, 19807548
19124972, 22229116, 18910748, 19147509, 18849896, 19617592, 19398098
22840085, 19076165, 19498411, 19704993, 21294273, 19459023, 20634511
21229744, 21474380, 21484367, 20276459, 17208793, 19234907, 19538714
22867050, 19383028, 19513969, 22195657, 23111368, 19986391, 19314048
18834934, 19473851, 19458082, 18894342, 19522067, 18116578, 19241857
19076778, 19066699, 19068003, 18937186, 20267877, 19916737, 19770543
22863466, 19635215, 19613533, 21057239, 20783416, 19315522, 19141785
19163887, 19885321, 18715868, 18852058, 19701617, 19280860, 19139608
21453435, 18946768, 19135521, 18604143, 19627476, 20568884, 20317221
20177779, 19266658, 24522717, 18330979, 19027351, 21227730, 18707416
21118157, 21113068, 19925992, 20235486, 20006646, 15922206, 18439295
18991776, 21245437, 23170839, 19163425, 19550195, 18849021, 18861196
17940721, 18524834, 19287706, 19522313, 18835283, 18748932, 19184765
21639478, 14851828, 20598625, 19226141, 19053891, 18871287, 18998228
18980002, 19682778, 19325604, 18777835, 17338864, 18945906, 19513650
19702758, 18952577, 19473088, 19178517, 18729269, 19529729, 19455563
20877072, 23275930, 20340620, 18536826, 19292605, 22264951, 20920711
20660273, 20011635, 17592037, 18935857, 22989069, 19029647, 19179158
22302421, 23076471, 19140712, 20012766, 23049301, 19184276, 20510208
20001507, 20079414, 19680763, 19259765, 18962892, 19513888, 14661880
21519340, 21222147, 20218012, 19551830, 21616278, 19068333, 18945435
20775015, 19319192, 20839581, 19178629, 21694237, 19304104, 19284357
22594310, 20725084, 19457575, 21297657, 19069755, 19021575, 18715884
19584688, 18798573, 19812592, 19018001, 19292272, 19270956, 21967791
21794779, 23055224, 18700893, 21276514, 20467377, 18406774, 20607821
21976167, 21082737, 21694632, 20115586, 20883009, 19164099, 18283792
19155350, 19884033, 19146822, 18798432, 21123424, 19471836, 20936562
18750781, 19045143, 22483804, 19146980, 19601003, 20099624, 21378016
19537547, 19281106, 18862203, 19954372, 18235274, 19205617, 19513351
20219458, 19150313, 18843054, 18708349, 19332386, 18953639, 21423055
19371270, 19203996, 20038431, 19370846, 21053036, 18998379, 20157569
21357522, 19273577, 24709351, 22113170, 19632437, 18990354, 19288396
20620581, 19557558, 19127078, 21792881, 20900588, 20053557, 22088198
22664609, 19315567, 19444471, 19154933, 20953798, 18290252, 18813323
19915945, 19500293, 18643483, 19134098, 23034352, 18965694, 19226858
19591356, 19602208, 22884308, 20069197, 18417590, 18920408, 18636884
22289350, 19043795, 20746507, 18260170, 18919682, 18678829, 19319357
19273758, 18953878, 18999195, 18759724, 19066844, 13843841, 20290467
22934855, 19649640, 19062675, 21924910, 18859710, 20862812, 19504641
23101611, 19341481, 20293730, 19343245, 18304090, 19241655, 19470791
18242738, 20649573, 18372060, 18953889, 18827679, 19140711, 19023430
19259290, 19045388, 20186278, 19522571, 19709663, 18875012, 18861564
19273760, 19225265, 15885689, 20853464, 21696068, 18819158, 21394146
19049721, 19368917, 21255373, 22393909, 20408163, 20579351, 18868829
20350915, 21327402, 22024217, 23101338, 19820247, 20363708, 19538241
19804032
Patch 21436941 : applied on Wed Jan 11 06:37:11 EET 2017
Unique Patch ID: 19245012
Patch description: "WLM Patch Set Update: 12.1.0.2.5 (21436941)"
Created on 13 Aug 2015, 04:00:40 hrs PST8PDT
Bugs fixed:
20075154, 20641027, 20548410, 19016964, 19582630
Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
197506932 dbabg-rac1,hdbabg-rac2
--------------------------------------------------------------------------------
OPatch succeeded.
According to the first node, the Patching level on the nodes is the same - 197506932. This is very important to be equal on all nodes.
[grid@dbabg-rac1 OPatch]$
----
Prior to the non-rolling patches:
[root@hdbabg-rac2 24412235]# /u01/app/12.1.0/grid/OPatch/opatchauto apply /home/grid/24412235 -oh /u01/app/12.1.0/grid -ocmrf /home/grid/ocm.rsp
OPatchauto session is initiated at Wed Jan 11 08:59:35 2017
System initialization log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2017-01-11_08-59-39AM.log.
Session log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2017-01-11_08-59-59AM.log
WARNING: the option -ocmrf is deprecated and no longer needed. OPatch no longer checks for OCM configuration. It will be removed in a future release.
The id for this session is TTXZ
Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.1.0/grid
Patch applicablity verified successfully on home /u01/app/12.1.0/grid
Verifying patch inventory on home /u01/app/12.1.0/grid
Patch inventory verified successfully on home /u01/app/12.1.0/grid
Bringing down CRS service on home /u01/app/12.1.0/grid
Prepatch operation log file location: /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_hdbabg-rac2_2017-01-11_09-03-04AM.log
Failed to bring down CRS service on home /u01/app/12.1.0/grid
Execution of [GIShutDownAction] patch action failed, check log for more details. Failures:
Patch Target : hdbabg-rac2->/u01/app/12.1.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/12.1.0/grid, host: hdbabg-rac2.
Command failed: /u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/OPatch/auto/dbtmp/bootstrap_hdbabg-rac2/patchwork/crs/install /u01/app/12.1.0/grid/OPatch/autohdbabg-rac2/patchwork/crs/install/rootcrs.pl -prepatch
Command failure output:
Using configuration parameter file: /u01/app/12.1.0/grid/OPatch/auto/dbtmp/bootstrap_hdbabg-rac2/patchwork/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.storage' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.storage' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.evmd' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.asm' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.crf' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'hdbabg-rac2' succeeded
CRS-2675: Stop of 'ora.asm' on 'hdbabg-rac2' failed
CRS-2679: Attempting to clean 'ora.asm' on 'hdbabg-rac2'
CRS-2681: Clean of 'ora.asm' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.cssd' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.gipcd' on 'hdbabg-rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'hdbabg-rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'hdbabg-rac2'
CRS-2672: Attempting to start 'ora.evmd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.mdnsd' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.evmd' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.gpnpd' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.gipcd' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hdbabg-rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.diskmon' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'hdbabg-rac2'
CRS-2672: Attempting to start 'ora.ctssd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.ctssd' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.asm' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.storage' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.crf' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.crsd' on 'hdbabg-rac2' succeeded
CRS-6017: Processing resource auto-start for servers: hdbabg-rac2
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.net1.network' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.net1.network' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.MGMTLSNR' on 'hdbabg-rac2' succeeded
CRS-2679: Attempting to clean 'ora.mgmtdb' on 'hdbabg-rac2'
CRS-2681: Clean of 'ora.mgmtdb' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.mgmtdb' on 'hdbabg-rac2' succeeded
CRS-6016: Resource auto-start has completed for server hdbabg-rac2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [197506932].
PROC-28: Oracle Cluster Registry already in current version
CRS-1153: There was an error setting Oracle Clusterware to rolling patch mode.
CRS-4000: Command Start failed, or completed with errors.
2017/01/11 09:16:51 CLSRSC-430: Failed to start rolling patch mode
After fixing the cause of the failure, run opatchauto resume with session id "TTXZ"
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Wed Jan 11 09:17:00 2017
Time taken to complete the session 17 minutes, 26 seconds
opatchauto failed with error code 42
[root@hdbabg-rac2 24412235]#
--- In 12c there is one very important feature called patching session, by having that you can go back to the stage where the patching stopped and resume it from there.
---- I will prepare a small demo and test it to validate
--- Where to find session information and how this hasn't been recorded from oracle
---- go to session directory
[grid@dbabg-rac1 bin]$ cd /u01/app/12.1.0/grid/OPatch/auto/dbsessioninfo/
--- check for any existing session - this is only testing, but for the training purposes we are going to view one
[grid@dbabg-rac1 dbsessioninfo]$ ll
total 40
-rw-r--r--. 1 root root 2684 Jan 11 07:13 6YYX.json
-rw-r--r--. 1 root root 2161 Jan 11 00:34 7SCI.json
-rw-r--r--. 1 root root 1450 Jan 11 06:29 EZGW.json
-rw-r--r--. 1 root root 1450 Jan 11 06:25 FRN1.json
-rw-r--r--. 1 root root 1450 Jan 11 00:31 IBUE.json
-rw-r--r--. 1 grid oinstall 1 Jan 11 06:32 lastsessioninfo_dbabg-rac1.txt
-rw-r--r--. 1 grid oinstall 4 Jan 11 06:32 localSessionInfoFile_dbabg-rac1.txt
-rw-r--r--. 1 root root 1445 Jan 11 06:22 LR18.json
-rwxrwxr-x. 1 root root 1532 Jan 11 07:13 patchingsummary.xml
-rwxr-x---. 1 grid oinstall 0 Sep 27 03:22 README.txt
-rw-r--r--. 1 root root 2684 Jan 11 01:08 YX56.json
---- checking one session structure
[grid@dbabg-rac1 dbsessioninfo]$ cat YX56.json
{
"executions" : {
"execution" : [ {
"id" : 1,
"date" : 1484088444031
} ]
},
"options" : {
"option" : [ {
"key" : "patch.location",
"value" : "/home/grid/20091753"
}, {
"key" : "home",
"value" : "/u01/app/12.1.0/grid"
}, {
"key" : "inventory.pointer.location",
"value" : "/u01/app/12.1.0/grid/oraInst.loc"
}, {
"key" : "host",
"value" : "dbabg-rac1"
}, {
"key" : "patch.plan",
"value" : "crs-rolling"
}, {
"key" : "ocmrf",
"value" : "/home/grid/ocm.rsp"
}, {
"key" : "nonrolling",
"value" : "false"
}, {
"key" : "session",
"value" : "YX56"
} ]
},
"targets" : {
"target" : [ {
"type" : "home",
"product-type" : "default",
"home" : "/u01/app/12.1.0/grid",
"host" : "dbabg-rac1"
}, {
"actions" : {
"goal" : [ {
"action" : [ {
"name" : "com.oracle.glcm.patch.auto.db.integration.controller.action.OPatchAutoBinaryAction",
"status" : "success",
"execution" : 1
}, {
"name" : "com.oracle.glcm.patch.auto.db.integration.controller.action.crs.GIRACPrereqAction",
"status" : "success",
"execution" : 1
} ],
"name" : "init:init"
}, {
"action" : [ {
"name" : "com.oracle.glcm.patch.auto.db.integration.controller.action.crs.RemoteGIStopAction",
"status" : "skipped",
"execution" : 1
} ],
"name" : "shutdown:prepare-shutdown"
}, {
"action" : [ {
"name" : "com.oracle.glcm.patch.auto.db.integration.controller.action.crs.GIShutDownAction",
"status" : "success",
"execution" : 1
} ],
"name" : "shutdown:shutdown"
}, {
"action" : [ {
"name" : "com.oracle.glcm.patch.auto.db.integration.controller.action.OPatchAutoBinaryAction",
"status" : "success",
"execution" : 1
} ],
"name" : "offline:binary-patching"
}, {
"action" : [ {
"name" : "com.oracle.glcm.patch.auto.db.integration.controller.action.crs.GIStartupAction",
"status" : "failure",
"execution" : 1
} ],
"name" : "startup:startup"
} ]
},
"type" : "home",
"product-type" : "crs",
"home" : "/u01/app/12.1.0/grid",
"host" : "dbabg-rac1"
} ]
},
"session-id" : "YX56",
"host" : "dbabg-rac1",
"home" : "/u01/app/12.1.0/grid",
"command" : "apply",
"status" : "failure"
}[grid@dbabg-rac1 dbsessioninfo]$
-----
---- check ASM mode
As you already know, we are running patching in rolling mode
You can check the status after the patching issue from ASM instance
---- checking rolling mode from crsctl
[grid@hdbabg-rac2 OPatch]$ /u01/app/12.1.0/grid/bin/crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [197506932].
[grid@hdbabg-rac2 OPatch]$ s
Still ASM information shows rolling
---checking rolling mode from ASM with sqlplus
---
[grid@dbabg-rac1 bin]$ ./sqlplus /nolog
SQL*Plus: Release 12.1.0.2.0 Production on Wed Jan 11 09:54:39 2017
Copyright (c) 1982, 2014, Oracle. All rights reserved.
SQL> as sysdba
Connected.
SQL> SELECT SYS_CONTEXT('SYS_CLUSTER_PROPERTIES', 'CLUSTER_STATE') FROM DUAL;
SYS_CONTEXT('SYS_CLUSTER_PROPERTIES','CLUSTER_STATE')
--------------------------------------------------------------------------------
In Rolling Patch
SQL>
-----
[root@hdbabg-rac2 24412235]# /u01/app/12.1.0/grid/OPatch/opatchauto apply /home/grid/24412235 -oh /u01/app/12.1.0/grid -nonrolling
OPatchauto session is initiated at Wed Jan 11 09:41:36 2017
System initialization log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchautodb/systemconfig2017-01-11_09-41-39AM.log.
Session log file is /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/opatchauto2017-01-11_09-41-55AM.log
The id for this session is C283
Executing OPatch prereq operations to verify patch applicability on home /u01/app/12.1.0/grid
Patch applicablity verified successfully on home /u01/app/12.1.0/grid
Verifying patch inventory on home /u01/app/12.1.0/grid
Patch inventory verified successfully on home /u01/app/12.1.0/grid
Bringing down CRS service on home /u01/app/12.1.0/grid
Prepatch operation log file location: /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_hdbabg-rac2_2017-01-11_09-44-11AM.log
CRS service brought down successfully on home /u01/app/12.1.0/grid
Start applying binary patch on home /u01/app/12.1.0/grid
Binary patch applied successfully on home /u01/app/12.1.0/grid
Starting CRS service on home /u01/app/12.1.0/grid
Postpatch operation log file location: /u01/app/12.1.0/grid/cfgtoollogs/crsconfig/crspatch_hdbabg-rac2_2017-01-11_10-08-11AM.log
CRS service started successfully on home /u01/app/12.1.0/grid
Verifying patches applied on home /u01/app/12.1.0/grid
Patch verification completed with warning on home /u01/app/12.1.0/grid
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:hdbabg-rac2
CRS Home:/u01/app/12.1.0/grid
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /home/grid/24412235/21436941
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2017-01-11_09-45-14AM_1.log
Patch: /home/grid/24412235/23854735
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2017-01-11_09-45-14AM_1.log
Patch: /home/grid/24412235/24006101
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2017-01-11_09-45-14AM_1.log
Patch: /home/grid/24412235/24007012
Log: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2017-01-11_09-45-14AM_1.log
OPatchauto session completed at Wed Jan 11 10:21:20 2017
Time taken to complete the session 39 minutes, 45 seconds
[root@hdbabg-rac2 24412235]#
Now, we are going to show you one very useful tool - kfod
Metalink information for more information
CRS-6706: Oracle Clusterware Release patch level ('nnn') does not match Software patch level ('mmm') (Doc ID 1639285.1)
[grid@dbabg-rac1 bin]$ ./kfod op=patches
---------------
List of Patches
===============
19769480
20091753
20299023
20831110
21359755
21436941
21948354
22291127
23054246
23854735
24006101
24007012
[grid@dbabg-rac1 bin]$
[grid@hdbabg-rac2 bin]$ ./kfod op=patches
---------------
List of Patches
===============
19769480
20091753
20299023
20831110
21359755
21436941
21948354
22291127
23054246
23854735
24006101
24007012
[grid@hdbabg-rac2 bin]$
[grid@hdbabg-rac2 logs]$ /u01/app/12.1.0/grid//bin/kfod op=patchlvl
-------------------
Current Patch level
===================
2615413759
[grid@hdbabg-rac2 logs]$
--- Resolution from metalink
CRS-6706: Oracle Clusterware Release patch level ('nnn') does not match Software patch level ('mmm') (Doc ID 1639285.1)
Clusterware is not coming up after applying Grid PSU Patchset on Oracle GI 12cR1 (Doc ID 2138519.1)
We will implement solution from 1639285.1 with small changes
--- Start fixing Patch level
---- metalink info
For 12.1.0.2 onwards, rootcrs.sh -patch is deprecated. Execute the below command on the problematic node as root user.
<GI_HOME>/crs/install/rootcrs.sh -prepatch
<GI_HOME>/crs/install/rootcrs.sh -postpatch
------
[root@hdbabg-rac2 install]# ./rootcrs.sh -prepatch
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'hdbabg-rac2'
CRS-2672: Attempting to start 'ora.evmd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.mdnsd' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.evmd' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.gpnpd' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.gipcd' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hdbabg-rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.diskmon' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'hdbabg-rac2'
CRS-2672: Attempting to start 'ora.ctssd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.ctssd' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.asm' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.storage' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.crf' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.crsd' on 'hdbabg-rac2' succeeded
CRS-6017: Processing resource auto-start for servers: hdbabg-rac2
CRS-2672: Attempting to start 'ora.net1.network' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.net1.network' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.hdbabg-rac2.vip' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.hdbabg-rac2.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.hdbabg-rac2.vip' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.ons' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.hdbabg-rac2.vip' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.MGMTLSNR' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.scan1.vip' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'hdbabg-rac2' succeeded
CRS-6016: Resource auto-start has completed for server hdbabg-rac2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
----- This is very important - as per the above information on node2 patch level is 2615413759 but in the cluster here is reported 197506932
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [197506932].
----------------------------------------
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'hdbabg-rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.hdbabg-rac2.vip' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.mgmtdb' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.CRS.dg' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.hdbabg-rac2.vip' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.hdbabg-rac2.vip' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.asm' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'dbabg-rac1'
CRS-2676: Start of 'ora.MGMTLSNR' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.hdbabg-rac2.vip' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'dbabg-rac1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.ons' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.net1.network' on 'hdbabg-rac2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'hdbabg-rac2' has completed
CRS-2677: Stop of 'ora.crsd' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.storage' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.storage' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.evmd' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.asm' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.gpnpd' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.crf' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.cssd' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.gipcd' on 'hdbabg-rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'hdbabg-rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2017/01/11 10:56:54 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2017/01/11 10:57:05 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2017/01/11 10:57:06 CLSRSC-347: Successfully unlock /u01/app/12.1.0/grid
[root@hdbabg-rac2 install]#
[root@hdbabg-rac2 install]# ./rootcrs.sh -postpatch
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2017/01/11 10:57:53 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2017/01/11 10:58:08 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'hdbabg-rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'hdbabg-rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'hdbabg-rac2'
CRS-2672: Attempting to start 'ora.evmd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.mdnsd' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.evmd' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.gpnpd' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.gipcd' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hdbabg-rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.diskmon' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'hdbabg-rac2'
CRS-2672: Attempting to start 'ora.ctssd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.ctssd' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.asm' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.storage' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'hdbabg-rac2'
CRS-5013: Agent "ORAROOTAGENT" failed to start process "/u01/app/12.1.0/grid/bin/osysmond" for action "start": details at "(:CLSN00008:)" in "/u01/app/grid/diag/crs/hdbabg-rac2/crs/trace/ohasd_orarootagent_root.trc"
CRS-2674: Start of 'ora.crf' on 'hdbabg-rac2' failed
CRS-2679: Attempting to clean 'ora.crf' on 'hdbabg-rac2'
CRS-2681: Clean of 'ora.crf' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.crsd' on 'hdbabg-rac2' succeeded
CRS-6017: Processing resource auto-start for servers: hdbabg-rac2
CRS-2672: Attempting to start 'ora.net1.network' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.net1.network' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'hdbabg-rac2'
CRS-2673: Attempting to stop 'ora.hdbabg-rac2.vip' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.hdbabg-rac2.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.hdbabg-rac2.vip' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.ons' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.hdbabg-rac2.vip' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'hdbabg-rac2'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.MGMTLSNR' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.scan1.vip' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'hdbabg-rac2' succeeded
CRS-6016: Resource auto-start has completed for server hdbabg-rac2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
----- This is very important - as per the above information on node2 patch level is 2615413759 but in the cluster here is reported 197506932
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [197506932].
----------------------------------------
PRCC-1012 : _mgmtdb was already disabled
PRCR-1003 : Resource ora.mgmtdb is already disabled
PRCC-1011 : _mgmtdb was already enabled on hdbabg-rac2
PRCR-1002 : Resource ora.mgmtdb is already enabled
SQL Patching tool version 12.1.0.2.0 on Wed Jan 11 11:02:24 2017
Copyright (c) 2016, Oracle. All rights reserved.
Connecting to database...OK
Note: Datapatch will only apply or rollback SQL fixes for PDBs
that are in an open state, no patches will be applied to closed PDBs.
Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
(Doc ID 1585822.1)
Determining current state...done
Adding patches to installation queue and performing prereq checks...done
Installation queue:
For the following PDBs: CDB$ROOT PDB$SEED HAUZER_SCAN
Nothing to roll back
Nothing to apply
SQL Patching tool complete on Wed Jan 11 11:06:10 2017
[root@hdbabg-rac2 install]#
--- Now, as per the below information we have 1:1 patch version on both nodes.
[grid@hdbabg-rac2 logs]$ ssh dbabg-rac1 /u01/app/12.1.0/grid//bin/kfod op=patchlvl
-------------------
Current Patch level
===================
2615413759
[grid@hdbabg-rac2 logs]$ ssh dbabg-rac1
Last login: Wed Jan 11 09:04:39 2017 from hdbabg-rac2.localdomain
[grid@dbabg-rac1 ~]$ /u01/app/12.1.0/grid//bin/kfod op=patchlvl
-------------------
Current Patch level
===================
2615413759
[grid@dbabg-rac1 ~]$
But the problem is that a different patch level was reported in the inventory.
Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
197506932 dbabg-rac1
2615413759 hdbabg-rac2
--- see
[grid@dbabg-rac1 OPatch]$ ./opatch lsinv
Oracle Interim Patch Installer version 12.2.0.1.8
Copyright (c) 2017, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/12.1.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/12.1.0/grid/oraInst.loc
OPatch version : 12.2.0.1.8
OUI version : 12.1.0.2.0
Log file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2017-01-11_11-09-34AM_1.log
Lsinventory Output file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2017-01-11_11-09-34AM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: dbabg-rac1.localdomain
ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1):
Oracle Grid Infrastructure 12c 12.1.0.2.0
There are 1 products installed in this Oracle Home.
Interim patches (4) :
Patch 24007012 : applied on Wed Jan 11 06:59:13 EET 2017
Unique Patch ID: 20511653
Patch description: "ACFS Patch Set Update : 12.1.0.2.161018 (24007012)"
Created on 30 Aug 2016, 00:17:17 hrs PST8PDT
Bugs fixed:
19060056, 21369858, 19355146, 19690653, 19526234, 20952520, 19475588
21231953, 18510745, 19149476, 19013966, 22222222, 19859183, 21519796
19469788, 21555038, 18877486, 20438706, 19610022, 19335268, 21058846
22472364, 22319558, 19183802, 19452723, 21124596, 19051391, 20010980
19526283, 22198405, 22689989, 19279106, 19078259, 19644505, 21077218
21233961, 19184398, 18321597, 19811784, 18185024, 22174238, 20098064
19353057, 19502657, 22661787, 19434261, 18951113, 20140148, 18900953
19053182, 22975177, 18955907, 18934139, 19134464, 19450090, 19517835
22241945, 20923224, 19446719, 20402287, 19201087, 21208140, 19649858
21076821, 19428756, 19001684, 19610001, 18899600, 20314907, 19509898
19528981, 18957085, 19127216, 23557970, 19901825, 22810422, 18915417
19270227, 22087780, 22198176, 19919907, 19593769, 23112252, 19557156
19195735, 23252830, 22846612, 20681968, 18851012, 19844362
Patch 24006101 : applied on Wed Jan 11 06:50:26 EET 2017
Unique Patch ID: 20683584
Patch description: "Database Patch Set Update : 12.1.0.2.161018 (24006101)"
Created on 1 Oct 2016, 12:33:50 hrs PST8PDT
Sub-patch 23054246; "Database Patch Set Update : 12.1.0.2.160719 (23054246)"
Sub-patch 22291127; "Database Patch Set Update : 12.1.0.2.160419 (22291127)"
Sub-patch 21948354; "Database Patch Set Update : 12.1.0.2.160119 (21948354)"
Sub-patch 21359755; "Database Patch Set Update : 12.1.0.2.5 (21359755)"
Sub-patch 20831110; "Database Patch Set Update : 12.1.0.2.4 (20831110)"
Sub-patch 20299023; "Database Patch Set Update : 12.1.0.2.3 (20299023)"
Sub-patch 19769480; "Database Patch Set Update : 12.1.0.2.2 (19769480)"
Bugs fixed:
21099555, 22175564, 19141838, 22083366, 20842388, 20117253, 19865345
19791273, 19243521, 20951038, 19908836, 21281532, 19238590, 24577566
21184223, 19134173, 20387265, 19149990, 21263635, 18886413, 17551063
22160989, 22507210, 19703301, 19366375, 19001390, 18202441, 20267166
19358317, 19706965, 18549238, 19068970, 18797519, 22148226, 20825533
19649152, 19670108, 18940497, 18948177, 18964978, 19035573, 19176326
20413820, 19176223, 20904530, 20134339, 19074147, 20868862, 18411216
21322887, 22507234, 20425790, 18966843, 21329301, 19333670, 19468991
20124446, 19883092, 19658708, 19591608, 19402853, 20618595, 21787056
22380919, 21266085, 17835294, 19721304, 19791377, 19068610, 22178855
22173980, 20746251, 20048359, 20898391, 19185876, 20281121, 20907061
6599380, 19577410, 22092979, 20603378, 19001359, 19490948, 21387964
20832516, 17532734, 19309466, 19081128, 20627866, 20844426, 21188532
18791688, 21442094, 20890311, 20596234, 18973548, 21296029, 19303936
19461270, 21479753, 20936905, 20235511, 21220620, 18964939, 19430401
22296366, 21153266, 19409212, 22657942, 20657441, 19879746, 19684504
20528052, 19024808, 20977794, 18799993, 20466322, 18740837, 19662635
20228093, 19065556, 20212067, 19524384, 17722075, 20446883, 18952989
16870214, 19928926, 19835133, 21629064, 20466628, 24386767, 19931709
19730508, 18819908, 23124895, 19188927, 20074391, 20356733, 14643995
19547370, 19065677, 21960504, 21225209, 20397490, 18967382, 19174430
21241829, 19536415, 19171086, 22465352, 22168163, 19335438, 20447445
18856999, 20471920, 19869255, 21620471, 18990693, 17890099, 18990023
20101006, 21300341, 20848335, 21744290, 20897759, 21668627, 19304354
19052488, 20794034, 23260854, 18681056, 20952966, 19896336, 18618122
20328248, 20440930, 18456643, 19699191, 19201867, 22865673, 18743542
20798891, 20347562, 22551446, 19777862, 19687159, 21373076, 19174942
20424899, 21899588, 18899974, 20598042, 19032777, 19058490, 22815955
19399918, 19434529, 21273804, 19018447, 22757364, 18851894, 19284031
19022470, 18043064, 20173897, 22062026, 20475845, 17274537, 19440586
16887946, 22374754, 17319928, 20708701, 17655240, 16439813, 19805359
19155797, 20859910, 19393542, 22024071, 17210525, 21847223, 19189525
21649497, 19075256, 22762046, 22075064, 19280225, 18845653, 20560611
19248799, 21756699, 18988834, 20245930, 18921743, 18799063, 20373598
20476175, 19571367, 20925795, 19018206, 20509482, 20711718, 20588502
18849537, 19183343, 21917884, 19189317, 19644859, 19390567, 19279273
20669434, 16863642, 22528741, 19619732, 18607546, 20348653, 19315691
19676905, 20165574, 17867700, 20558005, 20734332, 19532017, 20922010
19450314, 22353346, 20361671, 20009833, 22366558, 20294666, 18191823
19307662, 19371175, 19195895, 20043616, 19154375, 18914624, 20139391
21291274, 19174521, 19520602, 19382851, 21875360, 19676012, 19326908
20217801, 20093776, 21097043, 21246723, 21665897, 19143550, 20428621
19627012, 14283239, 19518079, 18610915, 18674024, 18306996, 19524158
19915271, 20122715, 20284155, 20017509, 19363645, 19597439, 21239530
19888853, 20880215, 21756677, 19534363, 19354335, 19044962, 19639483
22353199, 22243719, 22916353, 20378086, 21756661, 21260431, 22923409
20877664, 19028800, 20879889, 19723336, 19077215, 21421886, 19604659
19308965, 19048007, 18288842, 19689979, 21526048, 19180770, 19197175
19902195, 20318889, 19013183, 19012119, 20464614, 19067244, 21632821
19512341, 19841800, 20331945, 19587324, 24316947, 19578350, 19637186
18674047, 19054077, 20898997, 19708632, 21091431, 19289642, 20869721
19258504, 17365043, 19468347, 21373473, 16359751, 19439759, 19769480
19272708, 19978542, 20402832, 19329654, 19873610, 23229229, 21517440
13542050, 19291380, 21915719, 19076343, 19561643, 19990037, 19487147
18909599, 20831538, 18250893, 19016730, 16619249, 18354830, 21188584
19989009, 17414008, 20688221, 20704450, 20441797, 19157754, 18885870
21785691, 21450666, 18893947, 18705806, 22223463, 16923858, 18417036
20919320, 20474192, 22046677, 19385656, 19501299, 20920911, 20899461
21387128, 21315084, 18122373, 20581111, 19606174, 18436647, 19023822
19178851, 19124589, 19597583, 18499088, 19050649
Patch 23854735 : applied on Wed Jan 11 06:42:32 EET 2017
Unique Patch ID: 20511532
Patch description: "OCW Patch Set Update : 12.1.0.2.161018 (23854735)"
Created on 29 Sep 2016, 23:50:00 hrs PST8PDT
Bugs fixed:
21339083, 18589889, 20768643, 19061429, 19133945, 19341538, 19521169
20011424, 19361757, 19187207, 19302350, 19130141, 19530755, 19699720
19620918, 19168690, 18899171, 19244316, 19653795, 18634372, 19471722
19536337, 19184188, 19131709, 20160490, 20995001, 21131037, 19380733
20985167, 18943696, 20552947, 19591984, 18135723, 18461246, 20014326
19524857, 18890943, 19154753, 19499021, 19046190, 20923930, 20028666
19051385, 19050688, 19682695, 19831611, 21607972, 18922918, 21087799
18956780, 19683886, 23757020, 19026993, 20110156, 21269977, 18261648
18362971, 19558324, 22651583, 20185476, 18882642, 19478119, 19414274
17447588, 19262534, 20752167, 19147513, 22475834, 21242118, 19514085
20758997, 19659885, 19319904, 18703978, 23079844, 19700256, 19703246
20467806, 19192901, 21527118, 19479503, 19147751, 18901356, 21232394
18964974, 22162062, 18835366, 19013789, 19207286, 18950232, 22583325
19148791, 19556820, 16505171, 22261732, 22135419, 19449737, 21878613
21646169, 19187515, 19230771, 20795241, 19453778, 19853036, 18843572
18520351, 21430567, 19185148, 19232454, 21220597, 18541110, 18834955
19204743, 20175174, 19140891, 18747335, 20676340, 19270660, 19325701
20877003, 19222693, 19662663, 19684131, 19010177, 19272663, 18910576
18907170, 19426384, 9372680, 20737462, 19700294, 20289828, 19331454
22568500, 20278216, 20827053, 20119857, 19703199, 22456393, 20588164
18955644, 20991879, 18508710, 19908346, 19589221, 18952857, 19537762
21152052, 16286734, 20669114, 18762843, 18945249, 19184799, 19205086
19351363, 20091753, 19031737, 19079087, 19148367, 18968981, 19150517
20231741, 19217019, 18730096, 18975620, 19067804, 21259359, 19054979
19209951, 19318983, 18747382, 19154673, 18752378, 22905714, 19150088
19013444, 19234177, 18999857, 19075747, 19367276, 19612597, 19874047
19677813, 23049292, 19427050, 18910443, 20033787, 19148982, 19777496
20730170, 18523468, 19277814, 20562133, 19071526, 21420753, 18850051
20433932, 20061168, 19370739, 20822591, 19609388, 18776786, 20136892
18989446, 19148793, 19585454, 18317489, 19955755, 19059053, 19807548
19124972, 22229116, 18910748, 19147509, 18849896, 19617592, 19398098
22840085, 19076165, 19498411, 19704993, 21294273, 19459023, 20634511
21229744, 21474380, 21484367, 20276459, 17208793, 19234907, 19538714
22867050, 19383028, 19513969, 22195657, 23111368, 19986391, 19314048
18834934, 19473851, 19458082, 18894342, 19522067, 18116578, 19241857
19076778, 19066699, 19068003, 18937186, 20267877, 19916737, 19770543
22863466, 19635215, 19613533, 21057239, 20783416, 19315522, 19141785
19163887, 19885321, 18715868, 18852058, 19701617, 19280860, 19139608
21453435, 18946768, 19135521, 18604143, 19627476, 20568884, 20317221
20177779, 19266658, 24522717, 18330979, 19027351, 21227730, 18707416
21118157, 21113068, 19925992, 20235486, 20006646, 15922206, 18439295
18991776, 21245437, 23170839, 19163425, 19550195, 18849021, 18861196
17940721, 18524834, 19287706, 19522313, 18835283, 18748932, 19184765
21639478, 14851828, 20598625, 19226141, 19053891, 18871287, 18998228
18980002, 19682778, 19325604, 18777835, 17338864, 18945906, 19513650
19702758, 18952577, 19473088, 19178517, 18729269, 19529729, 19455563
20877072, 23275930, 20340620, 18536826, 19292605, 22264951, 20920711
20660273, 20011635, 17592037, 18935857, 22989069, 19029647, 19179158
22302421, 23076471, 19140712, 20012766, 23049301, 19184276, 20510208
20001507, 20079414, 19680763, 19259765, 18962892, 19513888, 14661880
21519340, 21222147, 20218012, 19551830, 21616278, 19068333, 18945435
20775015, 19319192, 20839581, 19178629, 21694237, 19304104, 19284357
22594310, 20725084, 19457575, 21297657, 19069755, 19021575, 18715884
19584688, 18798573, 19812592, 19018001, 19292272, 19270956, 21967791
21794779, 23055224, 18700893, 21276514, 20467377, 18406774, 20607821
21976167, 21082737, 21694632, 20115586, 20883009, 19164099, 18283792
19155350, 19884033, 19146822, 18798432, 21123424, 19471836, 20936562
18750781, 19045143, 22483804, 19146980, 19601003, 20099624, 21378016
19537547, 19281106, 18862203, 19954372, 18235274, 19205617, 19513351
20219458, 19150313, 18843054, 18708349, 19332386, 18953639, 21423055
19371270, 19203996, 20038431, 19370846, 21053036, 18998379, 20157569
21357522, 19273577, 24709351, 22113170, 19632437, 18990354, 19288396
20620581, 19557558, 19127078, 21792881, 20900588, 20053557, 22088198
22664609, 19315567, 19444471, 19154933, 20953798, 18290252, 18813323
19915945, 19500293, 18643483, 19134098, 23034352, 18965694, 19226858
19591356, 19602208, 22884308, 20069197, 18417590, 18920408, 18636884
22289350, 19043795, 20746507, 18260170, 18919682, 18678829, 19319357
19273758, 18953878, 18999195, 18759724, 19066844, 13843841, 20290467
22934855, 19649640, 19062675, 21924910, 18859710, 20862812, 19504641
23101611, 19341481, 20293730, 19343245, 18304090, 19241655, 19470791
18242738, 20649573, 18372060, 18953889, 18827679, 19140711, 19023430
19259290, 19045388, 20186278, 19522571, 19709663, 18875012, 18861564
19273760, 19225265, 15885689, 20853464, 21696068, 18819158, 21394146
19049721, 19368917, 21255373, 22393909, 20408163, 20579351, 18868829
20350915, 21327402, 22024217, 23101338, 19820247, 20363708, 19538241
19804032
Patch 21436941 : applied on Wed Jan 11 06:37:11 EET 2017
Unique Patch ID: 19245012
Patch description: "WLM Patch Set Update: 12.1.0.2.5 (21436941)"
Created on 13 Aug 2015, 04:00:40 hrs PST8PDT
Bugs fixed:
20075154, 20641027, 20548410, 19016964, 19582630
Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
197506932 dbabg-rac1
2615413759 hdbabg-rac2
--------------------------------------------------------------------------------
OPatch succeeded.
[grid@dbabg-rac1 OPatch]$
---apply fixes with patch level differences on first node as per the above metalink information
[root@dbabg-rac1 ~]# cd /u01/app/12.1.0/grid/crs/install
[root@dbabg-rac1 install]# ./rootcrs.sh -prepatch
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
----- This is very important - as per the above information on node2 patch level is 2615413759 but in the cluster here is reported 197506932
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [197506932].
----------------------------------------
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'dbabg-rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.cvu' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.dbabg-rac1.vip' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.CRS.dg' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.asm' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.cvu' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.dbabg-rac1.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.dbabg-rac1.vip' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.scan2.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.scan3.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.dbabg-rac1.vip' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.scan3.vip' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'hdbabg-rac2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'hdbabg-rac2' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.oc4j' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.ons' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.net1.network' on 'dbabg-rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dbabg-rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.storage' on 'dbabg-rac1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'dbabg-rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'dbabg-rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'dbabg-rac1' succeeded
CRS-2677: Stop of 'ora.crf' on 'dbabg-rac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'dbabg-rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.cssd' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.gipcd' on 'dbabg-rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dbabg-rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2017/01/11 11:12:33 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2017/01/11 11:12:46 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2017/01/11 11:12:46 CLSRSC-347: Successfully unlock /u01/app/12.1.0/grid
--- run postpatch
[root@dbabg-rac1 install]# ./rootcrs.sh -postpatch
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
2017/01/11 11:14:50 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2017/01/11 11:15:06 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'dbabg-rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dbabg-rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.evmd' on 'dbabg-rac1'
CRS-2676: Start of 'ora.mdnsd' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.evmd' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dbabg-rac1'
CRS-2676: Start of 'ora.gpnpd' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'dbabg-rac1'
CRS-2676: Start of 'ora.gipcd' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbabg-rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'dbabg-rac1'
CRS-2676: Start of 'ora.diskmon' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dbabg-rac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'dbabg-rac1'
CRS-2676: Start of 'ora.ctssd' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'dbabg-rac1'
CRS-2676: Start of 'ora.asm' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'dbabg-rac1'
CRS-2676: Start of 'ora.storage' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'dbabg-rac1'
CRS-5013: Agent "ORAROOTAGENT" failed to start process "/u01/app/12.1.0/grid/bin/osysmond" for action "start": details at "(:CLSN00008:)" in "/u01/app/grid/diag/crs/dbabg-rac1/crs/trace/ohasd_orarootagent_root.trc"
CRS-2674: Start of 'ora.crf' on 'dbabg-rac1' failed
CRS-2679: Attempting to clean 'ora.crf' on 'dbabg-rac1'
CRS-2681: Clean of 'ora.crf' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'dbabg-rac1'
CRS-2676: Start of 'ora.crsd' on 'dbabg-rac1' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: dbabg-rac1
CRS-2672: Attempting to start 'ora.net1.network' on 'dbabg-rac1'
CRS-2676: Start of 'ora.net1.network' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'dbabg-rac1'
CRS-2673: Attempting to stop 'ora.dbabg-rac1.vip' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.dbabg-rac1.vip' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.dbabg-rac1.vip' on 'dbabg-rac1'
CRS-2676: Start of 'ora.ons' on 'dbabg-rac1' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'hdbabg-rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'hdbabg-rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'hdbabg-rac2'
CRS-2676: Start of 'ora.dbabg-rac1.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'dbabg-rac1'
CRS-2677: Stop of 'ora.scan1.vip' on 'hdbabg-rac2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'dbabg-rac1'
CRS-2676: Start of 'ora.scan1.vip' on 'dbabg-rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'dbabg-rac1'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'dbabg-rac1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'dbabg-rac1' succeeded
CRS-6016: Resource auto-start has completed for server dbabg-rac1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
------------- :))))) hahah very nice, now we have a cluster with fixed patch level and upgrade state! ----------------------------------------------------
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [2615413759].
[root@dbabg-rac1 install]#
--- checking inventory
[grid@hdbabg-rac2 OPatch]$ ./opatch lsinv
Oracle Interim Patch Installer version 12.2.0.1.8
Copyright (c) 2017, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/12.1.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/12.1.0/grid/oraInst.loc
OPatch version : 12.2.0.1.8
OUI version : 12.1.0.2.0
Log file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2017-01-11_11-18-21AM_1.log
Lsinventory Output file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2017-01-11_11-18-21AM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: hdbabg-rac2.localdomain
ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1):
Oracle Grid Infrastructure 12c 12.1.0.2.0
There are 1 products installed in this Oracle Home.
Interim patches (4) :
Patch 24007012 : applied on Wed Jan 11 10:07:56 EET 2017
Unique Patch ID: 20511653
Patch description: "ACFS Patch Set Update : 12.1.0.2.161018 (24007012)"
Created on 30 Aug 2016, 00:17:17 hrs PST8PDT
Bugs fixed:
19060056, 21369858, 19355146, 19690653, 19526234, 20952520, 19475588
21231953, 18510745, 19149476, 19013966, 22222222, 19859183, 21519796
19469788, 21555038, 18877486, 20438706, 19610022, 19335268, 21058846
22472364, 22319558, 19183802, 19452723, 21124596, 19051391, 20010980
19526283, 22198405, 22689989, 19279106, 19078259, 19644505, 21077218
21233961, 19184398, 18321597, 19811784, 18185024, 22174238, 20098064
19353057, 19502657, 22661787, 19434261, 18951113, 20140148, 18900953
19053182, 22975177, 18955907, 18934139, 19134464, 19450090, 19517835
22241945, 20923224, 19446719, 20402287, 19201087, 21208140, 19649858
21076821, 19428756, 19001684, 19610001, 18899600, 20314907, 19509898
19528981, 18957085, 19127216, 23557970, 19901825, 22810422, 18915417
19270227, 22087780, 22198176, 19919907, 19593769, 23112252, 19557156
19195735, 23252830, 22846612, 20681968, 18851012, 19844362
Patch 24006101 : applied on Wed Jan 11 09:57:20 EET 2017
Unique Patch ID: 20683584
Patch description: "Database Patch Set Update : 12.1.0.2.161018 (24006101)"
Created on 1 Oct 2016, 12:33:50 hrs PST8PDT
Sub-patch 23054246; "Database Patch Set Update : 12.1.0.2.160719 (23054246)"
Sub-patch 22291127; "Database Patch Set Update : 12.1.0.2.160419 (22291127)"
Sub-patch 21948354; "Database Patch Set Update : 12.1.0.2.160119 (21948354)"
Sub-patch 21359755; "Database Patch Set Update : 12.1.0.2.5 (21359755)"
Sub-patch 20831110; "Database Patch Set Update : 12.1.0.2.4 (20831110)"
Sub-patch 20299023; "Database Patch Set Update : 12.1.0.2.3 (20299023)"
Sub-patch 19769480; "Database Patch Set Update : 12.1.0.2.2 (19769480)"
Bugs fixed:
21099555, 22175564, 19141838, 22083366, 20842388, 20117253, 19865345
19791273, 19243521, 20951038, 19908836, 21281532, 19238590, 24577566
21184223, 19134173, 20387265, 19149990, 21263635, 18886413, 17551063
22160989, 22507210, 19703301, 19366375, 19001390, 18202441, 20267166
19358317, 19706965, 18549238, 19068970, 18797519, 22148226, 20825533
19649152, 19670108, 18940497, 18948177, 18964978, 19035573, 19176326
20413820, 19176223, 20904530, 20134339, 19074147, 20868862, 18411216
21322887, 22507234, 20425790, 18966843, 21329301, 19333670, 19468991
20124446, 19883092, 19658708, 19591608, 19402853, 20618595, 21787056
22380919, 21266085, 17835294, 19721304, 19791377, 19068610, 22178855
22173980, 20746251, 20048359, 20898391, 19185876, 20281121, 20907061
6599380, 19577410, 22092979, 20603378, 19001359, 19490948, 21387964
20832516, 17532734, 19309466, 19081128, 20627866, 20844426, 21188532
18791688, 21442094, 20890311, 20596234, 18973548, 21296029, 19303936
19461270, 21479753, 20936905, 20235511, 21220620, 18964939, 19430401
22296366, 21153266, 19409212, 22657942, 20657441, 19879746, 19684504
20528052, 19024808, 20977794, 18799993, 20466322, 18740837, 19662635
20228093, 19065556, 20212067, 19524384, 17722075, 20446883, 18952989
16870214, 19928926, 19835133, 21629064, 20466628, 24386767, 19931709
19730508, 18819908, 23124895, 19188927, 20074391, 20356733, 14643995
19547370, 19065677, 21960504, 21225209, 20397490, 18967382, 19174430
21241829, 19536415, 19171086, 22465352, 22168163, 19335438, 20447445
18856999, 20471920, 19869255, 21620471, 18990693, 17890099, 18990023
20101006, 21300341, 20848335, 21744290, 20897759, 21668627, 19304354
19052488, 20794034, 23260854, 18681056, 20952966, 19896336, 18618122
20328248, 20440930, 18456643, 19699191, 19201867, 22865673, 18743542
20798891, 20347562, 22551446, 19777862, 19687159, 21373076, 19174942
20424899, 21899588, 18899974, 20598042, 19032777, 19058490, 22815955
19399918, 19434529, 21273804, 19018447, 22757364, 18851894, 19284031
19022470, 18043064, 20173897, 22062026, 20475845, 17274537, 19440586
16887946, 22374754, 17319928, 20708701, 17655240, 16439813, 19805359
19155797, 20859910, 19393542, 22024071, 17210525, 21847223, 19189525
21649497, 19075256, 22762046, 22075064, 19280225, 18845653, 20560611
19248799, 21756699, 18988834, 20245930, 18921743, 18799063, 20373598
20476175, 19571367, 20925795, 19018206, 20509482, 20711718, 20588502
18849537, 19183343, 21917884, 19189317, 19644859, 19390567, 19279273
20669434, 16863642, 22528741, 19619732, 18607546, 20348653, 19315691
19676905, 20165574, 17867700, 20558005, 20734332, 19532017, 20922010
19450314, 22353346, 20361671, 20009833, 22366558, 20294666, 18191823
19307662, 19371175, 19195895, 20043616, 19154375, 18914624, 20139391
21291274, 19174521, 19520602, 19382851, 21875360, 19676012, 19326908
20217801, 20093776, 21097043, 21246723, 21665897, 19143550, 20428621
19627012, 14283239, 19518079, 18610915, 18674024, 18306996, 19524158
19915271, 20122715, 20284155, 20017509, 19363645, 19597439, 21239530
19888853, 20880215, 21756677, 19534363, 19354335, 19044962, 19639483
22353199, 22243719, 22916353, 20378086, 21756661, 21260431, 22923409
20877664, 19028800, 20879889, 19723336, 19077215, 21421886, 19604659
19308965, 19048007, 18288842, 19689979, 21526048, 19180770, 19197175
19902195, 20318889, 19013183, 19012119, 20464614, 19067244, 21632821
19512341, 19841800, 20331945, 19587324, 24316947, 19578350, 19637186
18674047, 19054077, 20898997, 19708632, 21091431, 19289642, 20869721
19258504, 17365043, 19468347, 21373473, 16359751, 19439759, 19769480
19272708, 19978542, 20402832, 19329654, 19873610, 23229229, 21517440
13542050, 19291380, 21915719, 19076343, 19561643, 19990037, 19487147
18909599, 20831538, 18250893, 19016730, 16619249, 18354830, 21188584
19989009, 17414008, 20688221, 20704450, 20441797, 19157754, 18885870
21785691, 21450666, 18893947, 18705806, 22223463, 16923858, 18417036
20919320, 20474192, 22046677, 19385656, 19501299, 20920911, 20899461
21387128, 21315084, 18122373, 20581111, 19606174, 18436647, 19023822
19178851, 19124589, 19597583, 18499088, 19050649
Patch 23854735 : applied on Wed Jan 11 09:50:55 EET 2017
Unique Patch ID: 20511532
Patch description: "OCW Patch Set Update : 12.1.0.2.161018 (23854735)"
Created on 29 Sep 2016, 23:50:00 hrs PST8PDT
Bugs fixed:
21339083, 18589889, 20768643, 19061429, 19133945, 19341538, 19521169
20011424, 19361757, 19187207, 19302350, 19130141, 19530755, 19699720
19620918, 19168690, 18899171, 19244316, 19653795, 18634372, 19471722
19536337, 19184188, 19131709, 20160490, 20995001, 21131037, 19380733
20985167, 18943696, 20552947, 19591984, 18135723, 18461246, 20014326
19524857, 18890943, 19154753, 19499021, 19046190, 20923930, 20028666
19051385, 19050688, 19682695, 19831611, 21607972, 18922918, 21087799
18956780, 19683886, 23757020, 19026993, 20110156, 21269977, 18261648
18362971, 19558324, 22651583, 20185476, 18882642, 19478119, 19414274
17447588, 19262534, 20752167, 19147513, 22475834, 21242118, 19514085
20758997, 19659885, 19319904, 18703978, 23079844, 19700256, 19703246
20467806, 19192901, 21527118, 19479503, 19147751, 18901356, 21232394
18964974, 22162062, 18835366, 19013789, 19207286, 18950232, 22583325
19148791, 19556820, 16505171, 22261732, 22135419, 19449737, 21878613
21646169, 19187515, 19230771, 20795241, 19453778, 19853036, 18843572
18520351, 21430567, 19185148, 19232454, 21220597, 18541110, 18834955
19204743, 20175174, 19140891, 18747335, 20676340, 19270660, 19325701
20877003, 19222693, 19662663, 19684131, 19010177, 19272663, 18910576
18907170, 19426384, 9372680, 20737462, 19700294, 20289828, 19331454
22568500, 20278216, 20827053, 20119857, 19703199, 22456393, 20588164
18955644, 20991879, 18508710, 19908346, 19589221, 18952857, 19537762
21152052, 16286734, 20669114, 18762843, 18945249, 19184799, 19205086
19351363, 20091753, 19031737, 19079087, 19148367, 18968981, 19150517
20231741, 19217019, 18730096, 18975620, 19067804, 21259359, 19054979
19209951, 19318983, 18747382, 19154673, 18752378, 22905714, 19150088
19013444, 19234177, 18999857, 19075747, 19367276, 19612597, 19874047
19677813, 23049292, 19427050, 18910443, 20033787, 19148982, 19777496
20730170, 18523468, 19277814, 20562133, 19071526, 21420753, 18850051
20433932, 20061168, 19370739, 20822591, 19609388, 18776786, 20136892
18989446, 19148793, 19585454, 18317489, 19955755, 19059053, 19807548
19124972, 22229116, 18910748, 19147509, 18849896, 19617592, 19398098
22840085, 19076165, 19498411, 19704993, 21294273, 19459023, 20634511
21229744, 21474380, 21484367, 20276459, 17208793, 19234907, 19538714
22867050, 19383028, 19513969, 22195657, 23111368, 19986391, 19314048
18834934, 19473851, 19458082, 18894342, 19522067, 18116578, 19241857
19076778, 19066699, 19068003, 18937186, 20267877, 19916737, 19770543
22863466, 19635215, 19613533, 21057239, 20783416, 19315522, 19141785
19163887, 19885321, 18715868, 18852058, 19701617, 19280860, 19139608
21453435, 18946768, 19135521, 18604143, 19627476, 20568884, 20317221
20177779, 19266658, 24522717, 18330979, 19027351, 21227730, 18707416
21118157, 21113068, 19925992, 20235486, 20006646, 15922206, 18439295
18991776, 21245437, 23170839, 19163425, 19550195, 18849021, 18861196
17940721, 18524834, 19287706, 19522313, 18835283, 18748932, 19184765
21639478, 14851828, 20598625, 19226141, 19053891, 18871287, 18998228
18980002, 19682778, 19325604, 18777835, 17338864, 18945906, 19513650
19702758, 18952577, 19473088, 19178517, 18729269, 19529729, 19455563
20877072, 23275930, 20340620, 18536826, 19292605, 22264951, 20920711
20660273, 20011635, 17592037, 18935857, 22989069, 19029647, 19179158
22302421, 23076471, 19140712, 20012766, 23049301, 19184276, 20510208
20001507, 20079414, 19680763, 19259765, 18962892, 19513888, 14661880
21519340, 21222147, 20218012, 19551830, 21616278, 19068333, 18945435
20775015, 19319192, 20839581, 19178629, 21694237, 19304104, 19284357
22594310, 20725084, 19457575, 21297657, 19069755, 19021575, 18715884
19584688, 18798573, 19812592, 19018001, 19292272, 19270956, 21967791
21794779, 23055224, 18700893, 21276514, 20467377, 18406774, 20607821
21976167, 21082737, 21694632, 20115586, 20883009, 19164099, 18283792
19155350, 19884033, 19146822, 18798432, 21123424, 19471836, 20936562
18750781, 19045143, 22483804, 19146980, 19601003, 20099624, 21378016
19537547, 19281106, 18862203, 19954372, 18235274, 19205617, 19513351
20219458, 19150313, 18843054, 18708349, 19332386, 18953639, 21423055
19371270, 19203996, 20038431, 19370846, 21053036, 18998379, 20157569
21357522, 19273577, 24709351, 22113170, 19632437, 18990354, 19288396
20620581, 19557558, 19127078, 21792881, 20900588, 20053557, 22088198
22664609, 19315567, 19444471, 19154933, 20953798, 18290252, 18813323
19915945, 19500293, 18643483, 19134098, 23034352, 18965694, 19226858
19591356, 19602208, 22884308, 20069197, 18417590, 18920408, 18636884
22289350, 19043795, 20746507, 18260170, 18919682, 18678829, 19319357
19273758, 18953878, 18999195, 18759724, 19066844, 13843841, 20290467
22934855, 19649640, 19062675, 21924910, 18859710, 20862812, 19504641
23101611, 19341481, 20293730, 19343245, 18304090, 19241655, 19470791
18242738, 20649573, 18372060, 18953889, 18827679, 19140711, 19023430
19259290, 19045388, 20186278, 19522571, 19709663, 18875012, 18861564
19273760, 19225265, 15885689, 20853464, 21696068, 18819158, 21394146
19049721, 19368917, 21255373, 22393909, 20408163, 20579351, 18868829
20350915, 21327402, 22024217, 23101338, 19820247, 20363708, 19538241
19804032
Patch 21436941 : applied on Wed Jan 11 09:45:38 EET 2017
Unique Patch ID: 19245012
Patch description: "WLM Patch Set Update: 12.1.0.2.5 (21436941)"
Created on 13 Aug 2015, 04:00:40 hrs PST8PDT
Bugs fixed:
20075154, 20641027, 20548410, 19016964, 19582630
Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
2615413759 dbabg-rac1,hdbabg-rac2
--------------------------------------------------------------------------------
OPatch succeeded.
[grid@hdbabg-rac2 OPatch]$
----- As you can see, this process fixes the entire cluster version patch
- Additional checks
[root@dbabg-rac1 install]# /u01/app/12.1.0/grid/bin/crsctl query crs softwarepatch Oracle Clusterware patch level on node dbabg-rac1 is [2615413759]. [root@dbabg-rac1 install]# ssh hdbabg-rac2 /u01/app/12.1.0/grid/bin/crsctl query crs softwarepatch Oracle Clusterware patch level on node hdbabg-rac2 is [2615413759]. |
----
[root@dbabg-rac1 install]#
- Checking inventory of hdbabg-rac2
[grid@hdbabg-rac2 OPatch]$ ssh dbabg-rac1 Last login: Wed Jan 11 11:10:56 2017 from hdbabg-rac2.localdomain
[grid@dbabg-rac1 ~]$ cd /u01/app/12.1.0/grid/OPatch [grid@dbabg-rac1 OPatch]$ ./opatch lsinv Oracle Interim Patch Installer version 12.2.0.1.8 Copyright (c) 2017, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/12.1.0/grid Central Inventory : /u01/app/oraInventory from : /u01/app/12.1.0/grid//oraInst.loc OPatch version : 12.2.0.1.8 OUI version : 12.1.0.2.0 Log file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2017-01-11_11-23-04AM_1.log Lsinventory Output file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2017-01-11_11-23-04AM.txt -------------------------------------------------------------------------------- Local Machine Information:: Hostname: dbabg-rac1.localdomain ARU platform id: 226 ARU platform description:: Linux x86-64 Installed Top-level Products (1): Oracle Grid Infrastructure 12c 12.1.0.2.0 There are 1 products installed in this Oracle Home. Interim patches (4) : Patch 24007012 : applied on Wed Jan 11 06:59:13 EET 2017 Unique Patch ID: 20511653 Patch description: "ACFS Patch Set Update : 12.1.0.2.161018 (24007012)" Created on 30 Aug 2016, 00:17:17 hrs PST8PDT Bugs fixed: 19060056, 21369858, 19355146, 19690653, 19526234, 20952520, 19475588 21231953, 18510745, 19149476, 19013966, 22222222, 19859183, 21519796 19469788, 21555038, 18877486, 20438706, 19610022, 19335268, 21058846 22472364, 22319558, 19183802, 19452723, 21124596, 19051391, 20010980 19526283, 22198405, 22689989, 19279106, 19078259, 19644505, 21077218 21233961, 19184398, 18321597, 19811784, 18185024, 22174238, 20098064 19353057, 19502657, 22661787, 19434261, 18951113, 20140148, 18900953 19053182, 22975177, 18955907, 18934139, 19134464, 19450090, 19517835 22241945, 20923224, 19446719, 20402287, 19201087, 21208140, 19649858 21076821, 19428756, 19001684, 19610001, 18899600, 20314907, 19509898 19528981, 18957085, 19127216, 23557970, 19901825, 22810422, 18915417 19270227, 22087780, 22198176, 19919907, 19593769, 23112252, 19557156 19195735, 23252830, 22846612, 20681968, 18851012, 19844362
Patch 24006101 : applied on Wed Jan 11 06:50:26 EET 2017 Unique Patch ID: 20683584 Patch description: "Database Patch Set Update : 12.1.0.2.161018 (24006101)" Created on 1 Oct 2016, 12:33:50 hrs PST8PDT Sub-patch 23054246; "Database Patch Set Update : 12.1.0.2.160719 (23054246)" Sub-patch 22291127; "Database Patch Set Update : 12.1.0.2.160419 (22291127)" Sub-patch 21948354; "Database Patch Set Update : 12.1.0.2.160119 (21948354)" Sub-patch 21359755; "Database Patch Set Update : 12.1.0.2.5 (21359755)" Sub-patch 20831110; "Database Patch Set Update : 12.1.0.2.4 (20831110)" Sub-patch 20299023; "Database Patch Set Update : 12.1.0.2.3 (20299023)" Sub-patch 19769480; "Database Patch Set Update : 12.1.0.2.2 (19769480)" Bugs fixed: 21099555, 22175564, 19141838, 22083366, 20842388, 20117253, 19865345 19791273, 19243521, 20951038, 19908836, 21281532, 19238590, 24577566 21184223, 19134173, 20387265, 19149990, 21263635, 18886413, 17551063 22160989, 22507210, 19703301, 19366375, 19001390, 18202441, 20267166 19358317, 19706965, 18549238, 19068970, 18797519, 22148226, 20825533 19649152, 19670108, 18940497, 18948177, 18964978, 19035573, 19176326 20413820, 19176223, 20904530, 20134339, 19074147, 20868862, 18411216 21322887, 22507234, 20425790, 18966843, 21329301, 19333670, 19468991 20124446, 19883092, 19658708, 19591608, 19402853, 20618595, 21787056 22380919, 21266085, 17835294, 19721304, 19791377, 19068610, 22178855 22173980, 20746251, 20048359, 20898391, 19185876, 20281121, 20907061 6599380, 19577410, 22092979, 20603378, 19001359, 19490948, 21387964 20832516, 17532734, 19309466, 19081128, 20627866, 20844426, 21188532 18791688, 21442094, 20890311, 20596234, 18973548, 21296029, 19303936 19461270, 21479753, 20936905, 20235511, 21220620, 18964939, 19430401 22296366, 21153266, 19409212, 22657942, 20657441, 19879746, 19684504 20528052, 19024808, 20977794, 18799993, 20466322, 18740837, 19662635 20228093, 19065556, 20212067, 19524384, 17722075, 20446883, 18952989 16870214, 19928926, 19835133, 21629064, 20466628, 24386767, 19931709 19730508, 18819908, 23124895, 19188927, 20074391, 20356733, 14643995 19547370, 19065677, 21960504, 21225209, 20397490, 18967382, 19174430 21241829, 19536415, 19171086, 22465352, 22168163, 19335438, 20447445 18856999, 20471920, 19869255, 21620471, 18990693, 17890099, 18990023 20101006, 21300341, 20848335, 21744290, 20897759, 21668627, 19304354 19052488, 20794034, 23260854, 18681056, 20952966, 19896336, 18618122 20328248, 20440930, 18456643, 19699191, 19201867, 22865673, 18743542 20798891, 20347562, 22551446, 19777862, 19687159, 21373076, 19174942 20424899, 21899588, 18899974, 20598042, 19032777, 19058490, 22815955 19399918, 19434529, 21273804, 19018447, 22757364, 18851894, 19284031 19022470, 18043064, 20173897, 22062026, 20475845, 17274537, 19440586 16887946, 22374754, 17319928, 20708701, 17655240, 16439813, 19805359 19155797, 20859910, 19393542, 22024071, 17210525, 21847223, 19189525 21649497, 19075256, 22762046, 22075064, 19280225, 18845653, 20560611 19248799, 21756699, 18988834, 20245930, 18921743, 18799063, 20373598 20476175, 19571367, 20925795, 19018206, 20509482, 20711718, 20588502 18849537, 19183343, 21917884, 19189317, 19644859, 19390567, 19279273 20669434, 16863642, 22528741, 19619732, 18607546, 20348653, 19315691 19676905, 20165574, 17867700, 20558005, 20734332, 19532017, 20922010 19450314, 22353346, 20361671, 20009833, 22366558, 20294666, 18191823 19307662, 19371175, 19195895, 20043616, 19154375, 18914624, 20139391 21291274, 19174521, 19520602, 19382851, 21875360, 19676012, 19326908 20217801, 20093776, 21097043, 21246723, 21665897, 19143550, 20428621 19627012, 14283239, 19518079, 18610915, 18674024, 18306996, 19524158 19915271, 20122715, 20284155, 20017509, 19363645, 19597439, 21239530 19888853, 20880215, 21756677, 19534363, 19354335, 19044962, 19639483 22353199, 22243719, 22916353, 20378086, 21756661, 21260431, 22923409 20877664, 19028800, 20879889, 19723336, 19077215, 21421886, 19604659 19308965, 19048007, 18288842, 19689979, 21526048, 19180770, 19197175 19902195, 20318889, 19013183, 19012119, 20464614, 19067244, 21632821 19512341, 19841800, 20331945, 19587324, 24316947, 19578350, 19637186 18674047, 19054077, 20898997, 19708632, 21091431, 19289642, 20869721 19258504, 17365043, 19468347, 21373473, 16359751, 19439759, 19769480 19272708, 19978542, 20402832, 19329654, 19873610, 23229229, 21517440 13542050, 19291380, 21915719, 19076343, 19561643, 19990037, 19487147 18909599, 20831538, 18250893, 19016730, 16619249, 18354830, 21188584 19989009, 17414008, 20688221, 20704450, 20441797, 19157754, 18885870 21785691, 21450666, 18893947, 18705806, 22223463, 16923858, 18417036 20919320, 20474192, 22046677, 19385656, 19501299, 20920911, 20899461 21387128, 21315084, 18122373, 20581111, 19606174, 18436647, 19023822 19178851, 19124589, 19597583, 18499088, 19050649
Patch 23854735 : applied on Wed Jan 11 06:42:32 EET 2017 Unique Patch ID: 20511532 Patch description: "OCW Patch Set Update : 12.1.0.2.161018 (23854735)" Created on 29 Sep 2016, 23:50:00 hrs PST8PDT Bugs fixed: 21339083, 18589889, 20768643, 19061429, 19133945, 19341538, 19521169 20011424, 19361757, 19187207, 19302350, 19130141, 19530755, 19699720 19620918, 19168690, 18899171, 19244316, 19653795, 18634372, 19471722 19536337, 19184188, 19131709, 20160490, 20995001, 21131037, 19380733 20985167, 18943696, 20552947, 19591984, 18135723, 18461246, 20014326 19524857, 18890943, 19154753, 19499021, 19046190, 20923930, 20028666 19051385, 19050688, 19682695, 19831611, 21607972, 18922918, 21087799 18956780, 19683886, 23757020, 19026993, 20110156, 21269977, 18261648 18362971, 19558324, 22651583, 20185476, 18882642, 19478119, 19414274 17447588, 19262534, 20752167, 19147513, 22475834, 21242118, 19514085 20758997, 19659885, 19319904, 18703978, 23079844, 19700256, 19703246 20467806, 19192901, 21527118, 19479503, 19147751, 18901356, 21232394 18964974, 22162062, 18835366, 19013789, 19207286, 18950232, 22583325 19148791, 19556820, 16505171, 22261732, 22135419, 19449737, 21878613 21646169, 19187515, 19230771, 20795241, 19453778, 19853036, 18843572 18520351, 21430567, 19185148, 19232454, 21220597, 18541110, 18834955 19204743, 20175174, 19140891, 18747335, 20676340, 19270660, 19325701 20877003, 19222693, 19662663, 19684131, 19010177, 19272663, 18910576 18907170, 19426384, 9372680, 20737462, 19700294, 20289828, 19331454 22568500, 20278216, 20827053, 20119857, 19703199, 22456393, 20588164 18955644, 20991879, 18508710, 19908346, 19589221, 18952857, 19537762 21152052, 16286734, 20669114, 18762843, 18945249, 19184799, 19205086 19351363, 20091753, 19031737, 19079087, 19148367, 18968981, 19150517 20231741, 19217019, 18730096, 18975620, 19067804, 21259359, 19054979 19209951, 19318983, 18747382, 19154673, 18752378, 22905714, 19150088 19013444, 19234177, 18999857, 19075747, 19367276, 19612597, 19874047 19677813, 23049292, 19427050, 18910443, 20033787, 19148982, 19777496 20730170, 18523468, 19277814, 20562133, 19071526, 21420753, 18850051 20433932, 20061168, 19370739, 20822591, 19609388, 18776786, 20136892 18989446, 19148793, 19585454, 18317489, 19955755, 19059053, 19807548 19124972, 22229116, 18910748, 19147509, 18849896, 19617592, 19398098 22840085, 19076165, 19498411, 19704993, 21294273, 19459023, 20634511 21229744, 21474380, 21484367, 20276459, 17208793, 19234907, 19538714 22867050, 19383028, 19513969, 22195657, 23111368, 19986391, 19314048 18834934, 19473851, 19458082, 18894342, 19522067, 18116578, 19241857 19076778, 19066699, 19068003, 18937186, 20267877, 19916737, 19770543 22863466, 19635215, 19613533, 21057239, 20783416, 19315522, 19141785 19163887, 19885321, 18715868, 18852058, 19701617, 19280860, 19139608 21453435, 18946768, 19135521, 18604143, 19627476, 20568884, 20317221 20177779, 19266658, 24522717, 18330979, 19027351, 21227730, 18707416 21118157, 21113068, 19925992, 20235486, 20006646, 15922206, 18439295 18991776, 21245437, 23170839, 19163425, 19550195, 18849021, 18861196 17940721, 18524834, 19287706, 19522313, 18835283, 18748932, 19184765 21639478, 14851828, 20598625, 19226141, 19053891, 18871287, 18998228 18980002, 19682778, 19325604, 18777835, 17338864, 18945906, 19513650 19702758, 18952577, 19473088, 19178517, 18729269, 19529729, 19455563 20877072, 23275930, 20340620, 18536826, 19292605, 22264951, 20920711 20660273, 20011635, 17592037, 18935857, 22989069, 19029647, 19179158 22302421, 23076471, 19140712, 20012766, 23049301, 19184276, 20510208 20001507, 20079414, 19680763, 19259765, 18962892, 19513888, 14661880 21519340, 21222147, 20218012, 19551830, 21616278, 19068333, 18945435 20775015, 19319192, 20839581, 19178629, 21694237, 19304104, 19284357 22594310, 20725084, 19457575, 21297657, 19069755, 19021575, 18715884 19584688, 18798573, 19812592, 19018001, 19292272, 19270956, 21967791 21794779, 23055224, 18700893, 21276514, 20467377, 18406774, 20607821 21976167, 21082737, 21694632, 20115586, 20883009, 19164099, 18283792 19155350, 19884033, 19146822, 18798432, 21123424, 19471836, 20936562 18750781, 19045143, 22483804, 19146980, 19601003, 20099624, 21378016 19537547, 19281106, 18862203, 19954372, 18235274, 19205617, 19513351 20219458, 19150313, 18843054, 18708349, 19332386, 18953639, 21423055 19371270, 19203996, 20038431, 19370846, 21053036, 18998379, 20157569 21357522, 19273577, 24709351, 22113170, 19632437, 18990354, 19288396 20620581, 19557558, 19127078, 21792881, 20900588, 20053557, 22088198 22664609, 19315567, 19444471, 19154933, 20953798, 18290252, 18813323 19915945, 19500293, 18643483, 19134098, 23034352, 18965694, 19226858 19591356, 19602208, 22884308, 20069197, 18417590, 18920408, 18636884 22289350, 19043795, 20746507, 18260170, 18919682, 18678829, 19319357 19273758, 18953878, 18999195, 18759724, 19066844, 13843841, 20290467 22934855, 19649640, 19062675, 21924910, 18859710, 20862812, 19504641 23101611, 19341481, 20293730, 19343245, 18304090, 19241655, 19470791 18242738, 20649573, 18372060, 18953889, 18827679, 19140711, 19023430 19259290, 19045388, 20186278, 19522571, 19709663, 18875012, 18861564 19273760, 19225265, 15885689, 20853464, 21696068, 18819158, 21394146 19049721, 19368917, 21255373, 22393909, 20408163, 20579351, 18868829 20350915, 21327402, 22024217, 23101338, 19820247, 20363708, 19538241 19804032 Patch 21436941 : applied on Wed Jan 11 06:37:11 EET 2017 Unique Patch ID: 19245012 Patch description: "WLM Patch Set Update: 12.1.0.2.5 (21436941)" Created on 13 Aug 2015, 04:00:40 hrs PST8PDT Bugs fixed: 20075154, 20641027, 20548410, 19016964, 19582630 Patch level status of Cluster nodes : Patching Level Nodes -------------- ----- 2615413759 dbabg-rac1,hdbabg-rac2 -------------------------------------------------------------------------------- OPatch succeeded. [grid@dbabg-rac1 OPatch]$ |
- Now i should fix service startup issue, checking CRS alert.log
2017-01-11 11:31:53.122315 : AGFW:1734321920: {0:0:438} Preparing START command for: ora.crf 1 1 2017-01-11 11:31:53.122802 : AGFW:1734321920: {0:0:438} ora.crf 1 1 state changed from: OFFLINE to: STARTING 2017-01-11 11:31:53.124525 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] (:CLSN00107:) clsn_agent::start { 2017-01-11 11:31:53.126958 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] __IS_HASD_AGENT=TRUE 2017-01-11 11:31:53.129038 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] CSSD Mode= 2017-01-11 11:31:53.129092 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] }DaemonAgent::start entry 2017-01-11 11:31:53.129143 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] Utils::getOracleHomeAttrib getEnvVar oracle_home:/u01/app/12.1.0/grid 2017-01-11 11:31:53.129152 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] Utils::getOracleHomeAttrib oracle_home:/u01/app/12.1.0/grid 2017-01-11 11:31:53.129175 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] Utils::execCmd 1 USR_ORA_ENV: oracleHome:/u01/app/12.1.0/grid CrsHome:/u01/app/12.1.0/grid 2017-01-11 11:31:53.129197 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] Adding Environment Variables ORA_DAEMON_LOGGING_LEVELS=CRFMOND=0,CRFLDREP=0,CRFLOGD=0,CRFPROXY=0,OCLUMON=0,OCRAPI=0,OCRCLI=0,OCRMSG=0,CSSCLNT=0,CRFM=0,CRFCLI=0,CLSINET=0 2017-01-11 11:31:53.129207 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] Adding Environment Variables ORA_DAEMON_TRACING_LEVELS=CRFMOND=0,CRFLDREP=0,CRFLOGD=0,CRFPROXY=0,OCLUMON=0,OCRAPI=0,OCRCLI=0,OCRMSG=0,CSSCLNT=0,CRFM=0,CRFCLI=0,CLSINET=0 2017-01-11 11:31:53.129215 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] Adding Environment Variables START_MODE= 2017-01-11 11:31:53.129265 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] Adding Environment Variables _ORA_AGENT_ACTION=TRUE 2017-01-11 11:31:53.129274 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] Adding Environment Variables __IS_HASD_AGENT= 2017-01-11 11:31:53.129286 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] Utils:execCmd action = 1 flags = 5 ohome = (null) cmdname = osysmond. 2017-01-11 11:31:53.129311 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] Utils::getOracleHomeAttrib getEnvVar oracle_home:/u01/app/12.1.0/grid 2017-01-11 11:31:53.129318 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] Utils::getOracleHomeAttrib oracle_home:/u01/app/12.1.0/grid 2017-01-11 11:31:53.129679 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] (:CLSN00008:)Utils:execCmd scls_process_spawn() failed 1 2017-01-11 11:31:53.129692 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] (:CLSN00008:) category: -1, operation: fail, loc: canexec2, OS error: 0, other: no exe permission, file [/u01/app/12.1.0/grid/bin/osysmond] 2017-01-11 11:31:53.141128 :CLSDYNAM:1730119424: [ ora.crf]{0:0:438} [start] clsnUtils::error Exception type=2 string= CRS-5013: Agent "ORAROOTAGENT" failed to start process "/u01/app/12.1.0/grid/bin/osysmond" for action "start": details at "(:CLSN00008:)" in "/u01/app/grid/diag/crs/hdbabg-rac2/crs/trace/ohasd_orarootagent_root.trc" 2017-01-11 11:31:53.142089 : AGFW:1730119424: {0:0:438} sending status msg [CRS-5013: Agent "ORAROOTAGENT" failed to start process "/u01/app/12.1.0/grid/bin/osysmond" for action "start": details at "(:CLSN00008:)" in "/u01/app/grid/diag/crs/hdbabg-rac2/crs/trace/ohasd_orarootagent_root.trc" ] for start for resource: ora.crf 1 1 2017-01-11 11:31:53.143871 : AGFW:1734321920: {0:0:438} Agent sending reply for: RESOURCE_START[ora.crf 1 1] ID 4098:1702 |
- Checking trace
[grid@hdbabg-rac2 OPatch]$ vi "/u01/app/grid/diag/crs/hdbabg-rac2/crs/trace/ohasd_orarootagent_root.trc"
- As per the above trace, an error is visible related to the below executables
[grid@hdbabg-rac2 OPatch]$ ls -la /u01/app/12.1.0/grid/bin/osysmond -rwxr-x---. 1 grid oinstall 9526 Jan 11 10:09 /u01/app/12.1.0/grid/bin/osysmond [grid@hdbabg-rac2 OPatch]$ ls -la /u01/app/12.1.0/grid/bin/ologgerd -rwxr-x---. 1 grid oinstall 102183 Oct 18 13:00 /u01/app/12.1.0/grid/bin/ologgerd [grid@hdbabg-rac2 OPatch]$ ls -la /u01/app/12.1.0/grid/bin/ologgerd -rwxr-x---. 1 grid oinstall 102183 Oct 18 13:00 /u01/app/12.1.0/grid/bin/ologgerd [grid@hdbabg-rac2 OPatch]$ ls -la /u01/app/12.1.0/grid/bin/osysmond -rwxr-x---. 1 grid oinstall 9526 Jan 11 10:09 /u01/app/12.1.0/grid/bin/osysmond [grid@hdbabg-rac2 OPatch]$ ls -la /u01/app/12.1.0/grid/bin/osysmond.bin -rwxr-x---. 1 grid oinstall 102430 Oct 18 13:00 /u01/app/12.1.0/grid/bin/osysmond.bin |
Change user and group
- GO with Root user and execute chown command
[root@hdbabg-rac2 install]# chown root:oinstall /u01/app/12.1.0/grid/bin/osysmond [root@hdbabg-rac2 install]# chown root:oinstall /u01/app/12.1.0/grid/bin/ologgerd [root@hdbabg-rac2 install]# chown root:oinstall /u01/app/12.1.0/grid/bin/osysmond [root@hdbabg-rac2 install]# chown root:oinstall /u01/app/12.1.0/grid/bin/osysmond.bin |
- Checking changes
[root@hdbabg-rac2 install]# ls -la /u01/app/12.1.0/grid/bin/osysmond -rwxr-x---. 1 root oinstall 9526 Jan 11 10:09 /u01/app/12.1.0/grid/bin/osysmond [root@hdbabg-rac2 install]# ls -la /u01/app/12.1.0/grid/bin/ologgerd -rwxr-x---. 1 root oinstall 102183 Oct 18 13:00 /u01/app/12.1.0/grid/bin/ologgerd [root@hdbabg-rac2 install]# ls -la /u01/app/12.1.0/grid/bin/ologgerd -rwxr-x---. 1 root oinstall 102183 Oct 18 13:00 /u01/app/12.1.0/grid/bin/ologgerd [root@hdbabg-rac2 install]# ls -la /u01/app/12.1.0/grid/bin/osysmond -rwxr-x---. 1 root oinstall 9526 Jan 11 10:09 /u01/app/12.1.0/grid/bin/osysmond [root@hdbabg-rac2 install]# ls -la /u01/app/12.1.0/grid/bin/osysmond.bin -rwxr-x---. 1 root oinstall 102430 Oct 18 13:00 /u01/app/12.1.0/grid/bin/osysmond.bin |
- Try to start resources
[root@hdbabg-rac2 install]# /u01/app/12.1.0/grid/bin/crsctl start res ora.crf -init CRS-2672: Attempting to start 'ora.crf' on 'hdbabg-rac2' CRS-2676: Start of 'ora.crf' on 'hdbabg-rac2' succeeded |
:) hahaha now it's working, but the question here is why should we fix executable permissions like this.. Maybe due to the patching issue, the correct permissions were not applied.
But as per the metalink, by invoking -prepatch/-postpatch this should fix the issue.....
- checking all resources
[root@hdbabg-rac2 install]# /u01/app/12.1.0/grid/bin/crsctl stat res -t -init -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.asm ONLINE ONLINE hdbabg-rac2 Started,STABLE ora.cluster_interconnect.haip ONLINE ONLINE hdbabg-rac2 STABLE ora.crf ONLINE ONLINE hdbabg-rac2 STABLE ora.crsd ONLINE ONLINE hdbabg-rac2 STABLE ora.cssd ONLINE ONLINE hdbabg-rac2 STABLE ora.cssdmonitor ONLINE ONLINE hdbabg-rac2 STABLE ora.ctssd ONLINE ONLINE hdbabg-rac2 OBSERVER,STABL ora.diskmon OFFLINE OFFLINE STABLE ora.drivers.acfs ONLINE ONLINE hdbabg-rac2 STABLE ora.evmd ONLINE ONLINE hdbabg-rac2 STABLE ora.gipcd ONLINE ONLINE hdbabg-rac2 STABLE ora.gpnpd ONLINE ONLINE hdbabg-rac2 STABLE ora.mdnsd ONLINE ONLINE hdbabg-rac2 STABLE ora.storage ONLINE ONLINE hdbabg-rac2 STABLE -------------------------------------------------------------------------------- [root@hdbabg-rac2 install]# |
With all this, it should be obvious that you should first validate patching process on test environment prior to any production patching!!!!!
Thanks for your time and i hope this was useful!
Please let me know if you have any comments.
2 comments
Comment from: Simeon Visitor

Comment from: kkovachki Member

Hello Simo,
You are very welcome!!!!
Please send us your issue?
Regards,
Krasi
Simo: Krasi , again I used this article, thanks :)