拓冰网站定制 
由于某种故障导致crs的OCR_0001盘掉线votedisk从3个变为了2个WARNING: Write Failed. group:3 disk:1 AU:1 offset:4190208 size:4096WARNING: Hbeat write to PST disk 1.3915948466ingroup 3 failed. [4]Mon Jun 14 15:31:11 2021NOTE: process _b000_asm1 (21889) initiating offline of disk 1.3915948466 (OCR_0001) with mask 0x7eingroup 3NOTE: checking PST: grp 3GMON checking disk modesforgroup 3 at 14forpid 28, osid 21889NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: checking PSTforgrp 3done.NOTE: sendingsetoffline flag message 1047812201 to 1 disk(s)ingroup 3WARNING: Disk OCR_0001inmode 0x7f is now being offlinedINFO: Instance#2 could not find disk 1 in group 3NOTE: initiating PST update: grp 3, dsk 1/0xe968a1b2, mask 0x6a,opclearGMON updating disk modesforgroup 3 at 15forpid 28, osid 21889NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: PST update grp 3 completed successfullyNOTE: initiating PST update: grp 3, dsk 1/0xe968a1b2, mask 0x7e,opclearGMON updating disk modesforgroup 3 at 16forpid 28, osid 21889NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: cache closing disk 1 of grp 3: OCR_0001NOTE: PST update grp 3 completed successfullyMon Jun 14 15:31:13 2021NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 3 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Successful votingfilerelocation on diskgroup OCRNOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 2 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Successful votingfilerelocation on diskgroup OCRMon Jun 14 15:34:08 2021WARNING: PST-initiated drop of 1 disk(s)ingroup 3(.1918390620))SQL alter diskgroup OCR drop disk OCR_0001 force /* ASM SERVER */NOTE: GroupBlock outside rolling migration privileged regionNOTE: requesting all-instance membership refreshforgroup3Mon Jun 14 15:34:10 2021GMON updatingforreconfiguration, group 3 at 17forpid 28, osid 21889NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: cache closing disk 1 of grp 3: (notopen) OCR_0001NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: group 3 PST updated.Mon Jun 14 15:34:10 2021NOTE: membership refresh pendingforgroup 3/0x7258515c(OCR)NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 2 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Successful votingfilerelocation on diskgroup OCRGMON querying group 3 at 18forpid 18, osid 8900NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: cache closing disk 1 of grp 3: (notopen) _DROPPED_0001_OCRSUCCESS: refreshed membershipfor3/0x7258515c(OCR)SUCCESS: alter diskgroup OCR drop disk OCR_0001 force /* ASM SERVER */在第一次掉盘之后rebalance完成之后又掉一块盘,ocr磁盘组正常,表决盘因为就只有一个磁盘,无法在ocr磁盘组中refresh到其他磁盘上Tue Jun 15 04:41:42 2021WARNING: Waited 15 secsforwrite IO to PST disk 0ingroup 3.WARNING: Waited 15 secsforwrite IO to PST disk 0ingroup 3.Tue Jun 15 04:41:42 2021NOTE: process _b000_asm1 (58548) initiating offline of disk 0.3915948465 (OCR_0000) with mask 0x7eingroup 3NOTE: checking PST: grp 3GMON checking disk modesforgroup 3 at 23forpid 28, osid 58548NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: checking PSTforgrp 3done.NOTE: sendingsetoffline flag message 3615961191 to 1 disk(s)ingroup 3WARNING: Disk OCR_0000inmode 0x7f is now being offlinedINFO: Instance#2 could not find disk 1 in group 3NOTE: initiating PST update: grp 3, dsk 0/0xe968a1b1, mask 0x6a,opclearGMON updating disk modesforgroup 3 at 24forpid 28, osid 58548NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: PST update grp 3 completed successfullyNOTE: initiating PST update: grp 3, dsk 0/0xe968a1b1, mask 0x7e,opclearGMON updating disk modesforgroup 3 at 25forpid 28, osid 58548NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: cache closing disk 0 of grp 3: OCR_0000NOTE: PST update grp 3 completed successfullyTue Jun 15 04:41:44 2021NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 2 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRWARNING: Waited 18 secsforwrite IO to PST disk 0ingroup 3.WARNING: Waited 18 secsforwrite IO to PST disk 0ingroup 3.NOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRTue Jun 15 04:44:21 2021NOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRTue Jun 15 04:44:21 2021WARNING: PST-initiated drop of 1 disk(s)ingroup 3(.1918390620))SQL alter diskgroup OCR drop disk OCR_0000 force /* ASM SERVER */NOTE: GroupBlock outside rolling migration privileged regionNOTE: requesting all-instance membership refreshforgroup3NOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRTue Jun 15 04:44:24 2021GMON updatingforreconfiguration, group 3 at 26forpid 28, osid 58548NOTE: cache closing disk 0 of grp 3: (notopen) OCR_0000NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: group 3 PST updated.NOTE: membership refresh pendingforgroup 3/0x7258515c(OCR)NOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRGMON querying group 3 at 27forpid 18, osid 8900NOTE: cache closing disk 0 of grp 3: (notopen) _DROPPED_0000_OCRSUCCESS: refreshed membershipfor3/0x7258515c(OCR)NOTE: starting rebalance of group 3/0x7258515c(OCR) at power 1SUCCESS: alter diskgroup OCR drop disk OCR_0000 force /* ASM SERVER */查询这个时候的ocr磁盘组相关信息可以明显的看到ocr磁盘组只剩余1个disk,查询表决盘信息node1- crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 3619aee7c3b04fc1bfa5c4ce659acbf7 (/dev/emcpowerc) [OCR]2. ONLINE 00bc3e79f7404ff2bf60925a7b8a5a6d (/dev/emcpowere) [OCR]Located 2 voting disk(s).可以发现表决盘中的两个disk一个属于ocr磁盘组一个是被ocr磁盘组drop掉的磁盘尝试增加以前离线的磁盘到ocr磁盘组SQLalterdiskgroup OCRadddisk/dev/emcpowerc;alterdiskgroup OCRadddisk/dev/emcpowerc*ERRORatline 1:ORA-15032:notallalterations performedORA-15033: disk/dev/emcpowercbelongstodiskgroupOCRSQLalterdiskgroup OCRadddisk/dev/emcpowercforce2 ;alterdiskgroup OCRadddisk/dev/emcpowercforce*ERRORatline 1:ORA-03113:end-of-fileoncommunication channelProcess ID: 15191Session ID: 1613 Serial number: 7查看alert日志SQL alter diskgroup OCR add disk/dev/emcpowercforceNOTE: GroupBlock outside rolling migration privileged regionNOTE: Assigning number (3,4) to disk (/dev/emcpowerc)NOTE: requesting all-instance membership refreshforgroup3WARNING: ignoring disk/dev/emcpowerdindeep discoveryNOTE: initializing header on grp 3 disk OCR_0004WARNING: ignoring disk/dev/emcpowerdindeep discoveryNOTE: requesting all-instance disk validationforgroup3NOTE: skipping rediscoveryforgroup 3/0x725d2390(OCR) onlocalinstance.NOTE: requesting all-instance disk validationforgroup3NOTE: skipping rediscoveryforgroup 3/0x725d2390(OCR) onlocalinstance.NOTE: Attempting votingfilerelocation on diskgroup OCRErrorsinfile/u01/app/grid/diag/asm/asm/ASM1/trace/ASM1_rbal_12207.trc (incident311185):ORA-00600: internal error code, arguments: [kfdvfGetCurrent_baddsk], [], [], [], [], [], [], [], [], [], [], []Incident detailsin:/u01/app/grid/diag/asm/asm/ASM1/incident/incdir_311185/ASM1_rbal_12207_i311185.trcUse ADRCI or Support Workbench to package the incident.See Note 411.1 at My Oracle Supportforerror and packaging details.ERROR: ORA-600 throwninRBALforgroup number 3Errorsinfile/u01/app/grid/diag/asm/asm/ASM1/trace/ASM1_rbal_12207.trc:ORA-00600: internal error code, arguments: [kfdvfGetCurrent_baddsk], [], [], [], [], [], [], [], [], [], [], []Errorsinfile/u01/app/grid/diag/asm/asm/ASM1/trace/ASM1_rbal_12207.trc:ORA-00600: internal error code, arguments: [kfdvfGetCurrent_baddsk], [], [], [], [], [], [], [], [], [], [], []RBAL (ospid: 12207): terminating the instance due to error 488由于ORA-600 kfdvfGetCurrent_baddsk错误导致增加磁盘失败,通过上面查询的votedisk的信息,可以发现emcpowerc这个盘虽然ocr中离线,但是依旧还是votedisk盘,因此无法增加到该磁盘组中,采用变通方法,先加另外一块盘SQL alter diskgroup OCR add failgroup OCR_0001 disk/dev/emcpowerdforce;Diskgroup altered.SQLexitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management optionsnode1- crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 00bc3e79f7404ff2bf60925a7b8a5a6d (/dev/emcpowere) [OCR]2. ONLINE 0eef8152df5d4f41bf973ad5dc5a6cb1 (/dev/emcpowerd) [OCR]Located 2 voting disk(s).增加成功emcpowerd之后,emcpowerc已经不再是表决盘变为了emcpowerd,再次增加emcpowercSQL alter diskgroup OCR add failgroup OCR_0000 disk/dev/emcpowercforce;Diskgroup altered.SQLexitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management optionsnode1- crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 00bc3e79f7404ff2bf60925a7b8a5a6d (/dev/emcpowere) [OCR]2. ONLINE 0eef8152df5d4f41bf973ad5dc5a6cb1 (/dev/emcpowerd) [OCR]3. ONLINE 4f6201f808dc4ff3bf928b14eae0d4a6 (/dev/emcpowerc) [OCR]Located 3 voting disk(s).ASMCMD lsdsk -G ocrPath/dev/emcpowerc/dev/emcpowerd/dev/emcpowereSQL alter diskgroup OCR add failgroup OCR_0000 disk/dev/emcpowercforceNOTE: GroupBlock outside rolling migration privileged regionNOTE: Assigning number (3,0) to disk (/dev/emcpowerc)NOTE: requesting all-instance membership refreshforgroup3NOTE: initializing header on grp 3 disk OCR_0000NOTE: requesting all-instance disk validationforgroup3Mon Jan 24 17:47:42 2022NOTE: skipping rediscoveryforgroup 3/0x725dccb9(OCR) onlocalinstance.NOTE: requesting all-instance disk validationforgroup3NOTE: skipping rediscoveryforgroup 3/0x725dccb9(OCR) onlocalinstance.Mon Jan 24 17:47:48 2022GMON updatingforreconfiguration, group 3 at 20forpid 30, osid 16978NOTE: group 3 PST updated.NOTE: initiating PST update: grp 3GMON updating group 3 at 21forpid 30, osid 16978NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: group OCR: updated PST location: disk 0005 (PST copy 1)NOTE: group OCR: updated PST location: disk 0000 (PST copy 2)NOTE: PST update grp 3 completed successfullyNOTE: membership refresh pendingforgroup 3/0x725dccb9(OCR)NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 2 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRGMON querying group 3 at 22forpid 18, osid 15952NOTE: cache opening disk 0 of grp 3: OCR_0000 path:/dev/emcpowercMon Jan 24 17:47:53 2022NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 2 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRGMON querying group 3 at 23forpid 18, osid 15952SUCCESS: refreshed membershipfor3/0x725dccb9(OCR)Mon Jan 24 17:47:53 2022SUCCESS: alter diskgroup OCR add failgroup OCR_0000 disk/dev/emcpowercforceNOTE: starting rebalance of group 3/0x725dccb9(OCR) at power 1Starting background process ARB0Mon Jan 24 17:47:53 2022ARB0 started with pid31, OSid17092NOTE: assigning ARB0 to group 3/0x725dccb9(OCR) with 1 parallel I/Ocellip.ora not found.NOTE: F1X0 copy 3 relocating from 65534:4294967294 to 0:2fordiskgroup 3 (OCR)NOTE: stopping process ARB0SUCCESS: rebalance completedforgroup 3/0x725dccb9(OCR)NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 2 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: votingfileallocation on grp 3 disk OCR_0000NOTE: Successful votingfilerelocation on diskgroup OCRMon Jan 24 17:47:57 2022NOTE: GroupBlock outside rolling migration privileged regionNOTE: requesting all-instance membership refreshforgroup3NOTE: membership refresh pendingforgroup 3/0x725dccb9(OCR)Mon Jan 24 17:48:03 2022GMON querying group 3 at 24forpid 18, osid 15952SUCCESS: refreshed membershipfor3/0x725dccb9(OCR)Mon Jan 24 17:48:06 2022NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 3 votingfile(s).表决磁盘组从2个变为了3个,ocr磁盘组也恢复了正常的3个,至此OCR掉盘的故障处理完成 ocr磁盘组掉盘故障处理
由于某种故障导致crs的OCR_0001盘掉线votedisk从3个变为了2个WARNING: Write Failed. group:3 disk:1 AU:1 offset:4190208 size:4096WARNING: Hbeat write to PST disk 1.3915948466ingroup 3 failed. [4]Mon Jun 14 15:31:11 2021NOTE: process _b000_asm1 (21889) initiating offline of disk 1.3915948466 (OCR_0001) with mask 0x7eingroup 3NOTE: checking PST: grp 3GMON checking disk modesforgroup 3 at 14forpid 28, osid 21889NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: checking PSTforgrp 3done.NOTE: sendingsetoffline flag message 1047812201 to 1 disk(s)ingroup 3WARNING: Disk OCR_0001inmode 0x7f is now being offlinedINFO: Instance#2 could not find disk 1 in group 3NOTE: initiating PST update: grp 3, dsk 1/0xe968a1b2, mask 0x6a,opclearGMON updating disk modesforgroup 3 at 15forpid 28, osid 21889NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: PST update grp 3 completed successfullyNOTE: initiating PST update: grp 3, dsk 1/0xe968a1b2, mask 0x7e,opclearGMON updating disk modesforgroup 3 at 16forpid 28, osid 21889NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: cache closing disk 1 of grp 3: OCR_0001NOTE: PST update grp 3 completed successfullyMon Jun 14 15:31:13 2021NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 3 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Successful votingfilerelocation on diskgroup OCRNOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 2 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Successful votingfilerelocation on diskgroup OCRMon Jun 14 15:34:08 2021WARNING: PST-initiated drop of 1 disk(s)ingroup 3(.1918390620))SQL alter diskgroup OCR drop disk OCR_0001 force /* ASM SERVER */NOTE: GroupBlock outside rolling migration privileged regionNOTE: requesting all-instance membership refreshforgroup3Mon Jun 14 15:34:10 2021GMON updatingforreconfiguration, group 3 at 17forpid 28, osid 21889NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: cache closing disk 1 of grp 3: (notopen) OCR_0001NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: group 3 PST updated.Mon Jun 14 15:34:10 2021NOTE: membership refresh pendingforgroup 3/0x7258515c(OCR)NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 2 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Successful votingfilerelocation on diskgroup OCRGMON querying group 3 at 18forpid 18, osid 8900NOTE: group OCR: updated PST location: disk 0000 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 1)NOTE: cache closing disk 1 of grp 3: (notopen) _DROPPED_0001_OCRSUCCESS: refreshed membershipfor3/0x7258515c(OCR)SUCCESS: alter diskgroup OCR drop disk OCR_0001 force /* ASM SERVER */在第一次掉盘之后rebalance完成之后又掉一块盘,ocr磁盘组正常,表决盘因为就只有一个磁盘,无法在ocr磁盘组中refresh到其他磁盘上Tue Jun 15 04:41:42 2021WARNING: Waited 15 secsforwrite IO to PST disk 0ingroup 3.WARNING: Waited 15 secsforwrite IO to PST disk 0ingroup 3.Tue Jun 15 04:41:42 2021NOTE: process _b000_asm1 (58548) initiating offline of disk 0.3915948465 (OCR_0000) with mask 0x7eingroup 3NOTE: checking PST: grp 3GMON checking disk modesforgroup 3 at 23forpid 28, osid 58548NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: checking PSTforgrp 3done.NOTE: sendingsetoffline flag message 3615961191 to 1 disk(s)ingroup 3WARNING: Disk OCR_0000inmode 0x7f is now being offlinedINFO: Instance#2 could not find disk 1 in group 3NOTE: initiating PST update: grp 3, dsk 0/0xe968a1b1, mask 0x6a,opclearGMON updating disk modesforgroup 3 at 24forpid 28, osid 58548NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: PST update grp 3 completed successfullyNOTE: initiating PST update: grp 3, dsk 0/0xe968a1b1, mask 0x7e,opclearGMON updating disk modesforgroup 3 at 25forpid 28, osid 58548NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: cache closing disk 0 of grp 3: OCR_0000NOTE: PST update grp 3 completed successfullyTue Jun 15 04:41:44 2021NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 2 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRWARNING: Waited 18 secsforwrite IO to PST disk 0ingroup 3.WARNING: Waited 18 secsforwrite IO to PST disk 0ingroup 3.NOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRTue Jun 15 04:44:21 2021NOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRTue Jun 15 04:44:21 2021WARNING: PST-initiated drop of 1 disk(s)ingroup 3(.1918390620))SQL alter diskgroup OCR drop disk OCR_0000 force /* ASM SERVER */NOTE: GroupBlock outside rolling migration privileged regionNOTE: requesting all-instance membership refreshforgroup3NOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRTue Jun 15 04:44:24 2021GMON updatingforreconfiguration, group 3 at 26forpid 28, osid 58548NOTE: cache closing disk 0 of grp 3: (notopen) OCR_0000NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: group 3 PST updated.NOTE: membership refresh pendingforgroup 3/0x7258515c(OCR)NOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRGMON querying group 3 at 27forpid 18, osid 8900NOTE: cache closing disk 0 of grp 3: (notopen) _DROPPED_0000_OCRSUCCESS: refreshed membershipfor3/0x7258515c(OCR)NOTE: starting rebalance of group 3/0x7258515c(OCR) at power 1SUCCESS: alter diskgroup OCR drop disk OCR_0000 force /* ASM SERVER */查询这个时候的ocr磁盘组相关信息可以明显的看到ocr磁盘组只剩余1个disk,查询表决盘信息node1- crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 3619aee7c3b04fc1bfa5c4ce659acbf7 (/dev/emcpowerc) [OCR]2. ONLINE 00bc3e79f7404ff2bf60925a7b8a5a6d (/dev/emcpowere) [OCR]Located 2 voting disk(s).可以发现表决盘中的两个disk一个属于ocr磁盘组一个是被ocr磁盘组drop掉的磁盘尝试增加以前离线的磁盘到ocr磁盘组SQLalterdiskgroup OCRadddisk/dev/emcpowerc;alterdiskgroup OCRadddisk/dev/emcpowerc*ERRORatline 1:ORA-15032:notallalterations performedORA-15033: disk/dev/emcpowercbelongstodiskgroupOCRSQLalterdiskgroup OCRadddisk/dev/emcpowercforce2 ;alterdiskgroup OCRadddisk/dev/emcpowercforce*ERRORatline 1:ORA-03113:end-of-fileoncommunication channelProcess ID: 15191Session ID: 1613 Serial number: 7查看alert日志SQL alter diskgroup OCR add disk/dev/emcpowercforceNOTE: GroupBlock outside rolling migration privileged regionNOTE: Assigning number (3,4) to disk (/dev/emcpowerc)NOTE: requesting all-instance membership refreshforgroup3WARNING: ignoring disk/dev/emcpowerdindeep discoveryNOTE: initializing header on grp 3 disk OCR_0004WARNING: ignoring disk/dev/emcpowerdindeep discoveryNOTE: requesting all-instance disk validationforgroup3NOTE: skipping rediscoveryforgroup 3/0x725d2390(OCR) onlocalinstance.NOTE: requesting all-instance disk validationforgroup3NOTE: skipping rediscoveryforgroup 3/0x725d2390(OCR) onlocalinstance.NOTE: Attempting votingfilerelocation on diskgroup OCRErrorsinfile/u01/app/grid/diag/asm/asm/ASM1/trace/ASM1_rbal_12207.trc (incident311185):ORA-00600: internal error code, arguments: [kfdvfGetCurrent_baddsk], [], [], [], [], [], [], [], [], [], [], []Incident detailsin:/u01/app/grid/diag/asm/asm/ASM1/incident/incdir_311185/ASM1_rbal_12207_i311185.trcUse ADRCI or Support Workbench to package the incident.See Note 411.1 at My Oracle Supportforerror and packaging details.ERROR: ORA-600 throwninRBALforgroup number 3Errorsinfile/u01/app/grid/diag/asm/asm/ASM1/trace/ASM1_rbal_12207.trc:ORA-00600: internal error code, arguments: [kfdvfGetCurrent_baddsk], [], [], [], [], [], [], [], [], [], [], []Errorsinfile/u01/app/grid/diag/asm/asm/ASM1/trace/ASM1_rbal_12207.trc:ORA-00600: internal error code, arguments: [kfdvfGetCurrent_baddsk], [], [], [], [], [], [], [], [], [], [], []RBAL (ospid: 12207): terminating the instance due to error 488由于ORA-600 kfdvfGetCurrent_baddsk错误导致增加磁盘失败,通过上面查询的votedisk的信息,可以发现emcpowerc这个盘虽然ocr中离线,但是依旧还是votedisk盘,因此无法增加到该磁盘组中,采用变通方法,先加另外一块盘SQL alter diskgroup OCR add failgroup OCR_0001 disk/dev/emcpowerdforce;Diskgroup altered.SQLexitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management optionsnode1- crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 00bc3e79f7404ff2bf60925a7b8a5a6d (/dev/emcpowere) [OCR]2. ONLINE 0eef8152df5d4f41bf973ad5dc5a6cb1 (/dev/emcpowerd) [OCR]Located 2 voting disk(s).增加成功emcpowerd之后,emcpowerc已经不再是表决盘变为了emcpowerd,再次增加emcpowercSQL alter diskgroup OCR add failgroup OCR_0000 disk/dev/emcpowercforce;Diskgroup altered.SQLexitDisconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management optionsnode1- crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 00bc3e79f7404ff2bf60925a7b8a5a6d (/dev/emcpowere) [OCR]2. ONLINE 0eef8152df5d4f41bf973ad5dc5a6cb1 (/dev/emcpowerd) [OCR]3. ONLINE 4f6201f808dc4ff3bf928b14eae0d4a6 (/dev/emcpowerc) [OCR]Located 3 voting disk(s).ASMCMD lsdsk -G ocrPath/dev/emcpowerc/dev/emcpowerd/dev/emcpowereSQL alter diskgroup OCR add failgroup OCR_0000 disk/dev/emcpowercforceNOTE: GroupBlock outside rolling migration privileged regionNOTE: Assigning number (3,0) to disk (/dev/emcpowerc)NOTE: requesting all-instance membership refreshforgroup3NOTE: initializing header on grp 3 disk OCR_0000NOTE: requesting all-instance disk validationforgroup3Mon Jan 24 17:47:42 2022NOTE: skipping rediscoveryforgroup 3/0x725dccb9(OCR) onlocalinstance.NOTE: requesting all-instance disk validationforgroup3NOTE: skipping rediscoveryforgroup 3/0x725dccb9(OCR) onlocalinstance.Mon Jan 24 17:47:48 2022GMON updatingforreconfiguration, group 3 at 20forpid 30, osid 16978NOTE: group 3 PST updated.NOTE: initiating PST update: grp 3GMON updating group 3 at 21forpid 30, osid 16978NOTE: group OCR: updated PST location: disk 0002 (PST copy 0)NOTE: group OCR: updated PST location: disk 0005 (PST copy 1)NOTE: group OCR: updated PST location: disk 0000 (PST copy 2)NOTE: PST update grp 3 completed successfullyNOTE: membership refresh pendingforgroup 3/0x725dccb9(OCR)NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 2 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRGMON querying group 3 at 22forpid 18, osid 15952NOTE: cache opening disk 0 of grp 3: OCR_0000 path:/dev/emcpowercMon Jan 24 17:47:53 2022NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 2 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: Failed votingfilerelocation on diskgroup OCRGMON querying group 3 at 23forpid 18, osid 15952SUCCESS: refreshed membershipfor3/0x725dccb9(OCR)Mon Jan 24 17:47:53 2022SUCCESS: alter diskgroup OCR add failgroup OCR_0000 disk/dev/emcpowercforceNOTE: starting rebalance of group 3/0x725dccb9(OCR) at power 1Starting background process ARB0Mon Jan 24 17:47:53 2022ARB0 started with pid31, OSid17092NOTE: assigning ARB0 to group 3/0x725dccb9(OCR) with 1 parallel I/Ocellip.ora not found.NOTE: F1X0 copy 3 relocating from 65534:4294967294 to 0:2fordiskgroup 3 (OCR)NOTE: stopping process ARB0SUCCESS: rebalance completedforgroup 3/0x725dccb9(OCR)NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 2 votingfile(s).NOTE: Votingfilerelocation is requiredindiskgroup OCRNOTE: Attempting votingfilerelocation on diskgroup OCRNOTE: votingfileallocation on grp 3 disk OCR_0000NOTE: Successful votingfilerelocation on diskgroup OCRMon Jan 24 17:47:57 2022NOTE: GroupBlock outside rolling migration privileged regionNOTE: requesting all-instance membership refreshforgroup3NOTE: membership refresh pendingforgroup 3/0x725dccb9(OCR)Mon Jan 24 17:48:03 2022GMON querying group 3 at 24forpid 18, osid 15952SUCCESS: refreshed membershipfor3/0x725dccb9(OCR)Mon Jan 24 17:48:06 2022NOTE: Attempting votingfilerefresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 3 votingfile(s).表决磁盘组从2个变为了3个,ocr磁盘组也恢复了正常的3个,至此OCR掉盘的故障处理完成