ORACLE Bug 4431215 引发的血案—原因分析篇
抱歉上文编辑不下，才续写此文可为什么该目录会丢失呢，原因是因为ORACLE的BUG，且看故障元凶：Bug 4431215的定位过程，看如下三个日志，节点1在归档到8211发生错误，节点2在归档到6852发生错误，而1点钟有个RMAN定时JOB发起2级备份，备份后会删除归档日志，从RMAN日志可以看出，最大的备份号分别为8210和6851，这和节点ALERT日志是完全一致的，可见在RMAN备份后归档便发生了错误，而RMAN备份后的瞬间，归档目录是空的，这难免让人想到ORACLE在ASM环境下的一个BUG：“在ASM上建立的目录名称和SID前缀相同，则这个目录下的所有目录一旦为空，就会被ASM实例自动删除，不管这个目录是ASM自动建立，还是用户手工建立的。”。登陆ORACLE官方站点确认有该BUG：“Bug 4431215 - "backup archivelog all delete input" from RMAN removes ASM directory ”，至此故障调查完成。
[*]节点1日志：
[*]Wed Jul 13 01:22:55 2011
[*]Thread 1 advanced to log sequence 8210 (LGWR switch)
[*]Current log# 2 seq# 8210 mem# 0: +DATA/port/onlinelog/group_2.264.697238179
[*]Current log# 2 seq# 8210 mem# 1: +DATA/port/onlinelog/group_2.265.697238179
[*]Wed Jul 13 01:23:00 2011
[*]ALTER SYSTEM ARCHIVE LOG
[*]Wed Jul 13 01:23:01 2011
[*]Thread 1 cannot allocate new log, sequence 8211
[*]Checkpoint not complete
[*]Current log# 2 seq# 8210 mem# 0: +DATA/port/onlinelog/group_2.264.697238179
[*]Current log# 2 seq# 8210 mem# 1: +DATA/port/onlinelog/group_2.265.697238179
[*]
[*]节点2日志：
[*]Wed Jul 13 01:20:27 2011
[*]Thread 2 advanced to log sequence 6851 (LGWR switch)
[*]Current log# 3 seq# 6851 mem# 0: +DATA/port/onlinelog/group_3.268.697238217
[*]Current log# 3 seq# 6851 mem# 1: +DATA/port/onlinelog/group_3.269.697238219
[*]Thread 2 cannot allocate new log, sequence 6852
[*]Checkpoint not complete
[*]Current log# 3 seq# 6851 mem# 0: +DATA/port/onlinelog/group_3.268.697238217
[*]Current log# 3 seq# 6851 mem# 1: +DATA/port/onlinelog/group_3.269.697238219
[*]Wed Jul 13 01:20:36 2011
[*]Thread 2 advanced to log sequence 6852 (LGWR switch)
[*]Current log# 4 seq# 6852 mem# 0: +DATA/port/onlinelog/group_4.270.697238219
[*]Current log# 4 seq# 6852 mem# 1: +DATA/port/onlinelog/group_4.271.697238221
[*]
[*]故障发生前的RMAN备份日志，备份完成后删除备份的归档日志：
[*]channel d1: specifying archive log(s) in backup set
[*]input archive log thread=1 sequence=8180 recid=22515 stamp=756327382
[*]input archive log thread=1 sequence=8181 recid=22516 stamp=756327390
[*]input archive log thread=1 sequence=8182 recid=22517 stamp=756327399
[*]input archive log thread=1 sequence=8183 recid=22519 stamp=756327406
[*]input archive log thread=1 sequence=8184 recid=22520 stamp=756327418
[*]input archive log thread=1 sequence=8185 recid=22521 stamp=756327426
[*]input archive log thread=1 sequence=8186 recid=22522 stamp=756327434
[*]input archive log thread=1 sequence=8187 recid=22524 stamp=756327443
[*]input archive log thread=1 sequence=8188 recid=22525 stamp=756327453
[*]input archive log thread=1 sequence=8189 recid=22527 stamp=756327462
[*]input archive log thread=1 sequence=8190 recid=22528 stamp=756327469
[*]input archive log thread=1 sequence=8191 recid=22529 stamp=756327477
[*]input archive log thread=1 sequence=8192 recid=22531 stamp=756327488
[*]input archive log thread=1 sequence=8193 recid=22532 stamp=756327496
[*]input archive log thread=1 sequence=8194 recid=22533 stamp=756327505
[*]input archive log thread=1 sequence=8195 recid=22535 stamp=756327514
[*]input archive log thread=1 sequence=8196 recid=22536 stamp=756327523
[*]input archive log thread=1 sequence=8197 recid=22537 stamp=756327538
[*]input archive log thread=1 sequence=8198 recid=22538 stamp=756327653
[*]input archive log thread=1 sequence=8199 recid=22540 stamp=756327664
[*]input archive log thread=1 sequence=8200 recid=22541 stamp=756327673
[*]input archive log thread=1 sequence=8201 recid=22543 stamp=756327686
[*]input archive log thread=1 sequence=8202 recid=22544 stamp=756327695
[*]input archive log thread=1 sequence=8203 recid=22545 stamp=756327704
[*]input archive log thread=1 sequence=8204 recid=22547 stamp=756327717
[*]input archive log thread=1 sequence=8205 recid=22548 stamp=756329007
[*]input archive log thread=1 sequence=8206 recid=22549 stamp=756338512
[*]input archive log thread=1 sequence=8207 recid=22551 stamp=756343058
[*]input archive log thread=1 sequence=8208 recid=22552 stamp=756345806
[*]input archive log thread=1 sequence=8209 recid=22553 stamp=756350578
[*]input archive log thread=1 sequence=8210 recid=22555 stamp=756350584
[*]input archive log thread=2 sequence=6800 recid=22340 stamp=756264103
[*]input archive log thread=2 sequence=6801 recid=22345 stamp=756265938
[*]input archive log thread=2 sequence=6802 recid=22356 stamp=756266232
[*]input archive log thread=2 sequence=6803 recid=22357 stamp=756266255
[*]input archive log thread=2 sequence=6804 recid=22359 stamp=756266260
[*]input archive log thread=2 sequence=6805 recid=22362 stamp=756266269
[*]input archive log thread=2 sequence=6806 recid=22363 stamp=756266272
[*]input archive log thread=2 sequence=6807 recid=22365 stamp=756266287
[*]input archive log thread=2 sequence=6808 recid=22369 stamp=756267485
[*]input archive log thread=2 sequence=6809 recid=22376 stamp=756291174
[*]input archive log thread=2 sequence=6810 recid=22380 stamp=756293726
[*]input archive log thread=2 sequence=6811 recid=22386 stamp=756299397
[*]input archive log thread=2 sequence=6812 recid=22390 stamp=756311219
[*]input archive log thread=2 sequence=6813 recid=22396 stamp=756313749
[*]input archive log thread=2 sequence=6814 recid=22402 stamp=756316744
[*]input archive log thread=2 sequence=6815 recid=22406 stamp=756319318
[*]input archive log thread=2 sequence=6816 recid=22412 stamp=756325509
[*]input archive log thread=2 sequence=6817 recid=22415 stamp=756326651
[*]input archive log thread=2 sequence=6818 recid=22420 stamp=756326683
[*]input archive log thread=2 sequence=6819 recid=22425 stamp=756326710
[*]input archive log thread=2 sequence=6820 recid=22428 stamp=756326732
[*]input archive log thread=2 sequence=6821 recid=22433 stamp=756326756
[*]input archive log thread=2 sequence=6822 recid=22438 stamp=756326780
[*]input archive log thread=2 sequence=6823 recid=22441 stamp=756326801
[*]input archive log thread=2 sequence=6824 recid=22445 stamp=756326823
[*]input archive log thread=2 sequence=6825 recid=22450 stamp=756326850
[*]input archive log thread=2 sequence=6826 recid=22455 stamp=756326878
[*]input archive log thread=2 sequence=6827 recid=22460 stamp=756326905
[*]input archive log thread=2 sequence=6828 recid=22464 stamp=756326930
[*]input archive log thread=2 sequence=6829 recid=22468 stamp=756326951
[*]input archive log thread=2 sequence=6830 recid=22472 stamp=756326969
[*]input archive log thread=2 sequence=6831 recid=22477 stamp=756326991
[*]input archive log thread=2 sequence=6832 recid=22481 stamp=756327012
[*]channel d1: starting piece 1 at 13-JUL-11
[*]channel d1: finished piece 1 at 13-JUL-11
[*]piece handle=/u01/rmanbak/arch_ZJPORT_1hmha00r_1_1 tag=BKARCH comment=NONE
[*]channel d1: backup set complete, elapsed time: 00:03:17
[*]channel d1: deleting archive log(s)
[*]archive log filename=+DATA/zjport/archivelog/1_8180_697238176.dbf recid=22515 stamp=756327382
[*]archive log filename=+DATA/zjport/archivelog/1_8181_697238176.dbf recid=22516 stamp=756327390
[*]archive log filename=+DATA/zjport/archivelog/1_8182_697238176.dbf recid=22517 stamp=756327399
[*]archive log filename=+DATA/zjport/archivelog/1_8183_697238176.dbf recid=22519 stamp=756327406
[*]archive log filename=+DATA/zjport/archivelog/1_8184_697238176.dbf recid=22520 stamp=756327418
[*]archive log filename=+DATA/zjport/archivelog/1_8185_697238176.dbf recid=22521 stamp=756327426
[*]archive log filename=+DATA/zjport/archivelog/1_8186_697238176.dbf recid=22522 stamp=756327434
[*]archive log filename=+DATA/zjport/archivelog/1_8187_697238176.dbf recid=22524 stamp=756327443
[*]archive log filename=+DATA/zjport/archivelog/1_8188_697238176.dbf recid=22525 stamp=756327453
[*]archive log filename=+DATA/zjport/archivelog/1_8189_697238176.dbf recid=22527 stamp=756327462
[*]archive log filename=+DATA/zjport/archivelog/1_8190_697238176.dbf recid=22528 stamp=756327469
[*]archive log filename=+DATA/zjport/archivelog/1_8191_697238176.dbf recid=22529 stamp=756327477
[*]archive log filename=+DATA/zjport/archivelog/1_8192_697238176.dbf recid=22531 stamp=756327488
[*]archive log filename=+DATA/zjport/archivelog/1_8193_697238176.dbf recid=22532 stamp=756327496
[*]archive log filename=+DATA/zjport/archivelog/1_8194_697238176.dbf recid=22533 stamp=756327505
[*]archive log filename=+DATA/zjport/archivelog/1_8195_697238176.dbf recid=22535 stamp=756327514
[*]archive log filename=+DATA/zjport/archivelog/1_8196_697238176.dbf recid=22536 stamp=756327523
[*]archive log filename=+DATA/zjport/archivelog/1_8197_697238176.dbf recid=22537 stamp=756327538
[*]archive log filename=+DATA/zjport/archivelog/1_8198_697238176.dbf recid=22538 stamp=756327653
[*]archive log filename=+DATA/zjport/archivelog/1_8199_697238176.dbf recid=22540 stamp=756327664
[*]archive log filename=+DATA/zjport/archivelog/1_8200_697238176.dbf recid=22541 stamp=756327673
[*]archive log filename=+DATA/zjport/archivelog/1_8201_697238176.dbf recid=22543 stamp=756327686
[*]archive log filename=+DATA/zjport/archivelog/1_8202_697238176.dbf recid=22544 stamp=756327695
[*]archive log filename=+DATA/zjport/archivelog/1_8203_697238176.dbf recid=22545 stamp=756327704
[*]archive log filename=+DATA/zjport/archivelog/1_8204_697238176.dbf recid=22547 stamp=756327717
[*]archive log filename=+DATA/zjport/archivelog/1_8205_697238176.dbf recid=22548 stamp=756329007
[*]archive log filename=+DATA/zjport/archivelog/1_8206_697238176.dbf recid=22549 stamp=756338512
[*]archive log filename=+DATA/zjport/archivelog/1_8207_697238176.dbf recid=22551 stamp=756343058
[*]archive log filename=+DATA/zjport/archivelog/1_8208_697238176.dbf recid=22552 stamp=756345806
[*]archive log filename=+DATA/zjport/archivelog/1_8209_697238176.dbf recid=22553 stamp=756350578
[*]archive log filename=+DATA/zjport/archivelog/1_8210_697238176.dbf recid=22555 stamp=756350584=======节点1的归档备份到8210
[*]archive log filename=+DATA/zjport/archivelog/2_6800_697238176.dbf recid=22340 stamp=756264103
[*]archive log filename=+DATA/zjport/archivelog/2_6801_697238176.dbf recid=22345 stamp=756265938
[*]archive log filename=+DATA/zjport/archivelog/2_6802_697238176.dbf recid=22356 stamp=756266232
[*]archive log filename=+DATA/zjport/archivelog/2_6803_697238176.dbf recid=22357 stamp=756266255
[*]archive log filename=+DATA/zjport/archivelog/2_6804_697238176.dbf recid=22359 stamp=756266260
[*]archive log filename=+DATA/zjport/archivelog/2_6805_697238176.dbf recid=22362 stamp=756266269
[*]archive log filename=+DATA/zjport/archivelog/2_6806_697238176.dbf recid=22363 stamp=756266272
[*]archive log filename=+DATA/zjport/archivelog/2_6807_697238176.dbf recid=22365 stamp=756266287
[*]archive log filename=+DATA/zjport/archivelog/2_6808_697238176.dbf recid=22369 stamp=756267485
[*]archive log filename=+DATA/zjport/archivelog/2_6809_697238176.dbf recid=22376 stamp=756291174
[*]archive log filename=+DATA/zjport/archivelog/2_6810_697238176.dbf recid=22380 stamp=756293726
[*]archive log filename=+DATA/zjport/archivelog/2_6811_697238176.dbf recid=22386 stamp=756299397
[*]archive log filename=+DATA/zjport/archivelog/2_6812_697238176.dbf recid=22390 stamp=756311219
[*]archive log filename=+DATA/zjport/archivelog/2_6813_697238176.dbf recid=22396 stamp=756313749
[*]archive log filename=+DATA/zjport/archivelog/2_6814_697238176.dbf recid=22402 stamp=756316744
[*]archive log filename=+DATA/zjport/archivelog/2_6815_697238176.dbf recid=22406 stamp=756319318
[*]archive log filename=+DATA/zjport/archivelog/2_6816_697238176.dbf recid=22412 stamp=756325509
[*]archive log filename=+DATA/zjport/archivelog/2_6817_697238176.dbf recid=22415 stamp=756326651
[*]archive log filename=+DATA/zjport/archivelog/2_6818_697238176.dbf recid=22420 stamp=756326683
[*]archive log filename=+DATA/zjport/archivelog/2_6819_697238176.dbf recid=22425 stamp=756326710
[*]archive log filename=+DATA/zjport/archivelog/2_6820_697238176.dbf recid=22428 stamp=756326732
[*]archive log filename=+DATA/zjport/archivelog/2_6821_697238176.dbf recid=22433 stamp=756326756
[*]archive log filename=+DATA/zjport/archivelog/2_6822_697238176.dbf recid=22438 stamp=756326780
[*]archive log filename=+DATA/zjport/archivelog/2_6823_697238176.dbf recid=22441 stamp=756326801
[*]archive log filename=+DATA/zjport/archivelog/2_6824_697238176.dbf recid=22445 stamp=756326823
[*]archive log filename=+DATA/zjport/archivelog/2_6825_697238176.dbf recid=22450 stamp=756326850
[*]archive log filename=+DATA/zjport/archivelog/2_6826_697238176.dbf recid=22455 stamp=756326878
[*]archive log filename=+DATA/zjport/archivelog/2_6827_697238176.dbf recid=22460 stamp=756326905
[*]archive log filename=+DATA/zjport/archivelog/2_6828_697238176.dbf recid=22464 stamp=756326930
[*]archive log filename=+DATA/zjport/archivelog/2_6829_697238176.dbf recid=22468 stamp=756326951
[*]archive log filename=+DATA/zjport/archivelog/2_6830_697238176.dbf recid=22472 stamp=756326969
[*]archive log filename=+DATA/zjport/archivelog/2_6831_697238176.dbf recid=22477 stamp=756326991
[*]archive log filename=+DATA/zjport/archivelog/2_6832_697238176.dbf recid=22481 stamp=756327012
[*]channel d1: starting compressed archive log backupset
[*]channel d1: specifying archive log(s) in backup set
[*]input archive log thread=2 sequence=6833 recid=22484 stamp=756327033
[*]input archive log thread=2 sequence=6834 recid=22488 stamp=756327058
[*]input archive log thread=2 sequence=6835 recid=22492 stamp=756327082
[*]input archive log thread=2 sequence=6836 recid=22497 stamp=756327112
[*]input archive log thread=2 sequence=6837 recid=22501 stamp=756327145
[*]input archive log thread=2 sequence=6838 recid=22505 stamp=756327170
[*]input archive log thread=2 sequence=6839 recid=22510 stamp=756327198
[*]input archive log thread=2 sequence=6840 recid=22513 stamp=756327222
[*]input archive log thread=2 sequence=6841 recid=22518 stamp=756327253
[*]input archive log thread=2 sequence=6842 recid=22523 stamp=756327287
[*]input archive log thread=2 sequence=6843 recid=22526 stamp=756327311
[*]input archive log thread=2 sequence=6844 recid=22530 stamp=756327336
[*]input archive log thread=2 sequence=6845 recid=22534 stamp=756327363
[*]input archive log thread=2 sequence=6846 recid=22539 stamp=756327505
[*]input archive log thread=2 sequence=6847 recid=22542 stamp=756327535
[*]input archive log thread=2 sequence=6848 recid=22546 stamp=756327566
[*]input archive log thread=2 sequence=6849 recid=22550 stamp=756341841
[*]input archive log thread=2 sequence=6850 recid=22554 stamp=756350429
[*]input archive log thread=2 sequence=6851 recid=22556 stamp=756350436
[*]channel d1: starting piece 1 at 13-JUL-11
[*]channel d1: finished piece 1 at 13-JUL-11
[*]piece handle=/u01/rmanbak/arch_ZJPORT_1imha073_1_1 tag=BKARCH comment=NONE
[*]channel d1: backup set complete, elapsed time: 00:00:16
[*]channel d1: deleting archive log(s)
[*]archive log filename=+DATA/zjport/archivelog/2_6833_697238176.dbf recid=22484 stamp=756327033
[*]archive log filename=+DATA/zjport/archivelog/2_6834_697238176.dbf recid=22488 stamp=756327058
[*]archive log filename=+DATA/zjport/archivelog/2_6835_697238176.dbf recid=22492 stamp=756327082
[*]archive log filename=+DATA/zjport/archivelog/2_6836_697238176.dbf recid=22497 stamp=756327112
[*]archive log filename=+DATA/zjport/archivelog/2_6837_697238176.dbf recid=22501 stamp=756327145
[*]archive log filename=+DATA/zjport/archivelog/2_6838_697238176.dbf recid=22505 stamp=756327170
[*]archive log filename=+DATA/zjport/archivelog/2_6839_697238176.dbf recid=22510 stamp=756327198
[*]archive log filename=+DATA/zjport/archivelog/2_6840_697238176.dbf recid=22513 stamp=756327222
[*]archive log filename=+DATA/zjport/archivelog/2_6841_697238176.dbf recid=22518 stamp=756327253
[*]archive log filename=+DATA/zjport/archivelog/2_6842_697238176.dbf recid=22523 stamp=756327287
[*]archive log filename=+DATA/zjport/archivelog/2_6843_697238176.dbf recid=22526 stamp=756327311
[*]archive log filename=+DATA/zjport/archivelog/2_6844_697238176.dbf recid=22530 stamp=756327336
[*]archive log filename=+DATA/port/archivelog/2_6845_697238176.dbf recid=22534 stamp=756327363
[*]archive log filename=+DATA/port/archivelog/2_6846_697238176.dbf recid=22539 stamp=756327505
[*]archive log filename=+DATA/port/archivelog/2_6847_697238176.dbf recid=22542 stamp=756327535
[*]archive log filename=+DATA/port/archivelog/2_6848_697238176.dbf recid=22546 stamp=756327566
[*]archive log filename=+DATA/port/archivelog/2_6849_697238176.dbf recid=22550 stamp=756341841
[*]archive log filename=+DATA/port/archivelog/2_6850_697238176.dbf recid=22554 stamp=756350429
[*]archive log filename=+DATA/port/archivelog/2_6851_697238176.dbf recid=22556 stamp=756350436==========节点2的归档备份到8210
[*]Finished backup at 13-JUL-11
[*]
[*]released channel: d1
[*]
[*]executing global script: global_bkctl
[*]
[*]allocated channel: d1
[*]channel d1: sid=906 instance=port1 devtype=DISK
[*]
[*]Starting backup at 13-JUL-11
[*]channel d1: starting compressed full datafile backupset
[*]channel d1: specifying datafile(s) in backupset
[*]including current control file in backupset
[*]channel d1: starting piece 1 at 13-JUL-11
[*]channel d1: finished piece 1 at 13-JUL-11
[*]piece handle=/u01/rmanbak/backupctl.ctl tag=BKCTL comment=NONE
[*]channel d1: backup set complete, elapsed time: 00:00:02
[*]Finished backup at 13-JUL-11
[*]
[*]released channel: d1
[*]
[*]Recovery Manager complete.
事实上只要处理时间稍微充裕那么一点，我们不难发现更简便的方法，通过ASMCMD直接创建丢失的archivelog目录就可以了。
我们最后的处理：
通过ASMCDM在+DATA下新建了archivelog目录，然后在此目录下新建子目录，为什么不建立文件（因为笔者没有找到10G ASM下建文件的命令，如果有人知道，烦请指导，谢谢）
页:
[1]