RHCE ext3文件系统故障一例

huhahapz

　　好久没来了，博客长草了，我来除除草。
　　给我分了两人，一个统招，一个Java两年开发经验的社招，让我这从工具平台运维往Python开发方向转的工作是举步维艰啊~
　　领导看人还是真特么的不准，希望今年招聘的两位童鞋能来啊~
　　昨天下午，某客户打来电话，说文件系统只读，无法写入内容，导致系统无法正常使用——说实在的，现在转行做开发，问题接触得少（因为公司主要用Windows系统），所以也没听说过。
　　后来客户提供了账号密码，远程再远程登录上去（相当不稳定，看了不到20分钟的日志，我特么就登录了不下5次）查看dmesg和messages文件内容：

　　May 7 01:46:36 hostserver1 kernel: EXT3-fs error (device dm-10): ext3_free_blocks_sb: bit already cleared for block 154861
May  7 01:46:36 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_free_blocks_sb: Journal has aborted
May  7 01:46:36 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_free_blocks_sb: Journal has aborted
May  7 01:46:36 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_reserve_inode_write: Journal has aborted
May  7 01:46:36 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_truncate: Journal has aborted
May  7 01:46:36 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_reserve_inode_write: Journal has aborted
May  7 01:46:36 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_orphan_del: Journal has aborted
May  7 01:46:36 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_reserve_inode_write: Journal has aborted
May  7 01:46:36 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_delete_inode: Journal has aborted
May  7 01:46:36 hostserver1 kernel: EXT3-fs error (device dm-10): ext3_journal_start_sb: Detected aborted journal
May  7 13:37:08 hostserver1 kernel: usb 1-1: device not accepting address 2, error -71
May  7 13:37:35 hostserver1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
May  7 13:37:35 hostserver1 kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
May  7 13:49:23 hostserver1 kernel: EXT3-fs error (device dm-10): ext3_free_blocks_sb: bit already cleared for block 131687
May  7 13:49:23 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_free_blocks_sb: Journal has aborted
May  7 13:49:23 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_free_blocks_sb: Journal has aborted
May  7 13:49:23 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_reserve_inode_write: Journal has aborted
May  7 13:49:23 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_reserve_inode_write: Journal has aborted
May  7 13:49:23 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_orphan_del: Journal has aborted
May  7 13:49:23 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_truncate: Journal has aborted
May  7 13:49:23 hostserver1 kernel: EXT3-fs error (device dm-10): ext3_journal_start_sb: Detected aborted journal
May  7 13:50:26 hostserver1 kernel: batchtrans[6636]: segfault at 0000000000000001 rip 000000361005e587 rsp 00007fffc77df470 error 4
May  7 13:56:15 hostserver1 kernel: batchtrans[18111]: segfault at 0000000000000001 rip 000000361005e587 rsp 00007fff62ef30e0 error 4
May  7 14:08:02 hostserver1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
May  7 14:08:02 hostserver1 kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
May  7 14:30:40 hostserver1 kernel: EXT3-fs error (device dm-10): ext3_free_blocks_sb: bit already cleared for block 131731
May  7 14:30:40 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_reserve_inode_write: Journal has aborted
May  7 14:30:40 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_reserve_inode_write: Journal has aborted
May  7 14:30:40 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_orphan_del: Journal has aborted
May  7 14:30:40 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_truncate: Journal has aborted
May  7 14:30:40 hostserver1 kernel: EXT3-fs error (device dm-10): ext3_journal_start_sb: Detected aborted journal
May  7 16:28:20 hostserver1 kernel: usb 1-1: device not accepting address 2, error -71
May  7 16:28:40 hostserver1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
May  7 16:28:40 hostserver1 kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
May  7 20:08:24 hostserver1 kernel: EXT3-fs error (device dm-10): ext3_free_blocks_sb: bit already cleared for block 20605365
May  7 20:08:24 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_reserve_inode_write: Journal has aborted
May  7 20:08:24 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_reserve_inode_write: Journal has aborted
May  7 20:08:24 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_orphan_del: Journal has aborted
May  7 20:08:24 hostserver1 kernel: EXT3-fs error (device dm-10) in ext3_truncate: Journal has aborted
May  7 20:08:24 hostserver1 kernel: EXT3-fs error (device dm-10): ext3_journal_start_sb: Detected aborted journal
May  7 22:38:11 hostserver1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
May  7 22:38:11 hostserver1 kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended

　　外事不决问Google，技术问题当然也是——查得是因为Ext3日志型文件系统的原因，重启后没有进行磁盘检查，日志与文件数据不一致，累积多了文件系统就锁定了，只需要进行e2fsck修复即可。
　　device dm-10是什么意思呢？对应的是一个磁盘设备，用下面的命令判断吧：



lvdisplay|awk  '/LV Name/{n=$3} /Block device/{d=$3; sub(".*:","dm-",d); print d,n;}' | grep dm-10
　　实际上就是Block device对应的“:”后的内容。
　　处理办法也很简单：

• 备份分区数据，并进行修复分区操作，建议使用e2fsck修复。
  
• 如果此方法无效，建议卸载分区后重新格式化使用。
  
• 上述方法无效，估计硬件有问题了？可惜没到这一步。
  

　　很简单的问题处理了6个小时，原因是客户那边的系统管理员对系统不熟，业务系统并没有全关，导致数据备份做了很多次才OK。
　　另外，处理过程中客户不断强调要无损数据，但是文件系统已经受损，你让我怎么无损？幸好被开掉的前系统管理员来了，确认这个分区的文件都有备份，这才大胆的进行检查和恢复。
　　ps：整个过程中，感受到前系统管理员在某些方面经验都特么的比我丰富……泪奔啊