The most terrible thing a Linux could have may be some errors reported by the file system. It is usually caused by power outages or bad blocks/sectors within the hard drive. Eventually, as a Linux user, you may at least have suffered from file system errors once, where you have to reinstall the OS and for sure to lose all the existing data. This post is the summary of the recent file system errors happened on my Fedora 20. May give some hints.

0. File system errors seem to be inevitable

There was sth wrong with my Fedora 20 after a recent kernel update to 3.14.4. After megasas kernel module (a kernel module for LSI RAID cards) init’d, the whole system fell into emergency mode (single user mode). However, a manually CTRL-D would force the system going into the normal mode. I hunted for the changes of megasas which has been stable for years. Then I was looking forward to the next kernel release which may fix this weird issue. Interestingly, things were getting more weird.

After a ‘yum clean all‘ (trying to fix the broken dependency but not working anyway), I was not able to run ‘yum update’ anymore, always with the same error from Python: OSError [Errono 5]: Input/Output error. Tried to reboot the system and rebuild the yum cache manually but still got the same error. While I was wondering what happened within yum, there was sth going wrong within my file system.

1. Use tools to figure out what happens

1.1 dmesg

Please do use this powerful tool to check kernel’s complain and I found sth like below:
May 23 08:28:29 daveti kernel:EXT4-fs (dm-0): warning: mounting fs with errors, running e2fsck is recommended
May 23 08:28:29 daveti kernel:EXT4-fs (dm-0): re-mounted. Opts: (null)
May 23 08:28:29 daveti systemd: Started Remount Root and Kernel File Systems.
May 23 08:28:29 daveti systemd: Starting Configure read-only root support…
May 23 08:28:29 daveti systemd: Starting Import network configuration from initramfs…
May 23 08:28:29 daveti systemd: Starting Load/Save Random Seed…
May 23 08:28:29 daveti systemd: Starting Local File Systems (Pre).
May 23 08:28:29 daveti systemd: Reached target Local File Systems (Pre).
May 23 08:28:29 daveti systemd: Started Load/Save Random Seed.
May 23 08:28:29 daveti systemd: Started Configure read-only root support.
May 23 08:28:29 daveti systemd: Started Import network configuration from initramfs.
May 23 08:28:29 daveti systemd: Starting Emergency Shell…
May 23 08:28:29 daveti systemd: Started Emergency Shell.
May 23 08:28:29 daveti systemd: Starting Emergency Mode.
May 23 08:28:29 daveti systemd: Reached target Emergency Mode.
May 23 08:28:29 daveti systemd-fsck: /dev/sda1: clean, 410/128016 files, 244603/512000 blocks
May 23 08:28:29 daveti systemd: Started File System Check on /dev/mapper/fedora_dyn–184–171–61–7-home.
May 23 08:28:29 daveti systemd: Started File System Check on /dev/disk/by-uuid/c96cf320-40a7-42d5-9406-bf80c866faa6.
May 23 08:28:29 daveti systemd: Mounting /boot…
May 23 08:28:29 daveti systemd: Mounting /home…
May 23 08:33:08 daveti kernel:[ 301.924281] EXT4-fs (dm-0): error count: 45
May 23 08:33:08 daveti kernel:[ 301.924299] EXT4-fs (dm-0): initial error at 1400170854: __ext4_get_inode_loc:3919: inode 2238830: block 8913590
May 23 08:33:08 daveti kernel:[ 301.924307] EXT4-fs (dm-0): last error at 1400799080: __ext4_get_inode_loc:3919: inode 2238830: block 8913590
May 23 08:33:08 daveti kernel:EXT4-fs (dm-0): error count: 45
May 23 08:33:08 daveti kernel:EXT4-fs (dm-0): initial error at 1400170854: __ext4_get_inode_loc:3919: inode 2238830: block 8913590
May 23 08:33:08 daveti kernel:EXT4-fs (dm-0): last error at 1400799080: __ext4_get_inode_loc:3919: inode 2238830: block 8913590

What does this tell us? First, it tells my file system (EXT4) screws. Second, it tells the reason why the system falling into emergency mode (because of file system errors). Last, it tells some screwed inode operations.

1.2 smartctl

If your hard drive support SMART, try ‘smartctl -a /dev/yourDevice‘. Mine’s not. Let me know if this tool helps.

1.3 badblocks

This tool is used to see if there is any bad blocks within the drive – ‘badblocks /dev/yourDevice‘. NOTE, first it does NOT fix these bad blocks and second nothing from badblocks does not mean the file system in a good state. Again, dmesg!

2. Rescue mode?

We probably all know the right solution for this case – umount the bad disk and run fsck! The point is how to do that if you are not able to umount the disk even in single user mode when the root file system is screwed. If you have a second machine, you can unplug the issued hard drive and plug it into the other machine and run fsck there. If not, what else shall we do?

It is said that most Linux distribution installation ROMs have the so-called rescue mode where you can try to fix issues on the current system. I tried Fedora 20 live DVD and booted into ‘Try’ mode where a live image is loaded into the memory as the active OS. Unfortunately, I am still not able to umount the root file system for certain reasons.

3. Workaround

Fortunately, I have got a Ubuntu 14.04 LTS live DVD ROM. Booting from this system, I am able to umount all the partitions and run ‘fsck -yvf /dev/yourPartition‘ to fix all the bad blocks. After a reboot, my Fedora 20 runs smoothly into the Gnome login.

4. At last

File system errors may be inevitable for Linux (and eventually for all OSes). Once it happens, try to fix it ASAP. While I was trying to figure out how to umount the root file system, the number of errors (bad blocks) reported by EXT4 was increasing indeed! And please UMOUNT your disk/partition before fsck, otherwise you will hate Linux…