Tuesday, September 15, 2009

Recover a Non-booting Linux System

Ok, inevitably, it will come to pass that if you are testing out Alpha Releases, you will experience at least one time in your life where the system will not boot.

Recently, while working on Karmic (Ubuntu 9.10), I did an update an rebooted into a non functional system. Apparently just before freeze for Alpha 6, there were some uploads which caused the build systems to not completely build all the necessary packages. Hence I had a partial update.

Anyway, things seem to be resolved now, so on to recovery. I am documenting this mostly so I remember how to do it again (copy/paste is your friend) and maybe it will be useful to others.

First, you need a live CD/DVD/USB running your favorite Linux. In my case, it's Kubuntu. I used my usb-creator-kde to make a live USB from a known working and recent release.

After booting into your live environment, you need to open a terminal and switch to mount your partitions under /mnt.

To determine the partitions, you can use fdisk as follows:

ubuntu@ubuntu:~$ sudo fdisk -l

Disk /dev/sda: 320.1 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xd88dfd16

Device Boot Start End Blocks Id System
/dev/sda1 * 1 61 489951 83 Linux
/dev/sda2 62 38913 312078690 5 Extended
/dev/sda5 62 1306 10000431 82 Linux swap / Solaris
/dev/sda6 1307 7385 48829536 83 Linux
/dev/sda7 7386 38913 253248628+ 83 Linux

Disk /dev/sdb: 2057 MB, 2057306112 bytes
64 heads, 62 sectors/track, 1012 cylinders
Units = cylinders of 3968 * 512 = 2031616 bytes
Disk identifier: 0x0000a283

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 1012 2007777 b W95 FAT32

sdb is my USB drive and sda is my HDD. sda1 is /boot, sda6 is / and sda7 is home. Yours will likely be different.

After you figure out your partitions, you will mount them as follows:

ubuntu@ubuntu:~$ sudo mount /dev/sda6 /mnt
ubuntu@ubuntu:~$ sudo mount /dev/sda1 /mnt/boot
ubuntu@ubuntu:~$ sudo mount /dev/sda7 /mnt/home

Verify everything looks correct before proceeding (i.e. you got the partitions correct and mounted in the correct places).

Next, you need to take care of /proc and /dev as follows:

ubuntu@ubuntu:~$ sudo mount -t proc none /mnt/proc
ubuntu@ubuntu:~$ sudo mount -o bind /dev /mnt/dev

Ok, now you have setup your chroot environment, and may change into it:

ubuntu@ubuntu:~$ sudo chroot /mnt

At this point, you should be able to update your system, add/remove packages using apt-get, or edit any files you need to fix the system.

In my case, a simple apt-get update && apt-get upgrade will fix things.

After making your changes you need to exit the chroot and unmount the partitions in the reverse order.

Now you should be able to safely exit the live system and reboot.

Hope this helps.