software raid1 and lvm on debian etch

Background

I have a fileserver box, which currently has 2x200GB disks in lvm to give me a 400GB virtual disk. This arrangement gets good use of space, but if one disk has a failure, then the whole filesystem is trashed and cannot be recovered.

The solution is to start using raid. Before I go on, raid is not a backup solution. It cannot protect you from accidentally deleting all your files, and will not protect you from a virus or malicious user or hacker. Raid just reduces the damage if a disk happens to fail (which knowing my luck, is sometime soon).

The final solution I want is 2x500GB disks in raid1 (mirrored) with lvm on top to split into my partitions. This way I could add another pair of disks in raid, add them to the lvm and not have to worry about which partitions get new space, as lvm will allow me to expand any parition onto the new space, and have a partition across multiple disks.

Why not raid5? Raid 5 is great for getting space, as you have n+1 disks, and get the space of n disks out of it as one is the redundant disk. The problem with raid5 is it is limited to the smallest disk in the raid. So 2 500GB disks and one 200GB disk will only give 400GB as each disk can only be used up to 200GB. Raid5 is great if all your disks are the same size, but if I want to add disks, and not have to replace all 3+ disks, then with raid1 I just have to buy disks in pairs. My pc has 4 ide slots and 2 sata slots, so raid1 should be fine (disks are getting quite bit these days).

So the plan is to add 2 500GB disks. put them in raid1 with a partition for /boot (which cant be in lvm) and the rest becomes part of a lvm group, with my / and /home partitions in there (and /tmp, swap)

How I did it

Warning: This can seriously mess up your data. Please, please backup first – I didn’t and was sweating hard at one point when I thought I had lost my entire lvm array. I found the Debian From Scratch (DFS) a fabulous rescue CD

Note: This took me several attempts to actually finish writing, so there are probably some errors, so let me know if you spot anything that looks wrong or is ambiguous.

For background, my current setup has 2 disks. hda and sda. these are both 200GB, and /boot and swap are on hda, and the rest of hda and all of sdb are in my lvm group

I added my new 500GB disk as /dev/sdb and booted up.

Use fdisk to create a small (~100mb) parition for /boot. Set its type to fd (linux raid). This parition cannot be in lvm, as grub does not understand lvm.
Create another parition using the rest of the space for our lvm. Also set its type to fd

You may need to use partprobe or reboot to see the new partitions, however my Debian Etch box autodetected them when leaving fdisk

Installing raid

$apt-get install mdadm
$cat /proc/mdstat should look something like

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: <none>

if not, you may need to load some modules “modprobe md;modprobe raid1”

Next, you will need to create some raid device entries.

$mknod /dev/md0 b 9 0
$mknod /dev/md1 b 9 1

This creates a block device, major number (like a type) 9. As etch uses udev, you need to tell udev to do this next boot too

$echo M md0 b 9 0 >> /etc/udev/links.conf
$echo M md1 b 9 1 >> /etc/udev/links.conf

Now we can create our real raid array:

$mdadm –create /dev/md0 –verbose –level=1 –raid-disks=2 /dev/sdb1 missing
$mdadm –create /dev/md1–verbose –level=1 –raid-disks=2 /dev/sdb2 missing

or the short version

$mdadm -C /dev/md0 -n 2 -l 1 /dev/sdb1 missing
$mdadm -C /dev/md1 -n 2 -l 1 /dev/sdb2 missing

This creates a raid1 array md0 which has parition sdb1 in it and is expecting another disk soon, similarly raid1 array md1 has partition sdb2 and also expects another disk. A raid array without all its disks is called a degraded array, as it is running and works, but is not in good condition.

Next we need to save the state of this array so that it will be loaded at next boot

$mdadm –detail –scan >> /etc/mdadm/mdadm.conf

Adding/modifying VM

To create a new lvm setup, the following commands are a quick guide to creating a single partition in your raid. You will probably want to create a few more paritions, as this just creates a single lvm logical volume called root

$pvcreate /dev/md1
$vgcreate vg0 /dev/md1
$vgscan
$lvcreate -L10G -n root vg0
$mkfs -t ext3 /dev/vg0/root

Then edit /etc/fstab to mount /dev/vg0/root (you may need to refer to it as /dev/mapper/vg0-root) as /. Repeat for any other filesystems you may want, and then copy any data over

e.g.

$mkdir /mnt/root
$mount /dev/vg0/root /mnt/root
$rsync -auHxv –exclude=/proc/* –exclude=/sys/* –exclude=/boot/* –exclude=/mnt / /mnt/root/
$mkdir /mntroot/proc /mntroot/boot /mntroot/sys
$chmod 555 /mntroot/proc

I already had a LVM setup, so I wanted to migrate it from the old disks to the new raid disks

$pvcreate /dev/md1
$vgextend vg0 dev/md1
$pvmove /dev/hda3 (this is gonna be slow)
$vgreduce vg0 /dev/hda3
$pvremove /dev/hda3

Don’t forget a swap partition (can be in LVM too)

$lvcreate -L2G -n swap vg0
$mkswap /dev/vg0/swap

Make sure you copy over your /boot to /dev/md0 too!

Then make sure you have edited /boot/grub/devices.map and /boot/grub/menu.lst to have the correct new devices

Finishing off the config

rebuild initrd to scan for raid devices
!important, you need this for your initrd to load the raid so your kernel can find the root fs!

$dpkg-reconfigure linux-image-2.6.18-3-686

(or whatever your current kernel package is). This is better than mkinitrd or update-initramfs as you dont need to worry about which you are using and is the “debian way”

Someone suggested adding a fallback in grub and a second boot entry, but they don’t use the automagic grub updating, so it will just get trashed when they upgrade kernels
edit /boot/grub/menu.lst if you want to do this

[…]
default 0
fallback 1
[…]

install grub on new disk(s)

you can reboot now if you want to see it working

Add second disk to raid

create same partition structure on /dev/sda, then add them to the raid as follows

$mdadm –detail –scan
$mdadm /dev/md1 -a /dev/sda2
$mdadm /dev/md0 -a /dev/sda1

$watch cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[2] sdb1[0]
104320 blocks [2/1] [U_]
resync=DELAYED

md1 : active raid1 sda2[2] sdb2[0]
488279488 blocks [2/1] [U_]
[>………………..] recovery = 0.1% (779328/488279488) finish=93.8min speed=86592K/sec

unused devices: <none>

when done you should see:

$cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[1] sdb1[0]
104320 blocks [2/2] [UU]

md1 : active raid1 sda2[1] sdb2[0]
488279488 blocks [2/2] [UU]

unused devices: <none>

delete the array lines from /etc/mdadm/mdadm.conf and run

$mdadm –detail –scan >> /etc/mdadm/mdadm.conf

I am surprised that raid cannot figure this information out itself, but every webpages I saw says you need these lines in the config, and I am too scared to try not having those lines.

Make sure you rebuild your initrd so that it knows the raid setup at boot time and can initialise your root partition.

$dpkg-reconfigure linux-image-2.6.18-3-686

Add grub to second disk if desired

$install-grub /dev/sda

discussion

some sites suggest that dmadm.conf needs “devices=/dev/sdb6,/dev/sda6” for each array. Not sure if this is necessary. If you think you need it, other sites also suggest to use /dev/.static/dev/sdb6 instead of just /dev/sdb6 to make udev work properly. I dont have any devices set for my arrays – I would assume that is what the uuid is for, and mdadm finds them by scanning all paritions

I found the following page really helpful – http://xtronics.com/reference/SATA-RAID-debian-for-2.6.html

About Anton Piatek

Professional bit herder, amateur photographer. Linux and tech geek
This entry was posted in Debian and tagged , , , , , . Bookmark the permalink.

3 Responses to software raid1 and lvm on debian etch

  1. rotulet says:

    Hello,
    I have bought a new computer with 2 drives and created an lvm2 over raid. Exactly I have done 2 raid : raid1 for / /boot and /home, and raid0 for /usr /var and other thing … just for having something to complain with ;).
    I have done this install with the latest Debian netinstall CD and all was fine until I try to install a new kernel. This new kernel never booted, I tried update-initramfs, set correct module entries in initrd config, etc. but it did not work. But my first kernel still booted … don’t know why :(. Recently I made an apt-get upgrade and now none of my kernel boot now, even with update-initramfs (I did not try with a dpkg-reconfigure). At boot time, they find the raids and the lvm but I get a “mount : No such device”, I don’t know what device is not found … so sad 🙁
    With the rescue mode of the NetInstall CD, I even tried to compile my own kernel but this one did not find the lvm. I don’t know how to tell to the kernel to use lvm …
    Do you know the Debian way to build a kernel with lvm/raid ?

    Thanks !

  2. Anton says:

    The rescue mode was on the DFS (Debian From Scrach) cd.
    Try dpkg-reconfigure , as you may need to run a different initrd rebuilding script and dpkg-reconfigure should run the correct one.

    The kernel will have raid/lvm, but the initrd will need to be built to have those modules in it.
    make sure your /etc/mdadm/mdadm.conf is correct otherwise lvm wont see the correct drives.

  3. rotulet says:

    I have done a .deb file from the kernel sources, install this .deb and run update-initramfs to create the initrd but it did not work : “kernel panic, unable to load ramdisk” !!!, strange since I have built this kernel with ramdisk support.
    Then I have done a apt-get install linux-image-2.6.2X, to have a new kernel with all the stuff I should need. But it failed too. I add a “set -x” in the init script and saw one reason of these failures : he was not able to mount /sys to /root/sys … strange … lvm and raid seem ok and I think the init script create it before … I can’t go further this morning 🙁 I had to go to work

Leave a Reply