[MLUG] More praise for Linux software RAID

[Nick Nobody]

On Tue, 2011-05-31 at 17:27 -0400, Hendrik Boom wrote:
> On Tue, May 31, 2011 at 04:01:28PM -0400, Nick Nobody wrote:
> > I've said it before and I'll say it again, Linux software RAID is
> > awesome!
> > 
> > A few weeks ago I had a disk die in my NAS (1 of 3 1.5TB Seagate
> > drives). No problem, mdadm emailed me saying the RAID5 array was
> > degraded so I picked up a new disk (Western Digital this time) and
> > within a few hours the array was re-synced and running at full speed.
> > 
> > I RMA'd the dead disk and got the replacement back last week. Over the
> > weekend I added it to the array and used mdadm's "grow" feature to get
> > access to the extra 1.5TB of space. A little over 2 days later it was
> > done growing the array without any issues.
> 
> I do wonder how you managed to allocate parts of the various copies of 
> the RAID drive onto the new physical drive.  Or was it a third redundant 
> drive?

At this point I had a working RAID5 array with 3 disks. The procedure
for adding a disk is the following:

1) Run fdisk to create a partition on the new drive, in my case I just
created one partition (type 0x83, aka Linux native) which used all the
available space on the drive.

2) Run "mdadm --add /dev/[md device] /dev/sd[disk number]1". This adds
the disk to the array without telling the OS what to do with it.

3) Run "mdadm --grow /dev/[md device] --raid-devices=[number of disks in
the array]". This tells the OS to spread the data (and parity data)
across all of the disks.

When the growing process completes you'll see something like this in
your kernel logs (you can watch the progress in /proc/mdstat):
    md1: detected capacity change from 3000598396928 to 4500897595392

4) At this point you can tell the filesystem to take advantage of the
newly created space. This process varies depending on the filesystem
you're using and whether you're using LVM. I'm just using a plain-old
EXT3 filesystem, expanding it is as easy as: resize2fs /dev/md1

To be honest, I unmounted the filesystem before doing this step but only
because I wanted to fsck the filesystem beforehand (turns out it wasn't
even necessary). If I hadn't done that I wouldn't have had to unmount
the fs because the kernel version I'm using supports resizing of mounted
filesystems.

> > 
> > The coolest part of this story is that all throughout this process the
> > filesystem on the array was mounted and useable (albeit a tad slow).
> 
> It's impressive what you can do during a RAID recovery.  That said, I 
> probably wouldn't try doing a major system upgrade that involves a new 
> kernel and new RAID software during such a recovery!

I don't think that would be that big of a deal, the RAID software seems
to be very resilient. I've had my system shut off during a recovery
because of a power failure and when I turned the system back on it just
kept recovering. I've also moved RAID arrays into completely different
systems with different kernel versions without any problems.

> And presumably you did have to shut down the system while you 
> were physically replacing the drive?

Yes, it would have been pretty darn cool to hot swap the drives but I
needed to clean out the case anyway so I took the easy way out.

Damn Seagate made me lose upwards of 400 days of uptime on that box :P

nick