Difference between revisions of "Mirroring Disks"

Revision as of 04:30, 10 November 2010

Solaris:

In Solaris 10, you can mirror the disks during the build process with Jumpstart, or even use ZFS mirroring. However, on older systems, you need to mirror the disks the old fashioned way with DiskSuite 4.2.1. In this example, the file names for the two disks being mirrored are c1t0d0sX and c1t1d0sX. The disks are partitioned like this:

  /         c1t0d0s0
  swap      c1t0d0s1
  /var      c1t0d0s3
  /usr      c1t0d0s4
  /opt      c1t0d0s5
  /export   c1t0d0s6
  -         c1t0d0s7 

Slice 7 is a 50mb partition used to store the metadatabaes! IMO, it is best practice to never use slice 7, except for this purpose!

1. You need to make sure the following packages are installed first:

 SUNWmdr SUNWmdu SUNWmdx

This is the least you need to install! Oh and they will need to bzip to install successfully! They can be found on "Solaris 8 Software CD 2" (not sure for Solaris 9, but they could be part of the standard disto), in the following location:

 cd $PATH_TO_CDROM/Solaris_8/EA/products/DiskSuite_4.2.1/sparc/Packages
 pkgadd -d . SUNWmdr SUNWmdu SUNWmdx

Once they are installed, reboot the box. It wouldn't hurt to do an 'init 0' and then a 'boot -r'!

2. Copy the primary disks VTOC to the secondary disk:

  prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2

3. Create the metadatabases

  metadb -f -a -c3 /dev/dsk/c1t0d0s7
  metadb -f -a -c3 /dev/dsk/c1t1d0s7

4. Create and initialise the metadevices on the primary disk

  metainit -f d10 1 1 c1t0d0s0
  metainit -f d10 1 1 c1t0d0s1
  metainit -f d10 1 1 c1t0d0s3
  metainit -f d10 1 1 c1t0d0s4
  metainit -f d10 1 1 c1t0d0s5
  metainit -f d10 1 1 c1t0d0s6
  metainit d0 -m d10
  metainit d1 -m d11
  metainit d3 -m d13
  metainit d4 -m d14
  metainit d5 -m d15
  metainit d6 -m d16

5. Assign the root metadevice

  metaroot d0

6. Edit /etc/vfstab and make sure that all the devices are changed to metadevices. So each line that looked like this

  /dev/dsk/c1t0d0s3 /dev/rdsk/c1t0d0s3  /var  ufs   1 no    -

now looks like

  /dev/md/dsk/d3 /dev/md/rdsk/d3  /var  ufs   1 no    -

7. Run the lockfs command and reboot

  lockfs -fa
  reboot

8. Create the the metadevices on the secondary disk

  metainit d20 1 1 c1t1d0s0
  metainit d21 1 1 c1t1d0s1
  metainit d23 1 1 c1t1d0s3
  metainit d24 1 1 c1t1d0s4
  metainit d25 1 1 c1t1d0s5
  metainit d26 1 1 c1t1d0s6 

9. Attach the metadevices from the secondary disk to the metadevices we booted from to start the mirroring process

  metattach d0 d20
  metattach d1 d21
  metattach d3 d23
  metattach d4 d24
  metattach d5 d25
  metattach d6 d26

You can monitor the progress of mirroring with this command:

  metastat | grep -i progress

When it returns nothing, it's all finished.

10. Install the boot block on the secondary hard disk so that you can boot off it

  installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t1d0s0

11. Disable the following in /etc/inetd.conf

  100229/1 (rpc.metad)
  100230/1 (rpc.metamhd)

12. Add this line to /etc/system so that the machine will boot if there are only 50% of the metadatabases available, i.e. a disk has failed

  set md:mirrored_root_flag=1

You're now finished :)

CentOS/RedHat

OK, so software RAID on CentOS/RedHat sucks! Unless you build the box from the outset with mirrored disk, you're going to need to do a lot of work to get to a point where you're going to be happy. Wherever possible, mirror the disks during build. However, if the box is already built and you need to mirror the boot disk after the fact, here's how you do it:

1. Make sure that 'mdadm' is installed:

 yum install mdadm

2. Copy the partition table from one disk to the other

 sfdisk -d /dev/sda | sfdisk /dev/sdb

3. Change the partition type of the second disk to 'raid':

 fdisk /dev/sdb
 
 The number of cylinders for this disk is set to 8924.
 There is nothing wrong with that, but this is larger than 1024,
 and could in certain setups cause problems with:
 1) software that runs at boot time (e.g., old versions of LILO)
 2) booting and partitioning software from other OSs
    (e.g., DOS FDISK, OS/2 FDISK)
 
 Command (m for help): t
 Partition number (1-7): 1
 Hex code (type L to list codes): fd
 
 Command (m for help): t
 Partition number (1-7): 2
 Hex code (type L to list codes): fd

Etc. Repeat for all partitions and then exit:

 Command (m for help): w

4. Create the configuration file /etc/mdadm.conf

 vi /etc/mdadm.conf

with the following contents:

 DEVICE /dev/sda* /dev/sdb*

5. Build the RAID devices using the 'mdadm' command:

 mdadm -C /dev/md1 -n 2 -l 1 missing /dev/sdb1
 mdadm -C /dev/md2 -n 2 -l 1 missing /dev/sdb2
 .
 .
 .

Repeat for all partitions, then check your work:

 cat /proc/mdstat

6. Update /etc/mdadm.conf

 mdadm --detail --scan >> /etc/mdadm.conf 

7. Create file a file system on each partition

 mkfs.ext3 -j /dev/md1
 mkfs.ext3 -j /dev/md2
 .
 .
 mkswap /dev/md5

8. Make a mount point and mount all the partitions:

 mkdir /mnt/md
 mount /dev/md6 /mnt/md
 mkdir /mnt/md/boot
 mkdir /mnt/md/usr
 mkdir /mnt/md/var
 mkdir /mnt/md/home
 cd /mnt
 chmod -R 755 md/
 mount /dev/md1 /mnt/md/boot
 mount /dev/md3 /mnt/md/usr
 mount /dev/md7 /mnt/md/var
 mount /dev/md2 /mnt/md/home

9. Copy the files from disk 1 to the newly built raid

 rsync -aXv ––exclude=/proc/* ––exclude=/sys/* ––exclude=/dev/* ––exclude=/mnt/md / /mnt/md

10. Bind /proc, /sys & /dev to the new filesystem

 mount -o bind /proc /mnt/md/proc
 mount -o bind /sys /mnt/md/sys
 mount -o bind /dev /mnt/md/dev
 mount -o bind /dev/pts /mnt/md/dev/pts

11. Change the root device

 chroot /mnt/md

12. Edit /etc/fstab to use the appropriate metadevices.

 LABEL=/boot             /boot                   ext3    defaults        1 2

becomes

 /dev/md1                /boot                   ext3    defaults        1 2

etc....

13. Edit /boot/grub/device.map and add the following entry for the second disk.

 (hd1)     /dev/sdb

14. Install GRUB on the second disk

 grub --device-map=/boot/grub/device.map
 
 
     GNU GRUB  version 0.97  (640K lower / 3072K upper memory)

  [ Minimal BASH-like line editing is supported.  For the first word, TAB
    lists possible command completions.  Anywhere else TAB lists the possible
    completions of a device/filename.]
 grub> root (hd1,0)
 root (hd1,0)
  Filesystem type is ext2fs, partition type 0xfd
 grub> setup (hd1)
 setup (hd1)
  Checking if "/boot/grub/stage1" exists... no
  Checking if "/grub/stage1" exists... yes
  Checking if "/grub/stage2" exists... yes
  Checking if "/grub/e2fs_stage1_5" exists... yes
  Running "embed /grub/e2fs_stage1_5 (hd1)"...  15 sectors are embedded.
 succeeded
  Running "install /grub/stage1 (hd1) (hd1)1+15 p (hd1,0)/grub/stage2 /grub/grub.conf"... succeeded
 Done.
 grub> quit

15. Rebuild initramfs and force it to detect RAID, otherwise you'll get a kernel panic

 cd /boot
 mkinitrd -f --force-raid-probe initrd-`uname -r`.img `uname -r`

16. Edit grub.cfg to boot of second disk and use the metadevice. Change grub.cfg from:

 splashimage=(hd0,0)/grub/splash.xpm.gz
 hiddenmenu
 title CentOS (2.6.18-194.el5)
         root (hd0,0)
         kernel /vmlinuz-2.6.18-194.el5 ro root=LABEL=/ console=tty0 console=ttyS0,9600
         initrd /initrd-2.6.18-194.el5.img

to:

 splashimage=(hd1,0)/grub/splash.xpm.gz
 hiddenmenu
 title CentOS (2.6.18-194.el5)
         root (hd1,0)
         kernel /vmlinuz-2.6.18-194.el5 ro root=/dev/md6 console=tty0 console=ttyS0,9600
         initrd /initrd-2.6.18-194.el5.img