View source for Mirroring Disks

== Solaris: ==

In Solaris 10, you can mirror the disks during the build process with Jumpstart, or even use ZFS mirroring.  However, on older systems, you need to mirror the disks the old fashioned way with DiskSuite 4.2.1.  In this example, the file names for the two disks being mirrored are c1t0d0sX and c1t1d0sX.  The disks are partitioned like this:

   /         c1t0d0s0
   swap      c1t0d0s1
   /var      c1t0d0s3
   /usr      c1t0d0s4
   /opt      c1t0d0s5
   /export   c1t0d0s6
   -         c1t0d0s7 

Slice 7 is a 50mb partition used to store the metadatabaes!  IMO, it is best practice to never use slice 7, except for this purpose!

1. You need to make sure the following packages are installed first:

  SUNWmdr SUNWmdu SUNWmdx

This is the least you need to install!  Oh and they will need to bzip to install successfully!  They can be found on "Solaris 8 Software CD 2" (not sure for Solaris 9, but they could be part of the standard disto), in the following location:

  cd $PATH_TO_CDROM/Solaris_8/EA/products/DiskSuite_4.2.1/sparc/Packages
  pkgadd -d . SUNWmdr SUNWmdu SUNWmdx

Once they are installed, reboot the box.  It wouldn't hurt to do an 'init 0' and then a 'boot -r'!

2. Copy the primary disks VTOC to the secondary disk:

   prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2

3. Create the metadatabases

   metadb -f -a -c3 /dev/dsk/c1t0d0s7
   metadb -f -a -c3 /dev/dsk/c1t1d0s7

4. Create and initialise the metadevices on the primary disk

   metainit -f d10 1 1 c1t0d0s0
   metainit -f d10 1 1 c1t0d0s1
   metainit -f d10 1 1 c1t0d0s3
   metainit -f d10 1 1 c1t0d0s4
   metainit -f d10 1 1 c1t0d0s5
   metainit -f d10 1 1 c1t0d0s6
   metainit d0 -m d10
   metainit d1 -m d11
   metainit d3 -m d13
   metainit d4 -m d14
   metainit d5 -m d15
   metainit d6 -m d16

5. Assign the root metadevice

   metaroot d0

6. Edit /etc/vfstab and make sure that all the devices are changed to metadevices.  So each line that looked like this

   /dev/dsk/c1t0d0s3 /dev/rdsk/c1t0d0s3  /var  ufs   1 no    -

now looks like

   /dev/md/dsk/d3 /dev/md/rdsk/d3  /var  ufs   1 no    -

7. Run the lockfs command and reboot

   lockfs -fa
   reboot

8. Create the the metadevices on the secondary disk

   metainit d20 1 1 c1t1d0s0
   metainit d21 1 1 c1t1d0s1
   metainit d23 1 1 c1t1d0s3
   metainit d24 1 1 c1t1d0s4
   metainit d25 1 1 c1t1d0s5
   metainit d26 1 1 c1t1d0s6 

9. Attach the metadevices from the secondary disk to the metadevices we booted from to start the mirroring process

   metattach d0 d20
   metattach d1 d21
   metattach d3 d23
   metattach d4 d24
   metattach d5 d25
   metattach d6 d26

You can monitor the progress of mirroring with this command:

   metastat | grep -i progress

When it returns nothing, it's all finished.

10. Install the boot block on the secondary hard disk so that you can boot off it

   installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t1d0s0

11. Disable the following in /etc/inetd.conf

   100229/1 (rpc.metad)
   100230/1 (rpc.metamhd)

12. Add this line to /etc/system so that the machine will boot if there are only 50% of the metadatabases available, i.e. a disk has failed

   set md:mirrored_root_flag=1


You're now finished :)

== CentOS/RedHat ==

'''DON'T USE THESE INSTRUCTIONS!  I have yet to make them work properly!'''

OK, so software RAID on CentOS/RedHat sucks!  Unless you build the box from the outset with mirrored disk, you're going to need to do a lot of work to get to a point where you're going to be happy.  Wherever possible, mirror the disks during build!  However, if the box is already built and you need to mirror the boot disk after the fact, here's how you do it:

This is a worked example that uses the following disk layout:

  df -h 
  
  Filesystem            Size  Used Avail Use% Mounted on
  /dev/sda6             3.9G  283M  3.4G   8% /
  /dev/sda7              40G  251M   37G   1% /var
  /dev/sda3             7.8G  764M  6.7G  11% /usr
  /dev/sda2             7.8G  147M  7.3G   2% /home
  /dev/sda1              99M   12M   82M  13% /boot
  tmpfs                1006M     0 1006M   0% /dev/shm

  fdisk -l
  
  Disk /dev/sda: 73.4 GB, 73407865856 bytes
  255 heads, 63 sectors/track, 8924 cylinders
  Units = cylinders of 16065 * 512 = 8225280 bytes
  
     Device Boot      Start         End      Blocks   Id  System
  /dev/sda1   *           1          13      104391   83  Linux
  /dev/sda2              14        1057     8385930   83  Linux
  /dev/sda3            1058        2101     8385930   83  Linux
  /dev/sda4            2102        8924    54805747+   5  Extended
  /dev/sda5            2102        3145     8385898+  82  Linux swap / Solaris
  /dev/sda6            3146        3667     4192933+  83  Linux
  /dev/sda7            3668        8924    42226821   83  Linux

  parted -l
  
  Model: FUJITSU MAV2073RCSUN72G (scsi)
  Disk /dev/sda: 73.4GB
  Sector size (logical/physical): 512B/512B
  Partition Table: msdos
  
  Number  Start   End     Size    Type      File system  Flags
   1      32.3kB  107MB   107MB   primary   ext3         boot 
   2      107MB   8694MB  8587MB  primary   ext3              
   3      8694MB  17.3GB  8587MB  primary   ext3              
   4      17.3GB  73.4GB  56.1GB  extended                    
   5      17.3GB  25.9GB  8587MB  logical   linux-swap        
   6      25.9GB  30.2GB  4294MB  logical   ext3              
   7      30.2GB  73.4GB  43.2GB  logical   ext3 

1. Make sure that 'mdadm' is installed:

  yum install mdadm

2. Load the RAID kernel modules

  modprobe linear
  modprobe multipath
  modprobe raid0
  modprobe raid1
  modprobe raid5
  modprobe raid6
  modprobe raid10

3. If the new disk 2 has previously been used in a server with RAID, you will need to stop and remove any meta devices that it had

  mdadm --stop /dev/md0
  mdadm --stop /dev/md1
  mdadm --stop /dev/md2
  mdadm --stop /dev/md3
  mdadm --stop /dev/md4
  mdadm --stop /dev/md5
  mdadm --remove /dev/md0
  mdadm --remove /dev/md1
  mdadm --remove /dev/md2
  mdadm --remove /dev/md3
  mdadm --remove /dev/md4
  mdadm --remove /dev/md5

4. Copy the partition table from one disk to the other

  sfdisk -d /dev/sda | sfdisk /dev/sdb

5. Change the partition type of the second disk to 'raid':

  fdisk /dev/sdb
  
  The number of cylinders for this disk is set to 8924.
  There is nothing wrong with that, but this is larger than 1024,
  and could in certain setups cause problems with:
  1) software that runs at boot time (e.g., old versions of LILO)
  2) booting and partitioning software from other OSs
     (e.g., DOS FDISK, OS/2 FDISK)
  
  Command (m for help): '''t'''
  Partition number (1-7): '''1'''
  Hex code (type L to list codes): '''fd'''
  
  Command (m for help): '''t'''
  Partition number (1-7): '''2'''
  Hex code (type L to list codes): '''fd'''

Etc.  Repeat for all partitions and then exit:

  Command (m for help): '''w'''

6. To make sure that there are no remains from previous RAID installations on /dev/sdb, we run the following commands:

  mdadm --zero-superblock /dev/sdb1
  mdadm --zero-superblock /dev/sdb2
  mdadm --zero-superblock /dev/sdb3
  mdadm --zero-superblock /dev/sdb5
  mdadm --zero-superblock /dev/sdb6
  mdadm --zero-superblock /dev/sdb7

7. Build the RAID devices using the 'mdadm' command:

  mdadm -C /dev/md0 -n 2 -l 1 missing /dev/sdb1
  mdadm -C /dev/md1 -n 2 -l 1 missing /dev/sdb2
  mdadm -C /dev/md2 -n 2 -l 1 missing /dev/sdb3
  mdadm -C /dev/md3 -n 2 -l 1 missing /dev/sdb5
  mdadm -C /dev/md4 -n 2 -l 1 missing /dev/sdb6
  mdadm -C /dev/md5 -n 2 -l 1 missing /dev/sdb7

Check your work:

  cat /proc/mdstat

8. Create file a file system on each partition

  mkfs.ext3 -j /dev/md0
  mkfs.ext3 -j /dev/md1
  mkfs.ext3 -j /dev/md2
  mkfs.ext3 -j /dev/md4
  mkfs.ext3 -j /dev/md5
  mkswap /dev/md3

9. Update /etc/mdadm.conf

  mdadm --detail --scan > /etc/mdadm.conf 

10. Make a mount point and mount all the partitions:

  mkdir /mnt/md
  mount /dev/md4 /mnt/md
  mkdir /mnt/md/boot
  mkdir /mnt/md/usr
  mkdir /mnt/md/var
  mkdir /mnt/md/home
  cd /mnt
  chmod -R 755 md/
  mount /dev/md0 /mnt/md/boot
  mount /dev/md2 /mnt/md/usr
  mount /dev/md5 /mnt/md/var
  mount /dev/md1 /mnt/md/home

11. Copy the files from disk 1 to the newly built raid

  rsync -aXv ––exclude=/proc/* ––exclude=/sys/* ––exclude=/dev/* ––exclude=/mnt/md / /mnt/md

NOTE: Remember to escape the * if you're using a shell like tcsh.

12. Edit /etc/fstab to use the appropriate metadevices.

  LABEL=/                 /                       ext3    defaults        1 1
  LABEL=/var              /var                    ext3    defaults        1 2
  LABEL=/usr              /usr                    ext3    defaults        1 2
  LABEL=/home             /home                   ext3    defaults        1 2
  LABEL=/boot             /boot                   ext3    defaults        1 2
  tmpfs                   /dev/shm                tmpfs   defaults        0 0
  devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
  sysfs                   /sys                    sysfs   defaults        0 0
  proc                    /proc                   proc    defaults        0 0
  LABEL=SWAP-sda5         swap                    swap    defaults        0 0
  
becomes

  /dev/md4                /                       ext3    defaults        1 1
  /dev/md5                /var                    ext3    defaults        1 2
  /dev/md2                /usr                    ext3    defaults        1 2
  /dev/md1                /home                   ext3    defaults        1 2
  /dev/md0                /boot                   ext3    defaults        1 2
  tmpfs                   /dev/shm                tmpfs   defaults        0 0
  devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
  sysfs                   /sys                    sysfs   defaults        0 0
  proc                    /proc                   proc    defaults        0 0
  /dev/md3                swap                    swap    defaults        0 0

**13. Edit /boot/grub/device.map and add the following entry for the second disk.

  (hd1)     /dev/sdb

13. Edit /boot/grub/menu.lst to allow fallback to the next kernel in the event of a boot failure:

   vim /boot/grub/menu.lst

and add 

  default=0
  '''fallback=1'''

14. Add an entry for in menu.lst to allow booting off metadevice root

  '''title CentOS (2.6.18-194.26.1.el5 metadevices)
          root (hd1,0)
          kernel /vmlinuz-2.6.18-194.26.1.el5 ro root=/dev/md4
          initrd /initrd-2.6.18-194.26.1.el5.img'''
  title CentOS (2.6.18-194.26.1.el5)
          root (hd0,0)
          kernel /vmlinuz-2.6.18-194.26.1.el5 ro root=LABEL=/
          initrd /initrd-2.6.18-194.26.1.el5.img

15. Rebuild ramdisk to reflect our new reality:

  mv /boot/initrd-`uname -r`.img /boot/initrd-`uname -r`.img_orig
  mkinitrd /boot/initrd-`uname -r`.img `uname -r`

16. Setup GRUB on both disks

  grub

  Probing devices to guess BIOS drives. This may take a long time.


      GNU GRUB  version 0.97  (640K lower / 3072K upper memory)

   [ Minimal BASH-like line editing is supported.  For the first word, TAB
     lists possible command completions.  Anywhere else TAB lists the possible
     completions of a device/filename.]
  grub> '''root (hd0,0)'''
  root (hd0,0)
   Filesystem type is ext2fs, partition type 0x83
  grub> '''setup (hd0)'''
  setup (hd0)
   Checking if "/boot/grub/stage1" exists... no
   Checking if "/grub/stage1" exists... yes
   Checking if "/grub/stage2" exists... yes
   Checking if "/grub/e2fs_stage1_5" exists... yes
   Running "embed /grub/e2fs_stage1_5 (hd0)"...  15 sectors are embedded.
  succeeded
   Running "install /grub/stage1 (hd0) (hd0)1+15 p (hd0,0)/grub/stage2 /grub/grub.conf"... succeeded
  Done.
  grub> '''root (hd1,0)'''
  root (hd1,0)
   Filesystem type is ext2fs, partition type 0xfd
  grub> '''setup (hd1)'''
  setup (hd1)
   Checking if "/boot/grub/stage1" exists... no
   Checking if "/grub/stage1" exists... yes
   Checking if "/grub/stage2" exists... yes
   Checking if "/grub/e2fs_stage1_5" exists... yes
   Running "embed /grub/e2fs_stage1_5 (hd1)"...  15 sectors are embedded.
  succeeded
   Running "install /grub/stage1 (hd1) (hd1)1+15 p (hd1,0)/grub/stage2 /grub/grub.conf"... succeeded
  Done.
  grub> '''quit'''

17. Reboot and hope it works!

18. After the reboot, check that the metadevices have been used to boot from

  df -h

should show:

  

19. Run fdisk on the original disk to change all the partitions to RAID volumes

  fdisk /dev/sda

20. Add the disk 1 partitions to the meta devices

  mdadm --add /dev/md0 /dev/sda1
  mdadm --add /dev/md1 /dev/sda2
  mdadm --add /dev/md2 /dev/sda3
  mdadm --add /dev/md3 /dev/sda5
  mdadm --add /dev/md4 /dev/sda6
  mdadm --add /dev/md5 /dev/sda7

21. Monitor the re-sync with

  cat /proc/mdstat

until all meta devices have, for want of a better word, synced.

22. Update /etc/mdadm.conf to reflect the new situation

  mdadm --examine --scan > /etc/mdadm.conf

which should result in mdadm.conf looking something like:

  

23. Update grub so that in the event that disk 2 should fail, it will boot off disk 1

  vim /boot/grub/menu.lst

so that it looks like this:

  
24. Now finally update the ramdisk one more time

  mv /boot/initrd-`uname -r`.img /boot/initrd-`uname -r`.img_orig2
  mkinitrd /boot/initrd-`uname -r`.img `uname -r`

25. Reboot and you should be good to go!