Difference between revisions of "Mirroring Disks"

From Peter Pap's Technowiki
Jump to: navigation, search
(→‎CentOS/RedHat)
(→‎CentOS/RedHat)
Line 109: Line 109:
 
'''DON'T USE THEE INSTRUCTIONS!  I have yet to make them work properly!'''
 
'''DON'T USE THEE INSTRUCTIONS!  I have yet to make them work properly!'''
  
−
OK, so software RAID on CentOS/RedHat sucks!  Unless you build the box from the outset with mirrored disk, you're going to need to do a lot of work to get to a point where you're going to be happy.  Wherever possible, mirror the disks during build. However, if the box is already built and you need to mirror the boot disk after the fact, here's how you do it:
+
OK, so software RAID on CentOS/RedHat sucks!  Unless you build the box from the outset with mirrored disk, you're going to need to do a lot of work to get to a point where you're going to be happy.  Wherever possible, mirror the disks during build! However, if the box is already built and you need to mirror the boot disk after the fact, here's how you do it:
  
 
1. Make sure that 'mdadm' is installed:
 
1. Make sure that 'mdadm' is installed:
Line 115: Line 115:
 
   yum install mdadm
 
   yum install mdadm
  
−
2. Copy the partition table from one disk to the other
+
2. Load the RAID kernel modules
 +
 
 +
  modprobe linear
 +
  modprobe multipath
 +
  modprobe raid0
 +
  modprobe raid1
 +
  modprobe raid5
 +
  modprobe raid6
 +
  modprobe raid10
 +
 
 +
3. Copy the partition table from one disk to the other
  
 
   sfdisk -d /dev/sda | sfdisk /dev/sdb
 
   sfdisk -d /dev/sda | sfdisk /dev/sdb
  
−
3. Change the partition type of the second disk to 'raid':
+
4. Change the partition type of the second disk to 'raid':
  
 
   fdisk /dev/sdb
 
   fdisk /dev/sdb
Line 142: Line 152:
 
   Command (m for help): '''w'''
 
   Command (m for help): '''w'''
  
−
4. Create the configuration file /etc/mdadm.conf
+
5. To make sure that there are no remains from previous RAID installations on /dev/sdb, we run the following commands:
−
 
 
−
  vi /etc/mdadm.conf
 
−
 
 
−
with the following contents:
 
  
−
   DEVICE /dev/sda* /dev/sdb*
+
   mdadm --zero-superblock /dev/sdb1
 +
  mdadm --zero-superblock /dev/sdb2
 +
  mdadm --zero-superblock /dev/sdb3
 +
  mdadm --zero-superblock /dev/sdb5
 +
  mdadm --zero-superblock /dev/sdb6
 +
  mdadm --zero-superblock /dev/sdb7
  
−
5. Build the RAID devices using the 'mdadm' command:
+
6. Build the RAID devices using the 'mdadm' command:
  
−
   mdadm -C /dev/md1 -n 2 -l 1 missing /dev/sdb1
+
  mdadm -C /dev/md0 -n 2 -l 1 missing /dev/sdb1
−
   mdadm -C /dev/md2 -n 2 -l 1 missing /dev/sdb2
+
   mdadm -C /dev/md1 -n 2 -l 1 missing /dev/sdb2
−
   .
+
   mdadm -C /dev/md2 -n 2 -l 1 missing /dev/sdb3
−
   .
+
   mdadm -C /dev/md3 -n 2 -l 1 missing /dev/sdb5
−
   .
+
   mdadm -C /dev/md4 -n 2 -l 1 missing /dev/sdb6
 +
   mdadm -C /dev/md5 -n 2 -l 1 missing /dev/sdb7
  
−
Repeat for all partitions, then check your work:
+
Check your work:
  
 
   cat /proc/mdstat
 
   cat /proc/mdstat
−
 
−
6. Update /etc/mdadm.conf
 
−
 
−
  mdadm --detail --scan >> /etc/mdadm.conf
 
  
 
7. Create file a file system on each partition
 
7. Create file a file system on each partition
  
 +
  mkfs.ext3 -j /dev/md0
 
   mkfs.ext3 -j /dev/md1
 
   mkfs.ext3 -j /dev/md1
 
   mkfs.ext3 -j /dev/md2
 
   mkfs.ext3 -j /dev/md2
−
   .
+
   mkfs.ext3 -j /dev/md4
−
   .
+
   mkfs.ext3 -j /dev/md5
−
   mkswap /dev/md5
+
   mkswap /dev/md3
  
−
8. Make a mount point and mount all the partitions:
+
8. Update /etc/mdadm.conf
 +
 
 +
  mdadm --detail --scan > /etc/mdadm.conf
 +
 
 +
9. Make a mount point and mount all the partitions:
  
 
   mkdir /mnt/md
 
   mkdir /mnt/md
−
   mount /dev/md6 /mnt/md
+
   mount /dev/md4 /mnt/md
 
   mkdir /mnt/md/boot
 
   mkdir /mnt/md/boot
 
   mkdir /mnt/md/usr
 
   mkdir /mnt/md/usr
Line 184: Line 197:
 
   cd /mnt
 
   cd /mnt
 
   chmod -R 755 md/
 
   chmod -R 755 md/
−
   mount /dev/md1 /mnt/md/boot
+
   mount /dev/md0 /mnt/md/boot
−
   mount /dev/md3 /mnt/md/usr
+
   mount /dev/md2 /mnt/md/usr
−
   mount /dev/md7 /mnt/md/var
+
   mount /dev/md5 /mnt/md/var
−
   mount /dev/md2 /mnt/md/home
+
   mount /dev/md1 /mnt/md/home
  
−
9. Copy the files from disk 1 to the newly built raid
+
10. Copy the files from disk 1 to the newly built raid
  
 
   rsync -aXv ––exclude=/proc/* ––exclude=/sys/* ––exclude=/dev/* ––exclude=/mnt/md / /mnt/md
 
   rsync -aXv ––exclude=/proc/* ––exclude=/sys/* ––exclude=/dev/* ––exclude=/mnt/md / /mnt/md
Line 195: Line 208:
 
NOTE: Remember to escape the * if you're using a shell like tcsh.
 
NOTE: Remember to escape the * if you're using a shell like tcsh.
  
−
10. Bind /proc, /sys & /dev to the new filesystem
+
11. Edit /etc/fstab to use the appropriate metadevices.
−
 
 
−
  mount -o bind /proc /mnt/md/proc
 
−
  mount -o bind /sys /mnt/md/sys
 
−
  mount -o bind /dev /mnt/md/dev
 
−
  mount -o bind /dev/pts /mnt/md/dev/pts
 
−
 
 
−
11. Change the root device
 
−
 
 
−
  chroot /mnt/md
 
−
 
 
−
12. Edit /etc/fstab to use the appropriate metadevices.
 
  
 
   LABEL=/                /                      ext3    defaults        1 1
 
   LABEL=/                /                      ext3    defaults        1 1
Line 221: Line 223:
 
becomes
 
becomes
  
−
   /dev/md6               /                      ext3    defaults        1 1
+
   /dev/md4               /                      ext3    defaults        1 1
−
   /dev/md7               /var                    ext3    defaults        1 2
+
   /dev/md5               /var                    ext3    defaults        1 2
−
   /dev/md3               /usr                    ext3    defaults        1 2
+
   /dev/md2               /usr                    ext3    defaults        1 2
−
   /dev/md2               /home                  ext3    defaults        1 2
+
   /dev/md1               /home                  ext3    defaults        1 2
−
   /dev/md1               /boot                  ext3    defaults        1 2
+
   /dev/md0               /boot                  ext3    defaults        1 2
 
   tmpfs                  /dev/shm                tmpfs  defaults        0 0
 
   tmpfs                  /dev/shm                tmpfs  defaults        0 0
 
   devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
 
   devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
 
   sysfs                  /sys                    sysfs  defaults        0 0
 
   sysfs                  /sys                    sysfs  defaults        0 0
 
   proc                    /proc                  proc    defaults        0 0
 
   proc                    /proc                  proc    defaults        0 0
−
   /dev/md5               swap                    swap    defaults        0 0
+
   /dev/md3               swap                    swap    defaults        0 0
  
−
13. Edit /boot/grub/device.map and add the following entry for the second disk.
+
**13. Edit /boot/grub/device.map and add the following entry for the second disk.
  
 
   (hd1)    /dev/sdb
 
   (hd1)    /dev/sdb
  
−
14. Install GRUB on the second disk
+
14. Edit /boot/grub/menu.lst to allow fallback to the next kernel in the event of a boot failure:
 +
 
 +
  vim /boot/grub/menu.lst
 +
 
 +
and add
 +
 
 +
  default=0
 +
  '''fallback=1'''
 +
 
 +
15. Add an entry for in menu.lst to allow booting off metadevice root
 +
 
 +
  '''title CentOS (2.6.18-194.26.1.el5 metadevices)
 +
          root (hd1,0)
 +
          kernel /vmlinuz-2.6.18-194.26.1.el5 ro root=/dev/md4
 +
          initrd /initrd-2.6.18-194.26.1.el5.img'''
 +
  title CentOS (2.6.18-194.26.1.el5)
 +
          root (hd0,0)
 +
          kernel /vmlinuz-2.6.18-194.26.1.el5 ro root=LABEL=/
 +
          initrd /initrd-2.6.18-194.26.1.el5.img
 +
 
 +
16. Rebuild ramdisk to reflect our new reality:
 +
 
 +
  mv /boot/initrd-`uname -r`.img /boot/initrd-`uname -r`.img_orig
 +
  mkinitrd /boot/initrd-`uname -r`.img `uname -r`
 +
 
 +
17. Setup GRUB on both disks
 +
 
 +
  grub
 +
 
 +
  Probing devices to guess BIOS drives. This may take a long time.
 +
 
  
−
  grub --device-map=/boot/grub/device.map
 
−
 
 
−
 
 
 
       GNU GRUB  version 0.97  (640K lower / 3072K upper memory)
 
       GNU GRUB  version 0.97  (640K lower / 3072K upper memory)
  
Line 246: Line 275:
 
     lists possible command completions.  Anywhere else TAB lists the possible
 
     lists possible command completions.  Anywhere else TAB lists the possible
 
     completions of a device/filename.]
 
     completions of a device/filename.]
 +
  grub> '''root (hd0,0)'''
 +
  root (hd0,0)
 +
  Filesystem type is ext2fs, partition type 0x83
 +
  grub> '''setup (hd0)'''
 +
  setup (hd0)
 +
  Checking if "/boot/grub/stage1" exists... no
 +
  Checking if "/grub/stage1" exists... yes
 +
  Checking if "/grub/stage2" exists... yes
 +
  Checking if "/grub/e2fs_stage1_5" exists... yes
 +
  Running "embed /grub/e2fs_stage1_5 (hd0)"...  15 sectors are embedded.
 +
  succeeded
 +
  Running "install /grub/stage1 (hd0) (hd0)1+15 p (hd0,0)/grub/stage2 /grub/grub.conf"... succeeded
 +
  Done.
 
   grub> '''root (hd1,0)'''
 
   grub> '''root (hd1,0)'''
 
   root (hd1,0)
 
   root (hd1,0)
Line 261: Line 303:
 
   grub> '''quit'''
 
   grub> '''quit'''
  
−
15. Rebuild initramfs and force it to detect RAID, otherwise you'll get a kernel panic
+
18. Reboot and hope it works!
−
 
 
−
  cd /boot
 
−
  mkinitrd -f --force-raid-probe initrd-`uname -r`.img `uname -r`
 
−
 
 
−
16. Edit /boot/grub/grub.conf to boot of second disk and use the metadevice.  Change grub.cfg from:
 
−
 
 
−
  splashimage=(hd0,0)/grub/splash.xpm.gz
 
−
  hiddenmenu
 
−
  title CentOS (2.6.18-194.el5)
 
−
          root (hd0,0)
 
−
          kernel /vmlinuz-2.6.18-194.el5 ro root=LABEL=/ console=tty0 console=ttyS0,9600
 
−
          initrd /initrd-2.6.18-194.el5.img
 
−
 
 
−
to:
 
−
 
 
−
  splashimage=(hd'''1''',0)/grub/splash.xpm.gz
 
−
  hiddenmenu
 
−
  title CentOS (2.6.18-194.el5)
 
−
          root (hd'''1''',0)
 
−
          kernel /vmlinuz-2.6.18-194.el5 ro root='''/dev/md6''' console=tty0 console=ttyS0,9600
 
−
          initrd /initrd-2.6.18-194.el5.img
 
−
 
 
−
17. Exit back to the default environment and unmount all the meta device file systems
 
−
 
 
−
  exit
 
−
  umount /mnt/md/dev/pts
 
−
  umount /mnt/md/dev
 
−
  umount /mnt/md/sys
 
−
  umount /mnt/md/proc
 
−
  umount /mnt/md/boot
 
−
  umount /mnt/md/usr
 
−
  umount /mnt/md/var
 
−
  umount /mnt/md/home
 
−
  umount /mnt/md
 
−
 
 
−
18. Reboot the machine and go directly into the BIOS.
 

Revision as of 00:16, 11 November 2010

Solaris:

In Solaris 10, you can mirror the disks during the build process with Jumpstart, or even use ZFS mirroring. However, on older systems, you need to mirror the disks the old fashioned way with DiskSuite 4.2.1. In this example, the file names for the two disks being mirrored are c1t0d0sX and c1t1d0sX. The disks are partitioned like this: 

  /         c1t0d0s0
  swap      c1t0d0s1
  /var      c1t0d0s3
  /usr      c1t0d0s4
  /opt      c1t0d0s5
  /export   c1t0d0s6
  -         c1t0d0s7

Slice 7 is a 50mb partition used to store the metadatabaes! IMO, it is best practice to never use slice 7, except for this purpose!

1. You need to make sure the following packages are installed first: 

 SUNWmdr SUNWmdu SUNWmdx

This is the least you need to install! Oh and they will need to bzip to install successfully! They can be found on "Solaris 8 Software CD 2" (not sure for Solaris 9, but they could be part of the standard disto), in the following location: 

 cd $PATH_TO_CDROM/Solaris_8/EA/products/DiskSuite_4.2.1/sparc/Packages
 pkgadd -d . SUNWmdr SUNWmdu SUNWmdx

Once they are installed, reboot the box. It wouldn't hurt to do an 'init 0' and then a 'boot -r'!

2. Copy the primary disks VTOC to the secondary disk: 

  prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2

3. Create the metadatabases 

  metadb -f -a -c3 /dev/dsk/c1t0d0s7
  metadb -f -a -c3 /dev/dsk/c1t1d0s7

4. Create and initialise the metadevices on the primary disk 

  metainit -f d10 1 1 c1t0d0s0
  metainit -f d10 1 1 c1t0d0s1
  metainit -f d10 1 1 c1t0d0s3
  metainit -f d10 1 1 c1t0d0s4
  metainit -f d10 1 1 c1t0d0s5
  metainit -f d10 1 1 c1t0d0s6
  metainit d0 -m d10
  metainit d1 -m d11
  metainit d3 -m d13
  metainit d4 -m d14
  metainit d5 -m d15
  metainit d6 -m d16

5. Assign the root metadevice 

  metaroot d0

6. Edit /etc/vfstab and make sure that all the devices are changed to metadevices. So each line that looked like this 

  /dev/dsk/c1t0d0s3 /dev/rdsk/c1t0d0s3  /var  ufs   1 no    -

now looks like 

  /dev/md/dsk/d3 /dev/md/rdsk/d3  /var  ufs   1 no    -

7. Run the lockfs command and reboot 

  lockfs -fa
  reboot

8. Create the the metadevices on the secondary disk 

  metainit d20 1 1 c1t1d0s0
  metainit d21 1 1 c1t1d0s1
  metainit d23 1 1 c1t1d0s3
  metainit d24 1 1 c1t1d0s4
  metainit d25 1 1 c1t1d0s5
  metainit d26 1 1 c1t1d0s6

9. Attach the metadevices from the secondary disk to the metadevices we booted from to start the mirroring process 

  metattach d0 d20
  metattach d1 d21
  metattach d3 d23
  metattach d4 d24
  metattach d5 d25
  metattach d6 d26

You can monitor the progress of mirroring with this command: 

  metastat | grep -i progress

When it returns nothing, it's all finished.

10. Install the boot block on the secondary hard disk so that you can boot off it 

  installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t1d0s0

11. Disable the following in /etc/inetd.conf 

  100229/1 (rpc.metad)
  100230/1 (rpc.metamhd)

12. Add this line to /etc/system so that the machine will boot if there are only 50% of the metadatabases available, i.e. a disk has failed 

  set md:mirrored_root_flag=1


You're now finished :)

CentOS/RedHat

DON'T USE THEE INSTRUCTIONS! I have yet to make them work properly!

OK, so software RAID on CentOS/RedHat sucks! Unless you build the box from the outset with mirrored disk, you're going to need to do a lot of work to get to a point where you're going to be happy. Wherever possible, mirror the disks during build! However, if the box is already built and you need to mirror the boot disk after the fact, here's how you do it:

1. Make sure that 'mdadm' is installed: 

 yum install mdadm

2. Load the RAID kernel modules 

 modprobe linear
 modprobe multipath
 modprobe raid0
 modprobe raid1
 modprobe raid5
 modprobe raid6
 modprobe raid10

3. Copy the partition table from one disk to the other 

 sfdisk -d /dev/sda | sfdisk /dev/sdb

4. Change the partition type of the second disk to 'raid': 

 fdisk /dev/sdb
 
 The number of cylinders for this disk is set to 8924.
 There is nothing wrong with that, but this is larger than 1024,
 and could in certain setups cause problems with:
 1) software that runs at boot time (e.g., old versions of LILO)
 2) booting and partitioning software from other OSs
    (e.g., DOS FDISK, OS/2 FDISK)
 
 Command (m for help): t
 Partition number (1-7): 1
 Hex code (type L to list codes): fd
 
 Command (m for help): t
 Partition number (1-7): 2
 Hex code (type L to list codes): fd

Etc. Repeat for all partitions and then exit: 

 Command (m for help): w

5. To make sure that there are no remains from previous RAID installations on /dev/sdb, we run the following commands: 

 mdadm --zero-superblock /dev/sdb1
 mdadm --zero-superblock /dev/sdb2
 mdadm --zero-superblock /dev/sdb3
 mdadm --zero-superblock /dev/sdb5
 mdadm --zero-superblock /dev/sdb6
 mdadm --zero-superblock /dev/sdb7

6. Build the RAID devices using the 'mdadm' command: 

 mdadm -C /dev/md0 -n 2 -l 1 missing /dev/sdb1
 mdadm -C /dev/md1 -n 2 -l 1 missing /dev/sdb2
 mdadm -C /dev/md2 -n 2 -l 1 missing /dev/sdb3
 mdadm -C /dev/md3 -n 2 -l 1 missing /dev/sdb5
 mdadm -C /dev/md4 -n 2 -l 1 missing /dev/sdb6
 mdadm -C /dev/md5 -n 2 -l 1 missing /dev/sdb7

Check your work: 

 cat /proc/mdstat

7. Create file a file system on each partition 

 mkfs.ext3 -j /dev/md0
 mkfs.ext3 -j /dev/md1
 mkfs.ext3 -j /dev/md2
 mkfs.ext3 -j /dev/md4
 mkfs.ext3 -j /dev/md5
 mkswap /dev/md3

8. Update /etc/mdadm.conf 

 mdadm --detail --scan > /etc/mdadm.conf

9. Make a mount point and mount all the partitions: 

 mkdir /mnt/md
 mount /dev/md4 /mnt/md
 mkdir /mnt/md/boot
 mkdir /mnt/md/usr
 mkdir /mnt/md/var
 mkdir /mnt/md/home
 cd /mnt
 chmod -R 755 md/
 mount /dev/md0 /mnt/md/boot
 mount /dev/md2 /mnt/md/usr
 mount /dev/md5 /mnt/md/var
 mount /dev/md1 /mnt/md/home

10. Copy the files from disk 1 to the newly built raid 

 rsync -aXv ––exclude=/proc/* ––exclude=/sys/* ––exclude=/dev/* ––exclude=/mnt/md / /mnt/md

NOTE: Remember to escape the * if you're using a shell like tcsh.

11. Edit /etc/fstab to use the appropriate metadevices. 

 LABEL=/                 /                       ext3    defaults        1 1
 LABEL=/var              /var                    ext3    defaults        1 2
 LABEL=/usr              /usr                    ext3    defaults        1 2
 LABEL=/home             /home                   ext3    defaults        1 2
 LABEL=/boot             /boot                   ext3    defaults        1 2
 tmpfs                   /dev/shm                tmpfs   defaults        0 0
 devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
 sysfs                   /sys                    sysfs   defaults        0 0
 proc                    /proc                   proc    defaults        0 0
 LABEL=SWAP-sda5         swap                    swap    defaults        0 0

becomes 

 /dev/md4                /                       ext3    defaults        1 1
 /dev/md5                /var                    ext3    defaults        1 2
 /dev/md2                /usr                    ext3    defaults        1 2
 /dev/md1                /home                   ext3    defaults        1 2
 /dev/md0                /boot                   ext3    defaults        1 2
 tmpfs                   /dev/shm                tmpfs   defaults        0 0
 devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
 sysfs                   /sys                    sysfs   defaults        0 0
 proc                    /proc                   proc    defaults        0 0
 /dev/md3                swap                    swap    defaults        0 0
    • 13. Edit /boot/grub/device.map and add the following entry for the second disk.
 (hd1)     /dev/sdb

14. Edit /boot/grub/menu.lst to allow fallback to the next kernel in the event of a boot failure: 

  vim /boot/grub/menu.lst

and add 

 default=0
 fallback=1

15. Add an entry for in menu.lst to allow booting off metadevice root 

 title CentOS (2.6.18-194.26.1.el5 metadevices)
         root (hd1,0)
         kernel /vmlinuz-2.6.18-194.26.1.el5 ro root=/dev/md4
         initrd /initrd-2.6.18-194.26.1.el5.img
 title CentOS (2.6.18-194.26.1.el5)
         root (hd0,0)
         kernel /vmlinuz-2.6.18-194.26.1.el5 ro root=LABEL=/
         initrd /initrd-2.6.18-194.26.1.el5.img

16. Rebuild ramdisk to reflect our new reality: 

 mv /boot/initrd-`uname -r`.img /boot/initrd-`uname -r`.img_orig
 mkinitrd /boot/initrd-`uname -r`.img `uname -r`

17. Setup GRUB on both disks 

 grub

 Probing devices to guess BIOS drives. This may take a long time.



     GNU GRUB  version 0.97  (640K lower / 3072K upper memory)

  [ Minimal BASH-like line editing is supported.  For the first word, TAB
    lists possible command completions.  Anywhere else TAB lists the possible
    completions of a device/filename.]
 grub> root (hd0,0)
 root (hd0,0)
  Filesystem type is ext2fs, partition type 0x83
 grub> setup (hd0)
 setup (hd0)
  Checking if "/boot/grub/stage1" exists... no
  Checking if "/grub/stage1" exists... yes
  Checking if "/grub/stage2" exists... yes
  Checking if "/grub/e2fs_stage1_5" exists... yes
  Running "embed /grub/e2fs_stage1_5 (hd0)"...  15 sectors are embedded.
 succeeded
  Running "install /grub/stage1 (hd0) (hd0)1+15 p (hd0,0)/grub/stage2 /grub/grub.conf"... succeeded
 Done.
 grub> root (hd1,0)
 root (hd1,0)
  Filesystem type is ext2fs, partition type 0xfd
 grub> setup (hd1)
 setup (hd1)
  Checking if "/boot/grub/stage1" exists... no
  Checking if "/grub/stage1" exists... yes
  Checking if "/grub/stage2" exists... yes
  Checking if "/grub/e2fs_stage1_5" exists... yes
  Running "embed /grub/e2fs_stage1_5 (hd1)"...  15 sectors are embedded.
 succeeded
  Running "install /grub/stage1 (hd1) (hd1)1+15 p (hd1,0)/grub/stage2 /grub/grub.conf"... succeeded
 Done.
 grub> quit

18. Reboot and hope it works!

Retrieved from "https://peterpap.net/index.php?title=Mirroring_Disks&oldid=140"