Difference between revisions of "Patching with Live Upgrade"

From Peter Pap's Technowiki
Jump to: navigation, search
(→‎Live Upgrade)
(→‎Live Upgrade)
 
(4 intermediate revisions by the same user not shown)
Line 181: Line 181:
  
 
   ludelete base
 
   ludelete base
 +
 +
Output should look like:
 +
 +
  Determining the devices to be marked free.
 +
  Updating boot environment configuration database.
 +
  Updating boot environment description database on all BEs.
 +
  Updating all boot environment configuration databases.
 +
  Boot environment <base> deleted.
  
 
6. Use metaclear to clear old mirrors, d0, d20, d30 and  d40
 
6. Use metaclear to clear old mirrors, d0, d20, d30 and  d40
Line 189: Line 197:
 
   metaclear d40
 
   metaclear d40
  
−
7. Re-create the mirrors and attach the unused slices
+
7. Create a new boot environment re-using the the mirrors and the unused slices from the now deleted 'base' boot environment
  
 
   lucreate -n patched_20101126 \
 
   lucreate -n patched_20101126 \
Line 197: Line 205:
 
   -m /usr:/dev/md/dsk/d40:ufs,mirror -m /usr:/dev/md/dsk/d41:attach
 
   -m /usr:/dev/md/dsk/d40:ufs,mirror -m /usr:/dev/md/dsk/d41:attach
  
−
This will re-create the meta devices d0, d20, d30 and d40 with sub-devices d1, d21, d31 and d41 respectively.  It then copies the data from the current boot environment, 'after_patch' and create a new boot environment called 'patched_20101126'.
+
This will re-create the meta devices d0, d20, d30 and d40 with sub-devices d1, d21, d31 and d41 respectively.  It then copies the data from the current boot environment, 'after_patch' and create a new boot environment called 'patched_20101126'.  Out put should look like:
 +
 
 +
  Discovering physical storage devices
 +
  Discovering logical storage devices
 +
  Cross referencing storage devices with boot environment configurations
 +
  Determining types of file systems supported
 +
  Validating file system requests
 +
  The device name </dev/md/dsk/d0> expands to device path </dev/md/dsk/d0>
 +
  Preparing logical storage devices
 +
  Preparing physical storage devices
 +
  Configuring physical storage devices
 +
  Configuring logical storage devices
 +
  Analyzing system configuration.
 +
  Comparing source boot environment <after_patch> file systems with the file
 +
  system(s) you specified for the new boot environment. Determining which
 +
  file systems should be in the new boot environment.
 +
  Updating boot environment description database on all BEs.
 +
  Searching /dev for possible boot environment filesystem devices
 +
 
 +
  Updating system configuration files.
 +
  The device </dev/dsk/c0t0d0s0> is not a root device for any boot environment; cannot get BE ID.
 +
  Creating configuration for boot environment <patched_20101126>.
 +
  Source boot environment is <after_patch>.
 +
  Creating boot environment <patched_20101126>.
 +
  Creating file systems on boot environment <patched_20101126>.
 +
  Creating <ufs> file system for </> in zone <global> on </dev/md/dsk/d0>.
 +
  Creating <ufs> file system for </opt> in zone <global> on </dev/md/dsk/d30>.
 +
  Creating <ufs> file system for </usr> in zone <global> on </dev/md/dsk/d40>.
 +
  Creating <ufs> file system for </var> in zone <global> on </dev/md/dsk/d20>.
 +
  Mounting file systems for boot environment <patched_20101126>.
 +
  Calculating required sizes of file systems              for boot environment <patched_20101126>.
 +
  Populating file systems on boot environment <patched_20101126>.
 +
  Checking selection integrity.
 +
  Integrity check OK.
 +
  Populating contents of mount point </>.
 +
  Populating contents of mount point </opt>.
 +
  Populating contents of mount point </usr>.
 +
  Populating contents of mount point </var>.
 +
  Copying.
 +
  Creating shared file system mount points.
 +
  Creating compare databases for boot environment <patched_20101126>.
 +
  Creating compare database for file system </var>.
 +
  Creating compare database for file system </usr>.
 +
  Creating compare database for file system </opt>.
 +
  Creating compare database for file system </>.
 +
  Updating compare databases on boot environment <patched_20101126>.
 +
  Making boot environment <patched_20101126> bootable.
 +
  Setting root slice to Solaris Volume Manager metadevice </dev/md/dsk/d0>.
 +
  Population of boot environment <patched_20101126> successful.
 +
  Creation of boot environment <patched_20101126> successful.
  
 
8. Activate the new boot environment
 
8. Activate the new boot environment
  
−
   luactivate patched_30072009
+
   luactivate patched_20101126
 +
 
 +
Output should look like:
 +
 
 +
  A Live Upgrade Sync operation will be performed on startup of boot environment <patched_20101126>.
 +
 
 +
 
 +
  **********************************************************************
 +
 
 +
  The target boot environment has been activated. It will be used when you
 +
  reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
 +
  MUST USE either the init or the shutdown command when you reboot. If you
 +
  do not use either init or shutdown, the system will not boot using the
 +
  target BE.
 +
 
 +
  **********************************************************************
 +
 
 +
  In case of a failure while booting to the target BE, the following process
 +
  needs to be followed to fallback to the currently working boot environment:
 +
 
 +
  1. Enter the PROM monitor (ok prompt).
 +
 
 +
  2. Change the boot device back to the original boot environment by typing:
 +
 
 +
        setenv boot-device /pci@1c,600000/scsi@2/disk@1,0:a
 +
 
 +
  3. Boot to the original boot environment by typing:
 +
 
 +
        boot
 +
 
 +
  **********************************************************************
 +
 
 +
  Modifying boot archive service
 +
  Activation of boot environment <patched_20101126> successful.
 +
 
  
 
9. Reboot the machine with the 'shutdown' command
 
9. Reboot the machine with the 'shutdown' command
Line 207: Line 298:
 
   shutdown -g 0 -i 6 -y
 
   shutdown -g 0 -i 6 -y
  
 +
10. Use the 'lustatus' and df commands to check you've booted off the correct boot environment, 'patched_20101126'
  
−
10.  Check that you have indeed booted off the new boot environment
+
  lustatus
 +
  Boot Environment          Is      Active Active    Can    Copy     
 +
  Name                      Complete Now    On Reboot Delete Status   
 +
  -------------------------- -------- ------ --------- ------ ----------
 +
  after_patch                yes      no    no        yes    -       
 +
  patched_20101126          yes      yes    yes      no    -       
 +
 
 +
  df -h
 +
  Filesystem            Size  Used Avail Use% Mounted on
 +
  /dev/md/dsk/d0        4.0G 412M  3.5G  11% /
 +
  /dev/md/dsk/d40      4.0G  1.4G  2.6G  36% /usr
 +
  /dev/md/dsk/d20      7.9G  52M  7.8G  1% /var
 +
  /dev/md/dsk/d30      4.0G  352M  3.6G  9% /opt
  
 
11. Delete the old boot environment
 
11. Delete the old boot environment
Line 214: Line 318:
 
   ludelete after_patch
 
   ludelete after_patch
  
−
12. Re-attach sub-mirror devices that were used in the 'after_patch' boot environment to the new boot environment 'patched_20101126', and then make the second disk bootable again
+
Check it worked
 +
 
 +
  lustatus
 +
  Boot Environment          Is      Active Active    Can    Copy     
 +
  Name                      Complete Now    On Reboot Delete Status   
 +
  -------------------------- -------- ------ --------- ------ ----------
 +
  patched_20101126          yes      yes    yes      no    -
 +
 
 +
12. Re-attach sub-mirror devices that were used in the 'after_patch' boot environment to the new boot environment 'patched_20101126'
  
 
   metattach d0 d2
 
   metattach d0 d2
Line 220: Line 332:
 
   metattach d30 d32
 
   metattach d30 d32
 
   metattach d40 d42
 
   metattach d40 d42
 +
 +
Wait for the mirrors to finish re-syncing.
 +
 +
13.  Make sure the second drive is still bootable
 +
 
   installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0
 
   installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0
  
−
Wait for the mirrors to finish re-syncing.  Now you're done!
+
Now you're done!

Latest revision as of 05:03, 26 November 2010

So you have a critical system that needs patching, but you can't risk killing the box. Solaris uses "Live Upgrade" as a tool for altering a system, while maintaining the ability to rollback the change. This is ideal for patching a system, especially kernel patches and the like, as it means that you can apply you patches, try them out and rollback if it's failed. The method I describe here, assumes that the boot disks are mirrored. You can still use Live Upgrade in a non-mirrored environment, but I'm not going in to that here.

SO in this example, the OS is partitioned as follows: 

  [root@testboxen]$ df -h
  Filesystem             size   used  avail capacity  Mounted on
  /dev/md/dsk/d0         3.9G   395M   3.5G    10%    /
  /dev/md/dsk/d40        3.9G   1.4G   2.5G    36%    /usr
  /dev/md/dsk/d20        7.9G    51M   7.8G     1%    /var
  /dev/md/dsk/d30        3.9G   351M   3.6G     9%    /opt

Obviously d0, d20, d30 and d40 are all mirrored meta devices. d10, is also a meta device, for swap. It is not relevant here. These meta devices are set up as follows with the sub-mirrors: 

  d0  : d1  and d2
  d20 : d21 and d22
  d30 : d31 and d32
  d40 : d41 and d42


Preparation

So to use live upgrade, you first need to make sure that the necessary packages are installed and up to date.

1. Download and install the latest version of patch utilities patch 119254 (64 or higher) from sunsolve.sun.com 

  unzip 119254-66.zip
  patchadd -M . 119254-66

2. Download and install the latest version of pax patch 128330 from sunsolve.sun.com 

  unzip 128330-02.zip
  patchadd -M . 128330-02

3. Install the Live Upgrade packages if they don't already exist, from the Solaris install cd 

  cd PATH_TO_CDROM/Solaris_10/Tools/Installers/
  ./liveupgrade20 -noconsole - nodisplay

4. Check the installation worked 

  pkgchk -v SUNWlucfg SUNWlur SUNWluu

Once, you've got the necessary packages installed, you can proceed with the Live Upgrade.


Live Upgrade

In this example, we assume that Live Upgrade has never been used before, so we will have to set up a base boot environment as well as a new boot environment that we will patch.

1. Create the base and the new boot environments. This will break the mirror!!!! 

 lucreate -c base -m /:/dev/md/dsk/d2:ufs,detach \
 -m /var:/dev/md/dsk/d22:ufs,detach \
 -m /opt:/dev/md/dsk/d32:ufs,detach \
 -m /usr:/dev/md/dsk/d42:ufs,detach \
 -n after_patch

The output will look like this: 

  Discovering physical storage devices
  Discovering logical storage devices
  Cross referencing storage devices with boot environment configurations
  Determining types of file systems supported
  Validating file system requests
  Preparing logical storage devices
  Preparing physical storage devices
  Configuring physical storage devices
  Configuring logical storage devices
  Analyzing system configuration.
  No name for current boot environment.
  Current boot environment is named <base>.
  Creating initial configuration for primary boot environment <base>.
  WARNING: The device </dev/md/dsk/d0> for the root file system mount point </> is not a physical device.
  WARNING: The system boot prom identifies the physical device </dev/dsk/c0t0d0s0> as the system boot device.
  Is the physical device </dev/dsk/c0t0d0s0> the boot device for the logical device </dev/md/dsk/d0>? (yes or no) yes
  INFORMATION: Assuming the boot device </dev/dsk/c0t0d0s0> obtained from the system boot prom is the physical boot device for logical device </dev/md/dsk/d0>.
  The device </dev/dsk/c0t0d0s0> is not a root device for any boot environment; cannot get BE ID.
  PBE configuration successful: PBE name <base> PBE Boot Device </dev/dsk/c0t0d0s0>.
  Comparing source boot environment <base> file systems with the file 
  system(s) you specified for the new boot environment. Determining which 
  file systems should be in the new boot environment.
  Updating boot environment description database on all BEs.
  Searching /dev for possible boot environment filesystem devices
  
  Updating system configuration files.
  The device </dev/dsk/c0t1d0s0> is not a root device for any boot environment; cannot get BE ID.
  Creating configuration for boot environment <after_patch>.
  Source boot environment is <base>.
  Creating boot environment <after_patch>.
  Creating file systems on boot environment <after_patch>.
  Creating <ufs> file system for </> in zone <global> on </dev/md/dsk/d2>.
  Creating <ufs> file system for </opt> in zone <global> on </dev/md/dsk/d32>.
  Creating <ufs> file system for </usr> in zone <global> on </dev/md/dsk/d42>.
  Creating <ufs> file system for </var> in zone <global> on </dev/md/dsk/d22>.
  Mounting file systems for boot environment <after_patch>.
  Calculating required sizes of file systems              for boot environment <after_patch>.
  Populating file systems on boot environment <after_patch>.
  Checking selection integrity.
  Integrity check OK.
  Populating contents of mount point </>.
  Populating contents of mount point </opt>.
  Populating contents of mount point </usr>.
  Populating contents of mount point </var>.
  Copying.
  Creating compare databases for boot environment <after_patch>.
  Creating compare database for file system </var>.
  Creating compare database for file system </usr>.
  Creating compare database for file system </opt>.
  Creating compare database for file system </>.
  Updating compare databases on boot environment <after_patch>.

Check that it worked with the lustatus command: 

  lustatus
  Boot Environment           Is       Active Active    Can    Copy      
  Name                       Complete Now    On Reboot Delete Status    
  -------------------------- -------- ------ --------- ------ ----------
  base                       yes      yes    yes       no     -         
  after_patch                yes      no     no        yes    -

2. Activate the new boot environment 

  luactivate after_patch
  A Live Upgrade Sync operation will be performed on startup of boot environment <after_patch>.
  
  
  **********************************************************************
  
  The target boot environment has been activated. It will be used when you 
  reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You 
  MUST USE either the init or the shutdown command when you reboot. If you 
  do not use either init or shutdown, the system will not boot using the 
  target BE.
  
  **********************************************************************
  
  In case of a failure while booting to the target BE, the following process 
  needs to be followed to fallback to the currently working boot environment:
  
  1. Enter the PROM monitor (ok prompt).
  
  2. Change the boot device back to the original boot environment by typing:
  
       setenv boot-device /pci@1c,600000/scsi@2/disk@0,0:a
  
  3. Boot to the original boot environment by typing:
  
       boot
  
  **********************************************************************
  
  Modifying boot archive service
  Activation of boot environment <after_patch> successful.

lustatus output should now look like this 

  lustatus
  Boot Environment           Is       Active Active    Can    Copy      
  Name                       Complete Now    On Reboot Delete Status    
  -------------------------- -------- ------ --------- ------ ----------
  base                       yes      yes    no        no     -         
  after_patch                yes      no     yes       no     -

3. Reboot the server using the 'shutdown' command. DO NOT use the 'reboot' command as it will then ignore the changes you've made and boot off the original boot environment 

  shutdown -g 0 -i 6 -y

After the reboot, you should see the following disk layout from the 'df' command: 

  df -h
  Filesystem             size   used  avail capacity  Mounted on
  /dev/md/dsk/d2         3.9G   412M   3.5G    11%    /
  /dev/md/dsk/d42        3.9G   1.4G   2.5G    36%    /usr
  /dev/md/dsk/d22        7.9G    51M   7.8G     1%    /var
  /dev/md/dsk/d32        3.9G   351M   3.6G     9%    /opt

4. Patch the box with the latest patch cluster and reboot. If all is OK, repeat steps 2 and 3 and then go to step 5.

5. If you're happy that the machine is happy, then delete the old boot environment 

  ludelete base

Output should look like: 

  Determining the devices to be marked free.
  Updating boot environment configuration database.
  Updating boot environment description database on all BEs.
  Updating all boot environment configuration databases.
  Boot environment <base> deleted.

6. Use metaclear to clear old mirrors, d0, d20, d30 and d40 

  metaclear d0
  metaclear d20
  metaclear d30
  metaclear d40

7. Create a new boot environment re-using the the mirrors and the unused slices from the now deleted 'base' boot environment 

  lucreate -n patched_20101126 \
  -m /:/dev/md/dsk/d0:ufs,mirror -m /:/dev/md/dsk/d1:attach \
  -m /var:/dev/md/dsk/d20:ufs,mirror -m /var:/dev/md/dsk/d21:attach \
  -m /opt:/dev/md/dsk/d30:ufs,mirror -m /opt:/dev/md/dsk/d31:attach \
  -m /usr:/dev/md/dsk/d40:ufs,mirror -m /usr:/dev/md/dsk/d41:attach

This will re-create the meta devices d0, d20, d30 and d40 with sub-devices d1, d21, d31 and d41 respectively. It then copies the data from the current boot environment, 'after_patch' and create a new boot environment called 'patched_20101126'. Out put should look like: 

  Discovering physical storage devices
  Discovering logical storage devices
  Cross referencing storage devices with boot environment configurations
  Determining types of file systems supported
  Validating file system requests
  The device name </dev/md/dsk/d0> expands to device path </dev/md/dsk/d0>
  Preparing logical storage devices
  Preparing physical storage devices
  Configuring physical storage devices
  Configuring logical storage devices
  Analyzing system configuration.
  Comparing source boot environment <after_patch> file systems with the file 
  system(s) you specified for the new boot environment. Determining which 
  file systems should be in the new boot environment.
  Updating boot environment description database on all BEs.
  Searching /dev for possible boot environment filesystem devices
  
  Updating system configuration files.
  The device </dev/dsk/c0t0d0s0> is not a root device for any boot environment; cannot get BE ID.
  Creating configuration for boot environment <patched_20101126>.
  Source boot environment is <after_patch>.
  Creating boot environment <patched_20101126>.
  Creating file systems on boot environment <patched_20101126>.
  Creating <ufs> file system for </> in zone <global> on </dev/md/dsk/d0>.
  Creating <ufs> file system for </opt> in zone <global> on </dev/md/dsk/d30>.
  Creating <ufs> file system for </usr> in zone <global> on </dev/md/dsk/d40>.
  Creating <ufs> file system for </var> in zone <global> on </dev/md/dsk/d20>.
  Mounting file systems for boot environment <patched_20101126>.
  Calculating required sizes of file systems              for boot environment <patched_20101126>.
  Populating file systems on boot environment <patched_20101126>.
  Checking selection integrity.
  Integrity check OK.
  Populating contents of mount point </>.
  Populating contents of mount point </opt>.
  Populating contents of mount point </usr>.
  Populating contents of mount point </var>.
  Copying.
  Creating shared file system mount points.
  Creating compare databases for boot environment <patched_20101126>.
  Creating compare database for file system </var>.
  Creating compare database for file system </usr>.
  Creating compare database for file system </opt>.
  Creating compare database for file system </>.
  Updating compare databases on boot environment <patched_20101126>.
  Making boot environment <patched_20101126> bootable.
  Setting root slice to Solaris Volume Manager metadevice </dev/md/dsk/d0>.
  Population of boot environment <patched_20101126> successful.
  Creation of boot environment <patched_20101126> successful.

8. Activate the new boot environment 

  luactivate patched_20101126

Output should look like: 

  A Live Upgrade Sync operation will be performed on startup of boot environment <patched_20101126>.
  
  
  **********************************************************************
  
  The target boot environment has been activated. It will be used when you 
  reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You 
  MUST USE either the init or the shutdown command when you reboot. If you 
  do not use either init or shutdown, the system will not boot using the 
  target BE.
  
  **********************************************************************
  
  In case of a failure while booting to the target BE, the following process 
  needs to be followed to fallback to the currently working boot environment:
  
  1. Enter the PROM monitor (ok prompt).
  
  2. Change the boot device back to the original boot environment by typing:

       setenv boot-device /pci@1c,600000/scsi@2/disk@1,0:a
  
  3. Boot to the original boot environment by typing:
  
       boot
  
  **********************************************************************
  
  Modifying boot archive service
  Activation of boot environment <patched_20101126> successful.


9. Reboot the machine with the 'shutdown' command 

  shutdown -g 0 -i 6 -y

10. Use the 'lustatus' and df commands to check you've booted off the correct boot environment, 'patched_20101126' 

  lustatus
  Boot Environment           Is       Active Active    Can    Copy      
  Name                       Complete Now    On Reboot Delete Status    
  -------------------------- -------- ------ --------- ------ ----------
  after_patch                yes      no     no        yes    -         
  patched_20101126           yes      yes    yes       no     -         

  df -h
  Filesystem            Size  Used Avail Use% Mounted on
  /dev/md/dsk/d0        4.0G  412M  3.5G  11% /
  /dev/md/dsk/d40       4.0G  1.4G  2.6G  36% /usr
  /dev/md/dsk/d20       7.9G   52M  7.8G   1% /var
  /dev/md/dsk/d30       4.0G  352M  3.6G   9% /opt

11. Delete the old boot environment 

  ludelete after_patch

Check it worked 

  lustatus
  Boot Environment           Is       Active Active    Can    Copy      
  Name                       Complete Now    On Reboot Delete Status    
  -------------------------- -------- ------ --------- ------ ----------
  patched_20101126           yes      yes    yes       no     -

12. Re-attach sub-mirror devices that were used in the 'after_patch' boot environment to the new boot environment 'patched_20101126' 

  metattach d0 d2
  metattach d20 d22
  metattach d30 d32
  metattach d40 d42

Wait for the mirrors to finish re-syncing.

13. Make sure the second drive is still bootable 

  installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0

Now you're done!

Retrieved from "https://peterpap.net/index.php?title=Patching_with_Live_Upgrade&oldid=189"