View source for Patching with Live Upgrade

So you have a critical system that needs patching, but you can't risk killing the box.  Solaris uses "Live Upgrade" as a tool for altering a system, while maintaining the ability to rollback the change.  This is ideal for patching a system, especially kernel patches and the like, as it means that you can apply you patches, try them out and rollback if it's failed.  The method I describe here, assumes that the boot disks are mirrored.  You can still use Live Upgrade in a non-mirrored environment, but I'm not going in to that here.

SO in this example, the OS is partitioned as follows:

   [root@testboxen]$ df -h
   Filesystem             size   used  avail capacity  Mounted on
   /dev/md/dsk/d0         3.9G   395M   3.5G    10%    /
   /dev/md/dsk/d40        3.9G   1.4G   2.5G    36%    /usr
   /dev/md/dsk/d20        7.9G    51M   7.8G     1%    /var
   /dev/md/dsk/d30        3.9G   351M   3.6G     9%    /opt

Obviously d0, d20, d30 and d40 are all mirrored meta devices.  d10, is also a meta device, for swap.  It is not relevant here.  These meta devices are set up as follows with the sub-mirrors:

   d0  : d1  and d2
   d20 : d21 and d22
   d30 : d31 and d32
   d40 : d41 and d42



== Preparation ==

So to use live upgrade, you first need to make sure that the necessary packages are installed and up to date.  

1. Download and install the latest version of patch utilities patch 119254 (64 or higher) from sunsolve.sun.com

   unzip 119254-66.zip
   patchadd -M . 119254-66

2. Download and install the latest version of pax patch 128330 from sunsolve.sun.com

   unzip 128330-02.zip
   patchadd -M . 128330-02

3. Install the Live Upgrade packages if they don't already exist, from the Solaris install cd

   cd PATH_TO_CDROM/Solaris_10/Tools/Installers/
   ./liveupgrade20 -noconsole - nodisplay

4. Check the installation worked

   pkgchk -v SUNWlucfg SUNWlur SUNWluu

Once, you've got the necessary packages installed, you can proceed with the Live Upgrade.


== Live Upgrade ==

In this example, we assume that Live Upgrade has never been used before, so we will have to set up a base boot environment as well as a new boot environment that we will patch.

1. Create the base and the new boot environments. This will break the mirror!!!!

  lucreate -c base -m /:/dev/md/dsk/d2:ufs,detach \
  -m /var:/dev/md/dsk/d22:ufs,detach \
  -m /opt:/dev/md/dsk/d32:ufs,detach \
  -m /usr:/dev/md/dsk/d42:ufs,detach \
  -n after_patch

The output will look like this:

   Discovering physical storage devices
   Discovering logical storage devices
   Cross referencing storage devices with boot environment configurations
   Determining types of file systems supported
   Validating file system requests
   Preparing logical storage devices
   Preparing physical storage devices
   Configuring physical storage devices
   Configuring logical storage devices
   Analyzing system configuration.
   No name for current boot environment.
   Current boot environment is named <base>.
   Creating initial configuration for primary boot environment <base>.
   WARNING: The device </dev/md/dsk/d0> for the root file system mount point </> is not a physical device.
   WARNING: The system boot prom identifies the physical device </dev/dsk/c0t0d0s0> as the system boot device.
   Is the physical device </dev/dsk/c0t0d0s0> the boot device for the logical device </dev/md/dsk/d0>? (yes or no) '''yes'''
   INFORMATION: Assuming the boot device </dev/dsk/c0t0d0s0> obtained from the system boot prom is the physical boot device for logical device </dev/md/dsk/d0>.
   The device </dev/dsk/c0t0d0s0> is not a root device for any boot environment; cannot get BE ID.
   PBE configuration successful: PBE name <base> PBE Boot Device </dev/dsk/c0t0d0s0>.
   Comparing source boot environment <base> file systems with the file 
   system(s) you specified for the new boot environment. Determining which 
   file systems should be in the new boot environment.
   Updating boot environment description database on all BEs.
   Searching /dev for possible boot environment filesystem devices
   
   Updating system configuration files.
   The device </dev/dsk/c0t1d0s0> is not a root device for any boot environment; cannot get BE ID.
   Creating configuration for boot environment <after_patch>.
   Source boot environment is <base>.
   Creating boot environment <after_patch>.
   Creating file systems on boot environment <after_patch>.
   Creating <ufs> file system for </> in zone <global> on </dev/md/dsk/d2>.
   Creating <ufs> file system for </opt> in zone <global> on </dev/md/dsk/d32>.
   Creating <ufs> file system for </usr> in zone <global> on </dev/md/dsk/d42>.
   Creating <ufs> file system for </var> in zone <global> on </dev/md/dsk/d22>.
   Mounting file systems for boot environment <after_patch>.
   Calculating required sizes of file systems              for boot environment <after_patch>.
   Populating file systems on boot environment <after_patch>.
   Checking selection integrity.
   Integrity check OK.
   Populating contents of mount point </>.
   Populating contents of mount point </opt>.
   Populating contents of mount point </usr>.
   Populating contents of mount point </var>.
   Copying.
   Creating compare databases for boot environment <after_patch>.
   Creating compare database for file system </var>.
   Creating compare database for file system </usr>.
   Creating compare database for file system </opt>.
   Creating compare database for file system </>.
   Updating compare databases on boot environment <after_patch>.

Check that it worked with the lustatus command:

   lustatus
   Boot Environment           Is       Active Active    Can    Copy      
   Name                       Complete Now    On Reboot Delete Status    
   -------------------------- -------- ------ --------- ------ ----------
   base                       yes      yes    yes       no     -         
   after_patch                yes      no     no        yes    -

2. Activate the new boot environment

   luactivate after_patch
   A Live Upgrade Sync operation will be performed on startup of boot environment <after_patch>.
   
   
   **********************************************************************
   
   The target boot environment has been activated. It will be used when you 
   reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You 
   MUST USE either the init or the shutdown command when you reboot. If you 
   do not use either init or shutdown, the system will not boot using the 
   target BE.
   
   **********************************************************************
   
   In case of a failure while booting to the target BE, the following process 
   needs to be followed to fallback to the currently working boot environment:
   
   1. Enter the PROM monitor (ok prompt).
   
   2. Change the boot device back to the original boot environment by typing:
   
        setenv boot-device /pci@1c,600000/scsi@2/disk@0,0:a
   
   3. Boot to the original boot environment by typing:
   
        boot
   
   **********************************************************************
   
   Modifying boot archive service
   Activation of boot environment <after_patch> successful.

lustatus output should now look like this

   lustatus
   Boot Environment           Is       Active Active    Can    Copy      
   Name                       Complete Now    On Reboot Delete Status    
   -------------------------- -------- ------ --------- ------ ----------
   base                       yes      yes    no        no     -         
   after_patch                yes      no     yes       no     -

3. Reboot the server using the 'shutdown' command. '''DO NOT''' use the 'reboot' command as it will then ignore the changes you've made and boot off the original boot environment

   shutdown -g 0 -i 6 -y

After the reboot, you should see the following disk layout from the 'df' command:

   df -h
   Filesystem             size   used  avail capacity  Mounted on
   /dev/md/dsk/d2         3.9G   412M   3.5G    11%    /
   /dev/md/dsk/d42        3.9G   1.4G   2.5G    36%    /usr
   /dev/md/dsk/d22        7.9G    51M   7.8G     1%    /var
   /dev/md/dsk/d32        3.9G   351M   3.6G     9%    /opt

4. Patch the box with the latest patch cluster and reboot. If all is OK, repeat steps 2 and 3 and then go to step 5.

5. If you're happy that the machine is happy, then delete the old boot environment

   ludelete base

Output should look like:

   Determining the devices to be marked free.
   Updating boot environment configuration database.
   Updating boot environment description database on all BEs.
   Updating all boot environment configuration databases.
   Boot environment <base> deleted.

6. Use metaclear to clear old mirrors, d0, d20, d30 and  d40

   metaclear d0
   metaclear d20
   metaclear d30
   metaclear d40

7. Create a new boot environment re-using the the mirrors and the unused slices from the now deleted 'base' boot environment

   lucreate -n patched_20101126 \
   -m /:/dev/md/dsk/d0:ufs,mirror -m /:/dev/md/dsk/d1:attach \
   -m /var:/dev/md/dsk/d20:ufs,mirror -m /var:/dev/md/dsk/d21:attach \
   -m /opt:/dev/md/dsk/d30:ufs,mirror -m /opt:/dev/md/dsk/d31:attach \
   -m /usr:/dev/md/dsk/d40:ufs,mirror -m /usr:/dev/md/dsk/d41:attach

This will re-create the meta devices d0, d20, d30 and d40 with sub-devices d1, d21, d31 and d41 respectively.  It then copies the data from the current boot environment, 'after_patch' and create a new boot environment called 'patched_20101126'.  Out put should look like:

   Discovering physical storage devices
   Discovering logical storage devices
   Cross referencing storage devices with boot environment configurations
   Determining types of file systems supported
   Validating file system requests
   The device name </dev/md/dsk/d0> expands to device path </dev/md/dsk/d0>
   Preparing logical storage devices
   Preparing physical storage devices
   Configuring physical storage devices
   Configuring logical storage devices
   Analyzing system configuration.
   Comparing source boot environment <after_patch> file systems with the file 
   system(s) you specified for the new boot environment. Determining which 
   file systems should be in the new boot environment.
   Updating boot environment description database on all BEs.
   Searching /dev for possible boot environment filesystem devices
   
   Updating system configuration files.
   The device </dev/dsk/c0t0d0s0> is not a root device for any boot environment; cannot get BE ID.
   Creating configuration for boot environment <patched_20101126>.
   Source boot environment is <after_patch>.
   Creating boot environment <patched_20101126>.
   Creating file systems on boot environment <patched_20101126>.
   Creating <ufs> file system for </> in zone <global> on </dev/md/dsk/d0>.
   Creating <ufs> file system for </opt> in zone <global> on </dev/md/dsk/d30>.
   Creating <ufs> file system for </usr> in zone <global> on </dev/md/dsk/d40>.
   Creating <ufs> file system for </var> in zone <global> on </dev/md/dsk/d20>.
   Mounting file systems for boot environment <patched_20101126>.
   Calculating required sizes of file systems              for boot environment <patched_20101126>.
   Populating file systems on boot environment <patched_20101126>.
   Checking selection integrity.
   Integrity check OK.
   Populating contents of mount point </>.
   Populating contents of mount point </opt>.
   Populating contents of mount point </usr>.
   Populating contents of mount point </var>.
   Copying.
   Creating shared file system mount points.
   Creating compare databases for boot environment <patched_20101126>.
   Creating compare database for file system </var>.
   Creating compare database for file system </usr>.
   Creating compare database for file system </opt>.
   Creating compare database for file system </>.
   Updating compare databases on boot environment <patched_20101126>.
   Making boot environment <patched_20101126> bootable.
   Setting root slice to Solaris Volume Manager metadevice </dev/md/dsk/d0>.
   Population of boot environment <patched_20101126> successful.
   Creation of boot environment <patched_20101126> successful.

8. Activate the new boot environment

   luactivate patched_30072009

9. Reboot the machine with the 'shutdown' command

   shutdown -g 0 -i 6 -y


10.  Check that you have indeed booted off the new boot environment

11. Delete the old boot environment

   ludelete after_patch

12. Re-attach sub-mirror devices that were used in the 'after_patch' boot environment to the new boot environment 'patched_20101126', and then make the second disk bootable again

   metattach d0 d2
   metattach d20 d22
   metattach d30 d32
   metattach d40 d42
   installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0

Wait for the mirrors to finish re-syncing.  Now you're done!