Introduction

In my previous article, we ended up with a working cluster, with all appropriate cluster software installed. In this article, I’ll start to configure cluster resources. I want to configure two resource groups, ha-zone-1-rg and ha-zone-2-rg. Each resource group will contain a highly-available failover filesystem, a highly-available failover IP address and a highly-available Solaris Zone. I’ll illustrate the process for cloning a zone to save on installation time, as zones in Solaris 11 now use IPS and unless you have a local IPS repository, will connect to http://pkg.oracle.com to download all appropriate packages during zone installation - not something you want to repeat too many times.

A summary of the resources/resource groups I’m looking to create is as follows:

• ha-zone-1-rg - Resource group for the first set of failover resources
• ha-zone-1-hasp - a SUNW.HAStoragePlus resource for the first failover zpool used for the zonepath for the first failover zone, ha-zone-1
• ha-zone-1-lh-res - a SUNW.LogicalHostname resource for the first failover zone
• ha-zone-1-res - a SUNW.gds resource, coupled with SUNWsczone/sczbt zone boot registration to create a highly-available zone, ha-zone-1
• ha-zone-1-http-admin-smf-res - a SUNW.gds resource, coupled with SUNWsczone/sczsmf zone SMF service registration to create a highly-available iPlanet admin server instance
• ha-zone-1-http-instance-smf-res - a SUNW.gds resource, coupled with SUNWsczone/sczsmf zone SMF service registration to create a highly-available iPlanet instance
• ha-zone-2-rg - Resource group for the second set of failover resources
• ha-zone-2-hasp - a SUNW.HAStoragePlus resource for the second failover zpool used for the zonepath for the second failover zone, ha-zone-2
• ha-zone-2-lh-res - a SUNW.LogicalHostname resource for the second failover zone
• ha-zone-2-res - a SUNW.gds resource, coupled with SUNWsczone/sczbt boot registration to create a highly-available zone, ha-zone-2

This article will cover a lot of ground, much more so than the previous two parts. By the end of the article, you will see two HA resource groups in action, each with a failover zpool and logical hostname resource. I’ll also install the two zones, but won’t make them HA as yet - that’ll be in the next part of the series, as will the configuration of the HA SMF iPlanet resources.

As always, ensure that you read the Oracle Solaris Cluster 4.1 documentation library for full details.

Let’s make a start …

Preparation

Before we start configuring resource groups and their resources, it’s worth verifying the cluster and ensuring that the core components are healthy, and unmonitoring any local disks. We don’t need the cluster monitoring disk paths to local disk devices (which only have a single path) - only the shared devices upon which we’ll be deploying clustered services (which have a path from each cluster node).

Start by verifying that both nodes have correctly jointed the cluster, with the clnode status command:

Next, verify that both interconnects are healthy:

Finally, check that the quorum is healthy:

If those three basic checks look good, you can proceed with unmonitoring the local disk devices. First, obtain a list of the current cluster DID devices with the cldevice status command:

Here, we can see that d1 and d2 are our shared disk devices, and d4, d5, d7 and d8 are our node-specific local devices. This can be verified with cldevice list -v:

You can see two additional DID devices here, d3 and d6 (c7t0d0 on each node) which are the CD-ROM drives. These devices are automatically not eligible for monitoring, hence not appearing in the output of cldevice status.

Next, for each local device, use the cldevice unmonitor command to disable path monitoring:

If you do try to unmonitor one of the CD-ROM paths, you’ll receive an error:

With the local disk paths unmonitored, verify once again with cldevice status:

Next, we enable automatic rebooting of a node when all monitored disk paths fail - as this would indicate that the node has lost access to the shared storage (now that we are not monitoring the local disk paths). Using “+” for the final argument means to apply the configuration to “all nodes”. I could have just issued the command twice, specifying each node as the final argument, or even a list of nodes. The Solaris Cluster command syntax is very rich - read the documentation and manual pages for full details. Many other Solaris Cluster commands accept the “+” syntax to represent “all” of a type of object.

And verify:

At this point, verify what type of fencing is in use too - we want prefer3 (to specify the SCSI-3 protocol for device fencing for all devices) - which is the default in Solaris Cluster 4.1:

Each device then uses default_fencing type global (by default), to inherit the fencing type from the cluster-wide global_fencing attribute:

Excellent - we are now ready to begin provisioning our clustered resources. Our devices are correctly monitored, fencing defaults look sane, and the node is protected by rebooting if all paths to the shared storage fail.

Clustered Resources

In the previous article, I already created an fdisk partition on each of our iSCSI-backed shared disk devices, and applied an EFI label to each. They are now ready for the creation of our failover zpools. But first, let’s create the resource groups outlined in the introduction - each resource group will hold all of the resources appropriate for the operation of each highly-available zone: ha-zone-1-rg for ha-zone-1 and ha-zone-2-rg for ha-zone-2.

Resource groups are created with the clresourcegroup create command. It’s worth noting here that there are also abbreviated forms of each command. For example, I could also use the clrg command - these are simply hard links:

You will find that you’ll end up using the abbreviated forms more as you work with Solaris Cluster, to save yourself some typing. However, for documentation, you should always use the full command name.

Create the two resource groups:

Verify that they have been created with clresourcegroup status:

Both resource groups will be in an Unmanaged state at this time. An interesting point to note here is that clusternode2 appears before clusternode1 in the listing. This is because the second node of a two-node cluster is configured and rebooted first (via scinstall), thus obtaining a nodeid of 1:

clusternode1 is configured and rebooted into the cluster second, and obtains a nodeid of 2. Some admins choose to run scinstall from the second node of a cluster during build for this reason. I don’t bother with this, as the nodeids are only used internally.

We are now ready to create the failover zpool resources. Before doing anything on a cluster - verify, verify, verify! Double checking the output of cldevice status, we can see that d1 and d2 are the DID devices for the shared disks:

Verifying the full device paths:

And checking them against the LUN mapping on the storage server:

So - d1 is definitely for use by ha-zone-1 and d2 is definitely for use by ha-zone-2 - perfect.

Looking at the VTOC, we see that slice 0 is the slice for use on our EFI labelled disks:

From ONE NODE in the cluster, create the zpools on these slices:

Next, register the SUNW.HAStoragePlus resource type - which is the resource type we’ll be using to manage our failover zpools. From this point forward, unless otherwise noted, you only need to execute the commands on one node of the cluster:

Create the SUNW.HAStoragePlus resources to manage the storage:

We can now bring the two resource groups online (here, on clusternode1), and use the -M flag to set the resources to a managed state:

And verify:

On clusternode1, a zpool list will show the clustered zpools imported and available:

On clusternode2, the zpools will not be available:

We can now switch both resource groups over (i.e. fail them over) to clusternode2:

Note the use of the “+” argument again, to fail all resource groups over, rather than specifying each one.

Verify that the zpools are now available on clusternode2:

Excellent. So we know where we are, let’s fail them back to clusternode1 before we continue:

Next, add entries to /etc/hosts on BOTH NODES for the IP addresses of the highly available zones we are about to configure:

Whilst you can do things the “normal” way via zonecfg and specify IP addresses during zone configuration, these will only be protected via IPMP. If both interfaces fail, a failover will not be triggered. For this failover to occur, we need to use a LogicalHostname resource, and have that provide the zone’s IP (all will become clear during zone boot registration later in the process).

Add the LogicalHostname resources, referencing the hostnames (-h <hostname>) for the zones that were just added to /etc/hosts:

Check the status of all resources added so far:

At this point, make sure you can ping the logical hostnames:

Perform a failover or two (clrg switch -n <node> +) and ensure that you can still ping the logical hostnames. Once you’re happy, make sure everything is failed back to clusternode1.

Zone Creation, Installation and Boot Verification

Run through zonecfg on BOTH nodes for the first zone - ha-zone-1. Make sure you specify the zonepath on the shared storage. By default, the zpools are mounted at /<zpool-name>, so our zonepaths will be /<zpool-name>/<zone-name> - for example /ha-zone-1-pool/ha-zone-1 for the first zone. The ZFS filesystem will be created in the zpool during zone installation. So, on both nodes, run through zonecfg as follows:

A few things to note. Make sure you set autoboot to false as the zone boot will be managed by the cluster. Ensure you set ip-type to shared, otherwise it will not work with the LogicalHostname resource type created early. Finally, the osc-ha-zone boolean attribute must be created, and set to true.

On both nodes, verify that the zone has been configured:

Time to install the zone. Only do this on clusternode1 (or wherever the ha-zone-1-rg resource group is active) - verify that the resource group is active first:

Install the zone:

On clusternode1, verify that the zone is now installed:

Next, on BOTH NODES, configure the second zone - ha-zone-2:

Once the configuration is complete, clone the zone. Only do this on clusternode1:

And verify. At this point you should have two zones in an installed state:

Boot the first zone, connect to the zone console, and run through the system configuration screens (see my article on Solaris 11 Zones for a way to avoid this by using System Profiles):

Once done, shutdown and detach the zone:

Repeat for the second zone:

Once both zones are detached, they should appear in a configured state on clusternode1:

Next, as we are building failover zones (this step is not needed for multi-master zones), we need to set the current boot environment UUID on clusternode2 to the same UUID as the current boot environment on clusternode1. Grab the UUID for both boot environments with beadm:

You can see here that the current active environment (denoted by a “N” in the third field for “active now”) on clusternode1 has a UUID ending in eaccba35b384. We’ll replace the UUID on the second node (the one ending in 97b08ee9ac2c) with this value. On clusternode2, ascertain the correct ZFS filesystem to apply the UUID to:

Then, set the org.opensolaris.libbe property to the same UUID value present on clusternode1:

Verify that it’s been updated correctly:

OK - now fail the resource groups over to clusternode2:

Attach and boot both zones on clusternode2, and make sure they work. They should boot into the same fully configured state as you left them on clusternode1:

If all is well, shut the zones back down and detach them. I’ll leave the resource groups online on clusternode2 for now.

Conclusion

We now have a cluster containing two resource groups, and each resource group contains a failover filesystem resource and a failover logical hostname resource. I’ve also created the two zones that will be made highly-available in the next article, and verified that they can be booted on both nodes of the cluster (provided the resource group is available on the node the zone is booted on).

The next article will focus on making the two zones highly-available, configuring a couple of HA iPlanet SMF resources, and tying up any loose ends.