Introduction

This series of articles will cover the build of a two-node cluster running Solaris Cluster 4.1 on Solaris 11 x86. A few failover resource types will be introduced, and the end setup will have highly-available zones deployed to it. I’ll also install iPlanet on one of the zones, to illustrate how to incorporate an SMF service within a zone into the cluster-managed framework, after cloning it to save time in creating a second zone.

This is all being done under VMware Fusion 5.0 on Mac OS X. As VMware Fusion does not support disk sharing, I’ll also build a third Solaris 11 node for use as an iSCSI target and Quorum Server. Later in the process, I’ll remove the Quorum Server from the mix (as when I thought about it, an iSCSI LUN can be used for this purpose). I’ve still kept the Quorum Server details in the article, however, as it’s still of interest. Solaris 11 has totally changed the way iSCSI sharing happens, too. You have to configure COMSTAR - gone is zfs set shareiscsi=on :/

To start with, I installed Solaris 11/11 onto three VMs, clusternode1 and clusternode2 (each with 1.5GB RAM), and storagenode (with 1GB RAM). Both cluster nodes have a single 20GB disk for use as rpool, and the storagenode has a 20GB disk for rpool, and two additional 10GB disks for use as iSCSI LUNs on ZFS volumes. These LUNs will be presented to our cluster, and used for failover storage - which will then be used as the zonepaths for our highly-available zones. The node details are:

• 10.1.1.70 - storageserver - iSCSI target and quorum server
• 10.1.1.71 - ha-zone-1 - Zone to be provisioned to host iPlanet
• 10.1.1.72 - ha-zone-2 - Zone to be provisioned to illustrate cloning
• 10.1.1.80 - clusternode1 - Solaris Cluster 4.1 node
• 10.1.1.90 - clusternode2 - Solaris Cluster 4.1 node

Each cluster node has four network interfaces, as follows:

• net0 - e1000g0 - Public network
• net1 - e1000g1 - Public network
• net2 - e1000g2 - vmnet2 - a private host-only network (with no other hosts on it)
• net3 - e1000g3 - vmnet4 - a private host-only network (with no other hosts on it)

net0 and net1 will be configured as an IPMP group with transitive probing. net2 and net3 will be used as private cluster interconnects. It is important that no other hosts are using the interconnect networks, otherwise the cluster installation software will detect the traffic and complain, as it could interfere with cluster communications.

Let’s start with some basic preparation …

Network Preparation

On all three servers, make sure that NTP is configured correctly. If it isn’t, the cluster is not going to work. Time synchronisation across cluster nodes is very important. Needless to say, as this is being built under VMware, suspending VMs will cause massive time-shift and will break things. Shutdown your cluster every time you finish working with it. Anyway, I digress … Configure NTP using appropriate NTP servers - I use {1,2,3}.au.pool.ntp.org as they are geographically close to me:

Above, I created a minimal /etc/inet/ntp.conf file, performed an initial time sync with ntpdate, enabled the network/ntp service and then verified that the service was enabled (with svcs -xv) and that NTP itself was really working (with ntpq -pn).

For cluster communication to work, RPC must be enabled on all nodes too - do this with svccfg by setting the config/local_only value for the network/rpc/bind service to false from its default true:

And refresh the RPC service:

Configuring RPC correctly, and ensuring it’s refreshed prior to starting the cluster software installation, is very important!

Add entries into /etc/hosts on each node, for all cluster members and the storage node. This will guard against name resolution failure:

As discussed in the introduction, each cluster node has four network interfaces, as shown in the output of dladm show-phys:

net0 and net1 should be placed into an IPMP group for resilience. I’ve already written a detailed article on setting up transitive IPMP, so read that, follow the steps outlined in it, and place net0 and net1 into an IPMP group on each cluster node.

When you’re done, you should see something along the lines of

when you run ipmpstat -t.

If these servers were physical boxes using a multipathed storage topology (e.g. multiple HBAs, redundant paths back to multiple arrays, etc.) then now would be a good time to run stmsboot -e and enable MPXIO. But they’re not. So I won’t.

Storage Server - Initial Preparation / Repository

This section will cover a few things. I’ll first create a zpool, then set up a couple of ZFS filesystems shared via NFS. One will be for the raw ISO image of the Solaris Cluster 4.1 IPS repository downloaded from Oracle, and the other will be for the repository itself. The NFS share for the ISO isn’t strictly required, but it’s handy to have available. Finally, I’ll configure the appropriate package publishers on the storage node and the cluster nodes.

Earlier I said I had two 10GB disks available to my storage server VM - format shows that these disks are available at c8t1d0 and c8t2d0. I’ll create my zpool - datapool - just using the first disk for now, I’ll add the second disk to the pool prior to creating the iSCSI LUNs.

Next, create a ZFS filesystem for the ISO image (it is my intention to use this storage server for more than this single project, hence creating these filesystems), and set an appropriate mountpoint:

Next, configure the NFS share - I make this share available to 10.1.1.80 and 10.1.1.90 only - my cluster nodes:

Using zfs set share=... does not enable the share - it will only become available via NFS once the sharenfs property is set to on. After that’s done, check /etc/dfs/sharetab, and you’ll see the NFS share in all its glory:

To make sure this is working, from clusternode1, navigate the /net filesystem and verify:

Cool. Let’s copy the Solaris Cluster 4.1 repo ISO to the storage node.

Next, I repeat the process to create another NFS shared ZFS filesystem that will house the actual repository extracted from the ISO image, following the same procedure as for /isostore:

Create a LOFI loopback device for the ISO image, and mount it (“it” being the LOFI device returned when you created it with lofiadm, in my case /dev/lofi/1) as a HSFS filesystem on a suitable mountpoint:

Create a directory under the /pkgstore filesystem to hold the repository, and transfer it with rsync (or cp -Rp, etc.):

Unmount the LOFI device, then delete it.

On the storage server, set the new ha-cluster package publisher, referencing the local filesystem location, and verify that it’s been added correctly:

Perform the same steps on both cluster nodes, however this time ensure that the NFS path (i.e. /net/storagenode/...) is used instead:

On all three servers, force a rebuild of the package index, to speed up searches and so forth:

Conclusion

This article has covered a fair bit of ground already - configuring a Solaris Cluster is not a trivial task, and a lot has changed between the last clusters I built (Solaris 10 and Sun Cluster 3.2), and the new Solaris 11 / Solaris Cluster 4.1 combination I’m building here. Plus I’m adding a bit more complexity into the mix by using iSCSI instead of fibre-attached SAN storage.

That said, I’m writing these articles knowing that once it’s all set up, it’s easy enough to understand, and exhibits the usual standards, resilience and performance you would expect from Solaris and Sun/Solaris Cluster.

The next article covers iSCSI configuration, the Quorum Server, and Solaris Cluster software installation.