Solaris Cluster 4.1 Part Four: Highly Available Containers

Introduction

The previous article covered the configuration of two resource groups, each containing a failover zpool for use as the zonepath to a highly-available zone, and a failover IP address to be assigned to each zone. The two zones were also configured and installed, and we verified that they could be booted on either node of the cluster, provided that the storage had been failed over appropriately and was available on the node where the zone was being booted.

This final part in the series will cover the incorporation of the zone boot/shutdown/failover into the cluster framework, as well as the configuration of two iPlanet resources to illustrate how Solaris Cluster can manage SMF services deployed within a highly-available Solaris zone.

Highly-Available Zones

First, install the ha-zones data service, if you haven’t done so already. I installed the full cluster package suite, so already have all data services at my disposal:

Register the SUNW.gds resource type:

This is the Generic Data Service that is utilised by SUNWsczone (HA for Solaris Containers) for deploying highly-available zones. SUNWsczone supplies three highly-available mechanisms for zone deployment - sczbt (zone boot - used to start/stop/failover zones), sczsh (zone script resource - used for deploying highly-available services within zones, with start/stop scripts to control them) and sczsmf (zone SMF resource, used for deploying highly-available services within zones, with SMF services to control them). We’ll be using both sczbt and sczsmf.

Continue reading →

Solaris Cluster 4.1 Part Three: Cluster Resources

Introduction

In my previous article, we ended up with a working cluster, with all appropriate cluster software installed. In this article, I’ll start to configure cluster resources. I want to configure two resource groups, ha-zone-1-rg and ha-zone-2-rg. Each resource group will contain a highly-available failover filesystem, a highly-available failover IP address and a highly-available Solaris Zone. I’ll illustrate the process for cloning a zone to save on installation time, as zones in Solaris 11 now use IPS and unless you have a local IPS repository, will connect to http://pkg.oracle.com to download all appropriate packages during zone installation - not something you want to repeat too many times.

A summary of the resources/resource groups I’m looking to create is as follows:

  • ha-zone-1-rg - Resource group for the first set of failover resources
  • ha-zone-1-hasp - a SUNW.HAStoragePlus resource for the first failover zpool used for the zonepath for the first failover zone, ha-zone-1
  • ha-zone-1-lh-res - a SUNW.LogicalHostname resource for the first failover zone
  • ha-zone-1-res - a SUNW.gds resource, coupled with SUNWsczone/sczbt zone boot registration to create a highly-available zone, ha-zone-1
  • ha-zone-1-http-admin-smf-res - a SUNW.gds resource, coupled with SUNWsczone/sczsmf zone SMF service registration to create a highly-available iPlanet admin server instance
  • ha-zone-1-http-instance-smf-res - a SUNW.gds resource, coupled with SUNWsczone/sczsmf zone SMF service registration to create a highly-available iPlanet instance
  • ha-zone-2-rg - Resource group for the second set of failover resources
  • ha-zone-2-hasp - a SUNW.HAStoragePlus resource for the second failover zpool used for the zonepath for the second failover zone, ha-zone-2
  • ha-zone-2-lh-res - a SUNW.LogicalHostname resource for the second failover zone
  • ha-zone-2-res - a SUNW.gds resource, coupled with SUNWsczone/sczbt boot registration to create a highly-available zone, ha-zone-2

This article will cover a lot of ground, much more so than the previous two parts. By the end of the article, you will see two HA resource groups in action, each with a failover zpool and logical hostname resource. I’ll also install the two zones, but won’t make them HA as yet - that’ll be in the next part of the series, as will the configuration of the HA SMF iPlanet resources.

As always, ensure that you read the Oracle Solaris Cluster 4.1 documentation library for full details.

Let’s make a start …

Continue reading →

Solaris Cluster 4.1 Part Two: iSCSI, Quorum Server, and Cluster Software Installation

Introduction

The previous article in this series covered the initial preparation of our two cluster nodes, and the storage server. This article follows on from this by performing more work on the storage server - configuring the iSCSI LUNs that’ll be exported to our cluster nodes as shared disk devices, as well as installing the Solaris Cluster Quorum Server software. Then we move onto the cluster nodes, and install Solaris Cluster 4.1. By the end of this article, you’ll see an operational cluster - although it won’t have any resources created just yet.

iSCSI Configuration

Before we can configure iSCSI (which now requires COMSTAR configuration in Solaris 11), the appropriate package group needs to be installed - group/feature/storage-server. Install this package group on the storage server:

This will install quite a few packages (including things like AVS, Infiniband, Samba, etc.) but is the recommended method in the Oracle documentation. In any case, it provides the packages we want: scsi-target-mode-framework and iscsi/iscsi-target - and meets any dependencies. As an aside, you can find out what package owns a file via pkg search -l <filename> or pkg search file::<filename>:

Once the packages are installed, enable the SCSI target mode framework SMF service:

At this point, I’ll add a second disk to the datapool zpool to ensure there’s plenty of capacity for ZFS volume creation:

Let’s check how much free space we have:

OK - that’ll do - 39.6GB. Next, I’ll create two ZFS volumes, one for each zone that I’ll be deploying to the cluster. Each volume will be used as a failover zpool by the cluster, and will provide storage for a single failover zone. 8GB will suffice for each volume:

ZFS volumes are datasets that represent block devices, and are treated as such. They are useful for things such as this (and swap space, dump devices, etc.).

Continue reading →

Solaris Cluster 4.1 Part One: Initial Preparation

Introduction

This series of articles will cover the build of a two-node cluster running Solaris Cluster 4.1 on Solaris 11 x86. A few failover resource types will be introduced, and the end setup will have highly-available zones deployed to it. I’ll also install iPlanet on one of the zones, to illustrate how to incorporate an SMF service within a zone into the cluster-managed framework, after cloning it to save time in creating a second zone.

This is all being done under VMware Fusion 5.0 on Mac OS X. As VMware Fusion does not support disk sharing, I’ll also build a third Solaris 11 node for use as an iSCSI target and Quorum Server. Later in the process, I’ll remove the Quorum Server from the mix (as when I thought about it, an iSCSI LUN can be used for this purpose). I’ve still kept the Quorum Server details in the article, however, as it’s still of interest. Solaris 11 has totally changed the way iSCSI sharing happens, too. You have to configure COMSTAR - gone is zfs set shareiscsi=on :/

To start with, I installed Solaris 11/11 onto three VMs, clusternode1 and clusternode2 (each with 1.5GB RAM), and storagenode (with 1GB RAM). Both cluster nodes have a single 20GB disk for use as rpool, and the storagenode has a 20GB disk for rpool, and two additional 10GB disks for use as iSCSI LUNs on ZFS volumes. These LUNs will be presented to our cluster, and used for failover storage - which will then be used as the zonepaths for our highly-available zones. The node details are:

  • 10.1.1.70 - storageserver - iSCSI target and quorum server
  • 10.1.1.71 - ha-zone-1 - Zone to be provisioned to host iPlanet
  • 10.1.1.72 - ha-zone-2 - Zone to be provisioned to illustrate cloning
  • 10.1.1.80 - clusternode1 - Solaris Cluster 4.1 node
  • 10.1.1.90 - clusternode2 - Solaris Cluster 4.1 node

Each cluster node has four network interfaces, as follows:

  • net0 - e1000g0 - Public network
  • net1 - e1000g1 - Public network
  • net2 - e1000g2 - vmnet2 - a private host-only network (with no other hosts on it)
  • net3 - e1000g3 - vmnet4 - a private host-only network (with no other hosts on it)

net0 and net1 will be configured as an IPMP group with transitive probing. net2 and net3 will be used as private cluster interconnects. It is important that no other hosts are using the interconnect networks, otherwise the cluster installation software will detect the traffic and complain, as it could interfere with cluster communications.

Let’s start with some basic preparation …

Continue reading →

Adding Storage Devices to KVM-Based Guests

Introduction

During the process of configuring a CentOS KVM guest (hostname: mercury), I wanted to add a separate disk device for the /var/www filesystem so that I could configure the guest as a kickstart server with ample storage available.

To enable this, I wanted to do two things. First, add a new disk device to the virtualised guest for configuration as a new physical volume under LVM on the guest - this would be used to host /var/www. Second, I wanted to mount the CentOS (and possibly other) ISO images to the guest’s DVD drive so that I could populate the installation source tree.

Adding a Virtualised Hard Disk

Let’s start with adding the new hard disk device. dd is the command of choice here. You can create sparse files for test purposes (i.e. the full disk allocation is not consumed upon creation), but Red Hat warns against this as there can be data integrity issues, and obvious performance issues as the space is actually allocated whilst the device is being used. Therefore, the best choice is to pre-allocate the storage. This will obviously consume the full size of the device being created, but will be robust and will perform optimally.

I’ll create a 20GB volume for /var/www, under the default location for libvirt disk images - /var/lib/libvirt/images

As you can see, I have created the image file with a meaningful name (in this case, <hostname>-<device-associated-on-guest>). This will make managing the disk images easier as more guests are created. This will take some time - go and make a cup of tea.

Once done, dd should produce output along the following lines:

If your Linux distro comes with fallocate, you can do this in a second or too instead (depending on filesystem support - I use ext4 which does), rather than waiting on dd:

It’s worth noting that the original image created at guest installation (i.e. the one containing the OS) is available as /dev/sda (despite it being referenced as an IDE device in the configuration with a target of /dev/hda). So, the actual guest configuration shows an IDE drive:

However, within the guest, it’s detected as a SCSI drive at /dev/sda:

I’ll use virtio for any additional drives added, starting at /dev/vda. I could have added a SCSI HBA (via the virtio-scsi controller model) but there is no need to do this (and over-complicate things) for my requirements.

Continue reading →

How to Cluster Oracle Weblogic 12c via the Command Line

In this article, I will show you how to create a two-node Weblogic 12c cluster using only the command line. Oracle Weblogic (formerly BEA Weblogic) is one of the most resilient, reliable and high-performance J2EE application servers that I’ve worked with. I’ve used it to host both custom applications, as well as commercial applications that required an enterprise-grade J2EE container to serve them.

The lab topology will be as follows:

Weblogic Server

Hostname

IP Address

Port

Cluster Name
AdminServer dolan 172.16.18.169 7001 N/A
test_managed_server_1 dolan 172.16.18.169 7002 TestCluster
test_managed_server_2 gooby 172.16.18.172 7002 TestCluster

As you can see, AdminServer only runs on a single node. Once the managed servers have been configured and started for the first time by way of the administration server, they can be restarted independently - thus the loss of the administration server does not impact the general running of the cluster. It will, however, prevent administration of that cluster (configuration, deployments, etc.) until such time as the administration server is back online. There are ways around this (bind the administration to a failover VIP provided by keepalived or similar) but for all but the most demanding usage, a single administration server will suffice.

Both dolan and gooby run CentOS 6.3 x86_64. All steps should be run on both nodes unless otherwise noted.

Continue reading →

An Introduction to Solaris 11 Zones

Solaris Zones (or Containers) were first introduced in Solaris 10. I wrote about them a while back in How to Get Started with Solaris Containers, but a lot has changed in Solaris 11. Solaris Zones provide an easy way to either provide a sparse chroot-like environment so that applications can run in an environment that will not be detrimental to global system resources, or a full branded environment (running a Solaris 10 branded zone, for example). Delegated administration can be configured so that a zone can be managed by someone other than the global sysadmin. Zones provide an excellent way to split a system into several logical units, each with their own filesystem resources, system resources and management. Most of what I wrote about previously is still very pertinent, but Solaris 11 has built upon zone technology, placing it at its very core.

In Solaris 10, the default IP type for zones was shared, which meant that the zone shared the IP stack with the global zone. Within a zone on Solaris 10, an administrator was unable to configure network settings, unless exclusive IP was used, in which case the zone would be bound to a physical NIC in the global zone, and that NIC would only be available for exclusive use by that zone. With Solaris 11, and virtual networking, all zones can be created with an exclusive IP type. A Virtual NIC (VNIC) is created for each zone, over some physical NIC on the global zone. This network virtualisation allows each zone to maintain its own TCP/IP stack, and the zone administrator can change the zone’s network configuration from within the zone itself. A new anet interface type has been introduced within zonecfg to handle this.

Solaris 11 zones are now provisioned using the new Image Packaging System (IPS) and in a default configuration, packages will be installed from the repository configured (http://pkg.oracle.com, for example) in the global zone. It would make sense to have a local repository if you were rolling out large numbers of systems or zones, but for our testing purposes, downloading a couple of hundred megabytes of packages is no big issue.

This article will walk through the creation of a simple Solaris 11 zone, and introduce a method of installing zones without operator intervention using System Profiles.

Continue reading →

Amazon Web Services: How to Use ec2-api-tools to Manage EC2 Instances

Amazon Web Services provide a free usage tier to allow new users to trial many aspects of the offering for 12 months without charge. There are many benefits to this tier, and it is ideally suited to small-scale deployments or for just learning about AWS and EC2. At the time of writing, among the benefits of this tier are 750 hours of compute time for a t1.micro instance, along with 30GB of elastic block store (EBS) storage. So - I can run up 750 hours of CPU time using as many t1.micro instances as I want, as long as I stay within various other confines imposed by the tier.

30GB free EBS storage doesn’t go far if you use EBS-backed instances. EBS provides persistent storage, and with EBS-backed instances, the ability to shutdown and start up your instances. The alternative to the elastic block store is the instance store. Instance store-backed instances are launched, and must be left running. The storage is ephemeral and all changes will be lost (well - the entire instance is lost) when the instance is terminated. Note the terminology - terminated means just that - that the instance will be halted, and marked for deletion. The only way to have persistence is to use EBS-backed AMIs, which is obviously my preference.

This article will cover the provisioning of instances and volumes via the command line. For this, the Amazon EC2 API Tools can be used. These tools are Java-based, so an appropriate JDK must be used.

Essentially, the EC2 tools interact with the Amazon EC2 API using X.509 certificate-based authentication, and provide functionality much the same as the web-based EC2 Management Console. This allows for quick interaction with instances, automated monitoring, automation ability, and so on.

The article assumes that you have already signed up for Amazon Web Services, familiarised yourself with the concepts and functionality of the EC2 Management Console, and have generated and downloaded the following:

  • X.509 Certificate Pair (Public Certificate, Private Key)
  • SSH Private Key

You can create your X.509 certificate pair and download it via the AWS Management Console -> Your account name in menu bar -> Security Credentials. You can generate your SSH Private Key via the EC2 Management Console -> Network & Security -> Key Pairs. The SSH Public Key will automatically be installed on any instances provisioned so that you can log in.

Continue reading →

Shells: How to Respond to a Single Keystroke Within a Script

If you are using a newer Bourne-compatible shell such as (pd)ksh or bash, and have a modern terminal, you can take advantage of the -icanon stty mode to good effect. This mode allows your scripts to respond to a single keystroke rather than requiring the user to hit “return” after entering a character. This is useful for the familiar “Press any key to continue … ” type messages, but can also be used to provide control of programs.

Ensure that you restore your old stty settings before exiting. This is probably best achieved by using traps. The following script demostrates these concepts well, and gives the foundations of a Minesweeper-style grid, cursors, and cell-marking.

AWS: Ruby on Rails Deployment Part 2: Ruby, RubyGems, Rails, Thin and Nginx

The previous article in this series has left us with a minimally-configured Nginx installation running on an EBS-backed Ubuntu EC2 instance.

This article will pick up where we left off. The latest versions of Ruby and RubyGems will be downloaded and installed. Then the Rails and Thin gems will be installed. Nginx will then have the final configuration changes applied to enable it to proxy through to the Thin workers. Thin is a lean Ruby-based web-server that has been designed to replace Mongrel (which was the standard Ruby web server until development ceased), and uses various components lifted from Mongrel (e.g. the parser - giving us the same (or better?) speed and security as Mongrel).

It’s been a while since I deployed Ruby on Rails - and that was using Mongrel - so let’s see how Thin matches up.

Continue reading →