Introduction
The previous article in this series covered the initial preparation of our two cluster nodes, and the storage server. This article follows on from this by performing more work on the storage server - configuring the iSCSI LUNs that’ll be exported to our cluster nodes as shared disk devices, as well as installing the Solaris Cluster Quorum Server software. Then we move onto the cluster nodes, and install Solaris Cluster 4.1. By the end of this article, you’ll see an operational cluster - although it won’t have any resources created just yet.
iSCSI Configuration
Before we can configure iSCSI (which now requires COMSTAR configuration in Solaris 11), the appropriate package group needs to be installed - group/feature/storage-server. Install this package group on the storage server:
|
1 |
# pkg install group/feature/storage-server |
This will install quite a few packages (including things like AVS, Infiniband, Samba, etc.) but is the recommended method in the Oracle documentation. In any case, it provides the packages we want: scsi-target-mode-framework and iscsi/iscsi-target - and meets any dependencies. As an aside, you can find out what package owns a file via pkg search -l <filename> or pkg search file::<filename>:
|
1 2 3 4 5 6 7 8 |
# pkg search -l /usr/sbin/stmfadm INDEX ACTION VALUE PACKAGE path file usr/sbin/stmfadm pkg:/system/storage/scsi-target-mode-framework@0.5.11-0.175.1.0.0.24.2 # pkg search file::stmfadm INDEX ACTION VALUE PACKAGE basename file usr/sbin/stmfadm pkg:/system/storage/scsi-target-mode-framework@0.5.11-0.175.1.0.0.24.2 |
Once the packages are installed, enable the SCSI target mode framework SMF service:
|
1 2 3 4 |
# svcadm enable system/stmf # svcs stmf STATE STIME FMRI online 21:28:18 svc:/system/stmf:default |
At this point, I’ll add a second disk to the datapool zpool to ensure there’s plenty of capacity for ZFS volume creation:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# zpool add datapool c8t2d0 # zpool status datapool pool: datapool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM datapool ONLINE 0 0 0 c8t1d0 ONLINE 0 0 0 c8t2d0 ONLINE 0 0 0 errors: No known data errors |
Let’s check how much free space we have:
|
1 2 3 |
# zpool list datapool NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT datapool 39.8G 142M 39.6G 0% 1.00x ONLINE - |
OK - that’ll do - 39.6GB. Next, I’ll create two ZFS volumes, one for each zone that I’ll be deploying to the cluster. Each volume will be used as a failover zpool by the cluster, and will provide storage for a single failover zone. 8GB will suffice for each volume:
|
1 2 |
# zfs create -V 8G datapool/ha-zone-1 # zfs create -V 8G datapool/ha-zone-2 |
ZFS volumes are datasets that represent block devices, and are treated as such. They are useful for things such as this (and swap space, dump devices, etc.).
Next, use the stmfadm command to create two LUNs, one for each of our ZFS volumes:
|
1 2 3 4 |
# stmfadm create-lu /dev/zvol/rdsk/datapool/ha-zone-1 Logical unit created: 600144F06F9201000000519DE7880004 # stmfadm create-lu /dev/zvol/rdsk/datapool/ha-zone-2 Logical unit created: 600144F06F9201000000519DE78A0005 |
And verify their creation:
|
1 2 3 |
# stmfadm list-lu LU Name: 600144F06F9201000000519DE7880004 LU Name: 600144F06F9201000000519DE78A0005 |
Whilst we can use CHAP for security of our iSCSI LUNs, on my test network I can just lock down access to the initiator IQNs. First, on each cluster node, enable the iSCSI initiator service:
|
1 2 |
# svcadm enable network/iscsi/initiator # svcs network/iscsi/inititor |
Get the iSCSI initiator IQNs from both cluster nodes by issuing a command such as the following:
|
1 2 |
node1# iscsiadm list initiator-node | grep 'node.*name' Initiator node name: iqn.1986-03.com.sun:01:e00000000000.deadb33f |
|
1 2 |
node2# iscsiadm list initiator-node | grep 'node.*name' Initiator node name: iqn.1986-03.com.sun:01:e00000000000.c0ffee11 |
I’ve obfuscated the IQNs here… Back on the storage node, create a host group:
|
1 |
# stmfadm create-hg failover-1-group |
and add both initiator IQNs to it:
|
1 2 |
# stmfadm add-hg-member -g failover-1-group iqn.1986-03.com.sun:01:e00000000000.deadb33f # stmfadm add-hg-member -g failover-1-group iqn.1986-03.com.sun:01:e00000000000.c0ffee11 |
Verify the hostgroup:
|
1 2 3 4 |
# stmfadm list-hg -v failover-1-group Host Group: failover-1-group Member: iqn.1986-03.com.sun:01:e00000000000.deadb33f Member: iqn.1986-03.com.sun:01:e00000000000.c0ffee11 |
Then create a view to each of our LUNs, locked down to the new failover-1-group hostgroup:
|
1 2 |
# stmfadm add-view -h failover-1-group 600144F06F9201000000519DE7880004 # stmfadm add-view -h failover-1-group 600144F06F9201000000519DE78A0005 |
And verify the views:
|
1 2 3 4 5 6 7 8 9 10 |
# stmfadm list-view -l 600144F06F9201000000519DE7880004 View Entry: 0 Host group : failover-1-group Target Group : All LUN : Auto # stmfadm list-view -l 600144F06F9201000000519DE78A0005 View Entry: 0 Host group : failover-1-group Target Group : All LUN : Auto |
Next, enable the iSCSI target service:
|
1 2 |
# svcadm enable network/iscsi/target # svcs network/iscsi/target |
And then create the iSCSI target:
|
1 2 |
# itadm create-target Target iqn.1986-03.com.sun:02:b2dd308d-ddc9-c5be-9b44-a4db24281c4b successfully created |
Again, I’ve obfuscated my actual IQN a little. Verify that the target has been created:
|
1 2 3 4 5 6 7 8 |
# itadm list-target -v TARGET NAME STATE SESSIONS iqn.1986-03.com.sun:02:b2dd308d-ddc9-c5be-9b44-a4db24281c4b online 0 alias: - auth: none (defaults) targetchapuser: - targetchapsecret: unset tpg-tags: default |
On each cluster node, configure target discovery. I used sendtargets rather than static configuration, but either is easy to configure. Substitute the appropriate IP address for your iSCSI target:
|
1 2 |
# iscsiadm add discovery-address 10.1.1.70 # iscsiadm modify discovery --sendtargets enable |
You should now be able to see each of the LUNs when you issue iscsiadm list target -S on the cluster nodes:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# iscsiadm list target -S Target: iqn.1986-03.com.sun:02:b2dd308d-ddc9-c5be-9b44-a4db24281c4b Alias: - TPGT: 1 ISID: 4000002a0000 Connections: 1 LUN: 1 Vendor: SUN Product: COMSTAR OS Device Name: /dev/rdsk/c0t600144F06F9201000000519DE78A0005d0s2 LUN: 0 Vendor: SUN Product: COMSTAR OS Device Name: /dev/rdsk/c0t600144F06F9201000000519DE7880004d0s2 |
Keep in mind which LUN you intend to use for which zpool - as you’ll give yourself an administrative headache if you start mismatching them back to the ZFS volumes on the storage server. Issue the following command on the storage server to get a map:
|
1 2 3 4 5 |
# stmfadm list-lu -v | egrep 'LU|Alias' LU Name: 600144F06F9201000000519DE7880004 Alias : /dev/zvol/rdsk/datapool/ha-zone-1 LU Name: 600144F06F9201000000519DE78A0005 Alias : /dev/zvol/rdsk/datapool/ha-zone-2 |
Next, from only ONE of the cluster nodes, create an fdisk partition on each of the disks (as we’re using Solaris on x86). Note that I append p0 to the device path for partition 0:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# fdisk /dev/rdsk/c0t600144F06F9201000000519DE7880004d0p0 No fdisk table exists. The default partition for the disk is: a 100% "SOLARIS System" partition Type "y" to accept the default partition, otherwise type "n" to edit the partition table. y # fdisk /dev/rdsk/c0t600144F06F9201000000519DE78A0005d0p0 No fdisk table exists. The default partition for the disk is: a 100% "SOLARIS System" partition Type "y" to accept the default partition, otherwise type "n" to edit the partition table. y |
Again, from only ONE of the cluster nodes, add an EFI label to each of the disks. Use format -e to do this, otherwise a standard SMI label will be applied:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
# format -e Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0t600144F06F9201000000519DE78A0005d0 <SUN-COMSTAR-1.0 cyl 4093 alt 2 hd 128 sec 32> /scsi_vhci/disk@g600144f06f9201000000519de78a0005 1. c0t600144F06F9201000000519DE7880004d0 <SUN-COMSTAR-1.0 cyl 4093 alt 2 hd 128 sec 32> /scsi_vhci/disk@g600144f06f9201000000519de7880004 2. c8t0d0 <VMware,-VMware Virtual S-1.0-16.00GB> /pci@0,0/pci15ad,1976@10/sd@0,0 3. c8t1d0 <VMware,-VMware Virtual S-1.0 cyl 2608 alt 2 hd 255 sec 63> /pci@0,0/pci15ad,1976@10/sd@1,0 Specify disk (enter its number): 0 selecting c0t600144F06F9201000000519DE78A0005d0 [disk formatted] FORMAT MENU: disk - select a disk type - select (define) a disk type partition - select (define) a partition table current - describe the current disk format - format and analyze the disk fdisk - run the fdisk program repair - repair a defective sector label - write label to the disk analyze - surface analysis defect - defect list management backup - search for backup labels verify - read and display labels save - save new disk/partition definitions inquiry - show disk ID scsi - independent SCSI mode selects cache - enable, disable or query SCSI disk cache volname - set 8-character volume name !<cmd> - execute <cmd>, then return quit format> la [0] SMI Label [1] EFI Label Specify Label type[0]: 1 Warning: This disk has an SMI label. Changing to EFI label will erase all current partitions. Continue? y format> di AVAILABLE DISK SELECTIONS: 0. c0t600144F06F9201000000519DE78A0005d0 <SUN-COMSTAR-1.0-8.00GB> /scsi_vhci/disk@g600144f06f9201000000519de78a0005 1. c0t600144F06F9201000000519DE7880004d0 <SUN-COMSTAR-1.0 cyl 4093 alt 2 hd 128 sec 32> /scsi_vhci/disk@g600144f06f9201000000519de7880004 2. c8t0d0 <VMware,-VMware Virtual S-1.0-16.00GB> /pci@0,0/pci15ad,1976@10/sd@0,0 3. c8t1d0 <VMware,-VMware Virtual S-1.0 cyl 2608 alt 2 hd 255 sec 63> /pci@0,0/pci15ad,1976@10/sd@1,0 Specify disk (enter its number)[0]: 1 selecting c0t600144F06F9201000000519DE7880004d0 [disk formatted] format> la [0] SMI Label [1] EFI Label Specify Label type[0]: 1 Warning: This disk has an SMI label. Changing to EFI label will erase all current partitions. Continue? y format> q |
Verify the VTOCs with prtvtoc:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
# prtvtoc /dev/rdsk/c0t600144F06F9201000000519DE7880004d0s0 * /dev/rdsk/c0t600144F06F9201000000519DE7880004d0s0 partition map * * Dimensions: * 512 bytes/sector * 16777216 sectors * 16777149 accessible sectors * * Flags: * 1: unmountable * 10: read-only * * First Sector Last * Partition Tag Flags Sector Count Sector Mount Directory 0 4 00 34 16760765 16760798 8 11 00 16760799 16384 16777182 # prtvtoc /dev/rdsk/c0t600144F06F9201000000519DE78A0005d0s0 * /dev/rdsk/c0t600144F06F9201000000519DE78A0005d0s0 partition map * * Dimensions: * 512 bytes/sector * 16777216 sectors * 16777149 accessible sectors * * Flags: * 1: unmountable * 10: read-only * * First Sector Last * Partition Tag Flags Sector Count Sector Mount Directory 0 4 00 34 16760765 16760798 8 11 00 16760799 16384 16777182 |
You may need to run devfsadm on the “other” cluster node so that the new fdisk partition is recognised and the new VTOC is available, but the cluster nodes will receive reconfiguration reboots during cluster software installation and prior to any use being made of these devices anyway. At this point, you should see that you now have 2 sessions logged in to the iSCSI target (issue this on the storage server):
|
1 2 3 4 5 6 7 8 |
# itadm list-target -v TARGET NAME STATE SESSIONS iqn.1986-03.com.sun:02:b2dd308d-ddc9-c5be-9b44-a4db24281c4b online 2 alias: - auth: none (defaults) targetchapuser: - targetchapsecret: unset tpg-tags: default |
At this point, the storage configuration is complete. The two LUNs are presented to the two cluster nodes over iSCSI and the appropriate partitioning and labelling has occurred. We can now make use of these devices for highly-available failover storage once the cluster software has been installed.
Quorum Server Installation
The quorum server installation is very straight forward. First, verify that the ha-cluster publisher is configured:
|
1 2 3 4 |
# pkg publisher PUBLISHER TYPE STATUS P LOCATION solaris origin online F http://pkg.oracle.com/solaris/release/ ha-cluster origin online F file:///pkgstore/ha-cluster-4.1/repo/ |
Next, install the ha-cluster-quorum-server-full package:
|
1 |
# pkg install ha-cluster-quorum-server-full |
The default quorum server configuration is fine for our needs:
|
1 2 |
# grep '^[^#]' /etc/scqsd/scqsd.conf /usr/cluster/lib/sc/scqsd -d /var/scqsd -p 9000 |
So, enable the cluster/quorumserver SMF service:
|
1 2 |
# svcadm enable cluster/quorumserver # svcs cluster/quorumserver |
And verify that it’s working:
|
1 2 3 4 5 6 |
# /usr/cluster/bin/clqs show === Quorum Server on port 9000 === Disabled False Quorum server on port "9000" is not configured in any cluster. |
This output will change to show node reservations once we connect the cluster to it.
The quorum server is done!
Cluster Software Installation
Next, the full cluster software stack will be installed on both cluster nodes. Once you’ve checked with pkg publisher that the ha-cluster repository is available, install the package set with the following command:
|
1 |
# pkg install ha-cluster-full |
Verify that you have two spare interfaces for use as cluster interconnects:
|
1 2 3 4 5 6 |
# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net1 Ethernet up 1000 full e1000g1 net2 Ethernet unknown 0 unknown e1000g2 net3 Ethernet unknown 0 unknown e1000g3 net0 Ethernet up 1000 full e1000g0 |
OK - from the above I can see that net2 and net3 aren’t being used, as planned. I’ll use those.
Add /usr/cluster/bin to your PATH at this point, by editing your shell’s initialisation file(s) as appropriate. I’ll edit and source my ~/.profile:
|
1 2 3 4 |
# vi ~/.profile ... export PATH=$PATH:/usr/cluster/bin # . ~/.profile |
From clusternode1, install the new cluster via scinstall. This is an interactive program that will ask you simple and easy questions about your cluster name, node names, cluster interconnect configuration, and so on. Major points to note here are that I chose a Custom installation type so I could have fine control over all aspects of the installation, gave the cluster a name of spiffy-cluster, added clusternode2 as an additional cluster node, chose net2 and net3 for my cluster interconnects (and allowed scinstall to use auto-discovery to detect and configure the interfaces on the second node) and disabled automatic quorum device selection (as I want to use a Quorum Server). Since late in the Cluster 3.x series, a separate slice dedicated to the /.globaldevices filesystem is no longer required. Now, a LOFI-backed filesystem is created by default instead. Here is the output of my session - a lot of the time, as you can see, scinstall correctly assumes sensible defaults.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 |
# scinstall *** Main Menu *** Please select from one of the following (*) options: * 1) Create a new cluster or add a cluster node 2) Upgrade this cluster node 3) Manage a dual-partition upgrade * 4) Print release information for this cluster node * ?) Help with menu options * q) Quit Option: 1 *** New Cluster and Cluster Node Menu *** Please select from any one of the following options: 1) Create a new cluster 2) Create just the first node of a new cluster on this machine 3) Add this machine as a node in an existing cluster ?) Help with menu options q) Return to the Main Menu Option: 1 *** Create a New Cluster *** This option creates and configures a new cluster. Press Control-D at any time to return to the Main Menu. Do you want to continue (yes/no) [yes]? yes Checking the value of property "local_only" of service svc:/network/rpc/bind ... Property "local_only" of service svc:/network/rpc/bind is already correctly set to "false" on this node. Press Enter to continue: >>> Typical or Custom Mode <<< This tool supports two modes of operation, Typical mode and Custom mode. For most clusters, you can use Typical mode. However, you might need to select the Custom mode option if not all of the Typical mode defaults can be applied to your cluster. For more information about the differences between Typical and Custom modes, select the Help option from the menu. Please select from one of the following options: 1) Typical 2) Custom ?) Help q) Return to the Main Menu Option [1]: 2 >>> Cluster Name <<< Each cluster has a name assigned to it. The name can be made up of any characters other than whitespace. Each cluster name should be unique within the namespace of your enterprise. What is the name of the cluster you want to establish? spiffy-cluster >>> Cluster Nodes <<< This Oracle Solaris Cluster release supports a total of up to 16 nodes. List the names of the other nodes planned for the initial cluster configuration. List one node name per line. When finished, type Control-D: Node name (Control-D to finish): clusternode2 Node name (Control-D to finish): ^D This is the complete list of nodes: clusternode1 clusternode2 Is it correct (yes/no) [yes]? yes Attempting to contact "clusternode2" ... done Checking whether NWAM is enabled on local node ... Searching for a remote configuration method ... done The Oracle Solaris Cluster framework is able to complete the configuration process without remote shell access. Checking whether NWAM is enabled on all nodes ... >>> Authenticating Requests to Add Nodes <<< Once the first node establishes itself as a single node cluster, other nodes attempting to add themselves to the cluster configuration must be found on the list of nodes you just provided. You can modify this list by using claccess(1CL) or other tools once the cluster has been established. By default, nodes are not securely authenticated as they attempt to add themselves to the cluster configuration. This is generally considered adequate, since nodes which are not physically connected to the private cluster interconnect will never be able to actually join the cluster. However, DES authentication is available. If DES authentication is selected, you must configure all necessary encryption keys before any node will be allowed to join the cluster (see keyserv(1M), publickey(4)). Do you need to use DES authentication (yes/no) [no]? no >>> Minimum Number of Private Networks <<< Each cluster is typically configured with at least two private networks. Configuring a cluster with just one private interconnect provides less availability and will require the cluster to spend more time in automatic recovery if that private interconnect fails. Should this cluster use at least two private networks (yes/no) [yes]? >>> Point-to-Point Cables <<< The two nodes of a two-node cluster may use a directly-connected interconnect. That is, no cluster switches are configured. However, when there are greater than two nodes, this interactive form of scinstall assumes that there will be exactly one switch for each private network. Does this two-node cluster use switches (yes/no) [yes]? >>> Cluster Switches <<< All cluster transport adapters in this cluster must be cabled to a "switch". And, each adapter on a given node must be cabled to a different switch. Interactive scinstall requires that you identify one switch for each private network in the cluster. What is the name of the first switch in the cluster [switch1]? What is the name of the second switch in the cluster [switch2]? >>> Cluster Transport Adapters and Cables <<< Transport adapters are the adapters that attach to the private cluster interconnect. Select the first cluster transport adapter: 1) net2 2) net3 3) Other Option: 1 Adapter "net2" is an Ethernet adapter. Searching for any unexpected network traffic on "net2" ... done Verification completed. No traffic was detected over a 10 second sample period. The "dlpi" transport type will be set for this cluster. For node "clusternode1", Name of the switch to which "net2" is connected [switch1]? Each adapter is cabled to a particular port on a switch. And, each port is assigned a name. You can explicitly assign a name to each port. Or, for Ethernet and Infiniband switches, you can choose to allow scinstall to assign a default name for you. The default port name assignment sets the name to the node number of the node hosting the transport adapter at the other end of the cable. For node "clusternode1", Use the default port name for the "net2" connection (yes/no) [yes]? Select the second cluster transport adapter: 1) net2 2) net3 3) Other Option: 2 Adapter "net3" is an Ethernet adapter. Searching for any unexpected network traffic on "net3" ... done Verification completed. No traffic was detected over a 10 second sample period. The "dlpi" transport type will be set for this cluster. For node "clusternode1", Name of the switch to which "net3" is connected [switch2]? For node "clusternode1", Use the default port name for the "net3" connection (yes/no) [yes]? For all other nodes, Autodiscovery is the best method for configuring the cluster transport. However, you can choose to manually configure the remaining adapters and cables. Is it okay to use autodiscovery for the other nodes (yes/no) [yes]? >>> Network Address for the Cluster Transport <<< The cluster transport uses a default network address of 172.16.0.0. If this IP address is already in use elsewhere within your enterprise, specify another address from the range of recommended private addresses (see RFC 1918 for details). The default netmask is 255.255.240.0. You can select another netmask, as long as it minimally masks all bits that are given in the network address. The default private netmask and network address result in an IP address range that supports a cluster with a maximum of 32 nodes, 10 private networks, and 12 virtual clusters. Is it okay to accept the default network address (yes/no) [yes]? Is it okay to accept the default netmask (yes/no) [yes]? Plumbing network address 172.16.0.0 on adapter net2 >> NOT DUPLICATE ... done Plumbing network address 172.16.0.0 on adapter net3 >> NOT DUPLICATE ... done >>> Set Global Fencing <<< Fencing is a mechanism that a cluster uses to protect data integrity when the cluster interconnect between nodes is lost. By default, fencing is turned on for global fencing, and each disk uses the global fencing setting. This screen allows you to turn off the global fencing. Most of the time, leave fencing turned on. However, turn off fencing when at least one of the following conditions is true: 1) Your shared storage devices, such as Serial Advanced Technology Attachment (SATA) disks, do not support SCSI; 2) You want to allow systems outside your cluster to access storage devices attached to your cluster; 3) Oracle Corporation has not qualified the SCSI persistent group reservation (PGR) support for your shared storage devices. If you choose to turn off global fencing now, after your cluster starts you can still use the cluster(1CL) command to turn on global fencing. Do you want to turn off global fencing (yes/no) [no]? >>> Quorum Configuration <<< Every two-node cluster requires at least one quorum device. By default, scinstall selects and configures a shared disk quorum device for you. This screen allows you to disable the automatic selection and configuration of a quorum device. You have chosen to turn on the global fencing. If your shared storage devices do not support SCSI, such as Serial Advanced Technology Attachment (SATA) disks, or if your shared disks do not support SCSI-2, you must disable this feature. If you disable automatic quorum device selection now, or if you intend to use a quorum device that is not a shared disk, you must instead use clsetup(1M) to manually configure quorum once both nodes have joined the cluster for the first time. Do you want to disable automatic quorum device selection (yes/no) [no]? yes >>> Global Devices File System <<< Each node in the cluster must have a local file system mounted on /global/.devices/node@<nodeID> before it can successfully participate as a cluster member. Since the "nodeID" is not assigned until scinstall is run, scinstall will set this up for you. You must supply the name of either an already-mounted file system or a raw disk partition which scinstall can use to create the global devices file system. This file system or partition should be at least 512 MB in size. Alternatively, you can use a loopback file (lofi), with a new file system, and mount it on /global/.devices/node@<nodeid>. If an already-mounted file system is used, the file system must be empty. If a raw disk partition is used, a new file system will be created for you. If the lofi method is used, scinstall creates a new 100 MB file system from a lofi device by using the file /.globaldevices. The lofi method is typically preferred, since it does not require the allocation of a dedicated disk slice. The default is to use lofi. For node "clusternode1", For node "clusternode2", Configuring global device using lofi on clusternode2: done Is it okay to create the new cluster (yes/no) [yes]? yes During the cluster creation process, cluster check is run on each of the new cluster nodes. If cluster check detects problems, you can either interrupt the process or check the log files after the cluster has been established. Interrupt cluster creation for cluster check errors (yes/no) [no]? Cluster Creation Log file - /var/cluster/logs/install/scinstall.log.3176 Starting discovery of the cluster transport configuration. The following connections were discovered: clusternode1:net2 switch1 clusternode2:net2 clusternode1:net3 switch2 clusternode2:net3 Completed discovery of the cluster transport configuration. Started cluster check on "clusternode1". Started cluster check on "clusternode2". cluster check completed with no errors or warnings for "clusternode1". cluster check completed with no errors or warnings for "clusternode2". Refer to the log file for details. The name of the log file is /var/cluster/logs/install/scinstall.log.3176. Configuring "clusternode2" ... done Rebooting "clusternode2" ... done Configuring "clusternode1" ... done Rebooting "clusternode1" ... |
If you see messages regarding the cluster check failing, take a look at the messages in the /var/cluster/logs/install/cluster_check/checkresults.txt file. They may be important, or you may be able to ignore them, but ensure you read the logs! Anything specified as a “violation” must be addressed, or it may prevent proper cluster operation.
After the cluster has configured and rebooted both nodes, the cluster will form! Congratulations, you now have a cluster. Granted, it isn’t doing anything useful at the moment, and it’s still in installmode as it cannot reach quorum. We need to connect it to our quorum device.
You can verify that the cluster is in installmode by running the following command on one of the cluster nodes:
|
1 2 |
# cluster show -t global | grep installmode installmode: enabled |
To add the quorum server, run another interactive program called clsetup from one of the nodes. This will guide you through the process of adding the quorum server. Output from my session follows.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 |
# clsetup >>> Initial Cluster Setup <<< This program has detected that the cluster "installmode" attribute is still enabled. As such, certain initial cluster setup steps will be performed at this time. This includes adding any necessary quorum devices, then resetting both the quorum vote counts and the "installmode" property. Please do not proceed if any additional nodes have yet to join the cluster. Is it okay to continue (yes/no) [yes]? Do you want to add any quorum devices (yes/no) [yes]? Following are supported Quorum Devices types in Oracle Solaris Cluster. Please refer to Oracle Solaris Cluster documentation for detailed information on these supported quorum device topologies. What is the type of device you want to use? 1) Directly attached shared disk 2) Network Attached Storage (NAS) from Network Appliance 3) Quorum Server q) Return to the quorum menu Option: 3 >>> Add a Quorum Server Quorum Device <<< A Quorum Server process runs on a machine outside Oracle Solaris Cluster and serves the cluster as a quorum device. Before configuring the quorum server as a quorum device into the cluster, you will need to setup the quorum server machine and start the quorum server process. For detailed information on setting up a quorum server, refer to Oracle Solaris Cluster system administration guide. You will need to specify a device name for the quorum server quorum device, which must be unique across all quorum devices, the IP address of the quorum server machine, or hostname if the machine is added into /etc/hosts, and a port number on the quorum server machine used to communicate with the cluster nodes. Please refer to the clquorum(1M) man page and other Oracle Solaris Cluster documentation for details. Is it okay to continue (yes/no) [yes]? What name do you want to use for this quorum device? storagehost What is the IP address of the quorum server machine? 10.1.1.70 What is the port number on the quorum server machine? 9000 Is it okay to proceed with the update (yes/no) [yes]? /usr/cluster/bin/clquorum add -t quorum_server -p qshost=10.1.1.70 -p port=9000 storagehost Command completed successfully. Press Enter to continue: Do you want to add another quorum device (yes/no) [yes]? no Once the "installmode" property has been reset, this program will skip "Initial Cluster Setup" each time it is run again in the future. However, quorum devices can always be added to the cluster using the regular menu options. Resetting this property fully activates quorum settings and is necessary for the normal and safe operation of the cluster. Is it okay to reset "installmode" (yes/no) [yes]? /usr/cluster/bin/clquorum reset /usr/cluster/bin/claccess deny-all Cluster initialization is complete. Type ENTER to proceed to the main menu: *** Main Menu *** Please select from one of the following options: 1) Quorum 2) Resource groups 3) Data Services 4) Cluster interconnect 5) Device groups and volumes 6) Private hostnames 7) New nodes 8) Zone Cluster 9) Other cluster tasks ?) Help with menu options q) Quit Option: q |
You can now verify that installmode is disabled:
|
1 2 |
# cluster show -t global | grep installmode installmode: disabled |
And confirm that the cluster is quorate by running the clquorum status command:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# clquorum status === Cluster Quorum === --- Quorum Votes Summary from (latest node reconfiguration) --- Needed Present Possible ------ ------- -------- 2 3 3 --- Quorum Votes by Node (current status) --- Node Name Present Possible Status --------- ------- -------- ------ clusternode2 1 1 Online clusternode1 1 1 Online --- Quorum Votes by Device (current status) --- Device Name Present Possible Status ----------- ------- -------- ------ storagehost 1 1 Online |
Back on the quorum server, you can run clquorumserver show to view the quorum reservation made by the quorum server itself, as well as the registrations made by the two nodes.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# clquorumserver show === Quorum Server on port 9000 === Disabled False --- Cluster spiffy-cluster (id 0x519CC174) Reservation --- Node ID: 1 Reservation key: 0x519cc17400000001 --- Cluster spiffy-cluster (id 0x519CC174) Registrations --- Node ID: 1 Registration key: 0x519cc17400000001 Node ID: 2 Registration key: 0x519cc17400000002 |
As a final verification, let’s confirm that our shared storage is visible:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# cldevice list -v DID Device Full Device Path ---------- ---------------- d1 clusternode1:/dev/rdsk/c0t600144F06F9201000000519DE7880004d0 d1 clusternode2:/dev/rdsk/c0t600144F06F9201000000519DE7880004d0 d2 clusternode1:/dev/rdsk/c0t600144F06F9201000000519DE78A0005d0 d2 clusternode2:/dev/rdsk/c0t600144F06F9201000000519DE78A0005d0 d3 clusternode2:/dev/rdsk/c7t0d0 d4 clusternode2:/dev/rdsk/c8t0d0 d5 clusternode2:/dev/rdsk/c8t1d0 d6 clusternode1:/dev/rdsk/c7t0d0 d7 clusternode1:/dev/rdsk/c8t0d0 d8 clusternode1:/dev/rdsk/c8t1d0 |
We can see above that the d1 and d2 DID devices are available to both nodes, and the device path corresponds with to the LUNs we created earlier.
Conclusion
This article has covered a large piece of configuration work. First, iSCSI LUNs and the iSCSI target were built, with restricted-access to the IQNs of the two initiators. A quorum server was built, and Solaris Cluster 4.1 installed to both cluster nodes. Finally, the cluster quorum was configured and installmode disabled, thus giving us a complete cluster ready for the creation and configuration of resource groups and resources.
In the next article, the cluster will be put to use, with failover filesystems, logical hostnames (“floating IP addresses”) and other resources being created.