Cluster planning:
The area of cluster planning is a large one. Not only does it include planning for the types of hardware (CPUs, networks, and disks) to be used in the cluster, but it also includes other aspects. These include resource planning, that is, planning the desired behavior of the cluster in failure situations. Resource planning must take application loads and characteristics into account, as well as priorities.
High availability:
A high availability solution will ensure that any failure of any component of the solution, be it hardware, software, or system management, will not cause the application and its data to be inaccessible to the user community.
High availability is:
The masking or elimination of both planned and unplanned downtime.
The elimination of single points of failure (SPOFs)
Fault resilience, but not fault tolerance
High availability systems are an excellent solution for applications that can withstand a short interruption should a failure occur, but which must be restored quickly.
The difference between fault tolerance and high availability is:
A fault tolerant environment has no service interruption, while a highly available environment has a minimal service interruption.
System downtime is either planned or unplanned, with planned downtime accounting for the vast majority of the total.
HACMP allows you to minimize or eliminate this planned downtime from your operation, by allowing you to maintain a service to your customers while performing hardware upgrades, software upgrades, or other maintenance activity at the same time.
Services may be moved from one cluster node to another at will, allowing the original node to undergo maintenance without affecting the availability of the service. When the maintenance activity is completed, the service may be moved back to the node which was originally running it.
Unplanned downtime has one of two causes: periods caused by hardware failures, and those periods caused by software failures.
Hardware has been getting more and more reliable over time and will continue to do so, but hardware remains a cause of failures. Together with facilities provided by AIX, HACMP can protect your operation from a hardware failure, by automatically moving the services provided by the failing node to other nodes within the cluster.
Cluster nodes
One of HACMP's key design strengths is its ability to provide support across the entire range of RISC System/6000 products. Because of this built-in flexibility and the facility to mix and match RISC System/6000 products, the effort required to design a highly available cluster is significantly reduced.
The designated hardware should only be used on an appropriate IBM ^ pSeries, RS/6000 Platform, or 9076 Scalable POWERParallel Platform (SP).
The following sections will deal with the various:
Operating system levels
CPU options
Disk storages for CRM
Cluster node considerations available to you when you are planning your HACMP cluster.
Operating system level
Before installation of HACMP, make sure to have the proper version of the
operating system. Here is a list of required operating system levels for HACMP
versions (see Table2-1) and Parallel System Support Program versions (see
Table2-2).
Table 2-1 Required OS level for HACMP
Table 2-2 PSSP versions for SP installation
AIX OS levelHACMP Version
4.3.1
HACMP Version
4.4.0
HACMP Version
4.4.1
4.3.2 yesnono
4.3.3 yesyesyes
5.1 noyes*yes
* Note:
The following restrictions apply to HACMP for AIX Version 4.4.0 support of IBM AIX 5L for Power Version 5.1. IBM intends to remove these restrictions through further APARs on HACMP.
Enhanced Journaled File Systems are not supported on shared volume groups.
Fencing is not supported for concurrent mode volume groups created on 9333 disks.
HACMP can only run on 32-bit AIX kernels. Even if the hardware is capable of supporting 64-bit kernels, it must be booted using the the bosboot command with a 32-bit kernel.
The VSM-based xhacmpm configuration utility is not supported.
HACMP versionPrerequisite PSSP version
HACMP Version 4.3.1 for AIXPSSP Version 3.1
HACMP Version 4.4.0 for AIXPSSP Version 3.2
HACMP Version 4.4.1 for AIXPSSP Version 3.2
CPU options
HACMP is designed to execute with RS/6000 uniprocessors, SMP servers, and SP systems in a no single point of failure server configuration. HACMP supports the IBM ^ pSeries and the RS/6000 models that are designed for server application and meet the minimum requirements for internal memory, internal disk, and I/O slots.
Cluster node considerations:
Your major goal throughout the planning process is to eliminate single points of failure. A single point of failure exists when a critical cluster function is provided by a single component. If that component fails, the cluster has no other way of providing that function, and the service depending on that component becomes unavailable.
HACMP for AIX is designed to recover from a single hardware or software failure. It may not be able to handle multiple failures, depending on the sequence of failures. For example, the default event scripts cannot do an adapter swap after an IP address takeover (IPAT) has occurred if only one standby adapter exists for that network.
How to eliminate the single point of failure
Table2-6 summarizes potential single points of failure within an HACMP cluster and describes how to eliminate them.
Cluster networks
HACMP differentiates between two major types of networks: TCP/IP networks and non-TCP/IP networks. HACMP utilizes both of them for exchanging heartbeats. HACMP uses these heartbeats to diagnose failures in the cluster. Non-TCP/IP networks are used to distinguish an actual hardware failure from the failure of the TCP/IP software. If there were only TCP/IP networks being used, and the TCP/IP software failed, causing heartbeats to stop, HACMP could falsely diagnose a node failure when the node was really still functioning. Since a non-TCP/IP network would continue working in this event, the correct diagnosis could be made by HACMP. In general, all networks are also used for verification, synchronization, communication, and triggering events between nodes. Of course, TCP/IP networks are used for communication with client machines as well.
TCP/IP networks:
The following sections describe supported TCP/IP network types and network considerations.
Supported TCP/IP network types
Basically every adapter that is capable of running the TCP/IP protocol is a supported HACMP network type. There are some special considerations for certain types of adapters, however. The following gives a brief overview on the supported adapters and their special considerations.
Below is a list of TCP/IP network types as you will find them at the configuration time of an adapter for HACMP.
Generic IP
ATM
Ethernet
FCS
FDDI
SP Switch
SLIP
SOCC
Token-Ring
Heartbeating in HACMP
The primary task of HACMP™ is to recognize and respond to failures. HACMP uses heartbeating to monitor the activity of its network interfaces, devices and IP labels.
Heartbeating connections between cluster nodes are necessary because they enable HACMP to recognize the difference between a network failure and a node failure. For instance, if connectivity on the HACMP network (this network's IP labels are used in a resource group) is lost, and you have another TCP/IP based network and a non-IP network configured between the nodes, HACMP recognizes the failure of its cluster network and takes recovery actions that prevent the cluster from becoming partitioned.
To avoid cluster partitioning, we highly recommend configuring redundant networks in the HACMP cluster and using both IP and non-IP networks. Out of these networks, some networks will be used for heartbeating purposes.
In general, heartbeats in HACMP can be sent over:
TCP/IP networks.
Serial (non-IP) networks (RS232, TMSCSI, TMSSA and disk heartbeating).
The Topology Services component of RSCT carries out the heartbeating function in HACMP.
Topology services and heartbeat rings
HACMP uses the Topology Services component of RSCT for monitoring networks and network interfaces. Topology Services organizes all the interfaces in the topology into different heartbeat rings. The current version of RSCT Topology services has a limit of 48 heartbeat rings, which is usually sufficient to monitor networks and network interfaces.
Heartbeat rings are dynamically created and used internally by RSCT. They do not have a direct, one-to-one correlation to HACMP networks or number of network interfaces. The algorithm for allocating interfaces and networks to heartbeat rings is complex, but generally follows these rules:
In an HACMP network, there is one heartbeat ring to monitor the service interfaces, and one for each set of non-service interfaces that are on the same subnet. The number of non-service heartbeat rings is determined by the number of non-service interfaces in the node with the largest number of interfaces.
The number of heartbeat rings is approximately equal to the largest number of interfaces found on any one node in the cluster.
Note that during cluster verification, HACMP calls the RSCT verification API. This API performs a series of verifications, including a check for the heartbeat ring calculations, and issues an error if the limit is exceeded.
Heartbeating over IP aliases
This section contains information about heartbeating over IP aliases.
Overview
In general, HACMP subnetting requirements can be complicated to understand and may require that you reconfigure networks in AIX® to avoid features such as multiple subnet routes, which can lead to a single point of failure for network traffic
Overview
In general, HACMP™ subnetting requirements can be complicated to understand and may require that you reconfigure networks in AIX® to avoid features such as multiple subnet routes, which can lead to a single point of failure for network traffic.
When planning your cluster networks, you may need to:
Reconfigure IP addresses of HACMP interfaces that will be used at boot time or
Update/etc/hosts with the new boot time IP addresses.
Heartbeating over IP Aliases is useful because it:
Uses automatically generated IP aliases for heartbeating
Heartbeating over IP Aliasing provides an option where the addresses used for heartbeating can be automatically configured by HACMP in a subnet range that is outside of the range used for the base NIC or any service addresses.
Although Heartbeating over IP Aliasing automatically configures proper aliases for heartbeating, you must still be aware of the implications of subnet routing for all boot and service IP addresses. That is, failure to plan subnets properly can lead to application failures that are not detectable by HACMP. Reliable HACMP cluster communication still requires that the interfaces on a single network can communicate with the other nodes on that network.
Enables you to avoid reconfiguration of boot time addresses and /etc/hosts.
RSCT sets up the heartbeat rings to go over a separate range of IP aliases. This lets you use a specified subnet in a non-routable range for a heartbeat ring, preserving your other subnets for routable traffic. This also allows you to avoid reconfiguring boot time addresses and entries in /etc/hosts.
Makes HACMP topology configuration easier to understand.
Does not require that you obtain additional routable subnets from the network administrator.
For instance, you can use heartbeating over aliases in HACMP, if due to the network system administration restrictions, the IP addresses that your system can use at boot time must reside on the same subnet. (In general, if there are no system administration restrictions, the IP addresses that your system can use at boot time can reside on either the same or different subnets).
Heartbeating over disk
You can configure a non-IP point-to-point heartbeating network, called a disk heartbeating network, over any shared disk in an enhanced concurrent mode volume group. Heartbeating over disk provides another type of non-IP point-to-point network for failure detection.
Disk heartbeating networks provide an alternative to other point-to-point networks such as RS232 that have cable length restrictions, or TMSSA which require special disk adapter hardware and cabling. Heartbeating over disk does not require additional or specialized hardware, cabling or microcode; it can use any disk that is also used for data and for which volume groups and file systems are included in an HACMP™ resource group.
In a disk heartbeating network, two nodes connected to the disk periodically write heartbeat messages and read heartbeat messages (written by the other node) on a small, non-data portion of the disk. A disk heartbeating network, like the other non-IP heartbeating networks, connects only two nodes. In clusters with more than two nodes, use multiple disks for heartbeating. Each node should have a non-IP heartbeat path to at least one other node. If the disk heartbeating path is severed, at least one node cannot access the shared disk.
You have two different ways for configuring a disk heartbeating network in a cluster:
You can create an enhanced concurrent volume group shared by multiple nodes in your cluster. Then you use the HACMP Extended Configuration SMIT path to configure a point-to-point pair of discovered communication devices.
or
You can start by creating a cluster disk heartbeating network, and then add devices to it using the Add Pre-Defined Communication Interfaces and Devices panel in SMIT.
The HACMP cluster verification utility verifies that the disk heartbeating networks are properly configured.
Heartbeating over disk and fast method for node failure detection
With HACMP, you can reduce the time it takes for node failure to be realized throughout the cluster. If you have a disk heartbeating network configured, and specify a parameter for a disk heartbeating NIM, then when a node fails, HACMP uses a disk heartbeating network to place a departing message on the shared disk so neighboring nodes are aware of the node failure within one heartbeat period.
The area of cluster planning is a large one. Not only does it include planning for the types of hardware (CPUs, networks, and disks) to be used in the cluster, but it also includes other aspects. These include resource planning, that is, planning the desired behavior of the cluster in failure situations. Resource planning must take application loads and characteristics into account, as well as priorities.
High availability:
A high availability solution will ensure that any failure of any component of the solution, be it hardware, software, or system management, will not cause the application and its data to be inaccessible to the user community.
High availability is:
The masking or elimination of both planned and unplanned downtime.
The elimination of single points of failure (SPOFs)
Fault resilience, but not fault tolerance
High availability systems are an excellent solution for applications that can withstand a short interruption should a failure occur, but which must be restored quickly.
The difference between fault tolerance and high availability is:
A fault tolerant environment has no service interruption, while a highly available environment has a minimal service interruption.
System downtime is either planned or unplanned, with planned downtime accounting for the vast majority of the total.
HACMP allows you to minimize or eliminate this planned downtime from your operation, by allowing you to maintain a service to your customers while performing hardware upgrades, software upgrades, or other maintenance activity at the same time.
Services may be moved from one cluster node to another at will, allowing the original node to undergo maintenance without affecting the availability of the service. When the maintenance activity is completed, the service may be moved back to the node which was originally running it.
Unplanned downtime has one of two causes: periods caused by hardware failures, and those periods caused by software failures.
Hardware has been getting more and more reliable over time and will continue to do so, but hardware remains a cause of failures. Together with facilities provided by AIX, HACMP can protect your operation from a hardware failure, by automatically moving the services provided by the failing node to other nodes within the cluster.
Cluster nodes
One of HACMP's key design strengths is its ability to provide support across the entire range of RISC System/6000 products. Because of this built-in flexibility and the facility to mix and match RISC System/6000 products, the effort required to design a highly available cluster is significantly reduced.
The designated hardware should only be used on an appropriate IBM ^ pSeries, RS/6000 Platform, or 9076 Scalable POWERParallel Platform (SP).
The following sections will deal with the various:
Operating system levels
CPU options
Disk storages for CRM
Cluster node considerations available to you when you are planning your HACMP cluster.
Operating system level
Before installation of HACMP, make sure to have the proper version of the
operating system. Here is a list of required operating system levels for HACMP
versions (see Table2-1) and Parallel System Support Program versions (see
Table2-2).
Table 2-1 Required OS level for HACMP
Table 2-2 PSSP versions for SP installation
AIX OS levelHACMP Version
4.3.1
HACMP Version
4.4.0
HACMP Version
4.4.1
4.3.2 yesnono
4.3.3 yesyesyes
5.1 noyes*yes
* Note:
The following restrictions apply to HACMP for AIX Version 4.4.0 support of IBM AIX 5L for Power Version 5.1. IBM intends to remove these restrictions through further APARs on HACMP.
Enhanced Journaled File Systems are not supported on shared volume groups.
Fencing is not supported for concurrent mode volume groups created on 9333 disks.
HACMP can only run on 32-bit AIX kernels. Even if the hardware is capable of supporting 64-bit kernels, it must be booted using the the bosboot command with a 32-bit kernel.
The VSM-based xhacmpm configuration utility is not supported.
HACMP versionPrerequisite PSSP version
HACMP Version 4.3.1 for AIXPSSP Version 3.1
HACMP Version 4.4.0 for AIXPSSP Version 3.2
HACMP Version 4.4.1 for AIXPSSP Version 3.2
CPU options
HACMP is designed to execute with RS/6000 uniprocessors, SMP servers, and SP systems in a no single point of failure server configuration. HACMP supports the IBM ^ pSeries and the RS/6000 models that are designed for server application and meet the minimum requirements for internal memory, internal disk, and I/O slots.
Cluster node considerations:
Your major goal throughout the planning process is to eliminate single points of failure. A single point of failure exists when a critical cluster function is provided by a single component. If that component fails, the cluster has no other way of providing that function, and the service depending on that component becomes unavailable.
HACMP for AIX is designed to recover from a single hardware or software failure. It may not be able to handle multiple failures, depending on the sequence of failures. For example, the default event scripts cannot do an adapter swap after an IP address takeover (IPAT) has occurred if only one standby adapter exists for that network.
How to eliminate the single point of failure
Table2-6 summarizes potential single points of failure within an HACMP cluster and describes how to eliminate them.
Cluster networks
HACMP differentiates between two major types of networks: TCP/IP networks and non-TCP/IP networks. HACMP utilizes both of them for exchanging heartbeats. HACMP uses these heartbeats to diagnose failures in the cluster. Non-TCP/IP networks are used to distinguish an actual hardware failure from the failure of the TCP/IP software. If there were only TCP/IP networks being used, and the TCP/IP software failed, causing heartbeats to stop, HACMP could falsely diagnose a node failure when the node was really still functioning. Since a non-TCP/IP network would continue working in this event, the correct diagnosis could be made by HACMP. In general, all networks are also used for verification, synchronization, communication, and triggering events between nodes. Of course, TCP/IP networks are used for communication with client machines as well.
TCP/IP networks:
The following sections describe supported TCP/IP network types and network considerations.
Supported TCP/IP network types
Basically every adapter that is capable of running the TCP/IP protocol is a supported HACMP network type. There are some special considerations for certain types of adapters, however. The following gives a brief overview on the supported adapters and their special considerations.
Below is a list of TCP/IP network types as you will find them at the configuration time of an adapter for HACMP.
Generic IP
ATM
Ethernet
FCS
FDDI
SP Switch
SLIP
SOCC
Token-Ring
Heartbeating in HACMP
The primary task of HACMP™ is to recognize and respond to failures. HACMP uses heartbeating to monitor the activity of its network interfaces, devices and IP labels.
Heartbeating connections between cluster nodes are necessary because they enable HACMP to recognize the difference between a network failure and a node failure. For instance, if connectivity on the HACMP network (this network's IP labels are used in a resource group) is lost, and you have another TCP/IP based network and a non-IP network configured between the nodes, HACMP recognizes the failure of its cluster network and takes recovery actions that prevent the cluster from becoming partitioned.
To avoid cluster partitioning, we highly recommend configuring redundant networks in the HACMP cluster and using both IP and non-IP networks. Out of these networks, some networks will be used for heartbeating purposes.
In general, heartbeats in HACMP can be sent over:
TCP/IP networks.
Serial (non-IP) networks (RS232, TMSCSI, TMSSA and disk heartbeating).
The Topology Services component of RSCT carries out the heartbeating function in HACMP.
Topology services and heartbeat rings
HACMP uses the Topology Services component of RSCT for monitoring networks and network interfaces. Topology Services organizes all the interfaces in the topology into different heartbeat rings. The current version of RSCT Topology services has a limit of 48 heartbeat rings, which is usually sufficient to monitor networks and network interfaces.
Heartbeat rings are dynamically created and used internally by RSCT. They do not have a direct, one-to-one correlation to HACMP networks or number of network interfaces. The algorithm for allocating interfaces and networks to heartbeat rings is complex, but generally follows these rules:
In an HACMP network, there is one heartbeat ring to monitor the service interfaces, and one for each set of non-service interfaces that are on the same subnet. The number of non-service heartbeat rings is determined by the number of non-service interfaces in the node with the largest number of interfaces.
The number of heartbeat rings is approximately equal to the largest number of interfaces found on any one node in the cluster.
Note that during cluster verification, HACMP calls the RSCT verification API. This API performs a series of verifications, including a check for the heartbeat ring calculations, and issues an error if the limit is exceeded.
Heartbeating over IP aliases
This section contains information about heartbeating over IP aliases.
Overview
In general, HACMP subnetting requirements can be complicated to understand and may require that you reconfigure networks in AIX® to avoid features such as multiple subnet routes, which can lead to a single point of failure for network traffic
Overview
In general, HACMP™ subnetting requirements can be complicated to understand and may require that you reconfigure networks in AIX® to avoid features such as multiple subnet routes, which can lead to a single point of failure for network traffic.
When planning your cluster networks, you may need to:
Reconfigure IP addresses of HACMP interfaces that will be used at boot time or
Update/etc/hosts with the new boot time IP addresses.
Heartbeating over IP Aliases is useful because it:
Uses automatically generated IP aliases for heartbeating
Heartbeating over IP Aliasing provides an option where the addresses used for heartbeating can be automatically configured by HACMP in a subnet range that is outside of the range used for the base NIC or any service addresses.
Although Heartbeating over IP Aliasing automatically configures proper aliases for heartbeating, you must still be aware of the implications of subnet routing for all boot and service IP addresses. That is, failure to plan subnets properly can lead to application failures that are not detectable by HACMP. Reliable HACMP cluster communication still requires that the interfaces on a single network can communicate with the other nodes on that network.
Enables you to avoid reconfiguration of boot time addresses and /etc/hosts.
RSCT sets up the heartbeat rings to go over a separate range of IP aliases. This lets you use a specified subnet in a non-routable range for a heartbeat ring, preserving your other subnets for routable traffic. This also allows you to avoid reconfiguring boot time addresses and entries in /etc/hosts.
Makes HACMP topology configuration easier to understand.
Does not require that you obtain additional routable subnets from the network administrator.
For instance, you can use heartbeating over aliases in HACMP, if due to the network system administration restrictions, the IP addresses that your system can use at boot time must reside on the same subnet. (In general, if there are no system administration restrictions, the IP addresses that your system can use at boot time can reside on either the same or different subnets).
Heartbeating over disk
You can configure a non-IP point-to-point heartbeating network, called a disk heartbeating network, over any shared disk in an enhanced concurrent mode volume group. Heartbeating over disk provides another type of non-IP point-to-point network for failure detection.
Disk heartbeating networks provide an alternative to other point-to-point networks such as RS232 that have cable length restrictions, or TMSSA which require special disk adapter hardware and cabling. Heartbeating over disk does not require additional or specialized hardware, cabling or microcode; it can use any disk that is also used for data and for which volume groups and file systems are included in an HACMP™ resource group.
In a disk heartbeating network, two nodes connected to the disk periodically write heartbeat messages and read heartbeat messages (written by the other node) on a small, non-data portion of the disk. A disk heartbeating network, like the other non-IP heartbeating networks, connects only two nodes. In clusters with more than two nodes, use multiple disks for heartbeating. Each node should have a non-IP heartbeat path to at least one other node. If the disk heartbeating path is severed, at least one node cannot access the shared disk.
You have two different ways for configuring a disk heartbeating network in a cluster:
You can create an enhanced concurrent volume group shared by multiple nodes in your cluster. Then you use the HACMP Extended Configuration SMIT path to configure a point-to-point pair of discovered communication devices.
or
You can start by creating a cluster disk heartbeating network, and then add devices to it using the Add Pre-Defined Communication Interfaces and Devices panel in SMIT.
The HACMP cluster verification utility verifies that the disk heartbeating networks are properly configured.
Heartbeating over disk and fast method for node failure detection
With HACMP, you can reduce the time it takes for node failure to be realized throughout the cluster. If you have a disk heartbeating network configured, and specify a parameter for a disk heartbeating NIM, then when a node fails, HACMP uses a disk heartbeating network to place a departing message on the shared disk so neighboring nodes are aware of the node failure within one heartbeat period.
No comments:
Post a Comment