Oracle Essentials Oracle Database 11g (52 page)

The instance uses the control files, redo log files, and database files to perform crash recovery. The fact that the instance is now running on another machine is irrele-vant—the various Oracle files on disk are the key.

Most failover solutions include software that runs on the machine to monitor specific processes, such as the background processes of the Oracle instance. If the primary node itself has not failed but some process has, the monitoring software will detect the failure of the process and take some action based on scripts set up by the system administrator. For example, if the Oracle instance fails, the monitoring software may attempt to restart the Oracle instance three times. If all three attempts are unsuccessful, the software may initiate physical hardware failover, transferring control to the alternate node in the cluster.

Figures 11-4 and 11-5 illustrate the process of implementing a simple failover.

266

|

Chapter 11: Oracle and High Availability

Primary

Server

Heartbeat

Alternative

Server

Oracle Instance

Oracle Database

Figure 11-4. Before failover

Alternative

Server

Heartbeat

Primary

Server

Oracle Instance

Oracle Database

Figure 11-5. After failover

Protecting Against System Failure

|

267

Outage duration for hardware failover

The time for failover to take effect, and therefore the length of the associated database downtime, depends on the following intervals:

Time for the alternate node to detect the failure of the primary node
The alternate node monitors the primary node using a heartbeat mechanism.

The frequency of this check is usually configurable—for example, every 30 seconds—providing control over the maximum time that a primary-node failure will go undetected.

Time for the alternate node to execute various startup actions
The time needed for such actions (e.g., assuming control of the disks used to store the Oracle database) may vary by system and should be determined through testing. One important consideration is the time required for a filesystem check. The larger the database, the larger the number of filesystems that may have been used. When the alternate node assumes control of the disks, it must check the state of the various filesystems on the disks.

Time for Oracle crash recovery

As we mentioned, youcan effectively control this time period using checkpoints.

Oracle provides a simple way to control recovery times using the initialization parameter FAST_START_IO_TARGET or the more recently introduced FAST_

START_MTTR_TARGET parameter.

When the instance fails, users will typically receive some type of error message and will typically attempt to log in again. Application developers can deal with this sequence of failover events with generic or specific error handling in their applications, or they can use the Transparent Application Failover functionality described later in this chapter.

Failover and operating system platform

This type of failover capability has been available for many years. On Unix-based platforms, vendors typically offer a simple failover solution that includes two machines, a private network between them to monitor the heartbeat, shared disk, and cluster software. No additional software is required from Oracle.

On Windows, Oracle includes Fail Safe, software that provides a GUI interface for configuring the Oracle database for hardware failover that leverages Microsoft’s Cluster Server. The mechanics of the failover are the same—a GUI is provided for administrative convenience. (In recent releases, the Fail Safe Manager and Real Applications Cluster Guard Manager have merged.)

Real Application Clusters

Oracle introduced Oracle Parallel Server (OPS), the predecessor to Real Application Clusters, in 1989 on Digital Equipment Corporation’s VAX clusters running the
268

|

Chapter 11: Oracle and High Availability

VMS operating system, and on Unix in 1993. OPS clusters were often deployed to assure a highly available database. Real Application Clusters followed, in Oracle9
i
, and was first made generally available in 2001.

At first glance, Real Application Clusters (RAC) may look similar to the clustered solutions described earlier in the
“Simple Hardware Failover”
section. Both failover and Real Application Clusters involve clustered hardware with access to disks from multiple nodes. The key difference is that with simple hardware failover only one node is an active instance. With RAC, the Oracle database is spread across multiple nodes and each node has an active Oracle instance. Clients can connect to any of the instances to access the same database.

Because each Oracle instance runs on its own node, if a node fails, the instance on that node also fails. The overall Oracle database remains available since surviving instances are still running on other working nodes.

Figure 11-6 illustrates Real Application Clusters on a cluster.

Database

Database

Server

Server

Interconnect

Oracle Instance

Oracle Instance

Oracle Database

Figure 11-6. Oracle Real Application Clusters on a cluster
Real Application Clusters and hardware failover

The Real Application Clusters option can typically provide higher levels of availability than simple hardware failover. This option can also provide additional flexibility for scaling database applications across multiple machines, and manageability was simplified as of Oracle Database 10
g
when Enterprise Manager Grid Control was introduced.

Protecting Against System Failure

|

269

Real Application Clusters increases availability by enabling avoidance of complete database blackouts. With simple hardware failover, the database is completely unavailable until node failover, instance startup, and crash recovery are complete.

With RAC, clients can connect to a surviving instance any time. Clients may be able to continue working with no interruption, depending on whether the data they need to work on was under the control of the failed instance. You can think of the failure of a Real Application Clusters instance as a potential database “brownout,” as opposed to the guaranteed blackout caused by hardware failover.

Some other key differences between hardware failover and Real Application Clusters include the following:

• The Real Application Clusters option avoids the various activities involved in disk takeover: mounting volumes, validating filesystem integrity, opening Oracle database files, and so on. Not performing these activities can significantly reduce the time required to achieve full system availability.

• The Real Application Clusters option doesn’t require the creation and maintenance of the complex scripts typically used to control the activities for hardware failover. For example, there is no need to script which disk volumes will be taken over by a surviving node. The automatic nature of Real Application Clusters avoids the complex initial system administration to set up the failover environment, as well as the ongoing administration needed as additional disk volumes are used. In fact, adding disk volumes to your database but forgetting to add the volumes to the various failover scripts can cause a hardware failover solution to fail itself!

In a simple two-way cluster used for hardware failover, both machines should have equal processing power and should be sized so that each can handle the entire workload. This equivalence is clearly required since only one node of the cluster is used at any point for the entire workload. If one node fails, the other should be capable of running the same workload with equal performance.

With Real Application Clusters, you can use both nodes of the cluster concurrently to spread the workload, reducing the load on one machine or node. You must still make sure that each machine will be powerful enough to adequately handle the entire workload (albeit at a reduced performance level) to meet basic business requirements when a node is not available.

Of course, using Real Application Clusters to spread the workload over several machines will result in a lower percentage of each machine’s resources being used in normal operating conditions, typically more expensive than using fully utilized machines. Each machine in the cluster must devote some overhead to maintaining its role in the cluster, although this overhead is minimal. You will have to weigh the benefits of carrying on with some performance degradation in the event of a node failure versus the cost of buying more nodes or more powerful machines. The economics of your situation may dictate that a decrease in performance in the event of a node failure is more palatable than a larger initial outlay for more nodes or larger systems.

270

|

Chapter 11: Oracle and High Availability

Much of the complexity of tuning and programming for scalability has been removed since Oracle9
i
. Deployment was simplified in Oracle Database 10
g
when integrated clusterware was first introduced. Interested readers can find more details about Real Application Clusters scalability in the Oracle documentation and in
Chapter 9
of this book.

Node failure and Real Application Clusters

The database instances provide protection for each other—if an instance fails, one of the surviving instances will detect the failure and automatically initiate RAC recovery. This type of recovery is different from the hardware failover discussed previously. No actual “failover” occurs—no disk takeover is required, since all nodes already have access to the disks used for the database. There is no need to start an Oracle instance on the surviving node or nodes, since Oracle is already running on all the nodes. The Oracle software performs the necessary actions without using scripts; the required steps are an integral part of Real Application Clusters software.

The phases of Real Application Clusters recovery are the following:
Cluster reorganization

When an instance failure occurs, Real Application Clusters must first determine which nodes of the cluster remain in service. Oracle9
i
introduced a disk-based heartbeat in which each database group member votes on what members are part of the current group. Based on arbitration, a correct current group configuration is established. The time required for this operation is very brief.

Lock database rebuild

The lock database, which contains the information used to coordinate Real Application Clusters traffic, is distributed across the multiple active instances.

Therefore, a portion of that information is lost when a node fails. The remaining nodes have sufficient redundant data to reconstruct the lost information. Once the cluster membership is determined, the surviving instances reconstruct the lock database. The time for this phase depends on how many locks must be recovered, as well as whether the rebuild process involves a single surviving node or multiple surviving nodes. Oracle speeds the lock remastering process by allowing optimization of lock master locations in the background while users are accessing the system. In a two-node cluster, node failure leaves a single surviving node that acts as a dictator and processes the lock operations very quickly.

Instance recovery

Once the lock database is rebuilt, the redo logs from the failed instance perform crash recovery. This is similar to single-instance crash recovery—a rollforward phase followed by a nonblocking, deferred rollback phase. The key difference is that the recovery isn’t performed by restarting a failed instance. Rather, it’s performed by the instance that detected the failure.

Protecting Against System Failure

|

271

While Real Application Clusters recovery is in progress, clients connected to surviving instances remain connected and can continue working. In some cases users may experience a slight delay in response times, but their sessions aren’t terminated. Clients connected to the failed instance can reconnect to a surviving instance and can resume working. Uncommitted transactions will be rolled back and will have to be resubmitted. Queries that were active will also have been terminated; however, Transparent Application Failover (TAF) can be used to automatically continue query processing on a surviving node without requiring users to resubmit their queries.

You can also use TAF to resubmit transactions without user intervention.

Parallel Fail Safe/RACGuard

Oracle Parallel Fail Safe was renamed RACGuard in Oracle9
i
and integrated into the core RAC product in Oracle Database 10
g
. Prior to Oracle Database 10
g
, it was a feature in Real Application Clusters that leveraged the clustering software from systems vendors. As of Oracle Database 10
g
, the database includes a cluster filesystem.

RACGuard supported such features as:

• Automated, fast, and bounded recovery times from Oracle instance crashes

• Automatic capture of diagnostic data

• Guaranteed primary and secondary configuration

• Support for features such as Transparent Application Failover (described in the next section)

• Client preconnection to secondary instances to speed reconnection
Oracle Transparent Application Failover

Oracle introduced the Transparent Application Failover (TAF) capability in the first release of Oracle8. As the name implies, TAF provides a seamless migration of users’

sessions from one Oracle instance to another. You can use TAF to mask the failure of an instance for transparent high availability or to migrate users from an active instance to a less active one.
Figure 11-7
illustrates TAF with Real Application Clusters.