Fast Application Notification (FAN)

The Oracle RAC high availability framework monitors a database and its services, and sends Fast Application Notification (FAN) event notifications.

Oracle AI Database focuses on maintaining the highest possible service availability. In Oracle Real Application Clusters (Oracle RAC), services are designed to be continuously available with loads shared across one or more instances. The Oracle RAC high-availability framework maintains service availability by using Oracle Clusterware and resource profiles. Oracle Clusterware recovers and balances services according to business rules and the service attributes.

Overview of Fast Application Notification (FAN)

FAN provides immediate interrupt of clients following outages related to the database, nodes, and networks.

FAN is essential to break clients out of TCP/IP timeouts immediately following failures. FAN notifies clients immediately when resources become available and initiates draining of database sessions so clients experience no outages during planned maintenance. FAN also includes notifying configuration-level and service-level information that includes changes in service status.

The Oracle client drivers and Oracle Real Application Clusters (Oracle RAC) connection pools respond to FAN events, and take immediate action. FAN UP and DOWN events apply to services, databases, instances, networks, and nodes.

The Importance of Using Fast Application Notification

Using Fast Application Notification (FAN) events eliminates applications waiting on TCP timeouts, time wasted processing the last result at the client after a failure has occurred, and time wasted running work on slow, suspended, or terminated nodes.

Applications can waste time in many critical ways:

Waiting for TCP/IP timeouts when a node fails without closing sockets, and for every subsequent connection while that IP address is down.
Attempting to connect when services are down.
Not connecting when services resume.
Processing the last result at the client when the server goes down.
Attempting to run work on sub-optimal nodes.

When a node fails without closing sockets, all sessions that are blocked in an I/O wait (read or write) wait for tcp_keepalive. This wait status is the typical condition for an application connected by a socket. Sessions processing the last result are even worse off, not receiving an interrupt until the next data is requested.

How FAN is Used with Oracle AI Database and Applications

Fast Application Notification (FAN) is essential to prevent applications from stop responding on TCP/IP timeouts.

FAN events are published using Oracle Notification Service starting with Oracle Database 12.2. Advanced Queuing is used for FAN events only for older Oracle Call Interface (OCI) applications (OCI drivers before 12.2). The publication mechanisms are automatically configured as part of your Oracle RAC installation. There are specific settings needed on each client to enable the client to receive FAN events.

For OCI clients, the service attribute notification must be set on the server. For example srvctl modify service -db EMEA -service GOLD -notification TRUE . Also, for OCI clients you must set events to TRUE in the oraaccess.xml configuration file.
For ODP .Net clients, you must set HA events to TRUE in the oraaccess.xml in the connect string.
For Universal Connection Pool clients, set the pool property Fast Connection Failover to true ( setFastConnectionFailoverEnabled(true) ) in the properties file.

All clients can use the auto-configuration of ONS to receive events, but the clients still need settings to ensure they react to these events.

Oracle Net Services listeners and Global Data Services (GDS) are integrated with FAN events, enabling the listener and GDS to immediately de-register services provided by the failed instance and to avoid erroneously sending connection requests to failed instances.

Oracle connection pools use FAN to receive very fast notification of failures, to balance connections following failures, and to balance connections again after the failed components are repaired. So, when a service connecting to an Oracle AI Database instance starts, the connection pool uses the FAN event to route work to that resource, immediately. When a service for a database instance or node fails, the connection pool uses the FAN event to immediately interrupt applications to recover.

For cluster configuration changes, the Oracle Real Application Clusters (Oracle RAC) high availability framework publishes a FAN event immediately when a state change occurs in the cluster. Instead of waiting for the application to time out against the database and detect a problem, applications can receive FAN events and react immediately. With FAN, in-flight transactions are immediately terminated and the client notified when the instance fails.

FAN also publishes load balancing advisory events. Applications can take advantage of the load balancing advisory FAN events to direct work requests to the instance in the cluster that is currently providing the best service quality.

If you specify the connection load balancing goal CLB_GOAL_LONG for a database service, then the listener uses the load balancing advisory when the listener balances the connection loads. When load balancing advisory is enabled, the metrics used for the listener are finer grained.

You can take advantage of FAN events in the following ways:

Applications can use FAN if you use an integrated Oracle client. The integrated clients for FAN events include Oracle JDBC Universal Connection Pool, ODP.NET connection pool, OCI session pool, Oracle WebLogic Server Active Gridlink for Oracle RAC, and OCI, JDBC-thin, and ODP.NET Managed and Unmanaged clients. The pooled clients can take advantage of the load balancing advisory FAN events.
You can configure third-party application containers, such as those provided by Apache Tomcat and WebSphere, to use the built-in FAN support offered by using the Universal Connection Pool in place of the default pool, which is certified as a connection pool for third-party Java application servers including Apache Tomcat and WebSphere.
Use the FAN-aware capability of the Oracle drivers by using standard interfaces to test connections on get or release from the third-party connection pools in use by third-party application servers or custom applications.
This solution applies to standard Java applications through the use of the standard TNS connect string and ensures that the ons.jar and simpleFAN.jar files are available on the application CLASSPATH.
For the OCI/OCCI driver, the OCI_ATTR_SERVER_STATUS server context handle attribute is sensitive to FAN events and will return OCI_SERVER_NOT_CONNECTED if the connection has been affected by a FAN event.
You can implement FAN with server-side callouts on your database tier.
Applications can use FAN programmatically by using the JDBC and Oracle RAC FAN application programming interface (API) or by using callbacks with OCI and ODP.NET to subscribe to FAN events and to run event handling actions upon the receipt of an event.

For planned maintenance and applications using OCI or Pro* precompilers (and not using the OCI session pool or Tuxedo), an application must check OCI_ATTR_SERVER_STATUS. Add this check when sessions are returned to your own connection pool, and for idle connections, regularly. Following a FAN down event with planned maintenance, this attribute is set to OCI_SERVER_NOT_CONNECTED. The application closes the connection after reading this disconnected status. The session remains open for draining of active work until the application closes, providing error-free failover.

If you use one of the integrated clients listed in the first item of the preceding list, then, for DOWN events, the disruption to the application is minimized because the FAN-aware client terminates the connections to the failed instance or node before they are reused. Active work can be allowed to complete and, if there is a remaining instance, then continuous service can be maintained for ongoing work. Any sessions active when the instance or service stops are terminated and the application user is immediately notified. Incomplete transactions can be protected by Application Continuity, if it is enabled. Application users who request connections are directed only to available instances.

For UP events, when database services and instances are started, new connections are created so that the application can immediately take advantage of the extra hardware resources or additional capacity.

Requirements for Using FAN

Learn what you need to do to take advantage of FAN-aware capabilities in client drivers connecting to Oracle Real Application Clusters (Oracle RAC) databases.

Client drivers on releases after Oracle Database 12c release 2 (12.2) are FAN-aware, and FAN is enabled by default. This is also true for the JDBC Thin driver (12.2.0.1 and later), and Oracle Data Provider for Net (ODP.NET) drivers. A client driver can detect planned and unplanned FAN events, and take action beneath the application.

To take advantage of FAN-aware capabilities in the drivers, the following is required:

For the thin Java driver, beginning with release 12.2, FAN is automatically enabled by placing the ons.jar and simpleFAN.jar files on the CLASSPATH, and by using the recommended TNS format. Using the recommended TNS format automatically configures ONS. Also with the Java thin driver, FAN is supported for both planned and unplanned events. For unplanned outages, the FAN interrupt is immediate. For planned maintenance, configure the Java application servers or custom pools using standard interfaces to test connections on get or release from third-party connection pools. For example, depending on the application server, test TestConnectionsOnReserve, TestOnBorrow, or PreTest connections.
With this approach, when a FAN event is received during planned maintenance, Fast Connection Failover (FCF) closes sessions when they are tested, because the application does not have a connection to the database at this time, and can retry for a new connection. The connection tests may use isValid, isClosed, isUsable, or PingDatabase.
At the time the SQL command runs, the database will drain the connection, if it is affected by the upcoming planned maintenance. Connection pools, data sources, and, in the programmatic case, customer applications, must all be ready to manage the recoverable error that occurs when the SQL command runs, which usually closes the physical connection.
Third-party Java application servers and Java applications can use the PooledConnection standard interface when developing connection pools.
Beginning with the 11.2.0.3 release of the Oracle Call Interface (OCI) and Oracle C++ Call Interface (OCCI) drivers, when the OCI_ATTR_SERVER_STATUS server context handle attribute returns OCI_SERVER_NOT_CONNECTED, the application must terminate the connection. Work will be drained for planned maintenance. Releases of the driver after 12.2.0.1 can also detect OCISessionRelease and OCIRequestEnd when it receives a planned DOWN event.

FAN Callouts

Fast Application Notification (FAN) callouts are server-side scripts or executables that run whenever a FAN event is generated.

You can design and build callouts to do many things, such as:

Log status information
Page DBAs or to open support tickets when resources fail to start
Automatically start dependent external applications that must be co-located with a service
Shut down services when the number of available instances for a database decreases, for example, if nodes fail
Automate the fail back of a service to preferred instances, if the -failback parameter is not sufficient

Fast Application Notification High Availability Events

Learn how the Fast Application Notification (FAN) event delivers information to a callout program.

In the following example, FAN event types are listed always as the first entry when you receive FAN information through a callout, as in the following examples:

#service UP when the service starts
SERVICEMEMBER VERSION=1.0
   service=HRPDB1.example.com database=ractest
   instance=ractest2 host=prod_host01_1 status=up reason=BOOT
   card=1 timestamp=2019-10-24 09:11:51 timezone=+00:00
   db_domain=example.com
SERVICE VERSION=1.0
   service=HRPDB1.example.com database=ractest instance=ractest2
   host=prod_host01_1 status=up reason=BOOT
   timestamp=2019-10-24 09:11:51 timezone=+00:00
   db_domain=example.com

#service DOWN
SERVICEMEMBER VERSION=1.0 service=HRPDB1.example.com database=ractest
   instance=ractest2 host=prod_host01_1 status=down reason=USER
   timestamp=2019-10-25 17:59:43 timezone=+00:00 db_domain=example.com
   drain_timeout=120
SERVICE VERSION=1.0 service=HRPDB1.example.com database=ractest
   instance=ractest2 host=prod_host01_1 status=down reason=FAILURE
   timestamp=2019-10-24 21:25:57 timezone=+00:00 db_domain=example.com

Note that the preceding examples normally display as one line.

FAN event types include:

DATABASE
INSTANCE
NODE
SERVICE
SERVICEMEMBER
SERVICEMETRICS

The DATABASE and INSTANCE types list the default database service as DB_UNIQUE_NAME.

All events except for NODE events include a db_domain field.

Events of SERVICEMETRICS type are load balancing advisory events.

The following table describes name-value pairs for the event parameters, and provides more information about load balancing events:

Table: Event Parameter Name-Value Pairs and Descriptions

Parameter	Description
`VERSION`	Version of the event record. Used to identify release changes.
`database`	The unique name of the database supporting the service; matches the initialization parameter value for `DB_UNIQUE_NAME`, which defaults to the value of the `DB_NAME` initialization parameter.
`instance`	The name of the instance that supports the service.
`host`	The name of the node that supports the service or the node that has stopped; matches the node name known to Cluster Synchronization Services (CSS).
`service`	The service name; matches the name of the service as listed in `DBA_SERVICES` and is domain-qualified as appropriate. Refer to the following examples: SERVICEMEMBER VERSION=1.0 service=swingbench database=orcl instance=orcl_2 host=dev_host1 status=up reason=USER card=1 timestamp=2018-05-29 17:26:37 timezone=-07:00 db_domain=SERVICEMEMBER VERSION=1.0 service=swingbench.example.com database=orcl instance=orcl1 host=dev_host1 status=up reason=USER card=2 timestamp=2018-05-03 17:29:28 timezone=-07:00 db_domain=example.comSERVICEMEMBER VERSION=1.0 service=swingbench.example.com database=orcl instance=orcl2 host=dev_host1 status=up reason=USER card=1 timestamp=2018-07-03 17:29:18 timezone=-07:00 db_domain=example.com
`status`	Values are `UP`, `DOWN`, `NODEDOWN` (this only applies to the `NODE` event type), `NOT_RESTARTING`, and `UNKNOWN`. Notes: When the node is down, the status is `NODEDOWN`, as opposed to `DOWN` for other event types. When `STATUS=NODEDOWN` and `REASON=MEMBER_LEAVE`, a node has failed and is no longer part of the cluster, or a user has stopped a node. When `STATUS=NODEDOWN` and `REASON=PUBLIC_NW_DOWN`, the node is up but it is unreachable because the public network is down because of either a failure or a user action. Multiple public networks are supported by Oracle Clusterware. The FAN event reflects this fact.
`reason`	`AUTOSTART`, `BOOT`, `DEPENDENCY`, `FAILURE`, `MEMBER_LEAVE`, `PUBLIC_NW_DOWN`, `USER`. Notes: For `DATABASE` and `SERVICE` event types, `REASON=AUTOSTART` if, when the node started, the `AUTO_START` resource attribute was set to restore, and the resource was offline before the node started. For `DATABASE` and `SERVICE` event types, `REASON=BOOT` if, when the node started, the resource started because it was online before the node started. For SRVCTL and Oracle Enterprise Manager operations, `REASON=USER` describes planned actions for such operations as draining work.
`card` (cardinality)	The number of service members that are currently active; included in all `SERVICEMEMBER UP` events. Here is an example of a `SERVICEMEMBER UP` event: `SERVICEMEMBER VERSION=1.0 service=swingbench.example.comdatabase=orcl instance=orcl_2 host=dev_host3 status=up reason=USER card=1timestamp=2018-07-12 14:46:46 timezone=-07:00 db_domain=example.com`
`incarn` (incarnation)	For `NODEDOWN` events; the new cluster incarnation. This value changes each time a member joins or leaves the cluster. Here is an example of a `NODEDOWN` event: `VERSION=1.0 event_type=NODE host=dev_host2 incarn=175615351 status=nodedownreason=member_leave timestamp=2019-10-24 05:55:06 timezone=+00:00`
`timestamp`	The time according to Oracle Clusterware that an event occurs.
`timezone`	The time zone of Oracle Clusterware where the event occurred, given as GMT +/-hh:mm.
`drain_timeout`	Time in seconds during which a service will drain. Appears with SERVICEMEMBER events
`vip_ips`	VIP on a public network that has gone down. Part of a NODE event. Here is an example of a `NODEDOWN` event: `NODE VERSION=2.0 host=my-exa status=nodedown reason=public_nw_down incarn=0 timestamp=2019-10-24 09:02:35 timezone=+00:00 vip_ips=10.1.1.94`

Some of the FAN event record parameters have values that correspond to values returned by the SYS_CONTEXT function using the default namespace USERENV, as shown in the following table:

Table: FAN Parameters and Matching Session Information

FAN Parameter	Matching Session Information
`SERVICE`	`sys_context('userenv', 'service_name')`
`DATABASE_UNIQUE_NAME`	`sys_context('userenv', 'db_unique_name')`
`INSTANCE`	`sys_context('userenv', 'instance_name')`
`CLUSTER_NODE_NAME`	`sys_context('userenv', 'server_host')`

Subscription to High Availability Events

To monitor and notify applications about services, Oracle Real Application Clusters (Oracle RAC) uses Oracle RAC Fast Application Notification (FAN).

Oracle RAC uses FAN to notify applications about configuration changes and the current service level that is provided by each instance where the service is enabled. If you are using an Oracle Call Interface (OCI) client, or an ODP.NET client to receive FAN events, then you must enable the service used by that client to access the alert notification queue by using SRVCTL with the -notification parameter.

Using Fast Application Notification Callouts

Fast Application Notification (FAN) callouts are server-side program files that Oracle RAC runs immediately when high availability events occur.

You can use FAN callouts to automate activities when events occur in a cluster configuration, such as:

Opening fault tracking tickets
Sending messages to pagers
Sending e-mail
Starting and stopping server-side applications
Maintaining an up-time log by logging each event as it occurs
Relocating low-priority services when high priority services come online

To use FAN callouts, place a program file in the Grid_home/racg/usrco directory on every node that runs Oracle Clusterware. The program file must be able to run standalone when called, with optional arguments, from another program. The following is an example of a shell script, named callout.sh, which is placed in the Grid_home/racg/usrco directory:

#! /bin/bash
FAN_LOGFILE= [your_path_name]/admin/log/'hostname'_uptime'.log
echo $* "reported="'date' >> $FAN_LOGFILE &

The previous example adds entries similar to the following in the log file, indicated by $FAN_LOGFILE in the shell script, each time a FAN event is generated:

NODE VERSION=2.0 host=my-exa status=nodedown reason=public_nw_down
incarn=0 timestamp=2019-10-24  09:02:35 timezone=+00:00 vip_ips=10.1.1.94

The contents of a FAN event record matches the current session of the user logged on to the database. The user environment (USERENV) information is also available using Oracle Call Interface (OCI) connection handle and descriptor attributes (using OCIAttrGet()). Use this information to take actions on sessions that match the FAN event data.

In general, events are only posted to user callouts on the node from which the event originated. For example, if the database on node1 goes down, then the callout is posted to node1, only. The only exceptions to this are node down and VIP down events-these events are posted to all nodes, regardless of from where they originated.