1. Real Application Clusters Administration and Deployment Guide
  2. Ensuring Application Continuity

6 Ensuring Application Continuity

Application Continuity is a feature that enables the replay, in a non-disruptive and rapid manner, of a request against the database after a recoverable error that makes the database session unavailable so an outage appears to the user as no more than a delayed processing of the request.

AC needs request boundaries, which means AC depends on Oracle or third-party pools that embed request boundaries when connections are borrowed and returned to the pool. Whereas, TAC uses provided request boundaries and creates its own. With TAC request boundaries advance automatically when there is no transaction, the session state is restorable, and cursors are closed.

To use this chapter, you should be familiar with the major relevant concepts and techniques of the technology or product environment in which you are using Application Continuity, such as Oracle RAC or Oracle Active Data Guard (Oracle ADG).

See Also:

Oracle Database High Availability Overview and Best Practices describes how you can choose and implement the level of Application Continuity protection that is the best for your applications.

Understanding Application Continuity

Application Continuity can be used to mask outages from clients and recover in-flight transactions that would otherwise be lost.

Parent topic: Ensuring Application Continuity

About Application Continuity

Application Continuity provides high availability for your application workloads.

Application Continuity is a high availability feature that masks many recoverable Oracle Database outages (when replay is successful) from applications and users by restoring the database session: the full session, including all states, cursors, variables, and the last transaction if there is one. Application Continuity addresses the problem that arises when an application is trying to access the database and the database instance becomes unavailable due to an unplanned outage or planned maintenance (such as timeout, network outage, instance failure, repair, configuration change, and patch apply). Without Application Continuity in place, database recovery does not mask outages to applications and end users. In such scenarios, developers and users must handle the exception conditions, and users can be left not knowing what happened to their funds transfers, time sheets, orders, and bill payments. Users might lose screens of uncommitted data, and must log in again and reenter that data. In the worst cases, the administrator might be forced to restart the middle tier to recover from an overwhelming number of logins.

With Application Continuity, if the database instance becomes unavailable, then Application Continuity attempts to rebuild the session and any open transactions using the correct states; and if the transaction committed and need not be resubmitted, then the successful return status is returned to the application. If replay is successful, then the request can continue safely without risk of duplication. If replay cannot restore data that the application has already processed and potentially made decisions on, then the database rejects the replay and the application receives the original error.

Client requests can contain transactional and non-transactional work. After a successful replay on Oracle Database, the application can continue where that database session left off, instead of having users left in doubt, not knowing what happened to their transactions. Recovering these client requests also helps to avoid the need to reboot mid-tier servers to recover from an overload of logins when the application comes back online. With Application Continuity, the end-user experience is improved by masking many outages, planned and unplanned, without the application developer needing to attempt to recover the request.

Application Continuity performs the recovery of in-flight transactions and database session state, while ensuring the transaction idempotence provided by Transaction Guard. Each database session is tagged with a logical transaction ID (LTXID), so the database recognizes whether each replay committed any transactions, and if it did commit any transactions, whether the work also ran to completion. While Application Continuity attempts to replay, the replay appears to the application as a delayed processing, or the application receives the commit response for the original transaction (if the last transaction had completed before the outage).

Application Continuity is supported for Oracle RAC, Oracle Active Data Guard, and DBMS_ROLLING. It is supported for Oracle Database using the multitenant architecture (with failover at the Pluggable Database level). It is not currently supported for Oracle GoldenGate, Logical Standby, third-party replication solutions, or DML redirection if using Oracle Active Data Guard.

Related Topics

Parent topic: Understanding Application Continuity

Key Concepts for Application Continuity

This section describes several terms and concepts that you must understand to use Application Continuity.

The following terms are used throughout this chapter:

Database request

A database request is a unit of work submitted to the database from the application, such as a transaction. A request typically corresponds to the SQL and PL/SQL, and other database calls, of a single web request on a single database connection. A request is generally demarcated by the calls made to check-out and check-in the database connection from a connection pool.

Request Boundaries

Request Boundaries demarcate where applications and application servers borrow and return connections from their connection pools. Request Boundaries indicate when a session is not in use. When request boundaries are visible to the database, it enables functionality such as draining for planned maintenance, load balancing, and multiplexing to be isolated at the database layer. Sessions can be re-established with no visible disruption to the application layers above.

Recoverable error

A recoverable error is an error that arises due to an external system failure, independent of the application session logic that is running, such as a lost or invalid connection. Recoverable errors occur following planned and unplanned outages of foregrounds, networks, nodes, storage, and databases. The application receives an error code that can leave the application not knowing the status of the last operation submitted. Application Continuity reestablishes database sessions and resubmits the pending work for the class of recoverable errors.

Application Continuity does not resubmit work following call failures due to nonrecoverable errors. An example of a nonrecoverable error that would not be replayed is submission of invalid data values.

Commit outcome

A transaction is committed by updating its entry in the transaction table. Oracle Database generates a redo-log record corresponding to this update and writes out this redo-log record. Once this redo-log record is written out to the redo log on disk, the transaction is considered committed at the database. From the client perspective, the transaction is considered committed when an Oracle message (called the commit outcome), generated after that redo is written, is received by the client. However, if a COMMIT has been issued, then the COMMIT failure message cannot be retrieved if it is not received by the client or the application.

Restoring original function results

Restoring original function results are non-deterministic that can obtain a new value every time they are called, and thus their results can change frequently. Restoring original function results cause a problem for replay because the results can change at replay. Consider sequence.NEXTVAL and SYSDATE, often used in key values. If a primary key is built with values from these function calls, and is used in later foreign keys or other binds, at replay the same function result must be returned.

Application Continuity enables restoring original function results at replay for granted Oracle function calls to provide opaque bind-variable consistency. If the call uses database functions for which restoring original function results are possible, including sequence.NEXTVAL, SYSDATE, SYSTIMESTAMP, and SYSGUID, the original values returned from the function processing are saved and are reapplied at replay.

Session state consistency

After a COMMIT statement has processed, if state was changed in that transaction, it is not possible to replay the transaction to reestablish that state if the session is lost. Transparent Application Continuity creates a new checkpoint of the session state to reestablish a new starting point for replay. When configuring Transparent Application Continuity, use session state consistency set to AUTO.

  • A session has dynamic state if the session state changes during the request. When using Application Continuity, failover is internally disabled until the next request begins.

  • With Transparent Application Continuity, the state is managed for you when you set the session state consistency service attribute to AUTO. The session states are tracked and verified at failover. After a disable, when session state consistency service attribute is set to AUTO, failover is re-enabled automatically when possible.

  • Use session state AUTO so that the session state is managed as the application evolves. Session state AUTO is available with TAC. When using TAC, both the client visible and server visible session states must match for a successful failover or session migration. When there is an unrestorable session state at the beginning of a request, TAC does not enable because the state cannot be proven at the beginning of the request. Use ACCHK and the protection statistics to report unrestorable session state at the beginning of the request. Use RESET_STATE to clean session state automatically by Oracle Database between requests.

  • Note:

    The service attribute values FAILOVER_TYPE = TRANSACTION with SESSION_STATE_CONSISTENCY = STATIC are no longer a supported service attribute combination.

Stateless applications

A stateless application is an application program that does not use session state in one request – such as context and PL/SQL states that were set by a prior usage of that session by another web request or similar usage. The necessary state to handle the request is contained within the request itself, whether as part of the URL, query-string parameters, body, or headers. In a cloud environment, it is preferable that applications be stateless for the sake of scalability and portability. Statelessness enables greater scalability because the server does not have to maintain, update, or communicate that session state. Additionally, load balancers do not have to consider session affinity for stateless systems. Most modern Java Web applications are stateless. The service attribute RESET_STATE is recommended for all stateless applications to prevent leakage of session state to later reuses.

Parent topic: Understanding Application Continuity

How Application Continuity Works for Applications

If a recoverable error occurs and if you enabled replay, then Application Continuity attempts recovery of the database session.

The following figure is a graphical representation of how Application Continuity works.

Figure 6-1 Application Continuity

Description of Figure 6-1 follows
Description of "Figure 6-1 Application Continuity"
To attempt to recover a database session following a recoverable error, Application Continuity performs the following steps:

Note:

The steps to recover a database session apply for both unplanned and planned outages, although specific steps vary depending on the type of outage.

  1. The client application makes a request, which is passed to a middle tier (such as the Universal Connection Pool (UCP), ODP.NET, WebLogic Server, OCI session pool, Tuxedo, or third-party pool using UCP) and forwarded to the database. The application could also make a request directly to the database using the JDBC replay driver or ODP.NET managed provider 23ai.

  2. The middle tier, or Oracle drivers (JDBC replay driver, OSI, or ODPM 23ai).

  3. A planned or unplanned DOWN Fast Application Notification (FAN) event or recoverable error is received. The driver stops the terminated session.

  4. Application Continuity begins the replay and does the following:

    1. Replaces the terminated physical session with a new clean session.

    2. Prepares for replay by using Transaction Guard to determine the outcome of the in-flight transaction, if one was open.

    3. If FAILOVER_RESTORE=LEVEL2, FAILOVER_RESTORE=LEVEL1, or FAILOVER_TYPE=AUTO, then Application Continuity restores the initial session state. Use Wallets with FAILOVER_RESTORE to restore all modifiable parameters. Application Continuity uses a label callback or initial callback if an application also sets session states that are not provided by FAILOVER_RESTORE in the callback

    4. Rebuilds the database session, recovering the transactional and non-transactional states, and validating at each step that the data and messages seen by the client driver are the same as those that the client may have seen and used to make a decision.

    5. Ends the replay and returns to run-time mode.

    6. Submits the last queued call.

      This is the last call made when the outage was discovered. During replay, only this call can run a COMMIT. A COMMIT midway through rebuilding the session terminates replay (excluding autonomous transactions).

  5. The response is returned to the application.

    If replay succeeded, then the application can continue with the problem masked. If not, then the application must handle the original error.

The behavior of Application Continuity after a communication failure depends on the Oracle products and technologies involved. For example:

  • If you use Oracle RAC or an Oracle Active Data Guard farm, then, after the connection is reestablished on another running instance, Application Continuity attempts to rebuild the session and replay the last transaction if there is one in flight.

  • If you use Oracle Active Data Guard and fail over to a standby site, then Application Continuity connects to the failover instance and attempts to rebuild the session and replay the last transaction there, if a transaction was in-flight. Application Continuity does not replay if the Oracle Active Data Guard switchover or failover has lost data.

  • If you are using Oracle RAC or Oracle RAC One Node and not using Oracle Active Data Guard, and if an outage causes a break in all public networks or causes the database or database session to shut down briefly, then Application Continuity attempts to rebuild the session and replay the last transaction (if a transaction was in flight) against the database after connectivity is restored.

Related Topics

Parent topic: Understanding Application Continuity

Support for Oracle Application Continuity and Transparent Application Continuity

Support for Application Continuity (AC) is integrated into many Oracle applications.

Application Continuity is available for general use with the following Oracle technologies:

  • Oracle JDBC Replay Driver 12c or later. This is a JDBC driver feature provided with Oracle Database 12c for Application Continuity.
  • The Oracle Database 23ai JDBC Consolidated Data Source automatically enables replay when Application Continuity is enabled. There is no longer a need to choose the replay driver.
  • Oracle Universal Connection Pool (UCP) 12c or later
  • Oracle WebLogic Server Active GridLink or third-party JDBC application servers using UCP with Oracle JDBC Replay Driver 19c or later, or JDBC Consolidated Data Source.
  • Java connection pools or standalone Java applications using Oracle JDBC Replay Driver 12c or later with Request Boundaries.
  • Applications and language drivers using Oracle Call Interface (OCI) 12c Release 2 (12.2) or later, including Python with thick mode.
  • SQL*Plus 19c or later.
  • ODP.NET Unmanaged Provider 12c Release 2 (12.2) or later (Set “pooling=true” and “Application Continuity=true” as default in 12.2 and later).
  • Oracle Data Provider for .NET (ODP.NET), Managed Driver 23ai and later.

Transparent Application Continuity (TAC) is available for general use with the following Oracle technologies:

  • Oracle JDBC Replay Driver 19c or later. This is a JDBC driver feature provided with Oracle Database 19c for Application Continuity
  • The Oracle Database 21c or later JDBC Consolidated Data Source automatically enables replay when Application Continuity is enabled. There is no longer a need to choose the replay driver.
  • Oracle Universal Connection Pool (UCP) 19c or later with Oracle JDBC Replay Driver 19c or later
  • Oracle WebLogic Server Active GridLink, or third-party JDBC application servers using UCP with Oracle JDBC Replay Driver 19c or later
  • Java connection pools or standalone Java applications using Oracle JDBC Replay Driver 19c or later
  • Oracle Call Interface (OCI) 19c or later
  • Python (python-oracledb) and Node.js (node-oracledb) drivers in thick mode
  • SQL*Plus 19.3 or later
  • ODP.NET pooled, Unmanaged Driver 18c or later (Set “pooling=true” as default in 12.2 and later)
  • OCI-based applications using OCI 19c or later
  • Oracle Data Provider for .NET (ODP.NET), Managed Driver 23ai and later

Application Continuity for Java is embedded in the Universal Connection Pool, WebLogic data sources, including non-XA and XA data sources, and is available with the thin JDBC replay driver, standalone (which is a JDBC replay driver without Oracle connection pools, such as Apache Tomcat or a custom Java connection pool). Application Continuity for OCI is embedded in SQL*Plus, OCI 12.2 or later, ODP.NET, Unmanaged Provider including Python with thick mode, (ODP.NET), Managed Provider 23ai and later, and Python when using thick mode. With AC and TAC, JDBC applications auto enable starting with Oracle Database 21c, and OCI applications starting with Oracle Database 19c (19.3).

If a connection pool or container does not use an Oracle connection pool, then many third-party applications fully support replacing the connection pool with the Universal Connection Pool. This includes IBM WebSphere and Apache Tomcat. C applications that implement their own pooling can use OCIRequestBegin and OCIRequestEnd calls. Alternatively—for Java applications, only—an application can add its own request boundaries.

Request Boundaries

Request boundaries are embedded in Oracle connection pools starting with Oracle Database release 12.1. Request boundaries are also embedded for third party Java Application Servers that are standard with JDK9 or later. When you use the Oracle connection pools, request boundaries are marked explicitly at check-out and check-in, delimiting the size of each replay. When using third-party connection pools, use UCP if Java, or use Transparent Application Continuity, or add request boundaries, or use third party Java Application Servers that are standard with JDK9 or later. Request boundaries are discovered using state tracking when using Transparent Application Continuity. This type of request boundary is called an implicit request boundary. This functionality is available starting with the Oracle Database 19c Java replay driver, and the Oracle Database 19c OCI driver, which includes open source and ODP.NET Unmanaged Provider.

Note:

For Oracle Database 18c ONLY: Java requires an initial beginRequest. This is not needed when using later versions of the Java replay driver.

Related Topics

Parent topic: Understanding Application Continuity

Restrictions and Other Considerations for Application Continuity

Be aware of these restrictions and considerations when using Application Continuity.

Application Continuity is not supported for:
  • JDBC OCI driver (type 2)
  • OLE DB
  • ODBC
  • OCCI
  • Pre-compilers including C, COBOL, and FORTRAN
  • XA
  • python-oracledb in thin mode
  • node-oracledb in thin mode

Note:

If applications in these environments require support for planned maintenance, then consider draining using connection tests.

Note:

Oracle Data Provider for .NET (ODP.NET), Unmanaged Driver is deprecated in Oracle Database 23ai.

ODP.NET provides ADO.NET-based data access to Oracle Database. There are two primary Oracle data access drivers for Microsoft .NET Framework: ODP.NET, Managed Driver and ODP.NET, Unmanaged Driver. In Oracle Database 23ai, ODP.NET, Managed Driver supports all major features available in ODP.NET, Unmanaged Driver with the same application programming interfaces and configuration settings. Code migration from unmanaged ODP.NET to managed ODP.NET is straightforward for the vast majority of existing .NET applications. Oracle recommends that you migrate existing unmanaged ODP.NET applications to ODP.NET, Managed Driver. The ODP.NET, Unmanaged Driver can be desupported in a future release.

For OCI and ODP.NET Unmanaged drivers, Application Continuity does not work for ADTs, Advanced Queues (AQ), and some LOB APIs. These exclusions do not apply to Java.

For applications using JDBC, there is no support for oracle.sql deprecated concrete classes: OPAQUE, ANYDATA, or STRUCT.

If a statement cache at the application server level is enabled (for example, the WebLogic or third-party application server statement cache), this cache must be disabled when replay is used. Instead, configure the JDBC statement cache, which supports Application Continuity and is optimized for JDBC and Oracle Database (oracle.jdbc.implicitstatementcachesize=nnn).

Note the following restrictions when replay of transactions can occur:

  • Replay is disabled intentionally if a request issues an ALTER SYSTEM or ALTER DATABASE statement.

  • Replay is disabled at a request level for ALTER SESSION statements that are deemed unsafe to rebuild the session. This includes SERIALIZABLE mode. Starting with Oracle Database 23ai, statements for events and disabling and enabling COMMIT IN PROCEDURE and GUARD are supported.

    Note that other ALTER SESSION statements at an application level are supported for replay. These include statements for globalization support (NLS) settings, stored and private outlines, setting the container (CDB/PDB), SQL trace, and PL/SQL warnings.

  • The replay target database must be in the same database cluster (Oracle RAC, Oracle Data Guard, Oracle Active Data Guard, or Oracle Multitenant) as the source database. To protect the integrity of business transactions, Application Continuity does not replay if the target is a different database. Application Continuity also does not replay if the target database is the same as the source database (or Pluggable Database) but with data loss, such as one flashed back, recovered incompletely by media recovery, or opened by Oracle Data Guard to an earlier point in time.

  • For streams arguments, replay is on a "best effort" basis. For example, if the application is using physical addresses, the address has gone with the outage and cannot be repositioned. JDBC stream setters (such as setBinaryStream), for example, cause replay to be disabled.

  • Application Continuity is not currently supported for Oracle GoldenGate, Logical Standby, third-party replication solutions, or DML redirection if using Oracle Active Data Guard.

  • Replay does not support ISOLATION_LEVEL=SERIALIZABLE.

Note:

If you are creating clones of databases by splitting disk images (for example, BCVs) or by cloning so it is a "different" database for the purpose of making a logical standby or logical copy that is not a physical or Oracle Active Data Guard database, then the nid utility must be used to change the DBID to differentiate the databases.

Related Topics

Parent topic: Understanding Application Continuity

Application Continuity for Various Applications

Transparent Application Continuity covers applications that belong to three different groups, automatically tracked by the state tracking system.

Parent topic: Understanding Application Continuity

Applications That Use Containers with Request Boundaries

Applications that use containers with request boundaries enable Application Continuity to manage replay between explicit boundaries.

A request boundary is a tag that marks the beginning and end of a database request. Beginning with Oracle Database 12c release 2 (12.2.0.1), connection pools that embed request boundaries include Oracle Universal Connection Pool, all WebLogic server data sources, Tuxedo, Oracle Call Interface, ODP.NET Unmanaged Provider, and standard third-party application servers and standalone Java pools that use the JDBC drivers PooledConnection interface, in addition to SQL*Plus.

When Oracle Database is aware of request boundaries:

  • The database can process web requests effectively and with no performance overhead, including when to attach and release connections. It can multiplex, drain, rebalance, shed, and allow complex states inside requests. Without request boundaries, the lower layers of the database are not aware of web requests. Subsequently, the database relies on Oracle Client actions, advisory methods and heuristics, such as fast connection failover, connection validation, and state advice.

  • The length of replay is limited to the initial state, followed by the user calls in that request less those that are purged by Application Continuity. Request boundaries enable you to control the length of replay. You can also determine where to drain for planned maintenance (at the end of the request), and where to fail over for planned maintenance (at the beginning of the request).

  • When using Transparent Application Continuity with Java and Oracle Database 18c ONLY: Java requires an initial beginRequest (and only for the first request boundary). This is not needed in later versions.

  • When using Application Continuity, the replay driver detects safe places to move the request boundaries forward automatically. This feature is available when FAILOVER_TYPE is set to AUTO.

  • Applications deployed using middle-tier containers that set request boundaries have access to the full set of transparency features that the database server provides. The database detects when a client sets request boundaries and uses the boundaries to mark safe points for draining, failover, concentration, and throughput measures.

Request boundaries enable an application to use all complex, non-transactional session states within a request. The request boundary specification requires that these states are not dependent across boundaries.

Applications that are Database Agnostic

Database-agnostic applications set a state when the connection is established, and do not change non-transactional session states again, or change it rarely.

Database-agnostic applications (applications with no request boundaries) set basic, non-transactional states. These applications do not use features or sequences proprietary to Oracle Database. For these applications, Application Continuity identifies implicit boundaries. These applications often set state once when a connection is created, and then do not change state again, or change the state infrequently. This category of applications includes those applications that use anonymous PL/SQL that does not create server-side session states.

When using Transparent Application Continuity with Oracle Database 19.3 or later releases, explicit request boundaries are not required, but they are recommended. (For Oracle Database 18c ONLY: Java requires an initial beginRequest.) This allows support for SQL*Plus and third-party connection pools. When explicit request boundaries are present, they are used. Explicit request boundaries continue to be needed for Application Continuity. Oracle recommends that you return your connections to the connection pools, when not in use.

Transparent Application Continuity

A version of Application Continuity for simple applications that uses discovery to detect request boundaries.

Parent topic: Ensuring Application Continuity

About Transparent Application Continuity

In Oracle Database both Application Continuity and Transparent Application Continuity transparently tracks and records session and transactional state so that a database session can be recovered following recoverable outages.

Transparent Application Continuity uses this information to discover boundaries for basic applications. Recovery of the user database session is done safely and with no need for a DBA or developer to have any knowledge of the application or make application code changes. Transparency is achieved by using a state-tracking infrastructure that categorizes session state usage as an application issues user calls.

Transparent Application Continuity is enabled when FAILOVER_TYPE=AUTO.

You can enable Application Continuity or Transparent Application Continuity to protect applications during planned maintenance and when unplanned outages occur. For planned maintenance, database sessions that reach a safe place (such as a connection test or a known recoverable point) are drained at the database. For database sessions that do not drain, the database determines when to fail the database session over and triggers Transparent Application Continuity or Application Continuity to do so. Application Continuity hides unplanned outages for Java-based applications, OCI and ODP.NET applications including SQL*Plus, all Oracle connection pools, Tuxedo, WebLogic Server, and third-party application servers using Universal Connection Pool.

Starting with Oracle Database 23ai, Transparent Application Continuity (TAC) includes Transparent Application Failover (TAF) style cursors, known as Resumable Cursors. TAC supports basic SELECT cursors in FETCH. To support SELECT cursors in FETCH, not in transactions, set session state consistency to AUTO. In addition, the database repositions these cursors at failover for much faster failover. TAC provides higher protection because cursors can remain open across transaction boundaries. This provides better-unplanned outages and planned maintenance support for applications. It allows TAC to re-enable faster to provide higher protection and achieves speedier failover.

For unplanned outages, Transparent Application Continuity is invoked for outages that result in recoverable errors, typically related to underlying software, foreground, hardware, communications, network, or storage layers, hiding most failures from applications and users.

TAC supports basic applications using SQL and basic ALTER SESSION statements. With Transparent Application Continuity, DBAs no longer need to have knowledge of an application to:

  • Restore preset states—At run time, Transparent Application Continuity records the initial preset session states, monitors further states, and records session signatures sufficient to detect deviation in the state of a session at failover for monitored states. At failover, Transparent Application Continuity restores the preset session states before replay starts, and verifies that these session states fully match the original before replay starts. This also allows for session state that has been restored using both Application Continuity and other mechanisms, such as log-on triggers, labels, and connection call backs. You will continue to add log-on triggers, call backs, or labels if the state is outside the preset states.

  • Recognize and disable application-level side effects when recovering a session—Transparent Application Continuity records the transactions and session state from the client, which is referred to as capture, to enable replay. During normal run-time, Transparent Application Continuity detects side effects, or changes that occur as a result of the transaction, but are not part of the transaction itself. The type of side effect is distinguished between those that relate to an application’s logic and those that are internal, relating to database housekeeping. For applications that use statements that have side effects, capture is disabled when the statement is running. Once a new request starts, capture is re-enabled automatically.

  • Restoring original function values for owned functions—Such functions can return a new value each time they are run. Oracle provides support for keeping the original results of these functions SYSDATE, SYSTIMESTAMP, LOCAL_TIMESTAMP, CURRENT_TIMESTAMP, SYS_GUID, and sequence.NEXTVAL. If the original values are not kept and if different values are returned to the application at replay, then Transparent Application Continuity rejects replay. Use grants to keep your sequences, dates, and times. When an application is using its own schema, you can assign the grants for keeping to a role and then grant this role to users.

  • Know about request boundaries—Request Boundaries demarcate where applications and application servers borrow and return connections from their connection pools. For applications using Application Continuity with the JDBC thin driver (beginning with Oracle Database 18c), OCI, and ODP.NET Unmanaged Provider (beginning with Oracle Database 19c release 19.3), DBAs do not need to know about request boundaries but when they are in use, Transparent Application Continuity takes advantage of them. As a best practice, use explicitly delineated request boundaries in your application because it is not always possible for the database to identify a checkpoint where a request boundary can be inserted.

    Using Transparent Application Continuity, the server and the drivers are tracking transaction and session state usage. This allows the driver to detect implicit request boundaries. For an implicit boundary with SSS cursors, no objects may be open, cursors must have been returned to the statement cache, no transaction may be active, and the session state must have been recognized as fully restorable. The driver either discards the currently tracked information, and starts tracking again from this point, or it re-enables tracking if there had been a disabling event. On the next call to the server, the server verifies and, if applicable, creates a request boundary where there was previously no explicit boundary.

  • To use Transparent Application Continuity for more complex applications with unrestorable session states, use a pool like you use for Application Continuity and set RESET_STATE to clean the session state between usages. RESET_SATE is suitable for applications that must not leak session state from one pooled usage to a later usage.

Parent topic: Transparent Application Continuity

Deciding on Transparent Application Continuity

Transparent Application Continuity (TAC) is an application failover solution that is safe to enable by default.

TAC provides failover for planned events, called planned failover, as well as for unplanned events. Refer to Oracle Database High Availability Overview and Best Practices to know more about planned and unplanned failover and Oracle Maximum Availability Architecture (MAA) levels.

A successful failover is a delayed execution, with no error returned to the application. If TAC cannot produce the same logical session, TAC returns an error to the application. The correctness guarantees for a successful failover includes the following, but not limited to:
  • Within the SLA set for the service under replay_initiation_timeout- Failover starts if the timestamp for the failover is less than or equal to replay_initiation_timeout, where replay_initiation_timeout is set at the start of the request.
  • Same database forward in time- The failover checks that the target database is at the correct database (DBID or PDBID) and is at the same time or is forward in time (called SCN ancestry) from the original.
  • No duplication of transactions- The failover checks whether the last request committed and if the request committed returns COMMITTED to the client.
  • Same session state at the start of replay as the original session- If replay is needed, after restoring the session state using FAILOVER_RESTORE and before replaying, the session states are checked to ensure that they match the original session at the start of the current request.
  • Same results that the application has already processed- Cursors are repositioned and the results of that repositioning are checked to be the same as before. This is similar to TAF with the difference that the reposition in Oracle Database 23ai is at the server with the rows no longer returned to the client. To provide higher success of same results, TAC keep original function results for sequences, date, time, and GUIDs.
TAC does this automatically for your application and declares success when it is correct to do so. This enables application developers to focus on business functionality while TAC handles the failover in the best possible way.

Know about planned failover?

With TAC enabled, the planned failover engine in the Oracle Database looks for places where it is confident to failover and initiates the failover. This bounds your drain time and aims to prevent sessions from being aborted. This is automatic with both TAC and AC, but it is frequently with TAC.

Difference between TAC and AC (original)

AC needs request boundaries, which means AC depends on Oracle or third-party pools that embed request boundaries when connections are borrowed and returned to the pool. Whereas, TAC uses provided request boundaries and creates its own. With TAC request boundaries advance automatically when there is no transaction, the session state is restorable, and cursors are closed.

AC provides similar correctness guarantees for a successful failover as TAC, with a small variation. AC enforces same client visible session states at the start of replay as the original session. AC replays side effects, such as sending email by default. TAC does not replay side effects as the default. The action for side effects is customizable in Oracle Database 23ai.

TAC request boundaries

TAC creates its own request boundaries, when:
  • There is no transaction.
  • The session state is restorable (no PL/SQL global variables, temporary tables, OJVM, and session duration LOBs)
  • The cursors are resumable in Oracle Database 23ai or closed

Cleaning sessions for next usage

The service attribute RESET_STATE cleans your session state and cursors at the end of each request so these states do not leak to later requests. RESET_STATE saves you work and is future proofed, meaning that later changes to your application are also cleaned. It allows TAC to re-enable if TAC was unable to clean a session due to not restorable states.

Parent topic: Transparent Application Continuity

Using Transparent Application Continuity in Oracle Cloud Environments

Transparent Application Continuity is enabled by default on TP and TPURGENT services in an Oracle Cloud environment for Oracle Autonomous Database–Dedicated, and is available for Oracle Autonomous Database–Serverless.

In Oracle Cloud environments, the use of FAILOVER_RESTORE and wallets means that you should not have to add callbacks to set initial state, as was required for Transparent Application Failover (TAF) and Application Continuity in Oracle Database releases prior to 12.2.

There are two features that work together to enable Transparent Application Continuity automatically:

  • For planned outages, sessions that reach a safe place for transactions are drained from the instance and automatically failed over to another instance. For sessions that do not drain, Oracle Database determines where to fail over the session and invokes Application Continuity to fail over the session.
  • For unplanned outages, Transparent Application Continuity transfers the user sessions to a remaining instance, hiding the outage from users automatically without having to understand or change the application.

Enable Application Continuity on Your Service

You can change the service attributes specified on your service by using the generic package DBMS_APP_CONT_ADMIN. Use this procedure to enable Application Continuity or Transparent Application Continuity, or to disable failover. New sessions will use the new failover type. You must have the PDBADMIN user permissions to use these procedures. Use the full service name in these examples.

  • Use the dbms_app_cont_admin.enable_tac('TPURGENT') procedure to enable Transparent Application Continuity for your service:
    SQL> execute dbms_app_cont_admin.enable_tac('TPURGENT');
  • Use the dbms_app_cont_admin.enable_ac('TPURGENT') procedure to enable Application Continuity for your service:
    SQL> execute dbms_app_cont_admin.enable_ac('TPURGENT');
  • Use the dbms_app_cont_admin.disable_failover('HIGH') procedure to disable failover for your service:
    SQL> execute dbms_app_cont_admin.disable_failover('HIGH');
  • Use the dbms_app_cont_admin.acchk_set procedure to enable ACCHK for your service:
    SQL> execute dbms_app_cont_admin.acchk_set(true);
  • Use the dbms_app_cont_admin.acchk_set_filter procedure to set ACCHK filter for your service:
    SQL> execute dbms_app_cont_admin.acchk_set_filter(DBMS_APP_CONT_ADMIN.SERVICE_FIL
TER, 'TPURGENT');
  • Use the dbms_app_cont_admin.enable_reset_state procedure to enable RESET STATE for your service:
    SQL> execute dbms_app_cont_admin.enable_reset_state('TPURGENT', 'LEVEL1');
  • Use the dbms_app_cont_admin.set_draining procedure to configure draining for your service:
    SQL> execute dbms_app_cont_admin.set_draining('TPURGENT', 300, 'IMMEDIATE');

Parent topic: Transparent Application Continuity

Configuring Application Continuity

Application Continuity (AC) requires using an Oracle Pool or an application that allows Transparent Application Continuity (TAC) to discover request boundaries.

Parent topic: Ensuring Application Continuity

Overview of Application Continuity Configuration Tasks

The Application Continuity features in various Oracle applications are used automatically if you set the required service attributes.

The main actions for ensuring transparent replay for an application are the following:

  1. Ensure that you have the necessary CPU and memory resources.

    • CPU: Application Continuity is managed on the client and server sides and requires minimal CPU overhead to operate.

      At the client, CPU is used to build proxy objects and for garbage collection (GC).

      At the server, CPU is used for validation. CPU overhead is reduced for platforms with current Intel and SPARC chips where validation is assisted in the hardware.

    • Memory: When using Application Continuity, the replay driver requires more memory than the base driver because the calls are retained until the end of a request. At the end of the request, the calls are released to the garbage collector. This action differs from the base driver that releases closed calls.

      The memory consumption of the replay driver depends on the number of calls per request. If this number is small, then the memory consumption of the replay driver is less, and comparable to the base driver.

      To obtain the best performance, you must set the same value for both the -Xmx and -Xms parameters on the client. For example, if there is sufficient memory, then allocate 4 to 8 GB (or more) of memory for the Virtual Machine (VM), for example, by setting -Xms4g for 4 GB. If the -Xms parameter has a lower value, then the VM also uses a lower value from the operating system, and performance might suffer and garbage collection operations increase.

  2. Determine whether the application borrows and returns connections from the connection pool, for example WebLogic Server Pool, Universal Connection Pool, OCI Session Pool, Oracle Tuxedo request, or ODP.NET connection pool, for each request, or whether to add beginRequest and endRequest APIs to the application's own connection pool to identify request boundaries for Java, only.

    Caution:

    Do not use the beginRequest and endRequest Java API calls or the OCIRequestBegin and OCIRequestEnd OCI API calls anywhere other than at request boundaries (borrow and return connections from your connection pool). endRequest indicates that the request is complete, and that it is now stateless. Replay starts from the next beginRequest. If there is prior state, it must be reestablished using FAILOVER_RESTORE or callback.

  3. Application Continuity replays all states in a request. If the application sets states before vending connections, FAILOVER_RESTORE or a callback is needed. When using Oracle WebLogic Server or the Universal Connection Pool, use FAILOVER_RESTORE, connection labeling, or triggers. When using Oracle Call Interface (OCI) session pool, Oracle Tuxedo or ODP.NET with Oracle Database 18c or later clients, use FAILOVER_RESTORE, and only add the Transparent Application Failover (TAF) callback if it is needed. The labeling is used for both runtime and replay. Oracle strongly recommends setting FAILOVER_RESTORE to AUTO, LEVEL1, or LEVEL2 with wallets, to use database templates.

  4. Determine whether the application requires, and therefore needs to configure keeping original values for, SYSDATE, SYSTIMESTAMP, and SYS_GUID and sequences during failover.

  5. Assess the application style for the session_state_consistency value, and set the appropriate value on the service:

    • If session_state_consistency is set to AUTO, then Transparent Application Continuity monitors the session state and decides what to do. If you are unsure about state usage or know that states can change in the future, then use Transparent Application Continuity. See the list of preset session states because you may need to restore additional preset states. For .NET applications, use ODP.NET, Managed Driver 23ai or later, or ODP.NET Unmanaged Provider or later.

      Use session state AUTO so that the session state is managed as the application evolves. Session state AUTO is available with TAC. When using TAC, both the client visible and server visible session states must match for a successful failover or session migration. When there is unrestorable session state at the beginning of a request, TAC does not enable as the state cannot be proven at the beginning of the request. Use ACCHK and the protection statistics to detect unrestorable session state at the beginning of request. Use RESET_STATE to clean session state automatically by Oracle Database between requests.

    • If session_state_consistency is set to DYNAMIC, then the application changes the environment or settings during the request. Replay is disabled after the first COMMIT until the beginning of the next request. DYNAMIC is the default mode, appropriate for most applications.

    Note:

    The service attribute values FAILOVER_TYPE = TRANSACTION with SESSION_STATE_CONSISTENCY = STATIC are no longer a supported service attribute combination.

  6. Determine if any requests in the application should not be replayed.

    For example, replay may need to be disabled for requests using external PL/SQL actions.

  7. Follow these configuration guidelines:

    • For Java, use Oracle Database 12c release 1 (12.1.0.1), or later. For OCI-based applications, use Oracle Database 12c release 2 (12.2), or later.

    • For .NET applications, use ODP.NET, Managed Driver 23ai or later, or ODP.NET, Unmanaged Provider 12.2, or later, connecting to an Oracle Database 12c Release 2 (12.2) or later. By default, Application Continuity is enabled for ODP.NET applications in this configuration. When using OCI-based applications that do not use the OCI Session Pool, including SQL*Plus, use Transparent Application Continuity that adds boundaries for you.

      Note:

      Oracle Data Provider for .NET (ODP.NET), Unmanaged Driver is deprecated in Oracle Database 23ai.

    • For Java-based applications, use Universal Connection Pool 12.1 (or later) or WebLogic Server 12.1.2 (or later) configured with the JDBC replay data source; or for third party applications, including third party JDBC pools, use JDBC replay driver. For IBM WebSphere, Apache Tomcat, Red Hat Spring, and custom Java solutions, the most effective solution is to use UCP as the pooled data source.

      Custom Java pools and standalone Java applications can also use the JDBC replay data source directly. When using custom Java pools and standalone applications, Oracle recommends that you use Transparent Application Continuity which adds boundaries for you. You can also add beginRequest and endRequest Java APIs to your application.

    • If the application does not borrow and return from the Oracle connection pools, explicitly mark request boundaries. For example, if using custom JDBC pools, or other pools, Oracle recommends that you use Transparent Application Continuity which adds boundaries for you. You can also add beginRequest and endRequest Java APIs to your application. These APIs can also be used for standalone JDBC applications without a connection pool.

    • Enable FAN for fast interrupt on errors. This is essential to eliminate a TCP delay occurring before the failover can start. In 12.2 FAN is built into the JDBC and OCI drivers and is on by default for Java.

    • Use a database service to connect; never use a SID or an instance name, or the administration service that is the DB_NAME or DB_UNIQUE_NAME.

    • Use a connection string that sets retries for new incoming connections and a delay between these retries.

    • For the service, set FAILOVER_TYPE to TRANSACTION for Application Continuity or set FAILOVER_TYPE to AUTO for Transparent Application Continuity. Set COMMIT_OUTCOME to TRUE and, for OCI FAN, set NOTIFICATION to TRUE. Optionally to find the best connections to use, set GOAL to SERVICE_TIME and CLB_GOAL to LONG.

    • Use the statistics for request boundaries and protection level to monitor the level of coverage. If you need more details, then use Application Continuity Check Coverage (with the ORAchk utility) to report the percentage of requests that are fully protected by Application Continuity, and the location of those requests that are not fully protected. Use this coverage check before deployment and after application changes. Developers and management will know how well protected an application release is from failures of the underlying infrastructure. If there is a problem, then it can be fixed before the application is released, or waived knowing the level of coverage.

Related Topics

Parent topic: Configuring Application Continuity

Configuring Connections for High Availability and Application Continuity

These are general recommendations for configuring the connections used by applications for high availability.

If you are using Java, then you must use the oracle.jdbc.replay.OracleDataSourceImpl, oracle.jdbc.replay.OracleConnectionPoolDataSourceImpl, or oracle.jdbc.replay.driver.OracleXADataSourceImpl data source to obtain JDBC connections. These data sources support all the properties and configuration parameters of all the Oracle JDBC data sources, for example, the oracle.jdbc.pool.OracleDataSource.

For OCI based applications including SQL*Plus and ODP.NET, the OCI driver 12.2, and later, supports Application Continuity.

You must remember the following points while using the connection URL:

  • If the REMOTE_LISTENER setting for the database does not match the addresses in the ADDRESS_LIST at the client, then it does not connect, showing services cannot be found. So, the REMOTE_LISTENER setting for the database must match the addresses in the ADDRESS_LIST at the client:

    • If the connect string uses the SCAN Name, then REMOTE_LISTENER must be set to the SCAN name.
    • If the connect string uses an ADDRESS_LIST of host VIPs, then REMOTE_LISTENER must be set to an address list that includes all SCAN VIPs and all host VIPs

    Note:

    Use SCAN for location independence, to avoid having to reconfigure the client when you add or delete nodes, or when databases change to running on different nodes.

  • Set RETRY_COUNT, RETRY_DELAY, CONNECT_TIMEOUT, and TRANSPORT_CONNECT_TIMEOUT parameters in the connection string. These settings improve acquiring new connections at runtime, at replay, and during work drains for planned outages.

    The CONNECT_TIMEOUT parameter is equivalent to the SQLNET.OUTBOUND_CONNECT_TIMEOUT parameter in the sqlnet.ora file and applies to the full connection. The TRANSPORT_CONNECT_TIMEOUT parameter applies per address.

  • Set CONNECT_TIMEOUT to a high value to prevent an overabundance of log ins. Low values can result in log in storms to the application. Do not set (RETRY_COUNT+1)*RETRY_DELAY or CONNECT_TIMEOUT larger than your response time SLA. The application must either connect or receive an error within the response time SLA.

  • You are not recommended to use Easy*Connect for High Availability deployments.

Example 6-1 Recommended TNS Entry for High Availability

The following is an example of a Transparent Network Substrate (TNS entry). This is the required TNS format for Oracle Notification Service (ONS) to be auto configured. ONS is the transport system used for Fast Application Notification (FAN). Oracle recommends using FAN with Application Continuity to provide fast outage detection.

myAlias=(DESCRIPTION= 
   (CONNECT_TIMEOUT=90)(RETRY_COUNT=30)(RETRY_DELAY=3)(TRANSPORT_CONNECT_TIMEOUT=3)
   (ADDRESS_LIST=
      (LOAD_BALANCE=ON)
      (ADDRESS=(PROTOCOL=TCP)(HOST=RAC-scan)(PORT=1521)))
   (ADDRESS_LIST=
      (LOAD_BALANCE=ON)
      (ADDRESS=(PROTOCOL=TCP)(HOST=DG-Scan)(PORT=1521)))
   (CONNECT_DATA=(SERVICE_NAME=service_name)))

Related Topics

Parent topic: Configuring Application Continuity

Configuring Oracle Database for Application Continuity

Before you can use Application Continuity, you must ensure that your system is configured correctly.

Your Oracle Database configuration must include the following to use Application Continuity:

  • If you are using Oracle Real Application Clusters (Oracle RAC) or Oracle RAC One Node, Oracle Data Guard, or Oracle Active Data Guard, then ensure that Fast Application Notification (FAN) is configured with Oracle Notification Service (ONS) to communicate with pools and drivers that are Oracle Database 12c or later.

  • Set the service attributes on the service for replay and load balancing. For example, set:

    • FAILOVER_TYPE = AUTO | TRANSACTION: Use FAILOVER_TYPE=AUTO for Transparent Application Continuity or FAILOVER_TYPE=TRANSACTION for Application Continuity. This attribute enables the replay functionality for the replay drivers and Application Continuity. Oracle drivers keep track of all replayable statements issued during a database session. If all of the statements are replayable, and any in-flight transactions did not commit or the session is in conversation, then Oracle replays the uncommitted work following a planned or unplanned database outage. This mode re-establishes transactional and non-transaction states automatically with no additional application steps.

    • REPLAY_INITIATION_TIMEOUT = n: For setting the duration, in seconds, to allow replay to start. For example, you might set the value of n to 300.

    • FAILOVER_RETRIES = 3: For specifying the number of connection retries for each replay. Oracle recommends that you set retry_count in the tnsnames.ora file, rather than on the service.

    • FAILOVER_DELAY = 3: For specifying the delay in seconds between connection retries. Oracle recommends that you set retry_count in the tnsnames.ora file, rather than on the service.

    • COMMIT_OUTCOME = TRUE: If you are using Transaction Guard, the COMMIT_OUTCOME service parameter determines whether the transaction commit outcome is accessible after the COMMIT has run and an outage has occurred. Starting with Oracle Database 23ai, COMMIT_OUTCOME defaults to using Native Transaction Guard. Native Transaction Guard is used whenever the XID can be sent to the driver in advance of the COMMIT. While Oracle Database has always made the COMMIT action durable, Transaction Guard makes the outcome of the COMMIT durable.

    • FAILOVER_RESTORE = AUTO | LEVEL1 | LEVEL2: Use FAILOVER_RESTORE=AUTO or FAILOVER_RESTORE=LEVEL1 for Transparent Application Continuity and FAILOVER_RESTORE=LEVEL1 | LEVEL2 for Application Continuity. To automatically restore client states that are preset on the connection pool or driver before replay begins—including NLS states, TAGS (MODULE, ACTION, ECID, CLIENT_ID, CLIENT_INFO) states, and all server modifiable session states that use FAILOVER_RESTORE=LEVEL1 | LEVEL2 with Wallets.

Caution:

Do not use the default database service corresponding to the DB_NAME or DB_UNIQUE_NAME. Also, do not use the default database service for high availability, because this service cannot be enabled or disabled, and cannot be relocated on Oracle RAC or switched over to Oracle Data Guard. This service is reserved for Oracle Enterprise Manager Cloud Control (Cloud Control) and for DBAs.

Parent topic: Configuring Application Continuity

Establishing the Initial State Before Application Continuity Replays

At the beginning of replay, Application Continuity restores the initial session state.

The topics in this section describe how to configure the service and database to successfully restore the session state before replaying.

Parent topic: Configuring Application Continuity

Checking Initial States for Application Continuity

Learn about the session state that is restored.

If your application sets an initial state for the connection before allowing applications to use the connection, then Application Continuity must establish this initial state before replay starts. With common states, which are listed in the topics in this section, FAILOVER_RESTORE restores the states. However, you must review the topics that describe common states. If the states that your application presets are not listed, and the application needs initial states, then you must add an additional callback.

Examples of states that can be preset include:

  • PL/SQL package state
  • NLS Setting
  • Optimizer setting

During a request, Application Continuity reestablishes the entire state for the request. This prerequisite is for the initial state before Application Continuity starts replaying.

A callback is not required if FAILOVER_RESTORE restores all required states, which is the case for most applications.

See Also:

Oracle Database Release Notes for your platform, because more parameters are restored in each release

Parent topic: Establishing the Initial State Before Application Continuity Replays

FAILOVER_RESTORE

Setting FAILOVER_RESTORE to LEVEL1 (for AC with older 19c versions), LEVEL2 (for AC with database templates), or LEVEL1 or AUTO (for TAC) automatically restores all modifiable parameters before replaying the request.

FAILOVER_RESTORE is a setting on your service. Available with Oracle Database 12.2 and later, FAILOVER_RESTORE automatically restores all session states available for your application at the client-side.

Oracle recommends setting FAILOVER_RESTORE to LEVEL2, LEVEL1, or AUTO for all applications.

Refer to States Restored with FAILOVER_RESTORE for the client-side session states that are restored.

States Restored with FAILOVER_RESTORE

The following session states are restored when FAILOVER_RESTORE is set to LEVEL1, LEVEL2, or AUTO, with or without wallets.

Session States That Are Restored

  • NLS_CALENDAR
  • NLS_CURRENCY
  • NLS_DATE_FORMAT
  • NLS_DATE_LANGUAGE
  • NLS_DUAL_CURRENCY
  • NLS_ISO_CURRENCY
  • NLS_LANGUAGE
  • NLS_LENGTH_SEMANTICS
  • NLS_NCHAR_CONV_EXCP
  • NLS_NUMERIC_CHARACTER
  • NLS_SORT
  • NLS_TERRITORY
  • NLS_TIME_FORMAT
  • NLS_TIME_TZ_FORMAT
  • TIME_ZONE
  • NLS_TIMESTAMP_FORMAT
  • NLS_TIMESTAMP_TZ_FORMAT
  • CURRENT_SCHEMA
  • MODULE
  • ACTION
  • CLIENT_ID
  • AUTOCOMMIT states (for Java and SQL*Plus)
  • CONTAINER (PDB) and SERVICE
  • ROLES (excludes secure roles, which continue to require a call back)
  • ROW_ARCHIVAL
  • EDITION
  • ERROR_ON_OVERLAP_TIME
  • SQL_TRANSLATION_PROFILE
  • CLIENT_INFO. (JDBC)
Full FAILOVER_RESTORE with Database Templates

Oracle Database 23ai supports database templates. A database template is a set of session states with their values that describe a database session.

Database templates are created or assigned by the database automatically as the session state changes. TAC and AC use templates to restore the session state at replay.

Using database templates, a larger subset of sessions failover during planned maintenance and unplanned outages because their session states are restored and verified automatically, and clients use less memory. Database templates are enabled by default for TAC and you are recommended to use FAILOVER_RESTORE=LEVEL2 for AC.

At failover, FAILOVER_RESTORE restores session parameters that were altered in your session. Examples of session parameters restored include optimizer_capture_sql_plan_baselines and create_stored_outlines that were set in the session.

If you are already using a logon trigger, connection label, or callback to restore session parameters, you can continue to use them. Labels and callbacks are fully supported with and without extended FAILOVER_RESTORE. Using extended FAILOVER_RESTORE has the advantage that you do not need to update it as the application changes.

To use this feature, you must set FAILOVER_RESTORE to LEVEL2 or AUTO and ensure that the dictionary credentials are encrypted on your system. You can also set FAILOVER_RESTORE to LEVEL1 for AC and TAC that also restores all modifiable session parameters when wallets are used.

There are two methods of adding the wallet or keystore for dictionary credentials encryption:

Note:

You must use either one of the following methods to configure a keystore for FAILOVER_RESTORE.

  • Recommended: Use the WALLET_ROOT database instance initialization parameter to specify the wallet location. Using an initialization parameter for the wallet location ensures consistency across Oracle Real Application Clusters (Oracle RAC) and Oracle Data Guard. This method requires a rolling restart of the database.

  • Modify the sqlnet.ora file in your TNS_ADMIN directory on the database server to point to the wallet location. This method does not require a database restart,unless your database runs on the Microsoft Windows operating system. You are responsible for ensuring that the sqlnet.ora files are consistent in all ORACLE_HOME directories. Also, the sqlnet.ora might require additional maintenance when performing database upgrades.

FAILOVER_RESTORE with Database Templates

Starting with Oracle Database 23ai, Application Continuity uses database templates to checkpoint session state, restore the session state at the start of replay, and support session migration during planned maintenance.

Database templates restore server-side and client-visible session states at the beginning of the Application Continuity replay, thus increasing Application Continuity protection. To use this feature set FAILOVER_RESTORE=AUTO for Transparent Application Continuity or set FAILOVER_RESTORE=LEVEL2 for Application Continuity.

FAILOVER_RESTORE with database templates improves coverage for applications with a wider set of session states and lowers client memory usage. It enables you to migrate more sessions during planned maintenance and to have a higher number of successful and faster failovers during unplanned outages as protection by Application Continuity is higher.

Database templates for Application Continuity ensures the migration of sessions containing server-side session states that have not reached a boundary to drain during the draining period, and improves the Application Continuity protection for unplanned outages. Database templates for Application Continuity enables migration of sessions using secure and large session states for planned maintenance and for load balancing.

To enable this feature, you must set FAILOVER_RESTORE to LEVEL2 or AUTO. To retain the Oracle Database 19c restore functionality, set FAILOVER_RESTORE to LEVEL1 for both Application Continuity and Transparent Application Continuity. To restore all session states at failover, the database must have encryption enabled, for example using TDE wallets.

Configuring a Keystore using WALLET_ROOT for FAILOVER_RESTORE

Use these steps to configure encryption of dictionary credentials by using a software keystore (wallet) and Transparent Data Encryption (TDE) for use with FAILOVER_RESTORE.

  1. If you are using Oracle Autonomous Database, you do not need to perform these steps.
    For Oracle Autonomous Database, a software keystore already exists and dictionary credentials are encrypted
  2. If you are not using Oracle Autonomous Database, then check if your system is already configured to enforce dictionary credential encryption.
    1. Verify a wallet (a Keystore) exists using the following SQL query:
      SELECT con_id, wrl_type, status , wallet_type FROM V$ENCRYPTION_WALLET
ORDER BY con_id;
    CON_ID WRL_TYPE     STATUS   WALLET_TYPE
---------- ------------ -------- -----------
         0 FILE         OPEN     PASSWORD

      If no rows are returned by this SQL query, then a wallet, or keystore, does not exist.

    2. Verify that dictionary credentials are encrypted using the following SQL query:
      SQL> SELECT enforcement FROM DICTIONARY_CREDENTIALS_ENCRYPT;
ENFORCEMENT
---------------
ENABLED

      If this SQL query returns DISABLED, then the dictionary is not encrypted.

    If you have a wallet and dictionary credentials encrypted, you can use extended FAILOVER_RESTORE by setting the attribute on your service. You do not need to complete any more of the steps in this procedure.

    If you do not have an existing wallet, or if you need to enable dictionary credentials encryption, then continue with the following steps.

  3. Configure the database to use a software keystore.
    1. If necessary, create a directory to store the wallet.

      The location selected needs to be shared across Oracle RAC nodes and replicated to Oracle Data Guard sites. For Oracle RAC, the directory must be on shared storage.

    2. Change the initialization parameter WALLET_ROOT.
      The parameter value should be the directory where the wallet is stored.
      SQL> CONNECT / AS SYSDBA
SQL> ALTER SYSTEM SET WALLET_ROOT='/myOracleBase/admin/wallet/' SCOPE=spfile;

      You can also change the WALLET_ROOT initialization parameter using the init.ora file on all RAC nodes by adding wallet_root='/myOracleBase/admin/wallet/'.

      Starting with Oracle Database 23ai, the parameter ENCRYPTION_WALLET_LOCATION is desupported.

      To store and retrieve the TDE wallet, use the WALLET_ROOT structure (introduced with Oracle Database 18c).

      Note:

      If Transparent Data Encryption (TDE) is enabled, but WALLET_ROOT is not configured, then you will be blocked from upgrading to Oracle Database 23ai. This block for upgrades of databases using TDE is to prevent the possibility of not being able to open the database after the upgrade.
    3. Change the initialization parameter TDE_CONFIGURATION to specify a software keystore.
      SQL> CONNECT / AS SYSDBA
SQL> ALTER SYSTEM SET TDE_CONFIGURATION="KEYSTORE_CONFIGURATION=FILE" SCOPE=BOTH SID='*'

      You can also change the TDE_CONFIGURATION initialization parameter using the init.ora file on all RAC nodes by adding TDE_CONFIGURATION="KEYSTORE_CONFIGURATION=FILE" .

    4. Perform a rolling restart of the database instances to activate the new initialization parameters.

      For example, for a two node clustered database named orcl, where the instances are named orcl1 and orcl2, you would use the following commands to stop and restart each instance individually to avoid a complete outage of your database. 

      $ srvctl stop instance -db orcl -instance orcl1 -drain_timeout 600 -stopoption IMMEDIATE
$ srvctl start instance -db orcl -instance orcl1
 
$ srvctl stop instance -db orcl -instance orcl2 -drain_timeout 600 -stopoption IMMEDIATE
$ srvctl start instance -db orcl -instance orcl2

      Note:

      Fleet Patching and Provisioning, if used, automates this process and can be used instead if you are modifying the parameters during a patch upgrade.
    5. Verify that the parameters are set to the correct values after restarting the instances.
      SQL> SHOW PARAMETER WALLET_ROOT;
SQL> SHOW PARAMETER TDE_CONFIGURATION;
  4. Create a keystore with a password, if one does not already exist.

    In the following example password is the password for the keystore. The password is case sensitive. Keystore passwords adhere to the same rules as database user passwords. 

    SQL> CONNECT / AS SYSKM 
SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE IDENTIFIED BY "password";
  5. Open a keystore and set an encryption key.

    If your database is configured as an Oracle Multitenant database, then a keystore and encryption key must be set for each PDB using the CONTAINER=all clause. In the following example password is the password for the keystore. 

    SQL> CONNECT / AS SYSKM
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "password" CONTAINER=all;
SQL> ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY IDENTIFIED BY "password" 
SQL> WITH BACKUP CONTAINER=all;

    If your database is not configured as an Oracle Multitenant database, then use the following SQL commands, where password is the password for the keystore: 

    SQL> CONNECT / AS SYSKM
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "password";
SQL> ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY IDENTIFIED BY "password" WITH BACKUP;
  6. Encrypt the database dictionary credentials.

    Use an operator with the SYSKM role to run the following SQL command from within the Container Database (CDB) root and each PDB. 

    SQL> CONNECT / AS SYSKM
SQL> ALTER DATABASE DICTIONARY ENCRYPT CREDENTIALS;

Encryption and decryption of the information occurs automatically at the server during failover restoration.

WARNING:

It is recommended to backup the software keystore and the wallet location. Do not lose your TDE software keystore or WALLET_ROOT location. If you do, for Application Continuity or Transparent Application Continuity, a new keystore can be created but encrypted dictionary credentials will need to be re-instantiated. Failover will not succeed while there is a mismatch in the wallet keys.

Configuring a Keystore using SQLNET.ORA for FAILOVER_RESTORE

Use these steps to configure encryption of dictionary credentials by using SQLNET.ORA to point to the wallet location for use with FAILOVER_RESTORE.

Note:

Oracle recommends that you configure a Keystore using WALLET_ROOT for FAILOVER_RESTORE in preference to SQLNET.ORA.

This method does not require a database restart, unless your database runs on the Microsoft Windows operating system. You are responsible for ensuring that the sqlnet.ora files are consistent in all ORACLE_HOME directories.

  1. If you are using Oracle Autonomous Database, you do not need to perform these steps.
    For Oracle Autonomous Database, a software keystore already exists and dictionary credentials are encrypted
  2. If you are not using Oracle Autonomous Database, then check if your system is already configured to enforce dictionary credential encryption.
    1. Verify a wallet exists using the following SQL query:
      SQL> SELECT con_id, wrl_type, status , wallet_type FROM V$ENCRYPTION_WALLET
ORDER BY con_id;
    CON_ID WRL_TYPE     STATUS   WALLET_TYPE
---------- ------------ -------- -----------
         0 FILE         OPEN     PASSWORD

      If no rows are returned by this SQL query, then a wallet, or keystore, does not exist.

    2. Verify that dictionary credentials are encrypted using the following SQL query:
      SQL> SELECT enforcement FROM DICTIONARY_CREDENTIALS_ENCRYPT;
ENFORCEMENT
---------------
ENABLED

      If this SQL query returns DISABLED, then the dictionary is not encrypted.

    If you have a wallet and dictionary credentials encrypted, you can use extended FAILOVER_RESTORE by setting the attribute on your service. You do not need to complete any more of the steps in this procedure.

    If you do not have an existing wallet, or if you need to enable dictionary credentials encryption, then continue with the following steps.

  3. Configure the database to use a wallet.
    1. View the TNS_ADMIN environment variable to find the location of the network configuration files used by your database.

      • On Linux and UNIX systems, as the Oracle Home software owner, view the current setting of the TNS_ADMIN environment variable. 

        $ env | grep TNS_ADMIN

      • On Microsoft Windows systems, check the value set for TNS_ADMIN as both an environment variable and in the registry in the path Computer\HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\KEY_HOME_NAME.

      If the TNS_ADMIN variable is not set, then the default location of $ORACLE_BASE_HOME/network/admin is used for the Oracle Net configuration files with read-only Oracle homes.
    2. If necessary, create a directory to store the wallet.

      The location selected needs to be shared across Oracle RAC nodes and replicated to Oracle Data Guard sites. For Oracle RAC, the directory must be on shared storage.

    3. Change the initialization parameter TDE_CONFIGURATION to specify a software keystore.
      SQL> CONNECT / AS SYSDBA
SQL> ALTER SYSTEM SET TDE_CONFIGURATION="KEYSTORE_CONFIGURATION=FILE" SCOPE=BOTH SID='*'
  4. Create a keystore with a password, if one does not already exist.

    In the following example myOracleWalletLoc is the full path name of the directory created to store the wallet (or keystore) and password is the password for the keystore. The password is case sensitive. Keystore passwords adhere to the same rules as database user passwords. 

    SQL> CONNECT / AS SYSKM 
SQL> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '/myOracleWalletLoc' IDENTIFIED BY "password";
  5. Open a keystore and set an encryption key.

    If your database is configured as an Oracle Multitenant database, then a keystore and encryption key must be set for each PDB using the CONTAINER=all clause. In the following example password is the password for the keystore. 

    SQL> CONNECT / AS SYSKM
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "password" CONTAINER=all;
SQL> ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY IDENTIFIED BY "password" WITH BACKUP CONTAINER=all;

    If your database is not configured as an Oracle Multitenant database, then use the following SQL commands, where password is the password for the keystore: 

    SQL> CONNECT / AS SYSKM
SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "password";
SQL> ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY IDENTIFIED BY "password" WITH BACKUP;
  6. Encrypt the database dictionary credentials.

    Use an operator with the SYSKM role to run the following SQL command from within the Container Database (CDB) root and each PDB. 

    SQL> CONNECT / AS SYSKM
SQL> ALTER DATABASE DICTIONARY ENCRYPT CREDENTIALS;

Encryption and decryption of the information occurs automatically at the server during failover restoration.

Caution:

It is recommended to backup the wallet location. Do not lose your wallet or location. If you do, for Application Continuity or Transparent Application Continuity, a new wallet can be created but encrypted dictionary credentials will need to be re-instantiated.

Failover will not succeed while there is a mismatch in the wallet keys.

FAILOVER_RESTORE = NONE and No Callback

FAILOVER_RESTORE = NONE does not restore session state at failover.

Oracle recommends setting FAILOVER_RESTORE to LEVEL2, LEVEL1, or AUTO for all applications.

Connection Labeling

Connection Labeling is not required unless you want to restore more complex states, such as proxying a session.

This scenario is applicable to Universal Connection Pool (UCP) and Oracle WebLogic server. Connection Labeling APIs are required to match secure session states- SYS_CONTEXT and password protected roles. They are also useful in reporting a failover on the driver side. When a connection is borrowed, Connection Labeling populates the gap by using a callback.

Connection Initialization Callback

Use the Connection Initialization Callback only if you need to restore session states such as SYS_CONTEXT and password protected roles. FAILOVER_RESTORE with wallets restores all modifiable parameters.

If your replaying driver uses an application callback to set the initial state of the session during runtime and replay, then the callback interface depends on whether the driver is a Java Database Connectivity (JDBC) driver or an Oracle Call Interface (OCI) driver.

In this scenario, with either JDBC or OCI, the driver uses an application callback to set the initial state of the session during runtime and replay. With the JDBC replay driver, the driver provides a connection initialization callback interface and methods to register and unregister connection initialization callbacks in the oracle.jdbc.replay.OracleDataSource interface. With OCI and Oracle Data Provider for .NET (ODP.NET), you register the Transparent Application Failover (TAF) callback.

When registered, the initialization callback is run every time a connection is borrowed from the pool, and at each successful reconnection following a recoverable error. (This is true for the JDBC/UCP initialization callback, and should be the same for TAF.) Using the same callback during both runtime and replay ensures that the same initialization is established at replay, as it was when the session was first established. An application is responsible for ensuring that the initialization actions are the same as that on the original connection before failover. If the callback invocation fails, then replay is disabled on that connection. Use the connection initialization callback only when the application has not implemented UCP and WebLogic Connection Labeling and the state cannot be restored automatically by setting either FAILOVER_RESTORE=AUTO or FAILOVER_RESTORE=LEVEL1 for Transparent Application Continuity or FAILOVER_RESTORE=LEVEL2 for Application Continuity.

Potential Side Effects of Application Continuity

When you use Application Continuity with the service attribute FAILOVER_TYPE set to TRANSACTION, statements that perform side effects are replayed.

When you use Transparent Application Continuity with the FAILOVER_TYPE service attribute set to AUTO, then replay is disabled when a side effect is detected. Replay re-enables at the next discovered or explicit boundary.

Note:

As an application owner you can elect to disable replay for requests that contain side effects that you do not want to repeat. The simplest way to disable side effects is to use Transparent Application Continuity (set the service attribute FAILOVER_TYPE to AUTO, which disables side effects for you.

Application Continuity replays PL/SQL chronologically to restore database state. This serves to rebuild the session as if the user submission was delayed. Most applications want the full state rebuilt as if the submission was repeated, such as writing a report or completing some auditing. However, the actions that are replayed to build the state might include some for which you want to take action to accommodate or mitigate the effects of the replay. Some applications elect to disable replay for requests that contain calls that they do not want to repeat.

Examples of actions that create side effects include the following:

  • DBMS_ALERT calls (email or other notifications)

  • DBMS_FILE_TRANSFER calls (copying files)

  • DBMS_PIPE and RPC calls (to external sources)

  • UTL_FILE calls (writing text files)

  • UTL_HTTP calls (making HTTP callouts)

  • UTL_MAIL calls (sending email)

  • UTL_SMTP calls (sending SMTP messages)

  • UTL_TCP calls (sending TCP messages)

  • UTL_URL calls (accessing URLs)

For applications with external actions (such as autonomous transactions or using UTL_HTTP to issue a service-oriented application (SOA) call), Application Continuity is transparent when the application is satisfied with replaying external actions, such as resending email, auditing, and transferring a file.

Allowing and Disallowing Side Effects for 23ai

Use APPLY_REPLAY_RULE to allow or disallow side effects.

Replayable Interface Side Effect Use Cases

The following use cases cover various scenarios for replayable interface side effects:

  • Use the following interface to allow side effects with TAC:
    begin 
insert into emp_tab values (''Alan'', 10); 
DBMS_APP_CONT.APPLY_REPLAY_RULE(REPLAYABLE => TRUE, TARGETS => DBMS_APP_CONT.AUTONOMOUS_TRANSACTIONS);       
do_my_autonomous_txn;
commit;
end
  • Use the following interface to disallow side effects with AC:
    begin 
insert into emp_tab values (''Alan'', 10); 
DBMS_APP_CONT.APPLY_REPLAY_RULE(REPLAYABLE => FALSE, TARGETS => DBMS_APP_CONT.AUTONOMOUS_TRANSACTIONS);       
do_my_autonomous_txn;
commit;
end
  • Use the following statement to wrap a replayable interface function for TAC:
    create [or replace] procedure mark_replayable  as 
    begin
        DBMS_APP_CONT.APPLY_REPLAY_RULE(REPLAYABLE => TRUE, TARGETS => DBMS_APP_CONT.AUTONOMOUS_TRANSACTIONS,
        SCOPE => DBMS_APP_CONT.SCOPE_PARENT); 
        end mark_replayable;

    begin
        insert into emp_tab values (''Alan'', 10);
        mark_replayable;
        do_my_autonomous_txn; 
        commit; 
    end

Related Topics

Parent topic: Configuring Application Continuity

Administering Application Continuity Operation and Usage

Learn how to manage the use of Application Continuity, and how you can use it in applications.

Parent topic: Ensuring Application Continuity

Using Application Continuity for Planned Maintenance

For planned maintenance, Oracle recommends that you drain requests from Oracle connection pools in combination with Application Continuity for those requests that do not complete.

This procedure has the least impact when there is minimal recovery to complete. Instances do need to be stopped to switch over to the patched software.

  1. FAN-aware pool, such as OCI, UCP, WebLogic Server, or ODP.NET Managed and Unmanaged Drivers.

    The FAN planned event drains at request boundaries.

    Note:

    • ODP.NET Managed Driver does not support Application Continuity.
    • Oracle Data Provider for .NET (ODP.NET), Unmanaged Driver is deprecated in Oracle Database 23ai.

  2. Use the srvctl relocate service command to relocate the service from the instance without disrupting the sessions or, for a uniform service, use the srvctl stop service command on the instance (do not use the -force parameter).

    The FAN planned event clears the idle sessions immediately and marks the active sessions to be released at check-in (end of request). This drains the sessions from the instance without disrupting work.

  3. If not all sessions have checked in and the time to stop the instance has been reached, then stop the instance (terminate).

    For Application Continuity-enabled pools (UCP, WebLogic, Tuxedo, ODP.NET, and OCI), and any Java pool that adds beginRequest/endRequest , Application Continuity attempts to recover those remaining sessions.

  4. Restart the instance and service.

    Runtime load balancing, when enabled, balances the sessions back to the restored instance at the next request boundaries.

Related Topics

Parent topic: Administering Application Continuity Operation and Usage

Administering Restoring Original Function Results

To manage restoring original function results, you need to grant certain privileges.

Parent topic: Administering Application Continuity Operation and Usage

Restoring Original Oracle Function Values and Application Continuity

When a request is replayed, the default and desired treatment of restoring original Oracle function values can vary.

By default, for SQL the original value received is restored for sequences. This is the value that the application owns. For PL/SQL, DATE and TIME, and SYSGUID restoring original Oracle function values, the KEEP clause must be granted as a part of the schema.

Support for restoring original Oracle function values is currently provided for SYSDATE, SYSTIMESTAMP, LOCALTIMESTAMP, CURRENT_TIMESTAMP, SYS_GUID, and sequence.NEXTVAL. If the original values are not kept and if different values for these objects are returned to the client, then replay is rejected because the client observes different results. If the application can use original values, then configure restoring original Oracle function values using the KEEP clause for owned sequences and GRANT KEEP for other users. (Most applications need sequence values to be kept at replay, for bind variable consistency.)

Note:

Keeping SYS_GUID values is supported only for serial processing plans. When parallel query is used, Application Continuity is not able to restore original values for SYS_GUID.

The following table shows examples of the treatment of restoring original Oracle function values by products during replay. (Actual implementation depends on specific products and releases.)

Table 6-1 Example Treatment of Restoring Original Oracle Function Values by Products During Replay

Function Product 1 Product 2 Product 3

SYSDATE, SYSTIMESTAMP

Original

Original

Current

Sequence NEXTVAL and CURRVAL

Original

Original

(Not applicable)

SYS_GUID

Original

(Not applicable)

(Not applicable)

To allow Application Continuity to keep and use original function results at replay:

  • The database user running the application might have the KEEP DATE TIME and KEEP SYSGUID privileges granted, and the KEEP SEQUENCE object privilege on each sequence whose value is to be kept. For example: 

    GRANT KEEP DATE TIME TO user2;
GRANT KEEP SYSGUID TO user2;
GRANT KEEP SEQUENCE ON sales.seq1 TO user2;

    Notes:

    • Starting with Oracle Database 19c, grants are not required for restoring original Oracle function values for SQL for sequences.

    • GRANT ALL ON object does not include (that is, does not grant the access provided by) the KEEP DATE TIME and KEEP SYSGUID privileges, and the KEEP SEQUENCE object privilege.

    • Grant privileges related to restoring original Oracle function values support only to application users, and to each application user grant only the necessary privileges.

    • Do not grant DBA privileges to database users running applications for which you want replay to be enabled.

  • Sequences in the application can use the KEEP attribute, which keeps the original values of sequence.NEXTVAL for the sequence owner, so that the keys match during replay. Most applications need sequence values to be kept at replay. The following example sets the KEEP attribute for a sequence (in this case, one owned by the user running the statement; for others, use GRANT KEEP SEQUENCE): 

    SQL> CREATE SEQUENCE my_seq KEEP;
SQL> -- Or, if the sequence already exists but without KEEP:
SQL> ALTER SEQUENCE my_seq KEEP;

    Note:

    Specifying ALTER SEQUENCE ... KEEP/NOKEEP applies to the owner of the sequence. It does not affect other users (not the owner) that have the KEEP SEQUENCE object privileges. If you want KEEP for all users, then be sure to grant the KEEP SEQUENCE object privilege to these users (or to revoke it from each user if the privilege has been granted).

  • To keep function results (for named functions) at replay, the DBA must grant KEEP privileges to the user invoking the function. This security restriction ensures that it is valid for replay to save and restore function results for code that is not owned by that user.

For identity sequences, restoring original Oracle function values is supported for owned sequences. Restoring original Oracle function values at SQL level is automatic for identity sequences. To restore original Oracle function values in PL/SQL for identity sequences use the KEEP clause. The definition of procedure and table are as follows: 

create table tab_identity_mine( id NUMBER GENERATED ALWAYS AS IDENTITY keep, content varchar2(50));

Use the following statement to create a procedure:

insert_identity(cnt in varchar2,newid out number) as 
begin 
insert into tab_identity_mine(content) values(cnt) returning id into newid;
end insert_identity;
Checking Your Keep Permissions

You must ensure that you have required KEEP permissions to keep function results at replay.

  • To check permission to keep SYSDATE and SYSGUID:
    SELECT * FROM USER_SYS_PRIVS WHERE PRIVILEGE LIKE '%KEEP%';

    This query returns output similar to the following: 

    USERNAME   PRIVILEGE        ADM   COM   INH
--------   --------------   ---   ---   ---
SOE1       KEEP SYSGUID     NO    NO    NO
SOE1       KEEP DATE TIME   NO    NO    NO
  • To check permission to keep SEQUENCES:
    SELECT SEQUENCE_NAME, KEEP_VALUE FROM USER_SEQUENCES;

    This query returns output similar to the following: 

    SEQUENCE_NAME   KEEP_VALUE
-------------   ----------
SEQ_PERSON      N      
SEQ_PLSQL       N
SEQ_PRODUCTS    Y
SEQ_PRODUCT_ID  Y

    The KEEP_VALUE in the above example is Y or N.

    Note:

    For all sequences grants, run the SELECT SEQUENCE_NAME, KEEP_VALUE FROM ALL_SEQUENCES; statement.
Granting and Revoking Keep Permissions for Restoring Original Oracle Function Values

To keep function results at replay, you must grant KEEP privileges to the user invoking the function.

  • To grant permission to keep restoring original Oracle function values for SYSDATE and SYSTIMESTAMP, or SYSGUID:
    GRANT [KEEP DATE TIME | KEEP SYSGUID]...[to USER]

    For example, for possible Oracle E-Business Suite usage with original dates:

    GRANT KEEP DATE TIME, KEEP SYSGUID to [custom user];
GRANT KEEP DATE TIME, KEEP SYSGUID to [apps user];
  • To revoke permission to keep restoring original Oracle function values for SYSDATE and SYSTIMESTAMP, or SYSGUID:
    REVOKE [KEEP DATE TIME | KEEP SYSGUID]...[from USER]
Granting Permission to Keep Restoring Original Oracle Function Values for Oracle Sequences

To keep the original values of sequence.nextval for replaying so that keys match, you must grant permissions on the sequence.

  • To grant permission as the owner of the sequence:
    CREATE SEQUENCE [sequence object] [KEEP|NOKEEP];
ALTER SEQUENCE [sequence object] [KEEP|NOKEEP];
  • To grant and revoke permission for others using the sequence:
    GRANT KEEP SEQUENCE on sequence.object to [myUser|role];
REVOKE KEEP SEQUENCE on sequence.object from [myUser|role];

    For example, for possible Oracle E-Business Suite usage with original sequence values:

    GRANT KEEP SEQUENCE on sequence.object to apps-user;
REVOKE KEEP SEQUENCE on sequence.object from my-user ;

    For example, for identity sequences, use the KEEP clause on the table create or alter statements: 

    CREATE TABLE tab_identity_mine(id NUMBER GENERATED ALWAYS AS IDENTITY keep, 
content varchar2(50));
Rules for Grants on Restoring Original Oracle Function Values

These considerations apply to granting and revoking privileges on restoring original Oracle function values.

  • If you grant all on an object for a user, then restoring original Oracle function values are excluded. Restoring original Oracle function values require explicit grants. Oracle does not support granting restoring original Oracle function values to the users supplied or created by Oracle Database, such as SYS, AUDSYS, GSMUSER, and SYSTEM.

  • The DBA role includes restoring original Oracle function value permission.

  • If a user has restoring original Oracle function values granted, then the objects inherit restoring original Oracle function value access when the functions are called (in SYS_GUID, SYSDATE and SYSTIMESTAMP).

  • If keeping restoring original Oracle function values on a sequence object is revoked, then SQL or PL/SQL commands using that object does not allow restoring original Oracle function value collection or application for that sequence.

  • If grants are revoked between run time and failover, then the restoring original Oracle function values that were collected are not applied.

  • If grants are granted between run time and failover, then restoring original Oracle function values are not collected and so none are applied.

Protection-Level Statistics

Use the statistics for request boundaries and protection level to monitor the level of coverage.

Application Continuity collects statistics from the system, the session, and the service, enabling you to monitor your protection levels. The statistics are available in V$SESSTAT, V$SYSSTAT, and, when service statistics are enabled, in V$SERVICE_STATS. For example, if you query V$SESSTAT and join with V$STATNAME, you can view output like the following: 

NAME                                                             VALUE
---------------------------------------------------------------- ----------
cumulative begin requests                                               731
cumulative end requests                                                 739
cumulative user calls in requests                                      7285
cumulative user calls protected by Application Continuity              7228
cumulative time in requests                                      2665167909

These statistics are saved in the Automatic Workload Repository (AWR) and are available in AWR reports. Statistics include:

  • Requests completed per second
  • User calls in a request
  • Protected user calls

The AWR report output is similar to the following:

Statistic                                Total    per Second    per Trans
---------------------------------------- -------- ------------- ---------
cumulative requests                       177,406          49.2       5.0
cumulative user calls in request          493,329         136.8      13.8
cumulative user calls protected           493,329         136.8      13.8

To enable protection-level statistics, use (_request_boundaries = 3).

Parent topic: Administering Application Continuity Operation and Usage

Session State Consistency

Session state consistency describes how non-transactional state is changed during a request.

  • About Session State Consistency
    To ensure session state consistency, Oracle recommends that you set the service parameter session_state to AUTO, which is available with Transparent Application Continuity.
  • Auto Session State Consistency
    When you set the service parameter session_state to AUTO, Transparent Application Continuity tracks and records session and transactional states so the database session can be recovered following recoverable outages.
  • Dynamic Session State Consistency
    Application Continuity uses Dynamic Session State Consistency. The session state is tracked as it is for TAC when using Session State Consistency = AUTO.

Parent topic: Administering Application Continuity Operation and Usage

About Session State Consistency

To ensure session state consistency, Oracle recommends that you set the service parameter session_state to AUTO, which is available with Transparent Application Continuity.

Transparent Application Continuity tracks and manages session states. If you choose to use Transparent Application Continuity, then you do not have to do anything else to ensure session state consistency.

You can set session_state to DYNAMIC for Application Continuity. You should only set session_state to DYNAMIC if you fully understand the application, and the application is not expected to change from the value set.

Examples of session state are NLS settings (globalization support), optimizer preferences, event settings, PL/SQL global variables, temporary tables, advanced queues, Large Objects (LOBs), and result cache. If non-transactional values change in committed transactions, then use the default value, DYNAMIC.

Using DYNAMIC mode, after a COMMIT has run, if the state was changed in that transaction, then it is not possible to replay the transaction to reestablish that state if the session is lost. Applications can be categorized depending on whether the session state after the initial setup is dynamic, and hence whether it is correct to continue past a COMMIT operation.

AUTO mode is appropriate for almost all applications. If your customers or users can modify your application, then you must use AUTO or DYNAMIC mode. AUTO mode is a newer version of DYNAMIC mode with the additional feature that it re-enables automatically when possible.

Note:

Set session_state to AUTO for long-running, stateless applications. Unless you require Application Continuity, Oracle recommends setting session_state to AUTO.
Auto Session State Consistency

When you set the service parameter session_state to AUTO, Transparent Application Continuity tracks and records session and transactional states so the database session can be recovered following recoverable outages.

Setting session_state to AUTO is the only value permitted for Transparent Application Continuity. When set to AUTO, a state-tracking infrastructure categorizes session state usage as the application issues user calls. Tracked session states are monitored and verified.

Note:

If you set session_state to AUTO, then you must also set failovertype to AUTO.
Replay (that is, Transparent Application Continuity) is enabled at an explicit begin request (from an Oracle pool, which is the usual source, or added by OCI or JDBC thin APIs), and is disabled on a COMMIT, at the end of a request (most often return to the pool), or a restricted call. Transparent Application Continuity automatically re-enables replay when the session state is describable after a disable within a request. Following is the step logic for three application scenarios:

  • No transaction

  • A transaction with COMMIT as the last statement

  • A transaction with an embedded COMMIT statement

Note:

Enable RESET_STATE if the session state set should not leak to the next request. This also ensures that TAC is enabled in the next request when PL/SQL is used.

For the request with no transaction, the logical steps are as follows:

  1. Check out from a connection pool.

  2. Begin request and enable replay.

  3. Issue one or more SELECT statements and perhaps other PL/SQL statements.

  4. Other actions.

  5. Check in.

  6. End request.

For the request with a transaction with COMMIT as the last statement, the logical steps are as follows:

  1. Check out from a connection pool.

  2. Begin request and enable replay.

  3. Issue one or more SELECT statements and perhaps other PL/SQL statements.

  4. The transaction begins.

  5. Other actions.

  6. Commit (which disables replay).

  7. Check in.

  8. End request.

For the request with a transaction with an embedded COMMIT statement, the logical steps are as follows:

  1. Check out from a connection pool.

  2. Begin request and enable replay.

  3. Issue one or more SELECT statements and perhaps other PL/SQL statements.

  4. The transaction begins.

  5. Other actions.

  6. Commit (which disables replay).

  7. Next PL/SQL statement, which re-enables Transparent Application Continuity when possible.

  8. Check in.

  9. End request.

Dynamic Session State Consistency

Application Continuity uses Dynamic Session State Consistency. The session state is tracked as it is for TAC when using Session State Consistency = AUTO.

Replay (that is, Application Continuity) is enabled at the beginRequest call, and is disabled on a COMMIT , an endRequest call, or a restricted call. Following is the step logic for three application scenarios:

  • No transaction

  • A transaction with COMMIT as the last statement

  • A transaction with an embedded COMMIT statement

For the request with no transaction, the logical steps are as follows:

  1. Check out from a connection pool.

  2. Begin request and enable replay.

  3. Issue one or more SELECT statements and perhaps other PL/SQL statements.

  4. Other actions.

  5. Check in.

  6. End request and disable replay.

For the request with a transaction with COMMIT as the last statement, the logical steps are as follows:

  1. Check out from a connection pool.

  2. Begin request and enable replay.

  3. Issue one or more SELECT statements and perhaps other PL/SQL statements.

  4. The transaction begins.

  5. Other actions.

  6. Commit (which disables replay).

  7. Check in.

  8. End request.

For the request with a transaction with an embedded COMMIT statement, the logical steps are as follows:

  1. Check out from a connection pool.

  2. Begin request and enable replay.

  3. Issue one or more SELECT statements and perhaps other PL/SQL statements.

  4. The transaction begins.

  5. Other actions.

  6. Commit (which disables replay).

  7. Other actions, during which Application Continuity is not covering the application.

  8. Check in.

  9. End request.

Application Continuity Statistics

Once Application Continuity is configured, you can use the statistics to verify Application Continuity usage and to check how well Application Continuity is protecting user workloads.

You can read the following statistics from the V$SESSTATS and V$SYSSTAT views:
  • cumulative begin requests
  • cumulative end requests
  • cumulative user calls in requests
  • cumulative user calls protected by Application Continuity
  • cumulative time in requests
You can read the following statistics only from the V$SESSTATS view:
  • cumulative DB time in requests
  • cumulative DB time protected in requests
  • successful replays by Application Continuity
  • rejected replays by Application Continuity

The following example shows you how to read Application Continuity statistics values for each active session.

SQL> SELECT sn.name, s.value FROM V$SESSTAT s, V$STATNAME sn WHERE sn.statistic# = s.statistic# AND sn.name in(
    'cumulative begin requests','cumulative end requests','cumulative user calls in requests','cumulative user calls protected by Application
    Continuity','cumulative time in requests');

The following example shows you how to read Application Continuity statistics from the V$SYSSTAT view. 

SQL> SELECT name, value FROM V$SYSSTAT WHERE name in ('cumulative begin requests','cumulative end requests',
    'cumulative user calls in requests','cumulative user calls protected by Application Continuity','cumulative time in requests',
    'cumulative DB time in requests','cumulative DB time protected in requests','successful replays by Application Continuity',
    'rejected replays by Application Continuity');

Parent topic: Administering Application Continuity Operation and Usage

Application Continuity Protection Check

The Application Continuity Protection Check (ACCHK) feature generates Application Continuity coverage reports and views that describe the protection of your application by Application Continuity.

Parent topic: Administering Application Continuity Operation and Usage

About Application Continuity Protection Check

The Application Continuity Protection Check (ACCHK) utility provides protection guidance for applications that use Application Continuity.

ACCHK provides guidance on the level of protection for each application that uses Application Continuity and helps guide you to increase protection, if required. ACCHK uses Application Continuity traces to collect coverage for a workload and provides detailed information as per your request. You must enable Application Continuity tracing to collect coverage before you run a database workload. ACCHK also provides diagnostics for an unsuccessful failover.

Database views and PL/SQL-based reports show you the level of protection for your applications for failover. If an application is not fully protected, then ACCHK identifies that application, finds out the reason why the application is not fully protected, and guides you how to increase the protection.

For the protected applications, ACCHK also reports which operations of an application are protected, and which operations of an application are not protected. If any operation or configuration of an application is not protected by the Application Continuity, then you can make configuration changes to increase the protection coverage. ACCHK generates a report with coverage statement and percentage value for the workload. The ACCHK report also shows how many operations were performed, how many operations were fully protected, and how many operations were not fully protected.

Enabling and Disabling Application Continuity Protection Check

You can manually enable or disable the Application Continuity Protection Check (ACCHK) feature for applications that use Application Continuity.

Application Continuity Protection Check is not enabled by default. Follow this procedure to enable or disable ACCHK and generate reports to check protection level for the applications.
  1. Grant read access to the users, who will run the Application Continuity Protection Check report and views, using the ACCHK_READ role:
    GRANT ACCHK_READ TO USER;
  2. Enable Application Continuity tracing for your applications using the dbms_app_cont_admin.acchk_set(true) procedure:
    SQL> execute dbms_app_cont_admin.acchk_set(true);

    By default, ACCHK is disabled automatically after 600 seconds. You can specify a lower number to reduce the auto disable time. For example, use the dbms_app_cont_admin.acchk_set(true,300) procedure to disable ACCHK after 300 seconds.

    The dbms_app_cont_admin.acchk_set(true) procedure enables Application Continuity tracing at the database level to which you are connected. If you are connected at the CDB level, then tracing is enabled for the CDB, and if you are connected at the PDB level, then tracing is enabled for the PDB.

    Note:

    Set the COMPATIBLE parameter to 19c or greater. If the COMPATIBLE parameter was previously set to a lower value, then the acchk_set procedure creates the ACCHK views and roles when you run the procedure for the first time after updating the COMPATIBLE parameter.
  3. Use the dbms_app_cont_admin.acchk_set(false) procedure to disable Application Continuity tracing for new sessions in your applications:
    SQL> execute dbms_app_cont_admin.acchk_set(false);

    Note:

    • The tracing will be disabled for the current sessions after the time expires.
    • The tracing is enabled by default for the entire Oracle Real Application (Oracle RAC) Cluster.
Generating an Application Continuity Protection Check Report

Generate the Application Continuity Protection Check (ACCHK) report to get guidance for the level of protection, reason for incomplete protection, and methods to increase the protection level.

The ACCHK utility is a post-processing tool that uses ACCHK tables in AUX to report Application Continuity coverage. Enable the Application Continuity tracing and Application Continuity Protection Check before running a workload and generating the report.

When in read-only mode, traces can record event and statistics. The parameter source in ACCHK_REPORT is used to decide which input will use the procedure for displaying the report. You can choose either from tables or traces. When the database is in read-only mode, set this parameter to DBMS_APP_CONT_REPORT.FROM_TRACES.

  1. Run a set of database options after enabling ACCHK and tracing for your applications.
    ACCHK generates reports only for the Application Continuity sessions.
  2. Generate the Application Continuity Protection Check report using the dbms_app_cont_report.acchk_report procedure:
    SQL> SET LINESIZE 350;
SQL> SET SERVEROUTPUT ON FORMAT WRAPPED;
SQL> execute dbms_app_cont_report.acchk_report;
    You can specify the type of the report from FULL, WARNING, or SUMMARY. For example:
    SQL> SET LINESIZE 350;
SQL> SET SERVEROUTPUT ON FORMAT WRAPPED;
SQL> execute dbms_app_cont_report.acchk_report(dbms_app_cont_report.FULL);
SQL> execute dbms_app_cont_report.acchk_report(dbms_app_cont_report.WARNING);
SQL> execute dbms_app_cont_report.acchk_report(dbms_app_cont_report.SUMMARY);

    The default report type is SUMMARY.

  3. Analyze the report and increase the protection level for the applications that are not fully protected. For example, a summary report looks like the following:
    --------------------------------------
------------ ACCHK Report ------------
--------------------------------------
CON_ID SERVICE       FAILOVER PROTECTED_ PROTECTED_ REQUESTS AVG_CALLS/ PROTECTED_    AVG_TIME/  PROTECTED_TIME/ EVENT_  ERROR_ PROGRAM   MODULE            ACTION    SQL_ CALL      TOTAL
                              CALLS %    TIME %              REQUEST    CALLS/REQUEST REQUEST MS REQUEST MS      TYPE    CODE                                         ID                
------ ------------- -------- ---------- ---------- -------- ---------- ------------- ---------- --------------- ------- ------ --------- ----------------- –-------- ---- --------- –-----
3      srv_tacr_pdb1 AUTO     98.734     98.432     117      9.453      9.333         2279.751   2244.014        DISABLE 41409  JDBC Thin AddCustNewOrder   Action-20      COMMIT    1     
                                                                                                                                Client                                                     
3      srv_tacr_pdb1 AUTO     98.734     98.432     117      9.453      9.333         2279.751   2244.014        REPLAY_ 41412  JDBC Thin InsertNewChecksum Action-1       SQL/PLSQL 1     
                                                                                                                 FAILED         Client                                     Execu           
End of report.
The following examples show how to query detailed information from an ACCHK report using the ACCHK views.

Example 6-2 Using the DBA_ACCHK_EVENTS View

In this example, the last row indicates that the application that is using the srv_tacr_pdb1 service has an event that caused Application Continuity to fail. 

SQL> SELECT * FROM DBA_ACCHK_EVENTS ORDER BY TIMESTAMP;
INST_ID CON_ID TIMESTAMP        SESSION_ID SERIAL# SERVICE_NAME  PROGRAM  MODULE            ACTION    SQL_ID CALL_NAME EVENT_TYPE ERROR_CODE
------- ------ ---------------- ---------- ------- ------------- -------  ----------------- --------- ------ --------- ---------- ----------
2       3    21-SEP-20          9598       1644    srv_tacr_pdb1 JDBC     AddCustNewOrder   Action-36        COMMIT    DISABLE    41409
             06.54.18.191 PM                                     Thin                                         
             -07:00                                              Client                                                        
2       3    21-SEP-20          1703       61265   srv_tacr_pdb1 JDBC     InsertNewChecksum  Action-1        SQL/PLSQL REPLAY_    41412
             06.51.07.624 PM                                     Thin                                        Execution FAILED 
             -07:00                                               Client

Example 6-3 Using the DBA_ACCHK_EVENTS_SUMMARY View

In this example, the last row indicates that the application that is using the srv_tacr_pdb1 service has an event that caused Application Continuity to fail. 

SQL> SELECT * FROM DBA_ACCHK_EVENTS_SUMMARY ORDER BY SERVICE_NAME;
INST_ID CON_ID SERVICE_NAME  FAILOVER_TYPE FAILOVER_RESTORE RESET_STATE PROGRAM MODULE            ACTION    SQL_ID CALL_NAME EVENT_TYPE ERROR_CODE FREQUENCY
------- ------ ------------- ------------- ---------------- ----------- ------- ----------------- --------- ------ --------- ---------- ---------- ----------
2       3      srv_tacr_pdb1 AUTO          AUTO             LEVEL1      JDBC    AddCustNewOrder   Action-20        COMMIT    DISABLE    41409      1
                                                                        Thin                                       Execution    
                                                                        Client                               
2       3      srv_tacr_pdb1 AUTO          AUTO             LEVEL1      JDBC    InsertNewChecksum Action-1         SQL/PLSQL REPLAY_    41412      1
                                                                        Thin                                       Execution FAILED    
                                                                        Client

Example 6-4 Using the DBA_ACCHK_STATISTICS View

In this example, the first row indicates that the application that is using the srv_tacr_pdb1 service has 11 implicit requests from JDBC and 31 calls in the application. 30 calls in these requests are protected. 

SQL> SELECT * FROM DBA_ACCHK_STATISTICS ORDER BY TIMESTAMP;
INST_ID CON_ID TIMESTAMP        SESSION_ID SERIAL# STAT_TYPE  SERVICE_NAME  FAILOVER_ FAILOVER_ RESET_ PROGRAM  BEGIN_   END_     USER_CALLS_ PROTECTED_CALLS_ TIME_IN_ TIME_PROTECTED_
                                                                            TYPE      RESTORE   STATE           REQUESTS REQUESTS IN_REQUESTS IN_REQUESTS      REQUESTS IN_REQUEST
------- ------ ---------------- ---------- ------- ---------- ------------- --------- --------- ------ -------  -------- -------- ----------- ---------------- -------- --------------- 
2       3      21-SEP-20        5653       54237   SESSION_   srv_tacr_pdb1 AUTO      AUTO      LEVEL1 JDBC     11       11       31          30               13316750 12415247
               06.54.25.321 PM                     STATISTICS                                          Thin                                                              
               -07:00                                                                                  Client                                                    
2       3      21-SEP-20        11291      26560   SESSION_   srv_tacr_pdb1 AUTO      AUTO      LEVEL1 JDBC     3        3        50          49               13094072 13068259
               06.54.24.915 PM                     STATISTICS                                          Thin
               -07:00                                                                                  Client

Example 6-5 Using the DBA_ACCHK_STATISTICS_SUMMARY View

In this example, the application that is using the srv_tacr_pdb1 service has 144 implicit requests, 99.5688328 percent calls in these requests are protected by Application Continuity or Transparent Application Continuity. 

SQL> SELECT * FROM DBA_ACCHK_STATISTICS_SUMMARY ORDER BY SERVICE_NAME;
INST_ID CON_ID SERVICE_NAME  FAILOVER_ FAILOVER_ RESET_ TOTAL_   PROTECTED_CALLS_ PROTECTED_TIME_ AVG_USER_CALLS_ AVG_PROTECTED_    AVG_TIME_   AVG_TIME_
                             TYPE      RESTORE   STATE  REQUESTS PERCENT          PERCENT         IN_REQUESTS     CALLS_IN_REQUESTS IN_REQUESTS PROTECTED_IN_REQUESTS
------- ------ ------------- --------- --------- ------ -------- ---------------- --------------- --------------- ----------------- ----------- –--------------------
2       3      srv_tacr_pdb1 AUTO      AUTO      LEVEL1 144       99.5688328       99.0130288      22.5486111      22.4513889         3078654.35  3048268.92
Filtering Statistics and Events

ACCHK can gather the statistics and events by filtering the data through a service, module, program, or machine name.

When you provide single service filter or multiple service filters, ACCHK records only the statistics and events related to the specified services. This functionality also applies when you provide only a single or multiple program filters for only a single or multiple module filters. Provide a service, program, module, and machine filter for ACCHK to record only the statistics and events that match the service, program, module, and machine.

The DBMS_APP_CONT_ADMIN package contains the following procedures for filtering data:
  • ACCHK_SET_FILTER
  • ACCHK_CLEAR_FILTER
  • ACCHK_SHOW_FILTERS

Note:

After adding filters by using ACCHK_SET_FILTER, you must run ACCHK_SET(TRUE) to load the filters. If you add or clear a filter, but do not run ACCHK_SET(TRUE), then that particular filter is not updated.

Filter Life Span

  • Filters are accumulative.
  • The ACCHK_CLEAR_FILTER procedure clears one or all filters depending on how you run the procedure.
Cleaning Up ACCHK Collected Information

You can purge the information collected by ACCHK after the ACCHK reports are generated and data gathered from ACCHK views is analyzed.

Purging the ACCHK collected information is optional. You can initiate a new ACCHK process with new collected data after purging existing ACCHK information.

Use the following statement to purge all previously collected ACCHK information:

SQL> execute dbms_app_cont_admin.acchk_purge(purge_all => TRUE);

Delaying the Reconnection in Application Continuity

Learn about how you can set parameters to manage reconnects with Application Continuity, or Transparent Application Continuity, and see examples on single-instance and Oracle Real Application Clusters (Oracle RAC) databases.

Parent topic: Administering Application Continuity Operation and Usage

Understanding How to Wait for the Service to be Ready for Application Continuity

To manage planned and unplanned outages, learn about the parameters that you use to manage application continuity.

By default, when Application Continuity initiates a failover, the driver attempts to recover the in-flight work at an instance where the service is available. For recovering the work, the driver must establish a good connection with the instance. This reconnection can take some time if the database or the instance must be restarted before the service is relocated and published. For this reason, the failover must be delayed until the service is available from another instance or database.

To manage connecting and reconnecting, you must use the RETRY_COUNT and RETRY_DELAY parameters in your TNS or URL. These TNS parameters can work well in conjunction with a planned outage, for example, an outage that may make a service unavailable for several seconds. While setting the RETRY_COUNT and RETRY_DELAY parameters, check the value of the REPLAY_INITIATION_TIMEOUT parameter first. A high value for the RETRY_DELAY parameter can cause replay to be canceled.

Parameter Name Possible Value Default Value

RETRY_COUNT

Positive integer zero or above

30

RETRY_DELAY

Time in sec, ms, or cs

3
Creating Services on Oracle RAC with Application Continuity

You can create services on Oracle RAC that utilize Transparent Application Continuity or Application Continuity.

Note:

The policy-managed database deployment option is desupported in Oracle Database 23ai.

You can create services that use Transparent Application Continuity, as follows:

$ srvctl add service -db mydb -service TACSERVICE -pdb mypdb –preferred inst1 -available inst2 
  -failovertype AUTO -session_state AUTO -failover_restore AUTO -commit_outcome TRUE -replay_init_time 600 
  -retention 86400 -notification TRUE -drain_timeout 300 -stopoption IMMEDIATE -role PRIMARY

You can create services that use Application Continuity, as follows:

$ srvctl add service -db mydb -service ACSERVICE -pdb mypdb -preferred inst1 -available inst2 
  -failovertype TRANSACTION -session_state DYNAMIC -failover_restore LEVEL1 -commit_outcome TRUE  -replay_init_time 600 
  -retention 86400 - notification TRUE -drain_timeout 300 -stopoption IMMEDIATE -role PRIMARY
Modifying Services on Single-instance Databases to use Application Continuity

If you are using a single-instance database, then use the DBMS_SERVICE package to modify services.

For Application Continuity:
DECLARE
params dbms_service.svc_parameter_array;
BEGIN
params('FAILOVER_TYPE'):='TRANSACTION';
params('REPLAY_INITIATION_TIMEOUT'):=1800;
params('RETENTION_TIMEOUT'):=86400;
params('FAILOVER_DELAY'):=10;
params('FAILOVER_RETRIES'):=30;
params('FAILOVER_RESTORE'):='LEVEL1';
params('commit_outcome'):='true';
params('aq_ha_notifications'):='true';
dbms_service.modify_service('[your service]',params);
END;
/
For Transparent Application Continuity:
DECLARE
params dbms_service.svc_parameter_array;
BEGIN
params('FAILOVER_TYPE'):='AUTO';
params('REPLAY_INITIATION_TIMEOUT'):=1800;
params('RETENTION_TIMEOUT'):=86400;
params('FAILOVER_DELAY'):=10;
params('FAILOVER_RETRIES'):=30;
params('FAILOVER_RESTORE'):='AUTO';
params('commit_outcome'):='true';
params('aq_ha_notifications'):='true';
dbms_service.modify_service('[your service]',params);
END;
/

Running Without Application Continuity

Sometimes Application Continuity is not in effect because a disabling call has been issued.

Application Continuity is not in effect when it has not been started or when it has been disabled. If it has been disabled, it remains so through to the endRequest call.

Application Continuity is not started when the service property FAILOVER_TYPE does not have the value set to TRANSACTION or AUTO. For planned maintenance, set the FAILOVER_TYPE value to TRANSACTION or AUTO, beforehand; the setting applies to new connections, and existing connections retain their original service value.

Application Continuity is disabled for the current request when any of the following occurs:

  • The application runs a statement that is restricted for Application Continuity (for example, ALTER SYSTEM).

  • Application Continuity is explicitly disabled using disableReplay.

  • A COMMIT statement is issued when the service parameter session_state_consistency is set to Dynamic (the default, when not using Transparent Application Continuity).

  • An endRequest statement is issued until the next beginRequest is issued.

  • The session is terminated or disconnected and the NOREPLAY keyword is specified.

Related Topics

Parent topic: Administering Application Continuity Operation and Usage

Disabling Replay in Application Continuity

Learn about how you can disable replay with applications, and about specific rules and guidelines for disabling replay.

The rules described in this section are generic. They apply to all applications that replay work, including Application Continuity, and TAF (release 12.2 and after).

Parent topic: Administering Application Continuity Operation and Usage

Understanding Enabling and Disabling Replay in Application Continuity

Replay occurs following a recoverable error, but you can disable replay.

If an application has requests that you do not want the application to repeat, then the application either can take a connection to a service that does not have Application Continuity enabled, or the application can explicitly call an API to disable replay for those requests. If you use Transparent Application Continuity, then side effects are detected and disabled automatically. You do not need either to understand the application, or to disable requests with side effects.

When using Application Continuity, all calls are replayed. If an application uses UTL_SMTP, for example, and you do not want that application to repeat messages, then the application can use a connection to a different service, or use the disableReplay API on Java, or use the OCIRequestDisableReplay API for Oracle Call Interface (OCI). All other requests continue to be replayed.

For applications with external actions (for example, autonomous transactions or using UTL_HTTP to issue an SOA call), Application Continuity remains transparent if the application's correctness is preserved when these external actions are replayed after a failure.

Application Calls Autonomous Transactions, External PL/SQL, or Java Actions that Should Not Be Repeated

Autonomous transactions, external PL/SQL calls, and Java callouts can have side effects that are separate from the main transaction, and these side effects are replayed unless you specify otherwise.

Examples of side effects separate from the main transaction include the following:

  • Writing to an external table, sending email, forking sessions out of PL/SQL(including calls to UTL_HTTP, UTL_URL, UTL_FILE, UTL_FILE_TRANSFER, UTL_SMPT, UTL_TCP, UTL_MAIL, DBMS_PIPE, or DBMS_ALERT)
  • Writing to an external table, sending email, forking sessions out of Java (including running a shell script in the form Process proc = rt.exec(command);), transferring files, and accessing external URLs

Actions such as these leave persistent side effects. PL/SQL messaging and Java callouts can leave persistent results behind. For example, if a user walks away partway through some work without committing, and the session times out, or the user issues a Ctrl+C command, then the foreground or a component fails. As a result, the main transaction rolls back, while the side effects may have been applied.

Application developers decide whether to allow replay for external actions. Examples include using UTL_HTTP to issue an SOA call, or UTL_SMTP to send a message, or UTL_URL to access a website. If you do not want these kinds of external actions replayed, then use a connection without application continuity, or use one of the disable replay APIs.

Application Synchronizes Independent Sessions

If the application synchronizes independent sessions using volatile entities that are held until COMMIT, ROLLBACK, or session loss, then you must not configure an application for replay.

For example, if an application synchronizes multiple sessions connected to several data sources that are otherwise interdependent using resources such as a database lock, then this synchronization can be acceptable if the application is only serializing these sessions, and understands that any session can fail. However, if the application assumes that a lock or any other volatile resource held by one data source implies exclusive access to data on the same or a separate data source from other connections, then this assumption can be invalidated when replaying.

During replay, the client driver is not aware that the sessions are dependent on one session holding a lock or other volatile resource. To implement the synchronization lost by failures, you can also use pipes, buffered queues, or stored procedures taking a resource (such as a semaphore, device, or socket).

Application Uses Time at the Middle Tier in the Processing Logic

If the application uses the wall clock at the middle tier as part of the processing logic, then you must not configure an application for replay.

The client driver does not repeat the middle tier time logic, but instead uses the database calls that run as part of this logic. For example, an application using middle-tier time might assume that a statement processed at Time T1 is not re-processed at Time T2, unless the application explicitly does so.

Application Assumes that ROWIds Do Not Change

If an application caches ROWID values, then access to these ROWID values can be invalidated due to database changes.

Although a ROWID uniquely identifies a row in a table, a ROWID can change its value in the following situations:

  • The underlying table is reorganized.

  • An index is created on the table.

  • The underlying table is partitioned.

  • The underlying table is migrated.

  • The underlying table is exported and imported using EXP/IMP/DUL.

  • The underlying table is rebuilt using Oracle GoldenGate, or other replication technology.

  • The database of the underlying table is flashed back or restored.

In general, Oracle does not recommend that an application stores ROWID values for later use, because the corresponding row either might not exist, might contain completely different data. Note that using ROWID values do not prevent using Application Continuity. Replays can be rejected.

Application Assumes that Location Values Do Not Change

If you have applications that use physical identifiers, then review the guidelines and examples here to avoid issues.

SYSCONTEXT options comprise two sets:

  • A location-independent set, such as National Language Support (NLS) settings, ISDBA, CLIENT_IDENTIFIER, MODULE, and ACTION
  • A location-dependent set, which uses physical locators

Typically, an application does not use the physical identifiers, except in testing environments. If physical locators are used in mainline code, then the replay finds the mismatch and rejects it. However, it is acceptable to use physical locators between requests (before beginRequest) or in callbacks. A common issue is for QA to modify test applications to select V$INSTANCE. As V$INSTANCE can change, only put this check in the callback, or select the instance locally at the client and not from the database.

Example of Physical Identifier Use 

select 
    sys_context('USERENV','DB_NAME') 
    ,sys_context('USERENV','HOST') 
    ,sys_context('USERENV','INSTANCE') 
    ,sys_context('USERENV','IP_ADDRESS') 
    ,sys_context('USERENV','ISDBA')  
    ,sys_context('USERENV','SESSIONID') 
    ,sys_context('USERENV','TERMINAL') 
    ,sys_context('USERENV','SID') 
from dual;

Terminating or Disconnecting a Session Without Replay

Learn how to disable replay when a DBA terminates or disconnects a session by using the ALTER SYSTEM KILL SESSION or ALTER SYSTEM DISCONNECT SESSION statement.

If Application Continuity is configured and if a DBA terminates or disconnects a session by using the ALTER SYSTEM KILL SESSION or ALTER SYSTEM DISCONNECT SESSION statement, then Application Continuity, by default attempts, to recover the session. However, if you do not want the session to be replayed, then use the NOREPLAY keyword. For example: 


alter system kill session 'sid, serial#, @inst' noreplay;

alter system disconnect session 'sid, serial#, @inst' noreplay

$ srvctl stop service -db orcl -instance orcl2 –drain_timeout 60 -stopoption immediate -force -noreplay

$ srvctl stop service -db orcl -node myode3 –noreplay -drain_timeout 60 -stopoption immediate -force

$ srvctl stop instance -node mynode3 -drain_timeout 60 -stopoption immediate -force –noreplay

To terminate all sessions running on the local instance (rather that only one session) and not have the sessions replayed, you can also use the DBMS_SERVICE.DISCONNECT_SESSION PL/SQL procedure, and specify NOREPLAY for the disconnect_option parameter.

Related Topics

Parent topic: Administering Application Continuity Operation and Usage

Fast Application Notification (FAN)

The Oracle RAC high availability framework monitors a database and its services, and sends Fast Application Notification (FAN) event notifications.

Oracle Database focuses on maintaining the highest possible service availability. In Oracle Real Application Clusters (Oracle RAC), services are designed to be continuously available with loads shared across one or more instances. The Oracle RAC high-availability framework maintains service availability by using Oracle Clusterware and resource profiles. Oracle Clusterware recovers and balances services according to business rules and the service attributes.

Parent topic: Ensuring Application Continuity

Overview of Fast Application Notification (FAN)

FAN provides immediate interrupt of clients following outages related to the database, nodes, and networks.

FAN is essential to break clients out of TCP/IP timeouts immediately following failures. FAN notifies clients immediately when resources become available and initiates draining of database sessions so clients experience no outages during planned maintenance. FAN also includes notifying configuration-level and service-level information that includes changes in service status.

The Oracle client drivers and Oracle Real Application Clusters (Oracle RAC) connection pools respond to FAN events, and take immediate action. FAN UP and DOWN events apply to services, databases, instances, networks, and nodes.

Related Topics

Parent topic: Fast Application Notification (FAN)

The Importance of Using Fast Application Notification

Using Fast Application Notification (FAN) events eliminates applications waiting on TCP timeouts, time wasted processing the last result at the client after a failure has occurred, and time wasted running work on slow, suspended, or terminated nodes.

Applications can waste time in many critical ways:

  • Waiting for TCP/IP timeouts when a node fails without closing sockets, and for every subsequent connection while that IP address is down.
  • Attempting to connect when services are down.
  • Not connecting when services resume.
  • Processing the last result at the client when the server goes down.
  • Attempting to run work on sub-optimal nodes.

When a node fails without closing sockets, all sessions that are blocked in an I/O wait (read or write) wait for tcp_keepalive. This wait status is the typical condition for an application connected by a socket. Sessions processing the last result are even worse off, not receiving an interrupt until the next data is requested.

Parent topic: Fast Application Notification (FAN)

How FAN is Used with Oracle Database and Applications

Fast Application Notification (FAN) is essential to prevent applications from stop responding on TCP/IP timeouts.

FAN events are published using Oracle Notification Service starting with Oracle Database 12.2. Advanced Queuing is used for FAN events only for older Oracle Call Interface (OCI) applications (OCI drivers before 12.2). The publication mechanisms are automatically configured as part of your Oracle RAC installation. There are specific settings needed on each client to enable the client to receive FAN events.

  • For OCI clients, the service attribute notification must be set on the server. For example srvctl modify service -db EMEA -service GOLD -notification TRUE. Also, for OCI clients you must set events to TRUE in the oraaccess.xml configuration file.
  • For ODP .Net clients, you must set HA events to TRUE in the oraaccess.xml in the connect string.
  • For Universal Connection Pool clients, set the pool property Fast Connection Failover to true (setFastConnectionFailoverEnabled(true)) in the properties file.

All clients can use the auto-configuration of ONS to receive events, but the clients still need settings to ensure they react to these events.

Oracle Net Services listeners and Global Data Services (GDS) are integrated with FAN events, enabling the listener and GDS to immediately de-register services provided by the failed instance and to avoid erroneously sending connection requests to failed instances.

Oracle connection pools use FAN to receive very fast notification of failures, to balance connections following failures, and to balance connections again after the failed components are repaired. So, when a service connecting to an Oracle Database instance starts, the connection pool uses the FAN event to route work to that resource, immediately. When a service for a database instance or node fails, the connection pool uses the FAN event to immediately interrupt applications to recover.

For cluster configuration changes, the Oracle Real Application Clusters (Oracle RAC) high availability framework publishes a FAN event immediately when a state change occurs in the cluster. Instead of waiting for the application to time out against the database and detect a problem, applications can receive FAN events and react immediately. With FAN, in-flight transactions are immediately terminated and the client notified when the instance fails.

FAN also publishes load balancing advisory events. Applications can take advantage of the load balancing advisory FAN events to direct work requests to the instance in the cluster that is currently providing the best service quality.

If you specify the connection load balancing goal CLB_GOAL_SHORT for a database service, then the listener uses the load balancing advisory when the listener balances the connection loads. When load balancing advisory is enabled, the metrics used for the listener are finer grained.

You can take advantage of FAN events in the following ways:

  • Applications can use FAN if you use an integrated Oracle client. The integrated clients for FAN events include Oracle JDBC Universal Connection Pool, ODP.NET connection pool, OCI session pool, Oracle WebLogic Server Active Gridlink for Oracle RAC, and OCI, JDBC-thin, and ODP.NET Managed and Unmanaged clients. The integrated Oracle clients must be Oracle Database 10g release 2 (10.2) or later to take advantage of the FAN high-availability events. The pooled clients can also take advantage of the load balancing advisory FAN events.

  • You can configure third-party application containers, such as those provided by Apache Tomcat and WebSphere, to use the built-in FAN support offered by using the Universal Connection Pool in place of the default pool, which is certified as a connection pool for third-party Java application servers including Apache Tomcat and WebSphere.

  • Use the FAN-aware capability of the Oracle drivers by using standard interfaces to test connections on get or release from the third-party connection pools in use by third-party application servers or custom applications.

    • This solution applies to standard Java applications through the use of the standard TNS connect string and ensures that the ons.jar and simpleFAN.jar files are available on the application CLASSPATH.

    • For the OCI/OCCI driver, the OCI_ATTR_SERVER_STATUS server context handle attribute is sensitive to FAN events and will return OCI_SERVER_NOT_CONNECTED if the connection has been affected by a FAN event.

  • You can implement FAN with server-side callouts on your database tier.

  • Applications can use FAN programmatically by using the JDBC and Oracle RAC FAN application programming interface (API) or by using callbacks with OCI and ODP.NET to subscribe to FAN events and to run event handling actions upon the receipt of an event.

For planned maintenance and applications using OCI or Pro* precompilers (and not using the OCI session pool or Tuxedo), an application must check OCI_ATTR_SERVER_STATUS. Add this check when sessions are returned to your own connection pool, and for idle connections, regularly. Following a FAN down event with planned maintenance, this attribute is set to OCI_SERVER_NOT_CONNECTED. The application closes the connection after reading this disconnected status. The session remains open for draining of active work until the application closes, providing error-free failover.

If you use one of the integrated clients listed in the first item of the preceding list, then, for DOWN events, the disruption to the application is minimized because the FAN-aware client terminates the connections to the failed instance or node before they are reused. Active work can be allowed to complete and, if there is a remaining instance, then continuous service can be maintained for ongoing work. Any sessions active when the instance or service stops are terminated and the application user is immediately notified. Incomplete transactions can be protected by Application Continuity, if it is enabled. Application users who request connections are directed only to available instances.

For UP events, when database services and instances are started, new connections are created so that the application can immediately take advantage of the extra hardware resources or additional capacity.

Parent topic: Fast Application Notification (FAN)

Requirements for Using FAN

Learn what you need to do to take advantage of FAN-aware capabilities in client drivers connecting to Oracle Real Application Clusters (Oracle RAC) databases.

Client drivers on releases after Oracle Database 12c release 2 (12.2) are FAN-aware, and FAN is enabled by default. This is also true for the JDBC Thin driver (12.2.0.1 and later), and Oracle Data Provider for Net (ODP.NET) drivers. A client driver can detect planned and unplanned FAN events, and take action beneath the application.

To take advantage of FAN-aware capabilities in the drivers, the following is required:

  • For the thin Java driver, beginning with release 12.2, FAN is automatically enabled by placing the ons.jar and simpleFAN.jar files on the CLASSPATH, and by using the recommended TNS format. Using the recommended TNS format automatically configures ONS. Also with the Java thin driver, FAN is supported for both planned and unplanned events. For unplanned outages, the FAN interrupt is immediate. For planned maintenance, configure the Java application servers or custom pools using standard interfaces to test connections on get or release from third-party connection pools. For example, depending on the application server, test TestConnectionsOnReserve, TestOnBorrow, or PreTest connections.

    With this approach, when a FAN event is received during planned maintenance, Fast Connection Failover (FCF) closes sessions when they are tested, because the application does not have a connection to the database at this time, and can retry for a new connection. The connection tests may use isValid, isClosed, isUsable, or PingDatabase.

  • At the time the SQL command runs, the database will drain the connection, if it is affected by the upcoming planned maintenance. Connection pools, data sources, and, in the programmatic case, customer applications, must all be ready to manage the recoverable error that occurs when the SQL command runs, which usually closes the physical connection.

  • Third-party Java application servers and Java applications can use the PooledConnection standard interface when developing connection pools.

  • Beginning with the 11.2.0.3 release of the Oracle Call Interface (OCI) and Oracle C++ Call Interface (OCCI) drivers, when the OCI_ATTR_SERVER_STATUS server context handle attribute returns OCI_SERVER_NOT_CONNECTED, the application must terminate the connection. Work will be drained for planned maintenance. Releases of the driver after 12.2.0.1 can also detect OCISessionRelease and OCIRequestEnd when it receives a planned DOWN event.

Parent topic: Fast Application Notification (FAN)

FAN Callouts

Fast Application Notification (FAN) callouts are server-side scripts or executables that run whenever a FAN event is generated.

You can design and build callouts to do many things, such as:

  • Log status information
  • Page DBAs or to open support tickets when resources fail to start
  • Automatically start dependent external applications that must be co-located with a service
  • Shut down services when the number of available instances for a database decreases, for example, if nodes fail
  • Automate the fail back of a service to preferred instances, if the -failback parameter is not sufficient

Parent topic: Fast Application Notification (FAN)

Fast Application Notification High Availability Events

Learn how the Fast Application Notification (FAN) event delivers information to a callout program.

In the following example, FAN event types are listed always as the first entry when you receive FAN information through a callout, as in the following examples:

#service UP when the service starts
SERVICEMEMBER VERSION=1.0 
   service=HRPDB1.example.com database=ractest 
   instance=ractest2 host=prod_host01_1 status=up reason=BOOT 
   card=1 timestamp=2019-10-24 09:11:51 timezone=+00:00 
   db_domain=example.com 
SERVICE VERSION=1.0
   service=HRPDB1.example.com database=ractest instance=ractest2 
   host=prod_host01_1 status=up reason=BOOT 
   timestamp=2019-10-24 09:11:51 timezone=+00:00 
   db_domain=example.com

#service DOWN 
SERVICEMEMBER VERSION=1.0 service=HRPDB1.example.com database=ractest
   instance=ractest2 host=prod_host01_1 status=down reason=USER 
   timestamp=2019-10-25 17:59:43 timezone=+00:00 db_domain=example.com 
   drain_timeout=120  
SERVICE VERSION=1.0 service=HRPDB1.example.com database=ractest 
   instance=ractest2 host=prod_host01_1 status=down reason=FAILURE 
   timestamp=2019-10-24 21:25:57 timezone=+00:00 db_domain=example.com

Note that the preceding examples normally display as one line.

FAN event types include:

  • DATABASE
  • INSTANCE
  • NODE
  • SERVICE
  • SERVICEMEMBER
  • SERVICEMETRICS

The DATABASE and INSTANCE types list the default database service as DB_UNIQUE_NAME.

All events except for NODE events include a db_domain field.

Events of SERVICEMETRICS type are load balancing advisory events.

The following table describes name-value pairs for the event parameters, and provides more information about load balancing events:

Table 6-2 Event Parameter Name-Value Pairs and Descriptions

Parameter Description
VERSION

Version of the event record. Used to identify release changes.
database

The unique name of the database supporting the service; matches the initialization parameter value for DB_UNIQUE_NAME, which defaults to the value of the DB_NAME initialization parameter.

instance

The name of the instance that supports the service.
host

The name of the node that supports the service or the node that has stopped; matches the node name known to Cluster Synchronization Services (CSS).
service

The service name; matches the name of the service as listed in DBA_SERVICES and is domain-qualified as appropriate. Refer to the following examples: 

SERVICEMEMBER VERSION=1.0 service=swingbench
 database=orcl instance=orcl_2 host=dev_host1 status=up
 reason=USER card=1 timestamp=2018-05-29 17:26:37
 timezone=-07:00 db_domain=

SERVICEMEMBER VERSION=1.0 service=swingbench.example.com
 database=orcl instance=orcl1 host=dev_host1 status=up
 reason=USER card=2 timestamp=2018-05-03 17:29:28
 timezone=-07:00 db_domain=example.com

SERVICEMEMBER VERSION=1.0 service=swingbench.example.com
 database=orcl instance=orcl2 host=dev_host1 status=up
 reason=USER card=1 timestamp=2018-07-03 17:29:18
 timezone=-07:00 db_domain=example.com
status

Values are UP, DOWN, NODEDOWN (this only applies to the NODE event type), NOT_RESTARTING, and UNKNOWN.

Notes:

  • When the node is down, the status is NODEDOWN, as opposed to DOWN for other event types.

  • When STATUS=NODEDOWN and REASON=MEMBER_LEAVE, a node has failed and is no longer part of the cluster, or a user has stopped a node.

  • When STATUS=NODEDOWN and REASON=PUBLIC_NW_DOWN, the node is up but it is unreachable because the public network is down because of either a failure or a user action.

  • Multiple public networks are supported by Oracle Clusterware. The FAN event reflects this fact.
reason

AUTOSTART, BOOT, DEPENDENCY, FAILURE, MEMBER_LEAVE, PUBLIC_NW_DOWN, USER.

Notes:

  • For DATABASE and SERVICE event types, REASON=AUTOSTART if, when the node started, the AUTO_START resource attribute was set to restore, and the resource was offline before the node started.

  • For DATABASE and SERVICE event types, REASON=BOOT if, when the node started, the resource started because it was online before the node started.

  • For SRVCTL and Oracle Enterprise Manager operations, REASON=USER describes planned actions for such operations as draining work.

card (cardinality)

The number of service members that are currently active; included in all SERVICEMEMBER UP events.

Here is an example of a SERVICEMEMBER UP event: 

SERVICEMEMBER VERSION=1.0 service=swingbench.example.com
database=orcl instance=orcl_2 host=dev_host3 status=up reason=USER card=1
timestamp=2018-07-12 14:46:46  timezone=-07:00 db_domain=example.com
incarn (incarnation)

For NODEDOWN events; the new cluster incarnation. This value changes each time a member joins or leaves the cluster.

Here is an example of a NODEDOWN event: 

VERSION=1.0 event_type=NODE host=dev_host2 incarn=175615351 status=nodedown
reason=member_leave timestamp=2019-10-24 05:55:06 timezone=+00:00
timestamp

The time according to Oracle Clusterware that an event occurs.
timezone

The time zone of Oracle Clusterware where the event occurred, given as GMT +/-hh:mm.
drain_timeout

Time in seconds during which a service will drain. Appears with SERVICEMEMBER events

vip_ips

VIP on a public network that has gone down. Part of a NODE event.

Here is an example of a NODEDOWN event: 

NODE VERSION=2.0 host=my-exa status=nodedown reason=public_nw_down 
incarn=0 timestamp=2019-10-24 09:02:35 timezone=+00:00 vip_ips=10.1.1.94

Some of the FAN event record parameters have values that correspond to values returned by the SYS_CONTEXT function using the default namespace USERENV, as shown in the following table:

Table 6-3 FAN Parameters and Matching Session Information

FAN Parameter Matching Session Information
SERVICE sys_context('userenv', 'service_name')
DATABASE_UNIQUE_NAME sys_context('userenv', 'db_unique_name')
INSTANCE sys_context('userenv', 'instance_name')
CLUSTER_NODE_NAME sys_context('userenv', 'server_host')

Parent topic: Fast Application Notification (FAN)

Subscription to High Availability Events

To monitor and notify applications about services, Oracle Real Application Clusters (Oracle RAC) uses Oracle RAC Fast Application Notification (FAN).

Oracle RAC uses FAN to notify applications about configuration changes and the current service level that is provided by each instance where the service is enabled. If you are using an Oracle Call Interface (OCI) client, or an ODP.NET client to receive FAN events, then you must enable the service used by that client to access the alert notification queue by using SRVCTL with the -notification parameter.

Parent topic: Fast Application Notification (FAN)

Using Fast Application Notification Callouts

Fast Application Notification (FAN) callouts are server-side program files that Oracle RAC runs immediately when high availability events occur.

You can use FAN callouts to automate activities when events occur in a cluster configuration, such as:

  • Opening fault tracking tickets
  • Sending messages to pagers
  • Sending e-mail
  • Starting and stopping server-side applications
  • Maintaining an up-time log by logging each event as it occurs
  • Relocating low-priority services when high priority services come online

To use FAN callouts, place a program file in the Grid_home/racg/usrco directory on every node that runs Oracle Clusterware. The program file must be able to run standalone when called, with optional arguments, from another program. The following is an example of a shell script, named callout.sh, which is placed in the Grid_home/racg/usrco directory: 

#! /bin/bash
FAN_LOGFILE= [your_path_name]/admin/log/'hostname'_uptime'.log
echo $* "reported="'date' >> $FAN_LOGFILE &

The previous example adds entries similar to the following in the log file, indicated by $FAN_LOGFILE in the shell script, each time a FAN event is generated: 

NODE VERSION=2.0 host=my-exa status=nodedown reason=public_nw_down 
incarn=0 timestamp=2019-10-24  09:02:35 timezone=+00:00 vip_ips=10.1.1.94

The contents of a FAN event record matches the current session of the user logged on to the database. The user environment (USERENV) information is also available using Oracle Call Interface (OCI) connection handle and descriptor attributes (using OCIAttrGet()). Use this information to take actions on sessions that match the FAN event data.

In general, events are only posted to user callouts on the node from which the event originated. For example, if the database on node1 goes down, then the callout is posted to node1, only. The only exceptions to this are node down and VIP down events—these events are posted to all nodes, regardless of from where they originated.

Related Topics

Parent topic: Fast Application Notification (FAN)

Configure for Unplanned Outages

You can assign services to one or more instances in an administrator-managed Oracle RAC database to hide outages.

Note:

The policy-managed database deployment option is desupported in Oracle Database 23ai.

It is a complex task for application developers to mask outages of a database session (instance, node, storage or network, or any other related component). As a result, errors and timeouts are often exposed to the end users, which can lead to user frustration, lost productivity, and lost opportunities. Together, FAN and Application Continuity mask outages from users and applications by recovering the in-flight work for impacted database sessions following outages. Application Continuity performs this recovery beneath the application, so that the outage appears to the application as a slightly delayed processing of the request.

If Oracle Real Application Clusters (Oracle RAC) detects an outage, then Oracle Clusterware isolates the failed component and recovers the dependent components. For services, if the failed component is an instance, then Oracle Clusterware attempts to maintain the cardinality of the service. If the service definition allows for failover and that is necessary to maintain cardinality, then failover occurs.

Fast Application Notification (FAN) events can occur at various levels within the Oracle Database architecture. To provide backward compatibility with earlier release Oracle Call Interface (OCI) clients, they are published through Oracle Notification Service and Advanced Queuing. FAN callouts can also be written to run on the database server in response to FAN events.

Note:

Oracle Database does not run Oracle RAC callouts with guaranteed ordering. Callouts are run asynchronously, and they are subject to scheduling variability.

FAN is published from a remaining node when the failed node is out of service. The location and number of instances in an Oracle RAC environment that provide a service are transparent to applications. Restart and recovery are automatic, including the restarting of the subsystems, such as the listener and the Oracle Automatic Storage Management (Oracle ASM) processes, not only the database. You can use FAN callouts to report faults to your fault management system and to initiate repair jobs.

Related Topics

Parent topic: Ensuring Application Continuity

Managing Planned Maintenance

To minimize service disruption to application users, Oracle Real Application Clusters (Oracle RAC) provides interfaces that relocate, disable, and enable services.

Parent topic: Ensuring Application Continuity

About Planned Maintenance Management

For repairs, upgrades, and changes, drain your sessions before stopping or relocating the services.

When you relocate a service, you indicate the service should run on another instance. When a service is stopped or relocated, FAN is published with a planned reason code, typically reason=user. Once you complete the operation, you can return the service to normal operation or enable the service and then restart it. When a service restarts, FAN is published with UP status codes.

Due to dependencies, if you manually shut down your database, then all of your services for that database automatically stop. If you want your services to automatically start when you manually restart the database, then you must set the management policy of the service to automatic. If you want to shut down only one instance of the database and you want the service to remain offered, then you can either relocate the service using srvctl relocate service or stop the instance using srvctl stop instance with the -failover option, which enables the service to automatically failover according to its failover policy.

In either case, Oracle recommends that work running under the service drain at request boundaries. The drain interval is specified as an attribute of the service or you can provide a drain interval on the SRVCTL command line.

Related Topics

Parent topic: Managing Planned Maintenance

Planned Maintenance Without User Interruption

Oracle recommends that you drain database sessions from the instance over a controlled time period from FAN-enabled Oracle or non-Oracle connection pools, or, beginning with Oracle Database 18c at the database, itself.

Draining database sessions is the safest way to migrate work without interrupting applications. When draining occurs at connection tests and outside of request boundaries, it is 100% correct. Applications continue with no interruption as existing work completes and new work acquires a session for the same service functioning at another instance, resulting in no errors returned to applications and no risk of incorrect database session states. For connection tests, the caller expects to receive a good or bad return code and is ready to handle the result, making inspecting connection tests a widely applicable and very powerful solution.

The service attributes -drain_timeout and -stopoption control the drain time period, and then how the service manages sessions that have not completed once this time period expires. Requests that complete and then check back in to the pool or close, can be directed to a new location that is not affected by the planned maintenance.

Application Continuity provides additional cover, giving continuous service for those requests that do not complete within the allotted drain time. Using any FAN-aware pool allows sessions to drain at request boundaries after receipt of the FAN planned DOWN event.

Because not every application uses an Oracle connection pool and not every application is FAN-aware, the database inspects sessions during planned maintenance seeking safe places to stop a session so that the application is not disrupted. After stopping the service, the database looks for safe places where the connection can be closed. When the connection is closed, the database cleans up the session.

Stopping a session at a safe place enables the application to open a new connection with the states that it needs. Draining sessions may take a period of work to flow through each session. There is no requirement that closing a session is immediate, rather the close must occur at a safe place that exposes no errors to the application, and, preferably, before the drain timeout period has expired.

Requests are far more important than transactions because they enable the issued work to complete. For draining requests, the Oracle Universal Connection Pool uses the drain timeout to gradually drain, which prevents an overload of logins on the instances drained, by slowly releasing the original sessions across the time period rather than all at once. Gradual draining has the benefit of not disturbing the other work ongoing at the target instances.

Both DRAIN_TIMEOUT and STOP_OPTION are service attributes that you can define when you add the service or modify it after creation. You can also specify these attributes using SRVCTL, which will take precedence over what is defined on the service. You can specify the -drain_timeout and -stopoption parameters when using the following SRVCTL commands:

  • srvctl add service

  • srvctl modify service

  • srvctl relocate service

  • srvctl stop service

  • srvctl stop database

  • srvctl stop instance

  • srvctl stop pdb

  • srvctl relocate pdb

To manage planned maintenance without user interruption:

  1. Use SRVCTL to relocate a singleton service or a service not running on all nodes. Use the -force flag with the previously listed SRVCTL commands, except add and modify. You must use the -force flag if you specify the -stopoption parameter on the command line when you run either srvctl relocate service or srvctl stop service. For example:
    $ srvctl relocate service –db mycdb01 –service myservice –drain_timeout 120
  –stopoption IMMEDIATE –oldinst mycdb01_01 -force
    The preceding command relocates the service named myservice01 from the instance named mycdb01_01 to any instance on which it is configured to run. Oracle Clusterware chooses the instance if you do not specify a target on the command line, and waits two minutes (in this example) for any active sessions to drain, after which any sessions remaining on mycdb01_01 are forcibly disconnected. The connection pool automatically releases a connection at a request boundary.

    Note:

    If the service you want to relocate is a uniform service that is currently running on all nodes, then the preceding command returns an error, unless the service is not up on all instances, in which case the preceding command example would succeed for a uniform service.
  2. The FAN planned DOWN event clears idle sessions from the connection pool immediately and marks active sessions to be released at the next check-in. These FAN actions drain the sessions from the instance without disrupting the users.

    Existing connections on other instances remain usable, and new connections can be opened to these instances if needed. The database also marks the sessions to drain. The database looks for connection tests and, in Oracle Database 19c and later, for safe places to failover. An implicit connection boundary with Transparent Application Continuity is such a place.

  3. Not all sessions, in all cases, will check their connections into the pool. Oracle recommends, as a best practice, to have a timeout period (by setting the -drain_timeout parameter), after which the instance is forcibly shut down or the service stopped, evicting any remaining client connections.
    After the drain interval expires, the -stopoption parameter is implemented, which you can define against a service or a database, as follows:

    • When stopping a service (srvctl stop service), you can specify one of the following stop options using the -stopoption parameter: TRANSACTIONAL or IMMEDIATE

    • When stopping a database (srvctl stop database), you can specify one of the following stop options using the -stopoption parameter: NORMAL, TRANSACTIONAL, IMMEDIATE, or ABORT

    The database stop options correlate to the service stop options, as follows:
    • NORMAL=NONE
    • TRANSACTIONAL/TRANSACTIONAL LOCAL=TRANSACTIONAL
    • IMMEDIATE/ABORT=IMMEDIATE

    For those services that are configured to use Application Continuity, an attempt is made to recover these remaining sessions, after they are terminated, masking the outage from users and applications.

  4. Once maintenance is complete, restart the instance and the services on the original node.
  5. The FAN UP event for the service informs the connection pool that a new instance is available for use, allowing sessions to be created on this instance at next request boundaries.

Related Topics

Parent topic: Managing Planned Maintenance

Managing a Group of Services for Maintenance

With Oracle Real Application Clusters (Oracle RAC), you can use SRVCTL to manage groups of services in your cluster.

  • Stopping a Group of Services Example
    See how you can use SRVCTL to stop services by node name, database name, Pluggable Database name, or instance name.
  • Starting Services
    You can use the srvctl start service command to start all services on a node, all services offered by a database, or all services offered by a Pluggable Database.
  • Pluggable Database-Level Operations
    You can use SRVCTL to manage services on Pluggable Databases.
  • Relocating Services
    You can use the srvctl relocate service command to relocate services to a target destination, which can be an instance, a node, or a database.
  • Stopping Services
    You can use the srvctl stop service command to stop all services on a node, all services offered by a database, or all services offered by a Pluggable Database.

Parent topic: Managing Planned Maintenance

Stopping a Group of Services Example

See how you can use SRVCTL to stop services by node name, database name, Pluggable Database name, or instance name.

Many enterprises run a large number of services, whether it be many services offered by a single database or instance, or many databases offering a few services running on the same node. You no longer need to run SRVCTL commands for each individual service but need only specify the node name, database name, Pluggable Database name, or the instance name for all affected services.

For example, if you want to stop all of the services running on a node called racnode01, then you can use the following command: 

$ srvctl stop service –node racnode01 –drain_timeout 60 –stopoption IMMEDIATE

The command stops all services running on racnode01, allowing a drain interval of 60 seconds. After 60 seconds, any remaining sessions are stopped immediately. The 60-second drain timeout interval overrides any attribute setting on any of the services.

The command can also be qualified to stop the databases on a node, as in the following example:
$ srvctl stop instance 	-node racnode01 -drain_timeout 60 –stopoption TRANSACTIONAL LOCAL -failover –force
When you specify the -failover parameter:

  • All services are relocated, if possible, respecting the drain timeout interval and the stop option specified.

  • Any services that cannot be failed over are stopped, using the stop option specified.

  • Wait for the length of the drain timeout interval or until all sessions for targeted services are removed, whichever is sooner.

  • All instances stop according to the stop option specified.

When you specify the –stopoption TRANSACTIONAL LOCAL parameter:

  • Remaining services stop according to the drain timeout interval and stop option specified.

  • Wait for the length of the drain timeout interval or until all sessions for targeted services are removed, whichever is sooner.

  • The instance stops using the TRANSACTIONAL LOCAL stop option.

Starting Services

You can use the srvctl start service command to start all services on a node, all services offered by a database, or all services offered by a Pluggable Database.

To start services, you can also supply a list of services (a subset of all services) to the srvctl start service command that you want to start. Additionally, you can provide a node restriction, used in conjunction with the database option, for all services that can be started on a particular node. You can restrict the srvctl start service command to start only the parallel query service by specifying the -pq parameter.
The following examples illustrate how you can start services:

To start all of the services offered by a single Pluggable Database:

$ srvctl start service –db myRACCDB01 –pdb myPDB01 –startoption OPEN

To start all services on a given database and any of its Pluggable Databases:

$ srvctl start service –db myRACDB

To start a list of services on a given database, regardless of any Pluggable Database with which they are associated:

$ srvctl start service –db myRACDB –service "myFirstService,mySecondService,myThirdService"

To start all services for a database that can run on a given node:

$ srvctl start service –d myRACDB –node racnode01
Pluggable Database-Level Operations

You can use SRVCTL to manage services on Pluggable Databases.

To start all services for a Pluggable Database, for all instances or a single instance:
$ srvctl start service -db db_name -pdb pdb_name [-instance instance_name]

To stop all services for a Pluggable Database, for all instances or a single instance:

$ srvctl stop service -db db_name -pdb pdb_name [-node node_name | -instance 
   inst_name] [-stopoption stop_option] [-drain_timeout timeout]
   [-force [-noreplay]]

Note:

The -pdb pdb_name parameter is optional. If you omit the Pluggable Database name, then the operation occurs for the entire Container Database (all Pluggable Databases within this container).
Relocating Services

You can use the srvctl relocate service command to relocate services to a target destination, which can be an instance, a node, or a database.

In the following command examples, all services are relocated from the named database, Pluggable Database, instance, or node. The services will only relocate if the target can support that service, as defined by the service configuration. Any services that cannot be relocated remain at the original location. A placement error is recorded against any services that could not be relocated, or were already running at the new target. Services that fail to relocate remain running at their original location, and any sessions remain active.

$ srvctl relocate service –db myRACCDB –oldinst RACCDB_01 –newinst RACCDB_03
  -drain_timeout 30 -stopoption immediate

or

$ srvctl relocate service –db myRACCDB –pdb myPDB01 –currentnode racnode01
  –targetnode racnode02 -drain_timeout 30 -stopoption immediate

The relocate operation starts the service in the new location before stopping the service in its existing location.

If you do not specify a target destination, then Oracle Clusterware relocates all services or specific services from the specified database, Pluggable Database, instance, or node, as in the following examples:

$ srvctl relocate service –db myRACCDB –service "myService01,myService02"
  -drain_timeout 30 -stopoption immediate

or

$ srvctl relocate service –db myRACCDB –pdb myPDB01 -drain_timeout 30
  -stopoption transactional

If there is no valid target available, then the service remains at the original location and the sessions remain active. You must examine the services and stop them if that is what you want to do.

When you relocate a service, it starts at the new location before it stops at the original location. Oracle Clusterware can start that new instance or Pluggable Database as a dependency. When specified, the -drain_timeout and -stopoption parameters override the service attributes.

Stopping Services

You can use the srvctl stop service command to stop all services on a node, all services offered by a database, or all services offered by a Pluggable Database.

When you want to stop a subset of services, you can also supply a list of services (a subset of all services) that you want to stop to the srvctl stop service command. You can also restrict the srvctl stop service command to stop only the parallel query service by specifying the -pq parameter.

To stop all of the services offered by a single Pluggable Database:

$ srvctl stop service –db myRACCDB01 –pdb myPDB01 –drain_timeout 15 –stopoption TRANSACTIONAL

To stop all services on a given database and any of its Pluggable Databases:

$ srvctl stop service –db myRACDB –drain_timeout 15 –stopoption IMMEDIATE

To stop only a subset of the services offered by a database:

$ srvctl stop service –db myRACDB –service "myFirstService,mySecondService,
  myThirdService" –drain_timeout 60 –stopoption IMMEDIATE

Note:

If you use the –wait YES SRVCTL command line parameter, then the –stopoption parameter is not enforced until the entire drain timeout interval has expired, even if all of the sessions have exited prior to this interval completing.

Server Draining Ahead of Planned Maintenance

Before planned maintenance, drain or failover database sessions at the database instance so application work is not interrupted.

When you prepare for planned maintenance, you must stop or relocate the services that are using the server infrastructure. Relocating services is done over a period of time prior to the planned outage and is based on the nature of work associated with each service.

The procedure for rolling planned maintenance moves services in advance of maintenance to another database instance, and notifies the client-side drivers, connections pools, the database instance itself, and other subscribers that maintenance is pending, and what needs to be drained (either connections or sessions using this service). Once notified of draining, a Fast Application Notification (FAN) event is sent and the client pools behave as described elsewhere, in addition, the database begins to search for safe places to release connections and, if needed, to migrate the connections.

Moving or stopping a service triggers a FAN notification that is received by the subscribing Oracle drivers and Oracle connection pools. The FAN notification also triggers session draining at the server. Immediately, new work to that service is directed to another functioning instance of that service. Existing sessions are marked for release after their work completes. As work completes and the connections are returned to the connection pool, either the Oracle driver or the connection pool terminates these sessions.

Draining Sessions at the Database

For OLTP applications, application servers, and custom applications, which all have their own connection pools that borrow and return database sessions, it is safe to drain a database session when it is no longer borrowed. The optimal point for the Oracle server infrastructure to close a session is when the application server tests the validity of that connection. No error is returned to the application when the connection pool manager tests the validity of connections when borrowing and releasing, and also finds that the connection is no longer valid.

A safe place is a point where an application is not disturbed. In the case of connection pools, that means connections that are not borrowed (checked-in), and, in the case of applications, the same applies at the point of borrowing or returning a connection. At this time, all work is either complete or not started. The database can also fail over connections when all states can be restored transparent to the application.

Starting with Oracle Database 18c, the database uses an extensible set of rules and heuristics to detect when to take the database session away. When draining starts, the database session persists at the database until a rule is satisfied. The rules include the following:
  • Standard application server tests for validity
  • Custom SQL tests for validity
  • Request boundaries are in effect and no request is active
  • Request boundaries are in effect and the current request has ended
  • The session has one or more session states that are recoverable, and can be recreated at failover

In the case of connection tests, for example, it is standard practice for application servers, pooled applications, job schedulers, and others, to test connections when borrowed from connection pools, when returned to the pool, and at batch commits. When draining, the database intercepts the connection test, closes the connection and returns a failed status for the test. The application layer issuing the connection test is ready to handle a failed return status and, typically, issues a further request, to obtain a different connection. The application is not disturbed.

Not all sessions can be drained, such as when a connection is not returned to the pool or when FAN is not in use. If Transparent Application Continuity or Application Continuity is enabled, then the server detects request boundaries where Application Continuity can recover the session fast. The server can interrupt the session, which Application Continuity recovers elsewhere (such as, to another server in the Oracle RAC cluster) with no interruption.

For database sessions that do not drain, the database must finds a break point when the session can be replaced. At a break point, a connection can be failed over transparently when states are known and recoverable. Break points can be transaction boundaries, a request starting (beginRequest), before calls are processed in that request, and patterns, such as an audit call that signals that a request is starting or ending. Break points apply only when states are known to be restorable. Starting with Oracle Database 21c, the database software determines where to failover the session and invokes Application Continuity to failover the session.

Failing over connections requires that you enable Application Continuity, Transparent Application Continuity, or transparent application failover (TAF), depending on your application.

Adding, Disabling, Enabling, and Removing Connection Tests for Draining at the Server

You can add a SQL connection test to a service, a Pluggable Database. To add a new connection test to your PDB, use ALTER SESSION SET CONTAINER to switch to your PDB.

There are four SQL connection tests added for every database service and Pluggable Database service, by default, so, if an application uses these following SQL connection tests on the connection, then you do not to need to add them:
SELECT 1 FROM DUAL;
SELECT COUNT(*) FROM DUAL;
SELECT 1;
BEGIN NULL;END

Note:

  • When you use a connection test, the outcome of the connection test applies to that session only. Do not use connection tests to make general decisions about the instance and to make a determination to stop more than the session to which the test applies. Disable connection pool properties for flushing and destroying the pool on connection test failure. This is important for Oracle WebLogic Server data sources.
  • For external monitors, you are recommended to prefix the SQL used in that monitor with a hint /*+ MONITOR */ for SQL used inside the monitors, so that the monitor does not drain.
  • To add a server-side SQL connection test for a service, use a SQL statement similar to the following:
    SQL> execute dbms_app_cont_admin.add_sql_connection_test('select dummy from dual','sw_orcl');
    To add a server-side SQL connection test for a Pluggable Database, use a SQL statement similar to the following:
    SQL> execute dbms_app_cont_admin.add_sql_connection_test('begin null;end;');

    After you add the SQL connection test, it will be enabled by default.

  • You can disable a SQL connection test if you do not need it or it is not in use by logging on to a Pluggable Database and using a SQL statement similar to the following:
    SQL> execute dbms_app_cont_admin.disable_connection_test(dbms_app_cont_admin.sql_test,'select dummy from dual');

    By default, the ping test and end request test are disabled but, if you want to disable them after enabling them, then you can use either of the following SQL statements:

    If you want to disable the ping test, then use a SQL statement similar to the following:
    SQL> execute dbms_app_cont_admin.disable_connection_test(dbms_app_cont_admin.ping_test);
    If you want to disable the end request test, then use a SQL statement similar to the following:
    SQL> execute dbms_app_cont_admin.disable_connection_test(dbms_app_cont_admin.endrequest_test);
  • You can enable a SQL connection test after you disable it by logging on to the Pluggable Database and using a SQL statement similar to the following:
    SQL> execute dbms_app_cont_admin.enable_connection_test(dbms_app_cont_admin.sql_test,'select dummy from dual');

    You can also enable the ping test and end request test if they are disabled by using either of the following SQL statements:

    If you want to run any test that uses ping such as isValid, isUsable, OCIping, or connection.status, then use a SQL statement similar to the following:
    SQL> execute dbms_app_cont_admin.enable_connection_test(dbms_app_cont_admin.ping_test);
    If you want to enable draining at the end of a request, then use a SQL statement similar to the following:
    SQL> execute dbms_app_cont_admin.enable_connection_test(dbms_app_cont_admin.endrequest_test);
    If you want to disable draining on the end of a request, then use a SQL statement similar to the following:
    SQL> execute dbms_app_cont_admin.disable_connection_test(dbms_app_cont_admin.endrequest_test);
  • You can remove a SQL connection test if it is not needed by logging on to the Pluggable Database and running SQL statements similar to the following:
    SQL> execute dbms_app_cont_admin.delete_sql_connection_test('select dummy from dual','sw_orcl');
SQL> execute dbms_app_cont_admin.delete_sql_connection_test('begin null;end;');

  • If you want to modify rules for a service that belongs to a specific PDB, then switch to that PDB and modify the rules. For example, for eBusiness Suite:

    SQL> alter session set container=’VISPRD’;
SQL> execute dbms_app_cont_admin.add_sql_connection_test('Begin null; End ;');
SQL> executedbms_app_cont_admin.add_sql_connection_test('Begin null; End ;', ‘VISPRD’);

Every application server has a feature to test the validity of the connections in their respective connection pools, which is set either by a configuration property or at the administrative console. The purpose of the test is to prevent vending an unusable connection to an application, and when an unusable connection is detected, to remove it when released to the pool.

Across the various application servers, the tests have similar names. The tests offered use various approaches, the most common being a SQL statement. Oracle recommends that Java application servers use the standard Java call connection.isValid. Beginning with Oracle Database 18c, these tests are used to drain the database. Also beginning with Oracle Database 18c, the database drains sessions without using FAN by inspecting sessions for safe draining points.

The following table describes the standard connection tests available for several of the more common application servers:

Table 6-4 Standard Connection Tests for Some Common Application Servers

Application Server Connection Test to Database

Oracle WebLogic Server
The tests offered include: 
  • dbms_app_cont_admin.enable_connection_test(dbms_app_cont_admin.sql_test,'select 1 from dual');

  • TestConnectionsonReserve:

    isUsable, isValid, or PingDatabase

  • TestConnectionsOnCreate (SQL syntax) for server draining: 

    Select 1 from dual;

Oracle WebLogic Server Active Gridlink
The test is embedded:
isUsable

IBM WebSphere

 
dbms_app_cont_admin.enable_connection_test(dbms_app_cont_admin.sql_test,'select 1 from dual');

Pretest connections (SQL syntax) for server draining:

Select 1 from dual;

RedHat JBoss
check-valid-connection-sql (SQL syntax):
dbms_app_cont_admin.enable_connection_test(dbms_app_cont_admin.sql_test,'select 1 from
      dual');

Apache Tomcat
There are two tests available—testOnBorrow and testOnReturn—and they both use SQL syntax to test the connection to the database:
dbms_app_cont_admin.enable_connection_test(dbms_app_cont_admin.sql_test,'select 1 from dual');
Application server uses:
Select 1 from dual;

Oracle recommends that you use the following format for supporting automatic configuration of Oracle Notification Services (ONS), so that you can receive FAN events (over ONS):

Example 6-6 Automatic Configuration of FAN

alias =(DESCRIPTION =
 (CONNECT_TIMEOUT=90)(RETRY_COUNT=20)(RETRY_DELAY=3)(TRANSPORT_CONNECT_TIMEOUT=3) 
   (ADDRESS_LIST =
   (LOAD_BALANCE=on)
   ( ADDRESS = (PROTOCOL = TCP)(HOST=primary-scan)(PORT=1521)))
   (ADDRESS_LIST =
   (LOAD_BALANCE=on)
   ( ADDRESS = (PROTOCOL = TCP)(HOST=secondary-scan)(PORT=1521)))
  (CONNECT_DATA=(SERVICE_NAME = gold-cloud)))

Related Topics

Parent topic: Managing Planned Maintenance

Planned Failover with Application Continuity

Planned Failover is a failover that is invoked by Oracle Database at points where the database knows that the session is replayable using Application Continuity and that the session is expected not to drain.

Planned failover is an automatic solution that is used for relocating sessions during planned maintenance for batch and long running operations that are not expected to complete in the specified drain timeout period. Session migration is also used with ALTER SYSTEM DISCONNECT SESSION statement.

Planned failover is activated when draining starts. A rules engine decides when to invoke a failover.

  • The database maintains statistics regarding the rate and size of requests, and when a failover feature such as Application Continuity is enabled, then the database also maintains statistics for the level of protection for calls for replay, and the session state that needs to be recovered.

  • The database knows when Transparent Application Continuity or Application Continuity are enabled on a session and whether session state is tracked and recoverable, and when failover is enabled, if that failover is likely to be disabled before the drain timeout expires.

  • The database knows when a session is not expected to drain, and that a session is likely to recover, and how much replay it would need to run if it needs to replay.

  • The database knows if Fast Application Notification is enabled.

  • The database knows when request boundaries are discovered for Transparent Application Continuity.

A session failed over by the database is marked in the alert log so that you can find more information about the failed over session.

To use planned failover, follow these steps:

  1. Enable Application Continuity or Transparent Application Continuity.

  2. Set the service attributes -drain_timeout and -stopoption on your services.

  3. During maintenance, drain your services by relocating or stopping them. For Data Guard, you can use switchover wait with Data Guard Broker.

Parent topic: Managing Planned Maintenance

Transaction Guard for Improving Client Failover

Transaction Guard is a developer feature that your applications can use to determine COMMIT_OUTCOME following recoverable errors.

Parent topic: Ensuring Application Continuity

About Transaction Guard

Transaction Guard provides a fully integrated tool for applications to use to achieve idempotence automatically and transparently, and in a manner that scales.

Transaction Guard uses Logical Transaction ID (LTXID) to avoid submitting duplicate transactions. This function is referred to as transaction idempotence. The LTXID persists on commit, and is reused following a rollback. During normal runtime, a LTXID is automatically held in the session at both the client and server for each database transaction. At commit, the LTXID is persisted as part of committing the transaction and the next LTXID to use is returned to the client.

Note:

Application Continuity protects outages only for normal applications. Administrative tasks through AC-enabled services are not supported and you must avoid them.

Applications have a problem if they fail to recognize that the last submission has committed, or that it will commit sometime soon, or that the last submission has not run to completion. Applications failing to recognize these submission states can result in users who resubmit or applications that use their own replay to issue duplicate requests, repeating changes that are already committed to the database, and other forms of logical corruption. Transaction Guard can be used to solve this problem.

Application Continuity automatically enables and uses Transaction Guard, but you can also enable Transaction Guard independently. If the application has implemented an application-level replay, then it requires the application to be integrated with Transaction Guard to provide idempotence.

Transaction Guard for XA Transactions

Transaction Guard also supports XA-based transactions, which are transactions that are an option for transaction managers, such as Oracle WebLogic Server, Oracle Tuxedo, and Microsoft Transaction Server (exposed to Oracle Database through Oracle ODP.NET) when using one-phase commit with XA.

Related Topics

Parent topic: Transaction Guard for Improving Client Failover

Database Native Transaction Guard

Database Native Transaction Guard enhances the existing Transaction Guard protocols to use the database Transaction ID (DB XID) to impose at-most-once processing.

Each logical transaction ID (LTXID) is tagged to a single or multiple server-side transactions. A server-side transaction is identified by a database transaction identifier or DB XID. Persisting the LTXID in the LTXID_TRANS table as part of the commit of each DB XID introduces overheads in normal transaction operation. This overhead with respect to performance and redo generation is visible in CPU overhead in the case of small transactions because the extra work required by Transaction Guard is not insignificant relative to the work done by the user transaction.

Note:

Database Native Transaction Guard is the default option when you use Application Continuity.

The DB XID is unique within a Pluggable Database (PDB) with local undo and uniquely identifies a transaction in the database. Using the DB XID does not require persistence in a separate table, thus does not incur extra redo generation or performance overheads. The FORCE_OUTCOME procedure takes advantage of the DB XID, instead of relying on the persistence of the LTXID in the separate table.

Database Native Transaction Guard provides the Transaction Guard functionality with lower overheads by avoiding extra writes to the LTXID_TRANS table. This feature does not affect any client-side APIs and does not require administrative changes.

By default, the COMMIT_OUTCOME_FAST_PATH service parameter is set to TRUE if COMMIT_OUTCOME is set to TRUE. Set the COMMIT_OUTCOME_FAST_PATH service parameter to FALSE to disable Database Native Transaction Guard.

Parent topic: Transaction Guard for Improving Client Failover

Transaction Guard Support During Major Database Version Upgrades

Starting with Oracle Database 23ai, Transaction Guard works during DBMS_ROLLING operations to ensure continuous application functions during switchover, issued by DBMS_ROLLING to Transient Logical Standby.

Transaction Guard returns the commit outcome of the current in-flight transaction when an error or outage occurs. Applications embed the Transaction Guard APIs in their error handling procedures to ensure that work continues without any in-flight work lost or duplicate submissions after an outage. Transaction Guard provides idempotence support to ensure that a commit occurs not more than once when a transaction is re-processed (replay) after an outage.

Transaction Guard ensures continuous application operation during the DBMS_ROLLING switchover operation to Transient Logical Standby. Transaction Guard ensures that the last commit outcome of transactions in the in-flight sessions during a switchover outage is used to protect the applications from duplicate submissions of the transactions on replay.

Transaction Guard maintains a transaction history table called LTXID_TRANS that has the mapping of logical transaction identifiers (LTXIDs) to database transactions. For a failover to succeed after an outage, the changes to LTXID_TRANS from the primary database must first replicate and apply to Transient Logical Standby. With supplemental logging enabled for the DBMS_ROLLING procedure, Transaction Guard uses SQL to allow supplemental capture of LTXID_TRANS at CDB and PDB levels. The capture process replicates the LTXID_TRANS table and the apply process reads and recreates the LTXID_TRANS tables for the logical standby, along with the committed user transactions.

As a part of its support for the DBMS_ROLLING procedure, Transaction Guard performs the following functions:

  • Tracks when the primary database is in DBMS_ROLLING mode (when the database upgrade is initiated)

  • Checks that supplemental logging is in use

  • Records the redo vector for the primary key (PK) at runtime while in supplemental logging mode

  • Waits for all current updates to finish and replicate to the logical standby before performing the LTXID replication

  • Replicates the LTXID_TRANS tables and applies the redo to Transient Logical Standby for each PDB

  • Provides a mechanism for failover to know about successful LTXID replication

  • Enforces last commit outcome for inflight sessions on replay after an outage

  • Handles new users during supplemental capture and apply process to ensure that any apply does not create mismatched logged-in UIDs (user IDs) at the target database

Parent topic: Transaction Guard for Improving Client Failover

Transaction Guard Configuration Checklist

Oracle recommends that you use this configuration checklist before you configure services for Transaction Guard.

Before configuring services for Transaction Guard, complete each check on this list:

  • Grant permission to the application user who will call GET_LTXID_OUTCOME, as follows: 

    GRANT EXECUTE ON DBMS_APP_CONT to user_name;

    Note:

    Do not run this statement if you use Application Continuity.

  • Locate and define the transaction history table for optimal performance.

    The transaction history table (LTXID_HIST) is created, by default, in the SYSAUX tablespace when you create or upgrade an Oracle Database. New partitions are added when you add instances, using the storage of the last partition. If the location of transaction history table is not optimal for performance, then you can move it to another tablespace and create partitions there. For example, the following statement moves the transaction history table to a tablespace named FastPace: 

    ALTER TABLE LTXID_TRANS move partition LTXID_TRANS_1 tablespace FastPace
   storage ( initial 10G next 10G minextents 1 maxextents 121 );

  • Set values for the COMMIT_OUTCOME and RETENTION_TIMEOUT service parameters.

  • If you are using Oracle Real Application Clusters (Oracle RAC), Oracle Data Guard, or Oracle Active Data Guard, then to obtain rapid notification of an outage, Oracle recommends that you use Fast Application Notification (FAN).

Parent topic: Transaction Guard for Improving Client Failover

Configuring Services for Transaction Guard

To configure services to use Transaction Guard, review and set the required service parameters.

Review and set the following parameters:

  • COMMIT_OUTCOME: Set the COMMIT_OUTCOME service parameter to TRUE.

    The COMMIT_OUTCOME service parameter determines whether the transaction commit outcome is accessible after the COMMIT has run and an outage has occurred. While Oracle Database has always made COMMIT durable, Transaction Guard makes the outcome of the COMMIT durable. Applications use this durability of a commit to enforce the status of the last transaction run before an outage.

  • RETENTION_TIMEOUT: Use the RETENTION_TIMEOUT service parameter with COMMIT_OUTCOME. The RETENTION_TIMEOUT service parameter determines the amount of time, in seconds, that the COMMIT outcome is retained. Oracle recommends that most installations use the default value.

In the following example, the SRVCTL command configures a service named sales for Transaction Guard: 

$ srvctl add service -db crm -service sales -preferred crm_1,crm_2
  -available crm_3,crm_4 -commit_outcome TRUE -retention 86400
  -notification TRUE

You can also modify an existing service to configure it for Transaction Guard by using the srvctl modify service command.

Note:

Do not use the default database service, the service which has the name set to the value of db_name or db_unique_name. The default service is used for administrative purposes and does not have the same properties as user-created services.

Related Topics

Parent topic: Transaction Guard for Improving Client Failover

Application Continuity During Major Database Version Upgrades with DBMS_ROLLING

Starting with Oracle Database 23ai, Application Continuity and draining of database sessions are supported during major database version upgrades and database restructuring when using Application Continuity with DBMS_ROLLING.

This is the strategic approach for applying non-rolling patches. This feature also supports the upgrade from one major release to another major release. For the major release upgrade from Oracle Database 19c to Oracle Database 23ai, it is supported for Oracle Autonomous Database - Dedicated only.

Applications are continuously available during DBMS_ROLLING when you use Application Continuity. This means that your applications can continue through the database upgrade process.

DBMS_ROLLING support for Application Continuity enables transparent failover of database sessions across different major versions of a database so that the interruption is minimized during major database upgrades and restructuring of that database. This feature relocates the database sessions in real time to a different logical version of the database, where it is the same database in terms of data but is a different major version or major database restructuring.

This feature provides failover of the application during the transition period, termed switchover, to the new logical database. Although the transition to the new database is a planned maintenance operation, but the applications are unaware of when the switchover step will occur to the new logical database. The infrastructure monitors and starts the switchover automatically when the lag between the databases is sufficiently small enough for the switchover to run in low numbers of seconds to minutes.

The database sessions failover to the new primary database thereby keeping the application operational. The failover minimizes the interruption and thus enable business continuity during major upgrades and major restructures. This feature integrates support for Oracle Application Continuity with a Logical Standby when using DBMS_ROLLING.

Parent topic: Ensuring Application Continuity

Reset Database Session State

When you set the RESET_STATE service attribute to LEVEL1 or LEVEL2, the session state set by the application in a request is cleared when a request to the database ends.

Note:

Starting with Oracle Database 23ai, RESET_STATE does not have a dependency on the Application Continuity feature.

RESET_STATE is a very important database feature that enables developers to rely on the session state being clean when a session is returned to a connection pool with request boundaries. This can be an Oracle connection pool or a custom connection pool with added request boundaries. Set RESET_STATE to LEVEL1 to clear the unrestorable session state in the request.

Setting session state in a request leaves the session dirty, meaning that subsequent usages of that session can see that session state if it is not cleaned. For example, when an application borrows and returns a connection to a connection pool, the next usage of that connection can see the session state used previously. Without RESET_STATE or if RESET_STATE is set to NONE, application developers must cancel their cursors and clear session state that has been set before returning their connections to a pool for reuse.

The RESET_STATE service property also improves your protection when using Transparent Application Continuity because the session state is clean at the beginning of the next request.

Parent topic: Ensuring Application Continuity