Checkpointing, Recovery, and Delivery Semantics
Learn about the checkpoint, recovery, and delivery from data streams configuration.
The pos public contract
Every non-metadata DML or DDL record includes an opaque pos token by default. Internally, the token carries enough GoldenGate positioning information to identify a record, including commit or sequence information, transaction identity, and record sequencing. Its encoding is not a public source-coordinate format and can vary by source type and release.
| Do | Do not |
|---|---|
| Store the exact string after successful processing. | Parse it to obtain an Oracle SCN, PostgreSQL LSN, MySQL binlog coordinate, SQL Server LSN, or any other source-specific offset. |
| Scope it to the Data Stream, source trail contract, and consumer. | Compare positions lexically or numerically to determine order. |
Percent-encode it when used as begin in a URL. |
Trim, normalize, concatenate, decrypt, or rewrite it. |
Use separate csn, xid, opSeqno, and transInd fields for diagnostics or transaction analysis. |
Reuse a position from another stream, a different filter contract, or another environment. |
Source portability: The client recovery algorithm is intentionally the same for Oracle, PostgreSQL, MySQL, SQL Server, and other supported sources: persist the opaque pos and return it unchanged. Source-specific coordinate parsing is neither required nor supported by the Data Streams client contract.
A checkpoint must represent the last record whose business effect is durable. Saving before the business write can lose data after a crash. Saving after a non-idempotent write in a separate transaction can produce a duplicate after a crash.
End-to-end exactly-once patterns
| Pattern | How it works | Use when |
|---|---|---|
| Atomic sink and checkpoint transaction | Write the business record and new pos in the same database transaction. |
The sink and checkpoint can share a transactional datastore. |
| Idempotency key | Use pos, or a business key plus operation identity, as a unique key; retrying the same record becomes a no-op. |
The sink supports uniqueness or conditional writes. |
| Durable staging log | Append the record and pos atomically to a durable log, then process the log independently. |
Business processing is slow, multi-step, or spans systems. |
| At-least-once with reconciliation | Accept duplicates, deduplicate periodically, and reconcile counts or checksums. | The use case tolerates eventual correction. |
NOTE: Data Stream exactlyOnce describes service-side duplicate suppression and recovery response. It does not make an arbitrary downstream side effect exactly once because the WebSocket protocol has no per-record transactional acknowledgment with the client.
Quality of service
See the Quality of Service guidelines from the following table:
| Mode | Duplicate | Saved position unavailable | Recommended use |
|---|---|---|---|
exactlyOnce |
Suppresses duplicates recognized during recovery and source trail restart/abend handling. | Connection fails with an error. | Default for durable business processing. Pair with atomic or idempotent consumer logic. |
atLeastOnce |
Does not suppress duplicates during recovery and source trail restart/abend handling. | Connection fails with an error. | Use when replay is preferred and the consumer is designed to deduplicate. |
atMostOnce |
Suppresses recognized duplicates. | Service finds the next available record and continues, which can skip data. | Use only when data loss is explicitly acceptable and forward progress is more important than completeness. |
Transaction metadata and ordering
Refer to the following table for transaction metadata and ordering requirements:
| Field | Purpose | Limitations |
|---|---|---|
xid |
Groups records associated with a source transaction. | Format is source-dependent and should be treated as a string. |
csn |
Exposes the source commit sequence number when configured. | Meaning and format are source-dependent; do not use as the restart token. |
opSeqno |
Record sequence number within the transaction. | Enable and optionally rename it explicitly. |
transInd |
B first, M middle, E last, or W whole/single-operation transaction. |
Filtering can remove records and make the observed sequence incomplete. Coordinated processing can also reduce usefulness. |
pos |
Unique recovery position for the record. | Opaque; do not sort or derive transaction boundaries from its text. |
Do not assume that a connection always begins at a transaction boundary. A timestamp lookup or retained-trail boundary can begin after earlier operations are no longer available. If the application requires complete source transactions, design a transaction buffer, define limits for large transactions, and specify what to do when the beginning of a transaction is unavailable.
When a saved position is unavailable
To check on a saved position that becomes unavailable, check the following:
-
Confirm that the checkpoint belongs to the same Data Stream, source trail, and filter/output contract.
-
Confirm that the value was percent-encoded without being changed.
-
Confirm that the trail segment containing the position has not been purged.
-
With
exactlyOnceoratLeastOnce, treat the error as a recovery incident. Restore trail history, choose an approved timestamp or earliest position, and reconcile downstream state before resuming. -
With
atMostOnce, the service can continue at the next available record. Record the resulting data gap and execute the business-approved reconciliation procedure.
Trail retention planning
Set trail retention to exceed the recovery point objective plus operational detection and response time. For example, a consumer that can be offline for 24 hours and requires 12 hours for incident response needs more than 36 hours of available trail, with additional margin for large backlogs and maintenance. bufferSize does not extend trail retention.