8 Limiting Resource Use and Protecting Persisted Data

Bound supported serialization containers and preserve explicit trust boundaries for transactions, persistence metadata, stored values, and native indexes.

Resource limits and type validation protect different aspects of input handling. A well-typed container can still be too large, and a small object can still be an unexpected type. Apply both policies at their documented consumers. Limit storage-member heap and admission capacity using the deployment and service controls appropriate to the workload.

• Use Configuring Serialization Container Limits for generic serializer policy and POF consumers.

• Use Transaction Bridges and Persistence Metadata before upgrading transaction or persistence workflows.

• Use Protecting Memcached Values and Native Indexes for optional protocol and native-library boundaries.

Persisted Data Is Still Input

Restrict writes to persistence, snapshot, archive, and index directories to trusted processes. Restrict import and recovery operations. Keep backups of both application data and the configuration/classes required to interpret it. Restore in an isolated, appropriately patched environment before accepting production traffic.

Structural validation, class filters, and checksums are not keyed authentication of a snapshot or journal. Cryptographic snapshot/journal integrity and key lifecycle require a separately supported format. Do not infer tamper-proof persistence from the metadata parsing controls described here.

• Configuring Serialization Container Limits
  Set serializer-owned byte, element, and map-entry limits and verify the policy on the service that decodes the payload.
• Transaction Bridges and Persistence Metadata
  Account for typed metadata, bridge-local type restrictions, and legacy payload behavior when transaction and persistence data cross an upgrade boundary.
• Protecting Memcached Values and Native Indexes
  Treat optional Memcached listeners and HNSW index files as separate trust and resource boundaries.

Configuring Serialization Container Limits

Set serializer-owned byte, element, and map-entry limits and verify the policy on the service that decodes the payload.

Serialization container limits are owned by the serializer. POF consumes this policy; do not assume that every third-party serializer or every object-graph path enforces all three limits. Structural checks reject negative, impossible, truncated, and overflow-prone input independently of the security profile.

Limit	Hardened default	Property
Container bytes	64 MiB, or 67108864 bytes	coherence.serializer.limit.maxContainerBytes
Array, collection, and object elements	262144	coherence.serializer.limit.maxElements
Map entries	131072	coherence.serializer.limit.maxMapEntries

Defaults and explicit policy

The finite hardened defaults above apply starting with Coherence 26.1.0.0 when coherence.security.mode is unset. An explicitly selected compatibility profile permits default-unbounded behavior and warns when applicable shadow thresholds are exceeded. Hardened mode requires finite limits and rejects unlimited. A finite limit explicitly configured in compatibility mode still constrains its consumer.

Specify bytes using a supported size suffix, such as 64m, or a count of bytes. Element and map-entry values are counts. Choose limits from measured legitimate payloads. A per-container limit does not bound the total object graph, concurrent requests, or member heap.

A service-local named serializer override

Place the following serializer element in the cache service scheme that owns the decode path. The id refers to an operational serializer template. Each service receives its configured serializer policy; changing one service should not mutate the template used by other services.

<serializer id="pof">
  <limits>
    <max-container-bytes system-property="billing.serialization.maxBytes">64m</max-container-bytes>
    <max-elements system-property="billing.serialization.maxElements">262144</max-elements>
    <max-map-entries system-property="billing.serialization.maxMapEntries">131072</max-map-entries>
  </limits>
</serializer>

A limit element with system-property uses that property's value when set and its element body as the fallback. Missing values inherit. Global property-backed defaults form the base; named/global serializer configuration, service-local overrides, and explicitly configured serializer instances refine it. The existing text reference <serializer>pof</serializer> remains valid when no local override is required.

Operational serializer declarations support <limits> after their class/instance configuration. Cache-config inline instances use <serializer><instance>...</instance><limits>...</limits></serializer>. Preserve the instance syntax used by the relevant schema. Custom serializers must expose and consume the appropriate policy; configuration alone does not establish that a custom parser honors every bound.

Verification and rollout

1. Test the largest legitimate array, collection, map, and sparse structure on the actual serializer and service.

2. In an isolated deployment, submit a payload above each bound. Verify controlled rejection before excessive allocation or partial installation.

3. Repeat with property and service-local overrides to establish the effective configuration.

4. Coordinate producer/consumer limits, including replay and recovery readers. Increase a legitimate limit to a measured finite value rather than disabling policy.

Transaction Bridges and Persistence Metadata

Account for typed metadata, bridge-local type restrictions, and legacy payload behavior when transaction and persistence data cross an upgrade boundary.

Coherence distinguishes narrow product-owned Java serialization bridges from application data carried by the configured Coherence serializer. There is no OIS-specific escape-hatch property. When a supported application class is missing, update generated security-config.xml or coherence.serialization.allowed after security review. When a narrow bridge rejects a legitimate product shape, the remedy is a reviewed product allowlist or bridge-filter update, not a customer property that disables the bridge check.

Transaction result and XID bridges

The retained Java serialization byte fields in transaction Results, JCache entry-processor exceptions, and POF XidWrapper compose a field-specific expected-type filter with the central serialization hardening filter and any JVM filter before the nested object stream materializes a class. The Results ExternalizableLite exception field supports only exact classes in the runtime-exception bridge policy throughout the complete root, cause, and suppressed graph. Before emitting the exception tag, its ordinary writer uses a cycle-safe identity walk to validate every throwable, then emits null or a fixed Java-serialization tag. Its reader validates that tag before dispatch and filters the nested stream, including the narrow JDK list/array infrastructure required by ordinary suppressed exceptions. Every array admitted by the transaction and JCache exception bridges has a maximum length of 16,384. The reader rejects a larger exact Object[], stack-trace array, allowed-exception array, or primitive array before allocation. The Results writer applies the same maximum to each throwable's suppressed-exception and stack-trace arrays before emitting the exception tag. Applications with larger exception state must trim it before transmission. Unsupported custom exception classes anywhere in the graph, including ExternalizableLite or Externalizable runtime exceptions, fail on write before serialization callbacks instead of producing bytes the reader cannot consume. These explicit bridge checks apply in both security profiles, including when the profile property is unset. Applications that previously propagated a custom runtime exception through a transaction result must translate it to a supported JDK or Coherence transaction exception before transmission. An application serialization allowlist does not widen this exact product bridge.

The shipped transaction POF configuration serializes Results through its direct PortableObject implementation and retains POF fields 1 through 5. Configuring ExternalizableLitePofSerializer for Results is unsupported: that adapter exposes the exception property only through PofInputStream.readObject, which cannot enforce the exact bridge policy before POF type construction and callbacks. Results rejects that adapter on read and write before any property dispatch. Use the shipped direct PortableObject registration; there is no property that bypasses this check.

The XidWrapper ExternalizableLite/tagged DataInput field retains a broader generic serialization boundary than the tightened POF bridge. Its compatible format can carry ExternalizableXid or a custom serializable Xid. Compatibility does not automatically attach the central filter to an otherwise unfiltered BufferInput; hardened policy attaches it before tagged class dispatch. This path does not provide the exact XID-only field policy of the POF bridge. Keep the general application allowlist and any configured input filter appropriately constrained.

New POF XidWrapper writers canonicalize the retained field to ExternalizableXid, and tightened POF readers reject arbitrary legacy custom serializable Xid implementations. Before a mixed-version XA upgrade can deliver such legacy POF bytes to a new reader, drain or complete in-flight and prepared XA work that depends on custom serializable XID implementations. Apply this drain guidance to the tightened POF bridge; the tagged DataInput field retains the distinct boundary described above.

Transaction bindings during mixed-version operation

New transaction Message writers emit both the validated legacy POF binding map at field 2 and typed binding fields 4 through 9. New readers use only the typed fields and do not materialize field 2. Consequently, a legacy-only sender that supplies only POF field 2 is accepted by a new receiver with an empty binding set. Coordinate binding-dependent transactional traffic during rolling upgrade; do not assume a legacy-only binding map will be interpreted by a new reader. The tagged DataInput binding path uses the typed scalar codec directly.

Results application maps

The three Results maps carry application data: cache keys and values, query results, entry-processor results, aggregator results, and application natural keys used by session mutation metadata. They retain those application semantics during mixed-version operation. Configure their serializer and application type policy rather than expecting the narrow product-exception bridge to restrict ordinary cache data.

For tagged DataInput, nested Java serialization values retain the serialization hardening object-stream filter in both security profiles. Hardened security mode also attaches the central filter before general Coherence tagged class dispatch; compatibility mode preserves the prior generic application materialization behavior unless the input already has an operator/JVM filter. POF map values remain governed by the configured POF context and serializer policy. These maps are application serializer data, not narrow OIS bridge fields.

Memcached-origin stored values

Memcached binary pass-through writes and legacy DECO_MEMCACHED values are validated before storage or ordinary Coherence materialization. Dangerous terminal formats, including Java serialization, XML serialization, ExternalizableLite, unknown, and XML-bean shapes, reject before dispatch, including through supported binary, identity, or optional wrappers. FMT_EXT interop remains available only when materialization uses a proven non-default configured serializer. This stored-value guard is a hard boundary in both security profiles; ordinary Memcached scalar/binary behavior and supported POF interop remain available.

coherence.mode remains the runtime/license setting and does not select any of these object-stream bridge hardening behaviors. Use coherence.security.mode=compatibility or hardened only for the central profile differences described above.

Persistence Metadata and Application Data

Product metadata has a narrower type universe than application cache data. Partition counts and persistence versions use fixed parsing; quorum compatibility data retains a guarded reader, and snapshot/store names are validated before use. Arbitrary application keys, values, and result maps continue to use the selected application serializer and its configured class/size policy.

Do not replace the shipped direct PortableObject registration for transaction results with ExternalizableLitePofSerializer. That adapter is unsupported for the tightened result field. Do not assume an application allowlist widens a bridge-local throwable/XID type universe or raises its fixed array bound.

Before rollout, drain or explicitly test XA work using custom serializable XIDs and transaction bindings produced by older writers. Retain a recoverable copy of old persisted data. A rollback of security profile does not undo mode-independent bridge changes.

Protecting Memcached Values and Native Indexes

Treat optional Memcached listeners and HNSW index files as separate trust and resource boundaries.

Memcached-origin stored values

Memcached binary pass-through and legacy DECO_MEMCACHED values are validated before storage or ordinary Coherence materialization. Dangerous terminal Java, XML, ExternalizableLite, unknown, and XML-bean shapes reject, including through supported wrappers. Scalar/binary and supported POF interoperability remain available; FMT_EXT requires a proven non-default configured serializer.

These checks apply in both security profiles. They do not add authentication or encryption to a Memcached listener. Inventory that listener separately, restrict its network reachability, and expose it only where the protocol security capabilities meet the deployment's requirements. Extend identity configuration does not secure another protocol automatically.

HNSW input and file validation

The Java/JNI boundary validates dimensions, vector lengths, query counts, graph parameters, and signed-to-unsigned conversions before native work. Native index loading checks sizes, allocation arithmetic, link metadata, and neighbor identities. A failed load must not be treated as a usable newly loaded index.

Use the native library distributed with the corresponding patched Java module on every target platform. Replacing a Java JAR does not replace an older separately installed native binary. Keep index files in a protected directory and test load/rebuild procedures with the supported binary.

Invalid dimensions, oversized counts, and corrupt files should produce controlled errors. Preserve the original index before retrying or rebuilding. These checks do not cryptographically authenticate index files and do not establish bounds fixes in separate C++ or .NET clients.