39 Managing Thread Execution

You can control the execution behavior of Coherence service threads using task timeouts and the PriorityTask API for custom execution processing.

Security Prerequisites

An asynchronous continuation, CDI event callback or application executor can change the execution thread. Establish the intended authenticated subject when initiating protected work from that context. Do not rely on a thread-local inherited by accident or assume that a virtual thread changes authorization requirements.

For the complete configuration and policy, see Using Authenticated Subjects with Modern Java, Authorizing Access to Server-Side Operations in Securing Oracle Coherence.

Service Worker Models in 26.1.0.0.0

Choose platform or virtual workers per service, then configure the corresponding bounds or admission controls. The following sections cover pool selection, query scheduling and Extend pipelines alongside the existing priority-task, timeout and guardian guidance. Platform workers remain the default.

This chapter includes the following sections:

• Overview of Priority Tasks
  Coherence priority tasks provide applications that have critical response time requirements better control of the execution of processes within Coherence.
• Setting Priority Task Timeouts
  You can configure various execution timeout settings.
• Creating Priority Task Execution Objects
  The PriorityTask interface enables you to control the ordering in which a service schedules tasks for execution using a thread pool and hold the task execution time to a specified limit.
• Choosing Platform or Virtual Service Workers
  Select the worker-pool model per service according to blocking behavior, contention, and the work performed by each task.
• Configuring Virtual Service Workers and Admission
  Enable virtual workers on a service and use an optional aggregate task limit to protect memory or downstream capacity.
• Sizing Platform Service Worker Pools
  Use explicit worker bounds when required and let workload-aware resizing find a useful operating point within them.
• Preserving Ordering and Identity with Virtual Workers
  Await dependent operations and establish authenticated context explicitly at asynchronous boundaries.
• Understanding Partitioned Query Execution
  Distinguish query scheduling and partition parallelism from result consistency and client-side streaming.
• Configuring Extend Proxy Pipelines
  Process independent TCP connections through separate decode pipelines while retaining connection order and a shared request worker pool.
• Observing Service Execution and Proxy Pipelines
  Interpret pool and pipeline metrics together with request latency, workload shape, and downstream capacity.
• Migrating Service Execution Configuration
  Change pool selection and incompatible settings together, then qualify application ordering, identity, and resource behavior.

Overview of Priority Tasks

Coherence priority tasks provide applications that have critical response time requirements better control of the execution of processes within Coherence.Execution and request timeouts can be configured to limit wait time for long running threads. In addition, a custom task API allows applications to control queue processing. Use these features with extreme caution because they can dramatically affect performance and throughput of the data grid.

Setting Priority Task Timeouts

You can configure various execution timeout settings.Care should be taken when configuring Coherence task execution timeouts especially for Coherence applications that pre-date this feature and thus do not handle timeout exceptions. For example, if a write-through in a CacheStore is blocked and exceeds the configured timeout value, the Coherence task manager attempts to interrupt the execution of the thread and an exception is thrown. In a similar fashion, queries or aggregations that exceed configured timeouts are interrupted and an exception is thrown. Applications that use this feature should ensure that they handle these exceptions correctly to ensure system integrity. Since this configuration is performed on a service by service basis, changing these settings on existing caches/services not designed with this feature in mind should be done with great care.

This section includes the following topics:

• Configuring Execution Timeouts
  
• Execution Timeout Command Line Options
  

Configuring Execution Timeouts

The <request-timeout>, <task-timeout>, and the <task-hung-threshold> elements are used to configure execution timeouts for a service's worker threads. These timeout settings are configured for a service in a cache configuration file and can also be set using command line parameters. See also Using the Service Guardian.

Table 39-1 describes the execution timeout elements.

Table 39-1 Execution Timeout Elements

Element Name	Description
<request-timeout>	Specifies the default timeout value for requests that can time out (for example, implement the PriorityTask interface), but do not explicitly specify the request timeout value. The request time is measured on the client side as the time elapsed from the moment a request is sent for execution to the corresponding server node(s) and includes the following: The time it takes to deliver the request to a running node (server). The interval between the time the task is received and placed into a service queue until the execution starts. The task execution time. The time it takes to deliver a result back to the client. If the value does not contain a unit, a unit of milliseconds is assumed. Legal values are positive integers or zero (indicating no default timeout). Default value is an infinite timeout (0s) for clustered client requests and 30 seconds (30s) for non-clustered client requests.
<task-timeout>	Specifies the default timeout value for tasks that can be timed-out (for example, entry processors that implement the PriorityTask interface), but do not explicitly specify the task execution timeout value. The task execution time is measured on the server side and does not include the time spent waiting in a service backlog queue before being started. This attribute is applied only if the thread pool is used (the <thread-count-min> and <thread-count-max> values are positive). If zero is specified, then the default service-guardian <timeout-milliseconds> value is used.
<task-hung-threshold>	Specifies the amount of time in milliseconds that a task can run before it is considered "hung". Note: a posted task that has not yet started is never considered as hung. This attribute is applied only if the thread pool is used (the <thread-count-min> and <thread-count-max> values are positive).

The following distributed cache example explicitly configures the service dynamic thread pool, a task time out of 5000 milliseconds, and a task hung threshold of 10000 milliseconds:

<caching-schemes>
    <distributed-scheme>
      <scheme-name>example-distributed</scheme-name>
      <service-name>DistributedCache</service-name>
      <thread-count-max>20</thread-count-max>
      <thread-count-min>7</thread-count-min>
      <task-hung-threshold>10000</task-hung-threshold>
      <task-timeout>5000</task-timeout>
    </distributed-scheme>
</caching-schemes>

Setting the client request timeout to 15 milliseconds

<distributed-scheme>
      <scheme-name>example-distributed</scheme-name>
      <service-name>DistributedCache</service-name>
      <request-timeout>15000ms</request-timeout>
    </distributed-scheme>

Note:

The request-timeout should always be longer than the thread-hung-threshold or the task-timeout.

Execution Timeout Command Line Options

Use the command line options to set the service type default (such as distributed cache, invocation, proxy, and so on) for the node. Table 39-2 describes the options.

Table 39-2 Command Line Options for Setting Execution Timeout

Option	Description
coherence.replicated.request.timeout	The default client request timeout for the Replicated cache service
coherence.optimistic.request.timeout	The default client request timeout for the Optimistic cache service
coherence.distributed.request.timeout	The default client request timeout for distributed cache services
coherence.distributed.task.timeout	The default server execution timeout for distributed cache services
coherence.distributed.task.hung	The default time before a thread is reported as hung by distributed cache services
coherence.invocation.request.timeout	The default client request timeout for invocation services
coherence.invocation.task.hung	The default time before a thread is reported as hung by invocation services
coherence.invocation.task.timeout	The default server execution timeout invocation services
coherence.proxy.request.timeout	The default client request timeout for proxy services
coherence.proxy.task.timeout	The default server execution timeout proxy services
coherence.proxy.task.hung	The default time before a thread is reported as hung by proxy services

Creating Priority Task Execution Objects

The PriorityTask interface enables you to control the ordering in which a service schedules tasks for execution using a thread pool and hold the task execution time to a specified limit.Instances of PriorityTask typically also implement either the Invocable or Runnable interface. Priority Task Execution is only relevant when a task back log exists.

The API defines the following ways to schedule tasks for execution:

• SCHEDULE_STANDARD—a task is scheduled for execution in a natural (based on the request arrival time) order

• SCHEDULE_FIRST—a task is scheduled in front of any equal or lower scheduling priority tasks and executed as soon as any of worker threads become available

• SCHEDULE_IMMEDIATE—a task is immediately executed by any idle worker thread; if all of them are active, a new thread is created to execute this task

This section includes the following topics:

• APIs for Creating Priority Task Objects
  
• Errors Thrown by Task Timeouts
  

APIs for Creating Priority Task Objects

Coherence provides the following classes to help create priority task objects:

• PriorityProcessor can be extended to create a custom entry processor.

• PriorityFilter can be extended to create a custom priority filter.

• PriorityAggregator can be extended to create a custom aggregation.

• PriorityTask can be extended to create an priority invocation class.

After extending each of these classes, the developer must implement several methods. The return values for getRequestTimeoutMillis, getExecutionTimeoutMillis, and getSchedulingPriority should be stored on a class-by-class basis in your application configuration parameters. Table 39-3 describes these methods.

Table 39-3 Methods to Support Task Timeout

Method	Description
public long getRequestTimeoutMillis()	Obtains the maximum amount of time a calling thread is can wait for a result of the request execution. The request time is measured on the client side as the time elapsed from the moment a request is sent for execution to the corresponding server node(s) and includes: the time it takes to deliver the request to the executing node(s); the interval between the time the task is received and placed into a service queue until the execution starts; the task execution time; the time it takes to deliver a result back to the client. The value of TIMEOUT_DEFAULT indicates a default timeout value configured for the corresponding service; the value of TIMEOUT_NONE indicates that the client thread is can wait indefinitely until the task execution completes or is canceled by the service due to a task execution timeout specified by the getExecutionTimeoutMillis() value.
public long getExecutionTimeoutMillis()	Obtains the maximum amount of time this task is allowed to run before the corresponding service attempts to stop it. The value of TIMEOUT_DEFAULT indicates a default timeout value configured for the corresponding service; the value of TIMEOUT_NONE indicates that this task can execute indefinitely. If, by the time the specified amount of time passed, the task has not finished, the service attempts to stop the execution by using the Thread.interrupt() method. In the case that interrupting the thread does not result in the task's termination, the runCanceled method is called.
public int getSchedulingPriority()	Obtains this task's scheduling priority. Valid values are SCHEDULE_STANDARD, SCHEDULE_FIRST, SCHEDULE_IMMEDIATE
public void runCanceled(boolean fAbandoned)	This method is called if and only if all attempts to interrupt this task were unsuccessful in stopping the execution or if the execution was canceled before it had a chance to run at all. Since this method is usually called on a service thread, implementors must exercise extreme caution since any delay introduced by the implementation causes a delay of the corresponding service.

Errors Thrown by Task Timeouts

When a task timeout occurs the node gets a RequestTimeoutException. For example:

com.tangosol.net.RequestTimeoutException: Request timed out after 4015 millis
        at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.checkRequestTimeout(Service.CDB:8)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.poll(Service.CDB:52)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.poll(Service.CDB:18)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.InvocationService.query(InvocationService.CDB:17)
        at com.tangosol.coherence.component.util.safeService.SafeInvocationService.query(SafeInvocationService.CDB:1)

Choosing Platform or Virtual Service Workers

Select the worker-pool model per service according to blocking behavior, contention, and the work performed by each task.

The default platform daemon pool uses reusable platform workers and dynamic sizing. Select virtual to run service work using virtual threads and adaptive admission. Both are available on the Java 21 minimum runtime. For virtual-thread deployments, Java 25 avoids the monitor-pinning limitation present in Java 21; native blocking and application dependencies still need qualification.

Table 39-4 Choosing a service worker model

Service workload	Starting choice
Short in-memory operations and mostly CPU-bound tasks	Retain platform workers, especially for small members and latency-sensitive operations.
Blocking persistence, database or remote-service work	Evaluate virtual workers with representative downstream limits.
Many mutations contending on the same keys	Evaluate virtual keyed admission, while preserving application ordering requirements.
Proxy traffic	Measure the actual request path. Short nonblocking reads and blocking mutations can favor different choices.

Virtual threads reduce the carrier cost of supported blocking; they do not add CPU capacity, remove entry serialization, or make synchronous downstream systems unlimited. Qualify both member-local capacity and clusterwide throughput. Configure the choice on each service; there is no global switch that silently changes all service pools.

Configuring Virtual Service Workers and Admission

Enable virtual workers on a service and use an optional aggregate task limit to protect memory or downstream capacity.

Place this distributed-scheme fragment under cache-config/caching-schemes and map the application’s caches to its scheme name:

<distributed-scheme>
  <scheme-name>orders-virtual</scheme-name>
  <service-name>OrdersService</service-name>
  <daemon-pool>virtual</daemon-pool>
  <task-limit system-property="orders.task.limit">1000</task-limit>
  <backing-map-scheme><local-scheme/></backing-map-scheme>
  <autostart>true</autostart>
</distributed-scheme>

The application-defined orders.task.limit property can override this cap before startup. Omit task-limit, or set it to zero, when no additional aggregate cap is needed. Virtual admission adapts independently for ordinary direct tasks, read-only tasks, and keyed mailbox drainers. An explicit positive cap composes with those adaptive gates and limits ordinary running task bodies; it does not bound queued tasks, waiting virtual threads, or total request memory.

Do not combine virtual workers with thread-count, thread-count-min, thread-count-max or worker-priority. Startup rejects these combinations. Conversely, task-limit is not a platform-worker setting. lease-granularity remains applicable to both models.

A service init-param named daemon-pool can supply the operational default; an explicit cache-scheme selection takes precedence. Retain per-service configuration in deployment control so pool selection and its allowed settings change together.

Sizing Platform Service Worker Pools

Use explicit worker bounds when required and let workload-aware resizing find a useful operating point within them.

A fixed thread-count chooses a fixed pool; zero processes requests on the service thread. For dynamic sizing, use thread-count-min and thread-count-max. Equal minimum and maximum values pin the size. Mixing the fixed-count form with dynamic bounds emits a configuration warning. Avoid this combination in new configurations.

<distributed-scheme>
  <scheme-name>orders-platform</scheme-name>
  <service-name>OrdersService</service-name>
  <daemon-pool>platform</daemon-pool>
  <thread-count-max>32</thread-count-max>
  <thread-count-min>2</thread-count-min>
  <backing-map-scheme><local-scheme/></backing-map-scheme>
  <autostart>true</autostart>
</distributed-scheme>

This is a cache-scheme fragment. The values illustrate bounds, not a universal recommendation. If the maximum is omitted, a process-wide coordinator derives a platform-worker budget using available processors, heap, estimated thread-stack cost, process or container memory, and Linux VMA headroom where available. Automatic pools share that budget by role. Explicit finite maxima remain authoritative.

Resizing observes throughput, utilization, task occupancy, association width, and queue pressure. Above the effective CPU allocation, it requires evidence that added blocking concurrency is useful, establishes a stable baseline, and retains or rejects bounded probes using measured throughput benefit. A busy pool or a deep queue alone is not proof that more workers help. Diagnostic level 5 logs expose decisions and reasons.

Keep operational limits explicit when protecting a constrained database or external service. Measure latency, backlog, CPU, native memory, and downstream saturation together. Algorithm constants and main-branch tuning changes are not stable application configuration contracts.

Preserving Ordering and Identity with Virtual Workers

Await dependent operations and establish authenticated context explicitly at asynchronous boundaries.

Virtual workers route keyed mutations through per-key mailboxes. Truly read-only get and getAll work can use a direct route. A read submitted without waiting for an earlier write acknowledgment may therefore observe the earlier value. Await or chain the write before a dependent read:

NamedMap<String, String> orders = session.getMap("orders");
String value = orders.async().put("order-1", "confirmed")
        .thenCompose(previous -> orders.async().get("order-1"))
        .join();

Read-through and sliding-expiry reads can commit replicated state and remain write-shaped work, including write-quorum checks. A custom backing map used for direct reads must support concurrent reads and same-key read/write overlap. Do not rely on incidental worker arrival order for application correctness.

A virtual thread, completion callback, or application executor is also an identity boundary. Use the supported authenticated-subject APIs when a callback initiates another protected operation; do not assume that thread creation copies the required identity. See Using Authenticated Subjects with Modern Java. Existing timeout, guardian and re-entrant-call constraints continue to apply.

Understanding Partitioned Query Execution

Distinguish query scheduling and partition parallelism from result consistency and client-side streaming.

Partitioned query and aggregation execution bounds work dispatch rather than creating one unbounded task per selected partition. Streaming-aggregator characteristics guide execution: BY_MEMBER groups member work, BY_PARTITION requests partition-shaped work, and unqualified parallel aggregation permits the service to choose its supported parallel plan.

These characteristics describe execution and combination. They do not promise one task per entry, a stronger snapshot across partitions, or an entry-by-entry result stream to the client. A streaming aggregator consumes server-side entries and partial results; a large final collection can still require a large response.

Index the filter and aggregation attributes where appropriate, measure selected partition count and result size, and choose Continuous Aggregation for stable definitions queried frequently. Retain ordinary aggregation as the correctness and performance baseline. Timeout settings do not make arbitrary user code safely interruptible or cancel already committed mutations.

Configuring Extend Proxy Pipelines

Process independent TCP connections through separate decode pipelines while retaining connection order and a shared request worker pool.

A TCP Extend proxy assigns each accepted connection to one selector/decode pipeline for its lifetime. Frames, including the handshake, retain their wire order on that connection. Independent connections can decode concurrently, while ordinary requests still use the proxy service’s shared worker pool.

<proxy-scheme>
  <service-name>ExtendProxy</service-name>
  <acceptor-config>
    <tcp-acceptor>
      <local-address><address>127.0.0.1</address><port>9099</port></local-address>
      <pipeline-count>0</pipeline-count>
    </tcp-acceptor>
  </acceptor-config>
  <autostart>true</autostart>
</proxy-scheme>

This cache-scheme fragment binds the example proxy to loopback. Configure the application’s intended interface, TLS, and authentication before accepting remote clients. See Securing Extend Client Connections.

Table 39-5 TCP proxy pipeline count

Value	Behavior
Omitted or 0	Automatic mode: starts with a small processor-derived count and can grow for new connection load or decode pressure.
1	Single pipeline; useful for a controlled comparison with the previous topology.
2 through 256	Fixed count, retained until reconfiguration or restart.

Automatic growth does not move existing connections or remove pipelines. A newly added pipeline serves new or reconnecting connections. One busy connection remains limited to one pipeline. An explicitly configured proxy thread-count of zero keeps the original single-threaded request path and one automatic pipeline; request concurrency is not enabled by this setting.

The server change preserves the existing wire protocol. Updated Java Extend initiators also use 8 KiB framing input/output buffers; these internal buffers are distinct from socket buffers and maximum message size. Older compatible clients can benefit from concurrent independent connections without a protocol change.

Observing Service Execution and Proxy Pipelines

Interpret pool and pipeline metrics together with request latency, workload shape, and downstream capacity.

Use ServiceMBean.DaemonPoolType to distinguish platform and virtual models. ThreadCount reports live workers of the selected type; virtual counts do not describe a reusable idle platform pool. For virtual pools, ThreadCountMin is zero and ThreadCountMax is Integer.MAX_VALUE; admission is controlled separately.

Observe TaskBacklog, active tasks, TaskLimit, the direct/read-only/mailbox admission gauges and PoolSaturation with throughput and latency. Platform-only or virtual-only attributes can use sentinel values when inapplicable; do not graph those values as meaningful capacity. Secure management, metrics, and diagnostic endpoints before exposing them.

The TCP proxy’s ConnectionManager MBean supplies PipelineCountConfigured, PipelineCount and PipelineCountLimit. The configured value zero means automatic mode; active count can grow while the limit describes its safety ceiling. Inapplicable connection managers report -1. Existing message, byte and connection statistics aggregate across pipelines.

At diagnostic level 5, compare pipeline assignment/growth and worker resize/admission decisions with live connections and task backlog. Flat throughput with rising latency can indicate storage or downstream saturation rather than insufficient decoding. Change one execution setting at a time and retain a representative baseline.

Migrating Service Execution Configuration

Change pool selection and incompatible settings together, then qualify application ordering, identity, and resource behavior.

Retain platform workers initially when upgrading. For each candidate virtual service, remove platform thread-count and worker-priority elements, add daemon-pool set to virtual, and add a positive task cap only if the deployment needs an aggregate ceiling. Test blocking operations, hot keys, asynchronous write/read dependencies, authenticated callbacks, and shutdown under load.

Old virtual-threads-enabled, coherence.virtualthreads.enabled, service virtual-thread flags and the interim coherence.daemonpool.virtual property do not select this service model. Replace them with the supported per-service element or init-param. Do not keep obsolete switches as an apparent fallback.

To return a service to platform workers, select platform (or omit the element), remove task-limit and reinstate appropriate platform bounds together. For proxy comparisons, set pipeline-count to one and hold client connection count and worker settings constant. Complete the supported release upgrade procedure before using new configuration with older members.