Changes in 23.3.30

The following changes were made in Oracle NoSQL Database Release 23.3.30 Enterprise Edition.

Topics

New Features

Implemented three new SQL aggregate functions: count(distinct), array_collect(), and array_collect(distinct).
count(distinct expr) returns the number of distinct values that are returned by its input expression when computed on the rows belonging to the same group.

array_collect(expr) computes its input expression on each row of a group and collects all the resulting values into an array, returning this constructed array as its result.

array_collect(distinct expr) computes its input expression on each row of a group and collects all the resulting distinct values into an array, returning this constructed array as its result.
For example, the following query groups sales transactions by account number and returns, for each account number, an array containing the distinct categories of products purchased by the account:
```
select s.xact.acctno, array_collect(distinct s.xact.prodcat) as categories
from sales s
group by s.xact.acctno
```
[KVSTORE-1861]
Added JSON Collection as a new table type. This type allows a user to specify only key fields for a table, supplying other top-level fields as non-schema-defined JSON types. In addition MR Counter types can be defined as top-level paths in a JSON Collection for use in a multi-region configuration. Ad-hoc fields created in a JSON collection must be valid JSON types, which excludes records, binary, timestamp, uuid, and enum types. Maps, arrays, numeric values, string values, so forth, are valid.
An advantage of JSON Collection tables over a normal schema table is simplification. If, for example, an application is using unstructured (for example: JSON) data it is not necessary to create an intermediate field of type JSON to contain that data. All non-key fields are automatically added as JSON. Because of this semantic it is not necessary (or possible) to evolve the schema of a JSON Collection table. There are no fields to add or remove. It is possible to modify the TTL. JSON Collection tables are also very well suited to multi-region operation as only the primary keys must match across regions.

At this time JSON collection tables do not support a parent/child relationship. That is, a JSON collection must be a top-level parent with no child tables.
A JSON Collection table with a single primary key can be created this way:
```
create table jsonCollection(id integer, primary key(id)) as json collection
```
Arbitrary fields can then be added. This example uses the Direct Java Driver API:
```
    Table t = tableAPI.getTable("jsonCollection");
    Row row = t.createRow();
    row.put("id", 1);
    /* put some JSON -- these must be valid JSON types */
    row.put("name", "some_name"); // note that this isn't in the schema
    row.put("age", 15); // note that this isn't in the schema
    tableAPI.put(row, ...);
    PrimaryKey pkey = t.createPrimaryKey();
    pkey.put("id", 1);
    row = tableAPI.get(pkey, ...);
    /*
     * the row will look like this if toString is used:
     * {"id": 1, "name": "some_name", "age": 15}
     */
```
[KVSTORE-1717]
Added support for running multiple multi-region agents to replicate data from a remote region. When multiple agents are deployed, they divide up the data from the store so that each agent only replicates a portion of the data, to support increased throughput.
[KVSTORE-984]
Previous releases of the Full Text Search (FTS) capability were tested with Elasticsearch (ES) version 2. Starting with this release, FTS requires ES version 7 or later, and was tested with ES version 8.
In addition, previous releases of FTS required the use of the Shield ES plugin if users wanted secure communication between the store and the Elasticsearch cluster. Starting with this release, the Shield plugin is no longer supported. Users should configure ES with PKI security to enable secure communications.

Please note that this update requires changes to existing deployments to update the ES clusters to the newly supported versions. Be sure to review the FTS documentation for guidance on migrating the ES data.

[KVSTORE-1917]

Bug and Performance Fixes

Fixed a bug where, in some cases, the Streams API NoSQLSubscription.getOptimizedPosition method could return a stream position earlier than the stream position specified in the method parameter. The method now always returns a stream position equal to or later than the one provided.
[KVSTORE-2037]
Fixed a bug with queries using geojson indexes. Previously, the index would be used incorrectly if the index definition specified one or more non-geojson fields before the geojson field and the query did not contain equality predicates on the non-geojson fields. For example, the following index and query would return wrong results:
```
create index idx_kind_ptn on points(info.kind as string,
info.point as point)
```
```
select id, p.info.point
from points p
where geo_near(p.info.point,
               { "type" : "point", "coordinates" : [ 24.0175, 35.5156 ] },
               5000)
```
[KVSTORE-2134]
Fixed a bug in converting typed maps/arrays to JSON maps/arrays. Such a conversion is needed when the typed map/array is going to be inserted in a JSON map/array. In queries using the seq_transform function, the conversion would not be done in some cases, resulting in a runtime type error. For example, the following query (where 'info' is a column of type json) would raise an error, because, by itself, the expression {"num_phones": size($.phones)} has type map(integer), which must be converted to map(json), because the outer map constructor has type map(json).
```
select {
         "id" : t.id,
         "phones" : [seq_transform(t.info.addresses[],
                                   {"num_phones": size($.phones)})]
       }
from foo t
```
[KVSTORE-2139]
Fixed a bug where, in rare cases, the Streams API NoSQLSubscription.getCurrentPosition method could return a stream position that failed to include a few elements that had not yet been delivered to the subscriber via calls to NoSQLSubscriber.onNext. As a result, checkpointing the stream position returned by NoSQLSubscription.getCurrentPosition could have failed to stream the missing events after resuming from the checkpoint.
[KVSTORE-1998]
Fixed a bug that could have caused the Streams API to miss a few writes to the subscribed tables when performing elasticity operations on the associated store.
[KVSTORE-2062]
Fixed a bug where the XRegion Service agent might fail to start if the stores in local or remote regions were unavailable at start up time. With the fix, the agent retries until the stores become available.
[KVSTORE-2057]
Fixed a bug that, in rare cases, could cause a small number of operations to be missed in Streams API when the store was undergoing elastic operations involving partition migration.
[KVSTORE-1838]

Fixed an issue where SSL handshake task exceptions were silently ignored on the client causing the problem only to be observed as a connection timeout. The following is an example exception:

java.lang.IllegalAccessError:
org.bouncycastle.jcajce.provider.ProvSunTLSKDF$TLSExtendedMasterSecretGenerator$1
(in unnamed module @0xd504137) cannot access class
sun.security.internal.spec.TlsMasterSecretParameterSpec (in module
java.base) because module java.base does not export
sun.security.internal.spec to unnamed module @0xd504137
    at org.bouncycastle.jcajce.provider.ProvSunTLSKDF$TLSExtendedMasterSecretGenerator$1.run(Unknown Source)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
    at org.bouncycastle.jcajce.provider.ProvSunTLSKDF$TLSExtendedMasterSecretGenerator.engineInit(Unknown Source)
    at java.base/javax.crypto.KeyGenerator.init(KeyGenerator.java:476)
    at java.base/javax.crypto.KeyGenerator.init(KeyGenerator.java:452)
    at java.base/sun.security.ssl.SSLMasterKeyDerivation$LegacyMasterKeyDerivation.deriveKey(SSLMasterKeyDerivation.java:149)
    at java.base/sun.security.ssl.KAKeyDerivation.t12DeriveKey(KAKeyDerivation.java:89)
    at java.base/sun.security.ssl.KAKeyDerivation.deriveKey(KAKeyDerivation.java:61)
    at java.base/sun.security.ssl.ECDHClientKeyExchange$ECDHEClientKeyExchangeProducer.produce(ECDHClientKeyExchange.java:426)
    at java.base/sun.security.ssl.ClientKeyExchange$ClientKeyExchangeProducer.produce(ClientKeyExchange.java:65)
    at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:440)
    at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:182)
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
    at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:201)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)

Previously, such exceptions were only logged. However, since client-side logging is usually disabled, the exception was not being seen. With this fix, the request will time out with a message that looks like the following:

oracle.kv.RequestTimeoutException: Request dispatcher: c2409576586975841402, dispatch timed out after 0 attempt(s) to dispatch the request to an RN. Target: rg1-rn1 (23.2.6) on [2023-06-08 19:38:47.346 UTC]. Timeout: 5000ms. Dispatch event trace: (DISPATCH_REQUEST_RESPONDED, 2023-06-08 19:38:46.623 UTC, rg1-rn1, PersistentDialogException|ConnectionUnknownException|CompletionException|IllegalAccessError org.bouncycastle...

[KVSTORE-1816]

Fixed a bug that affected queries using GROUP BY that also ordered by the grouping expressions in descending order. If such a query performed index-based grouping, the query results were ordered in ascending order, instead of the requested descending order. For example, the following query exhibited the bug if there is also an index on address.state:
```
select u.address.state, count(*)
from users
group by u.address.state
order by u.address.state desc
```
[KVSTORE-2078]
Fixed a bug in resuming queries at RNs. The bug may occur when the row on which the query has to resume from has been deleted. The bug causes duplicate query results to be returned.
[KVSTORE-2144]
Fixed a bug with the 'any' comparison operators. The operators could return a wrong result when the right-hand-side operand would return more than 2 items.
[KVSTORE-2150]
Fixed a compatibility issue in communications between store processes running releases before and after release 22.3. For example, suppose the server had been upgraded to a new version with an old client. In that case, the client might observe requests throwing DialogNoBackoffException with messages Heartbeat timeout, no read event during last ... or endpoint already shutdown. Communication between server processes with different versions could also be affected by this issue. This issue was more likely to appear under light load and did not occur if there was at least one request/response exchange within each 10 second time period.
[KVSTORE-1935]
Fixed a bug when:
1. A SET TLL USING TABLE DEFAULT was used in an UPDATE statement and
2. The target table did not have an explicitly specified default TTL
In this case, the UPDATE statement failed to update the expiration time of the row.
[KVSTORE-1981]
Fix a bug in the Streams API that could cause subscribing or unsubscribing a table to hang.
[KVSTORE-1860]

Fixed a race condition in the implementation of table iteration that could occasionally produce a NullPointerException like:

java.lang.NullPointerException: Cannot invoke
"java.util.concurrent.CompletableFuture.whenComplete(java.util.function.BiConsumer)"
because the return value of
"oracle.kv.impl.api.table.TableScan$BasicMultiGetIteratorWrapper.getMoreElementsAsync()"
is null
      at oracle.kv.impl.api.table.TableScan$BasicMultiGetIteratorWrapper.nextLocal(TableScan.java:950)
      at oracle.kv.impl.async.AsyncIterationHandleImpl.notifyOneNext(AsyncIterationHandleImpl.java:478)
      at oracle.kv.impl.async.AsyncIterationHandleImpl.notifyNext(AsyncIterationHandleImpl.java:373)
      at oracle.kv.impl.async.AsyncIterationHandleImpl.subscribe(AsyncIterationHandleImpl.java:156)
      at oracle.kv.impl.async.AsyncPublisherImpl.subscribe(AsyncPublisherImpl.java:127)

[KVSTORE-1828]

Fixed a problem where the 'plan network-restore' Admin CLI command could report that it had succeeded when it had actually failed.
[KVSTORE-1811]
Fixed a problem introduced in the 22.1 release that could cause key stats scans to be throttled incorrectly, causing the scans to take significantly longer to complete in some cases.
[KVSTORE-1929]
Fixed thread safety flaws in the plan execution code that could cause plans to hang in rare cases.
[KVSTORE-1920]
Fixed a problem where the Admin CLI verify prerequisite command produced an incorrect error message when the version of the Admin CLI was earlier than the version of one or more of the storage nodes. Note that the verify prerequisite command checks to see if the store supports upgrading to the version being used by the Admin CLI.
Previously, the error message talked about downgrading the node but mentioned the prerequisite version instead of the intended upgrade version. Now the message refers to the upgrade version and the wording has been improved.
For example, if an SNA in the store is running version 23.2.1 and the Admin CLI version is 22.1.1, the error message will be:
```
  Node cannot be changed to use version 22.1.1 because that would
  require a downgrade from newer version 23.2.1 it is currently running,
  and that downgrade is not supported.
```
[KVSTORE-1803]
Fixed a SQL query bug that occurred when a map-filter step expression was applied directly to a row variable. For example, the following query would raise an exception, because the keys() step was applied to the "b" row variable.
```
select *
from bar $b, $b.keys() $k, $b.$k $v
```
[KVSTORE-1968]
Reverted changes made in the 22.1 release for KVSTORE-1194 to execute plans found in the APPROVED state during an Admin restart. The previous change was incorrect and meant that plans created with the -noexecute flag could be executed without plan execution being requested explicitly.
[KVSTORE-1591]
Modified the key distribution statistics collector to wait to perform scans until user tables have been created. The keys stats collector stores data in system tables, both for its own bookkeeping and to record the resulting statistics. Table rows and key/value entries share the same keyspace, so these system table entries could potentially collide with key/value data. Once a user table is created, though, it is up to the user to handle any potential collisions between table rows and key/value entries, so it is safe for the collector to write to the system tables. As a result of this change, key distribution statistics will only become available after the first user table is created, and will not be available for system tables or the key/value keyspace before then.
[KVSTORE-2007]
Fixed a bug where the ping command using the Admin REST API with a secure store always returned UNREACHABLE for the admin status and returned a response with HTTP status code 308 even if admin was healthy.
[KVSTORE-1862]
Changed the behavior of TableAPI.putIfPresent() when a GENERATED ALWAYS primary key column value is present. If the value is present, generation of a new value is skipped and the putIfPresent is performed on the row with the specified primary key. This way existing rows can be updated on tables using GENERATED ALWAYS in columns that are part of the primary key. Previously, the Row.put method threw IllegalArgumentException in this case.
[KVSTORE-152]

Fixed a problem where enabling FINE logging in a Java direct driver application used with a secure store could cause the application to get NoClassDefFoundErrors such as:

Exception in thread "main" java.lang.NoClassDefFoundError: oracle/kv/impl/security/login/LoginSession$Id
	at oracle.kv.impl.security.login.SessionId.hashId(SessionId.java:273)
	at oracle.kv.impl.security.login.LoginToken.hashId(LoginToken.java:178)
	at oracle.kv.impl.security.login.LoginToken.toString(LoginToken.java:173)
	at java.base/java.lang.String.valueOf(String.java:4218)
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:173)
	at oracle.kv.impl.security.login.AsyncUserLogin$LogoutCall.describeParams(AsyncUserLogin.java:558)
	at oracle.kv.impl.security.login.AsyncUserLogin$AbstractCall.describeCall(AsyncUserLogin.java:582)
	at oracle.kv.impl.async.AsyncVersionedRemoteDialogInitiator.toString(AsyncVersionedRemoteDialogInitiator.java:236)
	at java.base/java.lang.String.valueOf(String.java:4218)
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:173)
	at oracle.kv.impl.async.AsyncVersionedRemoteInitiator.lambda$startDialog$8(AsyncVersionedRemoteInitiator.java:239)
	at oracle.kv.impl.async.FutureUtils.lambda$unwrapExceptionVoid$2(FutureUtils.java:381)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at oracle.kv.impl.async.FutureUtils.checkedComplete(FutureUtils.java:245)
	at oracle.kv.impl.async.AsyncVersionedRemoteDialogInitiator.onCanRead(AsyncVersionedRemoteDialogInitiator.java:179)
	at oracle.kv.impl.util.registry.AsyncRegistryUtils$WrappedDialogHandler.onCanRead(AsyncRegistryUtils.java:670)
	at oracle.kv.impl.async.dialog.DialogContextImpl$OnCanReadTask.doWork(DialogContextImpl.java:1088)
	at oracle.kv.impl.async.dialog.DialogContextImpl$OnCanReadTask.run(DialogContextImpl.java:1059)
	at oracle.kv.impl.async.dialog.DialogContextImpl.callOnCanRead(DialogContextImpl.java:964)
	at oracle.kv.impl.async.dialog.DialogContextImpl.onReadDialogFrame(DialogContextImpl.java:648)
	at oracle.kv.impl.async.dialog.AbstractDialogEndpointHandler.onReadDialogFrame(AbstractDialogEndpointHandler.java:1783)
	at oracle.kv.impl.async.dialog.AbstractDialogEndpointHandler.onMessageReady(AbstractDialogEndpointHandler.java:1528)
	at oracle.kv.impl.async.dialog.AbstractDialogEndpointHandler.onChannelInputRead(AbstractDialogEndpointHandler.java:1088)
	at oracle.kv.impl.async.dialog.AbstractDialogEndpointHandler.onChannelInputRead(AbstractDialogEndpointHandler.java:1046)
	at oracle.kv.impl.async.dialog.nio.NioEndpointHandler.read(NioEndpointHandler.java:246)
	at oracle.kv.impl.async.dialog.nio.NioEndpointHandler.onRead(NioEndpointHandler.java:198)
	at oracle.kv.impl.async.dialog.nio.NioChannelExecutor.doProcess(NioChannelExecutor.java:1596)
	at oracle.kv.impl.async.dialog.nio.NioChannelExecutor.selectAndProcess(NioChannelExecutor.java:1318)
	at oracle.kv.impl.async.dialog.nio.NioChannelExecutor.runOnce(NioChannelExecutor.java:845)
	at oracle.kv.impl.async.dialog.nio.NioChannelExecutor.lambda$run$9(NioChannelExecutor.java:801)
	at oracle.kv.impl.fault.AsyncEndpointGroupFaultHandler.lambda$static$0(AsyncEndpointGroupFaultHandler.java:21)
	at oracle.kv.impl.async.dialog.nio.NioChannelExecutor.run(NioChannelExecutor.java:798)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.ClassNotFoundException: oracle.kv.impl.security.login.LoginSession$Id
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	... 36 more

[KVSTORE-1937]

Modified the Admin CLI to print out error messages when renewing an expired password fails due to password complexity violations even if debug mode is disabled. The error messages were previously only printed out in debug mode, making it difficult to understand why the password renewal failed.
[KVSTORE-1902]
Fixed an issue where the SQL shell's show ddl command could fail for a table with a geo-json index such as GEOMETRY or POINT.
[KVSTORE-1978]
Fixed an issue where a Storage Node did not shut down if its listening socket failed. The problem produced Listener shutting down messages in the snaboot log and the SN process did not exit. For fault tolerance of SNs, we expect administrators to install some mechanism (for example, a cron job) to monitor and restart SNs when they fail. With the fix, that mechanism will now restart the SN when a listening socket fails.
[KVSTORE-2041]
Fixed a resource leak that was accompanied by messages like Endpoint handler encounters that ... start unknown dialog type appearing in logs. There were no immediate symptoms when the leak started, but when the resource was exhausted, network communications with the problematic node failed, and eventually the node failed with an OutOfMemoryError.
[KVSTORE-2033]
Fixed a resource leak that produced the error message Maximum permit count exceeded either in logs or in the exception message for an exception thrown for a request. The issue was usually the result of other issues in the network layer (for example, a slow network or DNS failures) and could go away when the root cause was addressed.
[KVSTORE-2044]

Fixed an issue that could cause an RN failure during a topology elasticity operation. The issue could be observed with RN being either shut down or restarted with the sequences of logging messages like the following:

2023-07-15 02:55:51.295 UTC INFO [rg5-rn2] Cannot localize topology seq#: 387 because it is < migration topology seq#: 392
...
2023-07-15 02:55:52.297 UTC SEVERE [rg5-rn2] JE: Replica unexpected exception com.sleepycat.je.EnvironmentFailureException: (JE 23.3.3) rg5-rn2(2):/scratch/tests/workspace/kv_elasticity/kv/kvstore/build/test/standalone/elasticity-stress/scratch/kvtest-standalone.ElasticityStress/sn8/rg5-rn2/env java.lang.IllegalStateException: update of local topology failed from commit trigger for topology seq#: 392 UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed. Problem seen replaying entry Commit/22 vlsn=1306 isReplicated="1"  com.sleepycat.je.EnvironmentFailureException: (JE 23.3.3) rg5-rn2(2):/scratch/tests/workspace/kv_elasticity/kv/kvstore/build/test/standalone/elasticity-stress/scratch/kvtest-standalone.ElasticityStress/sn8/rg5-rn2/env java.lang.IllegalStateException: update of local topology failed from commit trigger for topology seq#: 392 UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed. Problem seen replaying entry Commit/22 vlsn=1306 isReplicated="1"
	at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:403)
	at com.sleepycat.je.dbi.TriggerManager.runTriggers(TriggerManager.java:347)
	at com.sleepycat.je.dbi.TriggerManager.runCommitTriggers(TriggerManager.java:231)
	at com.sleepycat.je.txn.Txn.commit(Txn.java:809)
	at com.sleepycat.je.rep.txn.ReplayTxn.commit(ReplayTxn.java:299)
	at com.sleepycat.je.rep.impl.node.Replay.replayEntry(Replay.java:770)
	at com.sleepycat.je.rep.impl.node.Replica$ReplayThread.run(Replica.java:1382)
Caused by: java.lang.IllegalStateException: update of local topology failed from commit trigger for topology seq#: 392
	at oracle.kv.impl.rep.migration.MigrationManager$CompletionTrigger.commit(MigrationManager.java:2196)
	at com.sleepycat.je.dbi.TriggerManager$7.run(TriggerManager.java:237)
	at com.sleepycat.je.dbi.TriggerManager.runTriggers(TriggerManager.java:343)
	... 5 more

[KVSTORE-1868]

Changed the error message of ConnectionTimeoutException to include endpoint information. The resulting message looks like the following for connections without SSL:

    oracle.kv.impl.async.exception.ConnectionTimeoutException: Problem with channel ((/127.0.0.1:33430->uranus/127.0.1.1:34455)NBL): Heartbeat timeout, no read event during last 1000 ms, endpointHandlerId=6479ecc3c752b3aa

And the message looks like the following for connections with SSL:

    oracle.kv.impl.async.exception.ConnectionTimeoutException: Problem with channel ((/127.0.0.1:35615->localhost/127.0.0.1:40019)NBL client): Heartbeat timeout, no read event during last 200 ms, endpointHandlerId=45c554258911aa74

[KVSTORE-1888]

Fixed a vulnerability to DNS failure. Before the fix, slow DNS resolution could cause the failure of some or all network communications on a node. The problem was fixed by moving potentially blocking DNS operations out of critical paths in the network layer.
[KVSTORE-1766]
Added the new -storage-type flag to the makebootconfig command. The flag can be used to specify the type of disk where the storage directory resides. Valid storage type values are HD, SSD, NVME, and UNKNOWN. Note that a somewhat similar -storage-dir-storage-type flag was previously documented but was not implemented.
[KVSTORE-1174]
Updated the version of transport layer security (TLS) used in communications between an Elasticsearch cluster and the store from TLSv1 to TLSv1.2 to improve security.
[KVSTORE-2031]

Fixed a problem that caused the 'plan change-parameters' Admin CLI command to fail if called to change SN parameters on a store that had been upgraded to 23.1. The failed plan would display a stack trace such as:

kv-> show plan -id 72;
Plan Change Storage Node Params
Owner:                 root(id:u1)                   
Id:                    72                            
State:                 ERROR                         
Attempt number:        1                             
Started:               2023-10-31 11:13:12 UTC       
Ended:                 2023-10-31 11:13:38 UTC       
Plan failures:         
	Failure 1: 1/WriteNewSNParams sn1 failed.: java.lang.NullPointerException:null (23.1.25)
Total tasks:           4                             
 Progress percent:     75                            
 Successful:           3                             
 Failed:               1                             

Failures:
   Task   1       ERROR at   2023-10-31 11:13:13 UTC: Plan 72 [Change Storage Node Params] task [WriteNewSNParams sn1]: 1/WriteNewSNParams sn1 failed.: java.lang.NullPointerException:null (23.1.25): oracle.kv.impl.sna.SNAFaultException: java.lang.NullPointerException:null (23.1.25)
Fault StackTrace: java.lang.NullPointerException
	at oracle.kv.impl.param.ParameterMap.merge(ParameterMap.java:316)
	at oracle.kv.impl.param.ParameterMap.merge(ParameterMap.java:290)
	at oracle.kv.impl.sna.StorageNodeAgent.replaceStorageNodeParams(StorageNodeAgent.java:4202)
	at oracle.kv.impl.sna.StorageNodeAgentImpl.lambda$newStorageNodeParameters$15(StorageNodeAgentImpl.java:605)
	at oracle.kv.impl.fault.ProcessFaultHandler.execute(ProcessFaultHandler.java:170)
	at oracle.kv.impl.sna.StorageNodeAgentImpl.newStorageNodeParameters(StorageNodeAgentImpl.java:602)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at oracle.kv.impl.util.AbstractInvocationHandler.invokeMethod(AbstractInvocationHandler.java:60)
	at oracle.kv.impl.security.SecureProxy$CheckingHandler.lambda$invoke$1(SecureProxy.java:568)
	at oracle.kv.impl.security.ExecutionContext.runWithContext(ExecutionContext.java:179)
	at oracle.kv.impl.security.SecureProxy$CheckingHandler.invoke(SecureProxy.java:566)
	at oracle.kv.impl.security.SecureProxy.invokeNonObject(SecureProxy.java:108)
	at oracle.kv.impl.util.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:107)
	at com.sun.proxy.$Proxy21.newStorageNodeParameters(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at oracle.kv.impl.util.AbstractInvocationHandler.invokeMethod(AbstractInvocationHandler.java:60)
	at oracle.kv.impl.async.JavaSerialMethodTable$JavaSerialMethodCall.callService(JavaSerialMethodTable.java:174)
	at oracle.kv.impl.async.JavaSerialResponder.callServer(JavaSerialResponder.java:104)
	at oracle.kv.impl.async.JavaSerialResponder.lambda$handleRequest$0(JavaSerialResponder.java:87)
	at oracle.kv.impl.async.AsyncVersionedRemoteDialogResponder.withThreadDialogContext(AsyncVersionedRemoteDialogResponder.java:315)
	at oracle.kv.impl.async.JavaSerialResponder.lambda$handleRequest$1(JavaSerialResponder.java:85)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

[KVSTORE-2152]

Storage Engine Changes (JE 23.3)

Recovery from backup will no longer fail due to missing reserve files.
To describe the problem and solution in more detail, backups do not include reserved files, only the active log set. As such when restoring from a backup recovery finds which reserve files are missing and removes them from the reservedFileDB and truncates the VLSNIndex to get rid of any references to VLSNs in the missing reserved files. However, it is possible in rare cases that the active log files do not contain any replicated logs, or at least do not contain the last matchpoint. In the case of KVSTORE-467, the active log files did not contain any replicated logs, thus causing startup to fail because the VLSNIndex was truncated till it was empty. In this case, the issue is that the matchpoint was not in the active log set, so the admin node could not start up because the VLSNIndex cannot truncate away the last matchpoint.

The fix is to determine during backup the last matchpoint, and if the matchpoint is not in the active log set, add reserved files to the active log set until the matchpoint is part of the backup set.

Note:
The JE backup is part of the implementation of the NoSQL Database snapshot facility, so the benefits of this fix will be seen there.

[KVSTORE-1931] (23.2.0)