Changes in 22.3.16

The following changes were made in Oracle NoSQL Database Release 22.3.16 Enterprise Edition.

Topics

• New Features
• Bug and Performance Fixes

New Features

1. Implemented functional indexing in SQL queries. With this feature, an index can also index the values of one or more functions over index paths. The functions that can be indexed are a subset of the builtin functions. Specifically, the following functions can be indexed:

   modification_time, expiration_time, expiration_time_millis, row_storage_size, year, month, day, hour, minute, second, millisecond, microsecond, nanosecond, week, length, replace, reverse, substring, trim, ltrim, rtrim, lower, upper.

   For example, the following index indexes the rows of the Users table by their latest modification time:

   create index idx_modtime on Users (modification_time())

   As another example, the following index indexes users in the Users table by the year of the start date of their address:

   create index idx_year on Users (substring(address.startDate as string, 0, 4))

   The above definition declares that the startDate field of the address column (which is of type JSON) must be of type STRING. Furthermore, it assumes that these date strings are in the YYYY-MM-DD format. So, the substring function is used to extract the YYYY part of the date and use it as the index key.

   [KVSTORE-1129]

2. Added support for compressing rotated log files. Compression is currently disabled by default.

   You can enable log file compression by setting the new serviceLogFileCompression Storage Node parameter to true. Restart the associated Storage Node Agent to make the new setting take effect for new files and to compress existing files.

   Compressed log files store significantly more logging output in the same amount of disk space. As a result, logging output can be retained for longer, making more of it available to debug older issues. Log files contain a lot of redundancy, so compressing them with the standard gzip algorithm is very effective. The implementation makes conservative assumptions about the compression ratio, allowing it to store approximately five times more data by default as compared with uncompressed files.

   When compression is enabled, the meaning of the adminLogFileLimit and serviceLogFileLimit parameters is adjusted to take the expected compression ratio into account, meaning that log files will be allowed to grow larger than the specified size. With the default maximum file count, the actual size limit will be approximately five times larger than the one specified.

   Log file sizes may temporarily exceed the requested limits in some cases when the value of the compression parameter is changed.

   Compression is applied to the rotated copies of the store combined debug logs, stat files, and perf files, plus the service debug log files for all services (RNs, Arbiters, Admins, and SNs). Bootstrap debug log files, GC log files, JE debug log files, and JE stats files are not currently compressed.

   The compressed rotated log files are renamed with a .gz suffix and the filename is modified to use the creation time rather than the standard log rotation number. For example, the standard rotated log file rg1-rn1_1.log would be renamed after it was compressed to a file with a name like rg1-rn1_20220720-201902.log.gz, where the date portion means the file was created at '2022-07-20 20:19:02 UTC'. In the unusual case that multiple rotated log files are created in the same second, a unique suffix is added to the date portion of the file name. For example:

   rg1-rn1_20220720-201902-1.log.gz

   When working with compressed log files, consider using the gzcat or zgrep commands, or uncompress the files into another directory, to avoid using extra disk space in the standard log directory. Manually uncompressed files will not be deleted automatically.

   [KVSTORE-879]

3. Made several improvements intended to make it easier to use the debug logs to diagnose store failures.
   • All services (Rep Nodes, Admins, Arbiters, and Storage Node Agents) now consistently log status changes with log entries that include the text "LIFECYCLE" in the logging message. The entries also identify when the service is starting after a crash.
   • Log entries for Rep Node and Admin replication state changes have been updated to include the text "LIFECYCLE", and the format has been modified slightly to make it easier to read.
   • Services now log software version information both on service startup and whenever a new service log file is started after log file rotation.

     [KVSTORE-1417]

4. Registering an Elasticsearch cluster with the store to enable Full Text Search now requires that the store be running Java 11 or later. Using the Admin CLI plan register-es command to register a Elasticsearch cluster when the store is running with an earlier Java version will fail with a message such as:

   kv-> plan register-es -clustername fts -host ftshost -port 20000
   oracle.kv.impl.admin.IllegalCommandException:Can not register ES Cluster, as it requires Java 11 or later, but running Java version 1.8.0_351

   If the nodes of the store are running different Java versions, only some of them Java 11 or later, the plan command may not fail, but Full Text Search will not work properly.

   [KVSTORE-1503]

Bug and Performance Fixes

1. Fixed a bug in the initialization of multi-region tables that could cause the copying of table data to fail to complete for very large tables.

   [KVSTORE-1595]

2. Fixed a bug that, in some uncommon cases, could cause the Streams API to resume streaming from the wrong checkpoint, possibly causing some stream operations to be missed.

   [KVSTORE-1630]

3. Fixed a bug where a multi-region table might miss some changes from a remote region if the table in the remote region was dropped and recreated around the same time that a replication node in local region restarted.

   [KVSTORE-1649]

4. Updated the output format of the verify configuration Admin CLI command, modifying the warning message Storage directory on rgX-rnY is running low ... to change from showing the amount of space used to showing the amount of space available. The format of the new warning message matches the availableStorageSize entry in the main output and correctly reflects the amount of space available before the node runs out of space.

   [KVSTORE-399]

5. Fixed a bug that caused the Admin CLI command 'show zones [-zn <id> | -znname <name>] -json' to not include the information about SNs.

   [KVSTORE-570]

6. Modified the Admin CLI plan deploy-sn -zn <zone-name> command to provide a better explanation of the problem in the failure message when an invalid zone name is specified.

   [KVSTORE-565]

7. Fixed a problem that a read operation using Consistency.Version performed on a multi-shard store could cause a RepNode to restart if the shard specified in the version did not match the shards available in the RepNode. This situation could occur if an application performed a read request using Consistency.Version returned by an update request for a different primary key, or if the record being read had been migrated by an elasticity operation. The problem could also occur when performing a multi-key operation, such as KVStore.storeIterator or TableAPI.tableIterator, using Consistency.Version on a multi-shard store.

   This bug would produce stack traces like the following:

   2022-09-01 18:13:37.187 UTC SEVERE [rg2-rn1] Process exiting
   java.lang.IllegalArgumentException: Replication environment mismatch. The UUID associated with the commit token is: 8c18cf27-b4fe-4795-8028-f1181ab6cdfc but this replica environment has the UUID: 79e879c7-bd33-4b73-a53c-eba0664c674b
           at com.sleepycat.je.rep.CommitPointConsistencyPolicy.ensureConsistency(CommitPointConsistencyPolicy.java:151)
           at com.sleepycat.je.rep.txn.ReadonlyTxn.checkConsistency(ReadonlyTxn.java:189)
           at com.sleepycat.je.rep.txn.ReadonlyTxn.txnBeginHook(ReadonlyTxn.java:156)
           at com.sleepycat.je.txn.Txn.<init>(Txn.java:353)
           at com.sleepycat.je.txn.Txn.<init>(Txn.java:303)
           at com.sleepycat.je.rep.txn.ReadonlyTxn.<init>(ReadonlyTxn.java:65)
           at com.sleepycat.je.rep.impl.RepImpl.createRepUserTxn(RepImpl.java:1225)
           at com.sleepycat.je.txn.Txn.createUserTxn(Txn.java:393)
           at com.sleepycat.je.txn.TxnManager.txnBegin(TxnManager.java:178)
           at com.sleepycat.je.dbi.EnvironmentImpl.txnBegin(EnvironmentImpl.java:2379)
           at com.sleepycat.je.Environment.beginTransactionInternal(Environment.java:1459)
           at com.sleepycat.je.Environment.beginTransaction(Environment.java:1350)
           at oracle.kv.impl.api.RequestHandlerImpl.executeInternal(RequestHandlerImpl.java:902)
           ...

   [KVSTORE-1635]

8. Fixed a bug that using the SQL INSERT statement to set the value of a field of type NUMBER to a numeric value with an exponent less than -324 was setting the value of the field to 0.

   [KVSTORE-1631]

9. Fixed a bug which could cause the thread in the Storage Node Agent responsible for master rebalancing to fail because of an attempt to divide by zero producing an java.lang.ArithmeticException. After the problem was encountered, the SNA would not perform the operations needed to distribute masters evenly across shards until the SNA was restarted.

   This bug would produce log entries like the following:

   2022-07-27 08:00:18.659 UTC SEVERE [sn5] MasterRebalanceThread thread exiting due to exception.
   java.lang.ArithmeticException: / by zero
           at oracle.kv.impl.sna.masterBalance.RebalanceThread.getLeaseAdjustedMD(RebalanceThread.java:858)
           at oracle.kv.impl.sna.masterBalance.RebalanceThread.needsRebalancing(RebalanceThread.java:242)
           at oracle.kv.impl.sna.masterBalance.RebalanceThread.run(RebalanceThread.java:329)
   2022-07-27 08:00:18.660 UTC INFO [sn5] Master balance manager shutdown
   2022-07-27 08:00:18.661 UTC INFO [sn5] MasterRebalanceThread thread exited.

   [KVSTORE-1603]

10. Fixed a problem that, in rare occasions, could cause a Replication Node to restart. The problem was likely introduced in the 22.1 release and could generate log entries such as:

    2022-09-19 17:23:26.934 UTC SEVERE [rg3-rn1] JE: Replica unexpected exception
    com.sleepycat.je.EnvironmentFailureException: (JE 22.3.5)
    rg3-rn1(1):/block1/opc/workspace/arbiter-stress/kv/workdir/scratch/kvroot/mystore/sn5/disk1/rg3-rn1/env
    java.lang.IllegalStateException: update of local topology failed from commit
    trigger UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to
    continue. Environment is invalid and must be closed. Problem seen replaying
    entry Commit/22 vlsn=7795150 isReplicated="1"
    com.sleepycat.je.EnvironmentFailureException: (JE 22.3.5)
    rg3-rn1(1):/block1/opc/workspace/arbiter-stress/kv/workdir/scratch/kvroot/mystore/sn5/disk1/rg3-rn1/env
    java.lang.IllegalStateException: update of local topology failed from commit
    trigger UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to
    continue. Environment is invalid and must be closed. Problem seen replaying
    entry Commit/22 vlsn=7795150 isReplicated="1"
    	at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:402)
    	at com.sleepycat.je.dbi.TriggerManager.runTriggers(TriggerManager.java:347)
    	at com.sleepycat.je.dbi.TriggerManager.runCommitTriggers(TriggerManager.java:231)
    	at com.sleepycat.je.txn.Txn.commit(Txn.java:809)
    	at com.sleepycat.je.rep.txn.ReplayTxn.commit(ReplayTxn.java:299)
    	at com.sleepycat.je.rep.impl.node.Replay.replayEntry(Replay.java:766)
    	at com.sleepycat.je.rep.impl.node.Replica$ReplayThread.run(Replica.java:1356)
    Caused by: java.lang.IllegalStateException: update of local topology failed
    from commit trigger
    	at oracle.kv.impl.rep.migration.MigrationManager$CompletionTrigger.commit(MigrationManager.java:2039)
    	at com.sleepycat.je.dbi.TriggerManager$7.run(TriggerManager.java:237)
    	at com.sleepycat.je.dbi.TriggerManager.runTriggers(TriggerManager.java:343)
    	... 5 more

    [KVSTORE-1657]

11. Fixed a problem where a request performed on an entry that has been migrated to another shard as part of an elasticity operation could fail with a FaultException rather than being automatically retried on the new shard. An example of the exception thrown:

    oracle.kv.FaultException: Partition: PARTITION-185 not present at RepNode
    rg2-rn3, topology seq#: 383 (22.2.6) on [2022-06-08 05:07:28.351 UTC]
    Fault class name: oracle.kv.impl.rep.IncorrectRoutingException
    Resource id is: rg2-rn1
    Remote stack trace: oracle.kv.FaultException: Partition: PARTITION-185 not
    present at RepNode rg2-rn3, topology seq#: 383 (22.2.6) on [2022-06-08
    05:07:28.351 UTC]
    Fault class name: oracle.kv.impl.rep.IncorrectRoutingException
    Resource id is: rg3-rn2
    	at oracle.kv.impl.fault.ServiceFaultHandler.getThrowException(ServiceFaultHandler.java:175)
    	at oracle.kv.impl.fault.ServiceFaultHandler.getThrowException(ServiceFaultHandler.java:156)
    	at oracle.kv.impl.fault.ProcessFaultHandler.rethrow(ProcessFaultHandler.java:231)
    	at oracle.kv.impl.fault.ProcessFaultHandler.execute(ProcessFaultHandler.java:149)
    	at oracle.kv.impl.api.RequestHandlerImpl.execute(RequestHandlerImpl.java:584)
    	at oracle.kv.impl.api.RequestHandlerImpl.lambda$executeAsyncRequest$4(RequestHandlerImpl.java:674)
    	at oracle.kv.impl.async.FutureUtils$CheckedSupplier.get(FutureUtils.java:560)
    	at oracle.kv.impl.async.AsyncVersionedRemoteDialogResponder.withThreadDialogContext(AsyncVersionedRemoteDialogResponder.java:312)
    	at oracle.kv.impl.api.RequestHandlerImpl.executeAsyncRequest(RequestHandlerImpl.java:673)
    	at oracle.kv.impl.api.RequestHandlerImpl.lambda$execute$1(RequestHandlerImpl.java:622)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    	at java.lang.Thread.run(Thread.java:833)

    [KVSTORE-1537]

12. Fixed a bug where an index query involving an array could return incorrect results.

    [KVSTORE-1643]

13. Note that the Ant build script no longer works to build the Community Edition. This change was not previously mentioned, but went into effect with the 21.2 release.