Changes in 22.1.16

Topics

New Features

Added new classes that allow programmatic creation and control of a small, standalone store instance, comprising a single partition. The major classes are
- KVLocal - represents the store instance
- KVLocalConfig - allows configuration of the KVLocal instance
KVLocal can be configured to use TCP/IP connections or, if using Java 16 or higher, Unix domain sockets.

In support of KVLocal, the requirement that all keys specified in Operation and TableOperation lists passed to KVStore.execute and TableAPI.execute, respectively, share the same Major Path or shard key is relaxed for single partition stores such as the ones managed by KVLocal.

[KVSTORE-1064]
Added three new SQL functions to do arithmetic operations on Timestamps:
- ```
timestamp(9) timestamp_add(timestamp, string)
```
  Adds a duration to a timestamp value and returns the new timestamp. The duration is a string with format [-]<n> (<UNIT>)+, where the <UNIT> can be YEAR, MONTH, DAY, HOUR, MINUTE, SECOND, MILLISECOND, NANOSECOND or the plural form of these keywords (e.g. YEARS). The UNIT keywords are case-insensitive. (e.g. 1 day 12 hours).
- ```
long timestamp_diff(timestamp, timestamp)
```
  Returns the number of milliseconds between two timestamp values (timestamp1 - timestamp2).
- ```
string get_duration(long)
```
  Converts the given number of milliseconds to a duration string.
[KVSTORE-1338]
In SQL queries, allow casting of a timestamp value to a long value that is the number of milliseconds since January 1, 1970, 00:00:00 GMT.
[KVSTORE-1247]
Added support for child tables to multi-region tables. Child tables always inherit the regions of the associated top level tables, so users cannot specify regions when creating child tables, nor can they explicitly add or drop regions for child tables.
[KVSTORE-1114]
Added support for MRCounter in JSON so that the MR_COUNTER data type can be used inside a schemaless JSON field. The syntax to declare an MR_COUNTER field for a JSON column is:
```
field-name AS field-type MR_COUNTER
```
A JSON MR_COUNTER can be updated the same way as any other MR_COUNTER column.

[KVSTORE-1223]
Switched the KeyStore and TrustStore type of the default security configuration created by makebootconfig or securityconfig utility to PKCS12.
The KeyStores and TrustStores used by the NoSQL Database server in the default security configuration will be created as PKCS12 password-protected stores. The TrustStore used by client applications (client.trust) in the default security configuration will be created as a PKCS12 password-less store if that feature is supported by the version of Java used to run the configuration utility. If not, the utilities will fall back to creating a JKS store if no password is specified for client.trust.
Two new optional flags are introduced for security configuration creation:
- ```
-kstype <JKS | PKCS12>
```
  Create KeyStores and TrustStores in the security configuration as PKCS12 or JKS store, default PKCS12.
- ```
-ctspwd <client.trust password>
```
  Create client.trust as a password-protected PKCS12 store using the specified password.
Upgrading to this release won't convert the store type of the existing KeyStores or TrustStores created by previous releases to PKCS12 automatically. To switch the store type of an existing configuration, a new command has been added to securityconfig utility:
```
securityconfig config update -secdir security -kstype PKCS12
```
Added two new client security properties to configure the password for client.trust if it's created as a password-protected PKCS12 store.
- ```
oracle.kv.ssl.trustStorePassword
```
- ```
oracle.kv.ssl.trustStorePasswordAlias
```
The password of client.trust can be specified either directly via oracle.kv.ssl.trustStorePassword or kept in the password store. If the latter, specify the alias name of the password in the password store via the oracle.kv.ssl.trustStorePasswordAlias parameter, and the NoSQL Database Driver will retrieve the password from the password store automatically.

Updated the merge-trust command in securityconfig utility. When merging a security configuration created with a PKCS12 store into a security configuration created with a JKS store, this command will automatically convert the store type of the TrustStore to PKCS12 after merging all trusted certificates. However, this command won't convert the store type when merging trusted certificates of security configurations that are both created with JKS store.

[KVSTORE-60]

Bug and Performance Fixes

Fixed a bug that could have caused a NullPointerException when executing "describe table" against a multi-region table.
[KVSTORE-1426]
Fixed a bug where adding or removing regions from a multi-region table might not have been executed correctly when the store was under high stress.
[KVSTORE-1374]
Fixed a bug where if a key-only table was later altered to add an identity column, an error would be raised if an attempt was made to read a key-only row that was inserted before the alter.
[KVSTORE-1370]
Fixed a bug where implicit casting of a string to a timestamp does not work when the string comes from JSON data.
[KVSTORE-1477]
Fixed a bug that the default value of an internal JE parameter preHeartbeatTimeoutMs is not set to a proper value. This bug will cause rare cases that the store takes more than one hour to recover from faults or a master transfer.
[KVSTORE-1384]
Fixed a bug that when a key-only MR table evolves by adding new columns, the newly added columns might be lost when being replicated to remote regions.
[KVSTORE-1333]
Fixed a problem where a row with a homogeneous JSON array of type STRING written with release 20.2 or earlier releases would fail to deserialize in release 20.3 or later releases. The issue shows up as a row that is not found in the result of a get() or in results of queries that need to deserialize the row value. The data remains intact but is not accessible.
[KVSTORE-1347]
Fixed a bug in SQL INSERT when the table has an auto-generated identity or UUID column and a column list is used before the VALUES clause. In this case, an exception would be thrown, but should not have been.
[KVSTORE-1371]
Fixed a query bug when the partition() function is used in the WHERE clause. The following example demonstrates the bug:
```
select id, partition($f) as part from foo $f
where $f.address.state = "MA" and partition($f) = 2
```
The above query will be sent for execution to an RN that contains partition 2. If there is an index on address.state, the query will use that index at that RN. In general, the index will contain entries from multiple partitions that are stored at the RN. The bug was that while the index was scanned, entries that do not belong to partition 2 were not filtered out.

[KVSTORE-1334]
Fixed a bug that prevented a table with a UUID column from being altered after the table was schema evolved: rows inserted before the schema evolution would not deserialize after the schema evolution.
[KVSTORE-1357]
Fixed a bug in computing the result type of a query. Specifically, the nullability flag of a field in the result type could be wrongly set to false. This would happen if the SELECT-clause expression that corresponds to that field may not return a NULL, but may return an empty result. The nullability flag should be set to true in this case because the SELECT clause converts EMPTY to NULL.
[KVSTORE-1342]
Fixed a bug where plans in the APPROVED state would not be run after an Admin restart.
[KVSTORE-1194]
Fixed a bug where the Streams API could lose writes to a new shard created during a store expansion.
[KVSTORE-1336]
Fixed a problem that could occur if the XRegion Agent was creating a new multi-region table and, at the same time, it found a previously missing multi-region table in a remote region. In this case, the agent could fail to add the remote region to the table, and the local table would not receive updates from the remote region.
[KVSTORE-1300]
Fixed compatibility problems in deserializing the server-side execution plan of proxy-based queries. Problems involving two cases are fixed:
1. A query is prepared with a kvclient that is newer than the servers.
2. A query is prepared with a kvclient at version V1. The prepared query is cached by the application. Then, kvclient and kvserver are upgraded to V2. The application executes the cached query again. In this case, the servers would attempt (and fail) to deserialize the query plan using V2, instead of V1.
[KVSTORE-1363]
Fixed a bug where adding a multi-region table could fail when there were network or remote region failures. The XRegion agent now has a retry mechanism so that it will add the table when the failures are resolved.
Also fixed a bug that when a Streams API subscription reconnects to a store, the subscription may resume from a position later than expected. This problem could affect both the Streams API and multi-region tables.

[KVSTORE-1265]
Fixed an incompatible change introduced in 20.3 in the performance JSON files (e.g., rnOp*.json). The incompatible change was that latency metrics (for example AllSingleKeyOperations_Interval_Avg) were changed to integer values where they were previously floating-point numbers. This change caused sub-millisecond latency values to appear as zero. Fixed by reverting back to using floating-point numbers.
[KVSTORE-1289]
Fixed a problem where a query run on a table with data that was created prior to release 19.5 and never modified might fail with an error that indicates "modification time not available".
[KVSTORE-1234]
Fixed a bug with unnesting queries. Query results would be lost if the query was not using an unnesting index and the batch size was reached in the midst of unnesting the arrays or maps specified in the query.
[KVSTORE-1228]
Fixed a bug with unnesting queries. Queries that specified count(*) were incorrectly using an unnesting index that unnested at a deeper level than the query.
[KVSTORE-1292]
Fixed two bugs related to query timeouts. The first problem was that a query could timeout at the driver but keep running at the RNs. The second problem was that the query timeout would not be respected if the query performed a generic (non-index-based) sort or group by.
[KVSTORE-353]
Modified the output of the 'show admins' Admin CLI command to display the admin storage directory path, and of the 'ping' command to display the size of the admin storage directory.
[KVSTORE-102]
Added TLSv1.3 to the default TLS protocols of security configurations created with the makebootconfig or securityconfig utilities. The default TLS protocols for security configuration are now "TLSv1.3, TLSv1.2". Note that the new protocols are only used when creating new security configurations with the above utilities. Upgrading to this release will not automatically update the existing security configuration of a running store. We recommended upgrading existing stores to use the TLSv1.3 protocol: see "Guidelines for Enabling TLSv1.3 Protocol" in Security Guide of Oracle NoSQL Database.
[KVSTORE-263]
The makebootconfig and securityconfig utilities now generate 2048 byte RSA keys. Note that updated keys are only created when generating new security configurations using the above utilities. Upgrading to this release will not automatically update the existing private keys used by a running store. We recommended upgrading existing stores to use new private keys so that they have 2048 byte RSA keys: see "Guidelines for Updating SSL Keys and Certificates" in Security Guide of Oracle NoSQL Database.
[KVSTORE-932]
Modified the output from the "describe table" DDL command to include multi-region table information if the table is multi-region.
[KVSTORE-1155]
Made changes to prevent a multi-region table that is missing at a remote region from disrupting existing multi-region tables in local and other remote regions. If an MR table is missing or has a incompatible schema in a remote region, with the fix the agent will check the particular table periodically while, at the same time, continuing to service other remote regions where the MR table is available. When the table in question is available again in the remote region, for example if the missing table is recreated or the incompatible schema is fixed, the agent will include the table and service it like other available MR tables.
[KVSTORE-395]
Fixed a bug that prevented key-only multi-region tables from working properly.
[KVSTORE-1302]
Fixed several bugs with partition migration, which is the underlying mechanism to move user data from one shard to another during store elasticity operations. The following problems were fixed:
- Record modification times could be migrated incorrectly, which caused problems with multi-region tables
- Client writes could be lost at the end of the migration under some edge cases
- The migration service might not be available after certain RN replication state changes.
[KVSTORE-1250] [KVSTORE-1244] [KVSTORE-1221]
Improved the robustness of the Java direct driver in handling network connection problems it encounters when contacting storage nodes.
[KVSTORE-1423]
Fix a bug that could cause RepNode crashes or RepNode metadata corruption in a short time window after an upgrade during which the store version had not been fully updated. This bug could cause RepNodes to crash if a new table with UUIDs or MRCounters was added, or an existing table was altered to add UUID or MRCounter fields, during the time window. For tables with UUIDs, RepNodes would crash during the time window, but the problem would fix itself after the store version was updated. For tables with MRCounters, the bug could cause persistent metadata corruption, which would result in RepNodes continuing to crash on restart.
[KVSTORE-1361]

API Changes

Row.getExpirationTime() no longer throws an exception if the underlying Row metadata is not available. Instead it will return 0 indicating no expiration time. Expiration time is 0 for rows that either do not have a TTL or were written prior to the introduction of the TTL feature.
Also added Row.getLastModificationTime() to return the last modification time of a Row in milliseconds since January 1, 1970. If the row was last modified using a version of the system older than release 19.5, the modification time is not available and 0 is returned.

[KVSTORE-1234]
Marked the IndexKeySizeLimitException class as hidden; it is for internal use only.
[NOSQL-408]