Migrating TTL Metadata for Table Rows

Learn how to migrate TTL data from source to the sink.

Time to Live (TTL) is a mechanism that allows you to automatically expire table rows. TTL is expressed as the amount of time, data is allowed to live in the store. Data that has reached its expiration timeout value can no longer be retrieved, and will not appear in any store statistics.

You can choose to include the TTL metadata for table rows along with the actual data when performing migration of Oracle NoSQL Database tables. The NoSQL Database Migrator provides configuration parameters to support the export and import of table row TTL metadata for the following source types:

Table 1 - Migrating TTL metadata

Source types	Source configuration parameter	Sink configuration parameter
Oracle NoSQL Database	`includeTTL`	`includeTTL`
Oracle NoSQL Database Cloud Service	`includeTTL`	`includeTTL`
DynamoDB-Formatted JSON File	`ttlAttributeName`	`includeTTL`
DynamoDB-Formatted JSON File stored in AWS S3	`ttlAttributeName`	`includeTTL`

Exporting TTL metadata in Oracle NoSQL Database and Oracle NoSQL Database Cloud Service

NoSQL Database Migrator provides the includeTTL configuration parameter to support the export of table row’s TTL metadata.

When a table is exported, the TTL data is exported for the table rows that have a valid expiration time. If a row does not expire, then the _metadata JSON object is not included explicitly in the exported data because its expiration value is always 0. The NoSQL Database Migrator exports the expiration time for each row as the number of milliseconds since the UNIX epoch (Jan 1st, 1970). For example,

//Row 1
{
    "id" : 1,
    "name" : "xyz",
    "age" : 45,
    "_metadata" : {
        "expiration" : 1629709200000   //Row Expiration time in milliseconds
    }
}

//Row 2
{
    "id" : 2,
    "name" : "abc",
    "age" : 52,
    "_metadata" : {
        "expiration" : 1629709400000   //Row Expiration time in milliseconds
    }
}

//Row 3 No Metadata for below row as it will not expire
{
    "id" : 3,
    "name" : "def",
    "age" : 15
}

Importing TTL metadata

You can optionally import TTL metadata using the includeTTL configuration parameter in the sink configuration template.

The default reference time of import operation is the current time in milliseconds, obtained from System.currentTimeMillis(), of the machine where the NoSQL Database Migrator tool is running. However, you can also set a custom reference time using the ttlRelativeDate configuration parameter if you want to extend the expiration time and import rows that would otherwise expire immediately. The extension is calculated as follows and added to the expiration time.

Extended time = expiration time - reference time

The import operation handles the following use cases when migrating table rows containing TTL metadata. These use cases are applicable only when the includeTTL configuration parameter is set to true.

Use-case 1: No TTL metadata information is present in the importing table row.

If the row you want to import does not contain TTL information, then the NoSQL Database Migrator sets the TTL=0 for the row.
Use-case 2: TTL value of the source table row is expired relative to the reference time when the table row gets imported.

The expired table row is ignored and not written into the store.
Use-case 3: TTL value of the source table row is not expired relative to the reference time when the table row gets imported.

The table row gets imported with a TTL value. However, the imported TTL value may not match the original exported TTL value because of the integer hour and day window constraints in the TimeToLive class. For example,

Consider an exported table row:
```
{
  "id" : 8,
  "name" : "xyz",
  "_metadata" : {
  "expiration" : 1734566400000 //Thursday, December 19, 2024 12:00:00 AM in UTC
  }
}
```
The reference time while importing is 1734480000000, which is Wednesday, December 18, 2024 12:00:00 AM.

Imported table row
```
{
  "id" : 8,
  "name" : "xyz",
  "_metadata" : {
    "ttl" :  1734739200000 //Saturday, December 21, 2024 12:00:00 AM
  }
}
```

Importing TTL Metadata in DynamoDB-Formatted JSON File and DynamoDB-Formatted JSON File stored in AWS S3

NoSQL Database Migrator provides an additional configuration parameter, ttlAttributeName to support the import of TTL metadata from the DynamoDB-formatted JSON file items.

DynamoDB exported JSON files include a specific attribute in each item to store the TTL expiration timestamp. To optionally import the TTL values from DynamoDB exported JSON files, you must supply the specific attribute’s name as a value to the ttlAttributeName configuration parameter in the DynamoDB-Formatted JSON File or DynamoDB-Formatted JSON File stored in AWS S3 source configuration files. Also, you must set the includeTTL configuration parameter in the sink configuration template. The valid sinks are Oracle NoSQL Database and Oracle NoSQL Database Cloud Service. NoSQL Database Migrator stores TTL information in the _metadata JSON object for the imported item.

The import operation manages the following use cases when migrating table items of the DynamoDB exported JSON files:

Use case 1: The ttlAttributeName configuration parameter value is set to the TTL attribute name specified in the DynamoDB exported JSON file.

NoSQL Database Migrator imports the expiration time for this item as the number of milliseconds since the UNIX epoch (Jan 1st, 1970).

For example, consider an item in the DynamoDB exported JSON file:
```
{
    "Item": {
        "DeptId": {
            "N": "1"
        },
        "DeptName": {
            "S": "Engineering"
        },
        "ttl": {
            "N": "1734616800"
        }
    }
}
```
Here, the attribute ttl specifies the time-to-live value for the item. If you set the ttlAttributeName configuration parameter as ttl in the DynamoDB-formatted JSON file or DynamoDB-formatted JSON file stored in AWS S3 source configuration file, NoSQL Database Migrator imports the expiration time for the item as follows:
```
{
  "DeptId": 1,
  "document": {
      "DeptName": "Engineering"
    }
  "_metadata": {
    "expiration": 1734616800000
  }
}
```
Note: You can supply the ttlRelativeDate configuration parameter in the sink configuration template as the reference time for calculating the expiration time.
Use case 2: The ttlAttributeName configuration parameter value is set, however, the value does not exist as an attribute in the item of the DynamoDB exported JSON file.

NoSQL Database Migrator does not import the TTL metadata information for the given item.
Use case 3: The ttlAttributeName configuration parameter value does not match the attribute name in the item of DynamoDB exported JSON file. NoSQL Database Migrator handles the import in one of the following ways based on the sink configuration:
- Copies the attribute as a normal field if configured to import using the default schema.
- Skips the attribute if configured to import using a user-defined schema.