1. Securing Oracle Enterprise Data Quality
  2. Configuring EDQ Encryption

11 Configuring EDQ Encryption

The encryption framework for EDQ now supports a pluggable module with implementations for OCI vaults and scripts as described in this chapter.

EDQ Enhanced Encryption Overview

To provide additional security, certain information stored in the EDQ configuration is encrypted.

  • Database passwords in director.properties (optional, Tomcat only)

  • Data store passwords

  • Stored credentials

  • Web push notification Vapid private keys

In EDQ versions up to 12.2.1.4.3, encryption is performed using the AES cipher in ECB mode with PKCS5Padding. At the first startup, a new random key is generated and stored. In Tomcat, the key is stored in the file localhome/kfile and in WebLogic the key is stored in the FMW key store.

The following limitations are seen in this implementation:

  • The default AES keysize (128 bits) is used. For additional security, 256 bits should be used.

  • The ECB cipher mode does not use an initialization vector (IV) so encrypting the same string always produces the same result. This can make it easier for attackers to guess passwords if they can see the encrypted values.

  • In Tomcat, the key file can be read by any user of the system.

  • If a new install is created using the existing schemas, a new key file is generated and existing encrypted data in the database cannot be recovered (this problem is common with the OCI marketplace images).

In EDQ 12.2.2.1.4.4 and later versions, encryption and decryption is handled by a pluggable security module. The module is configured by the new properties file localhome/security/module.properties. This must contain a type property defining the module type. The following types are supported:
  • legacy - The current mechanism used by default if module.properties does not exist.
  • default - A refactoring of the legacy module with improved security.
  • ocivault - Uses a key stored in an OCI vault. Encryption and decryption is performed by calls to the vault cryptographic endpoint. The key never leaves the vault so HSM or software protection modes can be used. Keys can be rotated for additional security - decrypt operations using the key version when the data was encrypted. It can be configured to act as a credentials store using vault secrets.
  • script - A JavaScript script can be configured to perform the encryption and decryption operations. This can be used to support additional key frameworks such as AWS Key Management Service (KMS) or GCP Cloud Key Management Service. The module can also be configured to act a as a credentials store using external services such as AWS and GCP Secrets Managers.
  • custom - Custom Java code. For more information, see Creating Custom Java Encryption and Credentials Store Modules.
The default module is used by new installations. The installations that are migrated from earlier versions continue to use the legacy module.

Default Security Module

The new default security module is similar to the legacy implementation but with these enhancements:
  • AES is used in GCM mode with a random initialization vector so encrypting the same data returns different results each time.
  • A 256-bit key is used if possible.
  • In Tomcat, the key is stored in the file localhome/masterkey which is set to mode 600 (-rw-------).

Ocivault Security Module

The ocivault security module type uses a key stored in an OCI vault. Authentication for calls to the OCI APIs uses either platform credentials or credentials configured with external properties. Stored credentials cannot be used here because the module can be used to decrypt passwords for the EDQ database schemas at startup time.

The following additional properties are used with the type in module.properties:

Table 11-1 Ocivault Security Module Properties

Property Description

vault

Vault name or OCID (required).

key

Key name or OCID (required).

region

OCI region name. If omitted, the region of the EDQ instance is used.

compartmentid

OCID of vault compartment. Used if vault or key are not specified by OCID. If omitted the compartment of the EDQ instance is used.

secrets.vault

Vault name or OCID for secrets queries. If omitted the key vault is used.

secrets.encrypt

Set to true if secret values are encrypted using the vault key. Secret values can be seen in the OCI console so they must be encrypted to hide them from the console.

secrets

Set to true to support credentials retrieval using vault secrets.

auth.X

Additional properties passed to the authenticator for OCI API calls. For example, set auth.profile to name a profile from ~/.oci/config.

The primary use case for the OCI module is with compute instances which can use platform credentials for authentication for OCI API calls. If the OCI dynamic groups imply the required permissions, no additional configuration is required.

Secrets Storage

If the module is enabled for secrets use, credentials for external integrations can be stored in a vault rather than in the EDQ configuration files. The examples include JMS broker connections, SMTP authentication, and LDAP credentials.

To specify credentials using a secret, replace the username and password properties with the following:

cred.secret = secretname

The secret vault in the value should be a JSON string containing username and password attributes, as shown below:

{ "username": "user1", "password": "mysecret2" }

To override the default secret encryption setting for the module, use the following:

cred.secret.encrypt = true | false

To specify the username in plain and retrieve the password from the vault, use the following:

cred.secret.username = username

The secret value in the vault is then just the password.

If no username is required, add the following:

cred.secret.passwordonly = true

The secret value in the vault is then just the password.

Sample OCI Vault module.properties Configurations

Using platform credentials with local compartment and region:


type            = ocivault
vault           = edqvault
key             = key1
secrets         = true
secrets.encrypt = true

Using a remote vault with credentials from ~/.oci/config: 


type            = ocivault
auth.profile    = default
region          = us-phoenix-1
compartmentid   = compartmentid
vault           = edqvault
key             = key1
secrets         = true

Script Security Module

The module definition for the script security module must include a script property which is the name of the script file. If the file is not absolute it is found relative to the EDQ local configuration directory. The script defines functions as described in the following table:

Table 11-2 Script Security Module Functions

Function Required Description

init(props)

No

Initialize the module. The argument is an object containing properties from module.properties.

encrypt(plain)

Yes

Encrypt some plain text. The argument and result are ArrayBuffers.

decrypt(plain)

Yes

Decrypt some ciphertext. The argument and result are ArrayBuffers.

getCredentials(props)

No

Retrieve credentials from a secrets store. The properties argument contains the cred.* settings from the EDQ configuration.

The properties passed to the init function include a type value set to "encryption". This allows the same script to be used as the security module and a credentials store.

base64 stuff

External encryption APIs require base64 encoding for plain and cipher text. EDQ scripts can use these base64 support functions:


var encoder = BASE64.getEncoder()
var enc     = encoder.encode(bytes)                 // Encode an ArrayBuffer to base64 string
var enc2    = encoder.encodeFromString(str)         // Encode a string to base 64 string
 
var urlenc  = BASE64.getUrlEncoder()                // Encoder using URL-safe alphabet
var mimeenc = BASE64.getMimeEncoder()               // Encoder with MIME line splitting
vat nopad   = BASE64.getEncoder().withoutPadding()  // Encoder which does not use = padding
 
var decoder = BASE64.getDecoder()
var dec     = decoder.decode(string)                // Decode a base64 string to ArrayBuffer
var dec2    = encoder.decodeToString(str)           // Decode a base 64 string to a string
 
var urldec  = BASE64.getUrlEncoder()                // Decoder using URL-safe alphabet
var mimdec  = BASE64.getMimeEncoder()               // Decoder with MIME line splitting

Sample Scripts

Using the AWS Key and Secrets Management Frameworks

module.properties


type         = script
keyid        = alias/rde1
region       = eu-west-1
script       = awsenc.js
secrets      = true
auth.profile = myprofile

awsenc.js


// Script security module sample for AWS KMS and secrets manager
// =============================================================
//
// Required properties:
//
//      region          AWS region
//      keyid           KMS keyid or alias/name
//
// Optional:
//
//      secrets         Set to true for secrets support
//
// Configure proxy server with https.proxyHost and https.proxyPort properties
 
addLibrary("http")
addLibrary("logging")
 
var client
var secclient
var encoder = BASE64.getEncoder()
var decoder = BASE64.getDecoder()
var kmsurl
var keyid
var secrets   = false
var secretenc = false
 
function init(props) {
 
  // Authentication properties
 
  var aprops = Object.keys(props).filter(k => k.startsWith("auth.")).reduce((o, k) => (o[k.substring(5)] = props[k], o), {})
 
  // Client for KMS
 
  client = HTTP.builder().withAWSAuthentication(aprops).build();
 
  // Properties
 
  var region   = props.region
 
  keyid = props.keyid
 
  if (!region || !keyid) {
    throw "awsenc: region or key not specified"
  }
 
  kmsurl  = "https://kms." + region + ".amazonaws.com"
 
  // Secrets handling
 
  secrets = props.secrets == "true"
   
  if (secrets) {
 
    // URL and client for secret queries
 
    securl    = "https://secretsmanager." + region + ".amazonaws.com"
    secclient = HTTP.builder().withAWSAuthentication(aprops).build();
  }
}
 
function encrypt(plain) {
  var req = client.requestbuilder().header("X-Amz-Target", "TrentService.Encrypt").build();
  var res = req.post(kmsurl, JSON.stringify({KeyId: keyid, Plaintext: encoder.encode(plain)}), "application/x-amz-json-1.1")
 
  return decoder.decode(JSON.parse(res.data).CiphertextBlob)
}
 
function decrypt(ctext) {
  var req = client.requestbuilder().header("X-Amz-Target", "TrentService.Decrypt").build();
  var res  = req.post(kmsurl, JSON.stringify({keyId: keyid, CiphertextBlob: encoder.encode(ctext)}), "application/x-amz-json-1.1")
 
  return decoder.decode(JSON.parse(res.data).Plaintext)
}
 
function getCredentials(props) {
  if (secrets) {
    var s = props.secret
 
    if (s) {
      var req = secclient.requestbuilder().failonerror(false).header("X-Amz-Target", "secretsmanager.GetSecretValue").build()
      var res = req.post(securl, JSON.stringify({SecretId: s}), "application/x-amz-json-1.1")
 
      if (res.code != 200) {
        logger.log(Level.WARNING, "AWS secrets manager call failed with code {0}", res.code)
      } else {
        var obj = JSON.parse(res.data)
        var str = obj.SecretString || decoder.decodeToString(obj.SecretBinary)
         
        if (props["secret.passwordonly"] == "true") {
          return {username: "", password: str}
        } else {
          var val = JSON.parse(str)
          return {username: val.username, password: val.password}
        }
      }
    }
  }
 
  return null
}

Using GCP Key and Secrets Frameworks

module.properties


type       = script
script     = gcpenc.js
project    = green-wombat-4252311
keyfile    = /opt/edq/gcp-encrypt.json
location   = europe-west2
keyring    = rde
key        = rde1
secrets    = true
seckeyfile = /opt/edq/gcp-secrets.json

gcpenc.js


// Script security module sample for GCP KMS and secrets manager
// =============================================================
//
// Required properties:
//
//      project         GCP project name
//      location        Key location (such as europe-west2)
//      keyring         Keyring name
//      key             Key name
//      keyfile         Service account key file location
//
// Optional:
//
//      version         Key version name
//
// Configure proxy server with https.proxyHost and https.proxyPort properties
 
addLibrary("http")
addLibrary("logging")
 
var client
var encoder = BASE64.getEncoder()
var decoder = BASE64.getDecoder()
var decurl
var encurl
var secrets   = false
 
function init(props) {
 
  // Need:
  //
  // project
  // location
  // keyring
  // key
  // keyfile
  //
  // Optional:
  //
  // version
 
  if (!props.project || !props.location || !props.keyring || !props.key || !props.keyfile) {
     throw "gcpenc: missing properties"
  }
 
  var project = props.project
  var location = props.location
 
  var base = "https://cloudkms.googleapis.com/v1"
        + "/projects/"   + props.project
        + "/locations/"  + props.location
        + "/keyRings/"   + props.keyring
        + "/cryptoKeys/" + props.key
 
  decurl = base + ":decrypt"
  encurl = base
 
  if (props.version) {
    encurl += "/cryptoKeyVersions/" + props.version
  }
 
  encurl += ":encrypt"
 
  // Client for KMS
 
  client = HTTP.builder().withAuthentication("GCP", {keyfile: props.keyfile, claim_scope: "https://www.googleapis.com/auth/cloud-platform"}).build();
 
  secrets = props.secrets == "true"
   
  if (secrets) {
 
    // URL and client for secret queries
 
    securl = "https://secretmanager.googleapis.com/v1/projects/" + props.project + "/secrets/"
  }
}
 
function encrypt(plain) {
  var req = client.requestbuilder().build();
  var res = req.post(encurl, JSON.stringify({"plaintext": encoder.encode(plain)}))
   
  return decoder.decode(JSON.parse(res.data).ciphertext)
}
 
function decrypt(ctext) {
  var req = client.requestbuilder().build();
  var res  = req.post(decurl, JSON.stringify({ciphertext: encoder.encode(ctext)}))
 
  return decoder.decode(JSON.parse(res.data).plaintext)
}
 
function getCredentials(props) {
  if (secrets) {
    var s = props.secret
 
    if (s) {
      var url = securl + s + "/versions/latest:access"
      var req = client.requestbuilder().failonerror(false).build()
      var res = req.get(url)
 
      if (res.code != 200) {
        logger.log(Level.WARNING, "GCP secrets manager call failed with code {0}", res.code)
      } else {
        var obj = JSON.parse(res.data)
        var str = decoder.decodeToString(obj.payload.data)
         
        if (props["secret.passwordonly"] == "true") {
          return {username: "", password: str}
        } else {
          var val = JSON.parse(str)
          return {username: val.username, password: val.password}
        }
      }
    }
  }
 
  return null
}

Encryption Migration

If the security module is replaced or reconfigured in an existing system, then encrypted data is no longer usable. If the system is a new install and encryption is not used in the director.properties, then this is not an issue. If there is an existing encrypted data, the new security module can be defined using a migration REST API.

There are two new system administration REST endpoints.

POST to /edq/admin/security/migrateencryption

The payload is a JSON object with the following structure:


{ "type" : "moduletype",
  "properties": {
 	... other settings for the module ...
  }
}
The migration process includes the following steps:
  1. A new security is created and initialized using the definition in the payload.
  2. Existing encrypted data is decrypted using the current module and encrypted using the new module.
  3. If the previous steps succeed, the new module replaces the current module and module.properties is written with the new definition.

The result of the migration call is a report summarizing the items which were updated. Any items which could not be decrypted are listed.

If the URL contains the query parameter?dryrun=true then the new module is created and decryption of existing data tested, but no updates are made.

Example Payload for Migration to a Remote OCI Vault


{ "type" : "ocivault",
  "properties": {
    "auth.profile"    : "default",
    "vault"           : "rde",
    "key"             : "rtest",
    "compartmentid"   : "compartmentid",
    "region"          : "us-phoenix-1",
    "secrets"         : true,
    "secrets.encrypt" : true
  }
}

POST to /etc/admin/security/rotateencryption

The payload is an empty JSON object. Existing encrypted data is decrypted using the current security module and then encrypted using the same module. This call can be used with a vault key after a new version is created so that the old version can be deleted.

The result of the call is a summary for the migrate call.

If the URL contains the query parameter ?dryrun=true, then decryption of existing data is tested but no updates are made.

Using Existing Schemas for New OCI Instance Without Losing Encrypted Data

If you create a new OCI instance but select the Use Existing Schemas option, the new instance starts with the legacy (or default) security module and creates a new secret key, rendering the existing encrypted data inaccessible. These are the steps to retain the data:

  1. Before shutting down the old system, ensure it is using some external encryption mechanism (OCI vaults or AWS).
  2. Create and setup the new system.
  3. Run encryption migration to setup the external module on the new system. Decryption fails for existing data because the current module is different but after migration is complete. The encrypted data is usable again.

Parent topic: Configuring EDQ Encryption

Configuring Pluggable Credentials Stores

The user name and password credentials used in EDQ configuration are normally defined in properties files, examples include:
  • Database schema passwords in director.properties
  • JMS broker authentication in "bucket" files
  • SMTP authentication in mail.properties
  • LDAP authentication in login.properties

When running an instance in WebLogic the credentials are stored in the FMW credentials store framework. The encryption module mechanism in EDQ 12.2.1.4.4 and later versions allows the security module to function as a credentials store but this does not support the definition of additional stores. For example, it is not possible to use a local encryption mechanism and retrieve credentials from external stores such as OCI Vaults or AWS Secrets Manager.

Configuring Additional Credential Stores

EDQ 12.2.1.4.4 and later versions extend the encryption module to allow additional credentials stores to be defined. The stores are configured by adding properties files to the localhome/security/credstores directory. Each properties file must contain a type property defining the store type along with additional settings which are specific to the store type. The following types are supported:

  • ocivault - The credentials are stored as secrets in an OCI vault and can be encrypted using a value key.
  • script - A JavaScript script can be configured to call out to external services such as AWS and GCP Secrets Managers.
  • custom - A custom Java class.

Credentials stored are queried in the alphabetical order of the associated file names.

Ocivault Credentials Store

The ocivault security module type queries secrets stored in an OCI vault. Authentication for calls to the OCI APIs uses either platform credentials or credentials configured with external properties. Stored credentials cannot be used here because the module can be used to query passwords for the EDQ database schemas at startup time.

The following additional properties are used with the type.

Table 11-3 Ocivault Credentials Store

Property Description

vault

Vault name or OCID (required).

secrets.encrypt

Set to true if secret values are encrypted using a key from the vault. The secret values can be seen in the OCI console so they must be encrypted to hide them from the console.

key

Key name or OCID (required if secrets.encrypt is true).

region

OCI region name. If omitted the region of the EDQ instance is used.

compartmentid

OCID of vault compartment. It is used if vault or key are not specified by OCID. If omitted the compartment of the EDQ instance is used.

auth.X

Additional properties passed to the authenticator for OCI API calls. For example, set auth.profile to name a profile from ~/.oci/config.

The primary use case for the OCI store is with compute instances which can use platform credentials for authentication for OCI API calls. As long as the OCI dynamic groups imply the require permissions no additional configuration is required.

Using Secrets

To specify credentials using a vault secret, replace the username and password properties with the following:

cred.secret = secretname

The secret vault in the value should be a JSON string containing username and passwordattributes as shown below:

{ "username": "user1", "password": "mysecret2" }

To override the default secret encryption setting for the module, use the following:

cred.secret.encrypt = true | false

To specify the username in plain and retrieve the password from the vault, use the following:

cred.secret.username = username

The secret value in the vault is then just the password.

If no username is required, add the following:

cred.secret.passwordonly = true

The secret value in the vault is then just the password.

Example

oci.properties


type            = ocivault
secrets.encrypt = true
vault           = vault1
key             = rtest
compartmentid   = ocid1.compartment.oc1..aaaaaaaaeq2s...
region          = us-phoenix-1

Script Credentials Store

The properties for a script credentials store must include a script property which is the name of the script file. If the file is not absolute, it is found relative to the EDQ local configuration directory. The script defines the following functions:

Table 11-4 Script Credentials Store

Function Required Description

init(props)

No

Initialize the module. The argument is an object containing properties from the properties file.

getCredentials(props)

Yes

Retrieve credentials from a secrets store. The properties argument contains the cred.* settings from the EDQ configuration.

The properties passed to the init function include a type value set to "credentials". This allows the same script to be used as the security module and a credentials store.

Example Using the AWS Secrets Management Framework

For more information about base64 support functions, see EDQ Enhanced Encryption Overview.

aws.properties


type         = script
region       = eu-west-1
script       = awscreds.js
auth.profile = myprofile

awscreds.js


// Script credentials module sample for AWS secrets manager
// =========================================================
//
// Required properties:
//
//      region          AWS region
//
// Configure proxy server with https.proxyHost and https.proxyPort properties
 
addLibrary("http")
addLibrary("logging")
 
var secclient
var encoder = BASE64.getEncoder()
var decoder = BASE64.getDecoder()
 
function init(props) {
 
  // Authentication properties
 
  var aprops = Object.keys(props).filter(k => k.startsWith("auth.")).reduce((o, k) => (o[k.substring(5)] = props[k], o), {})
 
  // Properties
 
  var region   = props.region
 
  if (!region) {
    throw "awscreds: region not specified"
  }
 
  // URL and client for secret queries
 
  securl    = "https://secretsmanager." + region + ".amazonaws.com"
  secclient = HTTP.builder().withAWSAuthentication(aprops).build();
}
 
function getCredentials(props) {
  var s = props.secret
 
  if (s) {
    var req = secclient.requestbuilder().failonerror(false).header("X-Amz-Target", "secretsmanager.GetSecretValue").build()
    var res = req.post(securl, JSON.stringify({SecretId: s}), "application/x-amz-json-1.1")
 
    // Error 400 can mean secret not found, so don't report it
 
    if (res.code != 200) {
      if (res.code != 400) {
        logger.log(Level.WARNING, "AWS secrets manager call failed with code {0}", res.code)
      }
    } else {
      var obj = JSON.parse(res.data)
      var str = obj.SecretString || decoder.decodeToString(obj.SecretBinary)
 
      if (props["secret.passwordonly"] == "true" || props["secret.username"]) {
        return {username: props["secret.username"] || "", password: str}
      } else {
        var val = JSON.parse(str)
        return {username: val.username, password: val.password}
      }
    }
  }
 
  return null
}

Parent topic: Configuring EDQ Encryption

Creating Custom Java Encryption and Credentials Store Modules

The 12.2.1.4.4 and 14.1.2.0.0 versions of EDQ supports pluggable modules for encryption and credentials lookup. The following section describes how to implement modules using custom Java classes which is necessary if it is not possible to write a script to interact with some security store because native code is required.

Custom modules are defined as shown below:


type  = custom
class = java class name
Perform the following steps to create a custom module:
  1. Create a Java class which implements the required interface.
  2. Compile the class using installdir/buildjars/customsecuritymodules.jar in the classpath.
  3. Package the classes into a jar file in localhome/security/jars.

Brief javadocs for the interfaces that are included in the docs subfolder in installdir/buildjars/customsecuritymodules.jar.

Custom Encryption Module

A custom encryption module must implement the interface oracle.edq.security.module.custom.interfaces.CustomEncryptionModule as shown below:

CustomEncryptionModule


/*
 * Copyright (C) 2023, Oracle and/or its affiliates. All rights reserved.
 */
package oracle.edq.security.module.custom.interfaces;
 
import java.nio.file.Path;
import java.util.Properties;
 
/**
 * Interface implemented by custom encryption modules.
 */
 
public interface CustomEncryptionModule {
 
  /**
   * Initialize the encryption module.
   *
   * @param localconfig The local configuration area.  This can be used to locate or store extra files
   * required by the module.
   * @param props The module properties
   * @param persist If <code>true</code> the module data can be persisted for future use.
   * This flag will be <code>false</code> if the module is being initalized for a "dryrun" encryption migration.
   * In this case no permanent changes should be made in the file system.
   *
   * @throws Exception If initialization failed.
   */
 
  void initEncryption(Path localconfig, Properties props, boolean persist) throws Exception;
 
  /**
   * Encrypt some data.
   *
   * @param in The plain text
   *
   * @return The cipher text
   *
   * @throws Exception If encryption failed
   */
 
  byte [] encrypt(byte [] in) throws Exception;
 
  /**
   * Decrypt some data.
   *
   * @param in The cipher text
   *
   * @return The plain text
   *
   * @throws Exception If decryption failed
   */
 
  byte [] decrypt(byte [] in) throws Exception;
}

If the encryption module implements oracle.edq.security.module.custom.interfaces.CustomCredentialsModule also then it acts as a credentials store. In this case the initCredentials method is not called.

Custom Credentials Store

A custom encryption module must implement the interface oracle.edq.security.module.custom.interfaces.CustomCredentialsModule as shown below:

CustomCredentialsModule


/*
 * Copyright (C) 2023, Oracle and/or its affiliates. All rights reserved.
 */
package oracle.edq.security.module.custom.interfaces;
 
import java.nio.file.Path;
import java.util.Properties;
 
/**
 * Interface implemented by custom credentials modules.
 */
 
public interface CustomCredentialsModule {
 
  /**
   * Get the prefix used to extract properties recognized by the module.  The default value is "cred".
   *
   * @return The prefix
   */
 
  default String credsPrefix() {
    return "cred";
  }
 
  /**
   * Initialize the credentials provider for credentials-only use.
   *
   * @param localconfig The local configuration area
   * @param props The module properties
   *
   * @throws Exception If initialization failed.
   */
 
  void initCredentials(Path localconfig, Properties props) throws Exception;
 
  /**
   * Attempt to get credentials.
   *
   * @param props The filtered property set.
   *
   * @return The credentials, or <code>null</code> if not found
   *
   * @throws Exception If an error ocurred
   */
 
  Credentials getCredentials(Properties props) throws Exception;
 
  /**
   * Credentials result class.
   */
 
  final class Credentials {
 
    private String  username;
    private String  password;
 
    /**
     * Constructor.
     *
     * @param username The user name
     * @param password The password
     */
 
    public Credentials(String username, String password) {
      this.username = username;
      this.password = password;
    }
 
    /**
     * Get the user name.
     *
     * @return The user name
     */
 
    public String getUsername() {
      return username;
    }
 
    /**
     * Get the password.
     *
     * @return The password
     */
 
    public String getPassword() {
      return password;
    }
  }
}

Parent topic: Configuring EDQ Encryption