15 Managing Online and Offline Secrets

You can store and manage credential files in Oracle Key Vault as opaque objects that an endpoint can retrieve when needed.

15.1 Uploading and Downloading Credential Files

The okvutil upload and okvutil download commands can upload and download credential files.

15.1.1 About Uploading and Downloading Credential Files

You use the okvutil utility to upload and download credential files.

Credential files are uploaded and stored as opaque objects in Oracle Key Vault, which means that Oracle Key Vault does not parse the contents of the file like an Oracle wallet or Java keystore. The upload process does not alter the credential file.

Examples of opaque objects are as follows:

  • Files that contain X.509 certificates

  • Kerberos keytabs

  • Files containing passwords

  • Files containing SSH keys

Uploading these credential files provides a central, secure location for long-term retention. After you have uploaded a credential file, you can download it in the same server location or share it with other trusted servers. Oracle Key Vault supports credential files up to 128 KB in size.

You can place the credential file anywhere in your server infrastructure (which includes database servers and application servers) that is accessible by an Oracle Key Vault endpoint.

15.1.2 Uploading a Credential File

The okvutil upload command can upload credential files.

  1. Ensure that the server that contains the credential file has been enrolled and provisioned as an Oracle Key Vault endpoint.
  2. Ensure that access control has been configured for the endpoint.

    If you are uploading the credential file to a virtual wallet, then ensure that the endpoint has read, modify, and manage wallet access to the wallet.

  3. Run the okvutil upload command.

    For example:

    # okvutil upload -l "/etc/oracle/app/creds/hr.keytab" -t kerberos -g HRWallet -d "Kerberos keytab file for HR group, 06_11_14"
    Enter Oracle Key Vault endpoint password: Key_Vault_endpoint_password
    

    In this example:

    • -l is the directory path to the hr.keytab credential file, which is being uploaded. Enclose the directory location in double quotation marks.

    • -t specifies the type of credential file, which in this example is a Kerberos keytab file. In addition to KERBEROS, other types that you can specify are as follows:

      • SSH for an SSH key file

      • OTHER for other files that store secrets, such as uploaded or downloaded files

    • -g adds the credential file to the virtual wallet HRWallet, which must already exist. This parameter enables you to upload the credential to a wallet that is specifically for the HR application users' needs, rather than to the default virtual wallet. In this example, HRWallet is the Oracle Key Vault virtual wallet to which access control was configured in Step 2.

    • -d is an optional description. As a best practice, include a brief description of what the credential file is used for and the date you performed the upload. This information helps for future reference and tracking of the credential file. You can modify this description later on in the Oracle Key Vault management console if necessary.

15.1.3 Downloading a Credential File

The okvutil download command can download credential files.

  1. Find the unique ID of the credential file that you must download, by using one of the following methods:
    • Oracle Key Vault management console: Log in as user who has the necessary access to the virtual wallet. In the Oracle Key Vault management console, from the Keys & Wallets tab, select Keys, Secrets & Objects to find the uploaded files. Note the unique ID of the uploaded file that you want to download. Credential files are listed as opaque objects.

    • okvutil list command: Run the okvutil list command from an endpoint that has access to the credential file or a virtual wallet that contains the credential file. Locate the unique ID of the credential file that you must download based on the description that you provided when you uploaded the file.

  2. From the command line, run the okvutil download command to download the credential file.

    For example:

    # okvutil download -l "/etc/oracle/app/newcreds/hr.keytab" -t kerberos -i 6ba7b810-9dad-11d1-80b4-00c04fd430c8
    Enter Oracle Key Vault endpoint password: Key_Vault_endpoint_password
    

    In this example:

    • -l is the directory to which you want to download the uploaded credential.

    • -t specifies the type of credential file, which in this example is a Kerberos keytab file. In addition to KERBEROS, other types that you can specify are as follows:

      • SSH for an SSH key file

      • OTHER for other files that store secrets, such as uploaded or downloaded files

    • -i is the unique ID of the credential file.

Related Topics

15.1.4 Guidelines for Uploading and Downloading Credential Files

Oracle provides recommendations for when you upload and download credential files.

  • After you complete the upload, upload the credential file again the next time it is changed. Otherwise, the uploaded (and subsequent downloaded version) file will be outdated. Periodically, you should compare the last modification date of the credential file with the timestamp of the uploaded version.

  • Use care if you use the okvutil upload and okvutil download commands, which provide an overwrite (-o) option. This option overwrites the uploaded credential file. You may want to create backups of the credential files before beginning the upload and download processes.

  • You can share one credential file among multiple server endpoints. Add the credential file as an opaque object to a virtual wallet using the -g option of the okvutil upload command. Grant access of that virtual wallet to all the server endpoints. Optionally, define an endpoint group and then make all the server endpoints members of that endpoint group. Grant this endpoint group access to that virtual wallet. Afterward, all the members of the group will have access to that wallet.

Related Topics

15.2 Managing Secrets and Credentials for SQL*Plus

To manage passwords in SQL*Plus scripts for a large number of Oracle databases, you can upload the passwords to Oracle Key Vault.

Many large sites use automated scripts to log in to Oracle databases to perform regular maintenance activities such as Oracle Recovery Manager (Oracle RMAN) backups, batch loading into an Oracle Database data warehouse, and similar tasks. Usually these scripts must connect as a highly privileged user. Logging in as a highly privileged user means that the scripts must have access to the user's password. Hard-coding a clear-text password into a script is, of course, very poor security. And if the user's password changes, then none of the scripts can work. One solution to avoiding the use of clear-text passwords in scripts is to put the passwords into an auto-open wallet (called the secure external password store) and then instruct the client application to retrieve the password from there. However, this idea only works if you only have a few databases. But if you have hundreds of databases, then the passwords are difficult to manage and require an update for each secure external password store.

If you have a large number of Oracle databases, you can store passwords centrally and then retrieve them securely for database connections. This capability has the following advantages:

  • Eliminates clear-text passwords from your maintenance scripts
  • Simplifies password changes
  • Does not require extra entries in the sqlnet.ora file
  • Makes the secure external password store obsolete

15.3 Managing Secrets and Credentials for SSH

You can perform public key authentication with private keys that are protected in Oracle Key Vault.

In many IT departments, public key authentication is the default approach to securely log into remote servers without having an administrator having to remember the password, for both on-premise or cloud-based hosts. If you are using Oracle Cloud Infrastructure (Oracle OCI), then the password of the user opc is unknown, so public key authentication is the only way to log into an OCI compute instance. This method is convenient, but not without risk. If the private key is lost, then remotely logging in is no longer possible. In addition, the process of provisioning a new public key is time consuming. Another problem is that because the private key uniquely identifies its owner, if it is stolen, copied, or compromised, then an intruder can easily cause a great deal of trouble, with the evidence pointing at the owner, not the intruder.

As a solution to these problems, consider uploading the SSH private keys into Oracle Key Vault by using the okvutil upload command. The benefits are as follows:

  • Because the SSH keys are not on your host computer, they cannot be copied or stolen, and furthermore, they are safe from disk or file corruptions.
  • The SSH keys are included in Oracle Key Vault backup operations, so losing them is impossible.
  • Oracle Key Vault provides continuous and fault-tolerant availability by allowing up to 16 Oracle Key Vault servers to connect to one multi-master cluster. Read-write pairs guarantee that highly sensitive information (such as encryption keys or passwords) are replicated to at least one or more Oracle Key Vault servers.

Related Topics

15.4 Integrating Oracle Key Vault with SSH Public Key Authentication

You can use Oracle Key Vault to store your public and private keys for SSH.

The Key Administrator or users with appropriate access can centrally manage the public or private key pairs for SSH public key authentication in Oracle Key Vault. When required, the key pairs can be rotated or revoked. As the keys are centrally managed, an authorized user can login to a remote machine without the private key existing on the local disk.

15.4.1 Step 1: Creating and Uploading the Key Pair

Use an Oracle Key Vault endpoint to create and upload the key pair.

OpenSSH must be at least at version 7.2p1 for endpoints that do not require a password and at version 8.1p1 for endpoints that do require a password.

  1. Create a key pair, both in PKCS#8 format. It must be an RSA key pair, with a key size of 1024, 2048, or 4096.

    Note:

    Do not enter a passphrase when prompted. Oracle Key Vault does not currently support using encrypted private keys for SSH.
    This generates the following files:
    • key.pem: The private key in PKCS#8 format, should start with -----BEGIN PRIVATE KEY-----.
    • key.pem.pub: The public key in RFC4716 format.
    • public.pem: The public key in PKCS#8 format, should start with -----BEGIN PUBLIC KEY-----.
    
            ssh-keygen -q -t rsa -m PKCS8 -f key.pem -b <key size>
            ssh-keygen -e -f key.pem.pub -mPKCS8 > public.pem
  2. Add the contents of key.pem.pub to the remote host's authorized_keys file.

    Note:

    While creating the authorized_keys file or its parent directory, if either does not exist, ensure they are created with the correct permissions.
  3. To test public key authentication at this stage (without using the Oracle Key Vault PKCS#11 library), use the following command: ssh -i ./key.pem <username>@<remote host>.
  4. Upload the private key to Oracle Key Vault using the RESTful services utility. ./bin/okv managed-object private-key register --object ./key.pem --algorithm RSA --length <key size> --activation-date now
    Make sure the <key size> should be same as chosen previously.
  5. Upload the public key to Oracle Key Vault using the RESTful services utility. ./bin/okv managed-object public-key register --object ./public.pem --algorithm RSA --length <key size> --private-key-uuid <private key UUID> --activation-date now
    Make sure that the private key UUID is the same as the UUID returned by the previous command.
  6. Delete the public and private keys from the local machine.
    You can retrieve the private and public key ;later in the PKCS#8 format using the RESTful services utility,
    rm key.pem key.pem.pub public.pem
    .

15.4.2 Step 2: Using an Endpoint to SSH a Remote Host

You can use an endpoint to SSH to a remote host.

The endpoints that SSH to the remote host must have access to both the public and private keys. Endpoints can be given access using the following:
  • being the endpoint that created the key pair
  • being directly given access to the wallet in which the keys are located
  • being added to an endpoint group that has been given access to the wallet in which the keys are located
Ensure that the following environment variable is set correctly:
  • OKV_HOME must be set to the directory in which the endpoint software was deployed. As a result,
    $OKV_HOME/lib/liborapkcs.so 
    should point to the location of the OKV PKCS#11 library.
  1. SSH to the remote host using the PKCS#11 library with the following command: .
    ssh -I $OKV_HOME/lib/liborapkcs.so <username>@<remote host>
  2. Enter the endpoint password, when prompted for a PIN, or enter NULL, if the endpoint does not have a password.

    Note:

    Oracle recommends that you use password protected endpoint. When you use auto-login endpoint, the protection of the SSH keys of the user is dependent on the file permissions only.

15.4.3 Step 3: Incorporating an ssh-agent

You can incorporate an ssh-agent to prevent specifying the PKCS#11 library.

You can use ssh-agent to prevent specifying the PKCS#11 library location and endpoint password each time you want to SSH into a remote host.

This requires having setup the key pair as described in,Step 2: Using an Endpoint to SSH a Remote Host.

  1. Identify the directory in which the
    liborapkcs.so
    is located.
  2. Setup ssh-agent, whitelisting the directory from the previous step:
    eval `ssh-agent -P "<full path to directory containing liborapkcs.so>/*"`
  3. Add PKCS#11 library provided keys to the agent via ssh-add.
  4. Enter the endpoint password, when prompted for the PIN.
  5. If the endpoint does not have a password, enter "NULL".
    ssh-add -s <full path to liborapkcs.so>
  6. Test the setup using ssh command without any option.
    ssh <username>@<remote host>

15.5 Centrally Managing Passwords in Oracle Key Vault

You can centrally manage passwords in Oracle Key Vault by using external keystores or adding them to shared virtual wallets as secrets.

15.5.1 About Centrally Managing Passwords in Oracle Key Vault

You can store passwords centrally in Oracle Key Vault and retrieve these passwords securely (for example, when logging into an Oracle database).

Storing passwords centrally in Oracle Key Vault has the following benefits:

  • It eliminates the need for clear text passwords in your maintenance scripts.
  • It eliminates the need for the secure external password store.
  • It simplifies password changes.
  • It does not require changes to the sqlnet.ora file.

Database administrators often use automated scripts to perform regular maintenance such as Oracle Recovery Manager (Oracle RMAN) backups, loading data into Oracle Data Warehouse, or refreshing materialized views. These scripts log in to the Oracle Database with the passwords of highly privileged users, which unfortunately is a poor security practice in that it entails hard-coded clear-text passwords in the script. And of course if the password changes, then all the scripts must be changed as well. One way to remove the need for clear-text passwords is to put the passwords into an auto-open wallet (called the secure external password store) and instruct the client to retrieve the password from there. This practice works well for a few databases, but if you have hundreds of databases, it is difficult to manage. Furthermore, password changes require you to update every secure external password store. By storing passwords in a central location, you eliminate this problem.

15.5.2 Creating and Sharing Centrally Managed Passwords

To create and share centrally managed passwords (external keystore passwords) for large database deployments, you first must use an Oracle Key Vault client and RESTful services utility commands.

  1. Ensure that you have installed the Oracle Key Vault RESTful services utility command-line interface.
    If you do not have the RESTful services utility command-line interface installed, then you can download it from the Oracle Key Vault management console as follows:
    curl -O -k https://Oracle_Key_Vault_IP_address:5695/okvrestclipackage.zip
  2. Install an Oracle Key Vault client that does not use a password.
    If you already have an Oracle Key Vault client installed (for example, for TDE key management), then do not use this endpoint. Instead, install an additional endpoint for secrets management. You must use an Oracle Key Vault client that is not password protected.

    Note:

    An OpenSSH must be at least at version 7.2p1 for endpoints that do not require a password and at version 8.1p1 for endpoints that do require a password.

    To install the Oracle Key Vault client without a password, execute the following RESTful services utility commands:
    okv admin endpoint create --endpoint endpoint_name 
    --description "Secrets management for endpoint_name" --type ORACLE_NON_DB 
    --platform platform_OS
    
    okv admin endpoint provision --endpoint endpoint_name 
    --location installation_directory --auto-login TRUE

    For this use case, executing the root script is not needed.

  3. Upload the passwords for your database accounts to Oracle Key Vault.
    1. Create text files that contain the names and passwords for each account that you want to upload. For example, suppose you have two maintenance scripts that log into the database. The first script uses an RMAN account nightly_backup, which logs into the PDB as Oracle Recovery Manager (Oracle RMAN) with the SYSBACKUP administrative privilege. The second script uses the refresh_dwh account, which refreshes a data warehouse by connecting to a PDB.
      The following shows the password text files for each account.
      $ more backup_pwd
      hV3t0ksxoSQIEe4VoF237o7t
      $ more refresh_pwd
      NfKmXHAi65kxqVqx2yiOd49s
    2. Register user names and passwords in Oracle Key Vault.
      First, the nightly_backup account is uploaded, followed by the refresh_dwh account. Oracle Key Vault returns the universally unique ID (UUID) of each of these managed objects. First, the registration:
      • Register the secret for the first managed object.
        $ PWD_RMAN=$(okv managed-object secret register --output_format "text" --object "./backup_pwd" --custom-attribute '[{"name": "x-NAME", "value": "NIGHTLY_BACKUP"}, {"name": "x-CONNECT-STRING", "value": "'${TWO_TASK}'"}]' --activation-date "now")
        echo $PWD_RMAN 
        E3CFC37C-875E-4F7E-BF2A-B37EAA86EA31
        Confirm with the command.
        okv managed-object attribute get-all --uuid $PWD_RMAN
      • Register the secret for the second managed object.
        PWD_REFRESH=$(okv managed-object secret register --output_format "text" --object "./refresh_pwd" --custom-attribute '[{"name": "x-NAME", "value" : "REFRESH_DWH"}, {"name": "x-CONNECT-STRING", "value": "'${TWO_TASK}'"}]' --activation-date "now")
        echo $PWD_REFRESH
        FDAFA443-8CD2-4FF7-BF27-9EB4AEC77611

        Edit the options and attributes in the generated JSON input file, reg_refresh.json:

        {
          "service" : {
            "category" : "managed-object",
            "resource" : "secret",
            "action" : "register",
            "options" : {
              "object" : "./refresh_pwd",
              "type" : "PASSWORD",
              "mask" : "ENCRYPT",
              "attributes" : {
                "name" : "REFRESH_DWH"
              }
            }
          }
        }

        Confirm with the command.

        okv managed-object attribute get-all --uuid $PWD_REFRESH
  4. On Linux, securely delete the file that contain passwords.
    shred -xzu ./backup_pwd ./refresh_pwd
After you complete this configuration, you can create a script that retrieves the password for these users from Oracle Key Vault and inserts the password into the SQL*Plus commands.

15.5.3 Example: Script for Using External Keystore Passwords in SQL*Plus Operations

You can create a script that retrieves the UUID of uploaded passwords by the user name and then inserts the password into the SQL*Plus command.

The benefit of this script is that no matter how many administrative or maintenance scripts log into that database, the script stays the same. For each new user account, you only upload the password and attributes file for the user, and then activate the secret.

For example, a script called log-me-in.sh can be as follows:

#!/bin/bash
set +x
# ************************************************
# ** Read user name and connect string from ******
# ** command line into variables *****************
# ************************************************

export user="${1}"
export TWO_TASK="${2}"
export PRIV="${3}"

KMIP_ID=$(okv managed-object object locate --output_format text --custom-attribute '[{"name": "x-NAME", "value" : "'${user}'"}, {"name": "x-CONNECT-STRING", "value": "'${TWO_TASK}'"}]')
pwd='"'$(okv managed-object secret get --output_format text --uuid ${KMIP_ID})'"'

if [ "${PRIV}" == 'AS SYSBACKUP' ]; then
  rman target ''"'${user}/${pwd}@${TWO_TASK} ${PRIV}'"''
else
  sqlplus ${user}/${pwd}
fi

You can make this file immutable by applying extended attributes. For example, in Linux as the root user, use the following chattr command:

# chattr +i log-me-in.sh

To execute this script, use the following syntax:

$ ./log-me-in.sh user_name hostname:port/service 'AS privilege'

For example to log in the nightly_backup user:

$ ./log-me-in.sh nightly_backup sales19c.us.example.com:1521/finpdb.us.example.com 'AS SYSBACKUP'

Output similar to the following appears:

Recovery Manager: Release 19.0.0.0.0 - Production on Fri Jun 26 12:22:04 2020
Version 19.7.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: sales19c:finpdb (DBID=3200396863)

RMAN> 

The following example command logs in as the refresh_dwh user:

$ ./log-me-in.sh refresh_dwh sales19c.us.example.com:1521/finpdb.us.example.com 

SQL> SHOW USER
REFRESH_DWH

If the owner of the target schema allows the refresh_dwh account user to log in, then the schema owner must grant the CONNECT THROUGH privilege to refresh_dwh, as follows:

  1. In SQL*Plus, execute the ALTER USER statement to create the proxy. For example:
    ALTER USER HR GRANT CONNECT THROUGH refresh_dwh;
  2. In the script, change the last line (sqlplus "${user}"/"${pwd}") to include the proxy. For example, to include HR:
    sqlplus "${user}"[HR]/"${pwd}"

15.5.4 Sharing Secrets with Other Databases

Sharing information in Oracle Key Vault requires a virtual wallet in Oracle Key Vault, and multiple databases connecting into their own endpoints that have access to the shared wallet.

  1. Log in to the server from which you want to share the secret (the source server).
  2. Create a virtual wallet and make it the default wallet of the endpoint.
    For example, to make a default wallet for an endpoint called secrets_db_name:
    okv manage-access wallet create --wallet shared_secrets
    okv manage-access wallet set-default --wallet shared_secrets --endpoint secrets_db_name
  3. Add the secret (password) to the shared virtual wallet.
    okv managed-object wallet add-member --uuid 994CB3E5-C5B3-4F75-BFED-CB41AE15D0B1 --wallet shared_secrets
    994CB3E5-C5B3-4F75-BFED-CB41AE15D0B1 is the UUID for the secret. In the event that you have forgotten the UUID, you can retrieve it. For example, to retrieve the UUID for nightly_backup:
    $ okv managed-object object locate --name nightly_backup
  4. Log in to the server that will use the shared secret (the destination server).
  5. Create an endpoint, and then make the shared_secrets wallet the default wallet of that new endpoint:
    okv admin endpoint create --endpoint secrets_db_name --description "Endpoint for Secrets Management in db_name" --type ORACLE_NON_DB --platform LINUX64
    okv manage-access wallet set-default --wallet shared_secrets --endpoint secrets_db_name
    okv admin endpoint provision --endpoint secrets_db_name --location /directory/for/new/endpoint/ --auto-login TRUE
  6. Securely copy the script that you created (for example, log-me-in.sh) from its source server to the destination server that will use the shared secret.
    $ scp log-me-in.sh destination_server:/remote_path
  7. In the destination server, use the following syntax to execute the script:
    $ ./log-me-in.sh user_name hostname:port/service 'AS privilege'

    For example:

    $ ./log-me-in.sh nightly_backup 192.0.2.1:1521/hr_pdb 'AS SYSBACKUP'

    Output similar to the following should appear:

    Recovery Manager: Release 19.0.0.0.0 - Production on Fri Jun 26 13:10:49 2020
    Version 19.7.0.0.0
    
    Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
    
    connected to target database: hr_db:hr_pdb (DBID=1932795327)
    
    RMAN>

15.5.5 Changing Passwords for a Large Database Deployment

For better security, on a regular basis, you should change passwords that are used by both humans and automated processes.

To change the passwords that are stored in Oracle Key Vault, you revoke and destroy the old password, and then upload and activate the new password. Finally, you must change the password in SQL*Plus.
  1. Log in to a server that has access to the stored password.
  2. Retrieve the UUID of this password.
    $ okv managed-object object locate --name nightly_backup

    UUID output similar to the following appears:

    994CB3E5-C5B3-4F75-BFED-CB41AE15D0B1
  3. Use this UUID to revoke this password.
    $ okv managed-object object revoke --code CESSATION_OF_OPERATION --uuid 994CB3E5-C5B3-4F75-BFED-CB41AE15D0B1

    You must provide a code for revoking the password. Possible values are as follows:

    • UNSPECIFIED
    • KEY_COMPROMISE
    • CA_COMPROMISE
    • AFFILIATION_CHANGED
    • SUPERSEDED
    • CESSATION_OF_OPERATION
    • PRIVILEGE_WITHDRAWN
  4. Destroy this password.
    $ okv managed-object object destroy --uuid 994CB3E5-C5B3-4F75-BFED-CB41AE15D0B1
    
  5. In the Oracle Key Vault management console, delete the secret.
  6. Create a text file to contain the new password, similar to the original password text file that was used to configure the original external keystore password.
    $ more backup_pwd
    cn0KpOnY9vNec2sLVHFnJwR6
  7. Upload the new password and attributes, and then activate the new secret.
    1. Generate JSON output.
      okv managed-object secret register --generate-json-input > reg_secret.json
    2. Check the generated output.
      $ more reg_secret.json
      
      {
        "service" : {
          "category" : "managed-object",
          "resource" : "secret",
          "action" : "register",
          "options" : {
            "object" : "./backup_pwd",
            "type" : "PASSWORD",
            "mask" : "ENCRYPT",
            "attributes" : {
              "name" : "NIGHTLY_BACKUP"
            }
          }
        }
      }
    3. Perform the registration.
      okv managed-object secret register --from-json ./reg_secret.json
    4. Activate the secret.
      $ okv managed-object object activate --uuid 0739649D-3058-4F34-BF84-50B2BD652C2D
      
  8. On Linux, securely delete the file that contain passwords.
    shred -xz backup_pwd
    rm backup_pwd
  9. In each Oracle database that shares the secret, log in to SQL*Plus and use the password command to change the password.
    For example:
    SYS> password nightly_backup
    Changing password for nightly_backup
    New password: cn0KpOnY9vNec2sLVHFnJwR6
    Retype new password: cn0KpOnY9vNec2sLVHFnJwR6
    Password changed