19 Managing Online and Offline Secrets

You can store and manage credential files in Oracle Key Vault as opaque objects that an endpoint can retrieve when needed.

• Uploading and Downloading Credential Files
  The okvutil upload and okvutil download commands can upload and download credential files.
• Managing Secrets and Credentials for SQL*Plus
  To manage passwords in SQL*Plus scripts for a large number of Oracle databases, you can upload the passwords to Oracle Key Vault.
• Managing Secrets and Credentials for SSH
  You can perform public key authentication with private keys that are protected in Oracle Key Vault.
• Managing Passwords in Oracle Key Vault in Scripts for Large Database Deployments
  You can create scripts to manage database password changes in large database deployments.

19.1 Uploading and Downloading Credential Files

The okvutil upload and okvutil download commands can upload and download credential files.

• About Uploading and Downloading Credential Files
  You use the okvutil utility to upload and download credential files.
• Uploading a Credential File
  The okvutil upload command can upload credential files.
• Downloading a Credential File
  The okvutil download command can download credential files.
• Guidelines for Uploading and Downloading Credential Files
  Oracle provides recommendations for when you upload and download credential files.

19.1.1 About Uploading and Downloading Credential Files

You use the okvutil utility to upload and download credential files.

Credential files are uploaded and stored as opaque objects in Oracle Key Vault, which means that Oracle Key Vault does not parse the contents of the file like an Oracle wallet or Java keystore. The upload process does not alter the credential file.

Examples of opaque objects are as follows:

• Files that contain X.509 certificates

• Kerberos keytabs

• Files containing passwords

• Files containing SSH keys

Uploading these credential files provides a central, secure location for long-term retention. After you have uploaded a credential file, you can download it in the same server location or share it with other trusted servers. Oracle Key Vault supports credential files up to 128 KB in size.

You can place the credential file anywhere in your server infrastructure (which includes database servers and application servers) that is accessible by an Oracle Key Vault endpoint.

19.1.2 Uploading a Credential File

The okvutil upload command can upload credential files.

1. Ensure that the server that contains the credential file has been enrolled and provisioned as an Oracle Key Vault endpoint.
2. Ensure that access control has been configured for the endpoint.

   If you are uploading the credential file to a virtual wallet, then ensure that the endpoint has read, modify, and manage wallet access to the wallet.

3. Run the okvutil upload command.

   For example:

   # okvutil upload -l "/etc/oracle/app/creds/hr.keytab" -t kerberos -g HRWallet -d "Kerberos keytab file for HR group, 06_11_14"
   Enter Oracle Key Vault endpoint password: Key_Vault_endpoint_password
   

   In this example:

   • -l is the directory path to the hr.keytab credential file, which is being uploaded. Enclose the directory location in double quotation marks.

   • -t specifies the type of credential file, which in this example is a Kerberos keytab file. In addition to KERBEROS, other types that you can specify are as follows:

     • SSH for an SSH key file

     • OTHER for other files that store secrets, such as uploaded or downloaded files

   • -g adds the credential file to the HRWallet group, which must already exist. This parameter enables you to upload the credential to a wallet that is specifically for the HR application users' needs, rather than to the default virtual wallet. In this example, HRWallet is the Oracle Key Vault virtual wallet to which access control was configured in Step 2.

   • -d is an optional description. As a best practice, include a brief description of what the credential file is used for and the date you performed the upload. This information helps for future reference and tracking of the credential file. You can modify this description later on in the Oracle Key Vault management console if necessary.

19.1.3 Downloading a Credential File

The okvutil download command can download credential files.

1. Find the unique ID of the credential file that you must download, by using one of the following methods:

   • Oracle Key Vault management console: Log in as an endpoint administrator or a user who has the necessary access to the virtual wallet. In the Oracle Key Vault management console, from the Keys & Wallets tab, select All Items to find the uploaded files. Note the unique ID of the uploaded file that you want to download. Credential files are listed as opaque objects.

   • okvutil list command: Run the okvutil list command from an endpoint that has access to the credential file or a virtual wallet that contains the credential file. Locate the unique ID of the credential file that you must download based on the description that you provided when you uploaded the file.

2. From the command line, run the okvutil download command to download the credential file.

   For example:

   # okvutil download -l "/etc/oracle/app/newcreds/hr.keytab" -t kerberos -i 6ba7b810-9dad-11d1-80b4-00c04fd430c8
   Enter Oracle Key Vault endpoint password: Key_Vault_endpoint_password
   

   In this example:

   • -l is the directory to which you want to download the uploaded credential.

   • -t specifies the type of credential file, which in this example is a Kerberos keytab file. In addition to KERBEROS, other types that you can specify are as follows:

     • SSH for an SSH key file

     • OTHER for other files that store secrets, such as uploaded or downloaded files

   • -i is the unique ID of the credential file.

19.1.4 Guidelines for Uploading and Downloading Credential Files

Oracle provides recommendations for when you upload and download credential files.

• After you complete the upload, upload the credential file again the next time it is changed. Otherwise, the uploaded (and subsequent downloaded version) file will be outdated. Periodically, you should compare the last modification date of the credential file with the timestamp of the uploaded version.

• Use care if you use the okvutil upload and okvutil download commands, which provide an overwrite (-o) option. This option overwrites the uploaded credential file. You may want to create backups of the credential files before beginning the upload and download processes.

• You can share one credential file among multiple server endpoints. Add the opaque object to a virtual wallet and then ensure that all of the endpoints have access to that virtual wallet. Optionally, define an endpoint group and then make all the server endpoints members of that group. Upload the credential file that you would like to share using this common wallet into Oracle Key Vault as a group, using the -g option of the okvutil upload command. Define a wallet and attach it to the endpoint group. Afterward, all the members of the group will have access to that wallet.

19.2 Managing Secrets and Credentials for SQL*Plus

To manage passwords in SQL*Plus scripts for a large number of Oracle databases, you can upload the passwords to Oracle Key Vault.

Many large sites use automated scripts to log in to Oracle databases to perform regular maintenance activities such as Oracle Recovery Manager (Oracle RMAN) backups, batch loading into an Oracle Database data warehouse, and similar tasks. Usually these scripts must connect as a highly privileged user. Logging in as a highly privileged user means that the scripts must have access to the user's password. Hard-coding a clear-text password into a script is, of course, very poor security. And if the user's password changes, then none of the scripts can work. One solution to avoiding the use of clear-text passwords in scripts is to put the passwords into an auto-open wallet (called the secure external password store) and then instruct the client application to retrieve the password from there. However, this idea only works if you only have a few databases. But if you have hundreds of databases, then the passwords are difficult to manage and require an update for each secure external password store.

If you have a large number of Oracle databases, you can centrally store passwords centrally and then retrieve them securely for database connections. This capability has the following advantages:

• Eliminates clear-text passwords from your maintenance scripts
• Simplifies password changes
• Does not require extra entries in the sqlnet.ora file
• Makes the secure external password store obsolete

19.3 Managing Secrets and Credentials for SSH

You can perform public key authentication with private keys that are protected in Oracle Key Vault.

In many IT departments, public key authentication is the default approach to securely log into remote servers without having an administrator having to remember the password, for both on-premise or cloud-based hosts. If you are using Oracle Cloud Infrastructure (Oracle OCI), then the password of the user opc is unknown, so public key authentication is the only way to log into an OCI compute instance. This method is convenient, but not without risk. If the private key is lost, then remotely logging in is no longer possible. In addition, the process of provisioning a new public key is time consuming. Another problem is that because the private key uniquely identifies its owner, if it is stolen, copied, or compromised, then an intruder can easily cause a great deal of trouble, with the evidence pointing at the owner, not the intruder.

As a solution to these problems, consider uploading the SSH private keys into Oracle Key Vault by using the okvutil upload command. The benefits are as follows:

• Because the SSH keys are not on your host computer, they cannot be copied or stolen, and furthermore, they are safe from disk or file corruptions.
• The SSH keys are included in Oracle Key Vault backup operations, so losing them is impossible.
• Oracle Key Vault provides continuous and fault-tolerant availability by allowing up to 16 Oracle Key Vault servers to connect to one multi-master cluster. Read-write pairs guarantee that highly sensitive information (such as encryption keys or passwords) are replicated to at least one or more Oracle Key Vault servers.

19.4 Managing Passwords in Oracle Key Vault in Scripts for Large Database Deployments

You can create scripts to manage database password changes in large database deployments.

• About Managing Passwords in Oracle Key Vault in Scripts for Large Database Deployments
  You can store external keystore passwords centrally and retrieve these passwords securely for use in a large deployment of Oracle Database connections.
• Configuring the External Keystore Password Upload
  To configure the external keystore password upload for large database deployments, you first must use an Oracle Key Vault client and RESTful APIs.
• Example: Script for Using External Keystore Passwords in SQL*Plus Operations
  You can create a script that uses retrieves the UUID of uploaded users by the user name and then inserts the password into the SQL*Plus command.
• Sharing Secrets with Other Databases
  Sharing information in Oracle Key Vault requires a virtual wallet in Oracle Key Vault, and multiple databases connecting into their own endpoints that have access to the shared wallet.
• Changing Passwords for a Large Database Deployment
  For better security, on a regular basis, you should change passwords that are used by both humans and automated processes.

19.4.1 About Managing Passwords in Oracle Key Vault in Scripts for Large Database Deployments

You can store external keystore passwords centrally and retrieve these passwords securely for use in a large deployment of Oracle Database connections.

Storing passwords centrally has the following benefits:

• It eliminates the use of clear text passwords from your maintenance scripts
• It eliminates the need for the secure external password store.
• It simplifies password changes.
• It does not require changes to the sqlnet.ora file.

In previous releases, users automated scripts to perform regular maintenance such as Oracle Recovery Manager (Oracle RMAN) backups, loading data into Oracle Data Warehouse, or refreshing materialized views. These scripts had to log in using the passwords of highly privileged users, which unfortunately is a poor security practice in that it entails hard-coded clear-text passwords in the script. And of course if the password changes, then all the scripts must be changed as well. One way to remove the need for clear-text passwords is to put the passwords into an auto-open wallet (called the secure external password store) and instruct the client to retrieve the password from there. This practice works well for a few databases, but if you have hundreds of databases, it is difficult to manage. Furthermore, password changes require you to update every secure external password store. By storing passwords in a central location, you eliminate this problem.

19.4.2 Configuring the External Keystore Password Upload

To configure the external keystore password upload for large database deployments, you first must use an Oracle Key Vault client and RESTful APIs.

1. Ensure that you have installed the Oracle Key Vault RESTful APIs.
   If you do not have the RESTful API installed, then you can download it from Oracle Key Vault by executing the following command:

   $ curl -O -k https://Oracle_Key_Vault_IP_address:5695/okvrestservices.jar

2. Install an Oracle Key Vault client that does not use a password.
   Most likely, you already have an Oracle Key Vault client that is installed but is password protected. Do not use this one; you must use an Oracle Key Vault client that is not password protected. Disk space and licensing should not be a problem, because the Oracle Key Vault client has a small disk footprint, and unlike other key management systems, Oracle Key Vault is not licensed based on the number of clients or keys.
   1. To install the Oracle Key Vault client without a password using RESTful commands, create a script that includes the following two commands:

      create_endpoint -e SECRETS_db_name -t ORACLE_NON_DB -q LINUX64 -d "Endpoint for Secrets Management for db_name"
      provision --autologin -e SECRETS_db_name -y /home/oracle/okv

   2. Execute the Oracle Key Vault RESTful services.

      java -jar okvrestservices.jar -c config.ini

   3. Because the remaining RESTful calls into Oracle Key Vault use the kmip parameter, the root script does not need to be executed (it is only needed for TDE-enabled Oracle databases).
3. Upload the passwords for your database accounts to Oracle Key Vault.
   1. Create text files that contain the names and passwords for each account that you want to upload. For example, suppose you have two maintenance scripts that log into the database. The first script uses the nightly backup account nightly_backup, which logs into the PDB as Oracle Recovery Manager (Oracle RMAN) with the SYSBACKUP administrative privilege. The second script uses the refresh_dwh account, which refreshes a data warehouse by connecting to a PDB.
      The following shows first the password text files for each account, and second, the attribute files with the account names.

      $ more backup_pwd
      hV3t0ksxoSQIEe4VoF237o7t

      $ more refresh_pwd
      NfKmXHAi65kxqVqx2yiOd49s

      $ more backup_name
      NAME=NIGHTLY_BACKUP

      $ more refresh_name
      NAME=REFRESH_DWH

   2. Use the kmip option in the okvrestservices.jar command to upload the users' passwords and attributes into Oracle Key Vault.
      First, the nightly_backup account is uploaded, followed by the refresh_dwh account. Oracle Key Vault returns the universally unique ID (UUID) of each of these managed objects. For RESTful commands with the kmip option, you must use okvclient.ora as the configuration file (-c option).

      $ java -jar okvrestservices.jar kmip -c $OKV_HOME/conf/okvclient.ora -s reg_secret -o ./backup_pwd -f ./backup_name -t PASSWORD -m ENCRYPT
      
      9E8AB7E0-0915-4FB4-BFE1-A85696525EA4

      $ java -jar okvrestservices.jar kmip -c $OKV_HOME/conf/okvclient.ora -s reg_secret -o ./refresh_pwd -f ./refresh_name -t PASSWORD -m ENCRYPT
      
      581D9330-5458-4FD2-BFF4-C0BF602644F4

4. Carefully delete the files that contain passwords.

   $ shred -xz backup_pwd
   $ rm backup_pwd
   $ shred -xz refresh_pwd
   $ rm refresh_pwd

5. Use the returned UUID values from the previous step to activate the secrets.

   $ java -jar okvrestservices.jar kmip -c $OKV_HOME/conf/okvclient.ora -s activate -u 581D9330-5458-4FD2-BFF4-C0BF602644F4

   $ java -jar okvrestservices.jar kmip -c $OKV_HOME/conf/okvclient.ora -s activate -u 9E8AB7E0-0915-4FB4-BFE1-A85696525EA4

After you complete this configuration, you can create a script that retrieves the password for these users from Oracle Key Vault and inserts the password into the SQL*Plus commands.

19.4.3 Example: Script for Using External Keystore Passwords in SQL*Plus Operations

You can create a script that uses retrieves the UUID of uploaded users by the user name and then inserts the password into the SQL*Plus command.

The benefit of this script is that no matter how many administrative or maintenance scripts log into that database, the script stays the same. For each new user account, you only upload the password and attributes file for the user, and then activate the secret.

For example, a script called log-me-in.sh can be as follows:

#!/bin/bash
user="${1}"
export TWO_TASK="${2}"
PRIV="${3}"
export OKV_HOME=/home/oracle/okv
id=$(java -jar okvrestservices.jar kmip -c $OKV_HOME/conf/okvclient.ora -s locate --name $user)
pwd=$(java -jar okvrestservices.jar kmip -c $OKV_HOME/conf/okvclient.ora -s get_secret -u ${id:1:36})
if [ "${PRIV}" == 'AS SYSBACKUP' ]; then
  rman target ''"'${user}/${pwd}@${TWO_TASK} ${PRIV}'"''
else
  sqlplus "${user}"/"${pwd}"
fi

You can make this file immutable by applying extended attributes. For example, in Linux as the root user, use the following chattr command:

# chattr +i log-me-in.sh

To execute this script, use the following syntax:

$ ./log-me-in.sh user_name hostname:port/service 'AS privilege'

For example to log in the nightly_backup user:

$ ./log-me-in.sh nightly_backup sales19c.us.example.com:1521/finpdb.us.example.com 'AS SYSBACKUP'

Output similar to the following appears:

Recovery Manager: Release 19.0.0.0.0 - Production on Fri Jun 26 12:22:04 2020
Version 19.7.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

connected to target database: sales19c:finpdb (DBID=3200396863)

RMAN> 

The following example command logs in as the refresh_dwh user:

$ ./log-me-in.sh refresh_dwh sales19c.us.example.com:1521/finpdb.us.example.com 

SQL> SHOW USER
REFRESH_DWH

If the owner of the target schema allows the refresh_dwh account user to log in, then the schema owner must grant the CONNECT THROUGH privilege to refresh_dwh, as follows:

1. In SQL*Plus, execute the ALTER USER statement to create the proxy. For example:

   ALTER USER HR GRANT CONNECT THROUGH refresh_dwh;

2. In the script, change the last line (sqlplus "${user}"/"${pwd}") to include the proxy. For example, to include HR:

   sqlplus "${user}"[HR]/"${pwd}"

19.4.4 Sharing Secrets with Other Databases

Sharing information in Oracle Key Vault requires a virtual wallet in Oracle Key Vault, and multiple databases connecting into their own endpoints that have access to the shared wallet.

1. Log in to the server from which you want to share the secret (the source server).
2. Create a virtual wallet and make it the default wallet of the endpoint.
   For example, to make a default wallet for an endpoint called SECRETS_db_name:

   create_wallet -w shared_secrets
   set_default_wallet -e secrets_db_name -w shared_secrets

3. Add the secret (password) to the shared virtual wallet.

   $ java -jar okvrestservices.jar kmip -c ~/bin/okv/conf/okvclient.ora -s add_member -w shared_secrets -u 994CB3E5-C5B3-4F75-BFED-CB41AE15D0B1

   994CB3E5-C5B3-4F75-BFED-CB41AE15D0B1 is the UUID for the secret. In the event that you have forgotten the secret, you can retrieve it. For example, to retrieve the secret for nightly_backup:

   $ java -jar okvrestservices.jar kmip -c ~/bin/okv/conf/okvclient.ora -s locate --name nightly_backup

4. Log in to the server that will use the shared secret (the destination server).
5. Create an endpoint, and then make the shared_secrets wallet the default wallet of that new endpoint:

   create_endpoint -e secrets_db_name -t ORACLE_NON_DB -q LINUX64 -d "Endpoint for Secrets Management in db_name"
   set_default_wallet -e secrets_db_name -w shared_secrets
   provision --autologin -e secrets_db_name -y /home/oracle/okv

6. Securely copy the script that you created (for example, log-me-in.sh) for using external keystore passwords from its source server to the destination server that will use the shared secret.

   $ scp log-me-in.sh destination_server:/remote_path

7. In the destination server, use the following syntax to execute the script:

   $ ./log-me-in.sh user_name hostname:port/service 'AS privilege'

   For example:

   $ ./log-me-in.sh nightly_backup 192.0.2.1:1521/hr_pdb 'AS SYSBACKUP'

   Output similar to the following should appear:

   Recovery Manager: Release 19.0.0.0.0 - Production on Fri Jun 26 13:10:49 2020
   Version 19.7.0.0.0
   
   Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
   
   connected to target database: hr_db:hr_pdb (DBID=1932795327)
   
   RMAN>

19.4.5 Changing Passwords for a Large Database Deployment

For better security, on a regular basis, you should change passwords that are used by both humans and automated processes.

To change the passwords that are stored in Oracle Key Vault, you can use okvrestservices.jar to revoke and destroy the old password, and then to upload and activate the new password. Finally, you must change the password in SQL*Plus.

1. Log in to a server that has access to the stored password.
2. Retrieve the UUID of this password.

   $ java -jar okvrestservices.jar kmip -c $OKV_HOME/conf/okvclient.ora -s locate --name nightly_backup

   UUID output similar to the following appears:

   994CB3E5-C5B3-4F75-BFED-CB41AE15D0B1

3. Use this UUID to revoke this password.

   $ java -jar okvrestservices.jar kmip -c $OKV_HOME/conf/okvclient.ora -s revoke -u 994CB3E5-C5B3-4F75-BFED-CB41AE15D0B1 -q CESSATION_OF_OPERATION

   You must provide a reason for revoking the password by using the -q option. Possible values are as follows:

   • UNSPECIFIED
   • KEY_COMPROMISE
   • CA_COMPROMISE
   • AFFILIATION_CHANGED
   • SUPERSEDED
   • CESSATION_OF_OPERATION
   • PRIVILEGE_WITHDRAWN
4. Destroy this password.

   $ java -jar okvrestservices.jar kmip -c $OKV_HOME/conf/okvclient.ora -s destroy -u 994CB3E5-C5B3-4F75-BFED-CB41AE15D0B1
   

5. In the Oracle Key Vault management console, delete the secret.
6. Create a text file to contain the new password, similar to the original password text file that was used to configure the original external keystore password.

   $ more backup_pwd
   cn0KpOnY9vNec2sLVHFnJwR6

7. Upload the new password and attributes, and then activate the new secret.

   $ java -jar okvrestservices.jar kmip -c $OKV_HOME/conf/okvclient.ora -s reg_secret -t PASSWORD -m ENCRYPT -o ./backup_pwd -f ./backup_name
   0739649D-3058-4F34-BF84-50B2BD652C2D

   $ java -jar okvrestservices.jar kmip -c $OKV_HOME/conf/okvclient.ora -s activate -u 0739649D-3058-4F34-BF84-50B2BD652C2D
   

8. On Linux, securely delete the file that contain passwords.

   shred -xz backup_pwd
   rm backup_pwd

9. In each Oracle database that shares the secret, log in to SQL*Plus and use the password command to change the password.
   For example:

   SYS> password nightly_backup
   Changing password for nightly_backup
   New password: cn0KpOnY9vNec2sLVHFnJwR6
   Retype new password: cn0KpOnY9vNec2sLVHFnJwR6
   Password changed