1. Administrator's Guide
  2. Monitoring and Auditing Oracle Key Vault

22 Monitoring and Auditing Oracle Key Vault

Oracle Key Vault administrators can monitor and audit the Oracle Key Vault system, configure alerts and use reports.

  • Managing System Monitoring
    System monitoring refers to tasks such as configuring SNMP connections, email notifications, the syslog destination, and system diagnostics.
  • Configuring Oracle Key Vault Alerts
    You can select the type of alerts that you want to see in the Oracle Key Vault dashboard.
  • Managing System Auditing
    Auditing entails tasks such as capturing audit records in a syslog file or downloading the audit records to a local file.
  • Using Oracle Key Vault Reports
    Oracle Key Vault collects statistical information on a range of activities that impact Key Vault operations.

22.1 Managing System Monitoring

System monitoring refers to tasks such as configuring SNMP connections, email notifications, the syslog destination, and system diagnostics.

Parent topic: Monitoring and Auditing Oracle Key Vault

22.1.1 Configuring Remote Monitoring to Use SNMP

With Simple Network Management Protocol (SNMP) enabled, system administrators can remotely monitor the Oracle Key Vault appliance and its services.

The collected data can be further processed and presented for the needs of the enterprise.

Parent topic: Managing System Monitoring

22.1.1.1 About Using SNMP for Oracle Key Vault

You can use the Simple Network Management Protocol (SNMP) to monitor devices on a network for resource usage.

Monitoring Oracle Key Vault is an important aspect how critical Oracle Key Vault's availability is when hundreds or thousands of Oracle and MySQL databases store their TDE master encryption keys in an Oracle Key Vault multi-master cluster. The types of resource usage that you should monitor include memory, CPU utilization, and processes. Even though Oracle Key Vault provides continuous key availability by allowing up to 16 (geographically distributed) instances to be connected to a single cluster, the health of each individual node contributes to the performance and availability of the entire cluster.

You can use Simple Network Management Protocol (SNMP) third-party tool to monitor remote systems that access Oracle Key Vault. The benefits of using SNMP to monitor Oracle Key Vault are as follows:

  • There is no need to allow SSH access to Oracle Key Vault. (SSH access should only be enabled for the window of time in which it is being used.)
  • You do not need to install additional tools to perform an SNMP monitoring operation.

Oracle Key Vault uses SNMP version 3 for user authentication and data encryption features. Unlike SNMP versions 1 and 2 that communicate in readable, insecure plaintext, SNMP 3 authenticates users and encrypts data on the communication channel between the monitoring server and the target. The information from Oracle Key Vault is unreadable to an intruder, even if the communication channel is intercepted.

In addition, with SNMP enabled on Oracle Key Vault, you can determine whether the key management server (KMIP daemon) is running. To track this information, you must use a third-party SNMP client to poll the Oracle Key Vault instance, because Oracle Key Vault does not provide SNMP client software.

Oracle Key Vault audits the creation and modification of SNMP credentials.

You must be a user with the System Administrator role to configure the SNMP account with a user name and password. These SNMP credentials are needed to access SNMP data.

In a multi-master cluster, the SNMP account with a user name and password can be set for all nodes of the cluster at once. It can also be set for each individual node.

Note:

You must ensure that the SNMP username and password is not the same username and password as any of the Oracle Key Vault administrative user accounts with the System Administrator, Key Administrator, or Audit Manager role.

22.1.1.2 Granting SNMP Access to Users

You can grant any user, including users who are not Oracle Key Vault administrators, access to SNMP data.

  1. Log in to the Oracle Key Vault management console as a user with the System Administrator role.
  2. Select the System tab, and then select Settings from the left navigation bar.
  3. In the Monitoring and Alerts area, click SNMP.
  4. In the Monitoring Settings page, enter the following information:
    • SNMP Access allowed from: Select All to enable a client at any IP address to poll Oracle Key Vault for information, Disabled to prevent any client, regardless of the client IP address, to poll Oracle Key Vault for information, or IP Address(es) if you want to restrict polling to clients with specific IP addresses. If you select IP Address(es), then enter the IP addresses of the users you want to grant access to in the IP Address field. Separate multiple IP addresses by a space. You cannot enter a range of IP addresses. You must list each IP address individually.
    • User Name: Enter a name to associate with the SNMP configuration that will perform the monitoring.
    • Password and Re-enter Password: Enter a secure password for this user that is at least 8 or more characters and contains at least one of each of the following: an uppercase letter, a lowercase letter, a number, and a special character from the set: period (.), comma (,), underscore (_), plus sign (+), colon (:), space. The SNMP password must not be the same as the password used to login into the Oracle Key Vault management console in any of the administrative roles.
  5. Click Save.
22.1.1.3 Changing the SNMP User Name and Password

You can change the SNMP user name and password for a node at any time.

  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Select the System tab, and then select Settings from the left navigation bar.
  3. In the Monitoring and Alerts area, click SNMP.
  4. In the User Name, Password, and Re-enter Password fields, enter the user name and password information.
  5. Click Save.
22.1.1.4 Changing SNMP Settings on the Standby Server

You change the SNMP settings from the command line on the standby server.

To add SNMP support in a primary-standby environment, you should configure SNMP on both the primary and standby servers before pairing them. This is because the standby server is no longer accessible from the Oracle Key Vault management console because all requests are forwarded to the primary server. However, you can change SNMP settings on the standby server in a primary-standby environment.

  1. Log in to the standby server as the support user.
  2. Switch to the root user. 
    su -
  3. Go to the Oracle Key Vault bin directory.
    cd /usr/local/okv/bin/
  4. Run the stdby_snmp_enable script.
    ./stdby_snmp_enable parameter "options"
    In this specification:
    • parameter can be the following:
      • -a, which sets the SNMP access. It accepts the following options:
        • all grants SNMP access.
        • disabled disables SNMP access.
        • IP_addresses specifies one or more IP addresses to be granted SNMP access. Separate each IP address with a space.
      • -u sets the user's SNMP name. This is the user name that was configured as the snmpuser when SNMP was enabled.
      • -p sets the user's SNMP password. This password was created when for the snmpuser when SNMP was enabled.
    • options is only used with the -a parameter.

The following examples show how to change SNMP settings on a standby server:

To grant SNMP access to all IP addresses and assign a user name snmpuser and password password:
./stdby_snmp_enable -a "all" -u "snmpuser" -p "password"
To disable SNMP access from all IP addresses: 
./stdby_snmp_enable -a "disabled"
To grant SNMP access from certain IP addresses and assign user name snmpuser and the password password:
./stdby_snmp_enable -a "192.0.2.1 192.0.2.2 192.0.2.3" -u "snmpuser" -p "password"
22.1.1.5 Remotely Monitoring Oracle Key Vault Using SNMP

SNMP enables you to monitor the vital components of Oracle Key Vault remotely without having to install new software in Oracle Key Vault.

Though there are third-party tools that graphically display the information that SNMP extracts from Oracle Key Vault, the examples shown here are given with snmpwalk and snmpget from the command line on a remote computer that has a network connection into the SNMP account in Oracle Key Vault.
  1. Log in to the remote host that will monitor Oracle Key Vault.
  2. Confirm that the UCD-SNMP-MIB is installed on the remote host from which Oracle Key Vault is monitored.
  3. Query the object ID for an Oracle Key Vault-supported SNMP Management Information Base (MIB) variable.
    For example, suppose you wanted to track the number of processes running for the SNMP host. You can use a third-party SNMP client utility to query the status of the KMIP MIB whose object ID is 1.3.6.1.4.1.2021.2, as follows:
    third_party_snmp_client_command -v 3 OKV_IP_address -u SNMP_user -a SHA -A SNMP_password -x AES -X SNMP_password -l authPriv iso.3.6.1.4.1.2021.2.1.2

    The output is similar to the following:

    iso.3.6.1.4.1.2021.2.1.2.1 = STRING: "mwecsvc"              <== Event collector
iso.3.6.1.4.1.2021.2.1.2.2 = STRING: "httpd"                <== httpd
iso.3.6.1.4.1.2021.2.1.2.3 = STRING: "kmipd"                <== KMIP daemon
iso.3.6.1.4.1.2021.2.1.2.4 = STRING: "ora_pmon_dbfwdb"      <== embedded DB
iso.3.6.1.4.1.2021.2.1.2.5 = STRING: "ServiceManager"       <== Golden Gate Service Manager (Monitors other processes and reports status)
iso.3.6.1.4.1.2021.2.1.2.6 = STRING: "adminsrvr"            <== Golden Gate Admin Server (Communicates with the DB to perform certain maintenance/admin tasks)
iso.3.6.1.4.1.2021.2.1.2.7 = STRING: "distsrvr"             <== Golden Gate Distribution Server (Sends the OGG changes to other nodes)
iso.3.6.1.4.1.2021.2.1.2.8 = STRING: "recvsrvr"             <== Golden Gate Receiver Server
22.1.1.6 SNMP Management Information Base Variables for Oracle Key Vault

Oracle Key Vault provides a set of SNMP Management Information Base (MIB) variables that you can track.

The following table lists the MIB variables that are supported.

Table 22-1 MIBs That SNMP Tracks for Oracle Key Vault

MIB Variable Object ID Description

hrSystemUptime

1.3.6.1.2.1.25.1.1

Tracks the amount of time that an Oracle Key Vault instance has been running

ifAdminStatus.x

1.3.6.1.2.1.2.2.1.7

Tracks if the Oracle Key Vault network interface (x) are running, not running, or being tested. Values are as follows:

  • 1: Instance is running

  • 2: Instance is down

  • 3: Instance is being tested

memAvailReal

1.3.6.1.4.1.2021.4.6

Tracks the available RAM

memTotalReal

1.3.6.1.4.1.2021.4.5

Tracks the total amount of RAM being used

ssCpuRawIdle

1.3.6.1.4.1.2021.11.53

For CPU monitoring; tracks the number of ticks (typically 1/100s) spent idle

ssCpuRawInterrupt

1.3.6.1.4.1.2021.11.56

For CPU monitoring; tracks the number of ticks (typically 1/100s) spent processing hardware interrupts

ssCpuRawKernel

1.3.6.1.4.1.2021.11.55

For CPU monitoring; tracks the number of ticks (typically 1/100s) spent processing kernel-level code

ssCpuRawNice

1.3.6.1.4.1.2021.11.51

For CPU monitoring; tracks the number of ticks (typically 1/100s) spent processing reduced-priority code

ssCpuRawSystem

1.3.6.1.4.1.2021.11.52

For CPU monitoring; tracks the number of ticks (typically 1/100s) spent processing system-level code

ssCpuRawUser

1.3.6.1.4.1.2021.11.50

For CPU monitoring; tracks the number of ticks (typically 1/100s) spent processing user-level code

ssCpuRawWait

1.3.6.1.4.1.2021.11.54

For CPU monitoring; tracks the number of ticks (typically 1/100s) spent waiting for input-output (IO)

UCD-SNMP-MIB.prTable

1.3.6.1.4.1.2021.2

Tracks the number of processes running under a certain name. Names we monitor are httpd (the http server), kmipd (the kmip daemon), and ora_pmon_dbfwdb (an indicator if the DB is down)

nsExtendOutputFull

1.3.6.1.4.1.8072.1.3.2.3.1.2

For monitoring Fast Recovery Area Space utilization; displays the size, used and free space. The alert also shows the CA and the server certificate expiration date and time, as well as the status of the Oracle Audit Vault agent and the Apache Tomcat Web server. For the certificate expiration, the time zone that is shown for the date and time is in UTC. The output may be inconsistent if Oracle Key Vault is in the middle of a certification rotation.
sysDescr

1.3.6.1.2.1.1.1

Represents the product name and version identification of Oracle Key Vault.
sysUpTime

1.3.6.1.2.1.1.3

Represents the time (in hundredths of a second) since the network management portion of the system was last re-initialized.
sysName

1.3.6.1.2.1.1.5

Represents an administratively-assigned name. By convention, this is the node's fully-qualified domain name.

See Also:

For more information refer to the Net-SNMP documentation at http://www.net-snmp.org

Parent topic: Configuring Remote Monitoring to Use SNMP

22.1.1.7 Example: Simplified Remote Monitoring of Oracle Key Vault Using SNMP

In Linux, you can simplify the SNMP commands you manually enter to find Oracle Key Vault information, yet still have useful and detailed output.

The configuration in this section assumes that you have granted SNMP access to a trusted user. It also assumes that the you have installed the SNMP Management Information Base (MIB) variables on the remote host that will monitor Oracle Key Vault.

For example, a lengthy version of the snmpwalk command for an SNMP user named snmp_admin is as follows: 

snmpwalk -v3 OKV_IP_address -n "" -l authPriv -u snmp_admin -a SHA -A snmp_user_password -x AES -X snmp_user_password

This command lists the vital services that are running on Oracle Key Vault. However, you can modify the command (and other SNMP commands) to be not only shorter, but to show additional information, such as whether the services are running or not running.

To simplify this type of command, you can edit the /etc/snmp/snmp.conf configuration file so that the SNMP commands you enter will automatically include commonly used settings, such as the default user or the default security level. The example in this topic omits password parameters so that users can enter the password at the command line interactively.

  1. Log in to the remote host that will monitor Oracle Key Vault.
  2. Edit the /etc/snmp/snmp.conf, which appears as follows: 
    # As the snmp packages come without MIB files due to license reasons, 
# loading MIBs is disabled by default. If you added the MIBs you 
# can reenable loading them by commenting out the following line. 
  mibs :
  3. Comment out the # mibs : line and then add the following lines, as follows:
    # loading MIBs is disabled by default. If you added the MIBs you 
# can reenable loading them by commenting out the following line. 
# mibs : 
defSecurityName snmp_admin 
defSecurityLevel authPriv 
defAuthType SHA1
defPrivType AES128

    In this example:

    • defSecurityName: Enter the name of the user to whom you granted SNMP access. This example uses snmp_admin.
    • defSecurityLevel: Enter the default security level to use. This example uses authPriv, which enables communication with authentication and privacy.
    • defAuthType: Enter the default authorization type. This example uses SHA1.
    • defPrivType: Enter the default privilege type. This example uses AES128.
  4. Restart snmpd to load the configuration file.

    For example, for Linux 7 and later:

    systemctl restart snmpd

    For Linux 6:

    service snmpd restart
  5. To run the simplified version of the snmpwalk command that was shown earlier, enter the following command: 
    snmpwalk okv_ip_address prNames -A snmp_user_pwd -X snmp_user_pwd

    In this command, prNames refers to "process names", which displays the names of processes instead of numbers. For example: 

    $ snmpwalk 192.0.2.254 prNames -A snmp_user_pwd -X snmp_user_pwd
UCD-SNMP-MIB::prNames.1 = STRING: httpd
UCD-SNMP-MIB::prNames.2 = STRING: kmipd
UCD-SNMP-MIB::prNames.3 = STRING: kmipusd
UCD-SNMP-MIB::prNames.4 = STRING: ora_pmon_dbfwdb
UCD-SNMP-MIB::prNames.5 = STRING: ServiceManager
UCD-SNMP-MIB::prNames.6 = STRING: adminsrvr
UCD-SNMP-MIB::prNames.7 = STRING: distsrvr
UCD-SNMP-MIB::prNames.8 = STRING: recvsrvr
UCD-SNMP-MIB::prNames.9 = STRING: av_agent_monitor
An example of running the snmptable command now becomes the following. 
snmptable okv_ip_address prTable -A snmp_user_pwd -X snmp_user_pwd

Output similar to the following appears. 

$ snmptable 192.168.1.181 -A Manager_1 -X Manager_1 prTable
SNMP table: UCD-SNMP-MIB::prTable

 prIndex          prNames prMin prMax prCount prErrorFlag                        prErrMessage prErrFix prErrFixCmd
       1            httpd     1    20       8     noError                                      noError            
       2            kmipd     1     2       2     noError                                      noError            
       3          kmipusd     1     2       2     noError                                      noError            
       4  ora_pmon_dbfwdb     1     1       1     noError                                      noError            
       5   ServiceManager     1     1       1     noError                                      noError            
       6        adminsrvr     1     1       1     noError                                      noError            
       7         distsrvr     1     1       1     noError                                      noError            
       8         recvsrvr     1     1       1     noError                                      noError            
       9 av_agent_monitor     1     1       0       error No av_agent_monitor process running  noError

The next example shows how you would now run the snmpdf command: 

snmpdf okv_ip_address -A snmp_user_pwd -X snmp_user_pwd

Output similar to the following appears. 

Description                Size (kB)      Used   Available   Used% 
/                          20027260    7247856    12779404     36%  
/usr/local/dbfw/tmp         6932408      15764     6916644      0%  
/var/log                    5932616      19932     5912684      0% 
/tmp                        1999184       3072     1996112      0% 
/var/lib/oracle           143592160   35023900   108568260     24%

22.1.2 Configuring Email Notification

You can use email notifications to directly notify administrators of Key Vault status changes without logging into the Oracle Key Vault management console.

Parent topic: Managing System Monitoring

22.1.2.1 About Email Notification

Email notifications alert users of status changes and are used to complete the processes of endpoint enrollment and user password reset operations.

To enable email notification you must set your email preferences in Oracle Key Vault. You can choose the events that you want updates to. The events include Oracle Key Vault system status like disk utilization, backup, and primary-standby, or user and endpoint status like expiration of user passwords, endpoint certificates, and keys, or cluster status like the heartbeat lag, naming conflicts, cluster-wide HSM status, and others.

In cluster deployments, you must configure and validate email settings on all nodes of the cluster. Email settings of a cluster node are local to that node.

Configuring Email Settings is driven by the SMTP provider. Once you confirm that the SMTP server is reachable from the Oracle Key Vault server. You need to follow the required setting from the SMTP provider.

You can modify the SMTP server configuration at any time. If a custom SMTP certificate was used initially, and you later decide to use the default, you can modify the trust store setting to default, instead of custom.

For example:

  • The enrollment token generated during endpoint enrollment can be mailed directly to the endpoint administrator from Oracle Key Vault.

  • An Oracle Key Vault system administrator can send the random temporary password directly to the user when the user password is reset.

To enable email notifications successfully, there must be a connection between Oracle Key Vault and the SMTP server.

You can disable email notifications at any time.

Note:

If you are using Oracle Key Vault in an Oracle Cloud Infrastructure (OCI) environment, then see My Oracle Support note 2501601.1 for information about how to configure Postfix to use email delivery on the Oracle Linux 6 and 7 platforms. After you complete the configuration, ensure that you populate the From Address field with the approved sender from OCI.
22.1.2.2 Configuring Email Settings

You can configure the Simple Mail Transfer Protocol (SMTP) server properties to receive email notifications from Oracle Key Vault.

  1. Log in to the Oracle Key Vault management console as a user with the System Administrator role.
  2. Select the System tab, and then select Settings from the left navigation bar.
  3. In the Network Services area, click Email.

    The Email Settings page appears.


    Description of 21_email.png follows
    Description of the illustration 21_email.png

  4. In the Email Settings page, enter the following values:

    • SMTP Server Address: Enter a valid SMTP server address or host name for the user account. This setting should match the SMTP server setting of the user's email account. Ensure that the SMTP server or hostname is reachable from Oracle Key Vault. If you enter the SMTP hostname, you must configure DNS from the System Settings menu, so the host name can be resolved.

    • SMTP Port: Enter the SMTP port number of the outgoing SMTP server, usually 465. This port number can be another number, if expressly configured that way in your organization.

    • Name: Enter an alias for the SMTP user that will appear in the From field of the email.

    • From Address: Enter the email address that you want to provide as a sender.

    • If the SMTP server requires a secure connection, select Require Secure Connection. If you are using anonymous relay on Microsoft Exchange Server, or an external SMTP server such as Gmail or Office 365, do not select Require Secure Connection. Ensure that your firewall rules allow forwarding of SMTP requests to an external SMTP server.

      If Require Secure Connection is selected, the Authentication Protocol field is displayed with two options, SSL and TLS. Select the authentication protocol for the email server, either SSL or TLS. The default is TLS.

    • If you have an SMTP user account, then check the box Require Credentials. When checked, the input fields User Name, Password, and Re-enter Password appear:

      • Enter the username of the SMTP user account.

      • Enter the password for the SMTP user account.

      • Reenter the password for the SMTP user account.

      Caution:

      Oracle strongly recommends that you have a secure connection to the SMTP server, because auto-generated tokens are sent over email for operations such as the creation of administrative users and Oracle Key Vault system alerts.

      Do not check Require Credentials for non-secure connections.

    • If Custom SMTP Server Certificate is checked, then the field Upload Certificate File appears with the Choose File button to its right. Select this option if you want to upload a custom SMTP server's certificate to establish a TLS session between SMTP and Oracle Key Vault. This is how you can add a custom truststore in cases where the default Java truststore does not contain a necessary certificate. After Upload Certificate File, click Browse to upload a custom certificate file.

  5. Click Configure.

    On successful configuration, a SMTP successfully configured message is displayed.

    If the configuration fails, then check that the SMTP server settings of the user email account are correct. Error messages highlight the field where the error has occurred to help isolate the problem.

22.1.2.3 Testing the Email Configuration

Oracle Key Vault management console enables you to send test emails to test the email configuration.

You can test the email configuration of the SMTP user account any time after you save the configuration. If you change an existing SMTP configuration, then you must save the configuration before you can test it.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Select the System tab, and then select Settings from the left navigation bar.
  3. In the Network Services area, click Email.
  4. In the Email Settings window, configure the user's SMTP settings.
  5. Save the configuration by clicking Configure.
    You must save the configuration before you can test it.
  6. In the Send Test Email section, enter the user email address in the Email Address field. Then click Test.

    An email is sent to the user with Oracle Key Vault: Test Message in the subject line.

    Depending on the Oracle Key Vault server timestamp, the email notification may not show up as the latest email.

    The email notification may also not show up in your inbox, in which case you must check the spam folder.

    If the email notification is not received, click the Reports tab and select System Reports from the left sidebar. On the System Reports page, click Notification Report. Check the list to determine the issue encountered while sending the email notification.

22.1.2.4 Disabling Email Notifications for a User

You can use the Oracle Key Vault management console to enable or disable email notifications.

An Oracle Key Vault user may elect not to receive email alerts. Only a user with the System Administrator role, or a user managing his own account can disable email notifications.
  1. Log in to the Oracle Key Vault management console as a user with the System Administrator role.
  2. Select the Users tab.

    The Manage Users page appears.

  3. Click user name of the user.

    The User Details page appears.

  4. Check the box to the left of text Do not receive email alerts.
  5. Click Save.

22.1.3 Configuring the Syslog Destination for Individual Multi-Master Cluster Nodes

On each node, you can forward syslog entries to a remote service such as Splunk or SIEM.

Parent topic: Managing System Monitoring

22.1.3.1 Setting the Syslog Destination Setting for the Node

You can set the syslog destination to use either Transmission Control Protocol (TCP) or User Datagram Protocol (UDP).

  1. Log into the Oracle Key Vault management console for the node as a user who has the System Administrator role.
  2. Select the System tab, and then Settings from the left navigation bar.
  3. In the Monitoring and Alerts area, click Syslog.
  4. In the Syslog window, select one of the following options:
    • TCP: Enables syslog using the TCP protocol.
    • UDP: Enables syslog using the UDP protocol.
  5. Enter the syslog destination IP addresses and port numbers in the Syslog Destinations field, in the format IP_address:port.
    You can enter multiple destinations, each separated by a space.
  6. Click Save.
22.1.3.2 Clearing the Syslog Destination Setting for the Node

You can clear the syslog destination setting for the node and then reset the node to the cluster setting.

  1. Log into the Oracle Key Vault management console for the node as a user who has the System Administrator role.
  2. Select the System tab, and then Settings from the left navigation bar.
  3. In the Monitoring and Alerts area, click Syslog.
  4. In the Syslog window, click Delete.

22.1.4 Capturing System Diagnostics

To troubleshoot problems that may arise, you can generate a diagnostics package.

  • About Capturing System Diagnostics
    The Oracle Key Vault diagnostics file provides advanced debugging and troubleshooting information for problems that you may encounter while using Oracle Key Vault.
  • Configuring the Oracle Key Vault Application Tracing Level
    The System Administrator can configure the tracing level for the unified application tracing from the configure diagnostics page.
  • Downloading the Diagnostics Package
    The system administrator can download the diagnostic files using the Oracle Key Vault management console.
  • Unpacking the Diagnostics Package
    When the generated diagnostics bundle file size is smaller than the partition size, the diagnostics package is available in a singular .zip file. Otherwise, the diagnostics bundle is split into and is available in parts with the extension .zip-partXX.
  • Deleting Trace Files
    You can delete the old tracing files after they are downloaded to free up the disk space.

Parent topic: Managing System Monitoring

22.1.4.1 About Capturing System Diagnostics

The Oracle Key Vault diagnostics file provides advanced debugging and troubleshooting information for problems that you may encounter while using Oracle Key Vault.

You can download diagnostics file and provide it to Oracle support for further analysis and debugging.

By default, diagnostics reporting is enabled on Oracle Key Vault. With the simplified diagnostics collection, system administrators are able to select which diagnostics components are to be packaged for the downloadable diagnostics bundle. Be aware that the first time you run the diagnostic utility or after the Oracle Key Vault system’s internal database has been restarted, it can take longer to produce the bundle compared to subsequent runs because it must gather all the diagnostic information of the system.

If you plan to perform an upgrade of the Oracle Key Vault server, then disable the diagnostics packaging utility by ensuring that there are no files available to download. This can be confirmed by checking if the Diagnostics page has a section called Diagnostics Package Files. If it does, click Clear to disable the utility.

During upgrades, the current trace level for each component will reset to the Mandatory trace level.

22.1.4.2 Configuring the Oracle Key Vault Application Tracing Level

The System Administrator can configure the tracing level for the unified application tracing from the configure diagnostics page.

The steps describe how the system administrator can configure the tracing level from the Oracle Key Vault management console.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Select System, and then Settings.
  3. Click Diagnostics in the Monitoring and Alerts area.
    The Diagnostics Configuration page appears.
    Description of 216_configure_dignostics.png follows
    Description of the illustration 216_configure_dignostics.png

  4. Select the tracing option for Application Trace Level from the Configurable Components area. By default, the Database Trace Level and System Trace Level are always set to MANDATORY.
    • Trace Levels
      • The Application Trace Level enables the LOG option of the earlier release's diagnostics utility as well as adjusts the tracing level of the Oracle Key Vault application.

        Note:

        Setting the application trace level does not affect the trace level of the KMIP server. Contact Oracle support for instructions on how to enable tracing for the KMIP server.
      • The Database Trace Level enables the DATABASE option of the earlier release's diagnostics utility.
      • The System Trace Level enables the SYSTEM, SOS_REPORT, and PLATFORM_COMMANDS options of the earlier release's diagnostics utility.
      • The MANDATORY level collects traces that are considered critical system conditions.
      • The ERROR level collects traces in instances of errors and exceptions.
      • The WARNING level collects traces in instances of warning conditions.
      • The INFO level collects traces for general operational information.
      • The DEBUG level collects all available traces.

    Note:

    Adjusting the trace level only sets the Configurable Components for the specific node or server. To update the trace level on other cluster nodes or the standby server of a Primary-Standby deployment, repeat these steps on the other nodes or servers.
  5. Click Save.

    The Oracle Key Vault tracing selections updated successfully message ensures the settings are saved successfully.

22.1.4.3 Downloading the Diagnostics Package

The system administrator can download the diagnostic files using the Oracle Key Vault management console.

The steps explains how you can customize the diagnostic package contents by only selecting the components of interest.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Select System, and then Diagnostics.

    The Download Diagnostics page appears.

  3. Select the component for downloading the related diagnostics.

    Note:

    The KMIP server tracing data is also included when you select the component Application.
  4. You can also adjust the Partition Size based on your requirement. The default diagnostics package file partition size is 500 MB.
  5. Click Save to save the download settings for future use.
  6. Click Create to generate the bundle.
  7. The Diagnostics Package Files pane appears with the list of files ready to be downloaded.
    Description of 216_diagnostic_package_files.png follows
    Description of the illustration 216_diagnostic_package_files.png

  8. Select the files to download and click Download.

    Note:

    Ensure all parts are downloaded before recombining them for the full diagnostics bundle. To combine the file parts, see Unpacking the Diagnostics Package.
22.1.4.4 Unpacking the Diagnostics Package

When the generated diagnostics bundle file size is smaller than the partition size, the diagnostics package is available in a singular .zip file. Otherwise, the diagnostics bundle is split into and is available in parts with the extension .zip-partXX.

  1. Ensure you have followed steps from section Downloading the Diagnostics Package, see Downloading the Diagnostics Package
  2. On the Diagnostics Package page, select the tracing file(s) to download and click Download from the Diagnostics Package Files pane.
    Description of 216_download_package_files.png follows
    Description of the illustration 216_download_package_files.png

  3. Download and save the .zip file that contains the diagnostic reports to a secure location.

    Note:

    As each node is traced independently, in a cluster deployment mode, trace level is required to be set manually for each node.

    If the diagnostics package file is with zip-partXX extension, recombine the files using the steps mentioned below.

    1. After the user has downloaded all the parts, run the following commands in a shell where the files have been downloaded to combine the parts into one zip file.
      For Linux:
      $ cat okv_traces.zip-part* > okv_traces.zip
      For Windows:
      $ type okv_traces.zip-part* > okv_traces.zip
    2. unzip the file using the following command:
      For Linux:
      $ unzip okv_traces.zip -d path_to_unpack_traces_to
      For Windows:
      $ Expand-Archive -LiteralPath okv_traces.zip -DestinationPath path_to_unpack_traces_to
    3. The application traces are available at <download_location>/var/lib/oracle/okv_application_traces/ path while the database and system diagnostics are located at the same directory locations as in earlier Oracle Key Vault releases.
      The application traces will have the following format: 
      VERSION | TIMESTAMP (YYYY-MM-DD HH:MM:SS.FF3 format) | HOSTNAME | TYPE | USER-CONTEXT IDENTIFIER | EXECUTION IDENTIFIER | TRACE SEVERITY LEVEL |  COMPONENT |  FILE NAME[LINE NUMBER] | MESSAGE
      v1 | 2023-01-19 18:49:45.810 | okv02001703c6fe | PLSQL | DD9E6226-742F-43BB-A725 -0CF6C8A7EBC0 | SID:1334,SPID:3460,CPID:1234 | 2 | GEN_SERVER | LDAP[3143] | Unable to reach the hostname when testing LDAP configuration
      • VERSION: Indicates the version of the trace line format.
      • HOSTNAME: Provides the host information (server or cluster) where the trace level are getting recorded.
      • TYPE: Indicates the file type that executed the line of code for the trace statement.
      • USER-CONTEXT IDENTIFIER: Indicates the user content identifier to identify the user who initiated the action. This can refer to an endpoint as well as a user.
      • EXECUTION IDENTIFIER: Identifies the execution context of a particular sequence of trace statement will include different types of IDs.
      • TRACE SEVERITY LEVEL : Identifies the trace level a specific message was printed. Possible trace levels are: 1, 2, 4, 8, 16.
      • COMPONENT: Indicates the component been traced. Currently, the componenet available is GEN_SERVER.
      • FILE NAME[LINE NUMBER]: Help in identifying the file name and line number for each trace statement.
      • MESSAGE: Indicates the specific trace message that contains the information about the trace level.
    4. Decompress the application trace files using the $ gzip -d okv_trace.trc.<date>.gz command.
  4. After downloading all parts of the diagnostics package, click Clear on the Oracle Key Vault management console page to clear disk space.
22.1.4.5 Deleting Trace Files

You can delete the old tracing files after they are downloaded to free up the disk space.

  1. Log in to the Oracle Key Vault management console as a user with the System Administrator role.
  2. Select System, and then Diagnostics. The Download Diagnostics page displays.
  3. Click Delete Trace Files.
    Review and confirm the prompt to proceed with the deletion.

    Note:

    The Delete Trace Files also deletes the older KMIP server tracing data. The tracing files that are currently in use are not deleted regardless of their size.

22.1.5 Monitoring System Metrics

You can use the System Metrics Monitoring feature to view and collect data for key system resource usage including CPU and Memory Usage in Oracle Key Vault.

  • About Capturing System Metrics
    The System Metrics Monitoring feature provide resource monitoring capabilities using the Oracle Key Vault management console.
  • Viewing System Metrics
    You can use the Oracle Key Vault management console to view and download the system monitoring data.

Parent topic: Managing System Monitoring

22.1.5.1 About Capturing System Metrics

The System Metrics Monitoring feature provide resource monitoring capabilities using the Oracle Key Vault management console.

Oracle Key Vault periodically collects CPU and Memory Usage, Disk I/O, Network and Application Metrics. You can view and collect system metrics data using the Oracle Key Vault management console. The Oracle Key Vault System Metrics Monitoring eliminates the need to first login to Oracle Key Vault server and then monitor the system manually.

22.1.5.2 Viewing System Metrics

You can use the Oracle Key Vault management console to view and download the system monitoring data.

The instructions also explain how you can customize the output to collect required data.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Select System.
  3. Click System Metrics.

    System Metrics page shows the status of metrics collection service. Up means the service is running and collecting the data and Down means there's an issue and Oracle Key Vault is not able to collect the data.

    System Metrics page appears with collapsed regions for each metrics category.

    Description of 216_system_metrics.png follows
    Description of the illustration 216_system_metrics.png

    Expand the System Metrics region to display the graph for:
    1. CPU and Memory usage
    2. Disk I/O Metrics
    3. Network Metrics
    4. Application Metrics

    Table 22-2 System Metrics

    System Metrics Description
    CPU and Memory usage percentage

    Hovering mouse over a point in CPU Usage graph shows following:

    • Time at which the data point collected
    • CPU usage percentage at the data point
    • Number of CPU cores at the data point
    • CPU load averages for last one, five and fifteen minutes at the data point
    Hovering mouse over a point in Memory Usage graph shows following:
    • Time at which the data point collected
    • Memory usage percentage at the data point
    • Total memory in GB at the data point
    • Free memory in GB at the data point
    Disk I/O Metrics
    Hovering mouse over a point in Disk Reads graph shows following:
    • Time at which the data point collected
    • Total number of disk reads
    Hovering mouse over a point in Disk Writes graph shows following:
    • Time at which the disk write data collected
    • Total number of disk writes
    Network Metrics
    Hovering mouse over a point in Data Received graph shows following:
    • Time at which the data point collected
    • Collected data in bytes
    • Average rate at which data received
    Hovering mouse over a point in Data Sent graph shows following:
    • Time at which the data point collected
    • Collected data in bytes
    • Average rate at which data sent
    Incoming TCP Connections
    Hovering mouse over a point in Incoming TCP Connections graph shows following:
    • Time at which the data point collected
    • Number of incoming TCP connections
    Application Metrics
    Hovering mouse over a point in KMIP Connections graph shows following:
    • Time at which the data point collected
    • Total number of KMIP connections
    • Number of KMIP connections on non-RESTFul interface
    • Number of KMIP connections on RESTFul interface
  4. Select the different options from the Period drop-down to view percentage usage in Last 1 Hour, Last 24 Hour, Last Week, Last Month, or Date Range(Period).

    Note:

    You can use the Date Range (Period) option to view and collect the usage data by specifying the From and To dates.
  5. Select the different options from the Interval drop-down to aggregate the displayed data. You can select the Auto option for optimized performance.
  6. Select an aggregate function from the Statistic drop-down.
    You can select from Mean, Min, Max or Count value.

    Note:

    The Count statistic is not applicable to all the metrics.
  7. Click Refresh to refresh and display the data according to the specified fields.
  8. Click Download to save the data in a .csv file.

    Note:

    • Clicking on Download button save the raw data for the period specified. Also, the downloaded data do not have interval and statistic filters applied on it.
    • You might not be able to view the data or make any changes to the System metrics when the Metrics Service is Down.

22.2 Configuring Oracle Key Vault Alerts

You can select the type of alerts that you want to see in the Oracle Key Vault dashboard.

  • About Configuring Alerts
    System administrators can configure alerts from the Oracle Key Vault dashboard, but all users can see alerts for the security objects to which they have access.
  • Configuring Alerts
    You can configure alerts in the Reports page of the Oracle Key Vault management console.
  • Viewing Open Alerts
    Users can view alerts depending on their privileges.

Parent topic: Monitoring and Auditing Oracle Key Vault

22.2.1 About Configuring Alerts

System administrators can configure alerts from the Oracle Key Vault dashboard, but all users can see alerts for the security objects to which they have access.

Email notifications must be enabled for users to receive alerts.

The Oracle Key Vault dashboard is the first page you see on logging into the management console. You can navigate to this page by clicking the Home tab. All users can see the alerts on security objects they have access to, but only users with the System Administrator role can configure alerts.

Oracle Key Vault offers several types of alerts that you can configure with appropriate thresholds according to your requirements. The alert types that appear are based on the type of environment that you are using: standalone, primary-standby, or multi-master cluster. You can also configure alerts for an HSM-enabled Oracle Key Vault server.

Oracle Key Vault alerts are categorized to one of the severity levels: CRITICAL, HIGH, MEDIUM, and LOW. You should resolve the higher severity alerts first.

An alert configuration consists of whether an alert is enabled or disabled and the threshold limit. In a multi-master cluster, by default, same alert configuration is applied to all of the nodes in a cluster. For following alerts, alert configuration can be on either Cluster or Node scope.
  • Fast Recovery Area Space Utilization
  • High CPU Usage
  • Failed System Backup
  • High Memory Usage
  • Disk Utilization
  • System Backup

You can configure the following alerts, which are listed in ascending order:

Table 22-3 Available Alerts

Alert Type Severity Environment Multi-master Cluster Applicability Purpose of Alert When Alert Is Deleted

Certificate Object Expiration

 HIGH

Standalone, primary-standby, and multi-master cluster environments

Cluster-wide

Raised when a certificate object's deactivation date is within the threshold value (default 7 days).

This alert is raised only if the certificate object is in the PRE-ACTIVE or ACTIVE state.

When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the certificate object expires.

Deleted if the certificate object is no longer expiring within the threshold value as a result of changes to either object's deactivation date or the configured threshold value. This alert is also deleted when a certificate object is revoked or destroyed.

Cluster FIPS Not Consistent

 MEDIUM

Multi-master cluster only

Cluster-wide

Raised when at least one, but not all, ACTIVE nodes in the cluster are in FIPS mode

Deleted when all cluster nodes are in FIPS mode or all nodes are not in FIPS mode

Cluster Heartbeat Lag

 HIGH

Multi-master cluster only

Node specific

Raised when a node has not received a heartbeat from another ACTIVE node in the cluster for over the threshold value (default 5 minutes)

Deleted when a node has once again received a heartbeat from the other node in the configured threshold period, as long as the node had received a heartbeat from the other node within the last Maximum Disable Node Duration period of time. This alert is also deleted when a node involved has been deleted from the cluster.

Cluster HSM Not Consistent

 MEDIUM

Multi-master cluster only

Cluster-wide

Raised when at least one, but not all, ACTIVE nodes in the cluster are HSM-enabled

Deleted when all nodes are HSM-enabled or all nodes are not HSM-enabled

Cluster Naming Conflict

 LOW

Multi-master cluster only

Cluster-wide

Raised when a naming conflict has been automatically resolved by Oracle Key Vault

Deleted when the object is deleted or renamed, or has had the new name explicitly accepted

Cluster Redo Shipping Status

 HIGH

Multi-master cluster only

Node specific

Raised when a read-write node is unable to ship redo to its read-write peer, and as a result, is in read-only restricted mode. In addition to redo-shipping inactive status information, the alert indicates that the node in the cluster is operating in read-only mode.

Deleted when a read-write node is once again able to ship redo to its read-write peer, or when the node is deleted. The email notification states that the redo-shipping status is back up and the node in the cluster is no longer operating in read-only mode.

Cluster Replication Lag

 HIGH

Multi-master cluster only

Node specific

Raised when incoming replication lag is greater than the threshold value (default 60 seconds)

Deleted when replication lag falls below the threshold value, or when any node in the replication link is deleted.

Disk Utilization

 MEDIUM

Standalone, primary-standby, and multi-master cluster environments

Node specific

Raised when the free disk space percentage of the /var/lib/oracle partition is lower than the threshold value (default 25 percent)

Deleted when free disk space is once again higher than the threshold

Endpoint Certificate Expiration

 HIGH

Standalone, primary-standby, and multi-master cluster environments

Cluster-wide

Raised when an endpoint's certificate is expiring within the threshold value (default 30 days).

When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the endpoint's certificate expires.

Deleted when the endpoint's certificate is no longer expiring within the threshold value or when the endpoint is deleted

Failed System Backup

 MEDIUM

Standalone, primary-standby, and multi-master cluster environments

Node specific

Raised when the last backup did not complete successfully

Deleted when the most recent backup completed successfully

Failed OKV Services

CRITICAL

Standalone, primary-standby, and multi-master cluster environments

Node Specific

Raised when DB, KMIP, REST, Email, Cluster, or Audit Vault service stops because of a failure.

Deleted when all of the failed services start running successfully.

Fast Recovery Area Space Utilization

 HIGH

Standalone, primary-standby, and multi-master cluster environments

Node specific

Raised when Fast Recovery Area Space utilization of Oracle Key Vault's embedded database exceeds the configured threshold value (default 70 percent).

To remedy this problem, try the following:

Reduce the Maximum Disable Node Duration setting of the cluster node. Minimize the duration when peer node is not available. Consider deleting the node from the cluster and adding it back later.

Deleted when Fast Recovery Area Space utilization of Oracle Key Vault's embedded database is once again within the configured threshold
High CPU Usage

HIGH

Standalone, primary-standby, and multi-master cluster environments

 Node Specific

Raised when average memory usage is greater than the threshold in last 5 minutes. Default value of threshold is 99%.

Setting a threshold above 90% takes memory swapping into consideration along with memory usage for raising an alert.

Deleted when 24 hours are passed since the first alert is raised and the CPU utilization is less than the threshold in last 5 minutes.

High Memory Usage

HIGH

Standalone, primary-standby, and multi-master cluster environments

 Node Specific

Raised when average memory usage is greater than the threshold in last 5 minutes. Default value of threshold is 99%.

Deleted when 24 hours are passed since the first alert is raised and the memory usage is less than the threshold in the last 5 minutes.

Invalid HSM Configuration

 CRITICAL

Standalone, primary-standby, and multi-master cluster environments

Node specific

Raised when there is an error in the HSM configuration (checked by default every 5 minutes)

Deleted when there is no longer an error in the HSM configuration

Key Rotations

 HIGH

Standalone, primary-standby, and multi-master cluster environments

Cluster-wide

Raised when a key's deactivation date is within the threshold value (default 7 days)

This alert is raised only if the key object is in the PRE-ACTIVE or ACTIVE state.

When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the certificate object expires.

Deleted if the key object is no longer expiring within the threshold value as a result of changes to either object's deactivation date or the configured threshold value. This alert is also deleted when a key object is revoked or destroyed.
OKV CA Certificate Expiration CRITICAL

Standalone, primary-standby, and multi-master cluster environments

Node specific

Raised when the Oracle Key Vault CA certificate is expiring within the threshold value (default 90 days).
When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the CA certificate expires.Be aware that if the CA certificate expires, then endpoints will no longer be able to communicate with Oracle Key Vault. This will result in downtime.

Deleted when the CA certificate is no longer expiring within the threshold value
OKV Server/Node Certificate Expiration CRITICAL


Standalone, primary-standby, and multi-master cluster environments

Node specific
  • Raised in a standalone or primary-standby deployment when the Oracle Key Vault server certificate is expiring within the threshold value (default 90 days)

  • Raised in a multi-master cluster environment when a node’s node certificate is expiring within the threshold value (default 90 days).

  • Be aware that if the server certificate expires in a standalone or primary-standby deployment, then endpoints will no longer be able to communicate with the Oracle Key Vault server. This will result in downtime for all endpoints.

  • If a node certificate expires in a multi-master cluster environment, endpoints will be able to use other nodes for endpoint operations (like fetching a key). However, the node will no longer be able to communicate with other multi-master cluster nodes, and operations like creating a new wallet will be impacted

Deleted when the server or node certificate is no longer expiring within the threshold value.
OKV Platform Certificates Expiration CRITICAL

Standalone, primary-standby, and multi-master cluster environments

Node specific
Raised when the Oracle Key Vault platform certificates are expiring within the threshold value (default 90 days). When an expiration alert is within its threshold, an email notification is sent once every 24 hours. The platform certificates are used when a new node is added to the cluster. The node whose platform certificates have expired cannot add any new node. Platform certificates are also used when shipping redo between read/write nodes of a cluster. The expired platform certificates may cause the redo shipping to fail resulting in read/write nodes of a cluster to go in read-only restricted mode.

Note:

Although expired platform certificates may impact communication between a node and its read/write peer, they do not impact endpoint communications with Oracle Key Vault.
You cannot upgrade an Oracle Key Vault node with the expired platform certificates.

Deleted when the platform certificates are no longer expiring within the threshold value.

OKV Server Certificate Expiration

  

Standalone, primary-standby, and multi-master cluster environments

Node specific

Raised when the Oracle Key Vault server certificate is expiring within the threshold value (default 90 days).

When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the certificate expires.

Be aware that if the server certificate expires, then endpoints will no longer be able to communicate with the Oracle Key Vault server. This will result in downtime.

Deleted when the server certificate is no longer expiring within the threshold value

Primary-Standby Data Guard Broker Status

 HIGH

Primary-standby only

-

Raised when the Oracle Data Guard Broker status is not ENABLED

Deleted when the broker status is once again ENABLED or when Oracle Key Vault is no longer in primary-standby mode

Primary-Standby Data Guard Fast-Start Failover Status

 MEDIUM

Primary-standby only

-

Raised when the fast-start failover status is not SYNCHRONIZED

Deleted when the fast-start failover status is once again SYNCHRONIZED or when Oracle Key Vault is no longer in a primary-standby configuration

Primary-Standby Destination Failure

 HIGH

Primary-standby only

-

Raised when the switchover status is FAILED DESTINATION

Deleted when the switchover status is no longer FAILED DESTINATION or when Oracle Key Vault is no longer in a primary-standby configuration

Primary-Standby Restricted Mode

 HIGH

Primary-standby only

-

Raised when in primary-standby environment and the primary is running in read-only restricted mode

Deleted when the primary is no longer in read-only restricted mode or when Oracle Key Vault is no longer in a primary-standby configuration

Primary-Standby Role Change

 LOW

Primary-standby only

-

Raised when there is a role change

Deleted when Oracle Key Vault is no longer in a primary-standby configuration

Secret Object Expiration

 HIGH

Standalone, primary-standby, and multi-master cluster environments

Cluster-wide

Raised when a secret object's deactivation date is within the threshold value (default 7 days). This alert is raised only if the object is in the PRE-ACTIVE or ACTIVE state.

When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the secret object expires.

Deleted if the secret object is no longer expiring within the threshold value as a result of changes to either object's deactivation date or the configured threshold value. This alert is also deleted when a secret object is revoked or destroyed.

SSH Tunnel Failure

 HIGH

Standalone, primary-standby, and multi-master cluster environments

Node specific

Raised when an SSH tunnel is not available

Deleted when the SSH tunnel is once again available or when the SSH tunnel is deleted

System Backup

 MEDIUM

Standalone, primary-standby, and multi-master cluster environments

Node specific

Raised when the last successful backup is over the threshold value (default 14 days)

Deleted when the last successful backup was within the threshold value

User Password Expiration

 MEDIUM

Standalone, primary-standby, and multi-master cluster environments

Cluster-wide

Raised when a user's password expires within the threshold value (default 14 days).

When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the password expires.

If the user password expires, user cannot login and Administrative tasks cannot be performed.

Deleted when a user's password no longer expire within the threshold value, or when the user is deleted

Related Topics

Parent topic: Configuring Oracle Key Vault Alerts

22.2.2 Configuring Alerts

You can configure alerts in the Reports page of the Oracle Key Vault management console.

  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Access the Alerts page by using one of the following methods:
    • Select the System tab, then Settings, and in the Monitoring and Alerts area, click Alerts.
    • Select the Reports tab, and then select Alert from the left navigation bar. In the Alerts page, click Configure.
    • On the Home page, expand Alerts at the top of the page, and then click All Alerts. Then click Configure.
    The Configure Alerts page appears, listing various alert types and for some, configurable threshold limit and unit (such as the days until expiration for key rotations alert type). If you are using a multi-master cluster, then the Configure Alerts page will provide cluster-specific alerts, such as the cluster heartbeat lag, redo shipping status, or whether naming conflicts resolution is enabled. The following image shows how the Configure Alerts page appears in a multi-master cluster environment.

    Note:

    Alerts are assigned severity based on their impact on the Oracle Key Vault and registered endpoints.


    Description of 218_configure_alerts.png follows
    Description of the illustration 218_configure_alerts.png

  3. Check the boxes in the Enabled column to the right of the alert types to enable the alert.
    Then set the threshold value in the box under Limit. This value determines when the alert will be sent. You can uncheck the boxes by alerts that you do not want to appear in the dashboard.

  4. You can apply node or cluster specific configuration for the following alerts:

    • Fast Recovery Area Space Utilization
    • High CPU Usage
    • Failed System Backup
    • High Memory Usage
    • Disk Utilization
    • System Backup

    Note:

    Node specific configuration overrides the configuration set at the cluster level.
  5. Click Save.

Related Topics

Parent topic: Configuring Oracle Key Vault Alerts

22.2.2.1 Guidance for Configuring Alerts

You must consider the following guidelines before configuring the Alerts.

For the alerts which support a configurable alert scope, following is the behavior when either the Cluster or the Node scope is selected.

When the Cluster scope is selected:
  • On changing the alert configuration, the changes are applied to all nodes which have the scope set to Cluster.
  • When an alert is disabled, all previously generated alerts, for selected alert, are deleted from all nodes which have the scope set to Cluster.
When the Node scope is selected:
  • On changing the alert configuration, the changes are applied to the current node only.
  • When an alert is generated on a node where the Node scope is set, the alert message has a post fix "node configuration in effect".
  • When an alert is disabled, the previously generated alert, for selected alert, is deleted from the current node.

For the alerts which support a configurable alert scope, following is the behavior when the scope is changed from Cluster to Node or vice versa.

  • When the selected scope for an alert is changed from Cluster to Node or vice versa, then the previously generated alert, for selected alert, is deleted from the current node.
  • When the selected scope for an alert is changed from Node to Cluster, then the node specific alert configuration is deleted.
  • When the selected scope for an alert is changed from Cluster to Node, the Enabled column and Limit column, for selected alert, on Configure Alerts page get populated with corresponding values from cluster alert configuration.

Parent topic: Configuring Alerts

22.2.3 Viewing Open Alerts

Users can view alerts depending on their privileges.

Users with the System Administrator role can view all alerts. Users without system administrator privileges can only view alerts related to objects they can access.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Access the Alerts page by using one of the following methods:
    • Select the System tab, then Settings, and in the Monitoring and Alerts area, click Alerts.
    • Select the Reports tab, and then select Alerts from the left navigation bar.
    • On the Home page, expand Alerts at the top of the page, and then click All Alerts.

      The Home page is also a convenient way to go immediately to how to solve a problem that an alert raises. Under Alerts, click Show Details, and then in the listing of alert types, click the appropriate link. For example, alerts describing upcoming key expirations will take you to the Alerts configuration page, where only alerts for key rotations are displayed. From there, you can examine the details of keys that are expiring.

The Alerts page appears, displaying all the unresolved alerts. Alerts are listed in the order of their severity and severity based color code scheme. The alerts with higher severity should be resolved first.

When you resolve the issue stated in the alert message, the alerts are automatically removed. To delete an alert message, select it and then click Delete. If the issue that caused the alert still exists, then the alert will be regenerated and appear again in this list.

Description of 21.5_view_alerts.png follows
Description of the illustration 21.5_view_alerts.png

Oracle Key Vault sends all system alerts to the syslog. The following is an example of a system alert in syslog:

Mar 29 18:36:29 okv080027361e7e logger[13171]: No successful backup done for 4 day(s)

The following table lists the conditions that trigger alerts, and the accompanying system alert message:

Condition System Alert Message

Certificate Object Expiration

Certificate object unique_ID expiration: date

Cluster FIPS Not Consistent

At least one, but not all, active OKV nodes are in FIPS Mode.

Cluster Heartbeat Lag

Replication lag from node node_name to node current_node_name exceeds threshold_value seconds

Cluster HSM Not Consistent

At least one, but not all, active OKV nodes are HSM-enabled.

Cluster Naming Conflict

Naming conflict for object_type: object_name

The label object_type can be endpoint, endpoint group, user, or user group, with object_name being the corresponding object.

Cluster Redo Shipping Status

Any of the following messages:

  • No heartbeats received from node source_node to node current_node.

  • Last heartbeat from node source_node to node current_node was more than threshold minutes ago.

  • Last heartbeat from node source_node to node current_node was more than one hour ago.

  • Last heartbeat from node source_node to node current_node was more than six hours ago.

  • Last heartbeat from node source_node to node current_node was more than Maximum_Disable_Node Duration hours (50% of the Maximum Disable Node Duration) ago. Take immediate action to restore communication from node source_node to avoid issues in the cluster.

  • Last heartbeat from node source_node to node current_node was more than Maximum_Disable_Node Duration hours (75% of the Maximum Disable Node Duration) ago. Take immediate action to restore communication from node source_node to avoid issues in the cluster.

  • Last heartbeat from node source_node to node current_node was more than Maximum_Disable_Node Duration hours (Maximum Disable Node Duration) ago. Node current_node may not have received all records from node source_node even if replication is restored.

Cluster Replication Lag

Replication lag from node node_name to node node_name is greater than threshold seconds. Current lag is current seconds.

Disk utilization
When Cluster scope is set:
  • Free disk space is below threshold_value (currently current_value)
  • Free disk space is below threshold_value (currently current_value) - node configuration in effect

Endpoint certificate expiration

Endpoint endpoint_name certificate expiration date
Failed OKV Services service_name service is failed or service_name1, service_name2 services are failed.

Failed system backup
When Cluster scope is set:
  • Most recent backup failed!
When Node scope is set:
  • Most recent backup failed! - node configuration in effect

Fast Recovery Area Space Utilization
When Cluster scope is set:
  • Fast Recovery Area space usage exceeds threshold_value% (currently current_value%)
When Node scope is set:
  • Fast Recovery Area space usage exceeds threshold_value% (currently current_value%) - node configuration in effect
High CPU usage
When Cluster scope is set:
  • CPU usage exceeds threshold_value (currently current_value)
When Node scope is set:
  • CPU usage exceeds threshold_value (currently current_value) - node configuration in effect
High Memory Usage
When Cluster scope is set:
  • Memory usage exceeds threshold_value (currently current_value)
When Node scope is set:
  • Memory usage exceeds threshold_value (currently current_value) - node configuration in effect

Invalid HSM Configuration

HSM configuration error. Please refer to the HSM Alert section in the Oracle Key Vault HSM Integration Guide

Key rotations

Key unique_ID expiration: date

Primary-standby destination failure

One or more standby servers are in an error state. HA destination failure.

Primary-standby Oracle Data Guard Broker status

Data Guard Broker is disabled

Primary-standby Oracle Data Guard fast-start failover status

HA FSFO is not synchronized. FSFO status is HA_status

Primary-standby restricted mode

HA running in read-only restricted mode

Primary-standby role change

HA role changed. Primary IP Address: IP_address

Secret Object Expiration

Secret object unique_ID expiration : date

SSH tunnel failure

SSH tunnel (IP IP_address) is not available

System backup
When Cluster scope is set:
  • No successful backup for number day(s)
When Node scope is set:
  • No successful backup for number day(s) - node configuration in effect

User password expiration

User user_name password expiration: date

Parent topic: Configuring Oracle Key Vault Alerts

22.3 Managing System Auditing

Auditing entails tasks such as capturing audit records in a syslog file or downloading the audit records to a local file.

Parent topic: Monitoring and Auditing Oracle Key Vault

22.3.1 About Auditing in Oracle Key Vault

Oracle Key Vault records and time-stamps all endpoint and user activity.

The audit records include endpoint groups and user groups, from endpoint enrollment and user password reset, to the management of keys and wallets, and changes to system settings and SNMP credentials. The audit trail captures details on who initiated which action, with what keys and tokens, and the result of the action. In addition, it records the success or failure of each action.

Only a user who has the Audit Manager role can manage the audit trail for Oracle Key Vault activity. Each user can see audit records of the objects that the user can access.

Auditing in Oracle Key Vault is enabled by default.

A user with the Audit Manager role can see and manage all the audit records. Other users can see only those audit records that pertain to security objects that they have created, or have been granted access to.

The audit manager can export audit records to view system activity off line. After exporting the records, the audit manager can delete them from the system to free up resources.

Related Topics

Parent topic: Managing System Auditing

22.3.2 Oracle Key Vault Audit Trail

The Oracle Key Vault audit trail captures information about activities that are performed in Oracle Key Vault, such as the name of an action and who performed it.

The following lists the contents of the Oracle Key Vault audit trail.

Table 22-4 Oracle Key Vault Audit Trail

Column Name Description
Event ID

ID to identify each audit operation type uniquely.

Client IP

The IP address of the client host or the IP address of the proxy server between the client and the Oracle Key Vault server that is making its IP address available to Oracle Key Vault

Node ID

ID of the Oracle Key Vault cluster node on which the operation was performed

Node IP Address

IP address of the Oracle Key Vault cluster node on which the operation was performed

Node Name

Name of the Oracle Key Vault cluster node on which the operation was performed

Object

Captures the name of object on which the operation is performed

Object Type

Type of object on which the operation is performed (for example, User, Endpoint)

Operation

Name of the operation performed

Result

Result of the operation indicating whether it was successful or failure

Subject

Captures the name of entity that performed the operation

Subject Type

Type of entity, User or Endpoint

Time

Timestamp of the operation

Parent topic: Managing System Auditing

22.3.2.1 Enabling Auditing and Configuring Syslog to Store Audit Records

You can enable or disable auditing and then configure the Oracle Key Vault syslog to store audit records if the System Administrator has enabled this functionality.

  1. Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
  2. Access the Audit Settings page by using one of the following methods:
    • Select the Systems tab, then Settings in the left navigation bar. In the Monitoring and Alerts area, click Audit.
    • Select the Reports tab, then Audit Trail in the left navigation bar. In the Audit Trail page, click Audit Settings.

    The Audit Settings page appears. The categories that you can configure are as follows:

    • Auto Purge Audit Records
    • Enable Auditing
    • Send Audit Records to Syslog
  3. Next, do the following:
    1. In a multi-master cluster environment, click the arrow on the right that appears next to the Save button to toggle between the current node and the entire cluster. Also, the Auto Purge Audit Records setting can only be configured at the node level only.
    2. Select either the Yes or No option for the auditing.
    3. Click Save.
  4. If syslog is configured, then perform additional steps as needed.
    If syslog is not configured, then the Syslog forwarding to remote machines not enabled error message appears. If this error appears, then enable syslog.
    1. Select the System tab, and then select Settings.
    2. In the Monitoring and Alerts area, select Syslog.
    3. In a multi-master cluster environment, toggle between Node Details - Effective on this Node and Cluster Details by clicking the arrow under the Save button.
    1. Select the protocol to use to transfer syslog files: TCP or UDP.
    2. Enter the IP address of the remote system where the syslog files will be stored.
    3. Click Save.

    Note:

    To avoid any accidental deletion of Audit records it is recommended to use TCP protocol for transferring syslog files.

22.3.3 Viewing Audit Records

To view audit records, access the Oracle Key Vault management console Audit Trail page.

The reports page shows the Audit Trail page by default. The Audit Trail page lists all system activity with details on who performed an operation, when the operation was performed, what object was used to perform the operation, and the result.
  1. Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
  2. Click the Reports tab, and then Audit Trail from the left navigation bar.
    The Audit Trail page appears. Optionally, filter records by selecting the table column heads, and from the drop-down list, select the type of sort order that you want.

    The private key is captured as the object in the audit records for the create key pair operation.

    Each Event ID indicates the specific event uniquely associated with an audited operation. For example, operations associated with login Attempted are grouped under the Event ID 1000. For example, to search the audit records for successful logins, filter the audit records with Event ID of 1001.

    Note:

    For the audit records that are generated prior to Oracle Key Vault 21.6, the Event ID column does not have any value.

Parent topic: Managing System Auditing

22.3.4 Exporting and Deleting Audit Records Manually

Oracle Key Vault audit records are stored in a .csv file.

A user with the Audit Manager role can export the audit trail to a .csv file that can be downloaded to the user's local system. The .csv file contains the same details found in the audit trail on the Reports page. The timestamp in the .csv file reflects the time zone of the particular Oracle Key Vault server whose records were exported. After you export the records, you can delete them from the Oracle Key Vault server to free up space.
  1. Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
  2. Click the Reports tab, then Audit Trail in the left navigation bar.

    The Audit Trail appears.

  3. Click Export/Delete Audit Records on the top right.

    The Export/Delete Audit Records page appears.

    Description of 21_export_delete_audit_records.png follows
    Description of the illustration 21_export_delete_audit_records.png

  4. Select the date by clicking the calendar icon.

    Based on the date that you select, the number of records appears after the Number of records to be exported/deleted label.

  5. Click Export to download the audit records in .csv file format to a local folder.

    After you export the records, you can delete them from Oracle Key Vault to free up resources.

  6. Click Delete to remove the audit records.
  7. Click OK to delete or Cancel to stop.

Parent topic: Managing System Auditing

22.3.5 Deleting Audit Records Automatically

You can configure Oracle Key Vault to automatically delete or purge the audit records that are older than the specified retention period.

The Audit Settings page shows the Auto Purge Audit Records pane. The user with the Audit Manager role can purge the Audit Records.
  1. Log in to the Oracle Key Vault management console as a user who has the audit manager role.
  2. Access the Audit Settings page by using one of the following methods:
    • Select the Systems tab, then Settings in the left navigation bar. In the Monitoring and Alerts area, click Audit.
    • Select the Reports tab, then Audit Trail in the left navigation bar. In the Audit Trail page, click Audit Settings.

    The Audit Settings page appears. The categories that you can configure are as follows:

    • Auto Purge Audit Records
    • Enable Auditing
    • Send Audit Records to Syslog
    • Replicate Audit Records, available in cluster environment.
  3. Select Yes, in the Auto Purge Audit Records pane.
    The Retention Period in Days field appears.
  4. Enter the number of days to retain the audit records. Oracle Key Vault will now periodically purge audit records that are older than the specified number of days.
  5. Click Save.

    Note:

    When Oracle Key Vault is integrated with Oracle Audit Vault, audit records are purged only after they are collected by the Audit Vault.

Related Topics

Parent topic: Managing System Auditing

22.3.6 Oracle Key Vault Audit Event IDs

Oracle Key Vault Audit Event ID identifies the audit operation type.

The Event ID in Oracle Key Vault audit records represents a stable identity that is uniquely associated with an audit operation type.

Table 22-5 Oracle Key Vault Audit Event IDs

Event ID Operation
1000 Login Attempted
1001 Login Attempted and Succeeded
1002 Logout
1003 Logout on Console Timeout
1100 System Recovery: Initiate Recovery Passphrase Change
1101 System Recovery: Reset Recovery Passphrase Change
1102 System Recovery: Change Recovery Passphrase
1103 System Recovery: Reset Administrative Accounts
1104 System Recovery: Manage Administrator
1105 System Recovery: Modify User Account Profile Parameters
1106 System Recovery: Reset User Account Profile Parameters
1107 System Recovery: Modify Enforce Separation of Administrator Roles Parameter
1200 Add User
1201 Delete User
1202 Modify User Attributes
1203 Reset User Password
1204 Change User Password
1205 Password Email Sent
1300 Add Endpoint
1301 Delete Endpoint
1302 Modify Endpoint Attributes
1303 Enroll Endpoint
1304 Enrollment Token Email Sent
1305 Get Enrollment Token
1306 Reenroll Endpoint
1307 Suspend Endpoint
1308 Resume Endpoint
1309 Rotate Endpoint Certificate
1310 Endpoint Certificate Rotation Initiated
1311 Endpoint Certificate Rotation Completed
1312 Modify Endpoint Configuration Parameters
1313 Reset Endpoint Configuration Parameters
1314 Clear Endpoint Configuration Parameters
1315 Modify Endpoint Settings for Keys & Secrets
1316 Reset Endpoint Settings for Keys & Secrets
1317 Clear Endpoint Settings for Keys & Secrets
1400 Modify Endpoint Self Enrollment Setting
1401 Modify Global Endpoint Configuration Parameters
1402 Reset Global Endpoint Configuration Parameters
1403 Modify Global Endpoint Settings for Keys & Secrets
1404 Reset Global Endpoint Settings for Keys & Secrets
2000 Add User Group
2001 Delete User Group
2002 Modify User Group Attributes
2003 Add User Group Member(s)
2004 Drop User Group Member(s)
2005 Add User Group Membership
2006 Drop User Group Membership
2100 Add Endpoint Group
2101 Delete Endpoint Group
2102 Modify Endpoint Group Attributes
2103 Add Endpoint Group Member(s)
2104 Drop Endpoint Group Member(s)
2105 Add Endpoint Group Membership
2106 Drop Endpoint Group Membership
2200 Create Wallet
2201 Delete Wallet
2202 Modify Wallet
2203 Add Wallet Membership
2204 Drop Wallet Membership
2205 Add Wallet Member(s)
2206 Remove Wallet Member(s)
2207 Add Access Mapping
2208 Delete Access Mapping
2209 Modify Access Mapping
2210 Assign Default Wallet
2211 Grant Endpoint Management Privilege
2212 Revoke Endpoint Management Privilege
2213 Grant Endpoint Group Management Privilege
2214 Revoke Endpoint Group Management Privilege
3000 Convert Server to First Cluster Node
3001 Convert Server to Candidate Node
3002 Add Candidate Node
3003 Abort Conversion of Candidate Node
3004 Abort Candidate Node
3005 Add Node to Cluster
3006 Abort Addition of Node to Cluster
3007 Finish Pairing on Candidate Node
3008 Initiate Delete Node
3009 Initiate Force Delete Node
3010 Cleanup Deleted Node
3011 Finish Delete Node
3012 Initiate Enable Node
3013 Finish Enable Node
3014 Initiate Disable Node
3015 Finish Disable Node
3016 Cancel Disable Node
3017 Edit Cluster Subgroup
3018 Start Replication for Node
3019 Stop Replication for Node
3020 Start Cluster Services
3021 Stop Cluster Services
3022 User Name Conflict Resolution
3023 User Group Name Conflict Resolution
3024 Endpoint Name Conflict Resolution
3025 Endpoint Group Name Conflict Resolution
3026 Wallet Name Conflict Resolution
3027 KMIP Attribute Name Conflict Resolution
3900 Initiate Primary-Standby Pairing
3901 Initiate Primary-Standby Unpairing
3902 Primary-Standby Role Switch
4000 Discover Versions
4001 Query
4002 Create
4003 Create Key Pair
4004 Register
4005 Delete
4006 Add Attribute(s)
4007 Delete Attribute(s)
4008 Modify Attribute(s)
4009 Get Attribute(s)
4010 Get Attribute List
4011 Check Object Status
4012 Activate
4013 Revoke
4014 Destroy
4015 Check
4016 Locate
4017 Get
4018 Rekey
4019 Store Endpoint Metadata
4020 Encrypt
4021 Decrypt
4022 Sign
4023 Signature Verify
5000 Power Off
5001 Reboot System
5002 Rebooting the Server Operating with FIPS Mode Enabled
5003 Rebooting the Server Operating with FIPS Mode Disabled
5100 Modify Trace Levels
5101 Download Diagnostics
5200 Configure Alerts
5201 Delete Alert(s)
5202 Critical Alert(s) Ignored
5203 Critical Alert(s) Viewed
5300 Modify Monitoring Settings
5301 Modify Monitoring Settings for Cluster
5302 Clear Monitoring Settings for Node
5303 Modify SNMP Credentials
5400 Enable Auditing
5401 Disable Auditing
5402 Enable Auditing in Cluster
5403 Disable Auditing in Cluster
5404 Modify Audit Settings
5405 Modify Audit Settings for Cluster
5406 Clear Audit Settings for Node
5407 Enable Audit Replication
5408 Disable Audit Replication
5409 Enable Audit Replication in Cluster
5410 Disable Audit Replication in Cluster
5411 Clear Audit Replication Settings for Node
5412 Start Sending Audit Records to SYSLOG
5413 Stop Sending Audit Records to SYSLOG
5414 Start Sending Audit Records to SYSLOG in Cluster
5415 Stop Sending Audit Records to SYSLOG in Cluster
5416 Clear Send Audit Records to SYSLOG Settings for Node
5417 Enable Audit Records Auto Purge
5418 Disable Audit Records Auto Purge
5419 Clear Audit Records Auto Purge Settings for Node
5420 Delete Audit Records
5600 Modify System Settings
5601 Modify Network Details
5602 Modify DNS Settings
5603 Modify DNS Settings for Cluster
5604 Clear DNS Settings for Node
5605 Modify System Time Settings
5606 Modify NTP Settings
5607 Modify NTP Settings for Cluster
5608 Clear NTP Settings for Node
5609 Modify Web Access
5610 Modify SSH Access
5611 Modify Management Console Timeout Settings
5612 Enable FIPS Mode
5613 Disable FIPS Mode
5614 Enable RESTful Services
5615 Disable RESTful Services
5616 Modify Syslog Settings
5617 Modify Syslog Settings for Cluster
5618 Clear Syslog Settings for Node
5619 Configure SMTP
5620 Unconfigure SMTP
5621 Modify SMTP Settings
5622 Modify Maximum Disable Node Duration Settings for Cluster
5623 Add SSH Tunnel
5624 Delete SSH Tunnel
5625 Enable SSH Tunnel
5626 Disable SSH Tunnel
5627 Tunnel Cleanup on Failover
5628 Delete Cloud IP Reservation
6000 Restore
6001 Create Backup
6002 Delete Backup
6003 Edit Backup
6004 Backup Initiated
6005 Backup Completed
6006 Pause Backup
6007 Resume Backup
6008 Purge Backup
6009 Create Backup Destination
6010 Delete Backup Destination
6011 Edit Backup Destination
6012 Reset Backup Host Public Key
6013 Create Backup Destination Policy
6014 Delete Backup Destination Policy
6015 Edit Backup Destination Policy
6016 Suspend Backup Destination Policy
6017 Resume Backup Destination Policy
6100 Generate New Server Certificate
6101 Generate New Node Certificate
6102 Modify Server Certificate Details
6103 Modify Server Certificate Alternate Name and IP Address Details
6104 Modify CA Certificate Details
6105 Generate and Download Intermediate CA Certificate Request
6106 Abort Generation of Intermediate CA Certificate
6107 Download Intermediate CA Certificate Request
6108 Upload Intermediate CA Certificate
6109 Upload Intermediate CA Trust Chain
6110 Generate New CA Certificate
6111 Unpack New CA Certificate Bundle
6112 Abort CA Certificate Rotation
6113 Activate New CA Certificate
6114 Update Endpoint Certificate Rotation Window Size
6115 Update Cluster Subgroup Order
6116 Remove Cluster Subgroup Order
6117 Update Endpoint Group Order
6118 Check Expired CA Certificate Rotation State
6119 Generate New CA Certificate for Manual Recovery
6120 Upload CA Certificate Bundle
6121 CA Certificate Rotation Manual Recovery Completed
6122 CA Certificate Rotation Manual Recovery Verified
6123 Abort CA Certificate Rotation Manual Recovery
6180 Generate Console Certificate
6181 Upload Console Certificate
6182 Remove Console Certificate
6183 Restore Console Certificate
8000 Add LDAP Configuration
8001 Delete LDAP Configuration
8002 Force Delete LDAP Configuration
8003 Test LDAP Configuration
8004 Edit LDAP Configuration
8005 Enable LDAP Configuration
8006 Disable LDAP Configuration
8007 Add LDAP Server
8008 Delete LDAP Server(s)
8009 Test LDAP Server
8010 Test LDAP Connections
8011 Add LDAP Group Access Mapping
8012 Delete LDAP Group Access Mapping
8013 Edit LDAP Group Access Mapping
8014 Remove User Group from LDAP Group Access Mapping
8015 Validate LDAP Group(s) for LDAP Configuration
8016 Validate LDAP User(s) for LDAP configuration
8018 Delete LDAP User
8019 Delete Stale LDAP User
8020 Modify LDAP User Attributes
8021 Download LDAP Log
8100 Add Single Sign-On Configuration
8101 Delete Single Sign-On Configuration
8104 Edit Single Sign-On Configuration
8105 Enable Single Sign-On Configuration
8106 Disable Single Sign-On Configuration
8200 Enable Audit Vault Integration
8201 Disable Audit Vault Integration
8202 Suspend Audit Vault Audit Record Collection
8203 Resume Audit Vault Audit Record Collection
8300 HSM Initialize
8301 HSM Set Credential
8302 HSM Create Bundle
8303 HSM Apply Bundle
8304 HSM Reverse Migrate

Parent topic: Managing System Auditing

22.3.7 Configuring Oracle Key Vault with Oracle Audit Vault

A user who has the Audit Manager role can configure Oracle Key Vault to send audit records to Oracle Audit Vault for centralized audit reporting and alerting.

Parent topic: Managing System Auditing

22.3.7.1 Integrating Oracle Audit Vault with Oracle Key Vault

You can perform this integration from the Oracle Key Vault management console.

Parent topic: Configuring Oracle Key Vault with Oracle Audit Vault

22.3.7.1.1 Step 1: Check the Environment

Before you begin the integration, you should ensure that the required components are all in place.

  1. Ensure that Oracle Audit Vault is properly installed and configured.
    This activity requires administrative access to the Oracle Audit Vault server in order to register Oracle Key Vault as a secured target.
  2. Ensure that you have the credentials of the Oracle Audit Vault administrator in order to register Oracle Key Vault as a secured target in Oracle Audit Vault server. This user does not need to be a super administrator.
  3. Enable SSH access to Oracle Audit Vault.

    Log in to the Oracle Audit Vault Server console as a user who has the Super Administrator role. Select the Settings tab, then System. In the Configuration area, click System Settings and then Web/SSH/SNMP. Turn on SSH Access, select IP addresses and then enter only the IP addresses that you need, or select All. Click Save.

  4. Verify if the user support can SSH in to the Oracle Audit Vault server. For more information, see Step 2: Configure Oracle Key Vault as a Registered Host and a Secured Target with Oracle Audit Vault.
22.3.7.1.2 Step 2: Configure Oracle Key Vault as a Registered Host and a Secured Target with Oracle Audit Vault

A user who has the Audit Manager role must configure the Oracle Key Vault server as a secured target on the Oracle Audit Vault server.

In a multi-master cluster environment, perform these steps on each node. Each node will send the audit records that were generated from that node to the Oracle Audit Vault server.
  1. Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
  2. Select the System tab, and then Audit Vault Integration from the left navigation bar.
    If you also have the System Administrator role, then the navigation will be slightly different. Click System, then in the left navigation pane, select Settings. In the Monitoring and Alerts area, click Audit Vault.
  3. In the Deployment pane, enter the following settings:
    • Hostname: Enter the host name or IP address of the Oracle Audit Vault server.
    • Public Host Key: Enter the public host key of the Oracle Audit Vault server by following the guidelines mentioned in the help text.
    • Support User Password: Enter the support user password of the Oracle Audit Vault server.
    • Administrator Name: Enter the user name of the Oracle Audit Vault server user who has the Administrator role.
    • Administrator Password: Enter the password of the Oracle Audit Vault server user who has the Administrator role.
    • Recovery Passphrase: Enter the recovery passphrase of the Oracle Key Vault server.
  4. Click Deploy.
    The integration may take about 10 minutes to complete. Do not attempt to re-initiate the Audit Vault integration during this interval. The Oracle Key Vault server may become unavailable for some time until the integration completes.
    After the integration completes, a Monitoring tab appears and will show the Audit Vault agent status.
22.3.7.1.3 Aborting an Oracle Audit Vault Integration

You can abort an Oracle Audit Vault integration by using the Oracle Key Vault management console.

The Audit Vault integration with Oracle Key Vault may take about 10 minutes to complete. If the integration gets stuck Oracle Key Vault displays a Server Error: 500.
  1. In the Recovery Passphrase field, under Key Vault Details pane, provide the recovery passphrase.
  2. Under the Progress window pane, on the Audit Vault Details page, click Abort to abort the integration.
  3. Perform the Integration again and start providing the information, see Step 2: Configure Oracle Key Vault as a Registered Host and a Secured Target with Oracle Audit Vault.
    The integration continues. Do not attempt to re-initiate the Audit Vault integration during this interval. The Oracle Key Vault server may become unavailable for some time until the integration completes.
    After the integration completes, a Monitoring tab appears and will show the Audit Vault agent status.
22.3.7.2 Viewing Oracle Key Vault Audit Data Collected by Oracle Audit Vault

You can use the Oracle Audit Vault Server console to view data that is collected by Oracle Key Vault and Oracle Audit Vault.

  1. Log in to the Oracle Audit Vault Server console as an auditor.
  2. Select the Reports tab.
  3. Select Activity Reports.
  4. Select All Activity.
  5. Filter the target to get all records belonging to okv_db_Oracle_Key_Vault_IP_address.
    Assuming that you filtered for the target okv_db_192.0.2.78, the report could be similar to the following: Description of avdf_all_activity.png follows
    Description of the illustration avdf_all_activity.png
22.3.7.3 Suspending an Oracle Audit Vault Monitoring Operation

You can suspend an Oracle Audit Vault monitoring operation from the Oracle Key Vault management console.

  1. Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
  2. Select the System tab, and then Audit Vault Integration in the left navigation bar.
    If you also have the System Administrator role, then the navigation will be slightly different. Click System, then in the left navigation pane, select Settings. In the Monitoring and Alerts area, click Audit Vault.
  3. Select the Monitoring tab.
  4. In the Audit Vault pane, click Suspend.
22.3.7.4 Resuming an Oracle Audit Vault Monitoring Operation

You can resume a suspended Oracle Audit Vault monitoring operation from the Oracle Key Vault management console.

  1. Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
  2. Select the System tab, and then Audit Vault Integration in the left navigation bar.
    If you also have the System Administrator role, then the navigation will be slightly different. Click System, then in the left navigation pane, select Settings. In the Monitoring and Alerts area, click Audit Vault.
  3. Select the Monitoring tab.
  4. In the Audit Vault pane, click Resume.
22.3.7.5 Deleting an Oracle Audit Vault Integration

You can delete an Oracle Audit Vault integration by using the Oracle Key Vault management console.

  1. Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
  2. Select the System tab, and then Audit Vault Integration in the left navigation bar.
    If you also have the System Administrator role, then the navigation will be slightly different. Click System, then in the left navigation pane, select Settings. In the Monitoring and Alerts area, click Audit Vault.
  3. Under Deployment, in the Audit Vault Details page, enter the following settings:
    • Public Host Key: Enter the public host key of the Oracle Audit Vault server by following the guidelines mentioned in the help text.
    • Support User Password: Enter the support user password of the Oracle Audit Vault server.
    • Administrator Name: Enter the user name of the Oracle Audit Vault server user who has the Administrator role.
    • Administrator Password: Enter the password of the Oracle Audit Vault server user who has the Administrator role.
  4. In the Deployment pane, click Delete.
  5. Click OK to confirm.
    Audit records that are already collected in Oracle Audit Vault are not affected when Oracle Key Vault integration with Oracle Audit Vault is deleted.
  6. Because you no longer need to copy files from one server to another, disable SSH access to Oracle Audit Vault.

    Log in to the Oracle Audit Vault Server console as a user who has the Super Administrator role. Select the Settings tab, then System. In the Configuration area, click System Settings and then Web/SSH/SNMP. Turn off SSH Access. Click Save.

22.3.7.6 Guidance for Integrating Oracle Audit Vault in a Multi-Master Cluster or Primary-Standby Environment

You must follow special guidelines to integrate Oracle Audit Vault with Oracle Key Vault in a multi-master cluster or primary-standby environment.

Multi-Master Cluster Environments

  • If Oracle Key Vault is configured to use multi-master clusters, then you must perform the Oracle Audit Vault integration individually on each node. Each node will send audit records that are generated only on that node to Oracle Audit Vault irrespective of whether audit record replication is enabled.

Primary-Standby Environments

  • Perform the integration only on the primary server, not the standby server.
  • If you must perform a switchover operation, then note the following:
    • You must switch back to the primary server if you want to suspend, resume, or delete the integration. You do not need to perform additional steps.
    • To integrate the new primary server with Oracle Audit Vault, optionally, ensure that you use the same Oracle Audit Vault host name and administrator credentials that were used in the old primary server.
    • If you perform an unpair operation after performing a switchover operation, then you must perform a new Oracle Audit Vault integration with the new primary server.
    • If you delete the new integration, then the old integration becomes non-functional. You must then delete this old integration by switching back to the old primary server.
  • If a failover operation occurs and the original primary server is no longer available, then you must perform a new Oracle Audit Vault integration with the new primary server.
  • However, if the original primary server is not lost and it is possible to bring back the original primary server as the new standby server, then you do not need to perform additional steps.

22.4 Using Oracle Key Vault Reports

Oracle Key Vault collects statistical information on a range of activities that impact Key Vault operations.

Parent topic: Monitoring and Auditing Oracle Key Vault

22.4.1 About Oracle Key Vault Reports

The reports cover system activity, certificate expiration, keys, passwords, entitlement status, extraction status, and metadata.

Oracle Key Vault provides seven types of reports for endpoints, users, keys and wallets, SSH keys configuration and usage, and system. In a multi-master cluster, some reports contain additional information, such as the node ID, node name, and IP address.

The seven report types are as follows:

  • Key management reports for Oracle endpoints, which includes information about TDE master encryption keys, GoldenGate master keys, and ACFS volume encryption key details

  • Keys and wallets reports list the access privileges granted to all keys and wallets, and the details of TDE master encryption keys managed by Oracle Key Vault

  • Secrets management reports for database passwords, secret data, and opaque objects

  • SSH reports for SSH user key management, SSH server access management and detailed information on SSH user keys inventory, authorization, and usage.

  • Endpoint reports contain details of all endpoint and endpoint group activity, certificate and password expiration, and access privileges

  • User reports contain details of all user and user group activity, their certificate and password expiration, and access privileges

  • System reports contain a history of system backups taken and scheduled, details of remote restoration points, and RESTful command-line interface usage

A user who has the Audit Manager role can view all reports, including reports that are accessible from the Audit Trail pages in the Oracle Key Vault management console. A user with the Key Administrator role can view user reports and keys and wallets reports. Users with the System Administrator role can view endpoint, user, and system reports.

Reports may include additional columns that are hidden by default. To include such columns in the displayed report, Click on Actions, then on Select Columns. The Select Columns dialog box appears. Select the columns that you want to include in the report from the list shown under Do Not Display section and move them to the list shown under Display In Report section by clicking on Move button (shown as >). Likewise, you can also remove columns from the report.

To view the reports:
  1. Log in to the Oracle Key Vault management console.
  2. Click the Reports tab and then Reports from left navigation bar.
  3. Expand the report type to see the corresponding reports.

Related Topics

Parent topic: Using Oracle Key Vault Reports

22.4.2 Viewing Key Management Reports for Oracle Endpoints

All users can view the key management reports for Oracle endpoints.

  1. Log in to the Oracle Key Vault management console.
  2. Click the Reports tab and then Reports from left navigation bar.
  3. Expand Key Management Reports for Oracle Endpoints Reports.

    The Key Management Reports for Oracle Endpoints page appears displaying the five types of reports.

    Description of 21_key_management_oracle_endpoints_report.png follows
    Description of the illustration 21_key_management_oracle_endpoints_report.png

  4. Click the report name to see the corresponding user report.

Parent topic: Using Oracle Key Vault Reports

22.4.3 Viewing Keys and Wallets Reports

The keys and wallets reports require different privileges for viewing, depending on the report.

  1. Log in to the Oracle Key Vault management console as a user who has the appropriate privileges.
    Only a user who has the Key Administrator role or Audit Manager role can view the Wallet Entitlement Report. All users can view the remaining reports.
  2. Click the Reports tab and then Reports from left navigation bar.
  3. Expand Keys and Wallets Reports.

    The Keys and Wallets Reports page appears displaying the reports.

  4. Click the report name to see the corresponding report.

Parent topic: Using Oracle Key Vault Reports

22.4.4 Viewing Secrets Management Reports

All users can view the secrets management reports.

  1. Log in to the Oracle Key Vault management console.
  2. Click the Reports tab and then Reports from left navigation bar.
  3. Expand Secrets Management Reports.

    The Secrets Management Reports page appears displaying the reports.

  4. Click the report name to see the corresponding report.

Parent topic: Using Oracle Key Vault Reports

22.4.5 Viewing SSH Reports

All users can view the SSH reports.

  1. Log in to the Oracle Key Vault management console.
  2. Click the Reports tab and then Reports from left navigation bar.
  3. Expand SSH Reports.

    The SSH Reports page appears displaying the reports.

  4. Click the report name to see the corresponding report.

Parent topic: Using Oracle Key Vault Reports

22.4.6 Viewing Endpoint Reports

You must have the System Administrator role or the Audit Manager role to view the five categories of endpoint reports.

Oracle Key Vault offers five endpoint reports: Endpoint Activity, Endpoint Entitlement, Endpoint Certificate Expiry, Endpoint Metadata, and Sign & Signature Verify Activity.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role or the Audit Manager role.
  2. Click the Reports tab and then Reports from left navigation bar.
  3. Expand Endpoint Reports.
  4. Select the name of the report that you want to view.

    For example, the Activity Report for endpoints appears similar to the following:

    Description of 21_endpoint_report.png follows
    Description of the illustration 21_endpoint_report.png

Parent topic: Using Oracle Key Vault Reports

22.4.7 Viewing User Reports

You must have the System Administrator role, the Key Management role, or the Audit Manager role to view the four categories of user reports.

Oracle Key Vault offers four user reports: User Activity, User Entitlement, User Expiry, and User Failed Login.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role, the Key Management role, or the Audit Manager role.
  2. Click the Reports tab and then Reports from left navigation bar.
  3. Expand User Reports to see user-specific reports.
  4. Click the report name to see the corresponding user report.

Parent topic: Using Oracle Key Vault Reports

22.4.8 Viewing System Reports

You must have the System Administrator role or the Audit Manager role to view the system reports.

Oracle Key Vault offers three system reports: Backup History, Notification, and RESTful Services Usage.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role or the Audit Manager role.
  2. Click the Reports tab and then Reports from left navigation bar.
  3. Expand System Reports.

    The System Reports page appears displaying the system reports available.

    Description of 21_system_report.png follows
    Description of the illustration 21_system_report.png

  4. Click the report type to see the corresponding system report.

    Note:

    For a multi-master cluster configuration, additional reports are available to monitor naming conflicts in the cluster. To view the Conflict Resolution reports, click the Cluster tab, and then Conflict Resolution from the left navigation bar.

Parent topic: Using Oracle Key Vault Reports