Monitoring and Auditing Oracle Key Vault

20 Monitoring and Auditing Oracle Key Vault

Oracle Key Vault administrators can monitor and audit the Oracle Key Vault system, configure alerts and use reports.

Managing System Monitoring
System monitoring refers to tasks such as configuring SNMP connections, email notifications, the syslog destination, and system diagnostics.
Configuring Oracle Key Vault Alerts
You can select the type of alerts that you want to see in the Oracle Key Vault dashboard.
Managing System Auditing
Auditing entails tasks such as capturing audit records in a syslog file or downloading the audit records to a local file.
Using Oracle Key Vault Reports
Oracle Key Vault collects statistical information on a range of activities that impact Key Vault operations.

20.1 Managing System Monitoring

System monitoring refers to tasks such as configuring SNMP connections, email notifications, the syslog destination, and system diagnostics.

Configuring Remote Monitoring to Use SNMP
With Simple Network Management Protocol (SNMP) enabled, system administrators can remotely monitor the Oracle Key Vault appliance and its services.
Configuring Email Notification
You can use email notifications to directly notify administrators of Key Vault status changes without logging into the Oracle Key Vault management console.
Configuring the Syslog Destination for Individual Multi-Master Cluster Nodes
On each node, you can forward syslog entries to a remote service such as Splunk or SIEM.
Capturing System Diagnostics
To troubleshoot problems that may arise, you can generate a system diagnostics file.
Monitoring System Metrics
You can use the System Metrics Monitoring feature to view and collect data for key system resource usage including CPU and Memory Usage in Oracle Key Vault.

Parent topic: Monitoring and Auditing Oracle Key Vault

20.1.1 Configuring Remote Monitoring to Use SNMP

With Simple Network Management Protocol (SNMP) enabled, system administrators can remotely monitor the Oracle Key Vault appliance and its services.

The collected data can be further processed and presented for the needs of the enterprise.

About Using SNMP for Oracle Key Vault
You can use the Simple Network Management Protocol (SNMP) to monitor devices on a network for resource usage.
Granting SNMP Access to Users
You can grant any user, including users who are not Oracle Key Vault administrators, access to SNMP data.
Changing the SNMP User Name and Password
You can change the SNMP user name and password for a node at any time.
Changing SNMP Settings on the Standby Server
You change the SNMP settings from the command line on the standby server.
Remotely Monitoring Oracle Key Vault Using SNMP
SNMP enables you to monitor the vital components of Oracle Key Vault remotely without having to install new software in Oracle Key Vault.
SNMP Management Information Base Variables for Oracle Key Vault
Oracle Key Vault provides a set of SNMP Management Information Base (MIB) variables that you can track.
Example: Simplified Remote Monitoring of Oracle Key Vault Using SNMP
In Linux, you can simplify the SNMP commands you manually enter to find Oracle Key Vault information, yet still have useful and detailed output.

Parent topic: Managing System Monitoring

20.1.1.1 About Using SNMP for Oracle Key Vault

You can use the Simple Network Management Protocol (SNMP) to monitor devices on a network for resource usage.

Monitoring Oracle Key Vault is an important aspect how critical Oracle Key Vault's availability is when hundreds or thousands of Oracle and MySQL databases store their TDE master encryption keys in an Oracle Key Vault multi-master cluster. The types of resource usage that you should monitor include memory, CPU utilization, and processes. Even though Oracle Key Vault provides continuous key availability by allowing up to 16 (geographically distributed) instances to be connected to a single cluster, the health of each individual node contributes to the performance and availability of the entire cluster.

You can use Simple Network Management Protocol (SNMP) third-party tool to monitor remote systems that access Oracle Key Vault. The benefits of using SNMP to monitor Oracle Key Vault are as follows:

There is no need to allow SSH access to Oracle Key Vault. (SSH access should only be enabled for the window of time in which it is being used.)
You do not need to install additional tools to perform an SNMP monitoring operation.

Oracle Key Vault uses SNMP version 3 for user authentication and data encryption features. Unlike SNMP versions 1 and 2 that communicate in readable, insecure plaintext, SNMP 3 authenticates users and encrypts data on the communication channel between the monitoring server and the target. The information from Oracle Key Vault is unreadable to an intruder, even if the communication channel is intercepted.

In addition, with SNMP enabled on Oracle Key Vault, you can determine whether the key management server (KMIP daemon) is running. To track this information, you must use a third-party SNMP client to poll the Oracle Key Vault instance, because Oracle Key Vault does not provide SNMP client software.

Oracle Key Vault audits the creation and modification of SNMP credentials.

You must be a user with the System Administrator role to configure the SNMP account with a user name and password. These SNMP credentials are needed to access SNMP data.

In a multi-master cluster, the SNMP account with a user name and password can be set for all nodes of the cluster at once. It can also be set for each individual node.

Note:

You must ensure that the SNMP username and password is not the same username and password as any of the Oracle Key Vault administrative user accounts with the System Administrator, Key Administrator, or Audit Manager role.

Parent topic: Configuring Remote Monitoring to Use SNMP

20.1.1.2 Granting SNMP Access to Users

You can grant any user, including users who are not Oracle Key Vault administrators, access to SNMP data.

Log in to the Oracle Key Vault management console as a user with the System Administrator role.
Select the System tab, and then select Settings from the left navigation bar.
In the Monitoring and Alerts area, click SNMP.

The Monitoring page appears.

Description of the illustration 21_snmp_grant_access.png
In the Monitoring Settings page, enter the following information:
- SNMP Access allowed from: Select All to enable a client at any IP address to poll Oracle Key Vault for information, Disabled to prevent any client, regardless of the client IP address, to poll Oracle Key Vault for information, or IP Address(es) if you want to restrict polling to clients with specific IP addresses. If you select IP Address(es), then enter the IP addresses of the users you want to grant access to in the IP Address field. Separate multiple IP addresses by a space. You cannot enter a range of IP addresses. You must list each IP address individually.
- User Name: Enter a name to associate with the SNMP configuration that will perform the monitoring.
- Password and Re-enter Password: Enter a secure password for this user that is at least 8 or more characters and contains at least one of each of the following: an uppercase letter, a lowercase letter, a number, and a special character from the set: period (.), comma (,), underscore (_), plus sign (+), colon (:), space. The SNMP password must not be the same as the password used to login into the Oracle Key Vault management console in any of the administrative roles.
Click Save.

Parent topic: Configuring Remote Monitoring to Use SNMP

20.1.1.3 Changing the SNMP User Name and Password

You can change the SNMP user name and password for a node at any time.

Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
Select the System tab, and then select Settings from the left navigation bar.
In the Monitoring and Alerts area, click SNMP.
In the User Name, Password, and Re-enter Password fields, enter the user name and password information.
Click Save.

Parent topic: Configuring Remote Monitoring to Use SNMP

20.1.1.4 Changing SNMP Settings on the Standby Server

You change the SNMP settings from the command line on the standby server.

To add SNMP support in a primary-standby environment, you should configure SNMP on both the primary and standby servers before pairing them. This is because the standby server is no longer accessible from the Oracle Key Vault management console because all requests are forwarded to the primary server. However, you can change SNMP settings on the standby server in a primary-standby environment.

Log in to the standby server as the support user.
Switch to the root user.
```
su -
```
Go to the Oracle Key Vault bin directory.
```
cd /usr/local/okv/bin/
```
Run the stdby_snmp_enable script.
```
./stdby_snmp_enable parameter "options"
```
In this specification:
- parameter can be the following:
  - -a, which sets the SNMP access. It accepts the following options:
    
    all grants SNMP access.
    
    disabled disables SNMP access.
    
    IP_addresses specifies one or more IP addresses to be granted SNMP access. Separate each IP address with a space.
  - -u sets the user's SNMP name. This is the user name that was configured as the snmpuser when SNMP was enabled.
  - -p sets the user's SNMP password. This password was created when for the snmpuser when SNMP was enabled.
- options is only used with the -a parameter.

The following examples show how to change SNMP settings on a standby server:

To grant SNMP access to all IP addresses and assign a user name snmpuser and password password:

./stdby_snmp_enable -a "all" -u "snmpuser" -p "password"

To disable SNMP access from all IP addresses:

./stdby_snmp_enable -a "disabled"

To grant SNMP access to certain IP addresses and assign user name snmpuser and password password:

./stdby_snmp_enable -a "192.0.2.1 192.0.2.3 192.0.2.3" -u "snmpuser" -p "password"

Parent topic: Configuring Remote Monitoring to Use SNMP

20.1.1.5 Remotely Monitoring Oracle Key Vault Using SNMP

SNMP enables you to monitor the vital components of Oracle Key Vault remotely without having to install new software in Oracle Key Vault.

Though there are third-party tools that graphically display the information that SNMP extracts from Oracle Key Vault, the examples shown here are given with snmpwalk and snmpget from the command line on a remote computer that has a network connection into the SNMP account in Oracle Key Vault.

Log in to the remote host that will monitor Oracle Key Vault.
Confirm that the UCD-SNMP-MIB is installed on the remote host from which Oracle Key Vault is monitored.

Query the object ID for an Oracle Key Vault-supported SNMP Management Information Base (MIB) variable.

For example, suppose you wanted to track the number of processes running for the SNMP host. You can use a third-party SNMP client utility to query the status of the KMIP MIB whose object ID is 1.3.6.1.4.1.2021.2, as follows:

third_party_snmp_client_command -v 3 OKV_IP_address -u SNMP_user -a SHA -A SNMP_password -x AES -X SNMP_password -l authPriv iso.3.6.1.4.1.2021.2.1.2

The output is similar to the following:

iso.3.6.1.4.1.2021.2.1.2.1 = STRING: "mwecsvc"              <== Event collector
iso.3.6.1.4.1.2021.2.1.2.2 = STRING: "httpd"                <== httpd
iso.3.6.1.4.1.2021.2.1.2.3 = STRING: "kmipd"                <== KMIP daemon
iso.3.6.1.4.1.2021.2.1.2.4 = STRING: "ora_pmon_dbfwdb"      <== embedded DB
iso.3.6.1.4.1.2021.2.1.2.5 = STRING: "ServiceManager"       <== Golden Gate Service Manager (Monitors other processes and reports status)
iso.3.6.1.4.1.2021.2.1.2.6 = STRING: "adminsrvr"            <== Golden Gate Admin Server (Communicates with the DB to perform certain maintenance/admin tasks)
iso.3.6.1.4.1.2021.2.1.2.7 = STRING: "distsrvr"             <== Golden Gate Distribution Server (Sends the OGG changes to other nodes)
iso.3.6.1.4.1.2021.2.1.2.8 = STRING: "recvsrvr"             <== Golden Gate Receiver Server

Related Topics

SNMP Management Information Base Variables for Oracle Key Vault

Parent topic: Configuring Remote Monitoring to Use SNMP

20.1.1.6 SNMP Management Information Base Variables for Oracle Key Vault

Oracle Key Vault provides a set of SNMP Management Information Base (MIB) variables that you can track.

The following table lists the MIB variables that are supported.

Table 20-1 MIBs That SNMP Tracks for Oracle Key Vault

MIB Variable	Object ID	Description
`hrSystemUptime`	1.3.6.1.2.1.25.1.1	Tracks the amount of time that an Oracle Key Vault instance has been running
`ifAdminStatus.x`	1.3.6.1.2.1.2.2.1.7	Tracks if the Oracle Key Vault network interface (`x`) are running, not running, or being tested. Values are as follows: 1: Instance is running 2: Instance is down 3: Instance is being tested
`memAvailReal`	1.3.6.1.4.1.2021.4.6	Tracks the available RAM
`memTotalReal`	1.3.6.1.4.1.2021.4.5	Tracks the total amount of RAM being used
`ssCpuRawIdle`	1.3.6.1.4.1.2021.11.53	For CPU monitoring; tracks the number of ticks (typically 1/100s) spent idle
`ssCpuRawInterrupt`	1.3.6.1.4.1.2021.11.56	For CPU monitoring; tracks the number of ticks (typically 1/100s) spent processing hardware interrupts
`ssCpuRawKernel`	1.3.6.1.4.1.2021.11.55	For CPU monitoring; tracks the number of ticks (typically 1/100s) spent processing kernel-level code
`ssCpuRawNice`	1.3.6.1.4.1.2021.11.51	For CPU monitoring; tracks the number of ticks (typically 1/100s) spent processing reduced-priority code
`ssCpuRawSystem`	1.3.6.1.4.1.2021.11.52	For CPU monitoring; tracks the number of ticks (typically 1/100s) spent processing system-level code
`ssCpuRawUser`	1.3.6.1.4.1.2021.11.50	For CPU monitoring; tracks the number of ticks (typically 1/100s) spent processing user-level code
`ssCpuRawWait`	1.3.6.1.4.1.2021.11.54	For CPU monitoring; tracks the number of ticks (typically 1/100s) spent waiting for input-output (IO)
`UCD-SNMP-MIB.prTable`	1.3.6.1.4.1.2021.2	Tracks the number of processes running under a certain name. Names we monitor are `httpd` (the http server), `kmipd` (the kmip daemon), and `ora_pmon_dbfwdb (an indicator if the DB is down)`
`nsExtendOutputFull`	1.3.6.1.4.1.8072.1.3.2.3.1.2	For monitoring Fast Recovery Area Space utilization; tracks the total limit, space used, and space free. The alert also shows the CA or server certificate expiration date and time (whichever expires sooner) as well as the status of the Oracle Audit Vault agent and the Oracle Audit Vault monitor. For the certificate expiration, the time zone that is shown for the date and time is in UTC. The output may be inconsistent if Oracle Key Vault is in the middle of a certification rotation.

20.1.1.7 Example: Simplified Remote Monitoring of Oracle Key Vault Using SNMP

In Linux, you can simplify the SNMP commands you manually enter to find Oracle Key Vault information, yet still have useful and detailed output.

The configuration in this section assumes that you have granted SNMP access to a trusted user. It also assumes that the you have installed the SNMP Management Information Base (MIB) variables on the remote host that will monitor Oracle Key Vault.

For example, a lengthy version of the snmpwalk command for an SNMP user named snmp_admin is as follows:

snmpwalk -v3 OKV_IP_address -n "" -l authPriv -u snmp_admin -a SHA -A snmp_user_password -x AES -X snmp_user_password

This command lists the vital services that are running on Oracle Key Vault. However, you can modify the command (and other SNMP commands) to be not only shorter, but to show additional information, such as whether the services are running or not running.

To simplify this type of command, you can edit the /etc/snmp/snmp.conf configuration file so that the SNMP commands you enter will automatically include commonly used settings, such as the default user or the default security level. The example in this topic omits password parameters so that users can enter the password at the command line interactively.

Edit the /etc/snmp/snmp.conf, which appears as follows:

# As the snmp packages come without MIB files due to license reasons, 
# loading MIBs is disabled by default. If you added the MIBs you 
# can reenable loading them by commenting out the following line. 
  mibs :

Comment out the # mibs : line and then add the following lines, as follows:
```
# loading MIBs is disabled by default. If you added the MIBs you 
# can reenable loading them by commenting out the following line. 
# mibs : 
defSecurityName snmp_admin 
defSecurityLevel authPriv 
defAuthType SHA 
defPrivType AES 
```
In this example:
- defSecurityName: Enter the name of the user to whom you granted SNMP access. This example uses snmp_admin.
- defSecurityLevel: Enter the default security level to use. This example uses authPriv, which enables communication with authentication and privacy.
- defAuthType: Enter the default authorization type. This example uses SHA.
- defPrivType: Enter the default privilege type. This example uses AES.
Restart snmpd to load the configuration file.
For example, for Linux 7:
```
systemctl restart snmpd
```
For Linux 6:
```
service snmpd restart
```

To run the simplified version of the snmpwalk command that was shown earlier, enter the following command:

snmpwalk okv_ip_address prNames -A snmp_user_pwd -X snmp_user_pwd

In this command, prNames refers to "process names", which displays the names of processes instead of numbers. For example:

$ snmpwalk 192.0.2.254 prNames -A snmp_user_pwd -X snmp_user_pwd
UCD-SNMP-MIB::prNames.1 = STRING: mwecsvc
UCD-SNMP-MIB::prNames.2 = STRING: httpd
UCD-SNMP-MIB::prNames.3 = STRING: kmipd
UCD-SNMP-MIB::prNames.4 = STRING: ora_pmon_dbfwdb
UCD-SNMP-MIB::prNames.5 = STRING: ServiceManager
UCD-SNMP-MIB::prNames.6 = STRING: adminsrvr
UCD-SNMP-MIB::prNames.7 = STRING: distsrvr
UCD-SNMP-MIB::prNames.8 = STRING: recvsrvr

An example of running the snmptable command now becomes the following.

snmptable okv_ip_address prTable -A snmp_user_pwd -X snmp_user_pwd

Output similar to the following appears.

SNMP table: UCD-SNMP-MIB::prTable 
prIndex         prNames prMin prMax prCount prErrorFlag prErrMessage prErrFix prErrFixCmd
      1         mwecsvc     1     1       1     noError      noError            
      2           httpd     1    20       9     noError      noError                
      3           kmipd     1     2       2     noError      noError                
      4 ora_pmon_dbfwdb     1     1       1     noError      noError                
      5  ServiceManager     1     1       1     noError      noError                
      6       adminsrvr     1     1       1     noError      noError               
      7        distsrvr     1     1       1     noError      noError                 
      8        recvsrvr     1     1       1     noError      noError

The next example shows how you would now run the snmpdf command:

snmpdf okv_ip_address -A snmp_user_pwd -X snmp_user_pwd

Output similar to the following appears.

Description                Size (kB)      Used   Available   Used% 
/                          20027260    7247856    12779404     36%  
/usr/local/dbfw/tmp         6932408      15764     6916644      0%  
/var/log                    5932616      19932     5912684      0% 
/tmp                        1999184       3072     1996112      0% 
/var/lib/oracle           143592160   35023900   108568260     24%

Parent topic: Configuring Remote Monitoring to Use SNMP

20.1.2 Configuring Email Notification

You can use email notifications to directly notify administrators of Key Vault status changes without logging into the Oracle Key Vault management console.

About Email Notification
Email notifications alert users of status changes and are used to complete the processes of endpoint enrollment and user password reset operations.
Configuring Email Settings
You can configure the Simple Mail Transfer Protocol (SMTP) server properties to receive email notifications from Oracle Key Vault.
Testing the Email Configuration
Oracle Key Vault management console enables you to send test emails to test the email configuration.
Disabling Email Notifications for a User
You can use the Oracle Key Vault management console to enable or disable email notifications.

Parent topic: Managing System Monitoring

20.1.2.1 About Email Notification

Email notifications alert users of status changes and are used to complete the processes of endpoint enrollment and user password reset operations.

To enable email notification you must set your email preferences in Oracle Key Vault. You can choose the events that you want updates to. The events include Oracle Key Vault system status like disk utilization, backup, and primary-standby, or user and endpoint status like expiration of user passwords, endpoint certificates, and keys, or cluster status like the heartbeat lag, naming conflicts, cluster-wide HSM status, and others.

In cluster deployments, you must configure and validate email settings on all nodes of the cluster. Email settings of a cluster node are local to that node.

Configuring Email Settings is driven by the SMTP provider. Once you confirm that the SMTP server is reachable from the Oracle Key Vault server. You need to follow the required setting from the SMTP provider.

You can modify the SMTP server configuration at any time. If a custom SMTP certificate was used initially, and you later decide to use the default, you can modify the trust store setting to default, instead of custom.

For example:

The enrollment token generated during endpoint enrollment can be mailed directly to the endpoint administrator from Oracle Key Vault.
An Oracle Key Vault system administrator can send the random temporary password directly to the user when the user password is reset.

To enable email notifications successfully, there must be a connection between Oracle Key Vault and the SMTP server.

You can disable email notifications at any time.

Note:

If you are using Oracle Key Vault in an Oracle Cloud Infrastructure (OCI) environment, then see My Oracle Support note 2501601.1 for information about how to configure Postfix to use email delivery on the Oracle Linux 6 and 7 platforms. After you complete the configuration, ensure that you populate the From Address field with the approved sender from OCI.

Parent topic: Configuring Email Notification

20.1.2.2 Configuring Email Settings

You can configure the Simple Mail Transfer Protocol (SMTP) server properties to receive email notifications from Oracle Key Vault.

Log in to the Oracle Key Vault management console as a user with the System Administrator role.
Select the System tab, and then select Settings from the left navigation bar.
In the Network Services area, click Email.

The Email Settings page appears.

Description of the illustration 21_email.png
In the Email Settings page, enter the following values:
- SMTP Server Address: Enter a valid SMTP server address or host name for the user account. This setting should match the SMTP server setting of the user's email account. Ensure that the SMTP server or hostname is reachable from Oracle Key Vault. If you enter the SMTP hostname, you must configure DNS from the System Settings menu, so the host name can be resolved.
- SMTP Port: Enter the SMTP port number of the outgoing SMTP server, usually 465. This port number can be another number, if expressly configured that way in your organization.
- Name: Enter an alias for the SMTP user that will appear in the From field of the email.
- From Address: Enter the email address that you want to provide as a sender.
- If the SMTP server requires a secure connection, select Require Secure Connection. If you are using anonymous relay on Microsoft Exchange Server, or an external SMTP server such as Gmail or Office 365, do not select Require Secure Connection. Ensure that your firewall rules allow forwarding of SMTP requests to an external SMTP server.
  
  If Require Secure Connection is selected, the Authentication Protocol field is displayed with two options, SSL and TLS. Select the authentication protocol for the email server, either SSL or TLS. The default is TLS.
- If you have an SMTP user account, then check the box Require Credentials. When checked, the input fields User Name, Password, and Re-enter Password appear:
  - Enter the username of the SMTP user account.
  - Enter the password for the SMTP user account.
  - Reenter the password for the SMTP user account.
  Caution:
  Oracle strongly recommends that you have a secure connection to the SMTP server, because auto-generated tokens are sent over email for operations such as the creation of administrative users and Oracle Key Vault system alerts.
  
  Do not check Require Credentials for non-secure connections.
- If Custom SMTP Server Certificate is checked, then the field Upload Certificate File appears with the Choose File button to its right. Select this option if you want to upload a custom SMTP server's certificate to establish a TLS session between SMTP and Oracle Key Vault. This is how you can add a custom truststore in cases where the default Java truststore does not contain a necessary certificate. After Upload Certificate File, click Browse to upload a custom certificate file.
Click Configure.

On successful configuration, a SMTP successfully configured message is displayed.

If the configuration fails, then check that the SMTP server settings of the user email account are correct. Error messages highlight the field where the error has occurred to help isolate the problem.

Parent topic: Configuring Email Notification

20.1.2.3 Testing the Email Configuration

Oracle Key Vault management console enables you to send test emails to test the email configuration.

You can test the email configuration of the SMTP user account any time after you save the configuration. If you change an existing SMTP configuration, then you must save the configuration before you can test it.

Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
Select the System tab, and then select Settings from the left navigation bar.
In the Network Services area, click Email.
In the Email Settings window, configure the user's SMTP settings.
Save the configuration by clicking Configure.
You must save the configuration before you can test it.
In the Send Test Email section, enter the user email address in the Email Address field. Then click Test.

An email is sent to the user with Oracle Key Vault: Test Message in the subject line.

Depending on the Oracle Key Vault server timestamp, the email notification may not show up as the latest email.

The email notification may also not show up in your inbox, in which case you must check the spam folder.

If the email notification is not received, click the Reports tab and select System Reports from the left sidebar. On the System Reports page, click Notification Report. Check the list to determine the issue encountered while sending the email notification.

Related Topics

Configuring Email Settings

Parent topic: Configuring Email Notification

20.1.2.4 Disabling Email Notifications for a User

You can use the Oracle Key Vault management console to enable or disable email notifications.

An Oracle Key Vault user may elect not to receive email alerts. Only a user with the System Administrator role, or a user managing his own account can disable email notifications.

Log in to the Oracle Key Vault management console as a user with the System Administrator role.
Select the Users tab.

The Manage Users page appears.
Click user name of the user.

The User Details page appears.
Check the box to the left of text Do not receive email alerts.
Click Save.

Parent topic: Configuring Email Notification

20.1.3 Configuring the Syslog Destination for Individual Multi-Master Cluster Nodes

On each node, you can forward syslog entries to a remote service such as Splunk or SIEM.

Setting the Syslog Destination Setting for the Node
You can set the syslog destination to use either Transmission Control Protocol (TCP) or User Datagram Protocol (UDP).
Clearing the Syslog Destination Setting for the Node
You can clear the syslog destination setting for the node and then reset the node to the cluster setting.

Parent topic: Managing System Monitoring

20.1.3.1 Setting the Syslog Destination Setting for the Node

You can set the syslog destination to use either Transmission Control Protocol (TCP) or User Datagram Protocol (UDP).

Log into the Oracle Key Vault management console for the node as a user who has the System Administrator role.
Select the System tab, and then Settings from the left navigation bar.
In the Monitoring and Alerts area, click Syslog.
In the Syslog window, select one of the following options:
- TCP: Enables syslog using the TCP protocol.
- UDP: Enables syslog using the UDP protocol.
Enter the syslog destination IP addresses and port numbers in the Syslog Destinations field, in the format IP_address:port.
You can enter multiple destinations, each separated by a space.
Click Save.

Parent topic: Configuring the Syslog Destination for Individual Multi-Master Cluster Nodes

20.1.3.2 Clearing the Syslog Destination Setting for the Node

You can clear the syslog destination setting for the node and then reset the node to the cluster setting.

Log into the Oracle Key Vault management console for the node as a user who has the System Administrator role.
Select the System tab, and then Settings from the left navigation bar.
In the Monitoring and Alerts area, click Syslog.
In the Syslog window, click Delete.

Parent topic: Configuring the Syslog Destination for Individual Multi-Master Cluster Nodes

20.1.4 Capturing System Diagnostics

To troubleshoot problems that may arise, you can generate a system diagnostics file.

About Capturing System Diagnostics
The Oracle Key Vault diagnostics file provides advanced debug and troubleshooting information for problems that you may encounter while using Oracle Key Vault.
Installing the Diagnostics Generation Utility
You can use the Oracle Key Vault management console to download instructions for installing and using the diagnostics generation utility.
Generating a System Diagnostics File
The system diagnostics file that you download is in a .zip file.
Removing the Diagnostic Generation Utility Temporary Files
Removing the diagnostic generation utility temporary files frees up space on your server.
Removing the Diagnostic Generation Utility
If you no longer need to generate system diagnostic reports, then you can remove the diagnostic generation utility.

Parent topic: Managing System Monitoring

20.1.4.1 About Capturing System Diagnostics

The Oracle Key Vault diagnostics file provides advanced debug and troubleshooting information for problems that you may encounter while using Oracle Key Vault.

You can download this file and provide it to Oracle support for further analysis and debugging. The diagnostics file includes information about free space and disk usage reported is space available to Oracle Key Vault and not based on total disk size.

Diagnostics reporting is not enabled by default. You must enable the feature to generate diagnostics reports. After you have enabled diagnostics, you can configure the necessary information to be captured in diagnostics reports. You then can customize and package diagnostics reports with flexibility. Be aware that the first time you run the diagnostic utility or after the Oracle Key Vault system's internal database has been restarted, it can take longer that it will in future runs because it must gather all the diagnostic information in the system.

If you plan to perform an upgrade of Oracle Key Vault, then you must remove the diagnostic generation utility before performing the upgrade.

Parent topic: Capturing System Diagnostics

20.1.4.2 Installing the Diagnostics Generation Utility

You can use the Oracle Key Vault management console to download instructions for installing and using the diagnostics generation utility.

The instructions also explain how you can customize the output in the reports to accommodate different categories.

Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
Select System.

The Status page appears.
Description of the illustration 21.5_system_diagnostics.png
Click Download Diagnostics.
If the diagnostics generation utility is not installed, then you will be prompted to download the diagnostics-not-enabled.readme file.
Save the diagnostics-not-enabled.readme file to a local directory.
Follow the directions in this readme file to install and run the diagnostics generation utility, and to customize the report output.
The readme file includes the following instructions, but you should double-check this file in case these instructions have changed:
1. Use SSH as to connect as user support , then switch user (su) to root .
2. Install the diagnostics generation utility:
```
/usr/local/dbfw/bin/priv/dbfw-diagnostics-package.rb --install
```
3. Enable the collection of diagnostics:
```
/usr/local/dbfw/bin/priv/dbfw-diagnostics-package.rb --enable ALL
```

Parent topic: Capturing System Diagnostics

20.1.4.3 Generating a System Diagnostics File

The system diagnostics file that you download is in a .zip file.

Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
Select the System tab and then Status from the left navigation bar.
Click Download Diagnostics.
You should be prompted to download a .zip file, which will contain the diagnostics reports. If you are prompted to download the diagnostics-not-enabled.readme file, then the diagnostics generation utility has not been installed and you will need to install it.
Download the .zip file that contains the diagnostic reports to a secure location.

Related Topics

Installing the Diagnostics Generation Utility

Parent topic: Capturing System Diagnostics

20.1.4.4 Removing the Diagnostic Generation Utility Temporary Files

Removing the diagnostic generation utility temporary files frees up space on your server.

After you have run diagnostic reports, temporary files will accumulate. You should periodically remove these files. You can execute the command to remove these files from any directory.

Log in to the server where you downloaded and installed the diagnostic generation utility.
Use SSH as to connect as user support , then switch user (su) to root .
Execute the following command:
```
/usr/local/dbfw/bin/priv/dbfw-diagnostics-package.rb --clean
```
This command removes any .zip files that are found in the /usr/local/dbfw/tmp directory.

Parent topic: Capturing System Diagnostics

20.1.4.5 Removing the Diagnostic Generation Utility

If you no longer need to generate system diagnostic reports, then you can remove the diagnostic generation utility.

If you plan to upgrade Oracle Key Vault, then you must remove the diagnostic generation utility before you perform the upgrade. Removing this utility does not remove its temporary files.

Log in to the server where you downloaded and installed the diagnostic generation utility.
Use SSH as to connect as user support , then switch user (su) to root .

Execute the following command:

/usr/local/dbfw/bin/priv/dbfw-diagnostics-package.rb --remove

Parent topic: Capturing System Diagnostics

20.1.5 Monitoring System Metrics

You can use the System Metrics Monitoring feature to view and collect data for key system resource usage including CPU and Memory Usage in Oracle Key Vault.

About Capturing System Metrics
The System Metrics Monitoring feature provide resource monitoring capabilities using Oracle Key Vault management console.
Viewing System Metrics
You can use the Oracle Key Vault management console to view and download the system monitoring data.

Parent topic: Managing System Monitoring

20.1.5.1 About Capturing System Metrics

The System Metrics Monitoring feature provide resource monitoring capabilities using Oracle Key Vault management console.

Oracle Key Vault periodically collects CPU and Memory Usage data. You can view and collect system metrics data using the Oracle Key Vault management console. The Oracle Key Vault System Metrics Monitoring eliminates the need to first login to Oracle Key Vault server and then monitor the system manually.

Parent topic: Monitoring System Metrics

20.1.5.2 Viewing System Metrics

You can use the Oracle Key Vault management console to view and download the system monitoring data.

The instructions also explain how you can customize the output to collect required data.

Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
Select System.

The Status page appears.

Description of the illustration 21.5_system_diagnostics.png
Click System Metrics.

System Metrics page appears.

Description of the illustration 21.5_system_metrics.png

By default, the page displays the graphs for CPU and Memory usage percentage for last one hour.
Hovering mouse over a point in CPU Usage graph shows following:
- Time at which the data point collected
- CPU usage percentage at the data point
- Number of CPU cores at the data point
- CPU load averages for last one, five and fifteen minutes at the data point
Hovering mouse over a point in Memory Usage graph shows following:
- Time at which the data point collected
- Memory usage percentage at the data point
- Total memory in GB at the data point
- Free memory in GB at the data point
Select the different options from the Period drop-down to view CPU and Memory Usage of Last 1 Hour, Last 24 Hour, Last Week, Last Month, or Date Range(Period).

Note:
You can use the Date Range (Period) option to view and collect the usage data by specifying the From and To dates.
Select the different options from the Interval drop-down to aggregate the displayed data. You can select the Auto option for optimized performance.

Note:
The Interval field collect the CPU and Memory usage data based on the selected option.
Select the different options from the Statistic drop-down to aggregate function.
You can select from Mean, Min, or Max value.
Click Refresh to refresh and display the data according to the specified fields.
Click Download to save the data in a .csv file.
System Metrics page shows the status of metrics collection service. Up means the service is running and collecting the data and Down means there's an issue and Oracle Key Vault is not able to collect the data.
Note:
- Clicking on Download button save the raw data for the period specified. Also, the downloaded data do not have interval and statistic filters applied on it.
- You cannot view the data or make any changes to the System metrics when the Metrics Service is Down.

Parent topic: Monitoring System Metrics

20.2 Configuring Oracle Key Vault Alerts

You can select the type of alerts that you want to see in the Oracle Key Vault dashboard.

About Configuring Alerts
System administrators can configure alerts from the Oracle Key Vault dashboard, but all users can see alerts for the security objects to which they have access.
Configuring Alerts
You can configure alerts in the Reports page of the Oracle Key Vault management console.
Viewing Open Alerts
Users can view alerts depending on their privileges.

Parent topic: Monitoring and Auditing Oracle Key Vault

20.2.1 About Configuring Alerts

System administrators can configure alerts from the Oracle Key Vault dashboard, but all users can see alerts for the security objects to which they have access.

Email notifications must be enabled for users to receive alerts.

The Oracle Key Vault dashboard is the first page you see on logging into the management console. You can navigate to this page by clicking the Home tab. All users can see the alerts on security objects they have access to, but only users with the System Administrator role can configure alerts.

Oracle Key Vault offers several types of alerts that you can configure with appropriate thresholds according to your requirements. The alert types that appear are based on the type of environment that you are using: standalone, primary-standby, or multi-master cluster. You can also configure alerts for an HSM-enabled Oracle Key Vault server.

Oracle Key Vault alerts are categorized to one of the severity levels: CRITICAL, HIGH, MEDIUM, and LOW. You should resolve the higher severity alerts first.

You can configure the following alerts, which are listed in ascending order:

Table 20-2 Available Alerts

Alert Type	Severity	Environment	Multi-master Cluster Applicability	Purpose of Alert	When Alert Is Deleted
`Certificate Object Expiration`	HIGH	Standalone, primary-standby, and multi-master cluster environments	Cluster-wide	Raised when a certificate object's deactivation date is within the threshold value (default 7 days). This alert is raised only if the certificate object is in the `PRE-ACTIVE` or `ACTIVE` state. When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the certificate object expires.	Deleted if the certificate object is no longer expiring within the threshold value as a result of changes to either object's deactivation date or the configured threshold value. This alert is also deleted when a certificate object is revoked or destroyed.
`Cluster FIPS Not Consistent`	MEDIUM	Multi-master cluster only	Cluster-wide	Raised when at least one, but not all, `ACTIVE` nodes in the cluster are in FIPS mode	Deleted when all cluster nodes are in FIPS mode or all nodes are not in FIPS mode
`Cluster Heartbeat Lag`	HIGH	Multi-master cluster only	Node specific	Raised when a node has not received a heartbeat from another `ACTIVE` node in the cluster for over the threshold value (default 5 minutes)	Deleted when a node has once again received a heartbeat from the other node in the configured threshold period, as long as the node had received a heartbeat from the other node within the last Maximum Disable Node Duration period of time. This alert is also deleted when a node involved has been deleted from the cluster.
`Cluster HSM Not Consistent`	MEDIUM	Multi-master cluster only	Cluster-wide	Raised when at least one, but not all, `ACTIVE` nodes in the cluster are HSM-enabled	Deleted when all nodes are HSM-enabled or all nodes are not HSM-enabled
`Cluster Naming Conflict`	LOW	Multi-master cluster only	Cluster-wide	Raised when a naming conflict has been automatically resolved by Oracle Key Vault	Deleted when the object is deleted or renamed, or has had the new name explicitly accepted
`Cluster Redo Shipping Status`	HIGH	Multi-master cluster only	Node specific	Raised when a read-write node is unable to ship redo to its read-write peer, and as a result, is in read-only restricted mode. In addition to redo-shipping inactive status information, the alert indicates that the node in the cluster is operating in read-only mode.	Deleted when a read-write node is once again able to ship redo to its read-write peer, or when the node is deleted. The email notification states that the redo-shipping status is back up and the node in the cluster is no longer operating in read-only mode.
`Cluster Replication Lag`	HIGH	Multi-master cluster only	Node specific	Raised when incoming replication lag is greater than the threshold value (default 60 seconds)	Deleted when replication lag falls below the threshold value, or when any node in the replication link is deleted.
`Disk Utilization`	MEDIUM	Standalone, primary-standby, and multi-master cluster environments	Node specific	Raised when the free disk space percentage of the `/var/lib/oracle` partition is lower than the threshold value (default 25 percent)	Deleted when free disk space is once again higher than the threshold
`Endpoint Certificate Expiration`	HIGH	Standalone, primary-standby, and multi-master cluster environments	Cluster-wide	Raised when an endpoint's certificate is expiring within the threshold value (default 30 days). When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the endpoint's certificate expires.	Deleted when the endpoint's certificate is no longer expiring within the threshold value or when the endpoint is deleted
`Failed System Backup`	MEDIUM	Standalone, primary-standby, and multi-master cluster environments	Node specific	Raised when the last backup did not complete successfully	Deleted when the most recent backup completed successfully
`Failed OKV Services`	CRITICAL	Standalone, primary-standby, and multi-master cluster environments	Node Specific	Raised when DB, KMIP, REST, Email, Cluster, or Audit Vault service stops because of a failure.	Deleted when all of the failed services start running successfully.
`Fast Recovery Area Space Utilization`	HIGH	Standalone, primary-standby, and multi-master cluster environments	Node specific	Raised when Fast Recovery Area Space utilization of Oracle Key Vault's embedded database exceeds the configured threshold value (default 70 percent). To remedy this problem, try the following: Reduce the Maximum Disable Node Duration setting of the cluster node. Minimize the duration when peer node is not available. Consider deleting the node from the cluster and adding it back later.	Deleted when Fast Recovery Area Space utilization of Oracle Key Vault's embedded database is once again within the configured threshold
`High CPU Usage`	HIGH	Standalone, primary-standby, and multi-master cluster environments	Node Specific	Raised when average CPU utilization is greater than the threshold in last 5 minutes. Default value of threshold is 70%.	Deleted when average CPU utilization is less than the threshold in last 5 minutes.
`High Memory Usage`	HIGH	Standalone, primary-standby, and multi-master cluster environments	Node Specific	Raised when average memory usage is greater than the threshold in last 5 minutes. Default value of threshold is 95%.	Deleted when average memory usage is less than the threshold in last 5 minutes.
`Invalid HSM Configuration`	CRITICAL	Standalone, primary-standby, and multi-master cluster environments	Node specific	Raised when there is an error in the HSM configuration (checked by default every 5 minutes)	Deleted when there is no longer an error in the HSM configuration
`Key Rotations`	HIGH	Standalone, primary-standby, and multi-master cluster environments	Cluster-wide	Raised when a key's deactivation date is within the threshold value (default 7 days) This alert is raised only if the key object is in the `PRE-ACTIVE` or `ACTIVE` state. When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the certificate object expires.	Deleted if the key object is no longer expiring within the threshold value as a result of changes to either object's deactivation date or the configured threshold value. This alert is also deleted when a key object is revoked or destroyed.
`OKV CA Certificate Expiration`	CRITICAL	Standalone, primary-standby, and multi-master cluster environments	Node specific	Raised when the Oracle Key Vault CA certificate is expiring within the threshold value (default 90 days). When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the CA certificate expires.Be aware that if the CA certificate expires, then endpoints will no longer be able to communicate with Oracle Key Vault. This will result in downtime.	Deleted when the CA certificate is no longer expiring within the threshold value
`OKV Server/Node Certificate Expiration`	CRITICAL	Standalone, primary-standby, and multi-master cluster environments	Node specific	Raised in a standalone or primary-standby deployment when the Oracle Key Vault server certificate is expiring within the threshold value (default 90 days)  Raised in a multi-master cluster environment when a node’s node certificate is expiring within the threshold value (default 90 days). Be aware that if the server certificate expires in a standalone or primary-standby deployment, then endpoints will no longer be able to communicate with the Oracle Key Vault server. This will result in downtime for all endpoints. If a node certificate expires in a multi-master cluster environment, endpoints will be able to use other nodes for endpoint operations (like fetching a key). However, the node will no longer be able to communicate with other multi-master cluster nodes, and operations like creating a new wallet will be impacted	Deleted when the server or node certificate is no longer expiring within the threshold value.
`OKV Server Certificate Expiration`		Standalone, primary-standby, and multi-master cluster environments	Node specific	Raised when the Oracle Key Vault server certificate is expiring within the threshold value (default 90 days). When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the certificate expires. Be aware that if the server certificate expires, then endpoints will no longer be able to communicate with the Oracle Key Vault server. This will result in downtime.	Deleted when the server certificate is no longer expiring within the threshold value
`Primary-Standby Data Guard Broker Status`	HIGH	Primary-standby only	-	Raised when the Oracle Data Guard Broker status is not `ENABLED`	Deleted when the broker status is once again `ENABLED` or when Oracle Key Vault is no longer in primary-standby mode
`Primary-Standby Data Guard Fast-Start Failover Status`	MEDIUM	Primary-standby only	-	Raised when the fast-start failover status is not `SYNCHRONIZED`	Deleted when the fast-start failover status is once again `SYNCHRONIZED` or when Oracle Key Vault is no longer in a primary-standby configuration
`Primary-Standby Destination Failure`	HIGH	Primary-standby only	-	Raised when the switchover status is `FAILED DESTINATION`	Deleted when the switchover status is no longer `FAILED DESTINATION` or when Oracle Key Vault is no longer in a primary-standby configuration
`Primary-Standby Restricted Mode`	HIGH	Primary-standby only	-	Raised when in primary-standby environment and the primary is running in read-only restricted mode	Deleted when the primary is no longer in read-only restricted mode or when Oracle Key Vault is no longer in a primary-standby configuration
`Primary-Standby Role Change`	LOW	Primary-standby only	-	Raised when there is a role change	Deleted when Oracle Key Vault is no longer in a primary-standby configuration
`Secret Object Expiration`	HIGH	Standalone, primary-standby, and multi-master cluster environments	Cluster-wide	Raised when a secret object's deactivation date is within the threshold value (default 7 days). This alert is raised only if the object is in the `PRE-ACTIVE` or `ACTIVE` state. When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the secret object expires.	Deleted if the secret object is no longer expiring within the threshold value as a result of changes to either object's deactivation date or the configured threshold value. This alert is also deleted when a secret object is revoked or destroyed.
`SSH Tunnel Failure`	HIGH	Standalone, primary-standby, and multi-master cluster environments	Node specific	Raised when an SSH tunnel is not available	Deleted when the SSH tunnel is once again available or when the SSH tunnel is deleted
`System Backup`	MEDIUM	Standalone, primary-standby, and multi-master cluster environments	Node specific	Raised when the last successful backup is over the threshold value (default 14 days)	Deleted when the last successful backup was within the threshold value
`User Password Expiration`	MEDIUM	Standalone, primary-standby, and multi-master cluster environments	Cluster-wide	Raised when a user's password expires within the threshold value (default 14 days). When an expiration alert is within its threshold, an email notification is sent once every 24 hours until the password expires. Note: If the user password expires, user cannot login and Administrative tasks cannot be performed.	Deleted when a user's password no longer expire within the threshold value, or when the user is deleted

Related Topics

Parent topic: Configuring Oracle Key Vault Alerts

20.2.2 Configuring Alerts

You can configure alerts in the Reports page of the Oracle Key Vault management console.

Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
Access the Alerts page by using one of the following methods:
- Select the System tab, then Settings, and in the Monitoring and Alerts area, click Alerts.
- Select the Reports tab, and then select Alert from the left navigation bar. In the Alerts page, click Configure.
- On the Home page, expand Alerts at the top of the page, and then click All Alerts. Then click Configure.
The Configure Alerts page appears, listing various alert types and for some, configurable threshold limit and unit (such as the days until expiration for key rotations alert type). If you are using a multi-master cluster, then the Configure Alerts page will provide cluster-specific alerts, such as the cluster heartbeat lag, redo shipping status, or whether naming conflicts resolution is enabled. The following image shows how the Configure Alerts page appears in a multi-master cluster environment.

Note:
Alerts are assigned severity based on their impact on the Oracle Key Vault and registered endpoints.

Description of the illustration 21.5_configure_alert.png
Check the boxes in the Enabled column to the right of the alert types to enable the alert.
Then set the threshold value in the box under Limit. This value determines when the alert will be sent. You can uncheck the boxes by alerts that you do not want to appear in the dashboard.
Click Save.

Related Topics

Viewing the Oracle Key Vault Dashboard

Parent topic: Configuring Oracle Key Vault Alerts

20.2.3 Viewing Open Alerts

Users can view alerts depending on their privileges.

Users with the System Administrator role can view all alerts. Users without system administrator privileges can only view alerts related to objects they can access.

Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
Access the Alerts page by using one of the following methods:
- Select the System tab, then Settings, and in the Monitoring and Alerts area, click Alerts.
- Select the Reports tab, and then select Alerts from the left navigation bar.
- On the Home page, expand Alerts at the top of the page, and then click All Alerts.
  The Home page is also a convenient way to go immediately to how to solve a problem that an alert raises. Under Alerts, click Show Details, and then in the listing of alert types, click the appropriate link. For example, alerts describing upcoming key expirations will take you to the Alerts configuration page, where only alerts for key rotations are displayed. From there, you can examine the details of keys that are expiring.

The Alerts page appears, displaying all the unresolved alerts. Alerts are listed in the order of their severity and severity based color code scheme. The alerts with higher severity should be resolved first.

When you resolve the issue stated in the alert message, the alerts are automatically removed. To delete an alert message, select it and then click Delete. If the issue that caused the alert still exists, then the alert will be regenerated and appear again in this list.

Description of 21.5_view_alerts.png follows
Description of the illustration 21.5_view_alerts.png

Oracle Key Vault sends all system alerts to the syslog. The following is an example of a system alert in syslog:

Mar 29 18:36:29 okv080027361e7e logger[13171]: No successful backup done for 4 day(s)

The following table lists the conditions that trigger alerts, and the accompanying system alert message:

Condition	System Alert Message
`Certificate Object Expiration`	`Certificate object unique_ID expiration: date`
`Cluster FIPS Not Consistent`	`At least one, but not all, active OKV nodes are in FIPS Mode.`
`Cluster Heartbeat Lag`	`Replication lag from node node_name to node current_node_name exceeds threshold_value seconds`
`Cluster HSM Not Consistent`	`At least one, but not all, active OKV nodes are HSM-enabled.`
`Cluster Naming Conflict`	`Naming conflict for object_type: object_name` The label `object_type` can be `endpoint`, `endpoint group`, `user`, or `user group`, with `object_name` being the corresponding object.
`Cluster Redo Shipping Status`	Any of the following messages: `No heartbeats received from node source_node to node current_node.` `Last heartbeat from node source_node to node current_node was more than threshold minutes ago.` `Last heartbeat from node source_node to node current_node was more than one hour ago.` `Last heartbeat from node source_node to node current_node was more than six hours ago.` `Last heartbeat from node source_node to node current_node was more than Maximum_Disable_Node Duration hours (50% of the Maximum Disable Node Duration) ago. Take immediate action to restore communication from node source_node to avoid issues in the cluster.` `Last heartbeat from node source_node to node current_node was more than Maximum_Disable_Node Duration hours (75% of the Maximum Disable Node Duration) ago. Take immediate action to restore communication from node source_node to avoid issues in the cluster.` `Last heartbeat from node source_node to node current_node was more than Maximum_Disable_Node Duration hours (Maximum Disable Node Duration) ago. Node current_node may not have received all records from node source_node even if replication is restored.`
`Cluster Replication Lag`	`Replication lag from node node_name to node node_name is greater than threshold seconds. Current lag is current seconds.`
`Disk utilization`	`Free disk space is below threshold_value (currently current_value)`
`Endpoint certificate expiration`	`Endpoint endpoint_name certificate expiration date`
`Failed OKV Services`	`service_name service is failed or service_name1, service_name2 services are failed.`
`Failed system backup`	`Most recent backup failed!`
`Fast Recovery Area Space Utilization`	`Fast Recovery Area space usage exceeds threshold_value% (currently current_value%)`
`High CPU usage`	`CPU usage exceeds threshold_value (currently current_value)`
`High Memory Usage`	`Memory usage exceeds threshold_value (currently current_value)`
`Invalid HSM Configuration`	`HSM configuration error. Please refer to the HSM Alert section in the Oracle Key Vault HSM Integration Guide`
`Key rotations`	`Key unique_ID expiration: date`
`Primary-standby destination failure`	`One or more standby servers are in an error state. HA destination failure.`
`Primary-standby Oracle Data Guard Broker status`	`Data Guard Broker is disabled`
`Primary-standby Oracle Data Guard fast-start failover status`	`HA FSFO is not synchronized. FSFO status is HA_status`
`Primary-standby restricted mode`	`HA running in read-only restricted mode`
`Primary-standby role change`	`HA role changed. Primary IP Address: IP_address`
`Secret Object Expiration`	`Secret object unique_ID expiration : date`
`SSH tunnel failure`	`SSH tunnel (IP IP_address) is not available`
`System backup`	`No successful backup for number day(s)`
`User password expiration`	`User user_name password expiration: date`

Parent topic: Configuring Oracle Key Vault Alerts

20.3 Managing System Auditing

Auditing entails tasks such as capturing audit records in a syslog file or downloading the audit records to a local file.

About Auditing in Oracle Key Vault
Oracle Key Vault records and time-stamps all endpoint and user activity.
Oracle Key Vault Audit Trail
The Oracle Key Vault audit trail captures information about activities that are performed in Oracle Key Vault, such as the name of an action and who performed it.
Enabling Auditing and Configuring Syslog to Store Audit Records
You can enable or disable auditing and then configure the Oracle Key Vault syslog to store audit records if the System Administrator has enabled this functionality.
Viewing Audit Records
To view audit records, access the Oracle Key Vault management console Audit Trail page.
Exporting and Deleting Audit Records Manually
Oracle Key Vault audit records are stored in a .csv file.
Deleting Audit Records Automatically
You can configure Oracle Key Vault to automatically delete or purge the audit records that are older than the specified retention period.
Configuring Oracle Key Vault with Oracle Audit Vault
A user who has the Audit Manager role can configure Oracle Key Vault to send audit records to Oracle Audit Vault for centralized audit reporting and alerting.

Parent topic: Monitoring and Auditing Oracle Key Vault

20.3.1 About Auditing in Oracle Key Vault

Oracle Key Vault records and time-stamps all endpoint and user activity.

The audit records include endpoint groups and user groups, from endpoint enrollment and user password reset, to the management of keys and wallets, and changes to system settings and SNMP credentials. The audit trail captures details on who initiated which action, with what keys and tokens, and the result of the action. In addition, it records the success or failure of each action.

Only a user who has the Audit Manager role can manage the audit trail for Oracle Key Vault activity. Each user can see audit records of the objects that the user can access.

Auditing in Oracle Key Vault is enabled by default.

A user with the Audit Manager role can see and manage all the audit records. Other users can see only those audit records that pertain to security objects that they have created, or have been granted access to.

The audit manager can export audit records to view system activity off line. After exporting the records, the audit manager can delete them from the system to free up resources.

Related Topics

Audit Manager Role Duties

Parent topic: Managing System Auditing

20.3.2 Oracle Key Vault Audit Trail

The Oracle Key Vault audit trail captures information about activities that are performed in Oracle Key Vault, such as the name of an action and who performed it.

The following lists the contents of the Oracle Key Vault audit trail.

Table 20-3 Oracle Key Vault Audit Trail

Column Name	Description
`Client IP`	The IP address of the client host or the IP address of the proxy server between the client and the Oracle Key Vault server that is making its IP address available to Oracle Key Vault
`Node ID`	ID of the Oracle Key Vault cluster node on which the operation was performed
`Node IP Address`	IP address of the Oracle Key Vault cluster node on which the operation was performed
`Node Name`	Name of the Oracle Key Vault cluster node on which the operation was performed
`Object`	Captures the name of object on which the operation is performed
`Object Type`	Type of object on which the operation is performed (for example, `User`, `Endpoint`)
`Operation`	Name of the operation performed
`Result`	Result of the operation indicating whether it was successful or failure
`Subject`	Captures the name of entity that performed the operation
`Subject Type`	Type of entity, `User` or `Endpoint`
`Time`	Timestamp of the operation

Parent topic: Managing System Auditing

20.3.3 Enabling Auditing and Configuring Syslog to Store Audit Records

You can enable or disable auditing and then configure the Oracle Key Vault syslog to store audit records if the System Administrator has enabled this functionality.

Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
Access the Audit Settings page by using one of the following methods:
- Select the Systems tab, then Settings in the left navigation bar. In the Monitoring and Alerts area, click Audit.
- Select the Reports tab, then Audit Trail in the left navigation bar. In the Audit Trail page, click Audit Settings.
The Audit Settings page appears. The categories that you can configure are as follows:
- Auto Purge Audit Records
- Enable Auditing
- Send Audit Records to Syslog
Next, do the following:
1. In a multi-master cluster environment, click the arrow on the right that appears next to the Save button to toggle between the current node and the entire cluster. Also, the Auto Purge Audit Records setting can only be configured at the node level only.
2. Select either the Yes or No option for the auditing.
3. Click Save.
If syslog is configured, then perform additional steps as needed.
If syslog is not configured, then the Syslog forwarding to remote machines not enabled error message appears. If this error appears, then enable syslog.
1. Select the System tab, and then select Settings.
2. In the Monitoring and Alerts area, select Syslog.
3. In a multi-master cluster environment, toggle between Node Details - Effective on this Node and Cluster Details by clicking the arrow under the Save button.
1. Select the protocol to use to transfer syslog files: TCP or UDP.
2. Enter the IP address of the remote system where the syslog files will be stored.
3. Click Save.
Note:
To avoid any accidental deletion of Audit records it is recommended to use TCP protocol for transferring syslog files.

Parent topic: Managing System Auditing

20.3.4 Viewing Audit Records

To view audit records, access the Oracle Key Vault management console Audit Trail page.

The reports page shows the Audit Trail page by default. The Audit Trail page lists all system activity with details on who performed an operation, when the operation was performed, what object was used to perform the operation, and the result.

Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
Click the Reports tab, and then Audit Trail from the left navigation bar.

The Audit Trail page appears. Optionally, filter records by selecting the table column heads, and from the drop-down list, select the type of sort order that you want.

Parent topic: Managing System Auditing

20.3.5 Exporting and Deleting Audit Records Manually

Oracle Key Vault audit records are stored in a .csv file.

A user with the Audit Manager role can export the audit trail to a .csv file that can be downloaded to the user's local system. The .csv file contains the same details found in the audit trail on the Reports page. The timestamp in the .csv file reflects the time zone of the particular Oracle Key Vault server whose records were exported. After you export the records, you can delete them from the Oracle Key Vault server to free up space.

Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
Click the Reports tab, then Audit Trail in the left navigation bar..

The Audit Trail appears.
Click Export/Delete Audit Records on the top right.

The Export/Delete Audit Records page appears.

Description of the illustration 21_export_delete_audit_records.png
Select the date by clicking the calendar icon.

Based on the date that you select, the number of records appears after the Number of records to be exported/deleted label.
Click Export to download the audit records in .csv file format to a local folder.

After you export the records, you can delete them from Oracle Key Vault to free up resources.
Click Delete to remove the audit records.
Click OK to delete or Cancel to stop.

Parent topic: Managing System Auditing

20.3.6 Deleting Audit Records Automatically

You can configure Oracle Key Vault to automatically delete or purge the audit records that are older than the specified retention period.

The Audit Settings page shows the Auto Purge Audit Records pane. The user with the Audit Manager role can purge the Audit Records.

Log in to the Oracle Key Vault management console as a user who has the audit manager role.
Access the Audit Settings page by using one of the following methods:
- Select the Systems tab, then Settings in the left navigation bar. In the Monitoring and Alerts area, click Audit.
- Select the Reports tab, then Audit Trail in the left navigation bar. In the Audit Trail page, click Audit Settings.
The Audit Settings page appears. The categories that you can configure are as follows:
- Auto Purge Audit Records
- Enable Auditing
- Send Audit Records to Syslog
- Replicate Audit Records, available in cluster environment.
Select Yes, in the Auto Purge Audit Records pane.
The Retention Period in Days field appears.

Description of the illustration 21.5_delete_audit_records_automatically.png
Enter the number of days to retain the audit records. Oracle Key Vault will now periodically purge audit records that are older than the specified number of days.
Click Save.

Note:
When Oracle Key Vault is integrated with Oracle Audit Vault, audit records are purged only after they are collected by the Audit Vault.

Related Topics

Exporting and Deleting Audit Records Manually

Parent topic: Managing System Auditing

20.3.7 Configuring Oracle Key Vault with Oracle Audit Vault

A user who has the Audit Manager role can configure Oracle Key Vault to send audit records to Oracle Audit Vault for centralized audit reporting and alerting.

Integrating Oracle Audit Vault with Oracle Key Vault
You can perform this integration from the Oracle Key Vault management console.
Viewing Oracle Key Vault Audit Data Collected by Oracle Audit Vault
You can use the Oracle Audit Vault Server console to view data that is collected by Oracle Key Vault and Oracle Audit Vault.
Suspending an Oracle Audit Vault Monitoring Operation
You can suspend an Oracle Audit Vault monitoring operation from the Oracle Key Vault management console.
Resuming an Oracle Audit Vault Monitoring Operation
You can resume a suspended Oracle Audit Vault monitoring operation from the Oracle Key Vault management console.
Deleting an Oracle Audit Vault Integration
You can delete an Oracle Audit Vault integration by using the Oracle Key Vault management console.
Guidance for Integrating Oracle Audit Vault in a Multi-Master Cluster or Primary-Standby Environment
You must follow special guidelines to integrate Oracle Audit Vault with Oracle Key Vault in a multi-master cluster or primary-standby environment.

Parent topic: Managing System Auditing

20.3.7.1 Integrating Oracle Audit Vault with Oracle Key Vault

You can perform this integration from the Oracle Key Vault management console.

Step 1: Check the Environment
Before you begin the integration, you should ensure that the required components are all in place.
Step 2: Configure Oracle Key Vault as a Registered Host and a Secured Target with Oracle Audit Vault
A user who has the Audit Manager role must configure the Oracle Key Vault server as a secured target on the Oracle Audit Vault server.

Parent topic: Configuring Oracle Key Vault with Oracle Audit Vault

20.3.7.1.1 Step 1: Check the Environment

Before you begin the integration, you should ensure that the required components are all in place.

Ensure that Oracle Audit Vault is properly installed and configured.
This activity requires administrative access to the Oracle Audit Vault server in order to register Oracle Key Vault as a secured target.
Ensure that you have the credentials of the Oracle Audit Vault administrator in order to register Oracle Key Vault as a secured target in Oracle Audit Vault server. This user does not need to be a super administrator.
Enable SSH access to Oracle Audit Vault.

Log in to the Oracle Audit Vault Server console as a user who has the Super Administrator role. Select the Settings tab, then System. In the Configuration area, click System Settings and then Web/SSH/SNMP. Turn on SSH Access, select IP addresses and then enter only the IP addresses that you need, or select All. Click Save.

Parent topic: Integrating Oracle Audit Vault with Oracle Key Vault

20.3.7.1.2 Step 2: Configure Oracle Key Vault as a Registered Host and a Secured Target with Oracle Audit Vault

A user who has the Audit Manager role must configure the Oracle Key Vault server as a secured target on the Oracle Audit Vault server.

In a multi-master cluster environment, perform these steps on each node. Each node will send the audit records that were generated from that node to the Oracle Audit Vault server.

Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
Select the System tab, and then Audit Vault Integration from the left navigation bar.
If you also have the System Administrator role, then the navigation will be slightly different. Click System, then in the left navigation pane, select Settings. In the Monitoring and Alerts area, click Audit Vault.
In the Deployment pane, enter the following settings:
- Hostname: Enter the host name or IP address of the Oracle Audit Vault server.
- Public Host Key: Enter the public host key of the Oracle Audit Vault server by following the guidelines mentioned in the help text.
- Support User Password: Enter the support user password of the Oracle Audit Vault server.
- Administrator Name: Enter the user name of the Oracle Audit Vault server user who has the Administrator role.
- Administrator Password: Enter the password of the Oracle Audit Vault server user who has the Administrator role.
- Recovery Passphrase: Enter the recovery passphrase of the Oracle Key Vault server.
Description of the illustration avdf_deployment.png
Click Deploy.
The integration may take about 10 minutes to complete. Do not attempt to re-initiate the Audit Vault integration during this interval. The Oracle Key Vault server may become unavailable for some time until the integration completes.

After the integration completes, a Monitoring tab appears and will show the Audit Vault agent status.

Parent topic: Integrating Oracle Audit Vault with Oracle Key Vault

20.3.7.2 Viewing Oracle Key Vault Audit Data Collected by Oracle Audit Vault

You can use the Oracle Audit Vault Server console to view data that is collected by Oracle Key Vault and Oracle Audit Vault.

Log in to the Oracle Audit Vault Server console as an auditor.
Select the Reports tab.
Select Activity Reports.
Select All Activity.
Filter the target to get all records belonging to okv_db_Oracle_Key_Vault_IP_address.
Assuming that you filtered for the target okv_db_192.0.2.78, the report could be similar to the following:
Description of the illustration avdf_all_activity.png

Parent topic: Configuring Oracle Key Vault with Oracle Audit Vault

20.3.7.3 Suspending an Oracle Audit Vault Monitoring Operation

You can suspend an Oracle Audit Vault monitoring operation from the Oracle Key Vault management console.

Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
Select the System tab, and then Audit Vault Integration in the left navigation bar.
If you also have the System Administrator role, then the navigation will be slightly different. Click System, then in the left navigation pane, select Settings. In the Monitoring and Alerts area, click Audit Vault.
Select the Monitoring tab.
In the Audit Vault pane, click Suspend.

Parent topic: Configuring Oracle Key Vault with Oracle Audit Vault

20.3.7.4 Resuming an Oracle Audit Vault Monitoring Operation

You can resume a suspended Oracle Audit Vault monitoring operation from the Oracle Key Vault management console.

Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
Select the System tab, and then Audit Vault Integration in the left navigation bar.
If you also have the System Administrator role, then the navigation will be slightly different. Click System, then in the left navigation pane, select Settings. In the Monitoring and Alerts area, click Audit Vault.
Select the Monitoring tab.
In the Audit Vault pane, click Resume.

Parent topic: Configuring Oracle Key Vault with Oracle Audit Vault

20.3.7.5 Deleting an Oracle Audit Vault Integration

You can delete an Oracle Audit Vault integration by using the Oracle Key Vault management console.

Log in to the Oracle Key Vault management console as a user who has the Audit Manager role.
Select the System tab, and then Audit Vault Integration in the left navigation bar.
If you also have the System Administrator role, then the navigation will be slightly different. Click System, then in the left navigation pane, select Settings. In the Monitoring and Alerts area, click Audit Vault.
Under Deployment, in the Audit Vault Details page, enter the following settings:
- Public Host Key: Enter the public host key of the Oracle Audit Vault server by following the guidelines mentioned in the help text.
- Support User Password: Enter the support user password of the Oracle Audit Vault server.
- Administrator Name: Enter the user name of the Oracle Audit Vault server user who has the Administrator role.
- Administrator Password: Enter the password of the Oracle Audit Vault server user who has the Administrator role.
Description of the illustration avdf_delete.png
In the Deployment pane, click Delete.
Click OK to confirm.
Audit records that are already collected in Oracle Audit Vault are not affected when Oracle Key Vault integration with Oracle Audit Vault is deleted.
Because you no longer need to copy files from one server to another, disable SSH access to Oracle Audit Vault.

Log in to the Oracle Audit Vault Server console as a user who has the Super Administrator role. Select the Settings tab, then System. In the Configuration area, click System Settings and then Web/SSH/SNMP. Turn off SSH Access. Click Save.

Parent topic: Configuring Oracle Key Vault with Oracle Audit Vault

20.3.7.6 Guidance for Integrating Oracle Audit Vault in a Multi-Master Cluster or Primary-Standby Environment

You must follow special guidelines to integrate Oracle Audit Vault with Oracle Key Vault in a multi-master cluster or primary-standby environment.

Multi-Master Cluster Environments

If Oracle Key Vault is configured to use multi-master clusters, then you must perform the Oracle Audit Vault integration individually on each node. Each node will send audit records that are generated only on that node to Oracle Audit Vault irrespective of whether audit record replication is enabled.

Primary-Standby Environments

Perform the integration only on the primary server, not the standby server.
If you must perform a switchover operation, then note the following:
- You must switch back to the primary server if you want to suspend, resume, or delete the integration. You do not need to perform additional steps.
- To integrate the new primary server with Oracle Audit Vault, optionally, ensure that you use the same Oracle Audit Vault host name and administrator credentials that were used in the old primary server.
- If you perform an unpair operation after performing a switchover operation, then you must perform a new Oracle Audit Vault integration with the new primary server.
- If you delete the new integration, then the old integration becomes non-functional. You must then delete this old integration by switching back to the old primary server.
If a failover operation occurs and the original primary server is no longer available, then you must perform a new Oracle Audit Vault integration with the new primary server.
However, if the original primary server is not lost and it is possible to bring back the original primary server as the new standby server, then you do not need to perform additional steps.

Parent topic: Configuring Oracle Key Vault with Oracle Audit Vault

20.4 Using Oracle Key Vault Reports

Oracle Key Vault collects statistical information on a range of activities that impact Key Vault operations.

About Oracle Key Vault Reports
The reports cover system activity, certificate expiration, keys, passwords, entitlement status, extraction status, and metadata.
Viewing Key Management Reports for Oracle Endpoints
All users can view the key management reports for Oracle endpoints.
Viewing Keys and Wallets Reports
The keys and wallets reports require different privileges for viewing, depending on the report.
Viewing Secrets Management Reports
All users can view the secrets management reports.
Viewing Endpoint Reports
You must have the System Administrator role or the Audit Manager role to view the four categories of endpoint reports.
Viewing User Reports
You must have the System Administrator role, the Key Management role, or the Audit Manager role to view the four categories of user reports.
Viewing System Reports
You must have the System Administrator role or the Audit Manager role to view the system reports.

Parent topic: Monitoring and Auditing Oracle Key Vault

20.4.1 About Oracle Key Vault Reports

The reports cover system activity, certificate expiration, keys, passwords, entitlement status, extraction status, and metadata.

Oracle Key Vault provides six types of reports for endpoints, users, keys and wallets, and system. In a multi-master cluster, some reports contain additional information, such as the node ID, node name, and IP address.

The six report types are as follows:

Key management reports for Oracle endpoints, which includes information about TDE master encryption keys, GoldenGate master keys, and ACFS volume encryption key details
Keys and wallets reports list the access privileges granted to all keys and wallets, and the details of TDE master encryption keys managed by Oracle Key Vault
Secrets management reports for database passwords, secret data, and opaque objects
Endpoint reports contain details of all endpoint and endpoint group activity, certificate and password expiration, and access privileges
User reports contain details of all user and user group activity, their certificate and password expiration, and access privileges
System reports contain a history of system backups taken and scheduled, details of remote restoration points, and RESTful services utility command-line interface usage

A user who has the Audit Manager role can view all reports, including reports that are accessible from the Audit Trail pages in the Oracle Key Vault management console. A user with the Key Administrator role can view user reports and keys and wallets reports. Users with the System Administrator role can view endpoint, user, and system reports.

Related Topics

Managing System Auditing

Parent topic: Using Oracle Key Vault Reports

20.4.2 Viewing Key Management Reports for Oracle Endpoints

All users can view the key management reports for Oracle endpoints.

Log in to the Oracle Key Vault management console.
Click the Reports tab and then Reports from left navigation bar.
Expand Key Management Reports for Oracle Endpoints Reports.

The Key Management Reports for Oracle Endpoints Reports page appears displaying the four types of user reports.

Description of the illustration 21_key_management_oracle_endpoints_report.png
Click the report name to see the corresponding user report.

Parent topic: Using Oracle Key Vault Reports

20.4.3 Viewing Keys and Wallets Reports

The keys and wallets reports require different privileges for viewing, depending on the report.

Log in to the Oracle Key Vault management console as a user who has the appropriate privileges.
Only a user who has the Key Administrator role or Audit Manager role can view the Wallet Entitlement Report. All users can view the remaining reports.
Click the Reports tab and then Reports from left navigation bar.
Expand Keys and Wallets Reports.

The Keys and Wallets Reports page appears displaying the reports.

Description of the illustration 215_keys_wallets_report.png
Click the report name to see the corresponding report.

Parent topic: Using Oracle Key Vault Reports

20.4.4 Viewing Secrets Management Reports

All users can view the secrets management reports.

Log in to the Oracle Key Vault management console.
Click the Reports tab and then Reports from left navigation bar.
Expand Secrets Management Reports.

The Secrets Management Reports page appears displaying the reports.

Description of the illustration 21_secrets_mgmt_report.png
Click the report name to see the corresponding report.

Parent topic: Using Oracle Key Vault Reports

20.4.5 Viewing Endpoint Reports

You must have the System Administrator role or the Audit Manager role to view the four categories of endpoint reports.

Oracle Key Vault offers four endpoint reports: Endpoint Activity, Endpoint Certificate Expiry, Endpoint Entitlement, and Endpoint Metadata.

Log in to the Oracle Key Vault management console as a user who has the System Administrator role or the Audit Manager role.
Click the Reports tab and then Reports from left navigation bar.
Expand Endpoint Reports.

Description of the illustration 21_view_reports.png
Select the name of the report that you want to view.

For example, the Activity Report for endpoints appears similar to the following:

Description of the illustration 21_endpoint_report.png

Parent topic: Using Oracle Key Vault Reports

20.4.6 Viewing User Reports

You must have the System Administrator role, the Key Management role, or the Audit Manager role to view the four categories of user reports.

Oracle Key Vault offers four user reports: User Activity, User Entitlement, User Expiry, and User Failed Login.

Log in to the Oracle Key Vault management console as a user who has the System Administrator role, the Key Management role, or the Audit Manager role.
Click the Reports tab and then Reports from left navigation bar.
Expand User Reports to see user-specific reports.

The User Reports page appears displaying the four types of user reports.

Description of the illustration 21_user_reports.png
Click the report name to see the corresponding user report.

Parent topic: Using Oracle Key Vault Reports

20.4.7 Viewing System Reports

You must have the System Administrator role or the Audit Manager role to view the system reports.

Oracle Key Vault offers three system reports: Backup History, Backup Restoration Catalog, and RESTful services utility API Usage. For a multi-master cluster configuration, the following additional reports are available: Cluster and Conflict Resolution.

Log in to the Oracle Key Vault management console as a user who has the System Administrator role or the Audit Manager role.
Click the Reports tab and then Reports from left navigation bar.
Expand System Reports.

The System Reports page appears displaying the system reports available.

Description of the illustration 21_system_report.png
Click the report type to see the corresponding system report.

Parent topic: Using Oracle Key Vault Reports