2 Secure Installation and Configuration for Oracle Private Cloud Appliance

The Oracle Private Cloud Appliance is an engineered systems and installation is generally performed for the customer. When the system is first powered on, there are various tasks that need to be performed in order to get the system initially set up. Tasks like defining the first "admin" user account (note that a default administrator account is no longer provided due to security requirements), configuring system parameters like realm and region, and configuring basic networking parameters. Oracle Private Cloud Appliance supports two ways for an operator to view and configure system information: a GUI and a session-based CLI. For the initial installation, it is envisioned that the GUI would be used to guide the operator through the various initial configurations that are needed. If desired, the operator may wish to use the CLI to perform these actions instead.

Installation Overview

As an engineered system, installation of Oracle Private Cloud Appliance is not usually performed by the customer. Nevertheless, security concerns are the responsibility of everyone who uses the system. Security issues must be addressed before the system is installed because security cannot be added on to a system later.

Pre-Installation Security Details

Before Oracle Private Cloud Appliance installation, create a document to outline the services provided. Have it reviewed and updated to address any shortcomings.

  • For each application or service, have those responsible for security review the information.

  • Provide all URLs and links needed so that reviewers can easily find the source material employed to create the pre-installation plan.

  • Repeat the process until all reviewers are satisfied that all initial security goals have been satisfied.

Make a list of all the roles needed to deploy the Oracle Private Cloud Appliance in a secure environment. Identify the personnel needed to fill these roles.

Note:

Make sure that the roles and users identified do not overlap and that capabilities are appropriately isolated.

  • Identify the various administrators for all layers of the Oracle Private Cloud Appliance: infrastructure, Service Enclave, andCompute Enclave.

  • Identify the users of services at all relevant layers of the Oracle Private Cloud Appliance. List privileges needed and restrictions necessary.

Produce a draft implementation plan with the virtual machines and network connections needed for the Oracle Private Cloud Appliance. Have this reviewed and modified until it is as complete as feasible before installation.

  • Describe the role of each virtual machine as clearly as possible.

  • If there are departures from the typical front-end, back-end, and load balancer arrangement, describe it in full.

  • Describe the circumstances for starting virtual machines, both for initial use and for handling increased loads.

Describe the network connections needed, if any, between the virtual machines at each layer of the Oracle Private Cloud Appliance architecture.

  • List the secure network protocols to be used to operate and maintain the system.

  • Provide initial policy rules for virtual machines communications, at least at a prose level.

  • Determine which network connections can be switched: that is, can be handled by a simple VLAN and single IP address space.

  • Determine which network connections must be routed: that is, must be handled by more than one VLAN and multiple of subnetted IP address spaces.

System Site Preparation

For pre-installation site preparation, see the Oracle Private Cloud Appliance Installation Guide.

Product Installation

Regardless of who installs the Oracle Private Cloud Appliance, each component of the Oracle Private Cloud Appliance product automatically installs into a secure state. Security options, such as SSL, are already configured and enabled.

You can tailor the installed product for your specific deployment scenario, as long as security is not compromised.

You can uninstall or disable any component of the system which are not used in your specific deployment.

Post-Installation Configuration

This section describes the security configuration changes that must be made after installation. Generally, do not weaken the security posture of the Oracle Private Cloud Appliance when making changes to configurations.

Securing the Hardware

After installation of Oracle Private Cloud Appliance, secure the hardware by restricting access to the hardware and recording the serial numbers.

Oracle recommends the following practices to restrict access:

  • Install Oracle Private Cloud Appliance and related equipment in a locked, restricted-access room.

  • Lock the rack door unless service is required on components within the rack.

  • Restrict access to hot-pluggable or hot-swappable devices because the components are designed to be easily removed.

  • Store spare field-replaceable units (FRUs) or customer-replaceable units (CRUs) in a locked cabinet. Restrict access to the locked cabinet to authorized personnel.

  • Limit SSH listener ports to the management and private networks. Use SSH protocol 2 (SSH-2) and FIPS 140-2 approved ciphers.

  • Limit SSH allowed authentication mechanisms. Inherently insecure methods are disabled.

  • Label all significant items of computer hardware, such as FRUs.

  • Keep hardware activation keys and licenses in a secure location that is easily accessible to the system managers in the case of a system emergency.

Retrieving the Appliance Serial Number

An Oracle Private Cloud Appliance has a serial number that identifies the appliance as a whole entity. When working with Oracle Support Services, the appliance serial number may be required.

There is a label on the rack with the serial number. The serial number label is the white label about midway up the front right rail of the rack.

There are several other techniques to obtaining the overall appliance serial number:
  • Use the Service Enclave console (Administrative Console) - To get the appliance serial number from the Service Enclave console:

    1. Use a supported web browser to log into the console with a username and password that has sufficient authorization (https://adminconsole.<domain> ).

    2. If you just completed the initial setup, then use the admin username with the configured password (otherwise the administrator may provide you with a specific user for access).

    3. Access the Appliance Details section from the menu.

    4. Record the ID and the Realm from this page. The Realm is the physical appliance serial number: record both the ID and the physical appliance serial number.

  • Use the appropriate monitoring dashboard (Grafana) - To obtain the appliance serial number from the monitoring dashboard:

    1. Use a supported web browser to log into the dashboard with a username and Password that has sufficient authorization (https://grafana.<domain> ).

    2. Go to the Dashboards → Manage Dashboards section.

    3. Open the Service Monitoring → Hardware Stats dashboard. The appliance serial number is displayed on this dashboard.

    4. Record the ID and the Realm from this page. The Realm is the physical appliance serial number: record both the ID and the physical appliance serial number.

  • Use the Admin Command Line Interface (CLI) - To obtain the appliance serial number using the CLI:

    1. Log into the Admin CLI with a username and password that has sufficient authorizations (use the command ssh -l <username> management host -p 30006). The management host is one of the three management nodes (pcamn01, pcamn02, or pcamn03) and the username, by default, is admin. If the section on rotating the passwords is not completed yet, then use the password supplied during initial setup.

    2. From the prompt, there are a variety of ways to get the rack serial number. A quick report can be obtained with the command: list Rack fields name,rackNumber,rackSerialnumber,rackType.

      For example,

      PCA-ADMIN> list Rack fields rackSerialnumber
      Command: list Rack fields rackSerialnumber
      Status: Success
      Time: 2022-02-17 20:15:41,773 UTC
      Data:
        id                                   rackSerialnumber
        --                                   ----------------
        x6d9f31a-b1ds-4a43-bc09-7270609abd8k CL00888510
      PCA-ADMIN>
    3. Store the information from the Appliance Details in a secure location.

Retrieving the Serial Numbers for Hardware Components in the Rack

An Oracle Private Cloud Appliance is an engineered system that is made up of many hardware components. For simplicity we will call these the rack components, though the components can span multiple racks. The most efficient way to collect the serial numbers for the rack components is to use the Admin CLI, though they can also be obtained from the Service Enclave console (Administrative Console). These serial numbers may also be needed when working with Oracle Support Services.

To get a list of rack component serial numbers from the Admin CLI:

  1. Log into the Admin CLI with a username and password that has sufficient authorizations (use the command ssh -l <username> management host -p 30006). More details are in the Retrieving the Appliance Serial Number section.

  2. Once logged in, get a report using a command such as the following: list RackUnit fields name,rackElevation,serialNumber,hostname.

  3. Store the information recorded above in a secure location.

To use the Service Enclave console, log into the console with a username and password that has sufficient authorization (https://adminconsole.<domain>). After logging in:

  1. Using the menu, navigate to the Rack Units context.

  2. For each unit in the list, choose to View Details:

    1. Once in the details, choose the System tab.

    2. Record the Serial Number along with other component information.

  3. Return to the Rack Units list and record the next component.

  4. Store the information recorded above in a secure location.

Securing the Software

After installation of Oracle Private Cloud Appliance, secure the software. In many cases, hardware security is implemented through software.

After initial installation of Oracle Private Cloud Appliance, Oracle recommends the following practices to restrict system access:

  • Limit use of the super-user account, such as root on the management nodes and SuperAdmin accounts in the Service Enclave and Compute Enclave. Create and use individual user accounts because they ensure positive identification in audit trails, and require less maintenance when administrators leave the team or company.

  • Do not create new users on the management nodes.

  • Disable unnecessary protocols and modules for layers under customer control.

  • Restrict physical access to USB ports, network ports, and system consoles because physical severs and network switches have ports and console connections providing direct access to the system.

  • Restrict the capability to restart the system over the network.

  • For more information on how to enable other security features, see Security Features for Oracle Private Cloud Appliance in this guide.

By using the privileged users and multi-factor access control, and other tools such as data encryption, auditing, monitoring, and data masking, customers can deploy reliable data security solutions that do not require any changes to existing applications.

Securing the Network

Oracle Private Cloud Appliance is a private cloud environment. During initialization, the appliance core network components are integrated with your existing data center network design. Uplink ports in the appliance switches connect to your next-level data center switches to provide a redundant high-speed and high-bandwidth physical connection that carries all traffic into and out of the appliance. In other words, from the private cloud environment perspective, the public network is your on-premises network, and internet access is always indirect through your data center edge router.

This private environment has several consequences when it comes to securing Oracle Private Cloud Appliance with regard to networking:

  • The rack is located on your premises, set up inside your data center, and connected directly to your on-premises network. There is no need for a secure tunnel over the internet to allow your cloud resources and your on-premises network to communicate. Access is enabled through a gateway between your virtual cloud network (VCN) and your on-premises network

  • When it comes to internet access, inbound or outbound, resources in your cloud environment have no direct internet access. In contrast with a public cloud environment, no gateway is capable of enabling direct internet connectivity for a VCN. The configuration of the networking components in the data center determines how cloud resources can connect to the internet and whether they can be reached from the internet.

  • Generally, this private environment means that theOracle Private Cloud Appliance virtual network is well-isolated from public internet threats. However, It is possible to use public IP addresses inside this private network. A public IP makes a resource reachable from outside the VCN it resides in. The IP addresses that are considered "public" are really part of the data center's private range.

Nevertheless, there are steps that can be taken to control cloud network security and access to compute instances:

  • Use private subnets if instances do not require a public IP address.

  • Configure firewall rules on the instance to control traffic into and out of an instance at the packet level. However, Oracle-provided images that run Oracle Linux automatically include default rules that allow ingress on TCP port 22 for SSH traffic. In addition, the Microsoft Windows images include default rules that allow ingress on TCP port 3389 for Remote Desktop access.

  • Configure gateways and route tables to allow only required connectivity. This can control traffic flow to "outside" destinations such as your on-premises network or another VCN.

  • Use IAM policies to control access to Oracle Private Cloud Appliance interfaces. You can control which cloud resources can be accessed and which type of access is allowed. For example, you can control who can set up your network and subnets, or who can update route tables, network security groups, or security lists.

For more information on Oracle Private Cloud Appliance network security, see the Oracle Private Cloud Appliance User Guide and Oracle Private Cloud Appliance Administrator Guide .

Networking and DNS

Oracle Private Cloud Appliance networks require DNS service in three areas:

  • Kubernetes DNS in Service Enclave.

  • Authoritative DNS service in Customer Data Center Network for <pca>.<domain>.

  • Authoritative DNS service inCompute Enclave (accessible from subnets in customer tenancy) for <pca>.<domain> and internalpca.local.

  • Recursive DNS in Compute Enclave for foreign zones.

Kubernetes provides a DNS service using CoreDNS that gets dynamically configured with service endpoints when the corresponding Kubernetes service is deployed. One or both of the other two DNS services may be able to leverage this already required CoreDNS deployment. In the worst case each could use a separate DNS deployment using CoreDNS or another implementation.

Creating and Maintaining Accounts and Passwords

When the Oracle Private Cloud Appliance system is first powered on, various tasks need to be performed in order to initially set up the system. These tasks include defining the first super-admin user account, configuring system parameters such as system and domain name, and configuring basic networking parameters that make the appliance part of the data center network.

There are three layers of the Oracle Private Cloud Appliance, each of these three layers will be administered in a different way:

  • Infrastructure - The rack hardware contains default users and passwords. Infrastructure does not usually need to be accessed in day-to-day operations. After installation, update any default passwords and store them in a secure location. Individual accounts are not supported in the infrastructure context. On most infrastructure components there are only two accessible users, the administrative user and an account for Oracle Support Services that can only be accessed with support from the administrative user.

  • Service Enclave - The administrative user created during installation applies to this layer. Additional users can be added to the Service Enclave to help manage the Oracle Private Cloud Appliance but, in practice, there are few of these users as their operations are across the entire appliance.

  • Compute Enclave - Users of the Compute Enclave have day-to-day tasks within a tenancy. They may be creating compute instances or monitoring cloud resources. When an administrative user creates a tenancy, they define an administrator for the tenancy that can then expand access to the tenancy from the Compute Enclave console.

Each layer has different requirements and techniques for maintaining the user accounts.

Password Maintenance in the Infrastructure Layer

The infrastructure layer is not accessed in day-to-day operations and is not intended to be a multi-user experience. Infrastructure is largely managed either through the Service Enclave layer, or by the individual software components running within the Service Enclave. Change any default passwords immediately after successful rack installation and configuration.

Passwords to be updated include:

  • Compute node passwords

  • Compute node Oracle Integrated Lights Out Manager (ILOM) passwords

  • Management node passwords

  • Management node ILOM passwords

  • Leaf switch password

  • Management switch password

  • Spine switch password

  • Oracle ZFS Storage Appliance password

  • Oracle ZFS Storage ApplianceILOM password

There is a tool available on the management nodes to check for default passwords in the infrastructure that must be changed. To run it:

  1. Log into a management node using the default administrative user and password supplied to you by the installation team.

  2. Run the following command: /var/lib/pca-foundation/scripts/healthcheck.py.

The output of the tool will show passwords to change from factory defaults.

All passwords except the MySQL database password must be updated using the pca-admin tool that resides on the management node. Using the various native infrastructure tools will result in Service Enclave failures. An example of why the tool must be used is with the management node passwords. Updating the management nodes using the pca-admin tool keeps all management node passwords synchronized. The pca-admin tool also stores the password for the management nodes in a service-accessible database, allowing the Service Enclave tools and services to manage the nodes.

The MySQL database password is updated using the following command from one of the management nodes: /var/lib/pca-foundation/scripts/pca_change_mysql_root_password.py

Passwords are stored in two locations:

  • A vault instance where they are stored using 256-bit Advanced Encryption Standard (AES) cipher in the Galois Counter Mode (GCM) with 96-bit nonces.

  • The native password tool for the infrastructure component

Refer to documentation for the infrastructure component information for password access information.

The password complexity policy for infrastructure components is:

  • Length is 8 characters to 20 characters

  • The password must contain a character from each of the groups

    • Lower case letters (a-z)

    • Upper case letters (A-Z)

    • Digits (0-9)

    • Symbols (@$!#%*?&)

The password complexity policy for the infrastructure cannot be changed.

To update an infrastructure password

  • Start the pca-admin tool on any of the management nodes (after logging in)

  • Use the change password <component> <password> command. For example: change password zfs <updated_password>.

The password update can take a short amount of time to complete even after a successful password update response is returned.

The Power Distribution Units (PDUs) in the PCA racks vary depending on the locale. Refer to the PDU hardware documentation for information regarding password updates. Refer to the Oracle Private Cloud Appliance Release Notes for potential updates on the infrastructure processes.

Service Enclave User and Password Maintenance

At installation and configuration time, an initial user with the SuperAdmin Authorization Group and password is set up for the Service Enclave, refer to the configuration chapter of the Oracle Private Cloud Appliance Installation Guide

The Service Enclave is a multi-user environment where users do not share credentials. Because actions in the Service Enclave affect all tenancies on the appliance, very few users are necessary in this space. General security guidelines are:

  • Do not share credentials.

  • Create a user for each individual that requires access to the Service Enclave administration tools. This practice enables better audit tracking and easier administration of individual needs.

  • Apply the rule of least privileges by choosing the authorization group most appropriate for the individual.

  • When creating a new user, do not use a common password and do not use a default initial password for new users.

  • Change passwords regularly. There are no proactive password change or timeout notifications in the Service Enclave.

There are 4 important authorization groups in the Service Enclave:

  • Admin - Authorization for most operations except user management

  • Monitor - A read-only role that can only manage their own profile or browse service enclave information without changing it

  • SuperAdmin - Authorization for all capabilities, only a SuperAdmin can create new users for the Service Enclave and change roles for existing users

  • DrAdmin - Authorization for setting up Disaster Recovery (this group is used only during Day-0 configuration)

In the Service Enclave, the list of authorization groups is static. Existing groups cannot be modified to change authorizations and new groups cannot be created with different authorizations.

The password policy for the Service Enclave is as follows:

  • Password has a minimum length of 12 characters

  • Password contains at least one uppercase letter

  • Password contains at least one lowercase letter

  • Password contains at least one symbol (@$!#%*?&)

  • Password contains at least one number

The Service Enclave password policy cannot be changed.

Password storage is in the service database and uses Password Based Key Derivation Function 2 (PBKDF2) with a 32 character salt to hash the password.

User maintenance and password maintenance for the Service Enclave for a user in the SuperAdmin group is done using the Service Enclave Administration Console (https://adminconsole.<domain> ) or the Admin CLI. A user in the MONITOR or ADMIN group must use the Admin CLI or request a password change from a user in the SuperAdmin authorization group.

From the Admin CLI, use the command to change the password:

changePassword id= <id> password= <new_password> confirmPassword= <new_password>

To get the ID for the current user, use the command:

show UserPreference

A user in the SuperAdmin group can change the password of any user in the Admin CLI. A list of user IDs can be obtained using the command: list user.

Additional notes on password management in the 3.0.1 software version:

  • A user in the MONITOR or ADMIN group cannot change their password in the Service Enclave Administration Console.

  • There is no password recovery or password reset functionality, as a result, it is strongly advised that a second user with SuperAdmin capability is created.

  • Accounts will not be locked out or disabled with any number of invalid login attempts.

For more information on use of Identity Providers with the Service Enclave layer, see Service Enclave Security Features.

Compute Enclave User and Password Maintenance

There are no default Compute Enclave users or tenancies immediately following an Oracle Private Cloud Appliance Installation Guide install and configuration.

When a Service Enclave administrator creates a tenancy, an initial user is created and a password is assigned.

Have the new tenancy administrator log into the account and change their password using the Compute Enclave console (https://console.<domain> ).

Once logged in, use the Change Password drop down located in the top right of the console where the user name is displayed. The tenancy administrator is the only user account that cannot be reset by any user (including themselves). The only option available to the primary tenancy administrator created by the Service Enclave SuperAdmin is to store their password securely and use the Change Password action in the user interface after a successful login.

The password policy for the Compute Enclave is as follows:

  • Password has a minimum length of 12 characters

  • Password contains at least one uppercase letter

  • Password contains at least one lowercase letter

  • Password contains at least one symbol (@$!#%*?&)

  • Password contains at least one number

The password policy cannot be changed.

Password storage is in the service database and uses Password Based Key Derivation Function 2 (PBKDF2) with a 32 character salt to hash the password.

The tenancy administrator also sets up the CLI with their account so they can start making use of Identity and Access Management (IAM) operations. For instructions on setting up the CLI to use with a tenancy and user on the Oracle Private Cloud Appliance refer to Working in the Service Enclave.

Every tenancy comes with a default administrators group. This group can perform any action on all resources in a tenancy (that is, they have root access to the tenancy). Oracle recommends that you keep the group of tenancy administrators as small as possible but have at least one backup administrator.

Some security recommendations on managing tenancy administrators:

  • Have security policies granting membership of tenancy administrator group strictly on a as-needed basis.

  • Tenancy administrators use high-complexity passwords, along with MFA, and periodically rotate their passwords.

  • After account set up and configuration, Oracle recommends that you don't use the tenancy administrator account for day-to-day operations. Instead, create less privileged users and groups.

  • Though administrator accounts are not used for daily operations, they are still needed to address emergency scenarios impacting customer tenancy and operations. Specify secure and auditable "break-glass" procedures for using administrator accounts in such emergencies.

  • Disable tenancy administration access immediately when an employee leaves the organization.

  • Because the tenancy administrator group membership is restricted, Oracle recommends that you create security policies which prevent administrator account lock-out (for example, if the tenancy administrator leaves the company and no current employees have administrator privileges).

Users in the Compute Enclave other than the default administrator for a tenancy can have their passwords reset by a user in the Administrator group through the Users context in the Compute Enclave console (https://adminconsole.<domain> ).

A user can reset their own password if they have the CLI installed and configured via the command:

oci iam user ui-password create-or-reset --user-id <id>

An administrator can use the CLI command to reset any other user's password. After reset, the administrator securely communicates the reset password to the user. The user will be prompted on login to the console to change the password.

For more information on the user of Identity Providers with the Compute Enclave and IAM security features to help administer the Compute Enclave layer, refer to Identity Provider Security Features.

Maintaining the Monitoring and Logging Password

The monitoring and logging facilities for Oracle Private Cloud Appliance are accessed via consoles at:

  • Grafana: https://grafana.<domain>

  • Prometheus: https://prometheus.<domain>

In Oracle Private Cloud Appliance 3.0.1, the infrastructure layer has a single user for both Grafana and Prometheus (admin), and they have a default password. Change this password after installation and configuration.

For proper security, it is required that admins set the new password by running the Python CLI tool sauron_credential_update.py from the management node.

The password policy requires that the password:

  • Must be 12-20 characters long

  • Must contain at least 1 uppercase, 1 lowercase and one digit

  • Can contain the symbols -_+=

The monitoring and logging tools in Oracle Private Cloud Appliance 3.0.1 have the following restrictions:

  • More users cannot be added

  • The credential update tool does not check the password or return information on success or failure of the request

  • The Grafana and Prometheus screens do not lock out users after invalid attempts

Due to these characteristics:

  • Choose a complex password

  • Designate a user as the focal point for monitoring and logging

  • Do not share the password for the admin account

  • Change the password regularly

  • Do not share the password with tenancy administrators (Grafana and Prometheus contains logs for all tenancies and should therefore not be shared with a tenancy administrator due to information leakage between tenancies)

Viewing Failed Password Log-in Attempts

Security monitoring for all layers (Infrastructure, Service Enclave and Compute Enclave) can be viewed in the Grafana instance (https://grafana.<domain>).

To view logs that include security notifications:

From that data source, choose the infrastructure or software component to view information. The various components can be filtered using log labels for:

  • Management and compute nodes, use the host filter, such as {host="pcamn01"}

  • Oracle ZFS Storage Appliance, use {log="audit"}

  • Compute Enclave, use {log="api-server"}

  • Other labels are possible

Some components cannot be monitored at this tier:

  • Switches (Spine, Leaf and Management)

  • ILOM

If audit logs are required for those components, log directly into the component.