Post-Installation Configuration

This section describes the security configuration changes that must be made after installation. Generally, do not weaken the security posture of the Oracle Private Cloud Appliance when making changes to configurations.

Securing the Hardware

After installation of Oracle Private Cloud Appliance, secure the hardware by restricting access to the hardware and recording the serial numbers.

Oracle recommends the following practices to restrict access:

Install Oracle Private Cloud Appliance and related equipment in a locked, restricted-access room.
Lock the rack door unless service is required on components within the rack.
Restrict access to hot-pluggable or hot-swappable devices because the components are designed to be easily removed.
Store spare field-replaceable units (FRUs) or customer-replaceable units (CRUs) in a locked cabinet. Restrict access to the locked cabinet to authorized personnel.
Limit SSH listener ports to the management and private networks. Use SSH protocol 2 (SSH-2) and FIPS 140-2 approved ciphers.
Limit SSH allowed authentication mechanisms. Inherently insecure methods are disabled.
Label all significant items of computer hardware, such as FRUs.
Keep hardware activation keys and licenses in a secure location that is easily accessible to the system managers in the case of a system emergency.

Retrieving the Appliance Serial Number

An Oracle Private Cloud Appliance has a serial number that identifies the appliance as a whole entity. When working with Oracle Support Services, the appliance serial number may be required.

There is a label on the rack with the serial number. The serial number label is the white label about midway up the front right rail of the rack.

There are several other techniques to obtaining the overall appliance serial number:

Use the Service Enclave console (Administrative Console) - To get the appliance serial number from the Service Enclave console:
1. Use a supported web browser to log into the console with a username and password that has sufficient authorization (https://adminconsole.<domain>).
2. If you just completed the initial setup, then use the admin username with the configured password (otherwise the administrator may provide you with a specific user for access).
3. Access the Appliance Details section from the menu.
4. Record the ID and the Realm from this page. The Realm is the physical appliance serial number: record both the ID and the physical appliance serial number.
Use the appropriate monitoring dashboard (Grafana) - To obtain the appliance serial number from the monitoring dashboard:
1. Use a supported web browser to log into the dashboard with a username and Password that has sufficient authorization (https://grafana.<domain>).
2. Go to the Dashboards → Manage Dashboards section.
3. Open the Service Monitoring → Hardware Stats dashboard. The appliance serial number is displayed on this dashboard.
4. Record the ID and the Realm from this page. The Realm is the physical appliance serial number: record both the ID and the physical appliance serial number.
Use the Admin Command Line Interface (CLI) - To obtain the appliance serial number using the CLI:
1. Log into the Admin CLI with a username and password that has sufficient authorizations (use the command ssh -l <username>management host -p 30006). The management host is one of the three management nodes (pcamn01, pcamn02, or pcamn03) and the username, by default, is admin. If the section on rotating the passwords is not completed yet, then use the password supplied during initial setup.
2. From the prompt, there are a variety of ways to get the rack serial number. A quick report can be obtained with the command: list Rack fields name,rackNumber,rackSerialnumber,rackType.
  
  For example,
```
PCA-ADMIN> list Rack fields rackSerialnumber
Command: list Rack fields rackSerialnumber
Status: Success
Time: 2022-02-17 20:15:41,773 UTC
Data:
  id                                   rackSerialnumber
  --                                   ----------------
  x6d9f31a-b1ds-4a43-bc09-7270609abd8k CL00888510
PCA-ADMIN>
```
3. Store the information from the Appliance Details in a secure location.

Retrieving the Serial Numbers for Hardware Components in the Rack

An Oracle Private Cloud Appliance is an engineered system that is made up of many hardware components. For simplicity we will call these the rack components, though the components can span multiple racks. The most efficient way to collect the serial numbers for the rack components is to use the Admin CLI, though they can also be obtained from the Service Enclave console (Administrative Console). These serial numbers may also be needed when working with Oracle Support Services.

To get a list of rack component serial numbers from the Admin CLI:

Log into the Admin CLI with a username and password that has sufficient authorizations (use the command ssh -l <username>management host -p 30006). More details are in the Retrieving the Appliance Serial Number section.
Once logged in, get a report using a command such as the following: list RackUnit fields name,rackElevation,serialNumber,hostname.
Store the information recorded above in a secure location.

To use the Service Enclave console, log into the console with a username and password that has sufficient authorization (https://adminconsole.<domain>). After logging in:

Using the menu, navigate to the Rack Units context.
For each unit in the list, choose to View Details:
1. Once in the details, choose the System tab.
2. Record the Serial Number along with other component information.
Return to the Rack Units list and record the next component.
Store the information recorded above in a secure location.

Securing the Software

After installation of Oracle Private Cloud Appliance, secure the software. In many cases, hardware security is implemented through software.

After initial installation of Oracle Private Cloud Appliance, Oracle recommends the following practices to restrict system access:

Limit use of the super-user account, such as root on the management nodes and SuperAdmin accounts in the Service Enclave and Compute Enclave. Create and use individual user accounts because they ensure positive identification in audit trails, and require less maintenance when administrators leave the team or company.
Do not create new users on the management nodes.
Disable unnecessary protocols and modules for layers under customer control.
Restrict physical access to USB ports, network ports, and system consoles because physical severs and network switches have ports and console connections providing direct access to the system.
Restrict the capability to restart the system over the network.
For more information on how to enable other security features, see Security Features for Oracle Private Cloud Appliance in this guide.

By using the privileged users and multi-factor access control, and other tools such as data encryption, auditing, monitoring, and data masking, customers can deploy reliable data security solutions that do not require any changes to existing applications.

Securing the Network

Oracle Private Cloud Appliance is a private cloud environment. During initialization, the appliance core network components are integrated with your existing data center network design. Uplink ports in the appliance switches connect to your next-level data center switches to provide a redundant high-speed and high-bandwidth physical connection that carries all traffic into and out of the appliance. In other words, from the private cloud environment perspective, the public network is your on-premises network, and internet access is always indirect through your data center edge router.

This private environment has several consequences when it comes to securing Oracle Private Cloud Appliance with regard to networking:

The rack is located on your premises, set up inside your data center, and connected directly to your on-premises network. There is no need for a secure tunnel over the internet to allow your cloud resources and your on-premises network to communicate. Access is enabled through a gateway between your virtual cloud network (VCN) and your on-premises network
When it comes to internet access, inbound or outbound, resources in your cloud environment have no direct internet access. In contrast with a public cloud environment, no gateway is capable of enabling direct internet connectivity for a VCN. The configuration of the networking components in the data center determines how cloud resources can connect to the internet and whether they can be reached from the internet.
Generally, this private environment means that theOracle Private Cloud Appliance virtual network is well-isolated from public internet threats. However, It is possible to use public IP addresses inside this private network. A public IP makes a resource reachable from outside the VCN it resides in. The IP addresses that are considered "public" are really part of the data center's private range.

Nevertheless, there are steps that can be taken to control cloud network security and access to compute instances:

Use private subnets if instances do not require a public IP address.
Configure firewall rules on the instance to control traffic into and out of an instance at the packet level. However, Oracle-provided images that run Oracle Linux automatically include default rules that allow ingress on TCP port 22 for SSH traffic. In addition, the Microsoft Windows images include default rules that allow ingress on TCP port 3389 for Remote Desktop access.
Configure gateways and route tables to allow only required connectivity. This can control traffic flow to "outside" destinations such as your on-premises network or another VCN.

Note:
Communication between VCNs on different DRGs residing on a PCA rack is possible if route entries and firewall access on the customer data center network that connects the two VCNs are provided by the customer.
Use IAM policies to control access to Oracle Private Cloud Appliance interfaces. You can control which cloud resources can be accessed and which type of access is allowed. For example, you can control who can set up your network and subnets, or who can update route tables, network security groups, or security lists.

For more information on Oracle Private Cloud Appliance network security, see the Oracle Private Cloud Appliance User Guide and Oracle Private Cloud Appliance Administrator Guide .

Networking and DNS

Oracle Private Cloud Appliance networks require DNS service in three areas:

Kubernetes DNS in Service Enclave.
Authoritative DNS service in Customer Data Center Network for <pca>.<domain>.
Authoritative DNS service inCompute Enclave (accessible from subnets in customer tenancy) for <pca>.<domain> and internalpca.local.
Recursive DNS in Compute Enclave for foreign zones.

Kubernetes provides a DNS service using CoreDNS that gets dynamically configured with service endpoints when the corresponding Kubernetes service is deployed. One or both of the other two DNS services may be able to leverage this already required CoreDNS deployment. In the worst case each could use a separate DNS deployment using CoreDNS or another implementation.

Port Matrix

The default security posture for almost all firewalls is to deny access. This applies to the firewalls used between the Oracle Private Cloud Appliance rack and the customer data center. In order to allow certain Oracle Private Cloud Appliance features to operate correctly, access must be granted specifically for certain IP address and related services. An "allow all" rule such as 0.0.0.0/0 is too broad for security purposes, so the best practice is to explicitly list addresses. ports, and protocols to allow.

See the following sections for specific information when the administration network is enabled and not enabled:

Access Configuration Without Administration Network

The following table lists the access permissions that should be granted for certain IP addresses, ports, and protocols, along with an explanation of their purpose. This table is for configurations without admin network isolation. That is, there is NO dedicated port used for the administration network traffic. For more information about using the administration network, see Editing Administration Network Information in the Hardware Administration chapter of the Oracle Private Cloud Appliance Administrator Guide.

Source IP Address	Destination IP Address	Port	Protocol	Description
All customer networks	SE/CE combined VIP		ICMP	Type 0/Echo reply
All customer networks	Management Nodes		ICMP	Type 0/Echo reply
All customer networks	Object Storage IP		ICMP	Type 0/Echo reply
All customer networks	SE/CE combined VIP		ICMP	Type 3/Unreachable
All customer networks	Management Nodes		ICMP	Type 3/Unreachable
All customer networks	Object Storage IP		ICMP	Type 3/Unreachable
All customer networks	SE/CE combined VIP		ICMP	Type 8/ping to VIP
PCA administrators	Management Nodes		ICMP	Type 8/ping to node
All customer networks	Object Storage IP		ICMP	Type 8/ping to VIP
PCA administrators	SE/CE combined VIP	22	TCP	SSH to Active Management Node
PCA administrators	Management Nodes	22	TCP	SSH to Specific Management Node
Initial installation DNS IPs	SE/CE combined VIP	53	UDP	PCA Authoritative Zones
Initial installation DNS IPs	Management Nodes	53	TCP	PCA Authoritative Zones
Management Nodes	Initial installation DNS IPs	53	UDP	External DNS Resolution
Management Nodes	Initial installation DNS IPs	53	TCP	External DNS Resolution
PCA administrators	SE/CE combined VIP	443	TCP	SE API Endpoints and SE BUI
All CE users	SE/CE combined VIP	443	TCP	CE API Endpoints and CE BUI
All CE users	SE/CE combined VIP	8079	TCP	Image download repository
PCA administrators	SE/CE combined VIP	30006	TCP	Admin CLI
PCA administrators	Management Nodes	30006	TCP	Admin CLI

SE/CE combined VIP	DNS Recursive Servers	53	UDP	DNS Forwarding
SE/CE combined VIP	DNS Recursive Servers	53	TCP	DNS Forwarding
SE/CE combined VIP	Customer NTP Servers	123	UDP	NTP
SE/CE combined VIP	Usually transport.oracle.com	443	TCP	ASR Targets (https://docs.oracle.com/en/engineered-systems/ private-cloud-appliance/3.0/admin-3.0.2/admin-adm-healthmonitor.html#adm-health-asr)
SE/CE combined VIP	Grafana Notification Targets	443	TCP	Grafana Notification Targets (https://docs.oracle.com/en/engineered-systems/ private-cloud-appliance/3.0/admin-3.0.2/admin-adm-healthmonitor.html#adm-health-status)
SE/CE combined VIP	Local ULN Mirror	443	TCP	PCA patching
SE/CE combined VIP	Local image repository	443	TCP	Custom image import from-object-uri

Access Configuration With Administration Network

The following table lists the access permissions that should be granted for configurations with admin network isolation, when there is a dedicated port used for the admin network traffic.

For added security, you can isolate the Admin Network from the normal tenant traffic flows. Admin network isolation achieves the following goals:

Separate the tenant and administrator access paths by using differentxternal port sets.
Separate the tenant and administrator access to selected services that are exposed externally and internally to the VCNs.
Separate the tenant and administrator access to DNS services exposed internally to infrastructure and to VCNs.

For more information about using the administration network, see Editing Administration Network Information in the Hardware Administration chapter of the Oracle Private Cloud Appliance Administrator Guide.

Source IP Address	Destination IP Address	Port	Protocol	Description
All customer networks	CE VIP		ICMP	Type 0/Echo reply
PCA administrators	SE VIP		ICMP	Type 0/Echo reply
All customer networks	Management Nodes		ICMP	Type 0/Echo reply
All customer networks	Object Storage IP		ICMP	Type 0/Echo reply
All customer networks	CE VIP		ICMP	Type 3/Unreachable
PCA administrators	SE VIP		ICMP	Type 3/Unreachable
All customer networks	Management Nodes		ICMP	Type 3/Unreachable
All customer networks	Object Storage IP		ICMP	Type 3/Unreachable
All customer networks	CE VIP		ICMP	Type 8/ping to VIP
PCA administrators	SE VIP		ICMP	Type 8/ping to VIP
PCA administrators	Management Nodes		ICMP	Type 8/ping to node
All customer networks	Object Storage IP		ICMP	Type 8/ping to VIP
PCA administrators	SE VIP	22	TCP	SSH to Active Management Node
PCA administrators	Management Nodes	22	TCP	SSH to Specific Management Node
Initial installation DNS IPs	CE VIP	53	UDP	PCA Authoritative Zones
Initial installation DNS IPs	CE VIP	53	TCP	PCA Authoritative Zones
Initial installation AdminDNS IPs	SE VIP	53	UDP	PCA Administrative Zone
Initial installation AdminDNS IPs	SE VIP	53	TCP	PCA Administrative Zone
Management Nodes	Initial installation DNS IPs	53	UDP	External DNS Resolution for Data Network
Management Nodes	Initial installation DNS IPs	53	TCP	External DNS Resolution for Data Network
Management Nodes	Initial installation AdminDNS IPs	53	UDP	External DNS Resolution for Admin Network
Management Nodes	Initial installation AdminDNS IPs	53	TCP	External DNS Resolution for Admin Network
PCA administrators	SE VIP	443	TCP	SE API Endpoints and SE BUI
All CE users	CE VIP	443	TCP	CE API Endpoints and CE BUI
All CE users	CE VIP	8079	TCP	Image download repository
PCA administrators	SE VIP	30006	TCP	Admin CLI
PCA administrators	Management Nodes	30006	TCP	Admin CLI

CE VIP	DNS Recursive Servers	53	UDP	DNS Forwarding
CE VIP	DNS Recursive Servers	53	TCP	DNS Forwarding
SE VIP	DNS Recursive Servers	53	UDP	DNS Forwarding
SE VIP	DNS Recursive Servers	53	TCP	DNS Forwarding

SE VIP	Customer NTP Servers	123	UDP	NTP
SE VIP	Usually transport.oracle.com	443	TCP	ASR Targets (https://docs.oracle.com/en/engineered-systems/ private-cloud-appliance/3.0/admin-3.0.2/admin-adm-healthmonitor.html#adm-health-asr)
SE VIP	Grafana Notification Targets	443	TCP	Grafana Notification Targets (https://docs.oracle.com/en/engineered-systems/ private-cloud-appliance/3.0/admin-3.0.2/admin-adm-healthmonitor.html#adm-health-monitor)
SE VIP	Local ULN Mirror	443	TCP	PCA patching
CE VIP	Local image repository	443	TCP	Custom image import from-object-uri
Admin management nodes	Load balancer public IP address	6443	TCP	Load balancer public IP address endpoint

BGP Authentication on Uplinks

BGP authentication is required on uplinks with dyanmic routing. For a dynamic network configuration with BGP authentication, enter the parameters on a single line:

PCA-ADMIN> setDay0DynamicRoutingParameters \ 
uplinkPortSpeed=100 \  
uplinkPortCount=2 \  
uplinkVlanMtu=9216 \  
spine1Ip=<ip_address> \  
spine2Ip=<ip_address> \  
uplinkNetmask=255.255.255.252 \  
mgmtVipHostname=<hostname> \  
mgmtVip=<ip_address> \  
ntpIps=<ip_address> \  
peer1Asn=50000 \  
peer1Ip=<ip_address> \  
peer2ASN=50000 \  
peer2Ip=<ip_address> \  
objectStorageIp=1<ip_address> \  
mgmt01Ip=<ip_address> \  
mgmt02Ip=<ip_address> \  
mgmt03Ip=<ip_address> \  
bgpTopology=topology-type  \
BGPAuthenticuation=true  \
BGPAuthenticationPassword=<bgp-password>  \
adminBGPAuthenticuation=true  \
adminBGPAuthenticationPassword=<admin-bgp-password>

The following example shows a dynamic mesh topology configuration using BGP authentication on the uplinks as well as the Admin network:

PCA-ADMIN> setDay0DynamicRoutingParameters \ 
uplinkPortSpeed=40 \  
uplinkPortCount=4 \  
uplinkVlanMtu=9216 \  
spine1Ip=10.nn.nn.17 \  
spine2Ip=10.nn.nn.25 \  
uplinkNetmask=255.255.255.252, 255.255.255.252 \  
mgmtVipHostname=example-vip \  
mgmtVip=10.nn.nn.46 \  
ntpIps=10.nn.nn.2 \  
peer1Asn=50000 \  
peer1Ip=10.nn.nn.34,10.nn.nn.30 \  
peer2ASN=50000 \  
peer2Ip=10.nn.nn.72 \  
objectStorageIp=10.nn.nn.1   
mgmt01Ip=10.nn.nn.44 \  
mgmt02Ip=10.nn.nn.45 \  
mgmt03Ip=10.nn.nn.46 \  
bgpTopology=mesh  \
BGPAuthenticuation=true  \
BGPAuthenticationPassword=<bgp-password>  \
adminBGPAuthenticuation=true  \
adminBGPAuthenticationPassword=<admin-bgp-password>

Note:

BGP authentication isn't enabled if you don't specify a password.

Because BGP often involves separate administrative domains, password coordination is necessary between those responsible for both ends of the BGP links.

The adminBGPpassword must be established and changed on both ends of the BGP links at the same time. This might require careful coordination between different administrators. If one BGP authentication password is changed and the other isn't, the link fails.

To verify success in BGP operation, run the command show bgp sessions.