TLS Overview and Configuration

7 TLS Overview and Configuration

Transport Layer Security (TLS) is used for end-to-end communication encryption.

TLS between a Recovery Appliance and client databases involves the use of certificates that authenticate and encrypt communication.

Certificates describe the server, who it belongs to, its connection string, etc. and is issued and signed by a trusted authority. Customers may choose third-party vendors or Oracle internal CA certificate authority.

For development and testing purpose, some customers choose to use self-signed certificate, which could be created by RACLI command.

Trusted Certificates are generally obtained from a trusted Certified Authority (CA) through an application process (at the corporate level). These certificates are generally used between external systems. Because they were created by the CA, these certificates do not contain any local host names. The file type is *.pem.
Signed Certificates are created as needed and contain the local host name as well as location and organization information as part of what authenticates it. These certificates are often used between local or internal systems. Signed certificates are specific to each Recovery Appliance. The file type is *.p12.

For TLS, both types of certificates are required.

This chapter provides general information on obtaining the certificates from a security website, as well as alternatively information on generating the certificates manually with RACLI commands. RACLI (racli create certificate) is a wrapper for openssl operations.

Whether obtained or generated, the created certificate is imported to the Recovery Appliance wallet using racli add certificate so that they are available for the network. Then, finally the

racli alter network establishes the needed encryption mode.

enable: dual mode allows both encrypted and un-encrypted data.
only: only encrypted data
disable: only un-encrypted data

Certificate Management

This section describes in general terms the process after obtaining TLS certificates from a Certified Authority (CA) and management with the Recovery Appliance.

A certificate authority (CA) is a company or organization that acts to validate the identities of entities (such as websites, email addresses, companies, or individual persons) and bind them to cryptographic keys through the issuance of electronic documents known as digital certificates. A CA acts as a trusted third party for both the subject (owner) of the certificate and the party relying upon the certificate.

A digital certificate provides:

Authentication; the certificate serves as a credential to validate the identity of its owner. In this case, it authenticates communication from the Recovery Appliance to its protected databases, to other replication Recovery Appliances, and to cloud archival storage.
Encryption for secure communication over insecure networks such as the Internet.
Integrity of documents signed with the certificate so that they cannot be altered by a third party in transit.

The format of these certificates is specified by the X.509 standard.

The techniques vary between CAs for validating the domain to prove that certificate applicant controls the given domain name.

Likewise each CA has its own application steps that are beyond the scope of this chapter to detail.

In general, upon completion of the certificate application process with your chosen CA, you (the applicant) downloads a bundle file containing all of your certificates.

The following assumes that you have that bundle file (*.pfx), named in this example YourCompany.pfx, generated by your chosen CA.

CA Bundle File (*.pfx) of Certificates

The Recovery Appliance TLS encryption requires both a trusted certificate (*.pem) and signed certificate (*.p12). Each certificate needs to be extracted from the bundle file (*.pfx) and then imported into the TLS wallet.

Certificate Creation Using Third Party Software

On the Recovery Appliance, obtain a list of all subject alternative names (SAN) by issuing the following command.
```
racli list san
```
Note:
If this returns nothing, patch to a newer version.
With the SAN information available particularly for common name (CN) and DNS entries, go to your security website and enter this information to obtain the certificate package.

Use the PKCS#8 format and make sure to specify separate files.
Download the certificate ZIP package.
Unzip the certificate ZIP package.

The certificate ZIP package contains several files including trusted certificate and user certificate.
- The trusted certificate may have chain or root in its name, and it is *.pem format.
- User certificate is in *.crt format.
- The *.key file should also in this directory from downloaded package.
With openssl pkcs12, sign the user certificate with the trusted certificate to create a *.p12 file.
```
openssl pkcs12 -export --in /<DIR>/<NAME>.crt 
--inkey /<DIR>/<NAME2>.key --certfile  /<DIR>/<NAME3>.pem 
--passin pass:<YOURPASSWORD> --passout pass: :<YOURPASSWORD>  
--out /<DIR>/<NAME4>.p12
```
Note:
Do not use ewallet or cwallet for <NAME4>. <NAME4> should refer to local host information or organization name used for <NAME>, <NAME2>, and <NAME3>.
Import both the trusted certificate and the signed user certificate into the Recovery Appliance wallet.
```
racli add certificate --signed_cert=/<DIR>/<NAME4>.p12 
--trusted_cert=/<DIR>/<NAME3>.pem
```
Verify the certificates are in the Recovery Appliance wallet.
```
racli list certificate 
```
Continue with Configuring TLS Data Security on the Recovery Appliance followed by Configuring TLS Data Security on the Client.

Using Your Organization's CA process for TLS Certificates

This section details how to create TLS certificates using openssl.

Large organizations or government bodies, as examples, may have their own PKIs (public key infrastructure), each containing their own CAs.

For the case that your organization has its own certificate process, this section explains how to integrate Recovery Appliance certificates.

Prepare Information for the Certificates

On a Recovery Appliance as an admin_user or root, run this command.

racli list san


    Created log /opt/oracle.RecoveryAppliance/log/racli_list_san.log
    Thu May  6 16:18:33 2021: Start: List SAN
    CN = zdlra09ingest-scan1.yourdomain.com
    DNS.1 = zdlra09adm01.yourdomain.com
    DNS.2 = zdlra09adm02.yourdomain.com
    DNS.3 = zdlra09ingest-scan1.yourdomain.com
    DNS.4 = zdlra09ingest01-vip.yourdomain.com
    DNS.5 = zdlra09ingest01.yourdomain.com
    DNS.6 = zdlra09ingest02-vip.yourdomain.com
    DNS.7 = zdlra09ingest02.yourdomain.com
    Thu May  6 16:18:39 2021: End: List SAN

The CN (Common Name) item from your host is <yourScanName> which later corresponds to certificate files.

In this example, <yourScanName> is "zdlra09ingest-scan1", the signed certificate file is <yourScanName>.p12, and the trusted certificate is <yourScanName>.pem.

Use an editor to create a CRT configuration file for your organization's certification/security process..

In this example, it is named <YOUR_CONFIG2>. In your environment, all of the constructs with YOUR_... or yourDir are replaced with specific information from the local instance. And the <YOUR_DNS> items are replace with information obtained in previous step using racli list san.

[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = v3_req
distinguished_name = dn
 
[ dn ]
C=$args{YOUR_COUNTRY}
ST=$args{YOUR_STATE}
L=$args{YOUR_LOCATION}
O=$args{YOUR_ORGANIZATION}
OU=$args{YOUR_ORGANIZATION_UNIT}
emailAddress=$args{YOUR_EMAIL_ADDRESS}
CN = $list_san->{CN}
[ v3_req ]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = \@alt_names
[alt_names]
DNS.1 = <YOUR_DNS.1>
DNS.2 = <YOUR_DNS.2>

Upload the <YOUR_CONFIG2>.CRT file to your organization's certification/security process.
Note:
When your CA organization generates the bundle:
- Choose the format PEM (OpenSSL).
- Check the option for including CRT file, because <YOUR_CONFIG2> is needed.
From your organization's certification/security process, download the whole package to a location designated as <yourDir>. This example assumes it is named <yourDownload>.crt.

The trusted certificate, <yourDownload>.pem, is within that package and is used from this package in later steps to generate a signed certificate.
On a Recovery Appliance as an admin_user or root, run this command to generate the key file. In this example, it is yourScanName.key.
```
openssl genrsa --passout pass:<yourPassword> --out <yourDir>/<yourScanName>.key 2048
```

Get the certificate signed by the trusted certificate using the pkcs12 format.

openssl pkcs12 --export --in <yourDir>/<yourDownload>.crt 
    --inkey <yourDir>/<yourScanName>.key
    --certfile <yourDir>/<yourDownload>.pem 
    --passin pass:<yourPassword> 
    --passout pass:<yourPassword> 
    --out <yourDir>/<yourScanName>.p12

Import the signed certificate into the TLS wallet.

racli add certificate --signed_cert=<yourDir>/<yourScanName>.p12

Import the trusted certificate into the TLS wallet.

racli add certificate --trust_cert=<yourDir>/<yourScanName>.pem

After importing the certificates, verify with "racli list certificate" that the certificates are in the raa_certs database table.

# racli list certificate

Created log /opt/oracle.RecoveryAppliance/log/racli_list_certificate.20230329.1146.log
Wed Mar 29 11:46:49 2023: Start: List Certificate
Serial: 9A15CB4B76BBC52D
    Expire Time:      2024-03-28
    Certificate Type: trusted_cert
 
Serial: 95B9181340F644F0
    Expire Time:      2024-03-28
    Certificate Type: signed_cert
 
Wed Mar 29 11:46:49 2023: End: List Certificate

Continue with Configuring TLS Data Security on the Recovery Appliance followed by Configuring TLS Data Security on the Client.

Manually Creating TLS Certificates with RACLI

This section details how to create TLS certificates with RACLI.

If your organization does not have or is not using a Certificate Authority (CA), these instructions allow you to create the needed trusted and signed certificates for TLS operations.

The following information is required for both the trusted and signed certificates:

Country Name
State Name
Organization Name
Organization Unit Name
Email Address

Generate Trusted and Signed Certificates using RACLI.

With the organization information on-hand, issue an RACLI command similar to:

# racli create certificate --country=US --state=CA --location=SF --organization=oracle 
--organization_unit=zdlra --email_address=<YOUR_EMAIL>

Created log /opt/oracle.RecoveryAppliance/log/racli_create_certificate.20230329.1110.log
Enter New Password for Certificate:
Confirm New Password for Certificate:
Wed Mar 29 11:11:22 2023: Start: Create TLS Trusted Certificate
Wed Mar 29 11:11:26 2023: End: Create TLS Trusted Certificate
Wed Mar 29 11:11:26 2023: Start: Create TLS Signed Certificate
Wed Mar 29 11:11:31 2023: End: Create TLS Signed Certificate
Certificate(s) created under /raacfs/raadmin/config/cert

The name of the certificate created is <yourScanName>.p12, where <yourScanName> is the CN for your environment.

To obtain <yourScanName> local host information and Common Name (CN) item.

racli list san

    Created log /opt/oracle.RecoveryAppliance/log/racli_list_san.log
    Thu May  6 16:18:33 2021: Start: List SAN
    CN = zdlra09ingest-scan1.yourdomain.com
    DNS.1 = zdlra09adm01.yourdomain.com
    DNS.2 = zdlra09adm02.yourdomain.com
    DNS.3 = zdlra09ingest-scan1.yourdomain.com
    DNS.4 = zdlra09ingest01-vip.yourdomain.com
    DNS.5 = zdlra09ingest01.yourdomain.com
    DNS.6 = zdlra09ingest02-vip.yourdomain.com
    DNS.7 = zdlra09ingest02.yourdomain.com
    Thu May  6 16:18:39 2021: End: List SAN

In this example, <yourScanName> is "zdlra09ingest-scan1" and the certificate file is <yourScanName>.p12.

You assign the certificate type (trusted or signed) later when added to the wallet.

Import Certificates into Wallet

Upon completion of creating the trusted and signed certificates, <yourScanName>.pem and <yourScanName>.p12 respectively, import them into the Recovery Appliance wallet for TLS.

Import the certificates (trusted or signed) from the previous steps into the wallet. Here is the generic command, while specific examples are in the next steps.
```
racli add certificate { [--trusted_cert=<VALUE>] |
      [--signed_cert=<VALUE>] | [--self_signed] }
```
Arguments:
- --trusted_cert=<VALUE>: Specify the full path and name of the trusted certificate to be added.
- --signed_cert=<VALUE>: Specify the full path and name of the signed certificate in the trusted store to be added.
- --self_signed: Specifies that Recovery Appliance will look for both certificates from designated locations. This should only be used when the certificates were created by "racli create certificate". This is not the Oracle recommended configuration, and is used only in a test environment.
  
  Note:
  Self-signed certificates should not be used long-term or for production. The recommendation is to use a (trusted) certificate signed by your Certification Authority.
Import the signed certificate into the TLS wallet. If the certificate was created by RACLI, include the --self_signed argument.
```
racli add certificate --signed_cert=<yourDir>/<yourScanName>.p12 [--self_signed]
```
Import the trusted certificate into the TLS wallet. If the certificate was created by RACLI, include the --self_signed argument.
```
racli add certificate --trust_cert=<yourDir>/<yourScanName>.pem [--self_signed]
```

After importing the certificates, verify with "racli list certificate" that the certificates are in the raa_certs database table.

# racli list certificate

Created log /opt/oracle.RecoveryAppliance/log/racli_list_certificate.20230329.1146.log
Wed Mar 29 11:46:49 2023: Start: List Certificate
Serial: 9A15CB4B76BBC52D
    Expire Time:      2024-03-28
    Certificate Type: trusted_cert
 
Serial: 95B9181340F644F0
    Expire Time:      2024-03-28
    Certificate Type: signed_cert
 
Wed Mar 29 11:46:49 2023: End: List Certificate

Continue with Configuring TLS Data Security on the Recovery Appliance followed by Configuring TLS Data Security on the Client.

After a certificate is in the raa_certs database table and when it has less than 90 validation days remaining, an incident is raised. If a certificate expires, the user is required to import a new valid certificate using RACLI to replace the old one.

Configuring TLS Data Security on the Recovery Appliance

This section provides the steps for configuring TLS Data Security on the Recovery Appliance.

RACLI commands configure the TLS (Transport Layer Security). The Recovery Appliance these TLS modes:

only: is https encryption alone.
enable: is http/https dual mode.
disable: is http, the default, without encryption.

The port numbers can be customized. The default ports for encryption are:

TCPS: 2484
HTTPS: 8002
REPL_TCPS: 2485

The default ports for non-encrypted operation are:

TCP: 1521
HTTP: 8001
REPL_TCP: 1522

Verify with "racli list certificate" that the certificates are in the raa_certs database table.

# racli list certificate

Created log /opt/oracle.RecoveryAppliance/log/racli_list_certificate.20230329.1146.log
Wed Mar 29 11:46:49 2023: Start: List Certificate
Serial: 9A15CB4B76BBC52D
    Expire Time:      2024-03-28
    Certificate Type: trusted_cert
 
Serial: 95B9181340F644F0
    Expire Time:      2024-03-28
    Certificate Type: signed_cert
 
Wed Mar 29 11:46:49 2023: End: List Certificate

To update the TLS mode on the Recovery Appliance that employs the certificates, issue a command similar to:
```
racli alter network --service=ra_server --encrypt=enable
```
Note:

A complete Recovery Appliance outage is expected, because the whole CRS stack is restarted as part of the procedure. Additional steps are required because of this outage: pause replication, pause any backup scheduler, etc.

The general form of the command is:
```
racli alter network --service=ra_server
{ --encrypt=[enable|only|disable] }
[ --tcps_port=<VALUE>|--tcp_port=<VALUE> ]
[ --https_port=<VALUE>|--http_port=<VALUE> ]
[ --repl_tcp_port=<VALUE>|--repl_tcps_port=<VALUE>]
[ --silent ]
```
--service

Indicate the service being modified on the system. Valid value is "ra_server". Cannot be used of --network_type or its arguments.

--network_type

Indicate network type on the system. Cannot be used of --service or its arguments.

--encrypt

Specifies TLS encryption status on the system: "only" means HTTPS encryption; "enable" means dual HTTPS and HTTP; and "disable" means HTTP.

--http_port

Specifies HTTP port number to use. Default port is 8001.

--https_port

Specifies HTTPS port number to use. Default port is 8005.

--tcp_port

Specifies the TCP port number to use. Default port is 1521.

--tcps_port

Specifies TCPS port number to use. Default port is 2484

--rep_tcps_port

Specifies the replication TCPS port number to use. Default port is 2485.

--rep_tcp_port

Specifies the replication TCP port number to use. Default port is 1522.

--silent

When present

Verify the health of the TLS.

# racli run check --check_name=tls_health

Changing TLS Encryption on the Recovery Appliance

The "racli alter network" command configures TCPS & HTTPS, and TCP & HTTP. It has three encryption modes of operation.

Enable TLS Encryption: This enables dual mode TCP/TCPS and HTTP/HTTPS, and will use default ports unless otherwise specified.

racli alter network 
-–service=ra_server –-encrypt=enable
[ --tcps_port=<VALUE> ]
[ --https_port=<VALUE> ]
[ --repl_tcps_port=<VALUE> ]

Disable TLS Encryption: This enables TCP and HTTP, and will use their default ports unless otherwise specified.

racli alter network 
-–service=ra_server –-encrypt=disable
[ --tcp_port=<VALUE> ]
[ --http_port=<VALUE> ]
[ --repl_tcp_port=<VALUE> ]

Enable Only TLS Encryption: This enables only TCPS and HTTPS. The TCP and HTTP are disabled. Default ports are used unless otherwise specified.
```
racli alter network 
-–service=ra_server –-encrypt=only
[ --tcps_port=<VALUE> ]
[ --https_port=<VALUE> ]
[ --repl_tcps_port=<VALUE> ]
```

Validating TLS Usage

The following commands assist in monitoring the various TLS objects.

racli run check --check_name=tls_health
racli run diagnostics --tag=tls
racli run diagnostics --tag=tls_high

Configuring TLS Data Security on the Client

This section provides the steps required to configure TLS Data Security on the Client (database).

The client requires some modifications to support TLS. The Recovery Appliance can use https encryption alone, in dual mode http/https, or without encryption http, the default.

Configuring Protected Databases to Support TLS

If you want to continue using non-TLS, update the RMAN settings by adding to CONFIGURE CHANNEL DEVICE TYPE "_RA_NO_SSL=TRUE"

CONFIGURE CHANNEL DEVICE TYPE
'SBT_TAPE' PARMS 
'SBT_LIBRARY=<LIB_DIR>/libra.so,
ENV=(_RA_NO_SSL=TRUE,RA_WALLET=location=file:/<WRL>
     credential_alias==<DBNAME>_TCPS,_RA_TRACE_LEVEL=1000)' FORMAT '%U_%d';

An example <LIB_DIR> is /u01/app/oracle/product/19.0.0.0/dbhome_1/lib.

If you want to start using TLS, you need to perform the following steps.

Run verification to see where TLS presently is.

racli run check --check_name=tls_health
racli list certificate

Copy the trusted certificate (example: raCA.pem) from Recovery Appliance host to client side <COPY_DIR>.

Permissions for the certificates should be "oracle:oinstall".
Update wallet, or create new wallet. If existing wallet was created with mkstore, create a new wallet using orapki that can accept certificates. For example:
```
orapki wallet create --wallet <WRL>
```
```
orapki wallet create --wallet $ORACLE_HOME/dbs/Sydney
```

Import the trusted certificate into the wallet from above.

orapki wallet add --wallet <WRL> --trusted_cert --cert <COPY_DIR>/<NAME3>.pem

orapki wallet add --wallet $ORACLE_HOME/dbs/sydney --trusted_cert --cert $ORACLE_HOME/dbs/sydney/raCA.pem

On a Recovery Appliance host, find the TCPS alias (example: zdlra_tcps) in $ORACLE_HOME/network/admin/tnsnames.ora file, copy it to tnsnames.ora file on client side.

Update wallet to --auto_login.

orapki wallet create --wallet <WRL> --auto_login

orapki wallet create --wallet $ORACLE_HOME/dbs/sydney --auto_login

Update the wallet with VPC user credentials for the new TCPS alias.

mkstore --wrl <WRL> --createCredential <DBNAME>_tcps <VPCUSER> <VPCPW>

mkstore --wrl $ORACLE_HOME/dbs/sydney --createCredential zdlra7_tcps <VPCUSER> <VPCPW>

Add the wallet path <WRL> to sqlnet.ora file.
Validate on the client side using tnsping.
```
tnsping <DBNAME>_TCPS
```
```
tnsping ZDLRA7_TCPS
```
Connect to RMAN and update “CONFIGURE CHANNEL DEVICE” adding wallet information.
```
rman target / catalog <VPCUSER>/<VPCPW>@<DBNAME>_TCPS
```
```
rman target / catalog <VPCUSER>/<VPCPW>@zdlra7_tcps
```
Alternatively, after you type in the following command, you are prompted for the VPC user name and password.
```
rman target / catalog @<DBNAME>_TCPS
```
```
rman target / catalog @zdlra7_tcps
```

Validate the whole process by attempting to create a backup.

run   
{     allocate CHANNEL c1 DEVICE TYPE 'SBT_TAPE' PARMS "SBT_LIBRARY=<LIB_DIR>
        libra.so,ENV=(RA_WALLET='location=file:/<WRL>
        credential_alias=<DBNAME>_TCPS,RA_FORMAT=TRUE)";   

        backup incremental level 1 filesperset 1 section size 64g database plus archivelog not backed up filesperset 32;
        }

Validating TLS Usage

The following commands assist in monitoring the various TLS objects.

racli run check --check_name=tls_health
racli run diagnostics --tag=tls
racli run diagnostics --tag=tls_high

Trouble-shooting TLS

This section provides some information about common TLS configuration errors.

If TLS isn't working, the following are items that can cause issues.

The certificates are not correct.
- Missing DNS information
- Wrong format.
- Certificate was not signed.
The port is not open or available.
The trusted certificate was not copied to the client side.
The client side wallet is mkstore type that doesn't support certificate import.
The RMAN settings were not updated after tnsname was updated and the certificate imported.
The upstream Recovery Appliance wallets do not have certificates from the downstream Recovery Appliance.
The upstream Recovery Appliance tnsnames.ora file does not have downstream Recovery Appliance new TCPS information.

Troubleshoot DNS

To obtain the DNS information, issues the following RACLI command on the Recovery Appliance.

racli list san

To check if the certificate has the the DNS information, make sure that the trusted certificate has no information and that the signed certificate has DNS information.

openssl x509 -text -noout -in cert.pem | grep  -i 'dns’
openssl x509 -text -noout -in <>.p12 | grep  -i 'dns'

Troubleshoot Certificates

Get certificate details from metadata table including type.

racli list certificate

Get certificate details from wallet.

orapki wallet display --wallet /raacfs/raadmin/config/ra_wallet/wallet/ --complete

User Certificates:
- Subject: CN=<>-scan.subnet1.<>.oraclevcn.com
  
  Note:
  The scan address is not the same as the trusted certificate
- Issuer: CN=Oracle DB Recovery Service Authority
Trusted Certificates:
- Subject: CN=Oracle DB Recovery Service Authority
- Issuer: CN=Oracle DB Recovery Service Authority

Tips

If the backup is not working and returns errors, check the certificates, wallet, and RMAN configuration settings.
If the backup is hanging, check the Recovery Appliance's port. Make sure TCPS port (default 8005) is open on the Recovery Appliance.

Check scan listener and the listener status of the Recovery Appliance.