4 Troubleshooting OCCM

This section provides information to troubleshoot common errors that occur while installing and upgrading OCCM.

4.1 Configuration Related Issues

This section describes the most common deployment related issues and their resolution steps. It is recommended to perform the resolution steps provided in this guide. If the issue still persists, then contact My Oracle Support.

4.1.1 Issuer and Certificate Related Errors

This section describes the following issuer and certificate related issues and their resolution steps:

Secret Name Format Error

Problem: The format in which the Kubernetes secret is provided is incorrect.

For example:

test_secret : Here, underscore is not allowed

Test-secret: Here, uppercase is not allowed

Solution: You must provide a valid string that is in compliance with kubernetes regex. It must have lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character. For more information, see Oracle Communications Cloud Native Core, Certificate Management Installation, Upgrade and Fault Recovery Guide.

For example:

occm-mac-secret

nrf-tls-secret

Namespace Format Error

Problem: The format in which the Kubernetes namespace is provided in the secret input is incorrect.

For example:

test_ns : Here, underscore is not allowed

Test-ns: Here, uppercase is not allowed

Solution: You must provide a valid string in compliance with Kubernetes regex. It must have lower case alphanumeric characters or '-', and must start and end with an alphanumeric character.

For example:

occm-ns

ocncc-thrust5-02

Secret Key Error

Problem: Secret data can't be found against key(s) and the configured secret key(s) are incorrect. Configured keys are not present in the Kubernetes secret or incorrect key names are provided in the input configuration.

Solution: Revisit the Kubernetes secret and provide the correct keys (filenames) in the configuration.

Input String Error

Problem: The number of characters in the string entered by the user exceeds the character limit.

Solution: The user must enter a string that does not exceed the character limit.

Incomplete TrustStore Secret Error

Problem: OCCM TrustStore secret input has missing fields. This could be because OCCM TrustStore input secret is incomplete. CA certs are missing for certificate validation.

Solution: Verify the OCCM TrustStore input secret and provide a valid one.

Invalid Secret Name

Problem: Secret name already in use as a certificate configuration already points to the same destination secret.

Solution: Provide a unique destination secret name.

Repeated Secret Error

Problem: Secret already exists on the server. In automatic life cycle management of certificate, a fresh secret is created.

Solution: Either provide a unique secret name or set the override secret flag to true. This will enable OCCM to override the existing secret.

Secret Doesn't Exist Error

Problem: Secret doesn't exist with this name. This could be because secret holding the manually created certificate (not via OCCM) doesn't exist on the server. For OCCM to start monitoring it, you must provide the corresponding input secret holding the key or certificate.

Solution: Provide details of the Kubernetes secret holding key or certificate.

Unique Secret Key Error

Problem: Secret key should be unique for same secret names. Same destination secrets may be having same keys in the Certificate configuration. Secret Key (FileNames) should be unique to avoid overriding of data.

For Example: 
"privateKeyK8sSecretOut": {
            "namespace": "occm",
            "name": "test-secret",
            "key": "occm.pem"
    }

"certK8sSecretOut": {
        "namespace": "occm",
        "name": "test-secret",
        "key": "occm.pem"
    }

Solution: Provide unique destination secret (name, namespace, and key) in the configuration.

For Example:
"privateKeyK8sSecretOut": {
            "namespace": "occm",
            "name": "test-secret",
            "key": "nrfkey.pem"
    }

"certK8sSecretOut": {
        "namespace": "occm",
        "name": "test-secret",
        "key": "nrfcert.pem"
    }

CA Bundle Secret Error

Problem: CA Bundle secret doesn't exist. This could be because CA bundle check is skipped if the input configuration is not provided. If provided, it is validated whether the provided secret exists or not.

Solution: Either skip providing input CA bundle secret details or provide a valid secret.

Invalid MAC Secret Error

Problem: Invalid MAC secret has been passed because the MAC secret provided in input MAC secret is not in valid format.

For example:

12345: Here, the secret doesn't start with the prefixes.

Solution: MAC secret is expected to have following arguments.

pass: password

env: var

file: pathname

fd: number

stdin

Invalid File Format

Problem: The key or certificate files provided don't have a valid file name. This could be because the file doesn't have an extension or a period in the end.

For example:

file: This has a period at the end

abc: This doesn't have an extension

Solution: Provide a valid file name with OCCM supported extensions.

For example:

occmkey.pem

occmcert.pem

Delete Issuer Error

Problem: Issuer can not be deleted as it is in use by certificate(s).

Solution: Delete the mapped certificates first, followed by the corresponding issuer.

Issuer ID Error

Problem: Issuer ID and name do not match because the issuer edit payload doesn't have corresponding issuer ID and name.

Solution: Verify the payload and provide the name corresponding to the issuer ID or vice versa.

Issuer Already Exists Error

Problem: Issuer already exists with given name.

Solution: Provide a unique issuer name.

4.1.2 CMP Related Issues

This section describes the following CMP related issues and their resolution steps:

Server URL Error

Problem: The issuer URL provided in the serverURL field of issuer configuration is not reachable.Could be an incorrect URL (incorrect port etc). This is causing the following errors:

CMP error: error sending server <server IP>

CMP error:transfer error

Error running CMP command

Solution: Provide a valid server URL by editing issuer configuration.

Issuer Configuration Error

Problem: Pre-shared key (MAC secret) configured in 'CMP protection for OCCM certificate' is incorrect. This is causing the following errors:

CMP error: wrong pbm value

CMP error: error validating protection

Solution: Update the issuer config with correct secret

CMP Server Certificate Error

Problem: The server certificate configured in the OCCM trust store configuration is different from that of the sender (CMP or CA server) certificate. This certificate is used for verifying signature-based protection of CMP response messages. This is causing the following errors:

CMP info: received IP

CMP info: actual name in sender DN field =<>

CMP info: does not match expected sender=<DN from server cert configured in OCCM trust store>

Solution: Update the issuer configuration with correct secret

Certificate Path Validation Error

Problem: The certificates configured in OCCM trust store are invalid or incomplete for certificate path validation of the CMP server certificate. This is causing the following errors:

CMP info: received IP

CMP error: no suitable sender cert:for msg sender name name= <CMP server DN>...

CMP error: error validating protection

Solution: Configure OCCM trust store with the corresponding CA server certificate or chain.

4.1.3 TLS Related Issues

This section describes the following TLS related issues and their resolution steps:

Hostname validation failed

Error: Hostname validation failed. The TLS server certificate presented does not have the expected server URL IP. This is causing the following errors:

CMP:apps/cmp.c:2088:CMP info: will contact https://<CA server Alias> CMP DEBUG: Starting new transaction with ID=3B:C4:18:32:75:E5:E5:C2:18:B6:5A:52:E4:AD:D2:93 CMP info: sending IR CMP DEBUG: connecting to CMP server <server IP> using TLS CMP DEBUG: disconnected from CMP server CMP error: certificate verification failed:Certificate verification at depth = 0 error = 64 (IP address mismatch)

CMP:apps/cmp.c:2088:CMP info: will contact https://<CA server Alias> CMP DEBUG: Starting new transaction with ID=<Transaction ID> CMP info: sending IR CMP DEBUG: connecting to CMP server <server IP> using TLS CMP DEBUG: disconnected from CMP server CMP error: certificate verification failed:Certificate verification at depth = 0 error = 64 (IP address mismatch)

Expected IP address = <IP> Failure for: certificate

CMP error: certificate verify failed CMP error: error sending CMP error: transfer error:request sent: IR, expected response: IP

CMP error: certificate verify failed CMP error: error sending CMP error: transfer error:request sent: IR, expected response: IP

Solution: Verifiy the TLS server certificate. One possibilty could be that the certificate has DNS name instead of IP address. In that case pass DNS in the server URL of issuer.

Certificates configured in TLS TrustStore do not provide a valid trust anchor to authenticate server identity

Error: Certificates configured in TLS TrustStore do not provide a valid trust anchor to authenticate server identity. This is causing the following errors:

CMP info: sending IR CMP DEBUG: connecting to CMP server master:8446 using TLS CMP DEBUG: disconnected from CMP server CMP error: certificate verification failed:Certificate verification at depth = 1 error = 19 (self-signed certificate in certificate chain) Failure for: certificate XXXXXX

CMP error: certificate verify failed CMP error: error sending CMP error: transfer error:request sent: IR, expected response: IP

Solution: Configure the TLS TrustStore configuration under issuer with valid trust anchor.

4.2 Miscellaneous Issues

This section describes the following miscellaneous issues and their resolution steps:

Stop infinite certificate request retries

Problem: How to stop infinite certificate request retries.

Solution: Delete the certificate configuration and recreate it by following the procedure mentioned in the Oracle Communications Cloud Native Core, Certificate Management User Guide.

Incorrect certificate expiry details

Problem: The certificate expiry details indicated in Grafana dashboard does not match with validity period of the certificate configured in corresponding Kubernetes secrets.

Solution:
  1. Check if the Kubernetes secret filled against the certificate configuration is same as what is configured in the NF for that particular interface.
  2. Check if the certificates are manually filled in. If yes, initiate certificate recreation. Except for the initial integration with NF certificates, OCCM can only manage certificates created by it. OCCM does not support manual update of the certificates being monitored.

Certificate is not renewed on time

Problem: Certificate is not getting renewed on time.

Solution:
  1. Check if the Kubernetes secret filled against the certificate configuration is same as what is configured in the NF for that particular interface
  2. Check if the certificates are manually filled in. If yes, initiate certificate recreation. Except for the initial integration with NF certificates, OCCM can only manage certificates created by it. OCCM does not support manual update of the certificates being monitored.

Failed certificate creation

Problem: Certificate creation has failed.

Solution: Certificate creation could fail due to various reasons, OCCM metrics, alert and logs can be used to identify the root cause.

Failed Certificate Renewal

Problem: Certificate renewal has failed.

Solution:
  • Check CA connection alerts and metrics.
  • Check if the current certificate being renewed is deleted.
  • Check if the current certificate is already expired in which case OCCM creates a critical alert indicating the same and stop retrying of renewal. Operator needs to initiate certificate recreation.

Critical certificate expiry alert while integrating with NFs

Problem: Critical alert indicating certificate expiry is raised on integrating with NFs.

Solution: Check if the current certificate is already expired in which case OCCM creates a critical alert. Operator needs to initiate certificate recreation. (Holds true for both NF and OCCM certificate)

Certificate(s) not created for integrated NFs

Problem: Certificate(s) is not created for integrated NFs.

Solution:
  1. Check the certificate configuration. Ensure that LCM type Automatic is selected for creation in the certificate configuration. If Manual is selected, OCCM expects that the certificate is already present.
  2. Check if the Kubernetes secret filled against the certificate configuration is same as what is configured in the NF for that particular interface.

Certificate(s) is not created for integrated NF(s)

Problem: Certificate(s) is not created for integrated NF(s)

Solution:
  1. Check the certificate configuration. Ensure that LCM type Automatic is selected for creation in the certificate configuration. If Manual is selected, OCCM expects that the certificate is already present.
  2. Check if the Kubernetes secret filled against the certificate configuration is same as what is configured in the NF for that particular interface

OCCM certificate is expired

Problem: OCCM Certificate gets expired due to any reason.

Solution:

1.Create the OCCM certificate manually using any of the configured issuers in OCCM.

2. Get the Kubernetes secret corresponding to OCCM key/certificate location from the mapped issuer. This information will be present under CMP client authentication options for Other Cert section of the issuer. Update the secret fetched with the manually created certificate from (1).

3. In order to make OCCM aware about the change, delete the existing OCCM cert configuration and create a new one with same configuration and set the CertType as MANUAL.

OCCM will start monitoring the certificate.

4.3 OCCM Error Codes

The following are the types of OCCM error codes and their descriptions:

Table 4-1 Kubernetes Secret Error Codes

Error Code Description
ERR_MISSING_K8s_SECRET When the K8s secret is not found for further processing.
ERR_INCORRECT_K8s_SECRET_KEY When the k8s secret doesn't contain the configured key.
ERR_INVALID_K8S_NAMESPACE_FORMAT When the format of k8s namespace doesn't comply with the accepted values. Refer installation guide for regex information.
ERR_INVALID_K8S_SECRET_NAME_FORMAT When the format of k8s secret name doesn't comply with the accepted values. Refer installation guide for regex information.

Table 4-2 Certificate Error Codes

Error Code Description
ERR_OCCM_CERT_NOT_READY NF cert creation fails with ERR_OCCM_CERT_NOT_READY, if OCCM cert is not yet created.
ERR_MISSING_CERT_TO_BE_RENEWED When the cert to be renewed doesn't exist in the K8s secret specified in the configuration.
ERR_RENEW_BEFORE_GREATER_THAN_OR_EQUALS_TO_CERT_ACTUAL_VALIDITY When the renew before period configured ends up being greater than the cert validity. The renewal is not triggered in this case.
ERR_INVALID_X509_CERT When the certificate received from CA is not a valid X.509 certificate.

Table 4-3 Missing mandatory fields in configuration

Error Code Description
ERR_MISSING_MANDATORY_FIELDS_IN_CMP_PROTECTION_SECRET_MAC Either name, namespace, password key or reference key is not provided while configuring MAC secret.
ERR_MISSING_MANDATORY_FIELDS_IN_CMP_PROTECTION_SECRET_SIGNATURE Either name, namespace, key or cert is not provided while configuring Sign secret.
ERR_MISSING_MANDATORY_FIELDS_IN_PRIVATE_KEY_OUTPUT_LOCATION_CONFIG Missing fields in private key secret output location in certificate configuration.
ERR_MISSING_MANDATORY_FIELDS_IN_CERT_OUTPUT_LOCATION_CONFIG Missing fields in certificate secret output location in certificate configuration.
ERR_MISSING_MANDATORY_FIELDS_IN_CERT_CHAIN_OUTPUT_LOCATION_CONFIG Missing fields in certificate chain secret output location in certificate configuration. Either all fields should be provided or none.
ERR_MISSING_MANDATORY_FIELDS_IN_CA_BUNDLE_SECRET Missing fields in CA bundle input secret in certificate configuration. Either all fields should be provided or none.

Table 4-4 CMP Error Codes

Error Code Description
ERR_CMP_COMMAND_FAILED When CMP command execution fails. It can be during key pair generation, CSR creation or CA interaction.
ERR_CMP_COMMAND_TIMEOUT When CMP command execution doesn't complete within the configured time.

Table 4-5 Private Key Error Codes

Error Code Description
ERR_MISSING_KEY_ALGO When key algorithm is not provided for private key generation.
ERR_MISSING_KEY_SIZE When key size is not provided for RSA key.
ERR_MISSING_ECCURVE When EcCurve is not provided for EC key.

Table 4-6 Invalid Input Error Codes

Error Code Description
ERR_INVALID_DN When the format of DN(recipientDN or issuerDN) doesn't comply with the accepted values. Refer installation guide for regex information.
ERR_INVALID_IP When the IP configured in SAN is not valid.
ERR_INVALID_DNS When the format of DNS (configured in SAN) doesn't comply with the accepted values. Refer installation guide for regex information.
ERR_INVALID_URIIDURN When the format of URN (configured in SAN) doesn't comply with the accepted values. Refer installation guide for regex information.
ERR_SYNTAX_ERROR_IN_URI When the URI (configured in server URL and in SAN) has synatx error.

Table 4-7 Miscellaneous Codes

Error Code Description
ERR_MAX_CERT_LIMIT_REACHED When the total number of certs created exceeds the configured limit.
ERR_MAX_ISSUER_LIMIT_REACHED When the total number of issuers created exceeds the configured limit.
ERR_CERT_NOT_FOUND When the certificate doesn't exist for further processing.
ERR_ISSUER_NOT_FOUND When the issuer doesn't exist for further processing.
ERR_ISSUER_IN_USE When the delete or edit is requested for issuer which is referenced by atleast one certificate.

Table 4-8 TLS Codes

Error Code Description
ERR_INVALID_SERVER_URL_SCHEME When TLS is enabled and server URL scheme is other than HTTPS.

OR

When the TLS is disabled but the server URL scheme is HTTPS.
ERR_MISSING_TLS_TRUST_STORE_DATA When TLS is enabled but TLS TrustStore is not provided to validate the server certificate.