OCCM Alerts

Alerts Levels / Severity Types	Definition
Critical	Indicates a severe issue that poses a significant risk to safety, security, or operational integrity. It requires immediate response to address the situation and prevent serious consequences. Raised for conditions may affect the service of OCCM.
Major	Indicates a more significant issue that has an impact on operations or poses a moderate risk. It requires prompt attention and action to mitigate potential escalation. Raised for conditions may affect the service of OCCM.
Minor	Indicates a situation that is low in severity and does not pose an immediate risk to safety, security, or operations. It requires attention but does not demand urgent action. Raised for conditions may affect the service of OCCM.
Info or Warn (Informational)	Provides general information or updates that are not related to immediate risks or actions. These alerts are for awareness and do not typically require any specific response. WARN and INFO alerts may not impact the service of OCCM.

7.1 OccmCmpIdentityCertExpirationMinor

Table 7-2 OccmCmpIdentityCertExpirationMinor

Field	Details
Description	CMP Identity (OCCM) Certificate Expiry Alert. The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} will expire within 90 days.
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} will expire soon within 90 days'
Severity	Minor
Condition	The CMP Identity (OCCM) certificate will expire within 90 days.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7001
Metric Used	occm_cmp_identity_cert_expiration_seconds
Recommended Actions	Cause: When Certificate is going to expire within 90 days. Diagonstic Information: Check certificate configuration for renew before days. Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Recovery:The alert is cleared when the certificate is renewed. If alert is not cleared: Check for remaining validity of the cert. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.2 OccmCmpIdentityCertExpirationMajor

Table 7-3 OccmCmpIdentityCertExpirationMajor

Field	Details
Description	CMP Identity (OCCM) Certificate Expiry Alert. The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} will expire within 30 days.
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} will expire soon within 30 days'
Severity	Major
Condition	The CMP Identity (OCCM) certificate will expire within 30 days.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7001
Metric Used	occm_cmp_identity_cert_expiration_seconds
Recommended Actions	Cause: When Certificate is going to expire within 30 days. Diagnostic Information: Check certificate configuration for renew before days. Recovery:The alert is cleared when the certificate is renewed or when the Certificate expiry days crosses the Critical threshold. If alert is not cleared: Check for remaining validity of the cert. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.3 OccmCmpIdentityCertExpirationCritical

Table 7-4 OccmCmpIdentityCertExpirationCritical

Field	Details
Description	CMP Identity (OCCM) Certificate Expiry Alert. The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} will expire within one week.
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} will expire soon within 1 week'
Severity	Critical
Condition	The CMP Identity (OCCM) certificate will expire within one week.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7001
Metric Used	occm_cmp_identity_cert_expiration_seconds
Recommended Actions	Cause: The Certificate is going to expire within 1 week. Diagnostic Information: Check certificate configuration for renew before days. Recovery:The alert is cleared when the certificate is renewed or when the Certificate expiry days crosses the Critical threshold. If alert is not cleared: Check for remaining validity of the cert. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.4 OccmCmpIdentityCertExpired

Table 7-5 OccmCmpIdentityCertExpired

Field	Details
Description	Alert is raised when the certificate expires and then recreation will be triggered. If the certificate recreation is successful then alert will be cleared automatically or the operator has to clear the alert manually. The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} is expired.
Summary	'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} is expired'
Severity	Critical
Condition	The CMP Identity (OCCM) certificate has expired.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7002
Metric Used	occm_cmp_identity_cert_expiration_seconds
Recommended Actions	Cause: The Certificate is going to expire. Diagnostic Information: Check certificate configuration for renew before days. Check logs to identify the root cause, one of the possible cause is CA connection failure. in this case operator needs to manually configure the CMP Identity certificate. Manual recreate can be triggered once CA connection is resume. Recovery: The alert is cleared when the certificate is recreated. Check logs to identify the root cause, one of the possible cause is CA connection failure. in this case operator needs to manually configure the CMP Identity certificate. Get the k8s secret name corresponding to OCCM key/certificate location from the mapped issuer. This information will be present under CMP client authentication options for Other Cert section of the issuer. Create CMP Identity (OCCM) Certificate manually and Update the secret. Manual recreate can be triggered once CA connection is resume. For more information, refer Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide for troubleshooting scenarios. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue.

7.5 OccmEndEntityCertExpirationMinor

Table 7-6 OccmEndEntityCertExpirationMinor

Field	Details
Description	End Entity (NF) Certificate Expiry Alert. The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} will expire within 90 days.
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} will expire soon within 90 days'
Severity	Minor
Condition	The End Entity (NF) certificate will expire within 90 days.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7003
Metric Used	occm_end_entity_cert_expiration_seconds
Recommended Actions	Cause: The Certificate is going to expire within 90 days. Diagnostic Information: Check certificate configuration for renew before days. Recovery: The alert is cleared when the certificate is renewed so that the Certificate expiry days falls below the Minor threshold or when the Certificate expiry days crosses the Major threshold If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.6 OccmEndEntityCertExpirationMajor

Table 7-7 OccmEndEntityCertExpirationMajor

Field	Details
Description	End Entity (NF) Certificate Expiry Alert. The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} will expire within 30 days.
Summary	'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} will expire soon within 30 days.
Severity	Major
Condition	End Entity (NF) certificate will expire soon within 30 days.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7003
Metric Used	occm_end_entity_cert_expiration_seconds
Recommended Actions	Cause: When Certificate is going to expire within 30 days. Diagnostic Information: Check certificate configuration for renew before days. Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Recovery: The alert is cleared when the certificate is renewed or when the Certificate expiry days crosses the Critical threshold. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.7 OccmEndEntityCertExpirationCritical

Table 7-8 OccmEndEntityCertExpirationCritical

Field	Details
Description	End Entity (NF) Certificate Expiry Alert. The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} will expire within one week.
Summary	'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} will expire soon within 1 week'
Severity	Critical
Condition	End Entity (NF) certificate will expire soon within one week.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7003
Metric Used	occm_end_entity_cert_expiration_seconds
Recommended Actions	Cause: The Certificate is going to expire within 1 week. Diagnostic Information: Check certificate configuration for renew before days. Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Recovery:The alert is cleared when the certificate is renewed or when the Certificate expiry days crosses the Critical threshold. If alert is not cleared: Check for remaining validity of the cert. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.8 OccmEndEntityCertExpired

Table 7-9 OccmEndEntityCertExpired

Field	Details
Description	Alert is raised when the certificate expires and then recreation will be triggered. If the certificate recreation is successful then alert will be cleared automatically or the operator has to clear the alert manually. The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} is expired'
Summary	'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} for {{$labels.certPurpose}} is expired'
Severity	Critical
Condition	End Entity (NF) certificate has expired.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7009
Metric Used	occm_end_entity_cert_expiration_seconds
Recommended Actions	Cause: The Certificate is going to expire. Diagnostic Information: Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Check logs to identify the root cause, one of the possible cause is CA connection failure. Recovery: The alert is cleared when the certificate is recreated. Perform recreate operation when is CA is accessible. Alert will be cleared once recreation is successful. If CA is still down then create End-Entity(NF) certificate manually and update details in secret which will be automatically monitored by OCCM. Manual recreate can be triggered once CA connection is resume. Manual recreate can be triggered once CA connection is resume. For more information, refer Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide for troubleshooting scenarios. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue.

7.9 OccmServiceDown

Table 7-10 OccmServiceDown

Field	Details
Description	OCCM Service Down Alert New certificates will not be created, and existing ones can not be renewed until OCCM is back
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: OCCM service is down
Severity	Critical
Condition	The pods of the occm service is unavailable.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7004
Metric Used	up Note:This is a prometheus metric used for instance availability monitoring. If this metric is not available, use the similar metric as exposed by the monitoring system.
Recommended Actions	Cause: The Certificate is going to expire within 1 week. Diagnostic Information: Check the orchestration logs of occm service and check for readiness probe failures. Refer to the application logs on Kibana and filter based on occm service names. Check for ERROR WARNING logs related to thread exceptions. Recovery:The alert is cleared when the occm service is available. If alert is not cleared: Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue.

7.10 OccmMemoryUsageMinorThreshold

Table 7-11 OccmMemoryUsageMinorThreshold

Field	Details
Description	OCCM Memory Usage Alert OCCM Memory Usage for pod {{ $labels.pod }} has crossed the configured minor threshold (70%) (value={{ $value }}) of its limit.
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: Memory Usage of pod exceeded 70% of its limit.
Severity	Minor
Condition	A pod has reached the configured minor threshold( 70%) of its memory resource limits.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7005
Metric Used	container_memory_usage_bytes, Note : This is a kubernetes metric used for instance availability monitoring. If the metric is not available, use the similar metric as exposed by the monitoring system.
Recommended Actions	Cause: The Certificate is going to expire within 1 week. Diagnostic Information: Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Recovery:The alert gets cleared when the memory utilization falls below the Minor Threshold or crosses the major threshold, in which case OccmMemoryUsageMajorThreshold alert shall be raised. If alert is not cleared: Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.11 OccmMemoryUsageMajorThreshold

Table 7-12 OccmMemoryUsageMajorThreshold

Field	Details
Description	OCCM Memory Usage Alert OCCM Memory Usage for pod {{ $labels.pod }} has crossed the configured major threshold (80%) (value={{ $value }}) of its limit.
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: Memory Usage of pod exceeded 80% of its limit.
Severity	Major
Condition	A pod has reached the configured major threshold( 80%) of its memory resource limits.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7005
Metric Used	container_memory_usage_bytes, Note : This is a kubernetes metric used for instance availability monitoring.If the metric is not available, use the similar metric as exposed by the monitoring system.
Recommended Actions	Cause: The Certificate is going to expire within 1 week. Diagnostic Information: Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Recovery:The alert gets cleared when the memory utilization falls below the Major Threshold or crosses the critical threshold, in which case OccmMemoryUsageMajorThreshold alert shall be raised If alert is not cleared: Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.12 OccmMemoryUsageCriticalThreshold

Table 7-13 OccmMemoryUsageCriticalThreshold

Field	Details
Description	OCCM Memory Usage Alert OCCM Memory Usage for pod {{ $labels.pod }} has crossed the configured critical threshold (90%) (value={{ $value }}) of its limit..
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: Memory Usage of pod exceeded 90% of its limit.
Severity	Critical
Condition	A pod has reached the configured critical threshold ( 90% ) of its memory resource limits
OID	1.3.6.1.4.1.323.5.3.54.1.2.7005
Metric Used	container_memory_usage_bytes, Note : This is a kubernetes metric used for instance availability monitoring.If the metric is not available, use the similar metric as exposed by the monitoring system.
Recommended Actions	Cause: The Certificate is going to expire within 1 week. Diagnostic Information: Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Recovery:The alert gets cleared when the memory utilization falls below the Critical Threshold. If alert is not cleared: Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.13 OccmCPUUsageMinorThreshold

Table 7-14 OccmCPUUsageMinorThreshold

Field	Details
Description	OCCM CPU Usage Alert OCCM Pod {{$labels.pod}} has high CPU usage detected.
Summary	namespace: {{ $labels.namespace}}, podname: {{ $labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: CPU usage is {{ $value \| printf "%.2f" }} which is usage is above 70% (current value is: {{ $value }})
Severity	Minor
Condition	CPU usage is above 70%
OID	1.3.6.1.4.1.323.5.3.54.1.2.7006
Metric Used	container_cpu_usage_seconds_total
Recommended Actions	Cause: The CPU usage is above 70%. Diagnostic Information: Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Recovery:The alert gets cleared when the CPU usage falls below the Minor Threshold. If alert is not cleared: Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.14 OccmCMPFailureMinor

Table 7-15 OccmCMPFailureMinor

Field	Details
Description	OCCM CMP Command Execution Failure Alert The certificate {{$labels.certName}} used by {{$labels.nfType}} has failed while executing CMP cmd with {{$labels.statusCode}}.
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} has failed while executing CMP cmd with {{$labels.statusCode}}.
Severity	Minor
Condition	Certificate has failed while executing CMP cmds.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7007
Metric Used	occm_cmp_responses_total
Recommended Actions	Cause: When the rate of certificate failure due to CMP command execution error has crossed the threshold. Diagonstic Information: Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Recovery:The alert is cleared when the rate of certificate failure due to CMP command execution error falls below the Minor threshold or when the error rate crosses the Major threshold, in which case the OccmCMPFailureMajor alert is raised. If alert is not cleared: Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.15 OccmCMPFailureMajor

Table 7-16 OccmCMPFailureMajor

Field	Details
Description	OCCM CMP Command Execution Failure Alert The certificate {{$labels.certName}} used by {{$labels.nfType}} has failed while executing CMP cmd with {{$labels.statusCode}}.
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} has failed while executing CMP cmd with {{$labels.statusCode}}.
Severity	Major
Condition	Certificate has failed while executing CMP cmds
OID	1.3.6.1.4.1.323.5.3.54.1.2.7007
Metric Used	occm_cmp_responses_total
Recommended Actions	Cause: When the rate of certificate failure due to CMP command execution error has crossed the threshold. Diagnostic Information: Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Recovery:The alert is cleared when the rate of certificate failure due to CMP command execution error falls below the Minor threshold or when the error rate crosses the Major threshold, in which case the OccmCMPFailureMajor alert is raised. If alert is not cleared: Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.16 OccmCMPFailureCritical

Table 7-17 OccmCMPFailureCritical

Field	Details
Description	OCCM CMP Command Execution Failure Alert The certificate {{$labels.certName}} used by {{$labels.nfType}} has failed while executing CMP cmd with {{$labels.statusCode}}.
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} has failed while executing CMP cmd with {{$labels.statusCode}}.
Severity	Critical
Condition	Certificate has failed while executing CMP cmds
OID	1.3.6.1.4.1.323.5.3.54.1.2.7007
Metric Used	occm_cmp_responses_total
Recommended Actions	Cause: When the rate of certificate failure due to CMP command execution error has crossed the threshold. Diagnostic Information: Refer to the application logs on Kibana and filter based on the OCCM service name. Check for ERROR WARNING logs related to thread exceptions. Recovery: The alert is cleared when the rate of certificate failure due to CMP command execution error falls below the Critical threshold. If the alert is not cleared: Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.17 OccmFailureMinor

Table 7-18 OccmFailureMinor

Field	Details
Description	OCCM Internal Failure Alert The certificate {{$labels.certName}} used by {{$labels.nfType}} has failed while creating cert with {{$labels.errorReason}}.
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} has failed while creating cert with {{$labels.errorReason}}.
Severity	Minor
Condition	Certificate has failed while creating
OID	1.3.6.1.4.1.323.5.3.54.1.2.7008
Metric Used	occm_cert_request_status_total
Recommended Actions	Cause: The rate of OCCM errors has crossed the threshold. Diagnostic Information: Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Recovery:The alert is cleared when the rate OCCM error falls below the Minor threshold or when the error rate crosses the Major threshold, in which case the OccmFailureMajor alert is raised. If alert is not cleared: Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.18 OccmFailureMajor

Table 7-19 OccmFailureMajor

Field	Details
Description	OCCM Internal Failure Alert The certificate {{$labels.certName}} used by {{$labels.nfType}} has failed while creating cert with {{$labels.errorReason}}.
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} has failed while creating cert with {{$labels.errorReason}}.
Severity	Major
Condition	Certificate has failed while creating
OID	1.3.6.1.4.1.323.5.3.54.1.2.7008
Metric Used	occm_cert_request_status_total
Recommended Actions	Cause: The rate of OCCM errors has crossed the threshold. Diagnostic Information: Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Recovery:The alert is cleared when the rate OCCM error falls below the Major threshold or when the error rate crosses the Critical threshold, in which case the OccmFailureCritical alert is raised. If alert is not cleared: Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.19 OccmFailureCritical

Table 7-20 OccmFailureCritical

Field	Details
Description	OCCM CMP Command Execution Failure Alert The certificate {{$labels.certName}} used by {{$labels.nfType}} has failed while creating cert with {{$labels.errorReason}}.
Summary	namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The certificate {{$labels.certName}} used by {{$labels.nfType}} has failed while creating cert with {{$labels.errorReason}}.
Severity	critical
Condition	Certificate has failed while creating
OID	1.3.6.1.4.1.323.5.3.54.1.2.7008
Metric Used	occm_cert_request_status_total
Recommended Actions	Cause: The rate of OCCM errors has crossed the threshold. Diagnostic Information: Refer to the application logs on Kibana and filter based on occm service name. Check for ERROR WARNING logs related to thread exceptions. Recovery:The alert is cleared when the rate OCCM error falls below the Major threshold or when the error rate crosses the Critical threshold, in which case the OccmFailureCritical alert is raised. If alert is not cleared: Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue. Note: Note: The threshold is configurable in the `occm_alertingrules_<version>.yaml` file.

7.20 OccmInputSecretModifyMajor

Table 7-21 OccmInputSecretModifyMajor

Field	Details
Description	Input secret is modified by non-OCCM user The Secret {{$labels.secret}} in {{$labels.secretNamespace}} is modified by non-occm user, which is used by {{$labels.name}}.'
Summary	'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The Secret {{$labels.secret}} in {{$labels.secretNamespace}} is modified by non-occm user, which is used by {{$labels.name}} and {{$labels.type}}.'
Severity	Major
Condition	Input secrets are modified by non-OCCM users or by the operator manually.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7010
Metric Used	occm_secret_event_total
Recommended Actions	Cause: The input secret is modified by non-occm user. Diagnostic Information: Check input secrets for any modifications. Recovery:The alert is cleared when the new input secret is created manually. If alert is not cleared: Refer to the alert's label for which namespace and for which secret alert is triggered. Update input secrets with correct data, if require. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue.

7.21 OccmOutputSecretModifyMinor

Table 7-22 OccmOutputSecretModifyMinor

Field	Details
Description	Output secret is modified by non-OCCM user The Secret {{$labels.secret}} in {{$labels.secretNamespace}} is modified by non-occm user, which is used by {{$labels.name}}.'
Summary	'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The Secret {{$labels.secret}} in {{$labels.secretNamespace}} is modified by non-occm user, which is used by {{$labels.name}} and {{$labels.type}}.'
Severity	Minor
Condition	Output secrets are modified by non-OCCM user or by operator manually
OID	1.3.6.1.4.1.323.5.3.54.1.2.7011
Metric Used	occm_secret_event_total
Recommended Actions	Cause: The output secret is modified by non-occm user. Diagnostic Information: Check output secrets for any modifications. Automatic recreation will be triggered if certificate which is modified doesn't match with cert config. The update for validity will be done, if modified certificate validation is successful with cert configuration. No recreation will be triggered in this case. Recovery:The alert is cleared when the new output secret is created by occm. If alert is not cleared: Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue.

7.22 OccmK8sResourceDeleteMajor

Table 7-23 OccmK8sResourceDeleteMajor

Field	Details
Description	Kubernetes resource (secret or namespace) is deleted by non-OCCM user The Kubernetes resource is deleted, which is used in {{$labels.name}} of type {{$labels.type}}. K8s resources, secretNamespace: {{$labels.secretNamespace}} and secret: {{$labels.secret}}'
Summary	{{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }}: The k8s resource is deleted, which is used in {{$labels.name}} of type {{$labels.type}}. K8s resources, namespace: {{$labels.secretNamespace}} and secret: {{$labels.secret}}.'
Severity	Major
Condition	Kubernetes resources (secret or namespace) are deleted by non-OCCM user or by operator manually.
OID	1.3.6.1.4.1.323.5.3.54.1.2.7012
Metric Used	occm_secret_event_total
Recommended Actions	Cause: The k8s resource (secret/namespace) is deleted by non-occm user. Diagnostic Information: Check secret or namespace for any deletion. If secret is deleted, Automatic recreation of cert will be triggered. If namespace is deleted, no auto-recreation will happen. Recovery:The alert is cleared when the new secret/namespace is created again. If alert is not cleared: Try to create the secret/namespace again and check for alert. If namespace is deleted then there might be issue even after recreation of namespace. So if namespaces is deleted, delete cert config and create again. Depending on the failure reason, take the resolution steps from Oracle Communications, Cloud Native Core, Certificate Management Troubleshooting Guide. If this is unexpected, contact My Oracle Support. Ensure to capture the `occm` pod logs and relevant metrics to help Support analyze the issue.

7 OCCM Alerts