8 CNC Console Alerts

This section provides information about CNC Console Alerts.

Note:

For OCI:

The only section applicable for OCI is CNC Console Alerts on OCI.

Note:

  • The user must use updated occncc_agent_alertrules_<version>.yaml file for agent cluster, in case of multi cluster deployment.
  • Use occncc_manager_alertrules_<version>.yaml file for single cluster deployment and in the manager cluster, in case of multi cluster deployment.

Note:

Added the four sample alert files:

  • CNCC alert rules file for CNE without Prometheus Operator: occncc_manager_alertrules_<version>.yaml file for manager only or manager plus agent deployments and occncc_agent_alertrules_<version>.yaml file for agent only deployments.
  • CNCC alert rules file supporting CNE with Prometheus HA Operator: occncc_manager_alerting_rules_promha_<version>.yaml file for deployments in manager cluster and occncc_agent_alertingrules_promha_<version>.yaml file for deployments in agent only cluster.

Table 8-1 Alerts Levels or Severity Types

Alerts Levels / Severity Types Definition
Critical Indicates a severe issue that poses a significant risk to safety, security, or operational integrity. It requires immediate response to address the situation and prevent serious consequences. Raised for conditions may affect the service of OCCM.
Major Indicates a more significant issue that has an impact on operations or poses a moderate risk. It requires prompt attention and action to mitigate potential escalation. Raised for conditions may affect the service of OCCM.
Minor Indicates a situation that is low in severity and does not pose an immediate risk to safety, security, or operations. It requires attention but does not demand urgent action. Raised for conditions may affect the service of OCCM.
Info or Warn (Informational) Provides general information or updates that are not related to immediate risks or actions. These alerts are for awareness and do not typically require any specific response. WARN and INFO alerts may not impact the service of OCCM.

8.1 CNC Console IAM Alerts

This section provides information about CNC Console IAM Alerts.

8.1.1 CnccIamTotalIngressTrafficRateAboveMinorThreshold

Table 8-2 CnccIamTotalIngressTrafficRateAboveMinorThreshold

Trigger Condition

The total CNCC IAM Ingress Message rate has crossed the configured minor threshold of 700 TPS.

Default value of this alert trigger point in occncc_alertrules_<version>.yamlis when CNCC IAM Ingress Rate crosses 70 % of 1000 (Maximum ingress request rate)

Severity Minor
Alert details provided

Description: CNCC IAM Ingress traffic Rate is above the configured minor threshold i.e. 700 requests per second (current value is: {{ $value }})

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 70 Percent of Max requests per second(1000)'

For CNE without Prometheus Operator :

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 70 Percent of Max requests per second(1000)'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7001
Metric Used oc_ingressgateway_http_requests_total
Resolution

The alert is cleared either when the total Ingress Traffic rate falls below the Minor threshold or when the total traffic rate crosses the Major threshold, in which case the CnccIamTotalIngressTrafficRateAboveMajorThreshold alert is raised.

Note: The threshold is configurable in the occncc_alertrules_<version>.yaml file.

Steps:

  1. Reassess why the CNCC IAM is receiving additional traffic.
  2. If this is unexpected, contact My Oracle Support.

8.1.2 CnccIamTotalIngressTrafficRateAboveMajorThreshold

Table 8-3 CnccIamTotalIngressTrafficRateAboveMajorThreshold

Trigger Condition

The total CNCC IAM Ingress Message rate has crossed the configured major threshold of 800 TPS.

Default value of this alert trigger point ino ccncc_alertrules_<version>.yaml is when CNCC IAM Ingress Rate crosses 80 % of 1000 (Maximum ingress request rate)

Severity Major
Alert details provided

Description: 'CNCC IAM Ingress traffic Rate is above the configured major threshold i.e. 800 requests per second (current value is: {{ $value }})'

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 80 Percent of Max requests per second(1000)'

For CNE without Prometheus Operator :

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 80 Percent of Max requests per second(1000)'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7001
Metric Used oc_ingressgateway_http_requests_total
Resolution

The alert is cleared when the total Ingress Traffic rate falls below the Major threshold or when the total traffic rate crosses the Critical threshold, in which case the CnccIamTotalIngressTrafficRateAboveCriticalThreshold alert is raised.

Note: The threshold is configurable in the occncc_alertrules_<version>.yaml file.

Steps:

  1. Reassess why the CNCC IAM is receiving additional traffic.
  2. If this is unexpected, contact My Oracle Support.

8.1.3 CnccIamTotalIngressTrafficRateAboveCriticalThreshold

Table 8-4 CnccIamTotalIngressTrafficRateAboveCriticalThreshold

Trigger Condition

The total CNCC IAM Ingress Message rate has crossed the configured critical threshold of 900TPS. Default value of this alert trigger point in occncc_alertrules_<version>.yaml is when CNCC IAM Ingress Rate crosses 90 % of 1000 (Maximum ingress request rate)

Severity Critical
Alert details provided

Description:CNCC IAM Ingress traffic Rate is above the configured critical threshold, that is, 900 requests per second (current value is: {{ $value }})

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 90 Percent of Max requests per second(1000)'

For CNE without Prometheus Operator:

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 90 Percent of Max requests per second(1000)'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7001
Metric Used oc_ingressgateway_http_requests_total
Resolution

The alert is cleared when the Ingress Traffic rate falls below the Critical threshold.

Note: The threshold is configurable in the occncc_alertrules_<version>.yaml file.

Steps:

  1. Reassess why the CNCC IAM is receiving additional traffic.
  2. If this is unexpected, contact My Oracle Support.

8.1.4 CnccIamMemoryUsageCrossedMinorThreshold

Table 8-5 CnccIamMemoryUsageCrossedMinorThreshold

Trigger Condition A pod has reached the configured minor threshold( 70%) of its memory resource limits.
Severity Minor
Alert details provided

Description: 'CNCC IAM Memory Usage for pod {{ $labels.pod }} has crossed the configured minor threshold (70%) (value={{ $value }}) of its limit.'

Summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} : Memory Usage of pod exceeded 70% of its limit.'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7002
Metric Used

container_memory_usage_bytes,

kube_pod_container_resource_limits

Note:This is a Kubernetes metric used for instance availability monitoring. If the metric is not available, use a similar metric as exposed by the monitoring system.

Resolution

The alert gets cleared when the memory utilization falls below the Minor Threshold or crosses the major threshold, in which case CnccIamMemoryUsageCrossedMajorThreshold alert is raised.

Note: The threshold is configurable in the occncc_alertrules_<version>.yaml file.

If guidance is required, contact My Oracle Support.

8.1.5 CnccIamMemoryUsageCrossedMajorThreshold

Table 8-6 CnccIamMemoryUsageCrossedMajorThreshold

Trigger Condition A pod has reached the configured major threshold( 80%) of its memory resource limits.
Severity Major
Alert details provided

Description: 'CNCC IAM Memory Usage for pod {{ $labels.pod }} has crossed the configured major threshold (80%) (value = {{ $value }}) of its limit.'

Summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} : Memory Usage of pod exceeded 80% of its limit.'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7002
Metric Used

container_memory_usage_bytes,

kube_pod_container_resource_limits

Note: This is a Kubernetes metric used for instance availability monitoring.If the metric is not available, use a similar metric as exposed by the monitoring system.

Resolution The alert gets cleared when the memory utilization falls below the Major Threshold or crosses the critical threshold, in which case CnccIamMemoryUsageCrossedCriticalThreshold alert shall be raised.

Note: The threshold is configurable in the occncc_alertrules_<version>.yaml file.

If guidance is required, contact My Oracle Support.

8.1.6 CnccIamMemoryUsageCrossedCriticalThreshold

Table 8-7 CnccIamMemoryUsageCrossedCriticalThreshold

Trigger Condition A pod has reached the configured critical threshold ( 90% ) of its memory resource limits
Severity Critical
Alert details provided

Description: 'CNCC IAM Memory Usage for pod {{ $labels.pod }} has crossed the configured critical threshold (90%) (value = {{ $value }}) of its limit.'

Summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} : Memory Usage of pod exceeded 90% of its limit.'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7002
Metric Used

container_memory_usage_bytes,

kube_pod_container_resource_limits

Note: This is a Kubernetes metric used for instance availability monitoring. If the metric is not available, use a similar metric as exposed by the monitoring system.

Resolution The alert gets cleared when the memory utilization falls below the Critical Threshold.

Note : The threshold is configurable in the occncc_alertrules_<version>.yaml file.

If guidance is required, contact My Oracle Support.

8.1.7 CnccIamTransactionErrorRateAbove0.1Percent

Table 8-8 CnccIamTransactionErrorRateAbove0.1Percent

Trigger Condition The number of failed transactions is above 0.1 percent of the total transactions.
Severity Warning
Alert details provided Description: 'CNCC IAM transaction Error rate is above 0.1 Percent of Total Transactions (current value is {{ $value }})'

Summary: 'CNCC IAM transaction Error Rate detected above 0.1 Percent of Total Transactions'
OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7003
Metric Used oc_ingressgateway_http_responses_total
Resolution The alert is cleared when the number of failed transactions is below 0.1% of the total transactions or when the number of failed transactions crosses the 1% threshold in which case the CnccIamTransactionErrorRateAbove1Percent is raised.

Steps:
  1. Check the Service specific metrics to understand the specific service request errors.
  2. If guidance is required, contact My Oracle Support.

8.1.8 CnccIamTransactionErrorRateAbove1Percent

Table 8-9 CnccIamTransactionErrorRateAbove1Percent

Trigger Condition The number of failed transactions is above 1 percent of the total transactions.
Severity Warning
Alert details provided Description: 'CNCC IAM transaction Error rate is above 1 Percent of Total Transactions (current value is {{ $value }})'

Summary: 'CNCC IAM transaction Error Rate detected above 1 Percent of Total Transactions'
OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7003
Metric Used oc_ingressgateway_http_responses_total
Resolution The alert is cleared when the number of failed transactions is below 1% of the total transactions or when the number of failed transactions crosses the 10% threshold in which case the CnccIamTransactionErrorRateAbove10Percent is raised.

Steps:

  1. Check the Service specific metrics to understand the specific service request errors.
  2. If guidance is required, contact My Oracle Support.

8.1.9 CnccIamTransactionErrorRateAbove10Percent

Table 8-10 CnccIamTransactionErrorRateAbove10Percent

Trigger Condition The number of failed transactions is above 10 percent of the total transactions.
Severity Minor
Alert details provided

Description: CNCC IAM transaction Error rate is above 10 Percent of Total Transactions (current value is {{ $value }})'

Summary: 'CNCC IAM transaction Error rate is above 10 Percent of Total Transactions (current value is {{ $value }})'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7003
Metric Used oc_ingressgateway_http_responses_total
Resolution The alert is cleared when the number of failed transactions is below 10% of the total transactions or when the number of failed transactions crosses the 25% threshold in which case the CnccIamTransactionErrorRateAbove25Percent is raised.

Steps:

  1. Check the Service specific metrics to understand the specific service request errors.
  2. If guidance is required, contact My Oracle Support.

8.1.10 CnccIamTransactionErrorRateAbove25Percent

Table 8-11 CnccIamTransactionErrorRateAbove25Percent

Trigger Condition

The number of failed transactions is above 25 percent of the total transactions.
Severity Major
Alert details provided

Description: 'CNCC IAM transaction Error Rate detected above 25 Percent of Total Transactions (current value is {{ $value }})'

Summary: 'CNCC IAM transaction Error Rate detected above 25 Percent of Total Transactions'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7003
Metric Used oc_ingressgateway_http_responses_total
Resolution

TThe alert is cleared when the number of failed transactions are below 25% of the total transactions or when the number of failed transactions cross the 50% threshold in which case the CnccIamTransactionErrorRateAbove50Percent is raised.

Steps:

  1. Check the Service specific metrics to understand the specific service request errors.
  2. If guidance is required, contact My Oracle Support.

8.1.11 CnccIamTransactionErrorRateAbove50Percent

Table 8-12 CnccIamTransactionErrorRateAbove50Percent

Trigger Condition

The number of failed transactions is above 50 percent of the total transactions.
Severity Critical
Alert details provided

Description: The number of failed transactions is above 50 percent of the total transactions.

Summary: 'CNCC IAM transaction Error Rate detected above 50 Percent of Total Transactions'.

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7003
Metric Used oc_ingressgateway_http_responses_total
Resolution

The alert is cleared when the number of failed transactions is below 50 percent of the total transactions.

The threshold is configurable in the occncc_alertrules_<version>.yaml file.

Steps:

  1. Check the Service specific metrics to understand the specific service request errors.
  2. If guidance is required, contact My Oracle Support.

8.1.12 CnccIamIngressGatewayServiceDown

Table 8-13 CnccIamIngressGatewayServiceDown

Trigger Condition

The pods of the CNCC IAM Ingress Gateway microservice is available.
Severity Critical
Alert details provided

Description:'CNCC IAM Ingress-Gateway service InstanceIdentifier=~".*cncc-iam_ingressgateway" is down'

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} : Ingress-gateway service down'

For CNE without Prometheus Operator:

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} : Ingress-gateway service down'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7004
Metric Used

'up'

Note: This is a Prometheus metric used for instance availability monitoring. If this metric is not available, use a similar metric as exposed by the monitoring system.

Resolution

The alert is cleared when the cncc-iam_ingressgateway service is available.

Steps:

  1. Check the orchestration logs of cncc-iam_ingressgateway service and check for liveness or readiness probe failures.
  2. Refer to the application logs on Kibana and filter based on cncc-iam_ingressgateway service names. Check for ERROR WARNING logs related to thread exceptions.
  3. Depending on the failure reason, take the resolution steps.
  4. In case the issue persists, contact My Oracle Support.

8.1.13 CnccIamFailedLogin

Table 8-14 CnccIamFailedLogin

Trigger Condition

The count of failed login attempts in CNCC-IAM by a user goes above '3'
Severity Warning
Alert details provided

Description:'{{ $value }} failed Login attempts have been detected in CNCC IAM for user {{$labels.UserName}}, the configured threshold value is 3 failed login attempts for every 5 min'

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: failed login attempts are more than the configured threshold value'

For CNE without Prometheus Operator:

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: failed login attempts are more than the configured threshold value'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7005
Metric Used oc_ingressgateway_http_responses_total
Resolution

The alert gets cleared when the total failed login attempts for a particular user goes below the threshold value (default value is '3') in the last 5 min (default value is 5 m).

Note: The threshold and time is configurable in the alerts.yaml file.

If guidance is required, contact My Oracle Support.

8.1.14 AdminUserCreation

Table 8-15 AdminUserCreation

Trigger Condition

If a new admin account is created in the last 5 min

Severity Warning
Alert details provided

For CNE with Prometheus HA Operator:

Description: '{{ $value }} admin users have been created by {{$labels.UserName}} '

summary: 'namespace: {{$labels.namespace}}

summary: {{$labels.pod}}, user: {{$labels.UserName}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Admin users have been created '

For CNE without Prometheus Operator:

Description: '{{ $value }} admin users have been created by {{$labels.UserName}} '

summary: 'namespace: {{$labels.kubernetes_namespace}}

summary: {{$labels.kubernetes_pod_name}}, user: {{$labels.UserName}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Admin users have been created '

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7006
Metric Used oc_ingressgateway_http_requests_total
Resolution

The alert gets cleared when the total failed login attempts for a particular user go below the threshold value (default value is '3') in the last 5 min (default value is 5 m)

Note: The threshold and time is configurable in the occncc_alertrules_<version>.yaml file.

Login to admin GUI and review the user created.

If guidance is required, contact My Oracle Support.

8.1.15 CnccIamAccessTokenFailure

Table 8-16 CnccIamAccessTokenFailure

Trigger Condition

If the count of failed token for CNCC-IAM goes above configured value of '3'
Severity Warning
Alert details provided

Description: 'CNCC Iam Access Token Failure count is above the configured value i.e. 3 for every 5 min. Failed access token request count per second is (current value is: {{ $value }})'

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Access Token Failure count is above the configured threshold value'

For CNE without Prometheus Operator:

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Access Token Failure count is above the configured threshold value'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.7007
Metric Used oc_ingressgateway_http_responses_total
Resolution

The alert gets cleared when the total failed tokens for a particular user go below the threshold value (default value is '3') in the last 5 min (default value is 5 m)

Note: The threshold and time is configurable in the occncc_alertrules_<version>.yaml file.

If guidance is required, contact My Oracle Support.

8.2 CNC Console Core Alerts

This section provides the information about CNC Console Core Alerts.

8.2.1 CNC Console Alert Configuration

This section describes how to configure alert rules for the CNC Consolo in both Prometheus and OCI environments. It provides guidance on setting up measurement-based alert rules, where the alerting system evaluates metrics reported by CNC Console microservices against specified rule conditions to generate alerts as needed.

Prometheus Alert Configuration

In a Prometheus environment, CNC Console alert rules are configured based on metrics reported by CNC Console components. The alerting workflow monitors these metrics and issues notifications when the defined conditions are met.

For more information about configuring CNC Console alerts in Prometheus, see the “CNC Console Alert configuration in Prometheus” section in Oracle Communications Cloud Native Configuration Console Installation, Upgrade, and Fault Recovery Guide.

OCI Alert Configuration

In OCI environments, CNC Console alert rules are defined using Metric Query Language (MQL). Configuring these alerts requires creating a dedicated CNC Console alert file for use with the OCI observability platform.

For more information about configuring CNC Console alerts in OCI, see the “Configure CNC Console Alerts on OCI” section in Oracle Communications Cloud Native Configuration Console Installation, Upgrade, and Fault Recovery Guide.

8.2.2 CnccCoreTotalIngressTrafficRateAboveMinorThreshold

Table 8-17 CnccCoreTotalIngressTrafficRateAboveMinorThreshold

Trigger Condition

The total CNCC Core Ingress Message rate has crossed the configured minor threshold of 700 TPS.

Default value of this alert trigger point in cncc_alert_rules.yaml is when CNCC Core Ingress Rate crosses 70 % of 1000 (Maximum ingress request rate)
Severity Minor
Alert details provided

Description: 'CNCC Core Ingress traffic Rate is above the configured minor threshold i.e. 700 requests per second (current value is: {{ $value }})'

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 70 Percent of Max requests per second(1000)'

For CNE without Prometheus Operator:

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 70 Percent of Max requests per second(1000)'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8001
Metric Used oc_ingressgateway_http_requests_total
Resolution

The alert is cleared either when the total Ingress traffic rate falls below the minor threshold or when the total traffic rate crosses the major threshold, in which case the CnccCoreTotalIngressTrafficRateAboveMajorThreshold alert is raised.

Note: The threshold is configurable in the occncc_alertrules_<version>.yaml file.

Steps:

  1. Reassess why the CNCC Core is receiving additional traffic.
  2. If this is unexpected, contact My Oracle Support.

8.2.3 CnccCoreTotalIngressTrafficRateAboveMajorThreshold

Table 8-18 CnccCoreTotalIngressTrafficRateAboveMajorThreshold

Trigger Condition

The total CNCC Core Ingress Message rate has crossed the configured major threshold of 800 TPS.

Default value of this alert trigger point in cncc_alert_rules.yaml is when CNCC Core Ingress Rate crosses 80 % of 1000 (Maximum ingress request rate)
Severity Major
Alert details provided

Description: 'CNCC Core Ingress traffic Rate is above the configured major threshold i.e. 800 requests per second (current value is: {{ $value }})'

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 80 Percent of Max requests per second(1000)'

For CNE without Prometheus Operator :

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 80 Percent of Max requests per second(1000)'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8001
Metric Used oc_ingressgateway_http_requests_total
Resolution The alert is cleared when the total Ingress Traffic ratefalls below the Major threshold or when the total traffic rate cross the Critical threshold, in which case the CnccCoreTotalIngressTrafficRate Above CriticalThreshold.

Note: The threshold is configurable in the alerts.yaml file.

Steps:

  1. Reassess why the CNCC Core is receiving additional traffic.
  2. If this is unexpected, contact My Oracle Support.

8.2.4 CnccCoreTotalIngressTrafficRateAboveCriticalThreshold

Table 8-19 CnccCoreTotalIngressTrafficRateAboveCriticalThreshold

Trigger Condition

The total CNCC Core Ingress Message rate has crossed the configured critical threshold of 900TPS.

Default value of this alert trigger point in cncc_alert_rules.yaml is when CNCC Core Ingress Rate crosses 90 % of 1000 (Maximum ingress request rate)
Severity Critical
Alert details provided

Description: 'CNCC Core Ingress traffic Rate is above the configured critical threshold i.e. 900 requests per second (current value is: {{ $value }})'

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 90 Percent of Max requests per second(1000)'

For CNE without Prometheus Operator :

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 90 Percent of Max requests per second(1000)'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8001
Metric Used oc_ingressgateway_http_requests_total
Resolution

The alert is cleared when the Ingress Traffic rate falls below the Critical threshold.

Note: The threshold is configurable in the occncc_alertrules_<version>.yaml file.

Steps:

  1. Reassess why the CNCC IAM is receiving additional traffic.
  2. If this is unexpected, contact My Oracle Support.

8.2.5 CnccCoreMemoryUsageCrossedMinorThreshold

Table 8-20 CnccCoreMemoryUsageCrossedMinorThreshold

Trigger Condition

A pod has reached the configured minor threshold( 70%) of its memory resource limits.
Severity Minor
Alert details provided

Description: 'CNCC Core Memory Usage for pod {{ $labels.pod }} has crossed the configured minor threshold (70%) (value={{ $value }}) of its limit.'

Summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} : Memory Usage of pod exceeded 70% of its limit.'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8002
Metric Used

container_memory_usage_bytes

kube_pod_container_resource_limits

Note : This is a kubernetes metric used for instance availability monitoring. If the metric is not available, use the similar metric as exposed by the monitoring system.
Resolution

The alert gets cleared when the memory utilization falls below the Minor Threshold or crosses the major threshold, in which case CnccCoreMemoryUsageCrossedMajorThreshold alert is raised.

Note: The threshold is configurable in the occncc_alertrules_<version>.yaml file.

If guidance is required, contact My Oracle Support.

8.2.6 CnccCoreMemoryUsageCrossedMajorThreshold

Table 8-21 CnccCoreMemoryUsageCrossedMajorThreshold

Trigger Condition

A pod has reached the configured major threshold( 80%) of its memory resource limits.
Severity Major
Alert details provided

Description: 'CNCC Core Memory Usage for pod {{ $labels.pod }} has crossed the configured major threshold (80%) (value = {{ $value }}) of its limit.'

Summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} : Memory Usage of pod exceeded 80% of its limit.'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8002
Metric Used

container_memory_usage_bytes

kube_pod_container_resource_limits

Note : This is a kubernetes metric used for instance availability monitoring. If the metric is not available, use the similar metric as exposed by the monitoring system.
Resolution

The alert gets cleared when the memory utilization falls below the Major Threshold or crosses the critical threshold, in which case CnccCoreMemoryUsageCrossedCriticalThreshold alert is raised

Note: The threshold is configurable in the occncc_alertrules_<version>.yaml file.

If guidance is required, contact My Oracle Support.

8.2.7 CnccCoreMemoryUsageCrossedCriticalThreshold

Table 8-22 CnccCoreMemoryUsageCrossedCriticalThreshold

Trigger Condition

A pod has reached the configured critical threshold ( 90% ) of its memory resource limits
Severity Critical
Alert details provided

Description: 'CNCC Core Memory Usage for pod {{ $labels.pod }} has crossed the configured critical threshold (90%) (value = {{ $value }}) of its limit.'

Summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} : Memory Usage of pod exceeded 90% of its limit.'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8002
Metric Used

container_memory_usage_bytes

kube_pod_container_resource_limits

Note : This is a Kubernetes metric used for instance availability monitoring. If the metric is not available, use the similar metric as exposed by the monitoring system.
Resolution

The alert gets cleared when the memory utilization falls below the Critical Threshold.

Note: The threshold is configurable in the occncc_alertrules_<version>.yaml file.

If guidance is required, contact My Oracle Support.

8.2.8 CnccCoreTransactionErrorRateAbove0.1Percent

Table 8-23 CnccCoreTransactionErrorRateAbove0.1Percent

Trigger Condition

The number of failed transactions is above 0.1 percent of the total transactions..
Severity Warning
Alert details provided

Description:'CNCC Core transaction Error rate is above 0.1 Percent of Total Transactions (current value is {{ $value }})'

Summary:'CNCC Core transaction Error rate is above 0.1 Percent of Total Transactions (current value is {{ $value }})'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8003
Metric Used oc_ingressgateway_http_responses_total
Resolution

The alert is cleared when the number of failed transactions are below 0.1% of the total transactions or when the number of failed transactions cross the 1% threshold in which case the CnccCoreTransactionErrorRateAbove1Percent is raised.

The threshold is configurable in the alerts.yaml file.

Steps:

  1. Check the Service specific metrics to understand the specific service request errors.
  2. If guidance is required, contact My Oracle Support.

8.2.9 CnccCoreTransactionErrorRateAbove1Percent

Table 8-24 CnccCoreTransactionErrorRateAbove1Percent

Trigger Condition The number of failed transactions is above 1 percent of the total transactions.
Severity Warning
Alert details provided

Description: 'CNCC Core transaction Error rate is above 1 Percent of Total Transactions (current value is {{ $value }})'

Summary:'CNCC Core transaction Error Rate detected above 1 Percent of Total Transactions'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8003
Metric Used oc_ingressgateway_http_responses_total
Resolution

The alert is cleared when the number of failed transactions are below 1% of the total transactions or when the number of failed transactions crosses the 10% threshold in which case the CnccCoreTransactionErrorRateAbove10Percent is raised.

Steps:

  1. Check the Service specific metrics to understand the specific service request errors.
  2. If guidance is required,contact My Oracle Support.

8.2.10 CnccCoreTransactionErrorRateAbove10Percent

Table 8-25 CnccCoreTransactionErrorRateAbove10Percent

Trigger Condition The number of failed transactions is above 10 percent of the total transactions.
Severity Minor
Alert details provided

Description: 'CNCC Core transaction Error rate is above 10 Percent of Total Transactions (current value is {{ $value }})'

summary: 'CNCC Core ransaction Error Rate detected above 10 Percent of Total Transactions'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8003
Metric Used oc_ingressgateway_http_responses_total
Resolution

The alert is cleared when the number of failed transactions are below 10% of the total transactions or when the number of failed transactions crosses the 25% threshold in which case the CnccCoreTransactionErrorRateAbove25Percent is raised.

Steps:

  1. Check the Service specific metrics to understand the specific service request errors.
  2. If guidance is required, contact My Oracle Support.

8.2.11 CnccCoreTransactionErrorRateAbove25Percent

Table 8-26 CnccCoreTransactionErrorRateAbove25Percent

Trigger Condition The number of failed transactions is above 25 percent of the total transactions.
Severity Major
Alert details provided

Description: 'CNCC Core transaction Error Rate detected above 25 Percent of Total Transactions (current value is {{ $value }})'

Summary: 'CNCC Core transaction Error Rate detected above 25 Percent of Total Transactions'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8003
Metric Used oc_ingressgateway_http_responses_total
Resolution

The alert is cleared when the number of failed transactions are below 25% of the total transactions or when the number of failed transactions crosses the 50% threshold in which case the CnccCoreTransactionErrorRateAbove50Percent is raised.

Steps:

  1. Check the Service specific metrics to understand the specific service request errors.
  2. If guidance is required, contact My Oracle Support.

8.2.12 CnccCoreTransactionErrorRateAbove50Percent

Table 8-27 CnccCoreTransactionErrorRateAbove50Percent

Trigger Condition The number of failed transactions is above 50 percent of the total transactions.
Severity Critical
Alert details provided

Description: 'CNCC Core transaction Error Rate detected above 50 Percent of Total Transactions (current value is {{ $value }})'

Summary: 'CNCC Core transaction Error Rate detected above 50 Percent of Total Transactions'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8003
Metric Used oc_ingressgateway_http_responses_total
Resolution

The alert is cleared when the number of failed transactions are below 50 percent of the total transactions

Steps:

  1. Check the Service specific metrics to understand the specific service request errors.
  2. If guidance is required, contact My Oracle Support.

8.2.13 CnccCoreIngressGatewayServiceDown

Table 8-28 CnccCoreIngressGatewayServiceDown

Trigger Condition Cncc Core Ingress Gateway service is down
Severity Critical
Alert details provided

Description: 'CNCC Core Ingress-Gateway service InstanceIdentifier=~".*core_ingressgateway" is down'

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} : Ingress-gateway service down'

For CNE without Prometheus Operator:

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} : Ingress-gateway service down'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8004
Metric Used

'up'

Note: This is a Prometheus metric used for instance availability monitoring. If this metric is not available, use a similar metric as exposed by the monitoring system.
Resolution

The alert is cleared when the cncc-core_ingressgateway service is available.

Steps:

  1. Check the orchestration logs of cncc-core_ingressgateway service and check for liveness or readiness probe failures.
  2. Refer the application logs on Kibana and filter based on cncc-core_ingressgateway service names. Check for ERROR WARNING logs related to thread exceptions.
  3. Depending on the failure reason, take the resolution steps.
  4. In case the issue persists, contact My Oracle Support.

8.2.14 CnccCoreFailedLogin

Table 8-29 CnccCoreFailedLogin

Trigger Condition

The count of failed login attempts in CNCC-Core by a user goes above '3'
Severity Warning
Alert details provided

Description:'{{ $value }} failed Login attempts have been detected in CNCC Core for user {{$labels.UserName}}, the configured threshold value is 3 failed login attempts for every 5 min'

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: failed login attempts are more than the configured threshold value'

For CNE without Prometheus Operator :

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: failed login attempts are more than the configured threshold value'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8005
Metric Used oc_ingressgateway_http_responses_total
Resolution

The alert gets cleared when the total failed login attempts for a particular user go below the threshold value (default value is '3') in the last 5 min (default value is 5 m)

Note: The threshold and time is configurable in the alerts.yaml file.

If guidance is required, contact My Oracle Support.

8.2.15 CnccCoreUnauthorizedAccess

Table 8-30 CnccCoreUnauthorizedAccess

Trigger Condition

The count of failed login attempts in CNCC-Core by a user goes above '3'
Severity Warning
Alert details provided

Description:'{{ $value }} Unauthorized Accesses have been detected in CNCC-Core for {{$labels.ResourceType}} for {{$labels.Method}} request. The configured threshold value is 3 for every 5 min'

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.pod}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Unauthorized Access for CNCC-Core are more than threshold value'

For CNE without Prometheus Operator :

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Unauthorized Access for CNCC-Core are more than threshold value'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8006
Metric Used oc_ingressgateway_http_responses_total
Resolution

The alert gets cleared when the total failed login attempts for a particular user go below the threshold value (default value is '3') in the last 5 min (default value is 5 m)

Note: The threshold and time is configurable in the alerts.yaml

If guidance is required, contact My Oracle Support.

8.2.16 CnccCoreAccessTokenFailure

Table 8-31 CnccCoreAccessTokenFailure

Trigger Condition

If the count of failed token for CNCC-Core goes above configured value of '3'
Severity Warning
Alert details provided

Description: 'CNCC Core Access Token Failure count is above the configured value i.e. 3 for every 5 min. Failed access token request count per second is (current value is: {{ $value }})'

For CNE with Prometheus HA Operator:

summary: 'namespace: {{$labels.namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Access Token Failure count is above the configured threshold value'

For CNE without Prometheus Operator :

summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Access Token Failure count is above the configured threshold value'

OID used for SNMP Traps 1.3.6.1.4.1.323.5.3.51.1.2.8007
Metric Used oc_ingressgateway_http_responses_total
Resolution

The alert gets cleared when the total failed tokens for a particular user go below the threshold value (default value is '3') in the last 5 min (default value is 5 m)

Note: The threshold and time is configurable in the alerts.yaml file.

If guidance is required, Contact My Oracle Support.

8.3 CNC Console MIB Files

There are two MIB files which are used to generate the traps. The user need to update these files along with the Alert file in order to fetch the traps in their environment.

occncc_mib_tc_<version>.mib

This file is considered as CNCC top level mib file, where the Objects and their data types are defined.

occncc_mib_<version>.mib

This file fetches the Objects from the top level mib file and based on the Alert notification, these objects can be selected for display.

8.4 CNC Console Alerts on OCI

This section provides information about CNC Console Alerts on OCI:

8.4.1 CnccCoreTotalIngressTrafficRateAboveMinorThreshold

Table 8-32 CnccCoreTotalIngressTrafficRateAboveMinorThreshold

Trigger Condition

The total CNCC Core Ingress Message rate has crossed the configured minor threshold of 700 TPS.

Default value of this alert trigger point in cncc_alert_rules.yaml is when CNCC Core Ingress Rate crosses 70 % of 1000 (Maximum ingress request rate)
Severity minor
Alert details provided CNCC Core Ingress traffic Rate is above the configured minor threshold i.e. 700 requests per second
Metric Used oc_ingressgateway_http_requests_total
Resolution The alert is cleared either when the total Ingress Traffic rate falls below the Minor threshold or when the total traffic rate cross the Major threshold, inwhich case the CnccCoreTotalIngressTrafficRateAboveMajorThresholdalert shall be raised.Note: The threshold is configurable in the alerts.yamlSteps:Reassess why the CNCC Core is receiving additionaltraffic.If this is unexpected, contact My Oracle Support.

8.4.2 CnccCoreTotalIngressTrafficRateAboveMajorThreshold

Table 8-33 CnccCoreTotalIngressTrafficRateAboveMajorThreshold

Trigger Condition

The total CNCC Core Ingress Message rate has crossed the configured major threshold of 800 TPS.

Default value of this alert trigger point in cncc_alert_rules.yaml is when CNCC Core Ingress Rate crosses 80 % of 1000 (Maximum ingress request rate)
Severity major
Alert details provided CNCC Core Ingress traffic Rate is above the configured major threshold i.e. 800 requests per second
Metric Used oc_ingressgateway_http_requests_total
Resolution The alert is cleared when the total Ingress Traffic ratefalls below the Major threshold or when the total traffic rate cross the Critical threshold, in which casethe CnccCoreTotalIngressTrafficRateAboveCriticalThresholdNote: The threshold is configurable in the alerts.yaml alert shall be raised.Steps:Reassess why the CNCC Core is receiving additionaltraffic.If this is unexpected, contact My Oracle Support.

8.4.3 CnccCoreTotalIngressTrafficRateAboveCriticalThreshold

Table 8-34 CnccCoreTotalIngressTrafficRateAboveCriticalThreshold

Trigger Condition

The total CNCC Core Ingress Message rate has crossed the configured critical threshold of 900TPS.

Default value of this alert trigger point in cncc_alert_rules.yaml is when CNCC Core Ingress Rate crosses 90 % of 1000 (Maximum ingress request rate)
Severity critical
Alert details provided CNCC Core Ingress traffic Rate is above the configured critical threshold i.e. 900 requests per second
Metric Used oc_ingressgateway_http_requests_total
Resolution The alert is cleared when the Ingress Traffic rate falls below the Critical threshold.Note: The threshold is configurable in the alerts.yamlSteps:Reassess why the CNCC Core is receiving additional traffic.If this is unexpected, contact My Oracle Support.

8.4.4 CnccCoreMemoryUsageCrossedMinorThreshold

Table 8-35 CnccCoreMemoryUsageCrossedMinorThreshold

Trigger Condition A pod has reached the configured minor threshold( 70%) of its memory resource limits.
Severity minor
Alert details provided CNCC Core Memory Usage for pod has crossed the configured minor threshold (70%) of its limit.
Metric Used

container_memory_usage_bytes

Note : This is a kubernetes metric used for instance availability monitoring. If the metric is not available, use the similar metric as exposed by the monitoring system.
Resolution

The alert gets cleared when the memory utilization falls below the Minor Threshold or crosses the major threshold, in which case CnccCoreMemoryUsageCrossedMajorThreshold alert shall be raised

Note : The threshold is configurable in the alerts.yaml

If guidance required, Contact My Oracle Support.

8.4.5 CnccCoreMemoryUsageCrossedMajorThreshold

Table 8-36 CnccCoreMemoryUsageCrossedMajorThreshold

Trigger Condition A pod has reached the configured major threshold( 80%) of its memory resource limits.
Severity major
Alert details provided CNCC Core Memory Usage for pod has crossed the configured major threshold (80%) of its limit.
Metric Used

container_memory_usage_bytes

Note : This is a kubernetes metric used for instance availability monitoring. If the metric is not available, use the similar metric as exposed by the monitoring system.
Resolution

The alert gets cleared when the memory utilization falls below the Major Threshold or crosses the critical threshold, in which case CnccCoreMemoryUsageCrossedCriticalThreshold alert shall be raised

Note : The threshold is configurable in the alerts.yaml

If guidance required, Contact My Oracle Support.

8.4.6 CnccCoreMemoryUsageCrossedCriticalThreshold

Table 8-37 CnccCoreMemoryUsageCrossedCriticalThreshold

Trigger Condition A pod has reached the configured critical threshold ( 90% ) of its memory resource limits
Severity critical
Alert details provided CNCC Core Memory Usage for pod has crossed the configured critical threshold (90%) of its limit.
Metric Used

container_memory_usage_bytes

Note : This is a kubernetes metric used for instance availability monitoring. If the metric is not available, use the similar metric as exposed by the monitoring system.
Resolution

The alert gets cleared when the memory utilization falls below the Critical Threshold.

Note : The threshold is configurable in the alerts.yaml

If guidance required, Contact My Oracle Support.

8.4.7 CnccCoreTransactionErrorRateAbovePointOnePercent

Table 8-38 CnccCoreTransactionErrorRateAbovePointOnePercent

Trigger Condition The number of failed transactions is above 0.1 percent of the total transactions.
Severity warning
Alert details provided CNCC Core transaction Error rate is above 0.1 Percent of Total Transactions
Metric Used oc_ingressgateway_http_responses_total
Resolution The alert is cleared when the number of failure transactions are below 0.1% of the total transactions or when the number of failure transactions cross the 1% threshold in which case the CnccCoreTransactionErrorRateAbove1Percent shall beraised.Steps:1. Check the Service specific metrics to understand the specific service request errors.2. If guidance required, contact My Oracle Support.

8.4.8 CnccCoreTransactionErrorRateAboveOnePercent

Table 8-39 CnccCoreTransactionErrorRateAboveOnePercent

Trigger Condition The number of failed transactions is above 1 percent of the total transactions.
Severity warning
Alert details provided CNCC Core transaction Error rate is above 1 Percent of Total Transactions
Metric Used oc_ingressgateway_http_responses_total
Resolution The alert is cleared when the number of failure transactions are below 1% of the total transactions or when the number of failure transactions cross the 10% threshold in which case the CnccCoreTransactionErrorRateAbove10Percent shall beraised.Steps:1. Check the Service specific metrics to understand the specific service request errors.2. If guidance required, contact My Oracle Support.

8.4.9 CnccCoreTransactionErrorRateAboveTenPercent

Table 8-40 CnccCoreTransactionErrorRateAboveTenPercent

Trigger Condition The number of failed transactions is above 10 percent of the total transactions.
Severity minor
Alert details provided CNCC Core transaction Error rate is above 10 Percent of Total Transactions
Metric Used oc_ingressgateway_http_responses_total
Resolution The alert is cleared when the number of failure transactions are below 10% of the total transactions or when the number of failure transactions cross the 25% threshold in which case the CnccCoreTransactionErrorRateAbove25Percent shall beraised.Steps:1. Check the Service specific metrics to understand the specific service request errors.2. If guidance required, contact My Oracle Support.

8.4.10 CnccCoreTransactionErrorRateAboveTwentyFivePercent

Table 8-41 CnccCoreTransactionErrorRateAboveTwentyFivePercent

Trigger Condition The number of failed transactions is above 25 percent of the total transactions.
Severity major
Alert details provided CNCC Core transaction Error Rate detected above 25 Percent of Total Transactions
Metric Used oc_ingressgateway_http_responses_total
Resolution The alert is cleared when the number of failure transactions are below 25% of the total transactions or when the number of failure transactions cross the 50% threshold in which case the CnccCoreTransactionErrorRateAbove50Percent shall beraised.Steps:1. Check the Service specific metrics to understand the specific service request errors.2. If guidance required, contact My Oracle Support.

8.4.11 CnccCoreTransactionErrorRateAboveFiftyPercent

Table 8-42 CnccCoreTransactionErrorRateAboveFiftyPercent

Trigger Condition The number of failed transactions is above 50 percent of the total transactions.
Severity critical
Alert details provided CNCC Core transaction Error Rate detected above 50 Percent of Total Transactions
Metric Used oc_ingressgateway_http_responses_total
Resolution The alert is cleared when the number of failure transactions are below 50 percent of the total transactions.Steps:1. Check the Service specific metrics to understand the specific service request errors.2. If guidance required, contact My Oracle Support.

8.4.12 CnccCoreUnauthorizedAccess

Table 8-43 CnccCoreUnauthorizedAccess

Trigger Condition If the count of unauthorized access goes above the configured value of '3'
Severity warning
Alert details provided Unauthorized Accesses have been detected in CNCC-Core for request. The configured threshold value is 3 for every 5 min
Metric Used oc_ingressgateway_http_responses_total
Resolution

The alert gets cleared when the total unauthorized accesses for a particular user go below the threshold value (default value is '3') in the last 5 min (default value is 5 m)

Note : The threshold and time is configurable in the alerts.yaml

If guidance required, Contact My Oracle Support.