2 Installing OSO

This chapter provides information about installing Oracle Communications Cloud Native Core, Operations Services Overlay (OSO) in a cloud native environment.

Note:

For release version 25.2.100, OSO supports both fresh installation and upgrade from 25.1.2xx. For more information on how to upgrade OSO, see Upgrading OSO.
OSO, packaged as a Cloud Service Archive (CSAR) file, is a ZIP file that contains the following components:
  • All the required OSO images, including opensource software as a tar file.
  • All the required OSO Helm charts.
  • The following custom values.yaml files:
    • ocoso_csar_25_2_100_0_0_0_prom_custom_values.yaml is used for configuring the Prometheus parameters.
    • ocoso_csar_25_2_100_0_0_0_alm_custom_values.yaml is used for configuring the Alertmanager parameters.
    • ocoso_csar_25_2_100_0_0_0_alr_config_custom_values.yaml is used for configuring the Alert configuration parameters.

    Note:

    The README doc contains the details to populate mandatory values in the ocoso_csar_25_2_100_0_0_0_prom_custom_values.yaml and ocoso_csar_25_2_100_0_0_0_alm_custom_values.yaml files. For more information about the configuration parameters, see OSO Configuration Parameters.

2.1 Prerequisites

Before installing and configuring OSO, ensure that the following prerequisites are met:

  1. CSAR packages are downloaded.
  2. Unzip and TAR utilities are installed.
  3. Docker or Podman (v5.4.0) is installed, and you must be able to run the docker or podman commands.
  4. Minimum supported Helm version (v3.18.2) is installed.
  5. kubectl is installed (1.33.1).
  6. A central repository is made available for all images, binaries, helm charts, and so on, before running this procedure.
  7. The following images are populated in the registry on the repository server:

    Table 2-1 Images in the Registry on the Repository Server

    Image Name Path in Repository with version
    OSO_ALERTMANAGER_IMAGE occne.io/oso/alertmanager:v0.28.0
    OSO_PROMETHEUS_IMAGE occne.io/oso/prometheus:v3.4.1
    OSO_CONFIGMAPRELOAD_IMAGE occne.io/oso/configmapreload:v0.15.0
    OSO_ALERT_CONFIG_IMAGE occne.io/occne/oso_alert_config:25.2.100
    OSO_HELM_TEST_IMAGE occne.io/occne/oso_helm_test:25.2.100
    OSO_SNAPSHOT_IMAGE occne.io/occne/oso_snapshot:25.2.100
    All above images are packed into tar format and will be present in OSO CSAR under (Artifacts/Images) folder except for oso_alert_config which is present in ocoso_alr_config_csar_25_2_100.zip. You can use the following commands to load these images into your cluster's registry.
    $ docker/podman load -i <image-name>.tar
$ docker/podman tag <image-url> <registry-address>:<port>/image-url
$ docker/podman push <registry-address>:<port>/image-url
  8. OSO can now be installed in one of the following ways:
    • without the snapshot utility feature (disabled): proceed as per existing procedure and resize the PVC storage.
    • with the snapshot utility feature (enabled): adjust the size of the PVC storage to fit the snapshot data. Otherwise, the Prometheus pod may crash. OSO recommends the users to configure thrice the existing storage capacity.

      If this feature is enabled, users experience variations in CPU and RAM usage. It is recommended that each NF should benchmark these figures and obtain them from the respective NFs.

      For example, if the PVC storage size required at user environment for OSO Prometheus is 50GB then in snapshot enabled OSO deployment, the storage configuration is given as 150GB (50GB*3). This value is defined in PROM-PV-SIZE for the required size. 

      
persistentVolume:
enabled: true
...
...
mountPath: /data
size: PROM-PV-SIZE
storageClass: "PROM-STORAGE-CLASS"

      Use the web.enable-admin-api parameter to enable the snapshot utility feature. For more information about this parameter, see Prometheus Configuration Parameters.

Note:

Tables Table 2-2 and Table 2-3 in this section offer a comprehensive list of software necessary for the proper functioning of during deployment. However, these tables are indicative, and the software used can vary based on the customer's specific requirements and solution.

The Software Requirement column in Table 2-2 and Table 2-3 tables indicates one of the following:
  • Mandatory: Absolutely essential; the software cannot function without it.
  • Recommended: Suggested for optimal performance or best practices but not strictly necessary.
  • Conditional: Required only under specific conditions or configurations.
  • Optional: Not essential; can be included based on specific use cases or preferences.

Table 2-2 Preinstalled Software Versions

Software 25.2.1xx 25.1.2xx 25.1.1xx Software Requirement Usage Description
Helm 3.18.2 3.16.2 3.15.2 Mandatory

Helm, a package manager, simplifies deploying and managing NFs on Kubernetes with reusable, versioned charts for easy automation and scaling.

Impact:

Preinstallation is required. Without this capability, management of NF versions and configurations becomes time-consuming and error-prone, impacting deployment consistency.
Kubernetes 1.33.1 1.32.0 1.31.1 Mandatory

Kubernetes orchestrates scalable, automated NF deployments for high availability and efficient resource utilization.

Impact:

Preinstallation is required. Without orchestration capabilities, deploying and managing network functions (NFs) can become complex, leading to inefficient resource utilization and potential downtime.
Podman 5.4.0 5.4.0 5.4.0 Recommended

Podman is a part of Oracle Linux. It manages and runs containerized NFs without requiring a daemon, offering flexibility and compatibility with Kubernetes.

Impact:

Preinstallation is required. Without efficient container management, the development and deployment of NFs could become cumbersome, impacting agility.
To check the versions for the preinstalled software in the cloud native environment, run the following commands:
kubectl version
helm version
podman version

If you are deploying in a cloud native environment, these following additional software are to be installed before installing .

Table 2-3 Additional Software Versions

Software 25.2.1xx 25.1.2xx 25.1.1xx Software Requirement Usage Description
AlertManager 0.28.0 0.28.0 0.27.0 Recommended

Alertmanager is a component that works in conjunction with Prometheus to manage and dispatch alerts. It handles the routing and notification of alerts to various receivers.

Impact:

Not implementing alerting mechanisms can lead to delayed responses to critical issues, potentially resulting in service outages or degraded performance.
Configmapreload 0.15.0 0.15.0 0.14.0 Mandatory

Configmapreload is a simple binary to trigger a reload when Kubernetes ConfigMaps or Secrets, mounted into pods, are updated. It watches mounted volume dirs and notifies the target process that the config map has been changed.

Impact:

Not employing this solution would impact downtime as pods would need to restart everytime a configmap is updated.
Prometheus 3.4.1 3.2.0 2.52.0 Mandatory

Prometheus is a popular open source monitoring and alerting toolkit. It collects and stores metrics from various sources and allows for alerting and querying.

Impact:

Not employing this monitoring solution could result in a lack of visibility into NF performance, making it difficult to troubleshoot issues and optimize resource usage.

2.2 Resource Utilization

The use of resources such as CPU and RAM by the OSO services are constrained to ensure that the CPU and RAM do not consume excess resources that could be used by other applications.

During the deployment of the services, each service gets an initial CPU and RAM allocation. Each service is allowed to consume the resources to a specified upper limit while it continues to run. The initial allocation limit and the upper limit are set to the same value for services where the resource consumption limit remains the same as the initial allocation or in a case where increasing the CPU or RAM limits underneath a running application can cause service disruption.

Note:

Each NF is supposed to benchmark their application against OSO services and set the RAM, CPU, and PVC values accordingly. These initial resource request is set by OSO team as a placeholder and it is not recommended for any NF.

The resource requests and limits are as follows:

Table 2-4 CPU, RAM, and Ephemeral Resource Requests and Limits

Service CPU Initial Request (m) CPU Limit (m) RAM Initial Request (Mi) RAM Limit (Mi) PVC (Gb) Ephemeral Storage Request (Gi) Ephemeral Storage Limit (Gi) Instances
Prometheus 2000 4000 16384 (16 Gb) 32768 (32 Gb) 50 4 6 1
Prometheus (with snapshot utility enabled) 4000 8000 32768 (32 Gb) 65536 (64 Gb) 150 8 12 1
Prometheus AlertManager 20 20 64 64 NA 1 1 2

The overall observability services resource usage varies on each worker node. The observability services listed in the table are distributed evenly across all worker nodes in the Kubernetes cluster.

2.3 Downloading the OSO package

To download the OSO package from My Oracle Support, perform the following procedure:
  1. Log in to My Oracle Support using your login credentials.
  2. Click the Patches & Updates tab to locate the patch.
  3. In the Patch Search console, click the Product or Family (Advanced) option.
  4. In the Product field, enter Oracle Communications Cloud Native Core - 5G and select the product from the Product drop-down list.
  5. From the Release drop-down list, select Oracle Communications Cloud Native Core Network Repository Function <release_number>.

    Where, <release_number> indicates the required release number of OSO.

  6. Click Search.

    The Patch Advanced Search Results list appears.

  7. Select the required patch from the list.

    The Patch Details window appears.

  8. Click Download.

    The File Download window appears.

  9. Click the <p********_<release_number>_Tekelec>.zip file to download the release package.

    Where, <p********> is the MOS patch number and <release_number> is the release number of OSO.

2.4 Installing OSO Using CSAR

Open Network Automation Platform (ONAP) compliant orchestrator uses CSAR format to onboard, validate, and install OSO. However, in the absence of an orchestrator, manual installation is possible using the CSAR file contents.

Perform the following steps to install OSO using CSAR artifacts:
  1. Download the OSO CSAR zip files from My Oracle Support (MOS).
  2. Extract the CSAR zip files:
    $ unzip <OSO CSAR package>
$ unzip -d ./oso_csar/ ocoso_csar_<25.2.100 version>.zip
$ unzip -d ./oso_alert_config_csar/ ocoso_alr_config_csar_<25.2.100 version>.zip
  3. Upload all the artifacts present in the Artifacts/Images folder to the configured repository.

    The following structure indicates the CSAR file containing the Prometheus and alertmanager files.

    ├── Artifacts
│   ├── ChangeLog.txt
│   ├── Images
│   │   ├── alertmanager-0.28.0.tar
│   │   ├── configmapreload-0.15.0.tar
│   │   ├── oso-helm-test-25.2.100.tar
│   │   ├── oso-snapshot-25.2.100.tar
│   │   └── prometheus-3.4.1.tar
│   ├── Licenses
│   ├── Scripts
│   │   ├── ocoso_csar_25_2_100_0_0_0_alertmanager_charts.tgz
│   │   ├── ocoso_csar_25_2_100_0_0_0_alm_custom_values.yaml
│   │   ├── ocoso_csar_25_2_100_0_0_0_prometheus_charts.tgz
│   │   ├── ocoso_csar_25_2_100_0_0_0_prom_custom_values.yaml
│   │   └── OSO_README.md
│   └── Tests
├── Definitions
│   ├── etsi_nfv_sol001_vnfd_2_5_1_types.yaml
│   ├── oso_vnfd.yml
│   └── helm_extensions_1_0.yaml
├── ocoso_csar_25_2_100_0_0_0.zip
├── oso_vnfd.mf
└── TOSCA-Metadata
    └── TOSCA.meta

    The following structure indicates the CSAR file containing the alert configuration files.

    .
├── Artifacts
│   ├── ChangeLog.txt
│   ├── Images
│   │   └── oso-alert-config-25.2.100.tar
│   ├── Licenses
│   ├── Scripts
│   │   ├── ocoso_alr_config_csar_25_2_100_0_0_0_alert_config_charts.tgz
│   │   ├── ocoso_alr_config_csar_25_2_100_0_0_0_alert_config_custom_values.yaml
│   │   └── OSO_ALR_CONFIG_README.md
│   └── Tests
├── Definitions
│   ├── etsi_nfv_sol001_vnfd_2_5_1_types.yaml
│   ├── oso_alr_config_vnfd.yml
│   └── helm_extensions_1_0.yaml
├── ocoso_alr_config_csar_25_2_100_0_0_0.zip
├── oso_alr_config_vnfd.mf
└── TOSCA-Metadata
    └── TOSCA.meta
    For more information about the artifacts, see the Prerequisites section.
  4. From 25.1.200 onwards, the custom_values.yaml file is split into three different files:
    • ocoso_csar_25_2_100_0_0_0_prom_custom_values.yaml is used for configuring the Prometheus parameters
    • ocoso_csar_25_2_100_0_0_0_alm_custom_values.yaml is used for configuring the Alertmanager parameters.
    • ocoso_csar_25_2_100_0_0_0_alert_config_custom_values.yaml is used for configuring the alerts.
    These files are required to be populated and installed separately. Update the respective custom_values.yaml files inside respective Artifacts/Images directory with the required values as mentioned in OSO Configuration Parameters.
  5. Enable IPv6 Dualstack.
    1. To enable IPv6 DualStack in Prometheus
      • Enable IPv6 Dualstack in the ocoso_csar_25_2_100_0_0_0_prom_custom_values.yaml file. Search for the following comment in the yaml file and uncomment the following four lines after this comment.
        # Custom section to enable IPV6, Uncomment below section in order to enable Dualstack OSO
      having both Ipv4 and Ipv6 addresses                   
#ipFamilies:
#- IPv4
#- IPv6
#ipFamilyPolicy: PreferDualStack
      • Change the service type from ClusterIP to LoadBalancer to assign IPv6.
      • Save the file and proceed with the normal installation.
    2. To enable IPv6 DualStack in AlertManager Replace "ENABLE_DUAL_STACK" with "true" in below section inside the ocoso_csar_25_2_100_0_0_0_alm_custom_values.yaml file before installing alertmanager.
      # ip dual stack
  ipDualStack:
    enabled: ENABLE_DUAL_STACK
    ipFamilies: ["IPv6", "IPv4"]
    ipFamilyPolicy: "PreferDualStack"
  6. Save the file and proceed with the normal installation.
  7. Install OSO using the following steps:
    1. Extract the Helm charts tgz files which are available in the Artifacts/Scripts directory and populate the three custom_values.yaml files.
      $ cd Artifacts/Scripts 
$ tar -xvzf ocoso_csar_25_2_100_0_0_0_alertmanager_charts.tgz
$ tar -xvzf ocoso_csar_25_2_100_0_0_0_prometheus_charts.tgz
$ tar -xvzf ocoso_csar_25_2_100_0_0_0_alert_config_charts.tgz
    2. Install Prometheus and Alertmanager using Helm charts provided and updated ocoso_csar_25_2_100_0_0_0_prom_custom_values.yaml and ocoso_csar_25_2_100_0_0_0_alm_custom_values.yaml file with the following commands. To install alert-config feature, see Automated Configuration of NF Alerts.

      Use the following command for installation using Helm, if custom labels are given:

      
$ kubectl create namespace <deployment-namespace-name> 
$ helm --namespace=<deployment-namespace-name> install <oso-prom-release-name> -f <ocoso_csar_25_2_100_0_0_0_prom_custom_values.yaml> ./ocoso_csar_25_2_100_0_0_0_prometheus_charts.tgz
$ helm --namespace=<deployment-namespace-name> install <oso-alertmanager-release-name> -f <ocoso_csar_25_2_100_0_0_0_alm_custom_values.yaml> ./ocoso_csar_25_2_100_0_0_0_alertmanager_charts.tgz

      For example:

      
$ kubectl create namespace <oso-namespace>
# Example for Prometheus:
$ helm --namespace=<oso-namespace> install <oso-prom-release-name> -f ocoso_csar_25_2_100_0_0_0_prom_custom_values.yaml
        ./ocoso_csar_25_2_100_0_0_0_prometheus_charts.tgz
# Similarly for Alertmanager
$ helm --namespace=<oso-namespace> install <oso-alertmanager-release-name> -f ocoso_csar_25_2_100_0_0_0_alm_custom_values.yaml
        ./ocoso_csar_25_2_100_0_0_0_alertmanager_charts.tgz
      
$ kubectl create namespace ns1
## Prometheus
$helm install -f ocoso_csar_25_2_100_0_0_0_prom_custom_values.yaml --namespace=ns1 --name-template=oso-p ./ocoso_csar_25_2_100_0_0_0_prometheus_charts.tgz
##  Alertmanager
$ helm install -f ocoso_csar_25_2_100_0_0_0_alm_custom_values.yaml --namespace=ns1 --name-template=oso-a ./ocoso_csar_25_2_100_0_0_0_alertmanager_charts.tgz
    3. Run the following command to perform a Helm test. Populate the values in the ocoso_csar_25_2_100_0_0_0_prom_custom_values.yaml and ocoso_csar_25_2_100_0_0_0_alm_custom_values.yaml files to run the Helm test. Run the Helm test separately for each Alertmanager and Prometheus.
      $ helm test <release-name> -n <oso-namespace>

      Note:

      Helm Test can be run for the first time smoothly, but if some issue occurs, and there's a need to re-run the helm test, first you will have to delete the existing test job and repeat the "helm test" command as shown above. 
      $ kubectl get jobs.batch -n <namespace>
      $ kubectl delete jobs.batch oso-test -n <namespace>

      Figure 2-1 Helm Test


      Helm Test

2.4.1 Automated Configuration of NF Alerts

This section explains the process of automating the NF alerts configuration in OSO after deploying all the relevant NFs and OSO components. The process manages the alert rule definitions and dynamically updates the OSO Prometheus configmap as part of the Helm chart installation or upgrade.

  1. Prerequisites
    1. Install the respective NF, CNC Console, cnDBTier, and OSO in their designated namespace.
    2. The existing structure of the nf-alertrule.yaml file remains unchanged. Operators must prepare the alert rule file(s) and place them in a directory on the server, which will be provided as input to the new Helm chart.
  2. Existing Functionality
    NF Alerts are configured in Prometheus manually post OSO deployment by performing the following steps:
    
kubectl get configmaps occne-prometheus-server -o yaml -n <oso-namespace> > /tmp/tempConfig.yaml
$ sed -i '/etc\/config\/alertscncc/d' /tmp/tempConfig.yaml
$ sed -i '/rule_files:/a\ \- /etc/config/alertscncc' /tmp/tempConfig.yaml
$ kubectl replace configmap occne-prometheus-server -f /tmp/tempConfig.yaml
$ kubectl patch configmap occne-prometheus-server -n occne-infra --type merge --patch "$(cat ~/occncc_alertrules_<version>.yaml)"
  3. Alert Automation
    1. Installing Alert Automation feature using Helm
      1. Extract the Helm charts tgz file and available in the Artifacts/Scripts directory. 
        $ cd Artifacts/Scripts
$ tar -xvzf ocoso_csar_25_2_100_0_0_0_alert_config_charts.tgz

        The following structure indicates the CSAR file containing the alert configuration files.

        .
├── Artifacts
│   ├── ChangeLog.txt
│   ├── Images
│   │   └── oso-alert-config-25.2.100.tar
│   ├── Licenses
│   ├── Scripts
│   │   ├── ocoso_alr_config_csar_25_2_100_0_0_0_alert_config_charts.tgz
│   │   ├── ocoso_alr_config_csar_25_2_100_0_0_0_alert_config_custom_values.yaml
│   │   └── OSO_ALR_CONFIG_README.md
│   └── Tests
├── Definitions
│   ├── etsi_nfv_sol001_vnfd_2_5_1_types.yaml
│   ├── oso_alr_config_vnfd.yml
│   └── helm_extensions_1_0.yaml
├── ocoso_alr_config_csar_25_2_100_0_0_0.zip
├── oso_alr_config_vnfd.mf
└── TOSCA-Metadata
    └── TOSCA.meta

        Note:

        Deploy the oso-alr-config Helm chart after OSO Prometheus and Alertmanager are installed using the helm install command.
      2. Provide the appropriate values in the ocoso_csar_25_2_100_0_0_0_alert_config_custom_values.yaml file for oso-alr-config configmap and proceed with the installation of the alert automation feature. The chart can accept one or more alert rule files during deployment.
        • Install the alert configuration feature by providing one alert-rule file:
          helm -n <oso-namespace> install <oso-alr-config-feature-name> ./oso-alr-config -f ocoso_csar_25_2_100_0_0_0_alert_config_custom_values.yaml -f path/to/nf_alertrules.yaml
        • Install the alert configuration feature by providing multiple alert-rule files:
          helm -n <oso-namespace> install <oso-alr-config-feature-name> ./oso-alr-config -f ocoso_csar_25_2_100_0_0_0_alert_config_custom_values.yaml -f path/to/nf1_alertrules.yaml -f path/to/nf2_alertrules.yaml
      3. Run the following command to verify if oso-alr-config configmap is created by Helm.
        $ kubectl -n <oso-namespace> get configmap
        Example:
        kubectl -n oso get cm
        Sample output 
        
NAME DATA AGE
oso-alr-config-cm 2 2d21h
oso-prom-svr 8 2d21h
      4. Verify if the alerts are successfully loaded in the Prometheus GUI. For more information, see Verifying Installation section to understand how to access OSO Prometheus GUI and how to verify alerts.
      5. If no alert rule file is provided during installation, the script will not perform any actions, but the oso-alr-config Helm chart will still be installed. It is strongly recommended to supply at least one alert rule file, as it is required for configuration.

        Note:

        The deployment fails in any of the following cases:
        • If OSO-PROM-CONFIGMAP-NAME does not exist (Helm deployment fails before triggering script).
        • If there are validation errors.
        • If K8s API returns error.
    2. Adding more alerts using Helm Upgrade
      • Prepare the NF alert rule file and trigger a Helm upgrade for the oso-alr-config chart by providing the alert rule file as input.
      • The deployment fails in any of the following cases:

        Note:

        • If OSO-PROM-CONFIGMAP-NAME or oso-alr-config-cm does not exist.
        • If there are validation errors.
        • If K8s API returns error.
      • The chart can accept one or more alert rule files during deployment. Run the following command to provide multiple files:
        Using Multiple -f Flags
helm upgrade oso-alr-config oso-alr-config/ -f <ocoso_csar_25_2_100_0_0_0_alert_config_custom_values.yaml> -f <path/to/nf1_alertrules.yaml> -f <path/to/nf2_alertrules.yaml> -n <oso-namespace>

        Note:

        It is recommended to provide all the alert files together like all NFs, cnDBTier, and CNC Console).
      • OSO_ALERTRULE_CONFIGMAP will be created by Helm and will be accessible to user using the following command:
        $ kubectl -n <oso-namespace> get configmap

        For example, kubectl -n oso get cm

        Sample output: 
        
NAME DATA AGE
oso-alr-config-cm 2 2d21h
oso-prom-svr 8 2d21h
      • After successfully verifying the upgrade, Route to the OSO Prometheus GUI to verify if the Alerts are successfully loaded, For more information about accessing Prometheus GUI and verifying the alerts, see Verifying Installation.
    3. Unpatching or removing the alert automation feature
      • Clear the alerts from the OSO Prometheus ConfigMap and Prometheus GUI by using an empty alert rule file will be provided during the Helm upgrade. However, the references are retained under rule_files "/etc/config/alertscncc" and alert rules will be empty "alertscncc: { }"

        Note:

        It is recommended to unpatch all alerts together. This can be achieved by providing all the empty alert files at once, which restores the alerts section in the OSO deployment to its original state.
      • Prepare empty alert files (empty_nf_alertrules.yaml) for all the respective NFs that should be removed from the Prometheus GUI.
        For example, CNC Console alerts should not be listed under Prometheus GUI.
        
apiVersion: v1
data:
  alertscncc: |
    {}
      • Run a helm upgrade (for more information, see step 3.b) and pass these empty files.

        The following example, indicates when only 1 empty file of CNC Console is given to display the effect of this empty file.

        Updated Prometheus ConfigMap (oso-prom-svr configMap output):
        kubectl get configmap/oso-prom-svr -n default -oyaml
apiVersion: v1
data:
  alerting_rules.yml: |
    {}
  alerts: |
    {}
  alertscncc: |
    {}
  alertsnrf: |
    groups:
    - name: OcnrfAlerts
      rules:
      - alert: OcnrfTotalIngressTrafficRateAboveMinorThreshold
        annotations:
          description: 'Ingress traffic Rate is above the configured minor threshold i.e. 800 requests per second (current value is: {{ $value }})'
          summary: 'timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 80 Percent of Max requests per second(1000)'
        expr: sum(rate(oc_ingressgateway_http_requests_total{app_kubernetes_io_name="ingressgateway",kubernetes_namespace="ocnrf"}[2m])) by (kubernetes_namespace) >= 800 < 900
        labels:
          severity: minor
          oid: "1.3.6.1.4.1.323.5.3.36.1.2.7001"
          kubernetes_namespace: ' {{ $labels.kubernetes_namespace }} '
      - alert: OcnrfTotalIngressTrafficRateAboveMajorThreshold
        annotations:
          description: 'Ingress traffic Rate is above the configured major threshold i.e. 900 requests per second (current value is: {{ $value }})'
          summary: 'timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Traffic Rate is above 90 Percent of Max requests per second(1000)'
        expr: sum(rate(oc_ingressgateway_http_requests_total{app_kubernetes_io_name="ingressgateway",kubernetes_namespace="ocnrf"}[2m])) by (kubernetes_namespace) >= 900 < 950
        labels:
          severity: major
          oid: "1.3.6.1.4.1.323.5.3.36.1.2.7002"
          kubernetes_namespace: ' {{ $labels.kubernetes_namespace }} '
  alertsscp: |
    groups:
    - name: SCPAlerts
      rules:
      - alert: SCPIngressTrafficRateAboveMinorThreshold
        annotations:
          description: 'Ingress Traffic Rate at locality: "{{$labels.ocscp_locality}}"  is above minor threshold (i.e. 2380 mps)'
          summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}},scpfqdn: {{$labels.ocscp_fqdn}},scpauthority:{{$labels.ocscp_authority}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Current Ingress Traffic Rate is {{ $value | printf "%.2f" }} mps which is above 70 Percent of Max MPS(3400)'
        expr: sum(rate(ocscp_metric_http_rx_req_total{app_kubernetes_io_name="scp-worker"}[2m])) by (kubernetes_namespace,ocscp_locality,kubernetes_pod_name,ocscp_fqdn,ocscp_authority) >= 2380 < 2720
        for: 1m
        labels:
          severity: minor
          alertname: "SCPIngressTrafficRateAboveMinorThreshold"
          oid: "1.3.6.1.4.1.323.5.3.35.1.2.7001"
          namespace: '{{$labels.kubernetes_namespace}}'
          podname: '{{$labels.kubernetes_pod_name}}'
          vendor: oracle
      - alert: SCPIngressTrafficRateAboveMajorThreshold
        annotations:
          timestamp: ' {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} '
          description: 'Ingress Traffic Rate at locality: {{$labels.ocscp_locality}} is above major threshold (i.e. 2720 mps)'
          summary: 'namespace: {{$labels.kubernetes_namespace}}, podname: {{$labels.kubernetes_pod_name}},scpfqdn: {{$labels.ocscp_fqdn}},scpauthority:{{$labels.ocscp_authority}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }}: Current Ingress Traffic Rate is {{ $value | printf "%.2f" }} mps which is above 80 Percent of Max MPS(3400)'
        expr: sum(rate(ocscp_metric_http_rx_req_total{app_kubernetes_io_name="scp-worker"}[2m])) by (kubernetes_namespace,ocscp_locality,kubernetes_pod_name,ocscp_fqdn,ocscp_authority)  >= 2720 < 3230
        for: 1m
        labels:
          severity: major
          alertname: "SCPIngressTrafficRateAboveMajorThreshold"
          oid: "1.3.6.1.4.1.323.5.3.35.1.2.7001"
          namespace: '{{$labels.kubernetes_namespace}}'
          podname: '{{$labels.kubernetes_pod_name}}'
          vendor: oracle
  allow-snippet-annotations: "false"
  prometheus.yml: |
    global:
      evaluation_interval: 1m
      scrape_interval: 1m
      scrape_timeout: 30s
    rule_files:
    - /etc/config/alertsnrf
    - /etc/config/recording_rules.yml
    - /etc/config/alerts
    - /etc/config/alerting_rules.yml
    - /etc/config/alertsscp
    - /etc/config/alertscncc
    - /etc/config/rules
......
......
......

        Screenshot of Prometheus GUI without CNC Console alerts:

        Figure 2-2 Prometheus GUI with No CNC Console Alert Files


        Prometheus GUI with No CNC Console Alert Files

    4. Helm Uninstall
      1. Run the following command to uninstall oso-alr-config Helm chart.
        helm uninstall oso-alr-config -n <oso-namespace>

        During oso-alr-config Helm chart uninstallation, no hooks are executed. As a result, the OSO Prometheus configmapMap and its alert configurations remain intact, while all other oso-alr-config resources are deleted.

      2. In the event of a failure during an upgrade followed by an uninstall, the failed jobs may persist. In such cases, you must manually clear all jobs using the following command:
        • Run the following command to retrieve the jobs:
          #Get the jobs
$ kubectl get jobs -n <oso-namespace>

          Sample output: 

          NAME                               COMPLETIONS   DURATION   AGE
oso-alr-config-hook-post-install   0/1           138m       138m
        • Run the following command to delete the jobs:
          
# Delete the jobs
$ kubectl delete job oso-alr-config-hook-post-install
          Sample output: 
          
job.batch "oso-alr-config-hook-post-install" deleted

2.5 Postinstallation Tasks

This section explains the postinstallation tasks for OSO.

2.5.1 Verifying Installation

To verify if OSO is installed:

  1. Run the following command to verify the OSO version.
    helm ls -n <oso-namespace>

    For example:

    helm ls -n oso-name
    Sample output:
    
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                APP VERSION
oso     oso          3               2024-03-27 18:06:42.575679972 +0000 UTC deployed        prometheus-15.16.1      24.1.0
  2. Run the following commands to verify if pods are up and running:
    $ kubectl get pods --namespace <deployment-namespace-name>

    For example:

    $ kubectl get pods -n occne-infra
    Sample output:
    NAME                            READY   STATUS    RESTARTS   AGE
oso-prom-alm-0                  2/2     Running   0          14h
oso-prom-alm-1                  2/2     Running   0          14h
oso-prom-svr-84c8c7d488-qsnvx   2/2     Running   0          14h
  3. Run the following commands to verify if services are up and running and are assigned an EXTERNAL-IP (if LoadBalancer is being used):
    $ kubectl get service --namespace <deployment-namespace-name>
    For example:
    $ kubectl get service -n occne-infra
    Sample output:
    NAME                                TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)           AGE
oso-prom-alm                        ClusterIP      10.233.16.83    <none>          80/TCP      14h
oso-prom-alm-headless               ClusterIP      None            <none>          80/TCP,6783/TCP   14h
oso-prom-svr                        ClusterIP      10.233.46.136   <none>          80/TCP      14h
  4. Verify that all the GUIs are accessible.

    Note:

    Prometheus and Alertmanager GUIs can be accessed only using the CNC Console. For more information about accessing Prometheus and Alertmanager GUIs using CNC Console, see Oracle Communications Cloud Native Configuration Console User Guide.

    If the service is of type LoadBalancer, use EXTERNAL-IP to open the Prometheus GUI. Refer to Step 2 to get the services and their EXTERNAL-IPs.

    Example to access service IP address with output:

    # kubectl get nodes -o wide
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                  KERNEL-VERSION                  CONTAINER-RUNTIME
master   Ready    master   87d   v1.17.1   10.75.226.13    &lt;none&gt;        Oracle Linux Server 7.5   4.1.12-112.16.4.el7uek.x86_64   docker://19.3.11
slave1   Ready    &lt;none&gt;   87d   v1.17.1   10.75.225.177   &lt;none&gt;        Oracle Linux Server 7.5   4.1.12-112.16.4.el7uek.x86_64   docker://19.3.11
slave2   Ready    &lt;none&gt;   87d   v1.17.1   10.75.225.47    &lt;none&gt;        Oracle Linux Server 7.5   4.1.12-112.16.4.el7uek.x86_64   docker://19.3.11
    
# kubectl get service -n ocnrf
NAME                                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)           AGE
oso-prom-alm                          ClusterIP      10.103.63.10     &lt;none&gt;         80/TCP            35m
oso-prom-alm-headless                 ClusterIP      None             &lt;none&gt;         80/TCP            35m
oso-prom-svr                          ClusterIP      10.101.91.81     &lt;none&gt;         80/TCP            35m

    Note:

    In case Dualstack with IPv6 is enabled, change the service type to LoadBalancer in the custom values file. You can see both IPs when calling the kubectl get service command, and the output is as follows:
    [root@master Scripts]# kubectl get service -n ocnrf
NAME                         TYPE           CLUSTER-IP      EXTERNAL-IP                          PORT(S)           AGE
oso-prom-alm                 LoadBalancer   10.103.63.10    10.75.202.205,2606:b400:605:b809::2  80/TCP            35m
oso-prom-alm-headless        ClusterIP      None            <none>                               80/TCP            35m
oso-prom-svr                 LoadBalancer   10.101.91.81    10.75.202.204,2606:b400:605:b809::1  80/TCP            35m

    Figure 2-3 Prometheus GUI


    Prometheus GUI

    Figure 2-4 Alert Manager GUI

    Alert Manager GUI
  5. On the Prometheus GUI, click Alerts to verify that all the alerts (NF Alerts) are visible.

    Note:

    OSO Prometheus doesn't have any alerts of its own, therefore the GUI must appear empty initially. You can patch any NF alert rules in this section.

    The following image displays the alerts for cnDBTier:

    Figure 2-5 Prometheus GUI


    Prometheus GUI

    After alerts are raised, the GUI must display the triggered alerts as shown in the following image:

    Figure 2-6 Prometheus GUI - Alerts


    Prometheus GUI - Alerts

  6. Select Alerts tab on the Alertmanager GUI to view the triggered alerts as shown in the following image:

    Figure 2-7 Alertmanager - Alerts


    Alertmanager - Alerts

  7. On the Prometheus GUI, click Graph to verify if the expected metrics (example, NF Metrics) are displayed. The following image displays a sample Prometheus Graph with metrics:

    Figure 2-8 Prometheus Graph


    Prometheus Graph

  8. On the Prometheus GUI, navigate to Status and then click Target to verify if the configured targets are displayed.
    The following image shows Prometheus targets that are being extracted:

    Figure 2-9 Prometheus Target


    Prometheus Target

Note:

Post installation, Prometheus can be vertically scaled to add more resources to an existing Prometheus server for load balancing. For more information about, Prometheus vertical scaling, see Prometheus Vertical Scaling.