Maintenance Procedures

4 Maintenance Procedures

This section provides details about the OSO maintenance procedures.

4.1 Postinstallation CNE Configuration

4.1.1 Changing Metrics Storage Allocation

The following procedure describes how to increase or decrease the amount of persistent storage allocated to Prometheus for metrics storage.

Prerequisites

The user must calculate the revised amount of persistent storage required by the metrics.

Procedure

A Prometheus resource is used to configure all Prometheus instances running in OSO. Run the following command to identify the Prometheus resource:
```
kubectl get prometheus -n <namespace>
```
Run the following command to resize the Prometheus metric allocation size by setting the value of allowVolumeExpansion to true.
```
$ kubectl -n <namespace> get sc 
$ '{"allowVolumeExpansion": true}'
```

Run the following command to scale the Prometheus pod down.

$ kubectl -n <namespace> scale deploy oso-prom-svr --replicas 0

Run the following command to change the pvc size of Prometheus pods:
```
$ kubectl -n <namespace> edit pvc oso-prom-svr
```
Note:
You will be placed in a vi editor session that contains all of the configurations for the OSO Prometheus pvc. Scroll down to the line that contains the "spec.Capacity" key, then update the value to the <desired increased pv size> as configured in the above step. The file must look similar to the following example:
```
spec.Capacity: 10Gi
```
Type ":wq" to exit the editor session and save the changes.
Run the following command to verify that the pvc size change was applied:
```
$ kubectl get pv | grep oso-prom-svr
```
Note:
Wait until the new desired size gets reflected "10Gi".
Once both the pv sizes are updated to the new desired size, run the following command to scale up the Prometheus pods:
```
$ kubectl -n <namespace> scale deploy oso-prom-svr --replicas 1
```
Note:
You will be placed in a vi editor session that contains all of the configurations for the OSO Prometheus instances. Scroll down to the line that contains the "replicas" key, then change the value back to 2. This scale backs up both the pods. The file must look similar to the following example:
Run the following command to verify that the Prometheus pods are up and running:
```
kubectl get pods -n <namespace> | grep oso-prometheus
```

4.2 Managing 5G NFs

This section describes procedures to manage 5G NFs in CNE OSO.

4.2.1 Updating Alert Rules for an NF

This section describes the procedure to add or update the alerting rules for any Cloud Native Core (CNC) 5G Network Functions (NF) in OSO Prometheus GUI.

Prerequisites

All NFs are required to create a separate Alert-rules.
For OSO Prometheus: A valid OSO release must be installed and an alert file describing all NF alert rules according to old format is required.

Add or Update Alert Rules

Perform the following steps to add alert rules in OSO Prometheus GUI:

Take the backup of current configuration map of OSO Prometheus.
```
$ kubectl get configmaps <OSO-prometheus-configmap-name> -o yaml -n <namespace> /tmp/tempPrometheusConfig.yaml
```
Where,
- <OSO-prometheus-configmap-name> is the name of the OSO Prometheus configuration map.
- <namespace> is the OSO namespace.
Check and add the NF Alert file name inside the Prometheus configuration map.
<nf-alertsname> varies from NF to NF, and can be retrieved from each individual NF alert rules file.

For example, in the following screenshot, "alertscndbtier" is the nf-alertsname for cnDBTier.

Figure 4-1 OSO Alert file
After retrieving the nf-alertsname run the following steps:
```
$ sed -i '/etc\/config\/<nf-alertsname>/d' /tmp/tempPrometheusConfig.yaml
$ sed -i '/rule_files:/a\    \- /etc/config/<nf-alertsname>' /tmp/tempPrometheusConfig.yaml
```

Update the configuration map with the updated file.

$ kubectl -n <namespace> replace configmap <OSO-prometheus-configmap-name> -f
    /tmp/tempPrometheusConfig.yaml

Patch the NF alert rules in OSO Prometheus configuration map by mentioning the Alert-rule file path.
```
$ kubectl patch configmap <OSO-prometheus-configmap-name> -n <namespace> --type merge --patch "$(cat ./NF_altertrules.yaml)"
```