CNC Console Troubleshooting

Unable to display the release version of the NF at CNCC banner

Problem: CNCC banner displays the release version of CNC Console, but not displaying the release version of the NF.

Solution:

The “About” section and Application name displayed next to Oracle logo use the envSystemName and envNFVersion helm fields.
The value set of envSystemName and envNFVersion combines to display the Application name (Application name = envSystemName + envNFVersion).
CNCC Core Custom values have envSystemName and envNFVersion mentioned in it, but these values can be overridden.

Unable to reach CNC Console Core IP or port directly

Problem: Unable to reach CNCC Core IP or port directly. redirect_uri is inserted instead of directly accessing the CNCC Core.

Solution: As per the design, CNC Console redirects requests to CNCC IAM for authentication. On successful authentication, CNCC IAM redirects the user back to CNCC GUI.

'Admin' user created under Cncc realm is unable to access CNCC IAM

Problem: The user with ‘Admin’ privileges is unable to access CNCC IAM.

Solution: Users created under the Cncc realm have access only to CNCC Core and not to CNCC IAM. To access CNCC IAM, create the admin user under the Master realm.

CNCC returns 403 error during NF Configuration

Problem: CNCConsole returns a 403 Error Code and error "Forbidden. Data could not be saved".

Error Code/Error Message:

403/Forbidden

Solution: Log into CNCC IAM to check the roles of the user. The user must have <NF>_READ and <NF>_WRITE roles assigned to perform the write operation on any NF through the CNC Console.

CNC Console returns 500 - Internal Server Error

Problem: CNC Console returns a 500 Error Code while accessing NF Resource.

Error Code/Error Message:

500/Internal Server Error

Solution: The internal server error occurs when the NF routes are not configured correctly. To resolve this error, ensure that correct routes for each NF are configured during deployment. You can provide routes in either of the IP/FQDN in the Instances section:


id: <Instance ID>
type: <NF type>
owner: <ID of cluster owning the Instance>
ip: <IP of NF deployment>
port: <Port of NF deployment

CNCC IAM is accessible, but CNCC Core is not accessible

Problem: CNCC IAM is accessible, but CNCC Core is not accessible.

Error Message:

The ID Token contains invalid claims, which is a JWT validation error, indicating that the system clock on your server is off.

Observation: This issue occurs when Ingress Gateway is behind in time and when CNCC IAM is ahead of time. For example, If IAM (node1) is ahead of time and Ingress Gateway (node2) is 5 minutes behind, the Ingress Gateway invalidates the received token and throws "The ID Token contains invalid claims: {iat=2020-05-26T08:32:12Z}" error.

Solution: To resolve the error, you must ensure that the same time is maintained in CNCC IAM and Ingress Gateway when they run in the same instance or different NTP server instances.

CNCC IAM admin password configured through Kubectl secret is not reflected

Problem:

CNCC IAM admin password change through cncc-iam-secret is not working (Example: if configured cncc-iam-secret).

Solution: During the first installation, CNCC IAM reads the password from the cncc-iam-secret and stores it in the database. So any further changes to the admin password must be done through the CNCC IAM GUI.

Access Error in CNCC Core GUI

Problem:

Unable to access CNCC Core GUI and an “Invalid redirect URI” error occurs.

Observation:

This error occurs when there is a mismatch between the Root URL provided in CNCC IAM Admin Console and the URI through which you access the CNCC Core GUI.

For example, In CNCC IAM, the Root URL is mentioned as http://cncc-core-ingress-gateway.cncc.svc.cluster.local:30075/ and if you are accessing the CNCC Core GUI with IP and NodePort, that is, http://10.75.xx.xx:30075/* or vice-versa, you get “invalid redirect_uri” error on CNCC Core GUI.

Solution: To resolve this error, ensure that the Root URL provided in CNCC IAM and the URI through which you access the CNCC Core GUI are the same.

Changing the CNCC IAM admin password

Problem:

How to change the CNCC IAM admin password using the REST API call.

Solution: Refer the following sections in CNC Console User Guide:

Accessing NF Resources through Curl or Postman
CNC Console IAM REST APIs

Unable to access Kibana

Problem:

Kibana Common Service is not accessible

Solution: To resolve this issue, ensure that you are accessing Kibana through the correct path. The default access path to Kibana is through "/kibana". You can also access Kibana through the URL <node-ip>:<node-port>/mycne-cluster/kibana.

CNC Console Installation failure while installing using cnDBTier

Problem:

While installing CNCC using cnDBTier, the cncc-iam-kc pod does not come up and goes into a crash state.

Solution: cnDBTier needs additional grants such as "REFERENCES, INDEX" due to the addition of db hook job.

CNCC IAM kc pod fails while ASM is enabled

Problem:

While ASM is enabled, CNCC IAM kc pod fails due to Readiness probe failure.

Solution: Check whether annotation "sidecar.istio.io/rewriteAppHTTPProbers" is enabled and set to true under 'nonlbStatefulSets' in custom_cncc-iam_values.yaml during CNCC IAM deployment.

Unable to Acess CNCC GUI when ASM is Enabled

Problem:

Unable to access CNCC GUI after installation as cncc-iam-ingress-gateway is listening on port 8080 instead of port 8081(ASM enabled).

Solution: After installing CNCC, the cncc-iam-ingress-gateway is listening on port 8080 instead of port 8081 when ASM is enabled. To resolve this issue, configure the parameters in the custom_cncc-iam_values.yaml file as follows:

Annotation: sidecar.istio.io/rewriteAppHTTPProbers: "\"true\""
serviceMeshCheck: true
Annotation: sidecar.istio.io/inject: "true"

CNCC Core GUI does not get loaded after logging in

Problem

CNCC Core microservices are up and running but CNCC Core GUI does not get loaded after logging in.

Solution

CNCC supports only single pod deployment, check the following configurations (must be set to 1).

ingress-gateway:
  # Number of Pods must always be available, even during a disruption.
  minAvailable: 1
  # Min replicas to scale to maintain an average CPU utilization
  minReplicas: 1
  # Max replicas to scale to maintain an average CPU utilization
  maxReplicas: 1

Note:

These are preset to 1 and these parameters are not exposed in custom values.

CNC Console is not supporting ASM with mTLS disabled configuration

Problem

When service mesh is enabled and mTLS is disabled with insecure HTTP connections, CNC Console Core microservice is not coming up.

Solution

Update serviceMeshHttpsEnabled to false in custom-cncc-core_values.yaml file to allow insecure HTTP connections.


#Mandatory: This parameter must be set to "true" when CNCC is deployed with the Service Mesh
serviceMeshCheck: true
# If Service Mesh is deployed with TLS/MTLS disabled then set this flag to false
serviceMeshHttpsEnabled: false

Failed to allocate IP for CNC Console IAM Ingress gateway

Problem

Installation of CNCC IAM is successful but while checking CNCC IAM service status, unable to assign the external IP for svc cncc-iam-ingress-gateway and received the following error: Warning Allocation Failed 61s (x3 over 8m48s) metallb-controller Failed to allocate IP for "cncc/cncc-iam-ingress-gateway": no available IPs.

Solution

Check if the annotations are missing from the cncc-iam-ingress-gateway service. Add the missing annotations, due to which the dynamic metalLbIpAllocation will work properly.

Unable to Create required tables in CNCC IAM DB

Problem

Deployment needs two instances of CNCC where only the first instance is deployed correctly. After installing the second instance of CNCC in a different namespace, the pod "cncc-voice-iam-kc-0" repeatedly crashes

Observation

After analyzing the logs, it was found that during the preinstall checks, the hook pods did not create all the required tables in the DB. For example, in the first instance DB, all tables created, while in the second instance DB, there are only 43 tables created. The cbDBTier has a maximum table limit of 512. So, during the deployment of the second instance of CNCC, the maximum table limit threshold has exceeded, and hence 43 tables were created.

Deployment needs two instances of CNCC, first instance is deployed correctly.

After the installation of the second cncc-iam in a different namespace, the pod cncc-voice-iam-kc-0 is crashed repeatedly. By analyzing the logs it seems that not all the tables has been created by the hook pods during the pre-install checks. In the first instance db we can see that there are all tables created while in the second instance we can see only 43 tables.

Solution

To resolve this issue, you must either increase the maximum table limit or clean up unwanted databases to bring table count within the threshold limits. For more information about configuring the table limits, see Oracle Communications Cloud Native Core DBTier Installation and Upgrade Guide.

Default limits in ndb:

MaxNoOfOrderedIndexes: 512

MaxNoOfTables: 512

NoOfFragmentLogFiles: 256

CNCC Core Installation Failing with Validation Hook Error ProblemCNCC Core installation is failing with the validation hook error.SolutionTo resolve this error, in the custom-cncc-core_values.yaml file, check if the multiClusterMultiInstanceEnabled parameter is set to true. When this flag is enabled, the preinstall hook "cncc-core-validation-hook" starts validating the multicluster deployment configurations.If you do not need CNCC multicluster deployment validation, then set the multiClusterMultiInstanceEnabled parameter to false.

Resolve CNC Console Validation hook error

Problem

Validation hook error occurs during CNCC Core Deployment.

Solution

To resolve this issue, enable Helm Configuration Validation for CNCC Deployment, applicable for M-CNCC Core and A-CNCC deployment.

Check the cncc-acore-validation-hook or cncc-mcore-validation-hook pod logs for the error codes. Make the required corrections in the custom-cncc-core_values.yaml file and reinstall M-CNCC Core or A-CNCC. For more information about validation hook and error details, see "CNC Console Multi Cluster Deployment Helm Configuration Validation" section in Oracle Communications Cloud Native Core Console Installation and Upgrade Guide.

Error Code	Error Message Format	Error Scenarios	Sample Error Messages
1001	Invalid value. Resource: <Configuration Name>, ID: <ID>, Attribute: <Attribute>. <More Info>	Port should be Numeric Scheme should be either HTTP/HTTPS IDs should follow the alphanumeric pattern Max Limit should be satisfied for M-CNCC IAM, A-CNCC and Instance Max Length for Instance Id Max Length for Self Cncc Id CS instance must have one of these CS subtypes <grafana, kibana, jaeger, prometheus, alertmanager> Both ip and fqdn cannot be provided. Unsupported type InvalidConfig multicluster flag should be false in case of single-cluster deployment multicluster flag should be true in case of multi-cluster deployment	Invalid value. Resource: mCnccIam, ID: Cluster1, Attribute: Port. It should be numeric value. Invalid value. Resource: instance, ID: Cluster3 Cluster3-instance1, Attribute: Scheme. Allowed values are: [http, https]. Invalid value. Resource: instance, ID: Cluster1-grafana##$$%, Attribute: id. Ids should be alphanumeric with hyphen allowed as special character. The count of mCnccIam exceeded max limit. Allowed Value:x. Actual Value: y Max limit exceeded. Allowed Value:x. Actual Value: y Invalid value. Resource: aCncc, ID: Cluster3, Attribute: N/A. Both ip and fqdn cannot be provided. Invalid value. Resource: isMultiClusterEnabled, ID:,Attribute: False. isMultiClusterEnabled is set as false, only single cluster configuration is allowed. Invalid value. Resource: isMultiClusterEnabled, ID:,Attribute: True. isMultiClusterEnabled is set as true, only multi cluster configuration is allowed.
1002	Duplicate value. Resource: <Configuration Name>, ID: <ID>, Attribute: <Attribute>. <More Info>	All A-CNCC IDs must be unique API prefix must be unique for all instances Owner(Cluster) must have unique CS subtype	Duplicate value(s). Resource: aCncc, ID: [Cluster3], Attribute: id.
1003	Invalid Reference. Resource: <Configuration Name>, ID: <ID>, Attribute: <Attribute>. <More Info>	All the Instance owners must be referenced in M-CNCC IAM IDs or A-CNCC IDs M-CNCC IAM IDs and M-CNCC Core IDs must be same	Invalid Reference. Resource: instance, ID: Cluster5, Attribute: Owner. Not present in mCncc ids or aCncc ids. Invalid Reference. Resource: instance, ID: N/A, Attribute: N/A. M-Cncc Iam ids and M-Cncc Core ids do not match.
1004	Missing value. Resource: <Configuration Name>, ID: <ID>, Attribute: <Attribute>. <More Info>	Missing apiPrefix parameter for type CS Either of IP/FQDN should be present	Missing value. Resource: instance, ID: Cluster4-grafana, Attribute: apiPrefix. Missing value. Resource: instance, ID: Cluster3-PolicyInstance, Attribute: N/A. Either ip or fqdn is required.

Does CNC Console support Command Line Interface (CLI)

Problem: Can NF APIs integrated with CNC Console be accessed through curl or postman.

Solution The NF configuration APIs can be accessed through CNC Console GUI or directly using postman or curl. CNCC providess authentication and authorization in both ways. For more information, see "Generating Access Tokens and Accessing NF Resources" section in Oracle Communications Cloud Native Core Console User Guide.

Upgrade or Rollback Failure

Problem: Upgrade or Rollback Failure

Solution

When CNC Console upgrade or rollback fails, perform the following procedure:

Check the pre or post upgrade or rollback hook logs as applicable.
If the failure occurs, then check the cause of the failure from the logs by running the following command:
```
kubectl logs <pod name> -n <namespace>
```
After detecting the cause of failure, do the following:
- For upgrade failure:
  - If the cause of upgrade failure is database or network connectivity issue, then resolve the issue and rerun the upgrade command.
  - If the upgrade failure occurs during the postupgrade phase, for example, post upgrade hook failure due to target release pod not moving to ready state, then perform a rollback.

For rollback failure: If the cause of rollback failure is database or network connectivity issue, then resolve the issue and rerun the rollback command.

4. If the issue persists, contact My Oracle Support.

CNC Console Upgrade Results IP in Pending state

Problem: CNC Console deployment using static IP is not allocated to the new mcore service during upgrade.

Solution

CNCC supports the single helm chart deployment for deploying all three components M-CNCC IAM, M-CNCC Core and A-CNCC Core.

Earlier CNCC IAM and CNCC Core were deployed independently, now with single helm chart all 3 components can be deployed using single helm install command.

Upgrade from two helm deployments to one helm deployment is supported but one of the helm deployment must be manually deleted.

CNCC IAM deployment can be upgraded which upgrades M-CNCC IAM and freshly install M-CNCC Core and A-CNCC Core services. User can manuallydelete CNCC Core deployment. For more information, see Upgrade and Rollback sections of Oracle Communications Cloud Native Core Console Installation and Upgrade Guide..

In case, if static LoadBalancer IP is used in existing deployment, after the upgrade, new mcore service IP will be shown as pending. IP will be allocated once the existing M-CNCC Core service is uninstalled.

CNC Console Upgrade Displays Port Already in Use Error

Problem: CNCC deployment using static node port throws Port already in use error during upgrade.

Solution

If static port is used in existing deployment, before upgrade, in custom values file port needs to be updated to use another port to avoid port conflict error.

CNC Console Helm Test Fails

Problem: CNCC helm test fails when there are stale jobs or pods.

Solution

In some cases, Helm RC builds have intermittent issues which blocks auto deletion of jobs.

Ensure stable helm version is installed in your environment.

CNC Console Helm Test Fails with Service Account Error

Problem: CNCC helm test fails when there are stale jobs or pods.

CNCC helm test fails with error message "Unauthorized! Configured service account doesn't have access. Service account may have been revoked.".

Solution

The time sync between worker nodes is must for helm test to work. Ensure CNE worker nodes time is in sync.