Troubleshooting Oracle Database Appliance

Troubleshooting Data Preserving Reprovisioning Upgrades

Understand the errors you may encounter during Data Preserving Reprovisioning upgrade and their solutions.

Checks related to TDE-enabled databases

Scenario: The database precheck Validate TDE wallet presence may fail with the following error, for a TDE-enabled database with TDE Wallet Management attribute set to EXTERNAL value and with software keystore used for TDE configuration.

 Pre-Check                      Status   Error                              Action
 ------------------------------ -------- -------------------------------------------------
 Validate TDE wallet presence    Failed  Both Password Protected Wallet        Make sure that both, Password        
                                        (ewallet.p12) and Autologin Wallet     Protected Wallet (ewallet.p12) and   
                                        (cwallet.sso) are not found at         Autologin Wallet (cwallet.sso) are   
                                        '/u01/app/odaorahome/oracle/product/   present at mentioned location        
                                        19.0.0.0/dbhome_1/admin/extdb/                                              
                                        wallets' location for database 'extdb'

Cause: The reason for the above failure is that both the TDE wallets (ewallet.p12 and cwallet.sso) of the database is not present at the location dbhome>/admin/db_uniquename/wallets. Note that db_uniquename must be in lowercase.

Action Required: Create the path dbhome>/admin/db_uniquename/wallets, if it does not exist and then copy both TDE wallets (ewallet.p12 and cwallet.sso) of the database to the same location. Create the preupgrade report again. Once Data Provisioning Reprovisioning completes, that is, after restoring the node with Oracle Grid Infrastructure and database, both TDE wallets at dbhome>/admin/db_uniquename/wallets can be deleted.

Checks related to Oracle Grid Infrastructure

Scenario: The Oracle Grid Infrastructure precheck

Check
                    custom filesystems

may fail with the following error:

Check custom filesystems       Failed   File systems /acfsmounts/acfs1 are     Remove the file systems determined in
                                        owned by OS users not provisioned by   the check                           
                                        ODA

Cause: The file system /acfsmounts/acfs1 is owned by the operating system user and is not created by Oracle Database Appliance.

Action Required: Remove the file system from Oracle Clusterware manually. After completing the Data Preserving Reprovisioning, remount this file system manually.

Checks related to Oracle ILOM

Scenario: Configuration of Oracle ILOM host name may fail with the following error:

Job details
----------------------------------------------------------------
                     ID:  da5079f6-875b-435f-918f-7cb2974121e3
            Description:  Restore node service - GI
                 Status:  Failure (To view Error Correlation report, run "odacli describe-job -i ... --ecr" command)
                Created:  January 16, 2024 7:46:23 AM GMT
                Message:  DCS-10001:Internal error encountered: Failed to configure hostname the ilom (none).

Cause: Incorrect Oracle ILOM metadata may persist during the detach-node operation.

Action Required: Run cleanup.pl on all the nodes sequentially without the -f, -erasedata, or -nodpr options. Create a backup copy of the file /opt/oracle/oak/restore/metadata/provisionInstance.json and edit the original file. Delete the Oracle ILOM section from the file. For high-availability systems, there are two entries. A sample section is as follows:

"ilom" : {
       "ilomName" : "...",
       "ipAddress" : "...",
       "subNetMask" : "...",
       "gateway" : "..."
     },

Save the file and retry the operation.

Checks related to operating system

Scenario: The operating system precheck

Check Required OS
                    files

may fail with the following error:

Check Required OS files        Failed   Required file                          Identify the cause why file is       
                                        '/opt/oracle/dcs/dcscli/dcscli_wallet/ missing, remediate that and then     
                                        cwallet.sso' not found                 retry the operation

Cause: The operating system file required for the system upgrade is missing.

Action Required: Contact My Oracle Support to create the file.

Scenario: The operating system precheck

Check Additional OS
                    RPMs

may display the following alert:

Check Additional OS RPMs       Alert    Additional OS RPMs, compared to the    None; the list of these RPMs can be  
                                        base ODA image, are installed on the   found at                             
                                        system                                 '/opt/oracle/dcs/log/                
                                                                               reprovision-custom-rpms.list'. The   
                                                                               upgraded versions of these rpms will 
                                                                               have to be reinstalled manually after
                                                                               reimage

Cause: The system may have extra RPMs installed which are not managed by Oracle Database Appliance.

Action Required: After completing the Data Preserving Reprovisioning flow, manually install the additional RPMs listed in the custom-rpms.list file, located at /opt/oracle/oak/restore/.

Scenario: The Storage precheck Check Required Storage files may fail with the following error:

Check Required Storage files   Failed   Required file '/etc/multipath.conf'    Identify the cause why file is       
                                        not found                              missing, remediate that and then     
                                                                               retry the operation

Cause: The required storage file needed for the operating system upgrade is missing.

Action Required: Contact My Oracle Support to create the file.

Scenario: The database precheck Validate Database Version may fail with the following error:

Validate Database Version      Failed   Version '19.10.0.0.210119' for         Please update the database to the    
                                        database 'odacn' is lower than         minimum supported version or higher  
                                        minimum supported version                                                   
                                        '19.17.0.0.221018'                                                          
                                         
Validate Database Version      Failed   Version '12.1.0.2.210119' for          Please update the database to the    
                                        database 'dbj3' is lower than minimum  minimum supported version or higher  
                                        supported version '12.1.0.2.220719'                                         
 
Validate Database Version      Failed   Version '12.2.0.1.210119' for          Please update the database to the    
                                        database 'dbj4' is lower than minimum  minimum supported version or higher  
                                        supported version '12.2.0.1.220118'

Cause: Note that for the bare metal systems, only Oracle Database release 19c is supported. No other database is currently supported. The supported Oracle Database releases are as follows:

Oracle Database release	Minimum Version	The last Oracle Database Appliance release that provided the Oracle Database clone file
11.2.0.4	1.2.0.4.210119	19.10
12.1	12.1.0.2.220719	19.16
12.2	12.2.0.1.220118	19.14
18c	18.14.0.0.210420	19.11

Action Required: For Oracle Database release 11g databases, there is no ODACLI support to update the database. You must manually update the database to 11.2.0.4.210119 using OPatch. Then use the odacli update-registry command to update the metadata.

For Oracle Database releases 12.1.x, 12.2.x, and 18c, do the following:

Update the database to the last supported Oracle Database Appliance release.
Run the odacli update-repository -f serverzip_for_ODA_release command.
Run the odacli update-repository -f ODA_DB_CLONE_for_minimum_version command.
Generate prepatch report using the version as ODA_RELEASE.
Update the database, which creates the new database home.
Use the odacli delete-dbhome command to delete the database home. Note that if there are additional databases that run from the database home, you must patch all these databases before you can delete the database home.
Use the odacli upgrade-database command to upgrade the database from an earlier release to Oracle Database release 19c.

For a Oracle Database 19c database, update the database to Oracle Database 19.17 or later.

Failure to restore Oracle ASR when running the odacli restore-node -g command

Scenario: If Oracle ASR configuration fails during the restore-node operation, then the restore-node job displays the status as Success but the Oracle ASR configuration task status displays the status as Failure. To verify if the Oracle ASR configuration was restored successfully, check the describe-job output of the restore-node job. Following is a sample job for the restore-node operation:

Registering ASR Manager                  December 12, 2023 6:51:09 AM UTC         December 12, 2023 6:51:17 AM UTC         Failure                                                                                                                          
ASR service creation                     December 12, 2023 6:51:55 AM UTC         December 12, 2023 6:51:56 AM UTC         Failure  
Registering Asset: ODA Host              December 12, 2023 6:51:55 AM UTC         December 12, 2023 6:51:56 AM UTC         Failure  
ASR service creation                     December 12, 2023 6:51:56 AM UTC         December 12, 2023 6:51:56 AM UTC         Failure  
ASR assets activation                    December 12, 2023 6:51:56 AM UTC         December 12, 2023 6:51:56 AM UTC         Failure

Check the /opt/oracle/dcs/log/dcs-agent.log file and the Oracle ASR logs in the /var/opt/asrmanager/log location to determine the cause of the failure for Oracle ASR configuration. Once the cause and the resolution are known, Oracle ASR must be configured manually using the odacli configure-asr command after the restore-node job completes.

Cause: If Oracle ASR restoration fails due to an incorrect SSO password, the following error message may be displayed in the dcs-agent.log file.

An Oracle Single Sign On (OSSO) account is required for data submission.
If you do not have an account or have forgotten your username or
password,
 ******** http://support.oracle.com
 
Username []: asr-qa_ca@oracle.com
Password: ********
Password ******** (to verify):
 
Contacting transport servers. Please wait...
 
Checking connection to https://transport.oracle.com/v1/
Connection is ok. Trying to register client.
Error: Invalid Oracle SSO Username and/or Password. ********
 
 
Registration failed.
2023-12-11 10:11:09,259 DEBUG [Registering ASR Manager : JobId=cfc269c4-211a-4297-b363-a2ac65aa65b0] [] c.o.d.c.n.MessageUtil: load locale as en_US
2023-12-11 10:11:09,260 ERROR [Registering ASR Manager : JobId=cfc269c4-211a-4297-b363-a2ac65aa65b0] [] c.o.d.a.r.s.a.AsrOperations: Exception:
com.oracle.dcs.commons.exception.DcsException: DCS-10045:Validation error encountered: Registration failed : Error: Invalid Oracle SSO Username and/or Password.

Action Required: Use the correct SSO password and retry configuring Oracle ASR using the odacli configure-asr command.

Scenario: If Oracle ASR restoration fails due to connectivity issues with the transport server of the Oracle ASR Manager the following error message may be displayed in the dcs-agent.log file

2023-12-06 15:08:46,839 DEBUG [Registering ASR Manager : JobId=b4e25721-3a4f-4650-9ab9-ceefac678627] [] c.o.d.c.u.CommonsUtils: Output :
spawn /opt/asrmanager/bin/asr register
 
1) transport.oracle.com
Select destination transport server or enter full URL for alternate server [1]:
1
 
If a proxy server is required for HTTPS communication to the internet,
 
enter the information below. If no proxy is needed,
 enter -
Proxy server name []:
 
An Oracle Single Sign On (OSSO) account is required for data submission.
If you do not have an account or have forgotten your username or
password,
 ******** http://support.oracle.com
 
Username []: asr-qa_ca@oracle.com
Password: ********
Password ******** (to verify):
 
Contacting transport servers. Please wait...
  
Registration failed.

Action Required: Retry registration of Oracle ASR registration After the restore-node job completes, configure Oracle ASR manually using the odacli configure-asr command.

Failure during restore of KVM and DB systems

Scenario: Restore of metadata of VMs may fail due to missing CPU pool. The dcs-agent.log file may display the following error:

ERROR [Restore VMs metadata : JobId=bae05eea-27f1-4ccc-b962-6f071d5d90d3] 
[] c.o.d.a.k.e.KvmExceptionFactory: Not found by name com.oracle.dcs.commons.exception.DcsException: 
DCS-10032:Resource of type 'CPU Pool' with name 'pool_59c70ac2-' is not found.

Action Required: Run the odacli restore-node -kvm command to restore the missing CPU pool.

Scenario: When you run the odacli restore-node -d command, during restoration of databases, there may be an error in restoring the backup configuration.

Cause: Restore of the backup configuration may have failed for because of inaccessibility of NFS location, changed Objectstore password, and others.

Action Required: Create the backup configuration using the odacli create-backupconfig command and also attach it to the database using the odacli modify-database command, if required. If you encounter errors when you run the odacli restore-node -kvm command, then run the command again to restore the missing resources.

Scenario: Restore of metadata of VMs may fail due to missing vnetwork. The dcs-agent.log file may display the followign error:

ERROR [Restore VMs metadata : JobId=7776e6ad-b8c5-4e23-a72c-fb2d0b82fda3] 
[] c.o.d.a.k.e.KvmExceptionFactory: Not found by name com.oracle.dcs.commons.exception.DcsException:
 DCS-10032:Resource of type 'Virtual Network' with name 'vnet48777' is not found.

Action Required: Run the odacli restore-node -kvm command again to restore the missing vnetwork.

Errors related to Oracle Data Guard

Cause of the failure: If Oracle Data Guard is configured on the system and the Oracle Data Guard configuration has errors or warnings, then the precheck displays these error or warnings.

Dataguard FAILED 
Warning: ORA-16853: apply lag has Make sure that dataguard is in exceeded specified threshold. Make sure that dataguard is in 'CONFIGURED' state.

Resolution: Oracle Data Guard must be in Configured state. You must fix all warnings or errors that are displayed in the precheck to move the Oracle Data Guard configuration to the Configuration state.

Checks related to Muti User Access enabled environments

Scenario: Token expiration duration is out of range (> 600 mins or < 10 mins)

Cause: You may have manually edited the token expiration duration in the file at /opt/oracle/dcs/idm/idm.conf.

Action Required: The pre-upgrade report that is generated prior to detach checks for this anomaly and displays an error message and a resolution for the same. Edit the idm.conf file as the root user and correct the value of token expiration so that it is in range. This applies to other configuration settings too.

Scenario: The odacli restore-node -g command may fail with an error message about inconsistent state of the system.

Cause: The state of the system prior to the detach operation has a different multi-user access setting from when you ran the odacli restore-node -g command. This can happen if you accessed the BUI before running the odacli restore-node -g command and chose to enable or disable multi-user access.

Action Required: Perform a Data Preserving Reprovisioning cleanup of the system and then run the odacli restore-node -g command using ODACLI or BUI.

Scenario: The odacli restore-node -g command may display an error about a UID or GID conflict.

Cause: When running the odacli restore-node -g command, all users in the system are restored with their original UID or GID. If there is a conflict with an existing user or group, then the odacli restore-node -g command operation fails.

Action Required: Change the UID or GID of the conflicting user or group.

Sample Pre-Upgrade Checks Report

Sample output from a system when running the pre-upgrade checks.

# odacli create-preupgradereport
------------------------------------------------------------------------
                 Job ID:  e73f3d0f-8e77-40a1-92cc-2dc825c3fd28
            Description:  Run pre-upgrade checks for Bare Metal
                 Status:  SUCCESS
                Created:  December 12, 2023 12:43:13 PM GMT
                 Result:  All pre-checks succeeded

Node Name       
---------------
scaoda703c1n1 

Check                          Status   Message                                Action                                
------------------------------ -------- -------------------------------------- --------------------------------------
__GI__ 
Check presence of databases    Success  No additional database found           None                                  
not managed by ODA                      registered in CRS                                                            
Check custom filesystems       Success  All file systems are owned and used    None                                  
                                        by OS users provisioned by ODA                                               
Check presence of HAVIP        Success  No HAVIP resources found registered    None                                  
resources not managed by ODA            in CRS                                                                       
Check presence of export       Success  No EXPORT resources found registered   None                                  
resources not managed by ODA            in CRS                                                                       

__OS__ 
Check Required OS files        Success  All the required files are present     None                                  
Check Additional OS RPMs       Success  No RPMs outside of base ISO were       None                                  
                                        found on the system                                                          

__STORAGE__ 
Check Required Storage files   Success  All the required files are present     None                                  
Validate OAK Disks             Success  All OAK disks are in valid state       None                                  
Validate ASM Disk Groups       Success  All ASM disk groups are in valid state None                                  
Validate ASM Disks             Success  All ASM disks are in valid state       None                                  
Check Database Home Storage    Success  The volume(s)                          None                                  
volumes                                 orahome_sh,odabase_n0,odabase_n1                                             
                                        state is CONFIGURED.                                                         
Check space under /opt         Success  Free space on /opt: 189495.58 MB is    None                                  
                                        more than required space: 1024 MB                                            
Check space in ASM disk        Success  Space required for creating local      None                                  
group(s)                                homes is present in ACFS database                                            
                                        home storage. Required: 0 GB                                                 
                                        Available: 774 GB                                                            

__SYS__ 
Validate Hardware Type         Success  Current hardware is supported          None                                  
Validate ILOM interconnect     Success  ILOM interconnect is not enabled       None                                  
Validate System Version        Success  System version 19.21.0.0.0 is          None                                  
                                        supported                                                                    
Verify System Timezone         Success  Succesfully verified the time zone     None                                  
                                        file                                                                         
Verify Grid User               Success  Grid user is verified                  None                                  
Verify Grid Version            Success  Oracle Grid Infrastructure is running  None                                  
                                        on the '19.18.0.0.230117' version on                                         
                                        all nodes                                                                    
Check Audit Files              Success  Local Audit files not found            None                                  

__DB__ 
Validate Database Status       Success  Database 'mydb' is running and is in   None                                  
                                        'CONFIGURED' state                                                           
Validate Database Version      Success  Version '19.18.0.0.230117' for         None                                  
                                        database 'mydb' is supported                                                 
Validate Database Datapatch    Success  Database 'mydb' is completely applied  None                                  
Application Status                      with datapatch                                                               
Validate TDE wallet presence   Success  Database 'mydb' is not TDE enabled.    None                                  
                                        Skipping TDE wallet presence check.                                          
Validate Database Home         Success  Database home location check passed    None                                  
location                                for database mydbu                                                           
Validate Database Status       Success  Database 'uxljY' is running on         None                                  
                                        'scaoda703c1n2'. This check is                                               
                                        skipped.                                                                     
Validate Database Version      Success  Version '19.18.0.0.230117' for         None                                  
                                        database 'uxljY' is supported                                                
Validate Database Datapatch    Success  The database is RACOne and is running  None                                  
Application Status                      on scaoda703c1n2. This check is                                              
                                        skipped.                                                                     
Validate TDE wallet presence   Success  Database 'uxljY' is not TDE enabled.   None                                  
                                        Skipping TDE wallet presence check.                                          
Validate Database Home         Success  Database home location check passed    None                                  
location                                for database uxljY                                                           

__CERTIFICATES__ 
Check using custom             Success  Using Default key pair                 None                                  
certificates                                                                                                         
Check the agent of the DB      Success  All the agents of the DB systems are   None                                  
System accessible                       accessible                                                                   

__DBSYSTEMS__ 
Validate DB System DCS         Success  scaoda703c4n1: SUCCESS                 None                                  
component versions                                                                                                   


Node Name       
---------------
scaoda703c1n2 

Check                          Status   Message                                Action                                
------------------------------ -------- -------------------------------------- --------------------------------------
__GI__ 
Check presence of databases    Success  No additional database found           None                                  
not managed by ODA                      registered in CRS                                                            
Check custom filesystems       Success  All file systems are owned and used    None                                  
                                        by OS users provisioned by ODA                                               
Check presence of HAVIP        Success  No HAVIP resources found registered    None                                  
resources not managed by ODA            in CRS                                                                       
Check presence of export       Success  No EXPORT resources found registered   None                                  
resources not managed by ODA            in CRS                                                                       

__OS__ 
Check Required OS files        Success  All the required files are present     None                                  
Check Additional OS RPMs       Success  No RPMs outside of base ISO were       None                                  
                                        found on the system                                                          

__STORAGE__ 
Check Required Storage files   Success  All the required files are present     None                                  
Validate OAK Disks             Success  All OAK disks are in valid state       None                                  
Validate ASM Disk Groups       Success  All ASM disk groups are in valid state None                                  
Validate ASM Disks             Success  All ASM disks are in valid state       None                                  
Check Database Home Storage    Success  The volume(s)                          None                                  
volumes                                 orahome_sh,odabase_n0,odabase_n1                                             
                                        state is CONFIGURED.                                                         
Check space under /opt         Success  Free space on /opt: 131591.74 MB is    None                                  
                                        more than required space: 1024 MB                                            
Check space in ASM disk        Success  Space required for creating local      None                                  
group(s)                                homes is present in ACFS database                                            
                                        home storage. Required: 0 GB                                                 
                                        Available: 774 GB                                                            

__SYS__ 
Validate Hardware Type         Success  Current hardware is supported          None                                  
Validate ILOM interconnect     Success  ILOM interconnect is not enabled       None                                  
Validate System Version        Success  System version 19.21.0.0.0 is          None                                  
                                        supported                                                                    
Verify System Timezone         Success  Succesfully verified the time zone     None                                  
                                        file                                                                         
Verify Grid User               Success  Grid user is verified                  None                                  
Verify Grid Version            Success  Oracle Grid Infrastructure is running  None                                  
                                        on the '19.18.0.0.230117' version on                                         
                                        all nodes                                                                    
Check Audit Files              Success  Local Audit files not found            None                                  

__DB__ 
Validate Database Status       Success  Database 'mydb' is running and is in   None                                  
                                        'CONFIGURED' state                                                           
Validate Database Version      Success  Version '19.18.0.0.230117' for         None                                  
                                        database 'mydb' is supported                                                 
Validate Database Datapatch    Success  Database 'mydb' is completely applied  None                                  
Application Status                      with datapatch                                                               
Validate TDE wallet presence   Success  Database 'mydb' is not TDE enabled.    None                                  
                                        Skipping TDE wallet presence check.                                          
Validate Database Home         Success  Database home location check passed    None                                  
location                                for database mydbu                                                           
Validate Database Status       Success  Database 'uxljY' is running and is in  None                                  
                                        'CONFIGURED' state                                                           
Validate Database Version      Success  Version '19.18.0.0.230117' for         None                                  
                                        database 'uxljY' is supported                                                
Validate Database Datapatch    Success  Database 'uxljY' is completely         None                                  
Application Status                      applied with datapatch                                                       
Validate TDE wallet presence   Success  Database 'uxljY' is not TDE enabled.   None                                  
                                        Skipping TDE wallet presence check.                                          
Validate Database Home         Success  Database home location check passed    None                                  
location                                for database uxljY                                                           

__CERTIFICATES__ 
Check using custom             Success  Using Default key pair                 None                                  
certificates                                                                                                         
Check the agent of the DB      Success  All the agents of the DB systems are   None                                  
System accessible                       accessible                                                                   

__DBSYSTEMS__ 
Validate DB System DCS         Success  scaoda703c4n1: SUCCESS                 None                                  
component versions

Parent topic: Troubleshooting Oracle Database Appliance

Viewing Oracle Database Appliance Error Correlation Reports

Understand how to view Error Correlation Report and how to interpret the report to troubleshoot your appliance.

About Error Correlation Reports

If a DCS job fails, an Error Correlation job is created automatically to generate an Error Correlation report. You can access and review the generated Error Correlation report from the BUI to explore possible ways of error resolution.

The Error Correlation Report contains the following:

Log Messages: Errors, exceptions and warnings from various log files.
Failed Task Messages: Error message displayed when the DCS job failed.
Release Notes: Relevant Known Issues from Oracle Database Appliance Release Notes to help resolve the issue.
Documentation: Relevant topics from the Oracle Database Appliance Documentation Library to help resolve the error.

The Error Correlation Report is generated for every failed DCS job and can be accessed from the BUI. On Oracle Database Appliance high-availability deployments, the Error Correlation report contains the error information derived from log files of both the nodes.

Viewing Error Correlation Reports using ODACLI Commands

You can view the Error Correlation report of a failed DCS job by running the odacli describe-job -i failed_dcs_job_id --ecr command. For an example output, see the topic odacli describe-job in this guide.

Viewing Error Correlation Reports from the BUI

To view the Error Correlation Report from the Activities page in the BUI:

Log into the Browser User Interface:
```
https://host-ip-address:7093/mgmt/index.html
```
Click the Activity tab.
In the Activities page, click the Failure or InternalError link in the failed DCS job for which you want to view the Error Correlation report. Note that only failed DCS jobs have associated Error Correlation Reports.
You can also view the Error Correlation Report for the failed DCS job when you click the Actions menu, and select View Error Correlation Report.
The Error Correlation Report contains the following tabs:
- Log Messages: Displays the logs for DCS agent, DCS admin, Oracle HAMI, MySQL, and Oracle FPP. You can expand each section to view the details. Only components that have logs are displayed. If no errors are found, then the message No errors or exceptions found in logs is displayed in the Log Messages section.
- Failed Task Messages: Displays the specific error message displayed when the task failed.
- Release Notes: Displays relevant Known Issues from Oracle Database Appliance Release Notes to help resolve the issue. You can click each of these links to view the Release Notes entry. If no relevant Known issues are found, then the message No matching results were found. is displayed.
- Documentation: Displays relevant topics from the Oracle Database Appliance Documentation Library to help resolve the error. You can click each of these links to view the documentation topic from the Oracle Database Appliance documentation.

To view the Error Correlation Report from the Diagnostics page in the BUI:

In the BUI, click the Diagnostics tab.
In the Diagnostics page, click Collect Diagnostic Data for a failed job.
The Collect Diagnostics page displays the Error Correlation Report and Job Details in separate tabs for the failed DCS job. Click the Report File Name link to download the Error Correlation Report to your local system.
The Job details tab displays the steps in the job and the Error Correlation Report contains the Log Messages, Failed Task Messages, Release Notes, and Documentation tabs.

odacli describe-job

Parent topic: Troubleshooting Oracle Database Appliance

About Enabling Linux Kernel Core Extractor for Troubleshooting

Understand how to manage Linux Kernel Core Extractor to troubleshoot your appliance.

About Linux Kernel Core Extractor

A Linux kernel panic can occur due to various reasons such as faulty hardware, driver crashes, or software bugs. To identify the cause of kernel panic, it is essential to collect and analyze the vmcore of the crashed kernel. The kdump service is used to collect the vmcore after the first kernel crash. This process is slow for systems with high memory and often fails to generate vmcore when the available space is not sufficient. When Linux Kernel Core Extractor is enabled on Oracle Database Appliance bare metal systems, the crash utility in the kdump kernel collects useful information for troubleshooting without generating vmcore.

Linux Kernel Core Extractor Commands

List generated crash reports:

# /usr/sbin/oled lkce list
Followings are the crash*out found in /var/oled/lkce dir:
/var/oled/lkce/crash_20220307-154542.out

Purge existing all but last three crash reports:

# /usr/sbin/oled lkce clean
lkce deletes all but last three /var/oled/lkce/crash*out files. do you want to proceed(yes/no)? [no]:

Purge all crash reports:

# /usr/sbin/oled lkce clean --all
lkce removes all the files in /var/oled/lkce dir. do you want to proceed(yes/no)? [no]:

By default, the crash report contains output for the following crash commands. You can add other crash commands to the /etc/oled/lkce/crash_cmds_file.

#
# This is the input file for crash utility. You can edit this manually
# Add your own list of crash commands one per line.
#
bt
bt -a
bt -FF
dev
kmem -s
foreach bt
log
mod
mount
net
ps -m
ps -S
runq
quit

By default, vmcore generation is disabled. You can enable vmcore generation as follows:

# oled lkce configure --vmcore=yes
Restarting kdump service... done!
lkce: set vmcore to yes

For additional Linux Kernel Core Extractor commands, refer to the Linux Kernel Core Extractor help.

# oled lkce help
Usage: lkce options
options:
    report report-options -- Generate a report from vmcore
    report-options:
        --vmcore=/path/to/vmcore        - path to vmcore
        [--vmlinux=/path/to/vmlinux]        - path to vmlinux
        [--crash_cmds=cmd1,cmd2,cmd3,..]    - crash commands to include
        [--outfile=/path/to/outfile]        - write output to a file
 
    configure [--default]   -- configure lkce with default values
    configure [--show]  -- show lkce configuration -- default
    configure [config-options]
    config-options:
        [--vmlinux_path=/path/to/vmlinux]   - set vmlinux_path
        [--crash_cmds_file=/path/to/file]   - set crash_cmds_file
        [--kdump_report=yes/no]         - set crash report in kdump kernel
        [--vmcore=yes/no]           - set vmcore generation in kdump kernel
        [--max_out_files=<number>]        - set max_out_files
 
    enable  -- enable lkce in kdump kernel
    disable -- disable lkce in kdump kernel
    status  -- status of lkce
 
    clean [--all]   -- clear crash report files
    list        -- list crash report files

Parent topic: Troubleshooting Oracle Database Appliance

Viewing Details About DCS Error Messages

Understand how to view details about DCS errors for troubleshooting them.

About Viewing Information About DCS Errors

To view more details about any errors during DCS operations, use the command dcserr error_code.

# /opt/oracle/dcs/bin/dcserr
dcserr error_code
 
# dcserr 10001
10001, Internal_Error, "Internal error encountered: {0}."
// *Cause: An internal error occurred.
// *Action: Contact Oracle Support Services for assistance.
/
# dcserr 1001
Unknown error code

To view more details about DCS errors in the Browser User Interface (BUI), you can provide the DCS error code in the Search box in the BUI. The Search results display the Cause and Action of DCS error codes.

Parent topic: Troubleshooting Oracle Database Appliance

Collecting Diagnostics Data Using the BUI

Understand how to collect diagnostics data to troubleshoot errors.

About Collecting Diagnostics Data

Use the Diagnostics tab in the Browser User Interface to view diagnostic information about your deployment and the installed components.

In the Diagnostic Collection page, you can view the available diagnostics collections. Click Collect Diagnostic Data to start diagnostics collection. Once the data is collected, click on the collection file path to download the file.

In the Collect Diagnostics page, specify the Job ID for the diagnostics data collection. Optionally, specify a tag and a description for the collection. The details of the Job ID are displayed. Click Collect to start the diagnostics data collection.

You can also collect diagnostics from the Activity page, by selecting Collect Diagnostics from the Actions drop down for a specific job. Click Collect to start the diagnostics data collection.

To delete a diagnostic collection, from the Diagnostic Collection page, select the specific collection, and click Delete.

This diagnostic collection feature does not replace the odaadmcli manage diagcollect command. You can use the odaadmcli manage diagcollect command also to enable diagnostics collections, independently of this new feature from the BUI. The odaadmcli manage diagcollect command and the diagnostics collection from BUI use the tfactl command internally. The diagnostics collection from BUI is aimed to collect other data from DCS metadata that is not collected through tfactl and provide greater context for root cause analysis of related DCS jobs failures.

Parent topic: Troubleshooting Oracle Database Appliance

Resolving Errors When Updating DCS Components During Patching

Understand how to troubleshoot errors when updating DCS components during patching.

.

About DCS Components

When you run the odacli update-dcscomponents command during patching, pre-checks for MySQL installation are automatically verified before update of Oracle HAMI, MySQL, and DCS components. If any of the pre-checks fail, then the command errors out with a reference to the pre-check report log file location /opt/oracle/dcs/log/jobfiles/jobId/dcscomponentsPreCheckReport.log. Review the pre-check report and take corrective actions and then rerun the odacli update-dcscomponents command. If there are no pre-check errors, then the patching process proceeds with updating Oracle HAMI, MySQL, and DCS components such as the DCS Agent, DCS CLI, and DCS Controller.

Note:

Run the

odacli
                        update-dcsadmin

command prior to running the

odacli
                        update-dcscomponents

command.

When the odacli update-dcscomponents command completes successfully:

The command output is as follows:

# ./odacli update-dcscomponents -v 19.23.0.0.0            
{
  "jobId" : "3ac3667a-fa22-40b6-a832-504a56aa3fdc",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s):DCS-Agent
shutdown is successful. MySQL upgrade is done before. Metadata migration is
successful. Agent rpm upgrade is successful. DCS-CLI rpm upgrade is successful.
DCS-Controller rpm upgrade is succ",
  "reports" : null,
  "createTimestamp" : "April 8, 2024 02:37:37 AM CST",
  "description" : "Update-dcscomponents job completed and is not part of Agent
job list",
  "updatedTime" : "April 8, 2024 02:39:10 AM CST"
}

The pre-check report log file at the location /opt/oracle/dcs/log/jobfiles/jobId/dcscomponentsPreCheckReport.log contains the following:

Pre-check Name: Space check
Status: Success
Comments: Required space 3 GB is available in /opt

Pre-check Name: Port check
Status: Success
Comments: Port 3306 is available for running ODA MySQL

Pre-check Name: ODA MySQL rpm installation dry-run check
Status: Success
Comments: ODA MySQL rpm dry-run passed

Pre-check Name: Check for the existence of MySQL connector/J library
Status: Success
Comments: ODA MySQL connector/J library found

Pre-check Name: Check for the existence of Metadata migration utility
Status: Success
Comments: Metadata migration utility found

dcs-admin version: 
Pre-check Name: dcs-admin version validation
Status: Success
Comments: dcs-admin is already updated :19.23.0.0.0

Config File Exist dcscontroller: 
Pre-check Name: Check DCS config files exists for dcscontroller
Status: Success
Comments: Files /opt/oracle/dcs/conf/dcs-controller.yml and /opt/oracle/dcs/conf/dcs-controller-logback.xml exist
 
Config File Exist dcsagent: 
Pre-check Name: Check DCS config files exists for dcsagent
Status: Success
Comments: Files /opt/oracle/dcs/conf/dcs-agent.yml and /opt/oracle/dcs/conf/dcs-agent-logback.xml exist

Validate scheduler cron expressions:
Pre-check Name: Validate scheduler cron expressions
Status: Success
Comments: Scheduler cron expressions for existing job schedules are up to date

When the odacli update-dcscomponents command fails:

On Oracle Database Appliance single-node systems, the command output is as follows:

# ./odacli update-dcscomponents -v 19.23.0.0.0            

DCS-10008:Failed to update DCScomponents: 19.22.0.0.0
Internal error while patching the DCS components :
DCS-10231:Cannot proceed. Pre-checks for update-dcscomponents failed. Refer to
/opt/oracle/dcs/log/jobfiles/jobId/dcscomponentsPreCheckReport.log
on node 0 for details.

On Oracle Database Appliance high-availability systems, the command output is as follows:

# ./odacli update-dcscomponents -v 19.23.0.0.0            

Internal error while patching the DCS components :
DCS-10231:Cannot proceed. Pre-checks for update-dcscomponents failed. Refer to
/opt/oracle/dcs/log/jobfiles/jobId/dcscomponentsPreCheckReport.log
on node 0 and /opt/oracle/dcs/log/jobfiles/jobId/dcscomponentsPreCheckReport.log
on node 1 for details.

The command runs all pre-checks one by one, and errors out at the end if any of the pre-checks is marked as Failed. When a pre-check fails, the error message is displayed on to the console along with the reference to pre-check report log location. The pre-check report log file is at the location /opt/oracle/dcs/log/jobfiles/jobId/dcscomponentsPreCheckReport.log.

Pre-check Name: Space check
Status: Failed
Comments: Available space in /opt is 2 GB but minimum required space in /opt is 3 GB 

Pre-check Name: Port check
Status: Success
Comments: Port 3306 is available for running ODA MySQL

Pre-check Name: ODA MySQL rpm installation dry-run check
Status: Success
Comments: ODA MySQL rpm dry-run passed

Pre-check Name: Check for the existence of MySQL connector/J library
Status: Success
Comments: ODA MySQL connector/J library found

Pre-check Name: Check for the existence of Metadata migration utility
Status: Success
Comments: Metadata migration utility found

Pre-check Name: dcs-admin version validation
Status: Success
Comments: dcs-admin is already updated :19.20.0.0.0

Config File Exist dcscontroller: 
Pre-check Name: Check DCS config files exists for dcscontroller
Status: Success
Comments: Files /opt/oracle/dcs/conf/dcs-controller.yml and /opt/oracle/dcs/conf/dcs-controller-logback.xml exist
 
Config File Exist dcsagent: 
Pre-check Name: Check DCS config files exists for dcsagent
Status: Success
Comments: Files /opt/oracle/dcs/conf/dcs-agent.yml and /opt/oracle/dcs/conf/dcs-agent-logback.xml exist

Validate scheduler cron expressions:
Pre-check Name: Validate scheduler cron expressions
Status: Success
Comments: Scheduler cron expressions for existing job schedules are up to date

When the odacli update-dcscomponents command fails due to space check error: