8 Troubleshoot Your OMCe Service

Here’s where you’ll find information to help you resolve any problems you come across.

It’s likely that at some point when you’re creating and maintaining your OMCe service, you’ll run into issues or just have questions about the proper way to proceed. Review the information presented here to see if solutions to your issues or questions are addressed and where to go to get help if they aren’t.

Where Do I Get Help?

You can get assistance if you need help resolving an issue with your OMCe service by phone or using chat. On the Oracle Cloud My Service dashboard, or from the Stack console or Service console, click to see the phone number and the Chat link.

Using Diagnostic Tools

If you contact Oracle Support they may ask you to download and run one of the diagnostic tools:

• Mobile Suite Doctor. See Using the OMCe Mobile Suite Doctor.

• Mobile Core Doctor. See Using the OMCe Mobile Core Doctor.

What Information Do I Need?

Before you contact Oracle Support, make sure you have your Customer Support Identifier (CSI) number, which is in the Welcome email. In addition, you should also have:

• The identity domain name which you can find on the Details tab of the Mobile Core POD IDCS app. See Use IDCS to Manage Users and Roles.

• The user name for OMCe.

• The OMCe stack name.

• Status information from My Services including the following. See Troubleshoot Mobile Suite Provisioning

  • Content of the Stack Create and Delete History.

  • Summary messages from the details page you see when you click the Status error link.

For more information about getting log messages when Oracle Support ask for them, see Getting Log Messages.

Troubleshoot Mobile Suite Provisioning

If stack creation fails, or one or more services fail when you are provisioning OMCe, use the Create and Delete histories available from the dashboard to try to identify where the problem is.

How to Recognise a Problem

The process of creating an OMCe stack can take a few hours. During this time you can see the status of different resources in the Stack Overview window:

• When a stack is being created, there is an hourglass icon and the status is a clickable link Creating Stack which you can use to get further information.

• When service creation fails, the relevant entry has an error icon and the status is Error. You can click on the status link to see more details in the Stack Create and Delete History pane.

  
  Description of the illustration stack-creation-failure.png

The service creation operation will clean up the failed service and its resources and retry service creation. It will do this twice, and if the failure has been caused by a temporary condition and if you wait you may see service creation succeed.

Service Creation Failure Analysis with the Stack Dashboard 

On the Oracle Cloud Stack dashboard, open the Stack Create and Delete History pane. Click Details to see if the list of status messages give you an idea of why provisioning failed.

Possible reasons are:

• Maybe you have insufficient resources

• Perhaps you entered invalid data when you provisioned OMCe

Description of the illustration stack-create-and-delete-history.png

Service Creation Failure Analysis with the Service Dashboard

For an individual Service, such as Mobile Core, Bots or CxA, you can do the same at the Service level by navigating to the individual Service dashboard and opening the Service Create and Delete History pane near the bottom of the Stack Overview. Click Details to see why provisioning failed.

Description of the illustration service-create-and-delete-history.png

Analytics Not Working

If there has been an issue during provisioning with passing the private key into the OMCe stack then Enrich, required by Analytics, may not be installed and so the Enrich Spark job will not run. You’ll be able to see this because of the following effects:

• Analytics appears to function, but there are no events displayed.

• When you check the Big Data Service Console and you can see that the Enrich Spark job is not running.

• The /tmp/id_rsa file doesn’t exist.

The solution is to perform the following tasks to input the private key, and to install Enrich and submit the Enrich Spark job.

Input the Private Key

• Copy the private key to the Analytics Pod Service VM:

  ssh -i privatekey opc@{PUBLIC_IP_CXAPOD_VM} 
  

  Create the file /tmp/id_rsa and paste in private key content into it. Then execute the following:

  chmod 600 id_rsa 
  chown oracle:oracle id_rsa

Install Enrich and Submit the Enrich Spark Job

1. Find the Analytics Pod Service VM’s IP address.

   1. From OMCe Stack Overview, click the resource with the name <stack>CXAPOD. See View Your OMCe Stack.

   2. In the Service Overview for the Analytics Pod Service POD Details, expand Resources and note the Public IP, for example 192.0.2.1.

2. Connect to the Core VM using ssh and change to the oracle user. You can log in as the default user, opc. The opc user has sudo privileges.

   1. Use:

      $ ssh -i <private_key_file> opc@<CXAPOD IP address>

      For example, ssh -i my_private_rsa_keyfile opc@192.0.2.1

   2. Switch to the oracle user using:

      $ sudo su - oracle

3. Change directories by executing:

   cd ~/../../cxpd/Manager/service_scripts/provisioning_check/

   Look for a file in this directory called data_bag.json.  If it is present, continue with step 4.

   If the file doesn’t exist in this directory, contact Oracle Support Services.

4. Install Enrich and submit the Enrich Spark job by executing:

   python check_provisioning_status_rex.py

Troubleshoot Service Creation Failure

If service creation fails during provisioning there are a couple of methods you can use to try to diagnose the cause of the failure.

Service Creation Failure Analysis with Mobile Doctor Provisioning Log Zip

You can use the Mobile Suite Doctor tool to download the provisioning dump. See Using the OMCe Mobile Suite Doctor.

Alternatively, you can use the Mobile Core Doctor to produce diagnostic log files which you can use to try to diagnose the root cause of the failure. See Using the OMCe Mobile Core Doctor. If you cannot diagnose the problem, you can contact Oracle Support and attach the zip file to a service request.

For Mobile Core Service (MobileCorePOD), the Mobile Core Doctor diagnostic dump is automatically uploaded to the storage container specified when OMCe is provisioned. The file name has the form provisioning_logs/mbcp_postCreate_<timestamp>.zip.

Download the zip file using cURL or ftmcli (Oracle Storage Cloud Service - File Transfer Manager Client). You can expand the zip file and see if you can diagnose the cause of the failure.

For information about downloading and installing cURL, see Update Settings for the Storage Container.

Operation	Command Format	Sample Output
List Mobile Doctor Dumps	curl -X GET https://storage.oraclecorp.com/v1/<storage domain>/<container>?prefix=provisioning_logs -u <user>:<password>	provisioning_logs/mobdr_collection_20170621T065111.zip
Download Mobile Doctor Dump	curl -X GET https://storage.oraclecorp.com/v1/<storage domain>/<container>?prefix=provisioning_logs/<mobdr zip filename> -u <user>:<password> -o <mbcp zip filename>	provisioning_logs/mbcp_postCreate_20170621T065111.zip

Service Creation Failure Analysis directly on the VM 

You can SSH to the Mobile Core VM, Service Creation logs and other files helpful in diagnosing provisioning issues. They are available in the following locations:

• /u01/app/oracle/tools/paas/state/logs/rcu.results.txt

• /u01/app/oracle/tools/paas/state/logs/jaas.log

• /u01/app/oracle/tools/paas/state/logs/*_provisioning*

• /u01/app/oracle/tools/paas/state/logs/OraJCSmon

• /u01/app/oracle/tools/paas/state/work/jcs/OraJCSmon/records/JCSmon_*_domain_*

• /tmp/pod.postCreate.json

• /tmp/OraInstall*

• /tmp/wlstOfflineLogs_oracle

• /tmp/RCU*/logs/logger*.properties

• /tmp/RCU*/logs/rcu.log

• /tmp/mobile/patches/*/README.txt

• /tmp/wlstOfflineLogs_oracle/*.log

Troubleshoot Runtime Mobile Suite Issues

If you have a runtime failure, you can use the Mobile Suite Doctor tool to try to analyze the source of the problems. See Using the OMCe Mobile Suite Doctor.

• Query and download log files

• Do a full diagnostic dump at Suite level or Service level

Install the Mobile Suite Doctor, then follow these instructions to access the logs using the Oracle Storage Cloud Service (OSS) REST API. Log files are automatically uploaded to the OSS and stored as a series of zip files.

• You can query the set of log files and download them using the OSS REST API.

• You can trigger the upload of the latest logs to OSS using the PSM REST API

Troubleshoot Runtime Mobile Core Issues

If there are runtime failures in the Mobile Core, you can use the Mobile Core Doctor to try to diagnose the cause of the failure.

Runtime Failure Analysis using Mobile Core Doctor directly on the VM  

You can use the Mobile Core Doctor to produce diagnostic log files which you can use to try to diagnose the root cause of the failure. See Using the OMCe Mobile Core Doctor. If you cannot diagnose the problem, you can contact Oracle Support and attach the zip file to a service request.

For Mobile Core Service (MobileCorePOD), the Mobile Core Doctor diagnostic dump is automatically uploaded to the storage container specified when OMCe was provisioned. It has the name provisioning_logs/mobdr_collection_timestamp.zip.

Using the OMCe Mobile Suite Doctor

If you have to contact Oracle Support about a problem with OMCe they may ask you to use the Mobile Suite Doctor to collect useful diagnostic and configuration information from Mobile Suite logs and attach it to a service request.

Download and install the Mobile Suite Doctor

The Mobile Suite Doctor is part of the Admin tools zip which you can get from the OMCe Downloads page on OTN at http://www.oracle.com/technetwork/topics/cloud/downloads/mobile-suite-3636471.html. Save the zip to a local location and unzip it.

You have to have Python 3 installed on the same machine where you install and run Mobile Suite Doctor. You can get it from https://www.python.org/downloads/.

Note:

The Mobile Suite Doctor uses the executable python. If the version of Python 3 you have has the python3 executable you should either modify the omcedr.sh/omcedr.cmd scripts to use python3, or symlink python to the python3 executable.

On OSX, if you get an error SSL: CERTIFICATE_VERIFY_FAILED when you run the tool it is probably because Python 3.6 on OSX has no certificates and can’t validate any SSL connections. You can install the certifi package of certificates. See the ReadMe at /Applications/Python\ 3.6/ReadMe.rtf .

Using a Configuration File for Properties

The Mobile Suite Doctor uses information about the service and the Storage Service that it will run against. You can use a configuration file to pass these properties and their values to the tool. If you choose not to use a configuration file, you will be prompted for the values as the tool runs. You can choose to have some of the information in the configuration file and enter other values when prompted. This allows you to keep sensitive information such as passwords secure.

Create a JSON file to contain some or all of the following properties and values. The default name of the file is omcedr.json in the same directory as omcedr. You can choose to use another filename name or location using the command line option –omcedr_config_file.

Name	Description	Example
psmUser	The username for My Services and the Service Console	[e-mail]
psmPassword	The password for My Services and the Service Console	password
psmUri	The URI for My Services. Use the PSM URI for your datacenter region: psm.us.oraclecloud.com psm.europe.oraclecloud.com psm.aucom.oraclecloud.com psm.lad.oraclecloud.com	https://psm.us.oraclecloud.com
psmIdentityDomain	The identity domain	idcs-e8917a96fe5464d9a893eb90be983f18
storageUser	Name of the Storage user	dae78aaf9b48be68a1250b249f5af383
storagePassword	Password for the Storage user	password
storageUri	Mobile Suite Container Storage URI	https://storage.oraclecorp.com/v1/Storage-dae78aaf9b48be68a1250b249f5af383/OmceContainer

omcedr Log File

The Mobile Suite Doctor generates its own log file, omcedr.log, that contains information and error messages generated by the tool. Mobile Suite Doctor can also generate detailed logs using the command line option -ll to use DEBUG level.

omcedr

Use to see the full syntax and help for running each of the command options.

Syntax

omcedr.sh

Parameters

All parameters are required unless otherwise noted.

Parameter	Description
--help	Displays the full help for the Mobile Suite Doctor. By default only the syntax is displayed

omcedr list_logs

Use to get a list of available logs either for the entire Mobile Suite or a single Service.

Syntax

omcedr.sh list_logs -s|--stack stackName [-n|--service_name serviceName] [-ft|--from_time fromTime] [-tt|--to_time toTime] [-f|--omcedr_config_file configFile] [-ll|--log_level logLevel]

Parameters

All parameters are required unless otherwise noted.

Parameter	Description
-s|–stack stackName	Mobile Suite stack name
-n|–service_name serviceName	(Optional) Mobile Suite service name
-ft|–from_time fromTime	(Optional) Start time period to query the list of logs, based on the log file last update time. Default 24 hours ago. Date format: YYYY-MM-DD HH:MM (e.g. 2017-07-26 20:04)
-tt|–to_time toTime	(Optional) End time period to query the list of logs, based on the log file last update time. Default is the current date and time. Date format: YYYY-MM-DD HH:MM (e.g. 2017-07-27 20:04)
-f|–omcedr_config_file configFile	(Optional) Configuration file
-ll|-log_level logLevel	(Optional) Log level to set for the tool's internal logging (WARNING, INFO, DEBUG). Default is INFO.

omcedr push_logs

Use to force the ACCS logs to be pushed to OSS for the entire Mobile Suite or a single Service.

Syntax

omcedr.sh push_logs -s|--stack stackName [-n|--service_name serviceName] [-f|--omcedr_config_file configFile] [-ll|--log_level logLevel]

Parameters

All parameters are required unless otherwise noted.

Parameter	Description
-f|–omcedr_config_file configFile	(Optional) Configuration file
-ll|-log_level logLevel	(Optional) Log level to set for the tool's internal logging (WARNING, INFO, DEBUG). Default is INFO.
-n|–service_name serviceName	(Optional) Mobile Suite service name
—s|–stack stackName	Mobile Suite stack name

omcedr download_logs

Use to download logs either for the entire Mobile Suite or a single Service. It is also possible to download a named single file. The downloaded logs will be written to a temporary directory.

Syntax

omcedr.sh download_logs -s|--stack stackName [-n|--service_name serviceName] [-ft|--from_time fromTime] [-tt|--to_time toTime] [-f|--omcedr_config_file configFile] [-l|--log logFilename] [-od|-output_directory] [-ll|--log_level logLevel]

Parameters

All parameters are required unless otherwise noted.

Parameter	Description
-s|–stack stackName	Mobile Suite stack name
-n|–service_name serviceName	(Optional) Mobile Suite service name
-ft|–from_time fromTime	(Optional) Start time period to download logs, based on the log file last update time. Default 24 hours ago. Date format: YYYY-MM-DD HH:MM (e.g. 2017-07-26 20:04)
-tt|–to_time toTime	(Optional) End time period to download logs, based on the log file last update time. Default is the current date and time. Date format: YYYY-MM-DD HH:MM (e.g. 2017-07-27 20:04)
-l|–log logFilename	(Optional) Full path and name of log file to download
-f|–omcedr_config_file configFile	(Optional) Configuration file with the PSM and OSS settings
-ll|-log_level logLevel	(Optional) Log level to set for the tool's internal logging (WARNING, INFO, DEBUG). Default is INFO.
-od|-output_directory outputDirectory	(Optional) Directory to write the logs to. Default is the system's "temp" directory.

omcedr check_health

Use this command to perform a health check for a named service.

Syntax

omcedr.sh check_health -n|--service_name serviceName -t|--service_type serviceType [-f|--omcedr_config_file configFile] [-ll|--log_level logLevel]

Parameters

All parameters are required unless otherwise noted.

Parameter	Description
-n|–service_nameserviceName	Mobile Suite service name
-t| --service_typeserviceType	Mobile Suite service type (e.g. MobileCorePOD)
-f|–omcedr_config_fileconfigFile	(Optional) Configuration file with the PSM and OSS settings
-ll|-log_level logLevel	(Optional) Log level to set for the tool's internal logging (WARNING, INFO, DEBUG). Default is INFO.

omcedr dump

Use to perform a diagnostic dump for the entire Mobile Suite or a single Service. The output from running this command is a zip file containing the captured diagnostics information.

Syntax

omcedr.sh dump -s|--stack stackName [-n|--service_name serviceName] [-ft|--from_time fromTime] [-tt|--to_time toTime] [-pd|--provisioning_dumps] [-f|--omcedr_config_file configFile] [-od|--output_directory outputDirectory] [-ll|--log_level logLevel]

Parameters

All parameters are required unless otherwise noted.

Parameter	Description
-s|–stack stackName	Mobile Suite stack name
-n|–service_name serviceName	(Optional) Mobile Suite service name
-ft|–from_time fromTime	(Optional) Start time period to download logs, based on the log file last update time. Default 24 hours ago. Date format: YYYY-MM-DD HH:MM (e.g. 2017-07-26 20:04)
-tt|–to_time toTime	(Optional) End time period to download logs, based on the log file last update time. Default is the current date and time. Date format: YYYY-MM-DD HH:MM (e.g. 2017-07-27 20:04)
-l|–log logFilename	(Optional) Full path and name of log file to download
-f|–omcedr_config_file configFile	(Optional) Configuration file with the PSM and OSS settings
-ll|-log_level logLevel	(Optional) Log level to set for the tool's internal logging (WARNING, INFO, DEBUG). Default is INFO.
-od|-output_directory outputDirectory	(Optional) Directory to write the logs to. Default is the system's "temp" directory.

omcedr stack_info

Use to get information from PSM about a Mobile Suite stack.

Syntax

omcedr.sh stack_info -s|--stack stackName [-f|--omcedr_config_file configFile] [-ll|--log_level logLevel]

Parameters

All parameters are required unless otherwise noted.

Parameter	Description
-s|–stack stackName	Mobile Suite stack name
-f|–omcedr_config_file configFile	(Optional) Configuration file with the PSM and OSS settings
-ll|-log_level logLevel	(Optional) Log level to set for the tool's internal logging (WARNING, INFO, DEBUG). Default is INFO.

omcedr service_info

Use to get information from PSM about a specific Mobile Suite service.

Syntax

omcedr.sh service_info -n|--service_name serviceName -t|--service_type serviceType [-f|--omcedr_config_file configFile] [-ll|--log_level logLevel]

Parameters

All parameters are required unless otherwise noted.

Parameter	Description
-n|–service_name serviceName	Mobile Suite service name
-t| --service_type serviceType	Mobile Suite service type (e.g. MobileCorePOD)
-f|–omcedr_config_file configFile	(Optional) Configuration file with the PSM and OSS settings
-ll|-log_level logLevel	(Optional) Log level to set for the tool's internal logging (WARNING, INFO, DEBUG). Default is INFO.

Using the OMCe Mobile Core Doctor

If you have to contact Oracle Support about a problem with OMCe they may ask you to use the Mobile Core Doctor to collect diagnostic information from the mobile core host VM and attach it to a service request.

Overview

You can run the Mobile Core Doctor on the mobile core host VM.

The Mobile Core Doctor also runs automatically at various points in the core service’s lifecycle. The output zip files are uploaded to the storage account and container associated with the service.

Lifecycle Point	Collections Performed	Cloud Storage Filename
Post-Create	local_provisioning, local_offline_runtime, wls_online_state, wls_online_health	provisioning_logs/mbcp_postCreate_<timestamp>.zip
Pre-Delete	local_provisioning, local_offline_runtime, wls_online_state, wls_online_health	provisioning_logs/mbcp_preDelete_<timestamp>.zip

The timestamp is of the form <YEAR><MONTH><DAY>T<HOUR><MIN><SECONDS>: (e.g. 20170911T105647).

Run the Mobile Core Doctor and Copy Log Files

If you have been asked by support to attach log files from the Mobile Core Doctor to a service request, follow the steps below.

1. Find the Core VM’s IP address.

   1. From OMCe Stack Overview, click the resource with the name <stack>CORE. See View Your OMCe Stack.

   2. In the Service Overview for the Mobile Core POD Details, expand Resources and note the Public IP, for example 192.0.2.1.

2. Connect to the Core VM using ssh and change to the oracle user. You can log in as the default user, opc. The opc user has sudo privileges.

   1. To make the private key not user-readable, use:

      chmod 600 <private_key_file>

   2. Log in using:

      $ ssh -i <private_key_file> opc@<Core IP address>

      For example, ssh -i my_private_rsa_keyfile opc@192.0.2.1

   3. Switch to the oracle user using:

      $ sudo su - oracle

3. Run the Mobile Core Doctor to generate the log file zip.

   1. Use cd /u01/app/oracle/suite/mcs/mobile_doctor.

   2. Run mobile_doctor.sh collect. You might be prompted to provide the username, password and URI for My Services. For details on commands and parameters, see Mobile Core Doctor Tool Options.

   A zip file is generated under /tmp. The name of the file is <prefix>_<timestamp>.zip, for example, /tmp/mobdr_collection_20170707T100806.zip

4. Copy the zip file from the Core VM using scp. You can use a tool such as pscp (which comes with PUTTY) which allows you to copy files from and to remote machines. Use:

   $ scp -i <private_key_file> opc@<Core IP address>:/tmp/file.zip file.zip

   where file.zip is the zip file generated in the previous step. For example:

   $ scp -i ~/.ssh/my_private_rsa_keyfile opc@192.0.2.1:/tmp/mobdr_collection_20170707T100806.zip mobdr_collection_20170707T100806.zip

   Note:

   You might need to change the commands above depending on the scp tool you use. For example, if you are using pscp you would start the command with $ pscp.

Mobile Core Doctor Tool Options

The Mobile Core Doctor Tool is located on the Mobile Core VM (/u01/app/oracle/suite/mcs/mobile_doctor/mobile_doctor.sh).

mobile_doctor.sh

When run without any options the tool will print information about the available options and their sub-options.

Syntax

mobile_doctor.sh

Parameters and Options

None

mobile_doctor.sh list_collections

Display the list of known collections and the summary description of each of them.

Syntax

mobile_doctor.sh list_collections

Parameters and Options

None

Details of the Collections

• local_offline_runtime

  This collection focuses on data that the WebLogic domain reads or writes to disk. It includes certain configuration files, log files, incidents and flight recordings. It tends to be the flight recordings (binary, don't compress well) that take up most space in the mobile doctor output, and the script might not collect them if they are too big.

• local_provisioning

  This collection focuses on the log files that result from provisioning mobile core components on the core host.

• wls_online_state

  This collection uses WLST to fetch various pieces of information from the WLS MBeans that represent the domain, servers, applications and data-sources hosted by that WLS instance.

• wls_online_health

  This collection uses WLST to inspect the runtime status of servers, applications and data-sources required to run mobile core.

mobile_doctor.sh collect

Collects the information and collates it in a zip file in the temporary directory on the host (typically /tmp). The name of the resulting zip file is of the form <prefix>_<timestamp>.zip, for example, /tmp/mobdr_collection_20170714T095656.zip.

You might be prompted to provide the username, password and URI for My Services.

Syntax

mobile_doctor.sh collect [maxsize=<size>] [fileprefix=<prefix>] [collection_names...]

Parameters and Options

Parameter/Option	Mandatory/Optional	Description
maxsize	Optional Default: 25M	The approximate maximum size of the zip file to be created. As the mobile doctor collects information it will constantly check the size of the zip file and predict if the next piece of content will push the zip file over the prescribed limit. If it will then that piece of content will not be recorded, but the tool will go on to try and collect other pieces of data.
fileprefix	Optional Default: mobdr_collection	Specifies the prefix of the resulting zip file
collection_names	Optional Default: local_offline_runtime	The names of one or more of the predefined collections.

Getting Log Messages

If you have to contact Oracle Support Services, they may ask you to download log messages to send to them. For example, Oracle Support may ask you to get the core and custom code logs. This section describes how to do this.

Which Tool to Use

There are two diagnostic tools

• Mobile Suite Doctor, which you use to get the MobileCCC, Bots, and CxA logs. See Using the OMCe Mobile Suite Doctor.

• Mobile Core Doctor, which you use to get the mobile logs. See Using the OMCe Mobile Core Doctor.

The procedures you should expect to run are these:

1. Install and run the Mobile Suite Doctor, omcedr, to collect MobileCCC, Bots, and CxA logs, described inUsing the OMCe Mobile Suite Doctor:

   1. Install the Mobile Suite Doctor.

   2. Define a configuration file.

   3. Run the download_logs command. Ensure that from_time and to_time are before and after the period of the issue that you are reporting so that any events will be captured in the messages.

2. Follow the procedure in Using the OMCe Mobile Core Doctor to use the Mobile Core Doctor to collect mobile logs.

   1. Use ssh to connect to the Mobile Core Doctor.

   2. Run the tool with the single collect argument to generate a zip file.

   3. Copy the zip file from the Core VM using scp.