The following sections describe how to troubleshoot the mobile server:
The following sections describe how to troubleshoot the synchronization process or what to do in the event of certain synchronization scenarios:
Section 2.1.2, "Situations Where the Client is Out of Sync that Triggers a Complete Refresh"
Section 2.1.3, "The "Inconsistent Datatypes" SQLException Received If Order is Not Correct in Query"
Section 2.1.4, "MGP Compose Postponed Due to Transaction in the In-Queue"
Consult the following sections for details on how to resolve any synchronization errors or conflicts:
Section 2.1.1.1, "General Synchronization Errors and Conflicts"
Section 2.1.1.2, "Synchronization Error if Client Device Clock is Inaccurate"
Section 2.1.1.3, "Synchronization Error After Modifying Client Password"
Section 2.1.1.4, "Synchronization Error if Synchronized with a Large Number of Tables"
With the mobile server, you can have the following errors when synchronizing: nullity violations, foreign key constraint violations, or the client updates a row at the same time that the server deletes it.
The mobile server does not automatically resolve synchronization errors. Instead, the mobile server rolls back the corresponding transactions, and moves the transaction operations into the error queue. It is up to the administrator to view the error queue and determine if the correct action occurred. If not, the administrator must correct and re-execute the transaction. If it did execute correctly, then purge the transaction from the error queue.
A mobile server synchronization conflict occurs if:
Nullity violations.
Foreign key constraint violations.
The client and the server update the same row.
The client and server create rows with the same primary key values.
The client deletes the same row that the server updates.
The client updates a row at the same time that the server deletes it.
See Section 2.10, "Resolving Conflict Resolution with Winning Rules" in the Oracle Database Mobile Server Developer's Guide for more information on conflict resolution techniques.
The client device clock must be accurate within the timezone set on the device before attempting to synchronize. An inaccurate time may result in the following exception during synchronization: CNS: 9026 "Wrong user name or password. Please enter correct value and reSync."
If you have an active client and change its password on the server, then the client cannot synchronize. Return the password back to its original value on the server and retry the synchronization.
Berkeley DB mobile client may fail to synchronize with error "out of memory" when a large number of tables need to be synchronized.
To avoid this error, you can tune Berkeley DB using the parameters in the configuration file (DB_CONFIG) before doing synchronization.
When a client is out of sync with the server, any outstanding uploaded transaction from the client is placed in the error queue and a complete refresh is triggered to re-initialize the client data with what is currently on the server.
The following are a list of the situations—ordered from most to least likely—that can trigger a complete refresh for the client:
Dropping and then republishing the application.
Synchronizing by the same mobile user from multiple devices on the same platform or from different platforms when the publications are not platform-specific.
Receiving unexpected server apply phase conditions—such as constraint violations, unresolved conflicts, other database exceptions.
Modifying the application—such as changing subsetting parameters, or adding or altering publication items.
Requesting a force refresh from either the server admin or client.
Two separate applications using the same backend store.
Unexpected client apply conditions—such as deleting, moving or restoring the client database, database corruption, memory corruption, and other general system failures.
Loss of transaction integrity between server and client. The server fails post processing after completing a download and disconnecting from the client.
Data transport corruptions.
If you are creating a fast refresh publication item on a table with a composite primary key, the snapshot query should list the primary key columns in the order that they are present in the table definition. This automatically happens during the column selection when MDW is used or when a SELECT * query is used. Note that the order of the primary key columns in the table definition may be different from those in the primary key constraint definition.
The following example demonstrates what is valid or invalid given the table definition for TAB1:
CREATE TABLE TAB1( ID1 NUMBER(10) NOT NULL, ID2 NUMBER NOT NULL, COL1 VARCHAR2(200), COL2 VARCHAR2(200), ID3 NUMBER(4) NOT NULL); ALTER TABLE TAB1 ADD CONSTRAINT TAB1_PK PRIMARY KEY (ID3, ID2, ID1);
The following are valid snapshot queries:
SELECT * FROM TAB1 SELECT ID1,ID2,ID3,COL1,COL2 FROM TAB1 SELECT ID1,ID2,COL1,COL2,ID3 FROM TAB1
The following are invalid snapshot queries:
SELECT ID3,ID2,ID1,COL1,COL2 FROM TAB1 SELECT ID3,ID2,COL1,COL2 ID1 FROM TAB1
Define the table columns where the primary key columns appear before other columns. The order of the primary key columns in the table definition order must match the constraint definition in the snapshot query.
If the user synchronized and uploaded some more changes after the last apply cycle for a particular user; by default, the MGP must first apply thse changes before it can compose. If this keeps happening, the compose could be postponed beyond what you would like. By default, the MGP tries to avoid a compose phase postponed due to a transaction in the in-queue by performing an apply for any unprocessed in-queue data before doing a new compose. However, if the MGP compose is postponed due to a transaction in the in-queue, you can modify the following parameters to avoid the error:
SKIP_INQ_CHK_BFR_COMPOSE: By default, this parameter is set to NO. Setting this parameter to YES, then a compose is performed for a client even if there is unprocessed data in the in-queue.
DO_APPLY_BFR_COMPOSE: By default, this parameter is set to YES. If set to YES, the unprocessed data in the in-queue is applied before a client compose. This parameter takes effect only if SKIP_INQ_CHK_BFR_COMPOSE is set to NO.
For most situations, preserving the default values for these two parameters avoids the occurrence of the MGP Compose postponed error.
The Server Busy warning can be thrown for one of the following reasons:
When the MGP is processing apply/compose for that user.—To avoid MGP contention with sychronization, MGP should be scheduled to run when few clients are synchronizing. Alternatively, you could use queue-based synchronization, which does not use the MGP at all; thus, avoiding MGP contention with synchronizaiton.
If a previous synchronization was interrupted for that user and Oracle Database Mobile Server rolls back the transaction—If the Server Busy warning is a result of a long rollback, then Oracle Database recommended tuning steps for rollback operation may reduce the Server Busy state for the client.
The following sections detail how to troubleshoot the mobile server and its repository:
If you experience any difficulty with the mobile server, you can enable tracing in the mobile server.
To enable tracing in the mobile server, set up your environment as described in Chapter 3, "Tracing and Logging".
The following sections describe how to evaluate, validate and recover the mobile server repository:
Section 2.3.2, "Inspecting Files in the Mobile Repository With the WSH Tool"
Section 2.3.3, "Modifying IP Address of Machine Where Mobile Repository Exists"
Customers may modify the mobile server repository in the back-end database and, without realizing it, violate some of the rules. The Mobile Server Repository Diagnostic Tool (MSRDT) provides a mechanism for the customer to analyze, validate, and debug the mobile server repository.
The Mobile Server Repository Diagnosis Tool will automatically correct errors that it can and prints out all results of what was modified incorrectly or missing.
Note:
The output generated is best viewed if you set your column width to 80 or if you pipe it into a file, to view it with Word or WordPad.You can use the Mobile Server Repository and Diagnostic Tool (MSRDT) to validate your environment and what is in the back-end mobile server repository. The following sections describe the validation that occurs:
Section 2.3.1.1.1, "Validate the Environment for the Mobile Server"
Section 2.3.1.1.2, "Validate Integrity of Mobile Server Tables and Data"
Section 2.3.1.1.3, "Validate the Structure and Contents of the Repository"
The MSRDT tool displays the system configuration environment for the host where the mobile server resides, as follows:
Consolidator version
Back-end Oracle database version
Definition of the Java library path
Definition of the Java CLASSPATH
Operating system architecture, type and version
Java VM vendor, version, name, home, and info (mode)
File encoding used
Path separator and file separator used
Country, time zone, and user language
Lists the mobile server administrators
Lists the contents/current configuration for the mobile.ora file
When the mobile server and the repository are installed, certain tables, constraints, objects, and so on cannot be modified. The MSRDT diagnostic tool checks that these requirements are consistent as with what was installed. This includes the following:
Required tables exist
All columns within required tables exist
Required table attributes exist
Required constraints exist or have not been modified
Required sequences exist
Extra tables have not been added to mobile server schemas
Application files integrity check: If you have published an application to the server, then check if the folder exists and is not empty. Also, if sharing a repository among multiple mobile servers, ensures that the application is published on this mobile server, where this tool is executed.
If you are experiencing trouble with your repository, execute the MSRDT tool to determine if any of the following have occurred:
The mapping between primary keys for the map table and corresponding base table must be consistent.
Every map table must have a corresponding base table.
Identify the user with the most records in the map table.
Invalid indexes
Consistency of the In Queue schema with the ( CPV ) view schema.
Check if the Error Queue and the In Queue have the same records; that is, both queues should not contain records with the same primary key, which consists of the CLIENTID, TRANID$$, and SEQNO$$.
The records in the In Queue master table have corresponding records in the In Queue detail table.
Any DML lock is held by any table in the C$ALL_LOGGED_TABLES.
Invalid triggers.
Every publication item is included in C$PUBITEM_PROPS.
Validate that all of the users subscribed to a publication also have all of the corresponding sequences assigned to the same publication.
Invalid sequence values.
Validate that every record in C$ALL_SEQUENCES corresponding C$WS_<ID> window sequence object, where <ID> is the values of ID column in C$ALL_SEQUENCES.
Validate that every window sequence has the same id as a record in the C$ALL_SEQUENCES.
Look for any orphaned objects in the repository.
Verify that the mobile server repository owner is granted with sufficient privileges.
Validate the MGP properties: If MGP_SUSPEND(apply suspend) is false, then MGP_RUN must be true.
Reports on the most recent jobs and their status.
Check for any automatic synchronization notification sent to a non-existent user in C$DATA_NOTIFICATION.
Check for any automatic synchronization notification that has not been sent by the Device Manager for more than two days.
You can specify that the application uses a database other than the one in which the mobile server repository resides for all application schema data. If any application databases are defined, then the MSRDT tool checks for the following in all of the defined application databases:
Check for consistency for the primary keys for both the map table and corresponding base table
Check for any records that are in the In Queue master table, but not in the corresponding In-Queue detail table
Check for any tables in the C$ALL_LOGGED_TABLES that hold any DML locks
Check for duplicate records in the Error Queue and the In Queue
Identify the user with the most records in the map table
Verify that the In-Queue schema is consistent with the CPV view schema
Check for any invalid triggers and recompile any invalid triggers
Check for any invalid application database connections
Check for any application databases that are registered more than once. This shows up as duplicate records in the C$DB_INST table.
If the publication is dropped from the main database, then the base tables for the publication must also be dropped in the application database. The MSRDT tool checks if the publication was dropped in both the main database and the application database. If the publication tables still exist in the application database, then the MSRDT tool removes the publication entries in C$ALL_PK_HINTS and C$EQ in the application database.
Check for any records in the C$IN_MESSAGES and C$INQ tables in the application database for any dropped clients in the main database. In this case, the MSRDT tool reports the records and moves the C$IN_MESSAGES records to the C$EQ table, and purges the C$IN_MESSAGES table.
You can use the Mobile Server Repository Diagnostics Tool (msrdt) to validate the repository and provide error reporting. It also performs some error recovery.
The following is the usage and syntax for the msrdt tool:
msrdt -v <username>/<password>@<jdbc_url>
Where:
<username>/<password>: The mobile server repository administrator user name and password.
<jdbc_url>: You can specify the JDBC URL of a single Oracle database or an Oracle RAC database, as follows:
The URL for a single Oracle database has the following structure: <host>:<port>:<SID>
The JDBC URL for an Oracle RAC database can have more than one address in it for multiple Oracle databases in the cluster and follows this URL structure:
jdbc:oracle:thin:@(DESCRIPTION= (ADDRESS_LIST= (ADDRESS=(PROTOCOL=TCP)(HOST=PRIMARY_NODE_HOSTNAME)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=SECONDARY_NODE_HOSTNAME)(PORT=1521)) ) (CONNECT_DATA=(SERVICE_NAME=DATABASE_SERVICENAME)))
Note:
if you supply a Oracle RAC URL as the JDBC URL, then enclose it within two double-quotes as the operating system treats the equal sign (=) as a delimiter, which truncates the Oracle RAC URL and throws thesyntax error: unexpected token '('. errorYou can use the mobile server shell utility (wsh) to inspect and modify the mobile server repository interactively. For full details, see Appendix B.2, "Running a Script File with the WSH Tool" in the Oracle Database Mobile Server Administration and Deployment Guide.
During the installation, the machine name or IP address is provided by the user where the repository is created. If the IP address of the machine changes, then perform one of the two options:
If the user provided the machine name; then even after the IP change, the machine name will still work.
If user provided the IP address—instead of machine name—then after changing the IP address of the repository machine, the user must change the ADMIN_JDBC_URL and THIN_JDBC_URL parameters in the mobile.ora file on the mobile server.
This section focuses on how to debug the following Java error:
When you are experiencing the OutOfMemory error, then you should have an understanding of JVM memory architecture when tuning mobile server performance.
The following may cause an OutOfMemory error:
A memory leak in the mobile server.
Not enough physical memory to handle your application.
In-appropriate allocation of the three memory areas that used by the JVM. See Section 2.4.1.1, "JVM Memory Settings" for a full description.
Memory being held by the mobile server. See Section 2.4.1.2, "Why is Memory Not Released?" for more information.
Understanding how threads are consuming your memory. See Section 2.4.1.3, "Thread Memory Consumption and Concurrency" for full details.
JVMs may have different implementations of memory management and garbage collection schemes. But at a higher level, they all arrange the memory in the following three areas:
Section 2.4.1.1.1, "Java Heap"—The Java heap is where the Java objects live. It is normally the largest of the three.
Section 2.4.1.1.2, "Permanent Generation"—The memory where the classes are loaded.
Section 2.4.1.1.3, "Native Space"—The memory used by native code, which includes JVM native code and application JNI calls.
Section 2.4.1.1.4, "Setting Java Options for Java Memory"—Setting options depending on your environment.
This section describes how to modify the allocation of memory to the JVM memory areas.
Note:
Memory should be allocated properly for the three areas. Otherwise, different kinds ofOutOfMemory error may surface.The Java heap is where Java objects live. It consists of both the young and tenured generations. The amount of Java heap memory that the JVM starts with is designated by the initial heap size option (-Xms) and the maximum heap size option (-Xmx).
If you see from the stack trace that a Java method throws an OutOfMemory error, then you have exhausted your Java heap space.
For example:
java.lang.OutOfMemoryError: Java heap space
The default settings for the Java heap for a Oracle UNIX JVM is as follows: Xmx:64M Xms:4M. However, the default for the mobile server—if you start up the mobile server with the runmobileserver.bat executable—is set to Xmx:256M Xms:512M.
The size of the space reserved can be specified with the -Xmx option. The -Xms specifies the space that is immediately committed to the virtual machine. We recommend to allocate ¼ to ½ of the available physical memory to Java Heap. If you set the maximum Java Heap size to be large—such as, 512M—and you still receive this error, then there may be a leak in the Java code.
The amounts specified should be based on the available resources. At the minimum, you should set both values to at least 256 MB. Of course, the amount of memory you allocate depends on what you have available.
Note:
Set the Java heap memory size before starting the mobile server with therunmobileserver executable.The permanent generation holds data needed by the JVM that describes objects which do not have an equivalence at the Java language level. For example, permanent generation is where the classes are loaded. It holds objects that describe classes and methods.
If a classloader method or a String intern method throws an OutOfMemory error in the stack trace, then you have run out of permanent generation space.
For example:
java.lang.OutOfMemoryError: PermGen space at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
The default for the permanent generation for a Oracle UNIX JVM is 64MB. To set a new initial size for the Oracle JVM, use the -XX:PermSize option when starting the virtual machine. To set the maximum permanent generation size use the -XX:MaxPermSize option.
The native space is the memory used by native code, which includes JVM native code and application JNI calls. If a native method throws an OutOfMemory error or the JVM crashes with such an error, then you run out of native space.
For example:
java.lang.OutOfMemoryError: requested 14892 bytes. Out of swap space? java.lang.OutOfMemoryError: unable to create new native thread
The native space is the (Available physical memory) – (Java heap + Permanent generation). There is no way to set the native space, except to decrease the Java heap or permanent space. If you allocate too much memory for the Java heap, then the native code is left with not enough memory and may run out. If you have to increase the native memory, then decrease the -Xmx parameter to a reasonable value to leave enough memory for the native space. If you still get this error, the native code may have a memory leak.
Set the Java options when you start the Java servlet container. The following example sets the initial Java heap to 256M, the maximum Java heap to 512M, and the permanent generation memory to 64M:
–Xms=256m –Xmx=512m -XX:MaxPermSize=64m
For the mobile server, all modifications for the Java options must be specified on the command-line or in the runmobileserver.bat file.
By default, the MGP executes as a job in the Job Scheduler in the mobile server. Thus, the MGP and other mobile server components, such as the Sync Server, share the same memory space. This provides efficiency and manageability; however, if the MGP has a memory leak, then the mobile server is affected. In this case, perform the following:
Disable the MGP job.
Restart the mobile server.
Restart the MGP in a separate JVM with either the mgp.bat or, if using UNIX, the mgp shell script. This JVM is restarted periodically and may hide the memory leak issue.
Note:
Set the Java memory options for the MGP in themgp.bat file.With a larger Java heap size, the garbage collector collects less often and consumes less CPU time. Therefore, a larger heap size is desired for better performance. For the mobile server, most of the code is Java code; for the JDBC connection and Java mSync client, most of the code is in native code. So, setting the Java heap size larger, helps the efficiency and performance of the mobile server; however, if it is set too high, the JDBC connection and mSync client may have memory issues.
You may expect the mobile server to release the free memory back to operating system after it has finished its work. However, the mobile server holds a large amount of memory even when it is idle. This may not be an indication of memory leak; instead, it may be for one or more of the following reasons:
If you set the -Xms option to a large number—such as, 1024 MB—then you should expect the mobile server process to use at least 1024 MB until the process is killed.
For performance reasons, the mobile server caches metadata in Java heap memory.
The garbage collector may not collect the objects right way when they are no longer referenced. In addition, the garbage collector keeps a large amount of free memory in the Java heap for future allocations, instead of returning them to the operating system. You can use Java options to adjust the free memory size; instead, view the mobile server total runtime Java heap size and free heap size in the Mobile Manager at Mobile Manager->Data Synchronization->Host.
The Java heap and permanent generation together are called managed heap, since the garbage collector manages them. The native space can be divided into native heap and thread stack space. Each thread consumes memory, as follows:
Each thread created consumes about 1MB stack space, although it is JVM dependent. Take this memory into consideration if you execute multiple threads. For example, on a 32-bit x86 system, the (managed heap + native heap + thread stack size * number of threads) could not exceed 2 GB. On any system, ensure that the total JVM memory is less than the available physical memory size.
Each thread allocates additional Java and native heap memory as it executes.
There is an overhead associated with multi-threading. Therefore, be careful when executing too many concurrent threads. If concurrency is set to larger than 20, then you are more likely decreasing the mobile server throughput—instead of increasing it.
You can configure for concurrency with the parameters described in Section 1.2.4, "Configuration Parameters in the MOBILE.ORA that Affect Synchronization Performance".
The following section describes how to troubleshoot security issues:
If you are using a reverse proxy and have configured SSL between the client and the reverse proxy, you may receive the following error:
A certificate is required to complete client authentication.
For all clients you can only use SSL authentication with a signed certificate. If you use a self-signed certificate, you must turn off SSL authentication by adding the following to the NETWORK section in the client devmgr.ini file:
DISABLE_SSL_CHECK=YES
This parameter tells the reverse proxy firewall to use SSL encryption for the communication from the client, but not to perform SSL authentication.
If the environment of your device is incorrect, you may get the following error when you install mobile client using setup.exe or when you use the test publication functionality in Mobile Development Kit (MDK): "Insufficient information to proceed: error code: invalid platform: the thread is already in background processing mode."
This error is reported by the device manager.
If you get this error when you install mobile client, you can do the following to clean up your environment:
Set system environment variable PATH=%MOBILE_CLIENT_HOME%\bin;%PATH%, where %MOBILE_CLIENT_HOME% is the directory where users install mobile client.
Clean all files in %MOBILE_CLIENT_HOME%, ensuring that it's a clean environment.
Restart the machine, which is used to clean the shared memory.
Try to install a mobile client from the scratch. It should succeed without the error if the environment is clean.
If you get this error when you test publication in MDK, you can do the following to clean up your environment:
Set system environment variable PATH=%MOBILE_HOME%\sdk\bin;%PATH%, where %MOBILE_HOME% is the directory where users install MDK.
Make sure no mobile client is installed on the same machine. Restart the machine, which is used to clean the shared memory.
Try to test publication from scratch. It should succeed without the error if the environment is clean.