Mobile Synchronization Scalability and Performance for the Oracle Mobile Field Service Store and Forward Applications
This appendix covers the following topics:
Overview
The Oracle Mobile Application Foundation (MAF) provides data synchronization for Oracle Mobile Field Service Store and Forward - Laptop and Oracle Mobile Field Service Store and Forward Pocket PC of the Oracle E-Business Suite. This appendix provides benchmark results that demonstrate the scalability of the MAF architecture to large numbers of mobile users on low-cost server hardware running Linux. You can use these results to determine hardware requirements for implementing the Oracle Mobile Field Service Store and Forward applications.
Mobile users periodically synchronize their mobile computer devices with the enterprise system to download new Oracle Mobile Field Service data and upload changes that they have made in their applications.
Synchronizing a mobile client with the server consists of three basic steps:
-
Preparing data and connecting to the network.
-
Communicating with the Oracle10g Lite mobile server.
-
Disconnecting from the network, processing and displaying the new data.
While all three steps are important factors in determining the total synchronization time experienced by mobile users, only the time spent communicating with the mobile server affects the ability of the system to scale to large numbers of mobile users.
Note: When modems are used with circuit-switched networks, the connection time may be a large portion (up to 30 to 40 seconds) of the total synchronization time.
In this benchmark, the server load is characterized by the number of connections per minute. Simulated connections are started at a constant rate and the behavior of the server is measured under this steady load. Eventually, as the connection rate is increased, the server is no longer able to complete the requests at the rate that they are being started.
When the incoming load exceeds the maximum steady-state performance of the system, connections are queued until the load decreases and the server is able to catch up. This enables the server to handle sudden bursts of activity without a failure. However, in this case connection times will be longer due to the queuing.
To prevent arbitrarily long connection times, the server can be configured to refuse new connections if the queue exceeds a specified size. Mobile users will instead receive a message stating that the server is unavailable and to try again later.
Test Environment
The test system consists of a load generator running on a set of PCs, a single Oracle10g Lite mobile server, plus a separate Oracle10g database server. The load generator simulates mobile client connections and can be programmed to initiate connections at a specified rate. The load generator is connected to the mobile server through a LAN connection.
The load generator simulates the actual Mobile Field Service client application. The Oracle10g Lite database and Mobile Field Service Store and Forward application are pre-installed and an initial full synchronization is performed prior to the test.
The connections are made over a high-speed LAN rather than a dial-up network, but the total load on the mobile server for a given synchronization rate does not depend strongly on the speed of the connections.
System Configuration
The following is the hardware and software configuration for the test system:
Load Generator
| Server Type: | 4 PCs |
| Number of CPUs per PC: | 1 Intel Pentium III CPU |
| CPU Speed: | 531-598 MHz |
| Main Memory per PC: | 512 MB |
| Network Interface: | 100 Mbps Fast Ethernet NIC |
| Operating System: | Microsoft Windows NT 4.0 SP6 |
Mobile Server and Database Server
| Server Type: | Dell PowerEdge 2650 |
| Number of CPUs: | 2 Intel Xeon CPUs |
| CPU Speed: | 2.8 GHz |
| Main Memory: | 6 GB |
| Network Interface: | Two Integrated Gigabit NICs |
| Operating System: | Red Hat Enterprise Linux AS 2.1 |
Software
| Oracle E-Business Suite: | Release 11i.8 Rapid Install |
| Oracle Mobile Field Service: | Patch set 8.3.5 |
| Oracle9i Lite: | Release 5.0.2.3d |
| Oracle9i Database: | Release 9.2.0.2 |
Note: Note: For the mobile and database servers, Red Hat Enterprise Linux AS 2.1 supports the Hyper-Threading feature of the Intel Xeon processors in the Dell PowerEdge 2650. The total cost of these systems is less than $10,000 USD each.
The Oracle E-Business Suite was installed using Rapid Install. Oracle10g Lite was installed as described in "Installing and Configuring Oracle10g Lite" . No special database or application tuning was performed.
Test Data
Each test run simulates connections from 100 distinct mobile users. Prior to starting the test, transactions are performed on the enterprise system to queue up data for each mobile user. To simulate a typical load for a field service technician, each mobile user is assigned three service requests, three tasks, and three notes, plus additional supporting records. A total of 21 rows from 10 tables are downloaded to each mobile user. The total compressed size of the downloaded data is 2.6 KB. No data is uploaded to the mobile server from the mobile clients during this test.
Test Results
This section describes the test applied to Oracle Mobile Field Service Store and Forward and the resulting performance specifications.
Synchronization Time
During the test, the time taken by each mobile user to communicate with the mobile server is recorded. The top and bottom 5% of the synchronization times are discarded before computing the minimum and maximum synchronization times.
The median synchronization time rises quite slowly as the synchronization rate is increased. Up to a synchronization rate of 40 synchronizations/minute, the synchronization time is essentially constant at just under 1.5 seconds. The median synchronization time increases from 1.4 seconds to 1.9 seconds (32%) as the synchronization rate increases from 10 to 55 connections per minute.
CPU Load
The average CPU load on the database computer and middle tier computer rises nearly linearly as the synchronization rate is increased. At 55 synchronizations per minute, the average database load is 13% while the average mobile server load is 20%. For a graphics representation of the CPU load, see the graph in "CPU Usage Tuning."
Note: Note: The Xeon Hyper-Threading is enabled during the test, so the average CPU usage is calculated over the four logical processors rather than the two physical processors.
Maximum Steady-state Performance
At a synchronization rate of 55 connections per minute, the average synchronization time remains very low and the peak CPU load on the database computer and middle tier computer is reasonable. However, beginning at a synchronization rate of 60 connections per minute, the server can no longer process the requests as fast as they arrive, so the excess requests are queued up. When the incoming load stops at the end of the test, the mobile server completes the remaining requests. All of the requests completed without errors.
Laptop End-End Synchronization Test
In addition to the server capacity, the end-user is interested in what is the end-end synchronization time. For Oracle Mobile Field Service Store and Forward - Laptop, this is the time that elapses from the point the user initiates synchronization in the application to the point the application returns to the calendar screen. This time includes the following:
-
Time to contact the mobile server and authenticate the user and process data for the user.
Server side scalability is only impacted by this and has been discussed in depth in the preceding sections.
-
Time taken to check if there is an application upgrade.
Apart from data transfer, the synchronization operation also checks if there is a new version of the software. If there is a new version of the software it downloads the same. This being a search against the server file system does not impact scalability
-
Time taken for the application to restart the calendar screen.
This does not depend on the number of users and is only a function of the device the user uses.
The following table describes the synchronization data amount and synchronization times for end-end synchronizations for Oracle Mobile Field Service Store and Forward.
| Scenario | Amount of Data Transferred | Fixed Overhead (Logon, Authentication and Application Upgrade) | Time Taken on LAN | Time Taken on Wireless Network (50 kbps) |
|---|---|---|---|---|
| Null End-End synchronization with no updates either on client or server | 1K | 5.6K | 16s | 29s |
| Synchronization with user updating one task with two notes, six debrief lines | 4K | 5.6K | 17s | 31s |
| Synchronization with three new service requests, three new tasks and three new customers and three new install base items | 4K | 5.6K | 17s | 33s |
Bulk Record Update Test
In addition to normal daily transactions, organizations periodically make changes to large numbers of records in the enterprise system, such as a quarterly update to the System Items table in a field service organization. After such a bulk update of records, the modified records are downloaded to each mobile user during their next synchronization.
To test the scalability of the Mobile Application Foundation for bulk updates, the synchronization time was measured for a single mobile user after modifying 96,000 system item records. The test was performed with the mobile client connecting over a high-speed local area network. The following table summarizes the results:
| Test Case | Total Time |
|---|---|
| Initial Download All application data, including 96,000 system items |
4:14 |
| Incremental Update After 96,000 system items were updated |
4:04 |
| Incremental Delete After 96,000 system items were deleted |
2:54 |
The test results show that synchronization times for bulk updates are essentially limited by the transfer size of the modified records rather than scalability of the mobile server. On a wireless network, transferring 12MB of data will take at least 40 minutes (even at an ideal connection rate of 40kbps), which is an order of magnitude greater than the processing time. The mobile server compresses all data transfers for optimum performance. In this case, the system items average 125 bytes each.
The key to limiting the worst-case synchronization time is to intelligently limit the maximum amount of data that is downloaded to the mobile users. For example, Oracle Mobile Field Service Store and Forward applications provide a category mechanism to specify which system items are needed for mobile users. Only those system items will be downloaded.
System Sizing
To determine the appropriate hardware for the mobile server and database, develop a model of the anticipated system load during the workday. For example, the graph in this section shows a load model for an organization with 3,000 mobile users working in four U.S. time zones.
The workforce is assumed to be divided among the time zones as described in the following table:
| Time Zone | Usage | Amount of Users |
|---|---|---|
| Eastern | 35% | 1,050 |
| Central | 30% | 900 |
| Mountain | 10% | 300 |
| Pacific | 25% | 750 |
| Total | 100% | 3,000 |
As described in the following table, it is assumed that users synchronize three times per day, according to the normal distributions:
| Synchronization | Mean Time | Standard Deviations |
|---|---|---|
| Start of workday | 7:55 | 10 min |
| Middle of workday | 13:00 | 3 hrs |
| End of workday | 17:30 | 30 min |
For this model, the peak load is 41 synchronizations per minute at 7:55 Eastern Time. This load is well within the capabilities of the reference configuration used for the benchmarks in this appendix. At the time of the peak load, the estimated synchronization time is 1.6 seconds and the average CPU load is 9% for the database and 13% for the mobile server.
Conclusion
This benchmark demonstrates the high scalability of the Oracle Mobile Application Foundation running on low-cost hardware using Linux. Synchronization times for typical Mobile Field Service Store and Forward connections are very short (one to two seconds), and they remain low as the incoming request rate is increased. The CPU usage scales nearly linearly with the incoming request rate.
For the reference test configuration, the maximum steady-state synchronization rate is shown to be at least 55 connections per minute. At this rate, approximately 500 mobile users can synchronization in a 10-minute period. Depending on how concentrated the synchronization activity is at the peak times during the workday, this rate can support a field organization of several thousand mobile users. To determine system requirements, develop a model of the anticipated number of synchronizations per minute throughout the day, taking into account time zones and business processes.
The Oracle Mobile Application Foundation also robustly handles surges in load. When the incoming request rate exceeds the maximum steady-state performance of the system, the mobile server queues the excess connections and completes all of the requests without error, once the load drops below the maximum rate.
Performance Tuning of Mobile Application Foundation
To achieve the test results described in this appendix, tune the Mobile Application Foundation as described in this section. Here, performance tuning issues specific to Oracle Mobile Field Service Store and Forward are addressed. The following figure illustrates the Oracle Mobile Field Service Store and Forward architecture.
The Oracle10g Lite mobile server functions as a conduit between the mobile computer device and the enterprise database, transferring data and the Oracle Mobile Field Service Store and Forward application between the two. The first or initial synchronization, from the mobile computer device installs the entire Oracle Mobile Field Service Store and Forward application as well as the data targeted for that mobile user. Subsequent synchronizations (also called incremental or fast synchronizations) only download application upgrades and incremental data. This incremental dataset consists of data updated on the mobile computer device since last synchronization and data ready on the server since the last synchronization. It can be expected that in a live or production instance, application upgrades are minimal and therefore incremental synchronizations mainly transfer new data. Synchronizations that do not involve either application upgrades or data transfer are called null synchronizations. Null synchronizations are useful for performance benchmarking since they provide the lowest possible synchronization time.
First-time synchronizations should be done on either a LAN or network with a high bandwidth so that the synchronization completes in a timely manner. First-time synchronizations over low-bandwidth (GPRS) networks can terminate due to low network time-out settings. In this case, the latency for preparing a first-time synchronization dataset could require minutes as opposed to seconds for subsequent synchronizations. If the first-time synchronization fails, the process must restart from the beginning.
You are not expected to tune the Oracle Mobile Field Service Store and Forward application on the mobile computer device, except to ensure that the minimum system requirements are met.
Administrators should use the following to assess system performance:
-
Synchronization time.
Synchronization reports are provided to track synchronization times. Reports can be generated with various search criteria. See "Viewing User Synchronization Statistics".
-
Mobile server CPU and memory usage.
This can be done in UNIX using the SAR command and in Windows using the Performance Monitor.
-
Concurrent program run times.
Each mobile concurrent program should be monitored for excessive run times.
-
Workflow background process run times.
Mobile Server Tuning
For best performance, install the Oracle10g Lite mobile server and Apps 12 database on separate servers. When Oracle10g Lite mobile server shares system resources with another server, ensure that the memory and CPU usage of the Oracle10g Lite mobile server is tuned to handle synchronization requests without overloading.
Tuning Memory Usage
For optimum mobile server performance, ensure that the server has ample memory. The rule of thumb is to allocate a minimum of 512MB to the mobile server and then increment this amount by 512MB for every 1000 users. Therefore, for an installation with 1500 users, a memory allocation of 512MB (base allocation) plus another 512MB for a total of 1GB is best.
Start the mobile server as a java process to ensure that the server utilizes any extra memory. The command syntax to do this is given below:
java -Xms1024M -Xmx1024M -XX:+DisableExplicitGC ?XX:NewSize=128m
-XX:MaxNewSize=128m -XX:SurvivorRatio=2 oracle.lite.web.JupServer mode=server config-file=/u01/olite/mobile/server/bin/webtogo.ora
- where -
?Xms1024M and ?Xmx1024M specify an initial and maximum heap size of 1024MB.
The rest of the java parameters optimize garbage collection.
To learn about performance tuning, see My Oracle Support.
CPU Usage Tuning
CPU usage is directly related to the number of concurrent users. Concurrent users are mobile users who are actively synchronizing with the enterprise system at a given point of time. Typically, the maximum number of concurrent users is about 10% of the total user base. For example, if 250 mobile users are created, then you can expect a maximum of 25 concurrent users.
The following graph shows the CPU load with increasing concurrency from a benchmarking study. It shows a linear rate of increase in CPU load as the synchronization rate is increased. (See "CPU Load" .) Synchronization rate is the number of synchronizations that are initiated every minute. In this benchmark, the synchronization time was between one and two seconds.
In order to limit the CPU load on the mobile server, it is possible to limit the number of concurrent users. The webtogo.ora parameter MAX_CONCURRENT can be used for this purpose.
Database Tuning
Mobile Field Service Store and Forward captures online business activities such as service request creation and task assignments and then downloads this data to the mobile users. Mobile Field Service Store and Forward uses both concurrent programs as well as workflow processes to find and prepare this data. Data that is prepared for synchronization is stored in the out queue. The out queue comprises two tables: ASG_SYSTEM_DIRTY_QUEUE and ASG_DELETE_QUEUE.
Tuning Out Queue
Better synchronization times can be achieved by tuning the out-queue. The out-queue performance is constantly being improved by tuning the queries executed on the out-queue tables. Since the tables are insert/delete intensive, one of the important storage parameters is PCTUSED. For best performance, set this parameter to a high value like 70%. The size of the out-queue is related to amount of business activity relevant to mobile. Data prepared in the out-queue is automatically removed after the mobile computer device receives it.
There are several indexes defined in the ASG_SYSTEM_DIRTY_QUEUE table that eliminate the need for full table scans and optimize access. However, it is recommended that these indexes be managed carefully. Due to the insert/delete intensive nature of data management languages (DMLs) on this table, index-stagnation could become an issue. This will result in longer synchronization times even though the size of ASG_SYSTEM_DIRTY_QUEUE table remains about the same. In this case, index-rebuilds have proven to be helpful.
It is possible to purge the out-queue for dormant users by scheduling the JTM Master Concurrent Program with category type "purge." This will purge records for dormant users. Dormant users are those who have not synchronized for the past N days, where N is the value of profile option, ASG: Dormancy Period. The JTM Master Concurrent Program will also remove any duplicate records in the out-queue. For best performance, schedule this program once every week.
Tuning Concurrent Programs
For mobile operations, mobile concurrent programs place a big demand on the database, even more so than synchronization. The JTM Master Concurrent Program is a good example of this. For this program, you must run three category types: Transaction, Lookup and Inventory. You should regularly monitor the times taken for each category type. You can use the following table to do this.
| Category Type | Scheduled Interval | Average Run Time |
|---|---|---|
| Transaction | 5 min | |
| Lookup | 1 hour | |
| Inventory | 1 day |
The actual scheduled interval for these programs depend on your business requirements. However, the average run time should be compared with the scheduled interval to make sure that it is a very small fraction of the scheduled interval (less than 5% for Lookup and Inventory, and 20% for Transaction). If the concurrent programs take a longer time, then increase the scheduled interval so that the system is not overloaded.
Workflow Background Process
The workflow background process is used only by Oracle Mobile Field Service Store and Forward - Pocket PC. You should submit this process with the Process Stuck parameter set to "No."
Finally, you should periodically analyze the database tables/indexes so that the optimizer can choose the best execution plan. The Oracle Mobile Field Service Store and Forward schemas are ASG, JTM, CSL and CSM. Tables in this schema should be analyzed together with tables from Field Service and other modules used by Oracle Mobile Field Service Store and Forward.