Development Performance Guidelines

This document includes information, guidelines, and strategies to help designers and developers understand performance impacts when developing a feature using the Oracle Utilities Application Framework.

Object-Relational Mapping: Background

OUAF uses an Object-Relational Mapping (ORM) engine, which maps tables to entities using the system's table, table/field, field, and constraint metadata to guide the creation of mapping definitions during artifact generation.

Entities represent database tables.  They are created as Java objects during a database "session", which has the lifetime of a single DB transaction.

DONT: Entities are not safe to use for reference, calling methods, etc., after the session that created them has ended. For example, don’t copy entities into application caches. DO: Instead, let the application cache do the data retrieval and return the data to the session.  ID objects are safe to store across sessions.  Note in the following example that the entity AlgorithmType is not stored:

public class AlgorithmTypeInfoCache implements ApplicationCache {

  private static final AlgorithmTypeInfoCache INSTANCE = new AlgorithmTypeInfoCache();

  private final ConcurrentMap<AlgorithmType_Id, AlgorithmTypeInfo> algorithmTypeInfoById = new ConcurrentHashMap<AlgorithmType_Id, AlgorithmTypeInfo>();

  protected AlgorithmTypeInfoCache() {  ContextHolder.getContext().registerCache(this);    }

  public String getName() {  return "AlgorithmTypeInfoCache";    }

  public void flush() {algorithmTypeInfoById.clear();    }

  public static AlgorithmTypeInfo getAlgorithmTypeInfo(AlgorithmType_Id algTypeId) {

    return INSTANCE.privateGetAlgorithmTypeInfo(algTypeId);

  }

 

  private AlgorithmTypeInfo privateGetAlgorithmTypeInfo(AlgorithmType_Id algTypeId) {

    AlgorithmTypeInfo algTypeInfo = algorithmTypeInfoById.get(algTypeId);

    if (algTypeInfo != null) return algTypeInfo;

    AlgorithmType type = algTypeId.getEntity();

    if (type == null) return null;

    AlgorithmTypeInfo info = new AlgorithmTypeInfo(type);

    AlgorithmTypeInfo prev = algorithmTypeInfoById.putIfAbsent(algTypeId, info);

    if (prev != null) return prev;

    return info;

  }

}

DO: it is safe to use XML documents  (to be consumed by BOs, BSs, or SSs) for moving data between sessions.

Every entity has a unique corresponding "id" class, e.g. BatchControl has BachControlId.  The ORM framework automatically generates correct SQL to perform the following essential tasks:

·         Read, update, insert, delete one entity (row) from the database

·         Navigate between related entities as per their key/constraint relationships, for example from a parent entity to a collection of children.

The ORM defers database calls for performance

The ORM tries to be as "lazy" as possible; its basic stance is to avoid loading any data from the DB until the last possible moment.  Let’s use the following example to describe how the data is only loaded at last moment possible:

BatchControl someBatchControl = batchControlId.getEntity();

BatchControlParameters parms = someBatchControl.getParameters();

for (BatchControlParameter each : parms) {

String name = each.getBatchParameterName();

}

In the above example, the getEntity() call only retrieves the parent ID as a proxy.  The “someBatchControl” is not fully “hydrated” until some other property is accessed.  “Hydrating Entities” is the process of taking a row from the database and turning it into an entity.

The getParameters() call only retrives the child IDs, again as proxies.

Only when the getBatchParameterName() is called, is a row (the child row) actually retrieved.

ID Objects

When you create an ID, the ID object will not be null.  After you use an ID to retrieve an entity (using getEntity()), that is when you find out if the entity actually exists.  Just because an ID exists, doesn’t mean the entity itself exists!  DO: So you must check for null before attempting to use the entity you retrieved.  For example:

BatchControlId id = ...

BatchControl batchControl = id.getEntity();

if (batchControl == null) { /* oh oh */ }

Counting a collection

DO: If you want to count the number of batch control parameters that belong to a parent batch control, use the size() method as in the following example:

BatchControl someBatchControl = ...;

BatchControlParameters parms = someBatchControl.getParameters();

int count = parms.size();

The framework implementation code has an optimized implementation of the size() method, which either counts the existing in-memory collection elements, if they are already present, or issues a SQL count(*) query, if they aren't.

Avoid unnecessary work

DON’T: In the example, below, the call to listSize() is unnessary.  In most cases, you shouldn’t need to write something to loop over a collection:

if(query.listSize() > 0) {

   while (iter.next()) { .... }

}

The call to listSize() will make an unnecessary call to “select count(*)”.  Let the iterator do the work.  Avoid the extra call to the database.

ORM “Navigation” is your friend

Don’t be tempted to hand-write queries that are equivalent to navigations between entities:

BatchControlId batchControlId = ...

Query<BatchControlParameter> query = createQuery("from BatchControlParameter parm where parm.id.batchControlId = :parentId");

query.bindId("parentId", batchControlId);

List<BatchControlParameter> list = query.list();

DO: Use this instead – it’ll use the cache and will almost certainly be faster:

BatchControl batchControl = id.getEntity();

if (batchControl == null) { /* oh oh */ }

BatchControlParameters list = batchControl.getParameters();

How to Pre-load Entities Using Fetch

This technique is for performance intensive jobs that are doing too many single-row SQL retrieves.  The “fetch” command will pre-load the entities, resulting in one fewer database calls.

Write a query using “left join fetch” to select all data.  The ORM will fetch the associated collection for every retrieved table into the session cache. Subsequent navigation to the underlying collection is then an in-memory operation with no database IO. Again, PREFER code that performs standard navigation.

As a general strategy:

·         For most jobs, navigation is just fine.

·         Write code using navigation first, then ADD the fetch query later, only if it’s needed.

This is a link to the hibernate help on “fetch”: http://docs.jboss.org/hibernate/stable/core/reference/en/html_single/#performance-fetching

Session Cache

If an entity is retrieved previously within a session - in most cases - it does not have to be retrieved again since it is stored in the session cache.

So, multiple BO reads against Java-backed MO's do not re-execute SQL.

The exception is when a call to COBOL is made.  Since Java does not know what COBOL could have changed, the entities must be refreshed from the database.  Similary BO Reads against COBOL will always re-execute SQL.

Level 2 Cache Applicable for Batch

Hibernate’s Level 2 cache is a second level of caching that allows sharing of data between sessions.  This is useful for static, admin data like rates, type codes, etc since objects that are added to this cache cannot be updated.  The caching is enabled per entity on the Table transaction’s Caching Regime Flag with values of “Not Cached” and “Cached for Batch.”

Flushing – COBOL and Save Points

Flushing means writing the changes to the database.  It syncs the database with the session cache.  Flushing is expensive but necessary to maintain data integrity. The system flushes under the following conditions:

Avoid Extra SQL

Inspect generated SQL for extra calls.  Tools like Oracle’s tkprof, Yourkit java profiler, or debug application logs can help identify extra database calls.  The screen capture below shows how Yourkit is able to reveal SQL statements behind PreparedStatement calls.

Yourkit Demo: http://www.yourkit.com/docs/demo/JavaEE/JavaEE.htm

j2ee_statistics

Prepared statement – use binding

DON’T: Never concatenate values – DO: use binding instead.  Besides helping to reduce security concerns with SQL injection, concatenation results in reparsing of SQL statements by the database.  You could also lose the benefits of any PreparedStatement caching by the jdbc drivers.

Service Script vs. Java services

Service Scripts perform slower than java services.  There is an overhead on scripting that comes from xml manipulation and xpath evaluation.  Lots of moves, complicated XPath - proportional to amount of XPath.  Here are some tips:

·         One complicated XPath expression should be faster than several smaller ones – the overhead is in the setup.

·         Smaller documents will process faster – think about that when designing script schemas – only send what you need.

Java-To-COBOL Interaction Overhead

DO: For optimal performance, avoid Java-to-COBOL and COBOL-to-Java calls.  Jumping between Java-to-COBOL interactions incur additional overhead that could lead to performance penalties.  The framework must execute additional flushes and object re-reads (since Java has to sync itself with COBOL’s objects).  These reads will only need to be made if the session is “dirty,” meaning modifications have been done by COBOL.

The framework will issue a log entry whenever the flush called is made such as in the following entry.  In some cases when we find that a session is not dirty, the log entry may be written, but no flush acutally occurs.  For example, the following log entries were issued in less than 1 millisecond duration and indicates that no flush occurred.

16:33:01,742 [CobolThread 2] INFO  (com.splwg.base.support.context.FrameworkSession) Issuing flush

16:33:01,742 [CobolThread 2] INFO  (com.splwg.base.support.context.FrameworkSession) Issuing flush

 

The following are general recommendations related to COBOL:

·         We recommend include rewriting COBOL code in Java

·         If the maintenance object is written in COBOL, keep the plug-in algorithms in COBOL as well.

Java Performance Patterns

·         Loop over entryset of a hashmap, not the entities

·         Concatenate strings using StringBuilder

·         Use Findbugs – it will help expose patterns to be avoided.

Batch Performance

Commit Considerations

DON’T: Do not commit too frequently.  For example, we do not commit ever record since each commit has overhead at the database; however, sessions with lots of objects in the cache should commit more frequently.  Adjust your default value accordingly.

Clustered vs. Distributed Mode Performance:  Clustered Is Preferred

No coding changes required for clustered mode and no reason to use the distributed mode anymore.

 

Clustered mode was created for stability, not performance; however, clustered mode should have less overhead because “tspace” table is not continually accessed.  This tspace table stored the batch job’s instructions and information used by the distributed mode and accessed by the batch threads.  There have been cases where this table is in high contention.

Use ThreadIterationStrategy

The ThreadIterationStrategy idea is for each thread to only get one piece ThreadWorkUnit at a time.  Regardless of volume, threads start executing almost immediately – unlike other strategies where lots of data can be selected in getJobWork(), leading to out-of-memory conditions if there is too much work.

Here are some technical highlights about how the ThreadIterationStrategy works:

·         The Thread worker class creates Query and QueryIterator and iterates over that – similar to how COBOL uses cursors

·         Frequent commits can slow down these types of batch jobs.  Each commit closes the database connection and query iterators, so it requires a new Query and QueryIterator each time.

·         Flat file processing works.

·         The specified commit frequency is adhered to.

·         There is no need to construct ThreadWorkUnit per record in getJobWork()

Data Explorer

It is important to understand that Data Explorers process ALL records returned from the database, even if they are not displayed.  For example, FK ref info strings, BS calls, SS calls, Inhibit Row in Explorers – all can cause per-row processing even if they are not displayed. 

Data Explorers are rendered using JavaScript.  They are not designed to display many records, and trying to do so will result in possibly unacceptable performance. DO: Consider limiting the results returned and asking Users to add additional filter criteria to narrow down the results.  DON’T: Don’t try to display hundreds of records.

Zone Configuration

·         DO: Consider limiting the number of rows retrieved by the database limiting the query size.  Specify this on the zone parameter and the query will use the “rownum” technique to restrict the number of rows returned.

·         DO: As a rule-of-thumb: 10 columns (even if not visible) in a data explorer zone should be an alert to really think about performance implications.

·         DO: Try to perform all processing in the SQL instead fkInfo, BS or SS calls in other columns.  As described before these would be additional processes run on a per-row basis.

Table Indices and SQL

Here are some more common patterns to look out for.  (This is not meant to be a complete SQL tuning guide.)

·         Put Indexes on the most commonly used criteria.  If there is no proper index, the optimizer does a full table scan.  Consider:

·         Primary keys, foreign keys, ORDER BY fields.

·         Secondary Unique Indexes

·         DO: Use a JOIN instead of EXISTS.  This is faster for unique scan indexes.

·         DO: Use EXISTS instead of IN when working with ID fields, use ‘=‘ instead of LIKE.  Using LIKE on a system-generated key isn’t “reasonable”

·         CONSIDER: Using functions like TO_DATE(), SUBSTR() etc. means indexes on those fields won’t be used!  Use only when necessary.

·         DO: Use the power of optional filters – and not just in the WHERE clause.

FROM d1_tou_map tm, d1_tou_map_l tml

FROM d1_tou_map tm,  [(F1) d1_tou_map_l tml,]

 

·         DO: Only include necessary tables:

SELECT A.usg_grp_cd, A.usg_rule_cd, A.exe_seq,A.referred_usg_grp_cd,A.usg_rule_cat_flg, B.crit_seq, C.descr100 DESCR

FROM D1_USG_RULE A, d1_usg_rule_elig_crit B, d1_usg_rule_elig_crit_l C

WHERE A.usg_grp_cd= :H1

AND A.usg_grp_cd = B.usg_grp_cd

AND A.usg_rule_cd = B.usg_rule_cd

AND b.usg_grp_cd = C.usg_grp_cd

AND b.usg_rule_cd = C.usg_rule_cd

AND b.crit_seq = C.crit_seq

AND C.language_cd= :language

 

Note that Table B is not necessary; you could instead simply link directly from A to C.

·         Offload tasks, like string manipulations, concatenations, row numbering, case conversions, type conversions etc., to the front-end applications

·         Do basic validations in the front-end during data entry.  This saves unnecessary network roundtrips.

·         Avoid using UNION - use UNION ALL if it is possible.

·         Operators <>  and  !=  will NOT use the index!  Also the word “NOT” Use the Greater Than or Less Than operators.

select * from ci_scr_step where  (scr_cd <> 'ZZCW03') has cost 68

select * from ci_scr_step where (scr_cd > 'ZZCW03' or scr_cd < 'ZZCW03') has cost 1!!!

UI Maps and BPAs

UI maps will not be able to display many rows very quickly.  DONT display hundreds of rows in a UI Map.  Alternatively, the zone type “SERVICE” can display a large number of records faster.

DO: Ensure that the html code is proper.  Malformed HTML in UI maps (for example, opening a <div> and not closing it) can cause significant performance degradations at the browser.  It is possible to copy and paste HTML into Eclipse to check its validity.  There are also various tools like html tidy that can help to identify bad html.

DO: Minimize browser-to-server calls.  Namely, invokeBO/BS/SS will perform a call to a server to retrieve the data, which can be slow.  Many of these such calls on load of the UI Map will result in slow performance.

·         Use service script instead of BPA if multiple calls need to be made to BO, BS, SS. 

·         Create a “bulk” processing service script instead of calling the same one multiple times. Instead of multiple invokeBS calls on load of a UI map, write a pre-processing service script instead.

Diagnosing Performance Issues

Execution times can be obtained in a number of ways.

Fiddler

In a UI-oriented scenario, the first recommended analysis tool is to use an http logger like fiddler (http://www.fiddler2.com).  This tool should make it apparent if there are excessive calls from the client browser to the server and the server response times as seen from the browser.  The timings can then be categorized as server-side or client side calls.  When using fidder be sure to enable the following:

·         "Time-to-Last-Byte"

·         "Response Timestamp"

OUAF “Show Trace” button

Enable debug mode by adding debug=true to the url.  Then use the  “Start Debug”, “Stop Debug” and “Show Trace” buttons

Log Service times in spl_service.log

in log4j.properties, add the following to log service execution times:

log4j.logger.com.splwg.base.api.service.ServiceDispatcher=debug

Optimization and Performance Profiling

To squeeze every second of a given program for mission critical optimizations, it may be necessary to craft a repeatable unit test and profile the results using a profiling tool such as YourKit (www.yourkit.com).  This section will include some code samples to log execution times.  Attaching a profiler could give clues to optimization points.  A common pattern to follow in testing code is to allow the System to “warm up,” for example to load up the necessary application caches which are only done once and are not relevant to the code being optimized.

Basic Logging

The following code can be placed in a junit test to log execution times:

long start = logger.debugStart("Starting process");

//... code for process

logger.debugTime("End process", start);

Timing code ("shootout"):

The code below will run a BO Update 100 times and report the amount of time taken.  Note the 5 “warmup” executions before the repeated 100 runs.

    public void testMultiplePluginScripts() throws Exception {

        String docString1 = "<DR_ShortCreateIntervalRecords><factId>219250542869</factId><longDescr>REEE</longDescr></DR_ShortCreateIntervalRecords>";

        Document doc1 = DocumentHelper.parseText(docString1);

 

        String docString2 = "<DR_ShortCreateIntervalRecords2><factId>219250542869</factId><longDescr>REEE</longDescr></DR_ShortCreateIntervalRecords2>";

        Document doc2 = DocumentHelper.parseText(docString2);

 

        // warmups

        for (int i = 0; i < 5; i++) {

            BusinessObjectDispatcher.execute(doc1, BusinessObjectActionLookup.constants.FAST_UPDATE);

            rollbackAndContinue();

            BusinessObjectDispatcher.execute(doc2, BusinessObjectActionLookup.constants.FAST_UPDATE);

            rollbackAndContinue();

        }

 

        long totalElapsed = 0;

        // speed

        for (int i = 0; i < 100; i++) {

            long start = System.nanoTime();

            BusinessObjectDispatcher.execute(doc1, BusinessObjectActionLookup.constants.FAST_UPDATE);

            flush();

            totalElapsed += System.nanoTime() - start;

            rollbackAndContinue();

        }

        System.out.println("Script (100): " + totalElapsed / 1000000 + "ms");

 

        totalElapsed = 0;

        for (int i = 0; i < 100; i++) {

            long start = System.nanoTime();

            BusinessObjectDispatcher.execute(doc2, BusinessObjectActionLookup.constants.FAST_UPDATE);

            flush();

            totalElapsed += System.nanoTime() - start;

            rollbackAndContinue();

        }

        System.out.println("Java (100): " + totalElapsed / 1000000 + "ms");

}

Using PerformanceTestResult helpers

A performance helper suite of classes was introduced, allowing "shoot-out"s like the above to be more simple:

          Callable<Void> exprCallable = new Callable<Void>() {

            @Override

            public Void call() throws Exception {

                expression.value(context);

                return null;

            }

        };

        Callable<Void> javaCallable = new Callable<Void>() {

            @Override

            public Void call() throws Exception {

                function(x);

                return null;

            }

        };

        PerformanceTestCallable exprPerfCallable = new PerformanceTestCallable("Expression "

                + expression.getExpressionString(), exprCallable);

        PerformanceTestCallable javaPerfCallable = new PerformanceTestCallable("Java", javaCallable);

 

        PerformanceTestResult compareResult = PerformanceTestHelper.compare(20, 200000, exprPerfCallable,

                javaPerfCallable);

        compareResult.printResults();

 

The API is com.splwg.base.api.testers.performance.PerformanceTestHelper:

    public static PerformanceTestResult compare(int warmups, int reps, PerformanceTestCallable... callables)

            throws Exception {

 

Each of the performance callables is treated the same.  It gets a series of warmup executions, in order to populate caches, and allow hotspot JVM optimizations of any methods.  Then the accurate system nano timing (i.e., System.nanoTime()) is called around the loop of the given number of reps.

Profiling:

The code below uses YourKit’s controll classes to create a snapshot.

    public void testProfilePluginScripts() throws Exception {

        String docString = "<DR_ShortCreateIntervalRecords><factId>219250542869</factId><longDescr>REEE</longDescr></DR_ShortCreateIntervalRecords>";

        Document doc = DocumentHelper.parseText(docString);

 

        // warmups

        for (int i = 0; i < 5; i++) {

            BusinessObjectDispatcher.execute(doc, BusinessObjectActionLookup.constants.FAST_UPDATE);

            rollbackAndContinue();

        }

 

        Controller controller = new Controller();

        controller.forceGC();

        controller.startCPUProfiling(ProfilingModes.CPU_SAMPLING, Controller.DEFAULT_FILTERS);

        for (int i = 0; i < 500; i++) {

            BusinessObjectDispatcher.execute(doc, BusinessObjectActionLookup.constants.FAST_UPDATE);

            rollbackAndContinue();

        }

        controller.captureSnapshot(ProfilingModes.SNAPSHOT_WITHOUT_HEAP);

    }

 

PerformanceTestHelper API

As before, the PerformanceTestHelper helps by providing a seamless interface into the yourkit profiler, for various options of sampling, tracing, monitoring threads or timing in threads:

    public static PerformanceTestCallableResult profileSample(int warmups, int reps, PerformanceTestCallable callable)

            throws Exception {

    public static PerformanceTestCallableResult profileTrace(int warmups, int reps, PerformanceTestCallable callable)

            throws Exception {

 

    public static PerformanceTestCallableResult monitor(int warmups, int reps, int numThreads,

            PerformanceTestCallable callable) throws Exception {

 

    public static PerformanceTestCallableResult timeInThreads(int warmups, int reps, int numThreads,

            PerformanceTestCallable callable) throws Exception {

 

The PerformanceTestHelper utility class uses reflection to know whether the yourkit library is available or not. If it is not available (such as on the build server), the behavior reverts to simple timing protocols of the corresponding callable iterations. If it is available (such as on a developer's workstation, and they want to profile a test), then the yourkit profiler is connected to.  This would require actually running the test under a profile session, else an error is produced.

Profiling a callable is somewhat similar to the simple timing of a callable, except for some added steps:

1.     Performs some warmups

2.     Forces garbage collection via the yourkit API

3.     Starts the given profile type (sample vs trace)- the test should be run without automatically starting the profiler

4.     Wrap the repetition loop in a timer

5.     Capture a snapshot

 

This design approach allows profile/performance tests to be checked into version control, for re-profiling at a later point, and for documentation examples of how to profile code, etc.

References

SQL tips:

//documentation/Dev Doc/Database_Documentation/SQL Development and Tuning Best Practices.doc

Hibernate fetching strategies:

http://docs.jboss.org/hibernate/stable/core/reference/en/html_single/#performance-fetching

Yourkit profiling demo:

http://www.yourkit.com/docs/demo/JavaEE/JavaEE.htm