Access Tester generates statistics file along with the target output script and execution log.
This section provides the following information:
You can evaluate test results using the "results bundle" that gets generated at the end of a test run. This bundle contains three documents: target script, execution log, and execution statistics.
Target script is an XML document containing new test cases. The matching pair of test cases in the original and target scripts shares the test case ID. This ID is represented by a UUID value, which makes it possible to compare individual test cases in the original script with those in the target script. For more information, see "Generated Input Test Script".
Note:
The target script is not created if the Access Tester is configured to run in multi-threaded mode.
The execution statistics document contains the test metrics, summary and detail statistics, and a list of test cases that did not match. The detailed statistics can be used for further analysis or to keep a historical trail of results. The summary statistics are the same statistics displayed at the end of the test run and can be used to quickly assess the state of a test run.
The list of mismatched test cases as created in the statistics document contains test case IDs that have triggered mismatch and includes the reason for the mismatch, as seen in Table 26-14.
Table 26-14 Mismatched Results Reasons in the Statistics Document
| Reason for a MisMatch | Description |
|---|---|
|
Result |
The test cases did not match because of the difference in OAM Server response codes (Yes versus No). |
|
Content |
The test cases did not match because of the differences in the specific data values that were returned by the OAM Server. The specific values from the last test run that have triggered the mismatch are included. |
The Saved Connection Configuration File is the output file that is saved using the Save Configuration command on the File menu; the default file name is config.xml.
This connection configuration file includes details that were specified in the Access Tester Console, Server Connection panel.
Note:
An input test script file is also generated as described in the following topic. The name of the configuration file is used in the input test script to ensure that running the Access Tester in command line mode picks up connection information defined in the connection file.
Connection Configuration File
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<oamtestconfig xmlns="http://xmlns.example.com/idm/oam/oamtest/schema"
version="1.0">
<connection timeout="30000" minnconn="1" mode="open">
<agent password="00030d05101b050c42" name="agent1"/>
<keystore rootstore="" keystore_password="" keystore=""
global_passphrase=""/>
<primary>
<server maxconn="1" port="2100" addr="oam_server1"/>
</primary>
<secondary>
<server maxconn="1" port="0" addr=""/>
</secondary>
</connection>
<uri getauthscheme="true">
<scheme>http</scheme>
<host>oam_server1</host>
<port>7777</port>
<resource>/index.html</resource>
<operation>Get</operation>
</uri>
<identity>
<id>admin1</id>
<password>00030d05101b050c42</password>
<certstore></certstore>
<ipaddr>111.222.3.4</ipaddr>
</identity>
</oamtestconfig>
The input test script is generated by using the Access Tester and capturing your own test cases. The "configfile" attribute of the "Control" element is updated after creation to specify the connection configuration file to be used in command-line mode for establishing a connection to the OAM Server.
Generated Input Test Script
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<oamtestscript xmlns="http://xmlns.example.com/idm/oam/oamtest/schema"
version="1.0">
<history description="Manually generated using agent 'agent1'"
createdon="2012-02-03T22:28:00.468-05:00" createdby="test_user"/>
<control numthreads="1" numiterations="1" ignorecontent="false"
testname="samplerun1" configfile="config.xml"/>
<cases numcases="4">
<case uuid="465a4fda-d814-4ab7-b81b-f3f1cd72bbc0">
<request code="Validate">
<uri getauthscheme="true">
<scheme>http</scheme>
<host>oam_server1</host>
<port>7777</port>
<resource>/index.html</resource>
<operation>Get</operation>
</uri>
</request>
<response elapsed="984" code="Yes">
<comment></comment>
<status>Major code: 4(ResrcOpProtected) Minor code:
2(NoCode)</status>
<content>
<line type="auth.scheme.id">LDAPScheme</line>
<line type="auth.scheme.level">2</line>
<line type="auth.scheme.required.creds">2</line>
<line type="auth.scheme.redirect.url">http://emerald.uk.example.com:14100/oam/server/</line>
</content>
</response>
</case>
<case uuid="009b44e3-1a94-4bfc-a0c3-84a38a9e0f2a">
<request code="Authenticate">
<uri getauthscheme="true">
<scheme>http</scheme>
<host>oam_server1</host>
<port>7777</port>
<resource>/index.html</resource>
<operation>Get</operation>
</uri>
<identity>
<id>weblogic</id>
<password>00030d05101b050c42</password>
<certstore></certstore>
<ipaddr>192.168.1.8</ipaddr>
</identity>
</request>
<response elapsed="187" code="Yes">
<comment></comment>
<status>Major code: 10(CredentialsAccepted) Minor code:
2(NoCode)</status>
<content>
<line type="user.dn">cn=weblogic,dc=uk,dc=example,dc=com</line>
</content>
</response>
</case>
<case uuid="84fe9b06-86d1-47df-a399-6311990743c3">
<request code="Authorize">
<uri getauthscheme="true">
<scheme>http</scheme>
<host>oam_server1</host>
<port>7777</port>
<resource>/index.html</resource>
<operation>Get</operation>
</uri>
<identity>
<id>weblogic</id>
<password>00030d05101b050c42</password>
<certstore></certstore>
<ipaddr>192.168.1.8</ipaddr>
</identity>
</request>
<response elapsed="188" code="Yes">
<comment></comment>
<status>Major code: 8(Allow) Minor code: 2(NoCode)</status>
<content/>
</response>
</case>
<case uuid="61579e47-5532-42c3-bbc7-a00828256bf4">
<request code="Validate">
<uri getauthscheme="false">
<scheme>http</scheme>
<host>oam_server1</host>
<port>7777</port>
<resource>/index.html</resource>
<operation>Get</operation>
</uri>
</request>
<response elapsed="172" code="Yes">
<comment></comment>
<status>Major code: 4(ResrcOpProtected) Minor code:
2(NoCode)</status>
<content/>
</response>
</case>
</cases>
</oamtestscript>
Here is an example was generated by running the Access Tester in command-line mode and specifying the script.xml file as input to execute the 4 captured test cases.
Dscript.scriptfile="script.xml" -jar oamtest.jar
Notice the various sections in the Example: Output File Generated During a Test Run. As shown in the execution log, this test run found no mismatches, and shows that 4 out of 4 requests matched.
Output File Generated During a Test Run
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<oamtestscript xmlns="http://xmlns.example.com/idm/oam/oamtest/schema"
version="1.0">
<history description="Generated from script 'script.xml' using agent 'agent1'"
createdon="2012-02-03T23:03:02.171-05:00" createdby="test_user"/>
<control numthreads="1" numiterations="1" ignorecontent="false"
testname="oamtest" configfile=""/>
<cases numcases="4">
<case uuid="465a4fda-d814-4ab7-b81b-f3f1cd72bbc0">
<request code="Validate">
<uri getauthscheme="true">
<scheme>http</scheme>
<host>oam_server1</host>
<port>7777</port>
<resource>/index.html</resource>
<operation>Get</operation>
</uri>
</request>
<response elapsed="969" code="Yes">
<comment></comment>
<status>Major code: 4(ResrcOpProtected) Minor code:
2(NoCode)</status>
<content>
<line type="auth.scheme.id">LDAPScheme</line>
<line type="auth.scheme.level">2</line>
<line type="auth.scheme.required.creds">2</line>
<line type="auth.scheme.redirect.url">http://emerald.uk.example.com:14100/oam/server/
</line>
</content>
</response>
</case>
<case uuid="009b44e3-1a94-4bfc-a0c3-84a38a9e0f2a">
<request code="Authenticate">
<uri getauthscheme="true">
<scheme>http</scheme>
<host>oam_server1</host>
<port>7777</port>
<resource>/index.html</resource>
<operation>Get</operation>
</uri>
<identity>
<id>weblogic</id>
<password>00030d05101b050c42</password>
<certstore></certstore>
<ipaddr>111.222.3.4</ipaddr>
</identity>
</request>
<response elapsed="187" code="Yes">
<comment></comment>
<status>Major code: 10(CredentialsAccepted) Minor code:
2(NoCode)</status>
<content>
<line type="user.dn">cn=weblogic,dc=us,dc=oracle,dc=com</line>
</content>
</response>
</case>
<case uuid="84fe9b06-86d1-47df-a399-6311990743c3">
<request code="Authorize">
<uri getauthscheme="true">
<scheme>http</scheme>
<host>oam_server1</host>
<port>7777</port>
<resource>/index.html</resource>
<operation>Get</operation>
</uri>
<identity>
<id>weblogic</id>
<password>00030d05101b050c42</password>
<certstore></certstore>
<ipaddr>111.222.3.4</ipaddr>
</identity>
</request>
<response elapsed="172" code="Yes">
<comment></comment>
<status>Major code: 8(Allow) Minor code: 2(NoCode)</status>
<content/>
</response>
</case>
<case uuid="61579e47-5532-42c3-bbc7-a00828256bf4">
<request code="Validate">
<uri getauthscheme="false">
<scheme>http</scheme>
<host>oam_server1</host>
<port>7777</port>
<resource>/index.html</resource>
<operation>Get</operation>
</uri>
</request>
<response elapsed="171" code="Yes">
<comment></comment>
<status>Major code: 4(ResrcOpProtected) Minor code:
2(NoCode)</status>
<content/>
</response>
</case>
</cases>
</oamtestscript>
The statistics file (_stats.xml) is generated together with the target output script during the test run identified in the Execution log.
The script.xml file was used as input to execute the 4 captured test cases. The test run found no mismatches, and shows that 4 out of 4 requests matched.
A sample statistics document is shown in the Example: Sample Statistics Document. The various sections that provide statistics for this run, which you can compare against statistics for an earlier "known good" run.
Sample Statistics Document
A sample statistics document is shown here. Notice,
<oamteststats xmlns="http://xmlns.example.com/idm/oam/oamtest/schema"
version="1.0">
<history description="Generated from script 'script.xml' using agent
'agent1'" createdon="2012-02-03T23:03:02.171-05:00" createdby="test_user"/>
<summary>
<total>
<nummatched>4</nummatched>
<numtotal>4</numtotal>
<avgelapsedsource>238</avgelapsedsource
<avgelapsedtarget>232</avgelapsedtarget>
</total>
<validate>
<nummatched>2</nummatched>
<numtotal>2</numtotal>
<avgelapsedsource>578</avgelapsedsource>
<avgelapsedtarget>570</avgelapsedtarget>
</validate>
<authenticate>
<nummatched>1</nummatched>
<numtotal>1</numtotal>
<avgelapsedsource>187</avgelapsedsource>
<avgelapsedtarget>187</avgelapsedtarget>
</authenticate>
<authorize>
<nummatched>1</nummatched>
<numtotal>1</numtotal>
<avgelapsedsource>188</avgelapsedsource>
<avgelapsedtarget>172</avgelapsedtarget>
</authorize>
<summary>
<detail>
<source>
<validate>
<yes>2</yes>
<no>0</no>
<error>0</error>
<mismatch>0</mismatch>
<elapsed>1156</elapsed>
</validate>
<authenticate>
<yes>1</yes>
<no>0</no>
<error>0</error>
<mismatch>0</mismatch>
<elapsed>187</elapsed>
</authenticate>
<authorize>
<yes>1</yes>
<no>0</no>
<error>0</error>
<mismatch>0</mismatch>
<elapsed>188</elapsed>
</authorize>
</source>
<target>
<validate>
<yes>2</yes>
<no>0</no>
<error>0</error>
<mismatch>0</mismatch>
<elapsed>1140</elapsed>
</validate>
<authenticate>
<yes>1</yes>
<no>0</no>
<error>0</error>
<mismatch>0</mismatch>
<elapsed>187</elapsed>
</authenticate>
<authorize>
<yes>1</yes>
<no>0</no>
<error>0</error>
<mismatch>0</mismatch>
<elapsed>172</elapsed>
</authorize>
<target>
</detail>
<mismatch numcases="0"/>
</oamteststats>
This sample execution log was generated together with the target output script during a test run using script.xml to execute 4 test cases.
The test run found no mismatches, and shows that 4 out of 4 requests matched.
As you review this example, notice the information provided which is the same as the information you see in the Status Messages panel of the Access Tester. Notice the test cases, test name, connection configuration file, agent name, connection status, request validation status, authentication scheme, redirect URL, credentials expected, authentication status and user DN, session ID, authorization status, validation status, and summary statistics. Also notice that the target script and statistics document were generated by this run.
Execution Log
[2/3/12 11:02 PM][info] Setting up to run script 'script.xml' [2/3/12 11:02 PM][info] Loading test cases and control parameters from script [2/3/12 11:02 PM][info] Loaded 4 cases [2/3/12 11:02 PM][info] Control data for this test run: [2/3/12 11:02 PM][info] Test name : 'samplerun1' [2/3/12 11:02 PM][info] Configuration file : 'config.xml' [2/3/12 11:02 PM][info] Ignore content : 'false' [2/3/12 11:02 PM][info] Loading server configuration from file [2/3/12 11:02 PM][info] Loaded server configuration [2/3/12 11:02 PM][info] Connecting to server as agent 'oam_agent1' [2/3/12 11:03 PM][info][request] Connect : Yes ... [2/3/12 11:03 PM][info] Test 'samplerun1' will process 4 cases [2/3/12 11:03 PM][info][request] Validate : Yes [2/3/12 11:03 PM][info] Authentication scheme : LDAPScheme, level : 2 [2/3/12 11:03 PM][info] Redirect URL : http://oam_server1.uk.example.com:2100/server/ [2/3/12 11:03 PM][info] Credentials expected: 0x01 (password) [2/3/12 11:03 PM][info][request] Authenticate : Yes [2/3/12 11:03 PM][info] User DN : cn=admin1,dc=us,dc=company,dc=com [2/3/12 11:03 PM][info] Session ID : -1 [2/3/12 11:03 PM][info][request] Authorize : Yes [2/3/12 11:03 PM][info][request] Validate : Yes [2/3/12 11:03 PM][info] Summary statistics [2/3/12 11:03 PM][info] Matched 4 of 4, avg latency 232ms vs 238ms [2/3/12 11:03 PM][info] Validate: matched 2 of 2, avg latency 570ms vs 578ms [2/3/12 11:03 PM][info] Authenticate: matched 1 of 1, avg latency 187ms vs 187ms [2/3/12 11:03 PM][info] Authorize: matched 1 of 1, avg latency 172ms vs 188ms [2/3/12 11:03 PM][info] Generated target script 'samplerun1_0302171__target.xml' [2/3/12 11:03 PM][info] Generated statistics log 'samplerun1_0302171__stats.xml'