Test Administrator gives access and quotas for environments on the cloud, test assets, and virtual user test limits for the Oracle Load Testing test driver.
Test Designer creates test assets (pre-packaged test scripts and scenarios) and test environments (templates for application-under-test, based on Oracle Assemblies) for users to consume.
Tester is the consumer of the test assets and test environments for testing created by the Test Designer. The Tester also plans, executes, and analyzes the test results.
Table 40-1 Tasks in TaaS
| Task | How To | Role |
|---|---|---|
|
Ensure Prerequisites Are Met |
Test Administrator, Test Designer, Tester |
|
|
Add application |
Test Administrator |
|
|
Define Test Drivers |
Setting Up Oracle Load Testing Test Drivers |
Test Administrator |
|
Set Up Quotas |
Test Administrator |
|
|
Publish Test Environments and Test Assets |
Test Administrator |
|
|
Set Up Chargeback Service |
Test Administrator |
|
|
Create Test Asset |
Creating Test Assets - General Information Creating Test Asset - Oracle Load Testing |
Test Designer |
|
Create Test Environments |
Test Designer |
|
|
Verify Available Test Assets and Environments |
Tester |
|
|
Create Tests |
Creating Tests - Oracle Load Testing Tests |
Tester |
|
Create and Run Trials |
Tester |
|
|
Monitor On-Going Trials |
Tester |
|
|
Compare Trial Results |
Tester |
|
|
View Activity and Usage Information |
Tester |
|
|
View Deployment Information |
Tester |
Ensure the following prerequisites are met when using the Testing Cloud Self Service Portal.
Before You Do Anything
Ensure Enterprise Manager Cloud Control is set up to manage the cloud environment.
Ensure Self Service Application Plug-in is installed because Testing as a Service is part of the SSA plug-in.
Prerequisites Specific to Test Designer
Has the privileges to access the Testing as a Service (TaaS) Test Designer options. This user must have the EM_TAAS_DESIGNER role.
Must have knowledge about what applications are to be tested using Enterprise Manager and their structure (such as database servers, application servers, and configurations for the servers.)
Has created test scripts using the Oracle Application Testing Suite scripting tool OpenScript, the Oracle Load Testing scenario, and the self-contained .scnzip file, and the Oracle Functional Testing scenario with its self-contained .zip file. For information on how to create and record scripts, refer to the Oracle Application Testing Suite documentation available by downloading the documentation set from:
http://www.oracle.com/technetwork/oem/downloads/index-084446.html
Must be familiar with Enterprise Manager Cloud Control features.
Has created Test Assets based on scripts and scenarios for Oracle Load Testing, Oracle Functional Testing, or other test drivers and Test Environments based on Oracle assemblies.
Has deploy privileges over the Cloud Zones, otherwise he cannot create deployment plans.
Has access to the Cloud Infrastructure.
Prerequisites Specific to Test Administrator
Has the privileges to access the Testing Cloud Self Service Portal Test Administrator options. This user must have the EM_SSA_ADMINISTRATOR role.
Must have knowledge about what applications are being deployed, managed, and tested using Enterprise Manager and their structure (such as database servers, application servers, and configurations for the servers.)
Must have basic DB administration knowledge and have a good understanding of virtualization technologies.
Must be familiar with Enterprise Manager Cloud Control features.
Ensures that the Test Designer has created Test Assets and Test Environments.
Ensures that the Testers have the privileges to access the Testing Cloud Self Service Portal user options.
Publishes the test assets, test environments, and test drivers to the Tester role and the EM_SSA_USER role.
Has deploy privileges over the Cloud Zones, otherwise he cannot create deployment plans.
Has access to the Cloud Infrastructure.
Prerequisites Specific to Tester and SSA Users
Must have the EM_SSA_USER role.
Ensures that the Test Designer has created Test Assets and Test Environments.
Ensures that the Test Administrator has assigned Test Assets, Test Environments, and Test Drivers to the Tester role (EM_SSA_USER).
Has the privileges to access the TaaS Tester options.
Has the privileges to access the Testing Cloud Self Service Portal user options.
Must be familiar with Enterprise Manager Cloud Control features.
Enterprise Manager allows TaaS Test Administrators to manage the application infrastructure for testing activities, define user activities for self-service testing, and manage test environments and test assets.
Specific responsibilities include:
Manage the test drivers.
Manage user roles and access to test drivers (specify which users have access to which plug-in.)
Manage access to test environments and test assets based on users and roles.
Assign the OVM and Zone privileges to the following roles and users:
OVM - View privilege to Test Designer and Test Administrator
Zone - Operator privilege to Test Designer and Test Administrator
Since TaaS cannot know before hand the zones and Oracle Virtual Machines present in user deployments, the Super Administrator, Cloud Administrator, or respective target owner will decide which zones and Oracle Virtual Machines are to be used.
The following tasks are performed by the Test Administrator.
The Testing Cloud Self Service Portal allows Test Administrators to manage the application infrastructure for testing activities, define user activities for self-service testing, and manage test environments and test assets. Specific responsibilities include:
Manage the test drivers.
Manage user roles and access to test drivers (specify which users have access to which plug-in.)
Manage access to test environments and test assets based on users and roles.
Assign the OVM and Zone privileges to the following roles and users:
OVM - View privilege to Test Designer and Test Administrator
Zone - Operator privilege to Test Designer and Test Administrator
Since TaaS cannot know before hand the zones and Oracle Virtual Machines present in user deployments, the Super Administrator, Cloud Administrator, or respective target owner will decide which zones and Oracle Virtual Machines are to be used.
Note: Ensure the prerequisites are met before using the Testing Cloud Self Service Portal (see Prerequisites).
Follow this process to set up application types to be tested.
A Test Driver is the package of software that Enterprise Manager uses to generate tests or workload against test systems based on a deployment plan.
Oracle provides two types of drivers: Oracle Load Testing and Oracle Functional Testing. The Oracle Load Testing test driver is used for load, scalability, and stress testing. This allows you to simulate hundreds or thousands of users using the software at the same time. Oracle Functional Testing test driver is used for functional and regression testing.
Since Oracle Load Testing and Oracle Functional Testing are created by Oracle, Oracle knows the specifics of the test drivers and provides seamless integration to these drivers.
You can register your own type of test drivers as long as you can build a deployment plan around your assembly which contains the testing product and software. However, when you create a custom test driver, you need to direct your scripts on how to launch and monitor your tests.
The Custom Test Driver allows Testers to use test tools from third party vendors together with TaaS. Use Custom Test Driver Types to define and register test-driver types which meet your specific testing needs.
The following sections explain how to register specific test drivers.
The Oracle Load Testing test driver is used for load, scalability, and stress testing. This allows you to simulate hundreds or thousands of users using the software at the same time.
Note: Ensure the prerequisites are met before setting up Oracle Load Testing Test Drivers (see Prerequisites).
To register an Oracle Load Testing test driver, follow these steps:
To view the details of a test driver that has already been registered, highlight the test driver in the table. The deployment plan, agent deployment plan, and assembly structure information are provided.
The Oracle Functional Testing Test Driver is used for functional and regression testing and validating application functionality.
Note: Ensure the prerequisites are met before setting up Oracle Load Testing Test Drivers and Oracle Functional Testing Test Drivers (see Prerequisites).
To register an Oracle Functional Testing test driver, follow these steps:
To view the details of a test driver that has already been registered, highlight the test driver in the table. The deployment plan and assembly structure information are provided.
The Custom Test Driver allows testers to use test tools from third party vendors together with TaaS. Use Custom Test Driver Types to define and register test-driver types which meet your specific testing needs.
Oracle Load Testing and Oracle Functional Testing are types of drivers, and since they are created by Oracle, we (Oracle) know the specifics of the drivers. However, when you create a test driver, you need to direct your scripts on how to launch and monitor your tests. You can register your own type of test drivers as long as you can build a deployment plan around your assembly which contains the testing product and software.
Note:
If you are using third party test automation tools, these tools need to be registered as Custom Test Drivers.
Ensure the prerequisites are met before setting up Oracle Load Testing Test Drivers and Oracle Functional Testing Test Drivers (see Prerequisites).
To register a custom test driver, follow these steps:
Log in to Enterprise Manager Cloud Control using Testing Cloud Self Service Portal Administrator credentials.
From the Setup menu, select Cloud, then select Testing. The Testing Cloud Self Service Setup page appears.
Click Custom Types located in the left panel. The Custom Test Driver Types page appears
Click Create.
On the Add Custom Test Driver Type dialog, provide the following information:
Enter a meaningful name and description.
The Working Directory is a folder created on your Test Driver Virtual Machine (VM) where the operations will occur.
The Results Directory is a folder created on your Test Driver Virtual Machine (VM) and where the results of your tests will appear, TaaS will zip up the entire folder, send it to Enterprise Manager.
Pre-Execution Script is a file which performs pre-execution steps for your test. Click Browse to locate the file.
Execution Script is a file which launches the test. Click Browse to locate the file.
Progress Monitoring Script is a script which TaaS will run on the test driver, to determine if the test is running or complete (and its outcome). Click Browse to locate the file.
When the test is complete, TaaS reports the outcome of the test. TaaS looks at the value returned by this script to determine the test outcome (passed, failed, warning).
Note: For TaaS to know the outcome of the trial, ensure that your script prints one of the following outcomes: passed, failed or warning, as the last line to the standard output.
Post-Execution Script is a file which performs the clean up steps after your test runs. Click Browse to locate the file.
Stop Execution Script is a script which TaaS will run on the test driver to stop the test execution.
Custom Parameters is a property file which contains the run-time parameters that will be passed to the test execution. You can configure these parameters during trial submission.
Click OK.
Once you provide these inputs, your type is defined and you can now register a new driver of this type.
To register a driver, on the Test Drivers page, select Custom from the Register menu.
Provide a meaningful name for this Test Driver.
Click the magnifying glass for selecting a deployment plan on which the test driver is based. In the resulting dialog box, select the Deployment Plan that should be used for the Custom test driver and click Select.
Enter a description for the Test Driver and click Register.
Select a Test Driver in the list then click on the component to drill down to the Deployment Plan details.
The Testing as a Service Portal allows you (Test Administrator) to Setup usage quotas for Testers by specific roles. To set up quotas, follow these steps:
Note: Ensure the prerequisites are met before setting up quotas (see Prerequisites).
The Testing as a Service (TaaS) Portal allows you (Test Administrator) to setup result file systems for Testers.
Use Result File Systems to store, manage, and access the Functional Testing results for a longer duration and without any size limitations. These results can also be accessed from your external web clients and servers using the http and nfs protocols.
Note: Ensure the prerequisites are met before publishing test environments and test assets (see Prerequisites).
To set up result file systems, follow these steps:
The Testing as a Service (TaaS) Portal allows you (Test Administrator) to publish test asset assets, test environments, and test drivers to selected roles so they can be used when creating tests.
Note: Ensure the prerequisites are met before publishing test environments and test assets (see Prerequisites).
To publish test environments, test assets, and test drivers, follow these steps:
Note: Ensure the prerequisites are met before setting up the chargeback service (see Prerequisites).
For information about Chargeback Service, refer to the Chargeback Administration chapter.
Follow this process to set up applications to be tested.
The following tasks are performed by the Test Designer:
A Test Asset is a collection of test artifacts (scenarios, scripts, tests) that can be executed on a test environment. A test asset defines the actions that need to be executed to reproduce the workload against the test environment. The Testing Cloud Self Service Portal allows the Test Designer to create Test Assets based upon the available Test Drivers defined by the Test Administrator.
The Test Designer should know the application under test and its relevant Key Performance Indicators (KPIs). The Test Designer should also know the business use-cases for the test scenarios being run. Specific responsibilities include:
The Test Designer should be skilled in Oracle OpenScript. If the Oracle Load Testing Test Driver is used, the Test Designer should be skilled in Oracle Load Testing. If a Custom Test Driver is used, the Test Designer should be skilled using the corresponding test scripting tool.
The Test Designer should be skilled as DBA.
Create Test Assets using the Testing as a Service portal.
Create Test Environments using the Testing as a Service portal.
If the Test Designer is required to create environments, the Test Designer must know how to create assemblies.
Note: Ensure the prerequisites are met before creating test assets (see Prerequisites).
Follow this process to create Oracle Load Testing test assets.
Follow this process to create Oracle Functional Testing test assets.
A Test Environment is a collection of entities that represents the entire application stack, thus an environment consists of the following elements:
Application under test (AuT): Contains the software bits and application data. For example, for a deployed Siebel application, end users can modify settings, configurations, and parameters of the application.
Infrastructure under test (IaT): These are entities the Application under test depends upon, for example, the cloud setup/infrastructure in a cloud deployment. Typically an infrastructure is something created for consumption of end users by administrators.
Note: Ensure the prerequisites are met before creating test environments (see Prerequisites).
Follow this process to create test environments.
Once the Test Administrator and Test Designer have performed their respective tasks, the Tester can now start testing.
The Testing Cloud Self Service Portal allows Testers to create tests from test assets and test environments to perform testing activities. Testers create tests using test assets and environments that have been assigned to them by the Test Administrator.
The Tester should know the application under test and its relevant Key Performance Indicators (KPIs). The Tester should also know the business use-cases for the test scenarios being run. Specific responsibilities include:
Running the tests by using the test assets and environments created by the Test Designer and assigned to them by the Test Administrator.
Reserving resources from the cloud, cancel upcoming reservations, update configurations of resources that have been reserved (CPU, memory, and so on).
Editing test scenarios set up by the Test Designer to specify run time test parameters and schedule the tests for execution.
Performing the test run analysis to compare various metrics that are relevant to the application and business use-case.
The following tasks are performed by the Tester.
Viewing Deployment Information (Browse My Deployments)
The Testing Cloud Self Service Portal allows Testers to verify which test assets and test environments have been assigned to them by the Test Administrator.
Note: Ensure the prerequisites are met before verifying available test assets and environments (see Prerequisites).
Follow this process to verify test assets and test environments assigned to your role.
The Testing Cloud Self Service Portal allows Testers to create tests from test assets and environments to perform testing activities. Testers create tests using test assets and test environments that have been assigned to them by the Test Administrator.
Note: Ensure the prerequisites are met before creating tests (see Prerequisites).
See the following section for the types of tests you can create:
Note: Ensure the prerequisites are met before creating Tests (see Prerequisites).
Follow this process to create Oracle Load Testing tests.
Note: Ensure the prerequisites are met before creating Tests (see Prerequisites).
Follow this process to create Oracle Functional Testing tests.
Note: Ensure the prerequisites are met before creating Tests (see Prerequisites).
Follow this process to create Custom tests.
The Testing Cloud Self Service Portal allows Testers to create and run trials from the test instances that have been created in the portal. The Tester runs trials on the test asset and environment, monitors ongoing trials, and analyzes the results of completed trials.
Note: Ensure the prerequisites are met before creating and running trials (see Prerequisites).
See the following section for the types of trials you can create and run:
Follow this process to create and run Oracle Load Testing trials:
Follow this process to create and run Oracle Functional Testing trials:
The Testing Cloud Self Service Portal allows Testers to monitor running trials and create charts of the key performance indicators for the application-under-test.
Note: Ensure the prerequisites are met before monitoring on-going trials (see Prerequisites).
Follow this process to monitor on-going tests.
The Testing Cloud Self Service Portal allows Testers to compare trial results and create charts of the key performance indicators for the application-under-test.
Note: This feature is only available for Oracle Load Testing Test Driver based trials.
Note: Ensure the prerequisites are met before comparing trial results (see Prerequisites).
Follow this process to compare trial results.
To avoid duplication of effort, you can share tests with other users. Note that the person who created and shared the tests will be charged for any resource usage associated with the shared tests.
To share tests, perform the following steps:
Log in to Enterprise Manager Cloud Control using Testing Cloud Self Service Portal Tester credentials. You are logged into Self Service Portal.
Click My Tests.
Click the My Tests tab in the Testing Cloud Self Service Portal.
Select a test from the list of Tests on the My Tests page.
Click Share.
Select a user to share the Test with.
Click Share.
Click Log Out.
To verify that the test is shared:
The Testing Cloud Self Service Portal allows Testers to view summary information for test assets and test environments available to you, your quota as defined by the Test Administrator for your role, your current utilization of hardware resources, and submitted and scheduled trials.
Note: Ensure the prerequisites are met before viewing activity and usage information (see Prerequisites).
Follow this process to view activity and usage information.
The Testing Cloud Self Service Portal allows Testers to create tests from test assets and test environments to perform testing activities using the Testing Cloud Self Service Portal. Testers create tests using test asset and environment that have been assigned to them by the Test Administrator. The Tester runs trials on the test asset and environment, monitors ongoing trials, and analyzes the results of completed trials.
Note: Ensure the prerequisites are met before viewing deployment information (see Prerequisites).
Follow this process to view activity and usage information.
The users HOME tab displays a summary of what the user has done, including resources consumed for testing and tests created by the user. From the HOME tab, users can quickly access previously executed test trials in order to compare results or to do more in depth post-run analysis.
The Testing home page provides information regarding the statistics of ongoing testing. General statistics are provided, as well as test assets and environments, ongoing test and trials, Oracle Load Testing, and the Infrastructure. It also provides an overview of the servers and infrastructure resources consumed by the user, like the number of servers provisioned, number of CPUs, as well as RAM and Storage.
When you log in as Test Designer, you only see the My Library tab. When you log in as Test Administrator, you see the Home tab. When you log in as a Super Administrator, you see both the Home and My Library tabs.
The sections on the Home tab are:
General - Total counts of various artifacts available in the system, for example, its “total tests present in system", “total trials ran across all trials", and so on.
Test Assets/Environments - Accumulation graph of test assets, environments, and machines in use in the Testing Cloud. Count and trend data are included. This is the Test Administrator's overview of the entire TaaS portal, regardless of Testers.
Tests and Trials - Daily totals of the number of tests created or daily totals of the number of trial runs.
Infrastructure - Daily consumption of infrastructure resources: CPU, Memory, and Disk space storage.
Oracle Load Testing (OLT) - Daily total of the number of OLT Virtual Users (VU) that were run. Oracle Load Testing uses OpenScript scripts to simulate users accessing an application under test.
The vertical axis shows daily usage totals based upon the View selection in each of the regions.
The My Library tab provides details on both Test Assets and Test Environments.
Log in to Enterprise manager using Tester credentials.
Click My Tests radio button.
Click the Home tab.
You will see a summary of test assets and test environments available to you, your quota as defined by the Test Administrator for your role, your current utilization of hardware resources, and the submitted and scheduled trials.
The Trial Page provides the Results, Test Environment, Configuration, and Performance information of the trial.
The top of the page provides the general information of the trial including: status of the trial, results of the trial, start time of the trial with the option to stop the trial, the test being run, and the test environment being used.
Results
Note: The Results tab is only for Oracle Functional Testing.
From the Results tab, you can:
Download the results to be viewed at another time.
Browse the results location
Generate a Report to be published in BI Publisher.
Test Environment
From the Test Environment tab, you can:
Start, stop, and restart the Guest Virtual Machines used for the test.
Start, stop, and restart the Test Driver Instance used for the test.
Configuration
From the Configuration tab, you can:
View the details of the application that was tested.
Test Driver used for the test.
List of the Test Asset Parameters used.
Schedule of the trial.
Performance
Note: The Performance tab is only available for Oracle Load Testing trials.
From the Performance tab, you can:
View all the collected Oracle Load Testing counters and Enterprise Manager target metrics.
Select interested counter(s)/metric(s) and show them in charts.
Summary
Note: The Summary tab is only available for Oracle Load Testing trials.
From the Summary tab, you can:
See the summary report of this trial and AWR report if it is available.
Generate Oracle Load Testing trial report and AWR report if it is available.
To facilitate the sharing of information about trials, TaaS provides reports using Business Intelligence Publisher (BI Publisher). These reports include the TaaS Oracle Functional Testing Trial Report and the TaaS Oracle Load Testing Trial Report.
To view these reports:
The Test Driver hardware resource requirement depends on factors such as: number of virtual users for your Trial, ramp-up rate, number of script steps, number of variables in the script and script type.
To ensure that TaaS provides the appropriate amount of resources for your test trial, TaaS provides a hardware estimation feature. TaaS calculates and creates agent machines based on estimation results of CPU and memory resources consumed by the test asset in your test trial.
Note:
To use the Hardware Estimation Available feature, you must have selected the Hardware Estimation Data option on the Export File screen when creating the scenario.zip file in the Oracle Load Testing application.
To take advantage of this feature, the Test Administrator and Tester need to perform the following steps:
Define an Agent deployment plan for an Oracle Load Testing Test Driver.
From Setup menu, select Cloud, then select Testing.
On the Testing Cloud Self Service Setup page, select Test Drivers in the navigation tree.
From the Register menu, select Oracle Load Testing.
Provide the information for Name, Deployment Plan, Version, and so on.
Ensure that the deployment plans you use are scalable.
View hardware estimation result in Test Oracle Load Testing Asset.
From Enterprise menu, select Cloud, then select Self Service Portal.
Select My Tests.
On the Testing Cloud Self Service Portal page, click the My Library tab.
Select Test Assets. Highlight the asset in which you are interested. The Scenario Configuration section states whether Hardware Estimation is available for this asset. An X denotes that the estimate is not available whereas a check mark denotes that it is.
To create a Test Asset, click Create and choose Oracle Load Testing. Provide the requisite information.
The scenario zip file is exported from the Oracle Load Testing application.
Note that it is through the scenario zip file that the Hardware Estimation Data feature is enabled. To learn how to create a scenario, refer to the Oracle Load Testing Load Testing User's Guide available from the 'Oracle Application Testing Suite Downloads' page: (http://www.oracle.com/technetwork/oem/downloads/index-084446.html).
By choosing the configurable parameters, you can change these settings when you create a trial using this asset.
Evaluate Test Driver with estimation resource result.
From the My Tests tab, select Create, then select Oracle Load Testing.
Provide the requisite information and ensure that the source (Test Asset) is one that has the Hardware Estimation Data feature.
Note: It is through the scenario zip file that the Hardware Estimation Data feature is enabled on the Export File page.
Also note that the Hardware Estimate region contains the recommended number of agent machines to use for testing the selected Test Asset.
Start an Oracle Load Testing Trial Using Estimation Resource Result
Click the My Tests tab. Highlight the test you want to use for the trial.
In the Trials section, select Create.
On the Create Trial screen:
- Provide the requisite information in the Overview and Application Under Test regions. Click Continue.
- For the Test Asset Parameters, if needed change the values for the Trial Virtual Users, Ramp-up Rate, and End Condition. Click Continue.
- Provide the parameters for the Test Driver, Hardware Estimate, and Schedule. Click OK.
Note that in the Hardware Estimate region, the number of agents is calculated by TaaS to meet the required resource needed for the trial. However, you can change the number of agents if you want.
View Agent Status in Trial Home
On the My Tests tab, highlight the trial of interest.
View Agent Status in My Deployment Tab.
On the My Deployments tab, highlight the deployment of interest. The Agent Status will be shown in the Agent tab.
Share/unshare the Agent between different test driver instances.