Running Multivariate Tests (MVT) for Email channel
Important: This feature is available only for campaigns created with the Email Message Designer.
After you create an Multivariate Test (MVT) campaign, you can run the test from Program using the Send email campaign activity.
Tip: In the Program Designer, 
indicates an MVT campaign.
You can test any number of versions, and select the sample size for testing. Note that you can run an MVT campaign without testing. In this case, the default version of the campaign is sent.
About winner selection
You can select the winning version either automatically or manually. After the winning version is selected, it will be used for future launches. If the winning version is deleted for any reason, the default version will be used.
About automatic and manual winner selection
If you choose automatic winner selection, Oracle Responsys automatically selects the winning version based on the campaign metric you select:
- Open rate
The number of messages opened. - Click-through rate
The number of messages that result in the recipient clicking a link in the message. - Conversion rate
The number of messages that result in the recipient making a purchase. - Unsubscribe rate
The number of messages that result in the recipient unsubscribing. - Unique open rate:
The total number of unique opens by a recipient based on the delivered email campaign message. - Responder rate
The rate of unique responders where responders is the number of recipients who clicked on one or more links (i.e. unique individuals).
After the winner is selected, Program will send a notification to the specified email address.
Manual winner selection
If you choose manual winner selection, you select the winning version yourself.
With manual winner selection, you wait for the test to run, examine the results, then designate the winning version. When the test completes and you need to pick a winner, Program will send a notification to the specified email address. If you do not pick a winner within 7 days, the default version will be used.
Running an MVT test
You specify the MVT test options in the properties dialog.
To run an MVT test:
- Add a Send email campaign activity to the program and double click it to open the properties dialog.
- In the Campaign section:
- Click Select and select the MVT campaign you want to use, then click OK.
Tip: MVT campaigns are indicated by an
icon in the list. - Select the Enable multivariate testing checkbox.
If you do not select this checkbox, the campaign will run in regular mode using the default version.
- Click Select and select the MVT campaign you want to use, then click OK.
- Click the Multivariate testing section to specify test options.
- To terminate any previously running tests when this test starts, make sure the Start a new test checkbox is selected.
If this checkbox is not selected, the winning version from the previous test will be used if one is already selected. If a test is currently running, it will continue to run.
- Select when the test starts:
- Start on publish—Starts the test when the program is published.
- Start on date/time—Starts the test on a specific date and time.
- Select either Automatic or Manual winner selection mode.
For more information, see About winner selection.
- Select the metric for choosing the winner:
- Open rate–The rate of opened messages to delivered messages.
- Click-through rate–The rate of click throughs to delivered messages.
- Conversion rate–The rate of conversions to delivered messages.
- Unsubscribe rate–The rate of unsubscribe requests to delivered messages.
Unique open rate-The total number of unique opens by a recipient based on the delivered email campaign message.
- Responder rate-The rate of unique Responders to delivered messages where Responders is the number of recipients who clicked on one or more links i.e., unique individuals.
- In Total sample %, type the percentage of your target audience to test.
Oracle Responsys distributes the sample equally between all versions you are testing.
- Select the checkboxes of the versions you want to test.
Note: If you delete a version while the test is running, the default version is used instead and the test results will exclude the deleted version.
Tip: To preview each version, click
. - In the Duration section, specify when the MVT test should end:
- Time: With this option, Program ends the MVT test after the amount of time you have indicated. For duration, you can specify 1-23 hours. Note that hourly tests are limited to one test a day per campaign within a program stage. This means that although all tests will run, results will be accurate only for the first test. However, you can test several campaigns within a program, and you can test the same campaign in multiple programs.
- Significance: With this option, Program ends the MVT test when enough behavioral information has been collected from the campaigns being tested. Statistical significance determines if the difference in version scores is significant enough to rule out random chance as the reason for the difference. Statistical significance can potentially mean using a smaller number of recipients to test because the MVT test might determine statistical significance well before the timeout. In the Days text box, enter when the MVT test should timeout if it doesn't determine a statistically significant winner.
Important: MVT with statistical significance is currently under measured deployment and will be rolled out over time. To request access to this feature, please log in to My Oracle Support and create a service request.
- In the Email notification field, enter the email addresses to receive notifications about the test.
Separate multiple addresses by commas or semicolons.
- Click Done.
Analyzing test results and selecting a winner
To see the test results, double-click on the Analyze tab. The Test Results page shows details of the test and the winning version if one was automatically selected. To select the winner manually or to change the selection, select Designate as winner of the version you want.