You can view the results of an experiment by clicking on the experiment name either in the Completed Experiments or In Progress Experiments pages, or by clicking on the Results button on the Experiment Configuration page.
This opens the Experiment Configuration screen with the Results panel for the most recent execution of the experiment on display.
Note: The Results panel was upgraded for the version of Commerce Cloud released in April 2017. Experiments executed using a previous version of Commerce Cloud will display the results in the format used by the relevant version of Commerce Cloud. For more information on the results displayed by previous versions of Commerce Cloud, please contact Oracle support and ask for the relevant documentation.
A result set is created each time an experiment runs. You can select which result set to display by clicking on the expand arrow for the relevant result set.
As you scroll through the result set you can return to the top of the result set by clicking on the Back to Top arrow at the bottom of the Results panel.
If the experiment is currently in progress you can refresh the data displayed in the Results panel by clicking on the refresh button in the top right of the Results panel.
The results panel displays the following information:
Winning Goal Tile: This tile displays the name of the primary goal for the experiment, and the name of the variant that won in the result set. The winning variant is the one which showed the greatest percentage improvement for the primary goal among the variants which have a confidence level greater than 95%. If there were no variants with a confidence level greater than 95% that showed an improvement on the control for the primary goal, then “No Winner” is displayed in this tile.
Visitor & Days Run Tile: This tile displays the total number of visitors who participated in the experiment and how many days the experiment ran for.
Targeting Tile: This tile displays traffic allocation and the audience that was defined for the experiment.
High Level Goal Tiles: There is a separate high level goal tile displayed for each custom goal defined for the experiment and for each of the commerce metrics for the experiment. The high level goal tile for the primary goal is indicated with a gold star beside the goal name.
For any custom goals and for the Add to Cart and Conversion commerce metric this tile displays the Engagement Percentage value. The Engagement Percentage is how many visitors met the criteria for the goal.
For the Average Order Value commerce metric this tile displays the average order value for the control and each variation. This is calculated by dividing the total revenue of all completed orders for the specified currency for the experiment variation by the number of completed orders in the specified currency for the experiment variation.
For the Revenue per Visitor commerce metric this tile displays the revenue per visitor for the control and each variation. This is calculated by dividing the total revenue for the specified currency for the experiment variation by the number of visitors who participated in that experiment variation.
The Average Order Value and Revenue per Visitor commerce metrics are calculated separately for each currency used for completed orders. A separate tile is displayed for these metrics for each currency used.
Goal Detail Tiles: There is a separate goal detail tile displayed for each custom goal defined for the experiment and for each of the commerce metrics for the experiment. The goal detail tile for the primary goal is indicated with a gold star beside the goal name.
Each of the goal detail tiles includes the chart shown in the high level goal tile for that goal. You can choose whether to show or hide these charts by clicking on the arrows to toggle between show and hide.
Each goal detail tile also displays a table showing the results gathered for that goal during the execution of the experiment.
For any custom goals and for the Add to Cart and Conversion commerce metric this table displays the following fields:
Engagement Percentage: This is the percentage of visitors who met the criteria for this goal in the control and each variation.
Differential: This is the difference between the engagement percentage for the relevant variation and the engagement percentage for the control. A positive value means the engagement percentage was higher for the variation and a negative value means the engagement percentage was higher for the control.
Percentage Improvement: This compares how the frequency with which the goal occurs in the variation against the frequency with which it occurs in the control for the experiment. If the goal occurred more frequently in the variation than in the control, this figure is green. If the goal occurred an equal number of times in the variation and the control, the figure is black. If the goal occurred less frequently in the variant than in the control, the figure is red.
Confidence Level: This is a rounded percentage figure which shows the statistical significance level for each goal. It is measured against a set statistical significance level of 95%. A value of 94% or lower is displayed in red and 95% or above is displayed in green.
Engagement: This is the number of visitors who met the criteria for the goal in the control and each variation.
Visitors: This is the number of visitors who participated in the experiment in the control and each variation.
Traffic Allocation: This is the traffic allocation defined in the experiment for the control and each variation.
For the Average Order Value metric, the table in the goal detail tile displays the following fields:
Average Order Value: This is calculated by dividing the total revenue of all completed orders for the specific currency for the experiment variation by the number of completed orders for the specific currency for the experiment variation.
Differential: This is the difference between the average order value for the control and the average order value for each of the variations. A positive value means the average order value for the variation was higher and a negative value means the average order value for the control was higher.
Percentage Improvement: This is a percentage figure comparing the performance in the Average Order Value metric for the control and each variation. A positive value means the variation performed better and a negative value means the control performed better.
Total Revenue: This is the total revenue for all completed orders in the specified currency for the control and each variation.
Number of Orders: This is the number of completed orders in the specified currency for the control and each variation.
Traffic Allocation: This is the traffic allocation defined in the experiment for the control and each variation.
For the Revenue per Visitor metric, the table in the goal detail tile displays the following fields:
Revenue Per: This is calculated by dividing the total revenue in the specified currency by the number of visitors who participated for the control and each variation.
Differential: This is the difference between the revenue per visitor for the control and the revenue per visitor for each of the variations. A positive value means the revenue per visitor for the variation was higher and a negative value means the revenue per visitor for the control was higher.
Percentage Improvement: This is a percentage figure comparing the performance in the Revenue per Visitor metric for the control and each variation. A positive value means the variation performed better and a negative value means the control performed better.
Total Revenue: This is the total revenue for all completed orders in the specified currency for the control and each variation.
Visitors: This is the number of visitors who participated in the control and each of the variants.
Traffic Allocation: This is the traffic allocation defined in the experiment for the control and each variation.
The results sets for any previous executions of the experiment are displayed as an accordion view ordered by End Date with the most recent result set at the top of the list. You can expand or collapse these result sets by clicking on the chevron beside the start and end dates for the result set. Expanding a result set will display the result data for that execution of the experiment.
