Campaign performance report
Important: Are you using our new Stats Engine? In October 2016 (16.5) we released our new Stats Engine. This topic tells you how to use the Campaign Performance report for Campaigns published after the October 2016 release. For campaigns published before the October 2016 release - read Legacy campaign performance report.
The Campaign Performance report shows your campaign's key metrics at a glance to help you understand the experience that gets you maximum conversion, engagement and revenue. You can use this report along with other reports to make key business decisions for your organization.
Note: Campaign Conclusion features are available for campaigns published after the October 2016 release. See campaign conclusion.
Campaign performance summary
The summary includes:
- Total Generations: The number of generations for the campaign.
- Live For: The total number of days where there has been at least one generation.
- Current Duration: The number of days from the start of the campaign to the current date, or the date the campaign finished.
- Experiences: The number of experiences in the test. A combination of variants make up an experience. For example: a welcome page without an image, a medium size banner, and a call-to-action button saying View Our Best Offers.
Top ranked experiences
The experiences card shows the top winning experiences compared to the control (the default variant). The control card (Ctrl) is always gray and its position changes based on how it ranks compared to other experiences.
You can use Options to quickly compare the top experiences for Conversion, Revenue, and Engagement against the control variant (see View reports based on revenue or engagement).
Experiences chart and table for the primary action
The interactive experiences chart shows conversion results for the top experiences and the control.
You can use Options to view results for revenue and engagement (see View reports based on revenue or engagement). By default the chart shows cumulative data, but you can view daily data to identify peaks on a particular day (see How my campaign performed on a daily basis).
The table shows detailed information for each experience based on the metric you select. You can use the check boxes to select up to 10 experiences (including the total) to appear on the chart. Click the column headings to sort the results in ascending or descending order. For more details about additional metrics, see Include more metrics in the Experiences table.
Experiences table for each secondary action
The report also shows results for any secondary actions specified in your campaign. Secondary actions support your primary action and allow you to gain more insight into how users interact with your website content. The table shows detailed information for each secondary action based on the metric you select.
View reports based on revenue or engagement
Click Options > View and Order Data By, choose from the metrics below and click Apply.
- Conversion (Rate): The percentage of Generations who have converted (i.e. performed the required action) out of all generators for that experience.
- Revenue per Generation: The revenue divided by generations for each experience.
- Average Order Value: The revenue divided by the multiple action count (i.e. the total number of actions by visitors who saw the experience).
- Revenue per Converter: Revenue/single action count. For example, for purchase actions, Revenue per Converter is the revenue per visitor who has completed at least one purchase through the campaign.
- Engagement: Also known as Actions per Generation. The average number of multiple actions per generation (usually visitors). It is calculated as the multiple action count (i.e. the total number of actions by visitors who saw the experience) divided by generations.
The top ranked experiences, chart and table are updated using the metric you select. For more details about additional metrics, see Include more metrics in the Experiences table.
Use funnel analysis to measure visitor conversion
You can use the conversion funnel to see how visitors are converting based on their actions.
Click Options > Page Widgets > select Funnel and click Apply.
The funnel shows visitor conversion for the control experience and the visitor drop-off points.
To create bespoke funnels, you can:
- Reorder the actions in the funnel by dragging and dropping the action name to the desired location.
- Add or remove actions from the funnel.
- Analyze how visitors who performed a particular action progressed through the funnel by changing the default starting point for the funnel (which is usually all visitors who saw the experience).
Compare the funnel for another experience with the funnel of the control experience
Click the Funnel menu > Compare Experiences, then select the check box for the experience.
The two funnels appear side by side showing the conversion rate for each action.
How to use action breakdown to gain more insight into visitor behavior
Use the action breakdown widget to track how visitors are converting based on action attributes. For example, you could set up an On-click action on different areas of a page and then track which part of the page gets the most clicks. Or you might set up attributes on purchase actions to store the product category.
Click Options > Page Widgets > select Action Breakdown and click Apply.
The pie chart shows the action breakdown for the primary action. In this example we see the action 'Confirmation' and two attributes showing how visitors paid for their purchase - using a 'credit card', 'debit card'.
Use the options menu to modify the action breakdown view:
- Compare Experiences - Use to compare the control (Ctrl) against other experiences in your campaign.
- Actions: Statistics for the primary action are displayed by default. Use this option to display details for a different action.
- Measure: Use to show the statistics for Conversion, Engagement or Revenue.
- Export: Use to export the data in one of the following formats: comma-separated values (.cvs), .xls or tab-separated values (.tsv).
View reports based on elements
Use the Elements view to evaluate the performance of test variants in terms of test Elements. You may find this option useful when viewing report data for MVT campaigns.
Click Options > View and Order Data By, choose Elements and click Apply.
The sample report table below shows the conversion rate for the variants inside each element.
- Element: The Element that was tested during the Campaign.
- Variant: The alternative content for the Element that was tested.
- Generations
- Action Count: The number of actions performed for the variant.
- Conversion Rate: The percentage of visitors who have converted out of all the visitors who have entered the campaign.
- Margin of Error: The standard deviation for the conversion rate (or other metric e.g. Revenue per Converter) of the experience multiplied by a factor that depends on the confidence level. For example, a conversion rate of 2.5% ± 0.5 means that the conversion rate can be between 2% and 3%.
- Uplift: The relative difference between a given variant and the control content. Uplift is calculated using the following formula: Uplift (%) = (Conversion rate of variant - Conversion rate of control variant) / (Conversion rate of control variant) x 100%.
- Confidence: Measures the observed test results and is used to test a statistical hypothesis on the equality between an experience and the control metric, e.g. conversion rates. The confidence is the probability of obtaining a result equal to or 'less extreme' than what was actually observed, assuming that the experience and the control have equal metric values, e.g. equal conversion rates.
How do I?
Disabled experiences are hidden by default. You can display the complete set of experiences to review data that was captured before the experiences were disabled.
Click Options > Show Disabled Experiences and click Apply.
The Top Ranked Experiences section, Experiences chart and table are updated. The row for the disabled experience is highlighted pink in the table.
Zero weight variants are hidden by default.
Click Options > Show Zero Weight Variants and click Apply. The Top Ranked Experiences section, Experiences chart and table are updated. The row for the zero weight variant is highlighted pink in the table.
You may want to view data that supports the primary action in your campaign.
Click Options > Actions, select the check box for the secondary action and click Apply.
To view the top ranked experiences for any of your secondary actions, select your secondary action from the Actions drop-down to update the data in the experiences card.
You can filter results by a specific date range.
- Click Filters > Date and select Filter by Action Date or Filter by Generation Date.
Filtering by generation date is useful when you want to ignore the first x number of days from your report. For example, the date range for your campaign is 1st -31st December but you want to ignore the first 7 days in your report.
Filtering by action date is useful when you want to see what your report looked like a few days ago. For example you want to review reporting data for a date range in the past. For details about filtering by dates, see Generation Date vs Action Date Filtering.
- Click on the calendar under Date Range and select the dates you require on the right.
- Click Apply.
You may want to add more columns to the Experiences table to review other metrics before making any conclusions about your campaign. Click the table options menu and choose from the following options:
- Rank: How the experience ranks compared to other experiences. For example, a conversion rate of 53% will be ranked higher than a conversion rate of 49%.
- Elements: The elements being tested (e.g. a landing page) and the list of variants for the element.
- Conversion
- Action Count: The number of converted generations for the experience. original definition - The number of actions performed for the experience.
- Conversion Rate: The percentage of Generations who have converted out of all the generations who saw the experience.
- Generations: The number of generations for the campaign.
- Revenue per Generation
- Multiple Action Count: The number of actions performed for the experience.
- Revenue: The monetary value assigned to an action for the given experience.
- Revenue per Generation: The revenue divided by generations for each experience.
- Average Order Value
- Multiple Action Count: The number of actions performed for the experience.
- Revenue: The monetary value assigned to an action for the given experience.
- AOV: The revenue divided by the multiple action count (the total number of all actions by visitors who saw the experience).
- Revenue per Converter
- Action Count: The number of actions performed for the experience.
- Revenue: The monetary value assigned to an action for the given experience.
- Revenue per Converter: The revenue per unique visitor for the experience. For example, for purchase actions, Revenue per Converter is the revenue per visitor who has completed at least one purchase through the campaign.
- Engagement
- Multiple Action Count: The number of actions performed for the experience.
- Revenue per Converter: Revenue / Action Count.
- Actions per Generation: The average number of conversions per visitor who enters the campaign. It is calculated as the multiple action count (i.e. the total number of actions by visitors who saw the experience) divided by generations.
- Margin of Error: The standard deviation for the conversion rate (or other metric e.g. Revenue per Converter) of the experience multiplied by a factor that depends on the confidence level. For example, a conversion rate (or Revenue per Converter, etc.) of 2.5% ± 0.5 means that the conversion rate (or Revenue per Converter, etc.) can be between 2% and 3%.
- Uplift: The relative difference between the experience and the control expressed as a percentage. Uplift is calculated using the following formula:
Uplift (%) = (Conversion rate of experience - Conversion rate of control experience) / (Conversion rate of control experience) x 100%
Tip: To calculate uplift for other metrics, replace the value for Conversion rate with the value for the metric, such as Revenue per Converter.
- Confidence: Measures the observed test results and is used to test a statistical hypothesis on the equality between an experience and the control metric, such as conversion rates. The confidence is the probability of obtaining a result equal to or less extreme than what was actually observed, assuming that the experience and the control have equal metric values, such as equal conversion rates.
Use the status tick to include or exclude columns in the report:
Status | Description |
---|---|
Include columns in the report. | |
Exclude columns from report. |
You can view your campaign's performance on a cumulative or daily basis.
- Cumulative report uses a cumulative average which calculates the conversion rate as the total number of actions for the entire period, divided by the total number of generations for that period. This means the chart looks smoother with time. The report provides a top-level perspective that identifies trends and is less affected by daily movements in the conversion rate.
- Daily report calculates conversion rates on a daily basis by taking the total action count for the day and dividing by the generation count for the day. The daily report remains volatile for any period of time since changes are recorded daily, but can be useful when investigating seasonal influences, periodicity, etc. You can use the smoothing feature to view chart changes recorded over 1,2 or 3 weeks. Smoothing over 1 week divides the total action count for the week by the generation count for the week, and so on.
You can use the time frame filter to include data from a specific hour, day or part of day. Use UTC or Local time to filter generation data by UTC time zone or visitor's local time zone respectively.
What is the difference between selecting UTC and the Local filter options?
If a campaign is targeted across different times zones, using local time attributes can help you analyze statistics during a particular time of day. Say you run a campaign across different US States or EU Countries, using local time attributes you can analyze the behavior of the visitors at the same time of day relative to each visitor's time zone.
Hour, Day of Week and Part of Day are all UTC times - If you select Hour and 9:00 AM for example, you will include visitors who generated into the campaign between UTC time 9:00 and 10:00 AM only - based on the UTC. If you run a campaign across Europe and 20 visitors generated into the campaign in the UK time between 9:00 AM and 10:00 AM and another 30 visitors generated in Germany at a their local time between 9:00 AM and 10:00 PM, you will only see 20 total generations in your report.
Local Hour, Local Day of Week and Local Part of Day - If you select Local Hour and 9:00 AM for example, you will include visitors who generated into the campaign between 9:00 and 10:00 AM in their local time zone. If you run a campaign across Europe and 20 visitors generated into the campaign in the UK time between 9:00 AM and 10:00 AM and another 30 visitors generated in Germany at a their local time between 9:00 AM and 10:00 PM, you will see 50 total generations in your report.
Note: Local time frames are used for filtering only. All data within the graph/charts is shown in UTC times.
You may want to filter out revenue values or multiple actions that are significantly different to your average values, so that these values do not skew your reporting data.
- Click Filters > Outlier Filtering > select Action Value or Multiple Actions and then select Enable Outlier Filtering check box.
- Use the Multiple Actions filter with 99% confidence level when filtering Engagement data.
- Use both the Multiple Actions and Action Value filters with 99% confidence level when filtering Revenue data.
- Note: A 99% confidence level filters out the most extreme observations from the report, e.g. actions with enormously large sales amount. You can change the confidence level if the recommendation of 99% does not eliminate the data spikes you strongly believe to be outliers. See Outliers in Reporting Data for more information on outliers.
- Click Apply to display the filter in the list of applied filters.
You can choose the widgets that you want to include in your report, for example, the Top Ranked Experiences or the Action Funnel for your campaign. You can also hide those that are not relevant to you.
Click Options > Page Widgets and select or deselect the check box for the widget you want to show or hide.
You may want to use the Campaign Performance Report to measure revenue data by the visitor's first action. For example, on a gaming site you may only be interested in the revenue of the visitor's first bet and ignore revenue from any subsequent bets.
Click Filters > Action Filters, Measure and select the First Action check box.
Now when you view Revenue per Generation in your report, values are calculated based on the visitor's first action only.
You can share reports with colleagues by exporting the data in the Experiences table. Click the options menu, select Export Table or Export Chart and choose the export format: comma-separated values (.cvs), .xls or tab-separated values (.tsv).
Tick the Include Applied Filters check box to display a list of applied filters at the top of the exported file.
Learn more: To export specific campaign data for further analysis or for use in third-party tools, see Campaign data export.
You may want to know whether a test is likely to produce a very small uplift and/or take too long to conclude so that you can pro-actively stop investing in testing ideas which do not give any benefit to your business. These are known as Flat Tests. For example, you run a test on the search results page using 'Booking_Revenue' as primary metric. You would accept even small positive changes to revenue so you set the minimal detectable uplift to 1%.
Set a Minimal Detectable Uplift threshold value as follows:
Click Options > Minimal Detectable Uplift, enter a percentage and click Apply.
Note: It is best to determine this value before the test starts and not to change this value during the test.