Engagement Engine offers an A/B testing solution that allows you to vary one element of one rule run on your site. You can vary either one condition, one action, or the running order of the actions associated with a rule. Engagement Engine then uses JavaScript events to track the visitor’s behavior while on your website.
The basic flow of an A/B test is:
Select test rule.
Configure variant rule.
Define test goals.
Schedule test.
Run test.
Analyze results.
Within this framework there are a number of points that should be highlighted;
Tests can track up to three goals simultaneously.
You can nominate up to three events to track in each test. One of these must be nominated as the primary goal. By tracking three goals, Engagement Engine maximizes the amount of information available for your decision making.
You can only run one test at a time.
With A/B testing you can only alter one condition or action within a rule, although you can make as many changes as you wish within that one condition or action, or you can change the order of actions within a rule. This is intended to prevent confusion about what has influenced a visitor’s behavior. Similarly, to ensure that results are linked to a specific test, only one test can be run at a time.
Tests can only run against sites that use the most recent published version of the rule selected for the test.
If you have a rule that is associated with multiple sites, but where some of the sites are not running the most recent version of the rule, the test will only monitor visitors’ activity on the sites with the most recent version of the rule.
Visitors only participate in a test when the test rule is evaluated.
A visitor is not considered a participant in a test until either the control or variant test rule has been evaluated during their session. If neither the control nor variant rule is evaluated, their behavior is not monitored for test purposes.
Visitors stay on their allocated test path for duration of test.
Once a visitor has been allocated to either the control or variant path of the test, they remain on their allocated path for the duration of the test.
Tests can be scheduled a maximum of 30 days in advance.
In an effort to ensure tests remain relevant and necessary, you can only schedule a test to begin up to 30 days in advance.
Tests can last for a maximum of 14 days.
As only one test can run at a time, there is a limit of 14 days on the length of a test to try and prevent a backlog of tests from building up.
Any publish event renders a test invalid.
The test ends automatically if any sites are published while the test is in progress. This happens whether the site is published by a user clicking the publish button for that site, or by an auto publish that occurs after certain related events. For more information, please refer to the Publishing and A/B Tests section of this document.
“Winning” variants are not automatically promoted.
Engagement Engine provides the information that allows you to make an informed choice about content displayed to visitors and does not assume that you wish to make changes based solely upon the primary goal.
