This is a two-stage process where you first create the test and then create the schedule for the test. Engagement Engine does not consider it a valid test instance until both steps have been created.
Create Test
You can start the process to create a new test by clicking the Create Test button on the A/B Testing screen.
This displays the Create New Test screen.
Create New Test screen
This screen allows you to provide all of the details necessary to create a new A/B Test. The fields displayed on this screen are:
Test Name: This is the name of the test that is displayed in the test list on the A/B Testing screen.
Test Description: This should be a meaningful description of the test. This is also displayed on the test list on the A/B Testing screen. If the description is too long to be displayed you can display the full description by hovering over the test description cell in the test list.
Variant Rule: This is a drop down containing all of the rules that have been published on any of your sites. The rules in this dropdown are displayed in alphabetical order. You should select the rule that you wish to alter to create the variant rule.
Note: When you select a rule, a message is displayed to the right of the dropdown box informing you which sites are currently running the rule and which sites, if any, are running an older version of the rule.
Configure Variant Rule: Once you have selected your test rule, the Configure Variant Rule button should appear. Click on this button to display the rule.
Variant Rule screen
From her you can perform one of the following acts;
Modify a true action
Delete a true action
Modify the order of true actions
Modify a condition
Delete a condition
Modify a false action
Delete a false action
Modify the order of false actions
Only one change is permitted and once you have made a change the other conditions and actions displayed become read-only and cannot be altered.
You can switch between the variant rule and the control rule by clicking on the Variant Rule and Control Rule buttons. This allows you to easily compare the two rules.
Once you have configured the variant, click on the Save button to return to the Create New Test screen. You should see a message above the Configure Variant button confirming that the variant has been configured.
If you click on the Close button, you should return to the Create New Test screen without saving any changes.
Clicking on the Reset Rule button reloads the variant rule to match the control rule.
Clicking on the Revert button undoes any unsaved changes you have made.
Test Goals: A goal comprises up to three components; event, invite, and rule. These components each have their own dropdown box in this section. These dropdown boxes are:
Event Name: This dropdown box allows you to select the event component of a test goal. It contains a list of system-defined and user-defined custom events, and events related to any invites or widgets used by currently published rules.
Invite Name: This dropdown box enables you to select the invite component of a test goal. This only applies if the event you selected relates to an invite. You can instruct the system to collect data based on a particular invite by selecting that invite from the dropdown list, or to collect data based on all invites by selecting Any Invite from the dropdown.
Rule Name: This dropdown box allows you to select the rule component of a test goal. It displays a list of all possible test goal rules based on the event and invite options you have selected. You can instruct the system to collect data based on a particular rule by selecting that rule from the dropdown list, or to collect data based on all rules by selecting Any Rule from the dropdown.
You must have at least one test goal and can have up to three test goals. When you have configured a test goal you can add additional test goals by clicking on the Add Goal button.
Once you have configured your test goals you can select which is the primary goal by clicking the appropriate Primary Goal radio button. The primary goal is the one which is used to determine whether or not the test has reached the point of statistical significance.
Click on the Save button to save the test you have created. This displays the Test Definition screen for the test you have created and you can now create a schedule for the test.
Clicking on the Cancel button at any stage during the test creation cancels any changes you have made and returns you to the A/B Testing screen.
Create Schedule
Once you have created a test, you can schedule the test by clicking on the Schedule button from the Test Definition screen. This displays the Schedule screen.
Schedule Screen
From this screen you can enter the Start Date, Start Time, End Date, and End Time for the test you have created. Engagement Engine defaults these values to the next available 14-day window. You can edit these fields as required.
As you can only run one test at a time, this screen displays all currently scheduled tests in chronological order so that you can identify a window in which you can schedule your test.
Note: Bear in mind that you can only schedule a Start Date up to a maximum of 30 days in advance, and that a test cannot run for longer than 14 days, so the End Date and End Time must be within 14 days of the Start Date and Start Time.
You must also specify the Minimum Number of Unique Participants. This number is the threshold below which Confidence Level is not calculated after the test has ended. This is a required field, but you can enter 0 if you do not wish to specify a minimum number of required participants.
