A/B testing campaigns

A/B testing is one of the easiest ways to increase conversion rates and learn more about your audience. In an A/B test, you are testing a control version of an email (A) against a different version (B) to measure which one is most successful. Depending on your goals, an A/B test can help you determine such things as the effectiveness of subject lines, content, and calls to action.

Important: Oracle Eloqua provides you with a workflow to A/B test emails. If you want to A/B test a multi-step campaign, check out the posts on Topliners to see how others have done this.

Find out more about A/B testing:

Before you start A/B testing

  • Identify the objective of the A/B test. Typically you want to identify one element of an email that you want to test. For example, you could focus on content elements like subject lines, headlines, or calls to action, or design elements like color choice, button design, and so on. This blog post can help identify the features you might want to test. The A/B Test Planner can help you capture your plans.
  • Identify the metrics you'll use to measure the effectiveness of your A/B test. When setting up the test, Oracle Eloqua lets you identify one metric that determines the winning email. The winning metric could be one of the following: total opens, unique opens, total clicks, unique clicks, total conversion, or click-through rate. Choose the metric that aligns with your goals for the test. If you're looking for high engagement, you might want to focus on opens or clicks. If you're focused on goal completion, the total conversion metric might be more appropriate. For more information on how these metrics are calculated, see Campaign reporting and metrics.
  • Send the A/B test emails to enough contacts that you're confident the results of the test are trustworthy. The size of your segment, the content of the email, and any time constraints should help you determine an appropriate test group size.
  • Plan to capture the results of the A/B test and share them. You'll want others to benefit from any new discoveries.
  • You cannot change an A/B test after it has started. Have a look at this information on changing an active A/B test.

Creating an A/B testing campaign

Using the Simple Email Campaign Wizard, you can create an A/B test to send two test emails to a part of a segment.

To create an A/B testing campaign:

  1. Navigate to Orchestration An image of the Orchestration icon., then click Campaigns.
  2. Click Create a Simple Email Campaign.

    The simple email campaign design wizard opens.

    An image of the simple email campaign design wizard

  3. Complete Step 1 of the design wizard, giving the A/B test a name and ensuring you select A/B Test.

    An image highlighting the A/B Test button in the design wizard

  4. Specify the segment to include in the testing by configuring Step 2 of the design wizard. Later you will indicate what percentage of the segment to include in the testing. You can create a new segment or upload a new one.

    An image showing step 2 of the simple email campaign design wizard where you select the recipients

    After you add a segment, the number of contacts included in the segment displays. To refresh the contacts, click An image of the Refresh icon.

    An image showing the contact count that appears after you select a segment

    Tip: As you configure the wizard, Oracle Eloqua validates your configuration. Steps that have errors appear with a caution symbol An image showing the caution symbol that appears if there are errors associated with the configuration. Click the caution symbol to view the error details.

  5. Specify the emails to use in the test by configuring Step 3 of the design wizard.

    An image of step 3 of the simple email campaign design wizard. In this step you choose the emails to include in the A/B test

  6. Specify the size of your test audience and the metric (such as total opens or total clicks) that determines the winning email by configuring Step 4 of the design wizard:
    • To configure the percentage of the segment that you want to participate in the A/B test, drag the slider. In the following example, 50% of the segment will participate - 25% of that will receive the control email (A) and 25% will receive the test email (B).

      An image showing the percentage of segment members to participate in the test

    • To select the metric to determine which email performed the best, select the Winning Metric. Later you can configure whether you want to automatically send the winning email to the rest of the segment.
    • To automatically send the winning email to the remaining contacts when the A/B test ends, select Automatically send the winning version. Oracle Eloqua determines the winner using the winning metric you selected earlier. Otherwise, Oracle Eloqua will send you a notification with the results of the A/B test and you can return to the testing campaign to choose a winner.

      Note: In the event of a tie, Oracle Eloqua sends an email and you must manually select a winner.

  7. Configure how long you want the A/B testing campaign to run. You can set the test to run for hours, days, weeks, or months or set an end date. If you selected the Automatically send the winning version option, the winning email will be sent out after this testing time ends.

    An image showing the options to configure the duration of the A/B testing campaign

  8. Specify when to launch the A/B testing campaign by configuring Step 5 of the design wizard:

    An image of step 5 of the simple email campaign design wizard. In this step you choose when you want to send the email.

    • To start the campaign immediately after you active it, click Send right now.
    • To start the campaign on a specific date and time, click Schedule for a later time and then specify the date and time to start the campaign.
  9. Click Advanced Sending Options to configure additional settings:

    An image of the advanced sending options available from the simple email campaign design wizard

    • Use a Signature: Add a signature to your email. For more information on signature rules, see Signature rules.
    • Include List-Unsubscribe header: Add an unsubscribe header to your email.
    • Break send into smaller batches: Break up sending emails into smaller batches. The period you use depends on the size of the segment. If you are not sure what period to set, start with about 1000 emails per hour. So, for a segment of 3000, sending the email over 3 hours will send 1000 per hour.
    • Only send during certain days and hours: Specify when to send the email. For example, use this option to send only at off-peak hours or at an optimal time that you've determined.
    • Allow emails to be re-sent to past recipients: Allow this email to be re-sent to past recipients. Typically, you do not want to select this option so that you don't send duplicate emails to your contacts.
  10. If you want to configure additional campaign fields, click Actions An image of the Actions button, then click Settings. Here you can set campaign fields like product, country, and so on required by your company, and change the CRM sync or closed-loop reporting settings.

    Note: The setting Allow contacts to enter the campaign more than once does not apply to an A/B test since you are only sending the email one time.

  11. Click Save.
  12. Click Activate Campaign. The campaign launches according to the email delivery settings you specified.

    Note: You must resolve all errors before you can active the campaign.

    Oracle Eloqua sends the emails or sends them according to the schedule you set. The campaign's status changes to ACTIVE or SCHEDULED. You cannot change the campaign once it is ACTIVE or SCHEDULED. Find out more about changing an active or scheduled campaign.

After the campaign is ACTIVE, you can see a real-time view of the email's performance. To view the latest information, click Refresh. To view details about contacts included or excluded from the emails, click View Participants.

You can return to the campaign after the testing completes to manually choose the winning email. If you automatically sent the winning email to the remaining segment, you can monitor the performance of the winning email. See About the results of an A/B test for more information.

Changing an active A/B test

After the A/B test is ACTIVE or SCHEDULED, you cannot change it.Here is some information to take into consideration if you need to make changes:

  • A SCHEDULED testing campaign can only be changed after it is unscheduled.
  • An ACTIVE testing campaign cannot be deactivated. If you want to stop sending the test emails, you must cancel the campaign. You could then copy the testing campaign and make the changes required. Use the Allow emails to be re-sent to past recipients setting to ensure contacts are not re-sent the same email.

Related

Choosing the winning email of an A/B testing campaign

About the results of an A/B test

Knowledge Base: Can A/B Testing Can Be Done on Multi-Step Campaigns?

Email Analysis Overview report

Campaign reporting and metrics