Understanding performance switches

Use performance switches in a program to compare the effectiveness of campaigns or other variables (such as timers) within the context of a program flow. For example, you can test:

For Email campaigns:
- Subject lines, to see which one has a higher open rate.
- Message content, to see which message has a better click-through rate.
- Landing page content, to see which page gives you a better conversion rate.
- Timing, to see whether you get a better overall response when you vary the interval between messages.
- Touch frequency, message relevance, or overall strategy, to see which approach leads to the lowest unsubscribe rate.

Once you know which elements yield the best results, you can use that information to improve the performance of your programs.

Here's a sample program that uses a performance switch to test the effectiveness of two email campaigns (the most common way to use a performance switch). An allocation switch sends a random group (10%) to each of two test paths and sends the remaining 80%, the holdout group, to an elapsed timer. After the timer elapses, the performance switch sends the holdout group to the path that produced the best results.

Sample Performance Switch

Although you can create many valid combinations of actions, timers, and switches, a performance switch will give you the clearest results when you limit the set of variables you're testing. The more variables (campaigns, timers, and so forth) you include on each test path, the less likely it is you'll be able to attribute the results to any one variable.

About winner selection

Depending on how you configure a performance switch, the system can use one of four campaign metrics to automatically identify the path with the campaign that produces the best results. Alternately, you can set the switch to pause while you evaluate the results of both paths before choosing the one you want.

When you set a performance switch to automatic winner selection, you choose one of the following metrics and set it to minimum or maximum as the criterion for winning:

Open rate: the number of messages opened
Click-through rate: the number of messages that result in the recipient clicking a link in the message
Conversion rate: the number of messages that result in the recipient making a purchase
Unsubscribe rate: the number of messages that result in the recipient unsubscribing

When you set a performance switch to manual winner selection, you wait for the test to run, examine the results (typically using Insight Interactive Dashboards), then return to the performance switch to designate the winning path. This approach is best suited for complex paths that contain multiple campaigns, timers, and other variables. The following example illustrates a performance switch that's best suited for manual winner selection.

Complex Performance Switch

Tip: Automatic selection is designed to measure the results of one campaign per path and works best with comparable paths. For example, if you have one campaign on each path, it's easy to determine which campaign achieves the best results. However, if you have multiple campaigns and timers on each path, the effects of the campaign metrics you're evaluating become diluted. In this case, your best bet is to set the performance switch to manual winner selection and analyze the aggregate performance of each path yourself.

How it works

Before you can use a performance switch to compare the effectiveness of a campaign on each path, you must divide your audience into test and holdout groups and determine how long the test will run. You must also define the paths that lead out of the switch.

You can identify your test and holdout groups using an allocation switch that selects the groups randomly. Alternately, you can place a data switch in front of the allocation switch to filter out the holdout group before dividing the test audience into two groups.

Next comes a timer to set the length of the test and keep the holdout group in a holding pattern until the path with the best results is identified. You can use an elapsed timer to set the duration of the test, or a target timer to end the test at a specific time and date.

On the other side of the performance switch are the test paths (typically, two of them). In theory, you could use a data switch to siphon off the holdout group and an allocation switch to create three test paths. In practice, however, two paths provide a more realistic and meaningful test.

Although each path can contain any number of activities and events, as best practice you should keep it simple by controlling the number of variables you're testing. If you want to evaluate two campaigns, use the same timers on both paths. Likewise, if you want to identify the most effective interval between messages (via manual selection), it's a good idea to test the timers against similar messages. If you want to test the same exact message with other variables, make a copy of the campaign so that you can get two sets of metrics.

With all other elements in place, you're now ready to add and configure a performance switch.

To use a performance switch in a program

Drag the initial allocation or data switch, timer, path elements, and any other required elements onto the canvas. Connect them as needed.
Drag a Performance switch onto the canvas and connect it to its input timer and two output paths.
Double-click the performance switch and set the following options:
- Email notifications: Enter a list of email addresses you want to notify when the test has finished. Separate multiple email addresses with commas.
  With automatic selection mode, the notification reports which campaign has been identified as the winner. With manual selection mode, the email informs recipients that it's time to examine the performance of the test paths and identify the winner. Program also sends a notification when a winner has been identified for a manual-mode performance switch.
- Winner selection mode: Choose Automatic or Manual.
  The mode you choose depends on how complex the test paths are and whether a single metric on one campaign per path will yield enough information for Program to determine the path with the best performance. If so, choose Automatic; otherwise, choose Manual. With manual selection mode you can use Insight to examine the aggregate results for each path and determine the optimal path yourself.
- Metric and criterion: For automatic selection mode, choose one of the available metrics (Open rate, Click-through rate, Conversion rate, or Unsubscribe rate) and a criterion (maximum or minimum) for the metric.
- Path names and campaign selection: Add a unique label for each test path. If you are using automatic selection mode, select one campaign from each path.
Choose whether data gathering for the switch should start over when this program is published.
The data gathered by the switch is the basis for identifying the path with the best results. See Managing published programs with a performance switch
Click Done.

Note: For more information about how performance switches fit into programs, see Designing programs.

Tips, tricks, and notes

Make sure that your test group is large enough to yield meaningful results. If your test group is too small, the difference in performance might not be statistically significant.
Regardless of the selection mode, take a look at the results of your test to make sure you are measuring the right things. For example, the following illustration shows how the path with the best performing campaign is not necessarily the overall winner.
The performance switch works with audiences created using a Scheduled filter, Customer activated, Customer deactivated, or Custom event entry point.
You can't select a campaign in more than one performance switch or more than once within the same performance switch.
If your performance switch is set to automatic selection and two or more campaigns yield the same performance, the path listed first in the performance switch properties is declared the winner.

Managing published programs with a performance switch

The Start on publish setting is automatically turned on when you add a new performance switch. When the program is published, this setting is automatically turned off. This lets the switch gather data from the time of publication until you turn the setting on and republish the program. You can change this setting as needed by editing the switch definition.

If you turn this setting on before selecting the winning path in manual selection mode and then republish the program, any audience members in the switch will remain there, awaiting your selection of the winning path.

Once the winning path has been identified (either automatically on manually), the behavior of the Start on publish option changes:

If you change the switch's definition and turn on Start on publish, republishing the program clears any existing data for the switch. The switch then uses the new definition to determine the winning path.
If you change the switch's definition and do not turn on Start on publish, republishing the program does not change the existing winner selection for that switch.

If you change the selection mode, Program automatically restarts the performance switch. Any audience members in the switch at the time remain there until new audience members enter the switch and a winning path is selected. At that point, all audience members are sent to the winning path.

Unpublishing programs with a performance switch

When you unpublish a program that contains a performance switch:

Holdout group members in the switch are retained unless you opt to terminate all on the Analyze tab after unpublishing the program. If you do, holdout members in the switch are removed from the enactment table.
If a winner has been selected, the switch keeps that selection.
If the switch is set to automatic selection and no winner has been selected, unpublishing the program halts the selection process until the program is republished and new audience members enter the switch.

When you edit a published program that contains a performance switch:

If you do not alter the performance switch, it continues to work as published.
If you delete a performance switch and republish the program, the members of the holdout group in the switch are removed from the enactment table.
If you add an exit path to a performance switch, or delete any exit path except the existing winner, the switch keeps the winning path selection when you republish the program.
If you delete the winning path and republish the program, the performance switch is reset to wait for new audience members.

Selecting a winning path manually

A performance switch runs (collects data) until the timer that precedes it expires. If the switch is set to automatic selection, the path that contains the best performing campaign is designated as the winner. If the switch is set to manual selection, Program displays a warning on the Programs page and sends an email to the addresses defined for the switch. This is when you should review the test results in Insight and identify the path that yielded the best results.

After you've identified the winning path, you need to designate it in the performance switch.

To select the winning path

Open the program that contains the performance switch and select Snapshot on the Analyze tab.
Double-click the performance switch, select the winning path, and click Done.

The path you choose is highlighted and Program sends a follow-up notification to the specified email addresses. The remaining audience members (and all future traffic arriving at the performance switch) are sent to the path you select.

To override a selected path

Open the program that contains the performance switch and select Snapshot on the Analyze tab.
Double-click the performance switch, choose a new winning path, and click Done.
Confirm that you want to change the path.
Program sends an email notification announcing the change to the distribution list for the performance switch. If you manually select a winner before an automatic selection occurs, the automatic selection will not override your manual selection when the timer expires.