Excluding experiences from multivariate tests

You can exclude experiences from an MVT (partial factorial).

An MVT campaign contains multiple experiences, one for each combination of variants chosen from each element. For example, a campaign with two elements, element Button Color with variants blue (default) and black and element Text Color with variants black (default) and red, will generate a total of four combinations:

Experience Element: Button Color Element: Text Color
1 Blue (default) Black (default)
2 Blue (default) Red
3 Red Black (default)
4 Red Red

Excluding experiences can be useful in certain scenarios:

  • When a particular combination of the variants does not serve a good user experience. For example, we would not want to show experience 4, which shows red text on a red button.
  • When you want to reduce the overall number of experiences to reach campaign conclusion more quickly. This is because both experiences and individual variants are exposed to more traffic.
  • If you only want an experience to be shown for a specific segment, you can exclude it from the campaign and only show it within the campaign's targeting segments.

To exclude an experience:

  1. In the Preview tab of your campaign, choose the experience you want to exclude.
  2. Click Show to toggle between including and excluding this experience.
  3. The excluded experience appears as crossed out in the list and is not served to visitors generating into the test.

Calculating experience weights in multivariate tests

By default, all variant weights are split equally within each element of an MVT campaign, which in turn results in all experiences also split equally. However, it may be required to increase the weight of certain variants or experiences (for example, using the Optimal Traffic Split feature). This section explains how altering variant weights affects each experience’s traffic share, and how to set up the campaign to achieve specific experience weightings.

Equal variant split:

Taking this simple example, the experiences and relative traffic weights would be:

  1. Default A + Default B: 50% x 50% = 25%

  2. Default A + Variant B: 50% x 50% = 25%

  3. Variant A + Default B: 50% x 50% = 25%

  4. Variant A + Variant B: 50% x 50% = 25%

    And if Experience 4 were to be excluded:

  1. Default A + Default B: 50% x 50% = 25%; 25%/75% = 33.3%

  2. Default A + Variant B: 50% x 50% = 25%; 25%/75% = 33.3%

  3. Variant A + Default B: 50% x 50% = 25%; 25%/75% = 33.3%

  4. Variant A + Variant B: 50% x 50% = 25%; 25%/0% = 0%

Unequal variant split:

With these adjusted variant weights, the experiences’ relative traffic weights would become:

  1. Default A + Default B: 20% x 60% = 12%

  2. Default A + Variant B: 20% x 40% = 8%

  3. Variant A + Default B: 80% x 60% = 48%

  4. Variant A + Variant B: 80% x 40% = 32%

And if Experience 4 were to be excluded:

  1. Default A + Default B: 20% x 60% = 12%; 12%/68% = 17.65%

  2. Default A + Variant B: 20% x 40% = 8%; 8%/68% = 11.76%

  3. Variant A + Default B: 80% x 60% = 48%; 48%/68% = 70.59%

  4. Variant A + Variant B: 80% x 40% = 32%; 32%/0% = 0%

Specific Experience split requirement example:

As the campaign creator, I want to serve the Control Experience (Default A + Default B) to 50% of the traffic and split the remaining 3 experiences equally (i.e. 16.67% each).

This cannot be achieved by simply adjusting the weights of the variants, instead an alternative default variant must be added to Element A (with no content changes so that it is the same as default). Use the following configuration and exclude certain experience combinations:

  1. Default A + Default B: 60% x 50% = 30%; 30%/60% = 50%

  2. Default A + Variant B: 60% x 50% = 30%; 30%/0% = 0%

  3. Variant A + Default B: 20% x 50% = 10%; 10%/60% = 16.67%

  4. Variant A + Variant B: 20% x 50% = 10%; 10%/60% = 16.67%

  5. Alt def. A + Default B: 20% x 50% = 10%; 10%/60% = 0%

  6. Alt def. A + Variant B: 20% x 50% = 10%; 10%/60% = 16.67%

Important: Unless using the ‘split evenly’ function, only integral (whole number) percentages are supported for variant weights, e.g. 4%, 58%. Decimals can’t be entered in Campaign Designer, so dummy variants can be used and excluded to achieve specific weightings if desired, similar to the example above.