A/B Tests

A/B Tests provides the tools to carry out experiments using different variations of content features, eg: titles and featured images, and/or different variations of content blocks.

Altis provides an A/B Test Block, which you can use to test different variations of a block or set of blocks in your content, widgets and layouts. There is also a developer friendly framework to add new custom tests that is used to power built in features like Title and Featured Image tests which can be controlled from the Block Editor sidebar.

Tests run until the end date or when a statistically significant improvement has been found, when a winner is found the winning variant will be shown to everyone.

Post titles and featured images A/B Tests

With this feature enabled it's simple to create A/B Tests for your post titles and/or featured images directly from the post edit screen.

It is enabled by default but can be disabled via the configuration file:

{
	"extra": {
		"altis": {
			"modules": {
				"analytics": {
					"native": {
						"experiments": {
							"titles": false,
							"featured_images": false
						}
					}
				}
			}
		}
	}
}

Within the post edit screen click on the A/B icon to access the Experiments panel. Here you can set up the variants for your tests, the amount of traffic to show the tests to as well as start and end dates.

Once you are ready to run the test click on the toggle to unpause it. Results are updated every hour until a statistically significant winner is found.

A/B Testing Titles user interface

By default post title and featured image A/B tests are enabled for Posts and Pages however custom post types can be supported using keys shown in the example below, either when registering the post type or on the init action, for example:

add_action( 'init', function () {
	// Add support for title A/B tests to the 'events' post type.
	add_post_type_support( 'events', 'altis.experiments.titles' );

	// Add support for featured image A/B tests to the 'events' post type.
	add_post_type_support( 'events', 'altis.experiments.featured_images' );
} );

Notes:

In order for the test to function correctly, it needs WordPress to output the <ab-test></ab-test> Web Component (generated by the output_ab_test_html_for_post() function) that includes all the variations of the feature representation, so the front-end would be able to switch to the proper variation to serve to the visitor. As such, the built-in features will only work if the native WordPress functions are used, which are the_title() or get_the_title() for post title and the_post_thumbnail() or get_the_post_thumbnail() for featured image.
In order for the A/B Testing feature to work properly, the variant switching needs to happen on the front-end, so the feature would only work with data the front-end can change in a Javascript-enabled browser.
While the_post_thumbnail() is filtered to output the different variations as expected, has_post_thumbnail() is not filtered as it can be used in different situations than what the A/B Test is concerned with, and hence can break the expected functionality elsewhere.

Creating Custom Tests

There is a programmatic API to register A/B tests for posts:

Altis\Analytics\Experiments\register_post_ab_test( string $test_id, array $options )

Sets up the test.

$test_id: A unique ID for the test.
$options: Configuration options for the test.
- label <string>: A human readable label for the test.
- singular_label <string>: A human readable label for a single variant of the test.
- rest_api_variants_field <string>: The field name to make variants available at.
- rest_api_variants_type <string>: The data type of the variants.
- goal <string>: The conversion goal event name, eg "click" or "click:.selector a".
- selector <string>: A CSS selector for a child element to bind the event to.
- closest <string>: A CSS selector for the closest matching parent element to bind the event to. Applied before selector.
- goal_filter <string | callable>: Elasticsearch bool query to filter goal results. If a callable is passed it receives the test ID and post ID as arguments.
- query_filter <string | callable>: Elasticsearch bool query to filter total events being queried. If a callable is passed it receives the test ID and post ID as arguments.
- variant_callback <callable>: An optional callback used to render variants based.
  - $value <mixed>: The variant value.
  - $post_id <int>: The post ID.
  - $args <array>: Optional args passed to output_ab_test_html_for_post().
- winner_callback <callable>: An optional callback used to perform updates to the post when a winner is found. Defaults to no-op.
  - $post_id <int>: The post ID.
  - $value <mixed>: The winning variant value.
- post_types <array>: An array of supported post types for the test.
- show_ui <bool>: Whether to show the test in the Experiments sidebar of the Block Editor.
- editor_scripts <array>: An array of scripts to load for this test within the Block Editor, and dependencies for each (see example).

output_ab_test_html_for_post( string $test_id, int $post_id, string $default_content, array $args = [] )

Returns the A/B Test markup for client side processing.

$test_id: A unique ID for the test.
$post_id: The post ID for the test.
$default_content: The default content for the control test.
$args: An optional array of data to pass through to variant_callback.

Custom Test Example

Check an example of a custom A/B test for a Call to Action button to understand the steps you need to create custom tests for your project needs.

Goal Tracking

The goal tracking framework is explained in more detail here. A built in click event goal is available out of the box which is used by the built in title A/B test feature.

Variant Data Storage

How you manage the variant data is up to you, for example you could use Fieldmanager, CMB2 or other custom meta boxes framework to save the variant data.

Note you should use the following functions to get and update the variants. All functions are under the Altis\Analytics\Experiments namespace:

get_ab_test_variants_for_post( string $test_id, int $post_id ) : array

update_ab_test_variants_for_post( string $test_id, int $post_id, array $variants )

Actions

altis.experiments.test.registered: (string) $test_id, (array) $options

Fired when a test has been registered using register_post_ab_test.

altis.experiments.test.ended: (string) $test_id, (int) $post_id

Fired when a test has ended either by finding a statistically significant difference or the end date was reached.

altis.experiments.test.ended.$test_id: (int) $post_id

This is the same as above but scoped to tests with the ID $test_id.

altis.experiments.test.winner_found: (string) $test_id, (int) $post_id, (mixed) $value

Fired when a winning variant has been found. The $value parameter is the stored variant value.

altis.experiments.test.winner_found.$test_id: (int) $post_id, (mixed) $value

This is the same as the above but scoped to tests with the ID $test_id.

Testing Experiments

In order to see your experiments working locally you need to generate enough traffic to get statistically significant results. You can direct traffic to your development environment using the Trafficator tool.

Install it by running npm install -g trafficator.

Create a file called .trafficator.js that will perform the necessary actions to trigger conversions. The following example tests the built in titles experiment by doing the following:

generate visits to the home page of your local site
check the stored tests value, for post ID 1 this is titles_1
return different probabilities for different variant IDs
click the title link if the probability is met

module.exports = {
	funnels: [
		{
			entry: 'https://my-project.altis.dev/',
			steps: [
				{
					name: 'Click Article Title',
					probability: async page => {
						const p = await page.evaluate(() => {
							const tests = JSON.parse(localStorage.getItem('_altis_ab_tests') || "{}");
							if (tests.titles_1) {
								if (tests.titles_1 === 1) {
									return 0.4;
								}
								if (tests.titles_1 === 2) {
									return 0.1;
								}
							}
							return 0.15;
						});
						return p;
					},
					action: async page => {
						await page.click('.post-1 .entry-title a');
					}
				}
			]
		}
	],
};

This should generate traffic that exhibits a trend towards the first title variant.