Video A/B Testing - How To Measure The Impact Of Your Video

How to measure the impact of your video with A/B testing

Here’s a quick guide (7-minute read) on how to set up a video A/B test on your website. We’ll look at two possible ways to test your video, with screen recording examples for you to follow along with.

*DISCLAIMER - Since writing this Google has announced the retirement of Optimize as of September 2023. The advice here will hold true until then! Expect updates soon.
Quickly start measuring your video's impact

Setting up an A/B test on your website is a great way to start unpicking the return on investment from your video marketing efforts. It’s simple to do (in most cases), and at most will require a little of your website developer’s input. This guide assumes you’re using Google tools like Analytics and Chrome as we’ll be using Google Optimize throughout the examples.

If you’re using another analytics platform (Adobe/HubSpot) to measure your website, you’ll need a paid tool such as Optimizely who provide an account manager to help throughout the process.

Anyway, here’s a step-by-step guide to help you start A/B testing your video’s impact on a website page:
1 - Create your first Optimize experience

First time using Google Optimize? No worries, it’s easy to get up and running. Simply sign in through Google using your email and you’re good to go. Do make sure that you’re using the same email that has access to your Google Analytics, or you’ll run into trouble later when you try to connect it to your test.

Now, hit the “Let’s go” button under “create your first experience” and give your test a name. Make sure you use the URL of the page you’d like to test. Alternatively, you can enter your domain and specifically target pages later.

You’ll see Optimize gives you a few options here, and we’re only interested in the A/B test for this example (although we do give a little guidance on a redirect test later on). The top three test types are all set up very similarly.

Hit “create” and we’re good to go to the next step.
2 - Create your page variant

Add your first page variant by clicking the “add variant” button and naming the new page.

We’re going to add the Optimize Chrome extension to our browser here too so that we can open the preview and build mode we need to edit our page. Click edit, and you’ll be prompted to add the extension. Follow the instructions, and return to the test setup page.

Next step.
3 - Edit your page variant

Now you’ve got the Optimize extension added, we can edit our page. Click the “edit” button next to your new variant and you’ll find yourself in the edit mode.

Here you can select all the different elements of your page and interact with them. You can change dimensions, edit their text, HTML and more - but we’d only suggest touching the more technical stuff with your developer.

Click on your video, scroll right and click the “edit element” button, we’re then going to remove the video from this variant.
Next, we stretch (roughly in this example) our title and value proposition across the page to fill our banner, and add some example text.

Then hit “save” and “finished” and you’ll be taken back to the test page. Your changes will show and we’re ready for the next step.
In this example, we hid our video and (very roughly) moved our title and value proposition to be the main focus of the page. In a perfect world, our example page already has the title and value proposition there (that way this would be a 100% fair test).

You could create a totally different page
If you have the help of a developer, or you’re a bit of a whizz on your content management system (CMS), you could create a completely different version of the page with/without the video. Just remember, the more you change, the less likely you are to find the impact of the video alone.

To test a totally different page, you can set up a redirect test from the options on the “experience setup” page (from step 1). It will ask for the variant page’s URL - simply add that and you’re good to go.
4 - Make sure your page targeting is spot-on

If you decided to use your domain and not the specific page when you set up your test (during step 1), you might want to choose a specific page for these tests to show. You could change that in the targeting section.

In most cases, you chose your specific URL in step 1 though, so this should all be accurate.
5 - Connect your test to Google Analytics

Click the “link to analytics” button and it will open a tab where you can select a property. If you followed our advice in step 1 then you’ll have created your Optimize account with the same email address you have access to Analytics on.

Now you just choose the property you want and hit “link”.

This will allow you to choose how you’ll measure your test (the most important part!), assuming you already have goals set up in your Analytics, you simply select the ones you want to measure.

If you have no goals set up in Analytics, then we strongly suggest bringing in a specialist to help you get that right before running any tests.
6 - Install Google Optimize on your website

There are two ways to install Optimize on your site. We showcase the Google Tag Manager approach here, but you can just install some header code on your site instead. If you know how to do that, grand, crack on! If you don’t and you have a developer, simply send them the code to install.

If you’re using Google Tag Manager and you have access, then the process is simple. Log into your account, and create a new tag. Follow your in-house naming conventions or simply name it “Optimize Tag” or similar.

Now, under the tag section, you can click the preset “Google Optimize” tag. You’ll need your container ID to set up the tag correctly, and you can find that by returning to Optimize and hitting return on the top left-hand side of the screen until you reach the container screen, then just copy and paste your container ID back into Tag Manager.

Set the trigger section to “all pages” and hit “save”.

Now you can publish your changes by first hitting “submit” in the top right, typing any notes you’d like about the container changes, and then hitting “publish” in the top right.

Next, return to Optimize, and hit the “check installation” button. You’ll be taken to a screen where it will look for the right code on your page and confirm you’ve successfully installed your tag.

That was the trickiest step - nice job.
7 - Set your test live

Now you’re ready to set your test live. Just give everything a once or twice over to make sure you’re happy and then hit “start”.

Once your tests are running you’ll be able to open an incognito tab and plop in your URL a few times to see the two variants showing randomly. The visitor will not know a test is running (or at least that’s the idea).
8 - Watch the data come in through the reports ta

Back in Google Optimize, you’ll be able to watch how your tests are performing in the reports tab - just to the right of where you set up your details.

Here, Google will look for a winning page by trying to find statistical significance.

You may find that the data shows a lot of promise early, or that you need to run tests for a longer period of time to see the pages start to separate in performance. We’d always suggest at least three months of test time.

You’ll need page traffic to run your A/B test
You don’t need loads of traffic to find A/B testing useful, but the more traffic you do have going to your test pages, the more likely you’ll find a winner, and get useful results. Also, you won’t be waiting a year to reach any conclusions. In short: the less traffic you have, the longer the test will take.

Around a thousand visits (or more) per month to your target page is ideal, but not mandatory. The more traffic heading through, the more effective your test will be.

Get the most out of your animations
Animated videos are great, and we know they work. But it’s not always easy to see that if you’re not measuring their effectiveness! We filmed our screens and wrote these words to help you better understand the marketing impact of your video.
But our main goal is to create killer animations for your fintech. If you’d like to talk to us about getting started, or even want some help with where your video fits within a wider content strategy, reach out to us.

Author: Josef James
Sign up for the Very Unboring Email 🚫🥱

One or two emails from us a month filled with some bite-sized advice that will help you:

- Not be boring.
- Create better animated and video content.
- Help you think about content strategy, creative and more.
- With hints and tips around measurement and testing (if you're into that!).
hi@financeshoots.com
LinkedIn
Careers
Based in London, working globally.

© 2023 The Shoots Group Ltd