post

Footer About

Footer Get Involved

Footer Links

Mind the Product

General

Product Unplugged Episode 3 with Pierce Healy

Product Unplugged Episode 2 with Lorilyn McCue

Product Strategy

Building Product Teams

Diversity and Inclusion

Digital Transformation

Artificial Intelligence (AI) & Machine Learning (ML)

Product Leadership

Product Management Career

Product Ethics

Minimum Viable Product

Enterprise Product Management

OKRs

Customer Research

Hypothesis and Testing

Business Strategy

User Research

👩‍🚀🚀   Live now for MTP members: <a href="https://mtp2017stg.wpengine.com/membership/#events" style="color:#FFFFFF;text-decoration:underline;">Driving Innovation AMA with Tendayi Viki</a>

Roadmaps: Break the rules without breaking the principles

Got questions about your product career?

Product roadmaps vs data – Jeremy Levy on The Product Experience

From engineering to product management at Google

Mind the Product joins Pendo

Kevin Shanahan is Head of Growth at Peak, the self-improvement startup that offers the #1 brain training app worldwide. He oversees product development of new apps under the Peak brand, from concept to prototype to MVP to launch.

Kevin Shanahan

<p>This is the second of two posts that deep dive on A/B tests, expanding on <a href="https://www.youtube.com/watch?v=KAFKKlFoJjU&amp;feature=youtu.be&amp;list=PLWz5rJ2EKKc_amR5JL4lSH4PHSWNUTV85">a talk I gave at Google Playtime 2016 in London</a>. In the first post I explained <a href="https://mtp2017stg.wpengine.com/stepping-ab-tests/" target="_blank" rel="noopener noreferrer">how to step up your A/B testing</a>, and in this post I’ll look at how using a tracker spreadsheet can help you manage your A/B tests and better retain learnings from past ones. <a href="https://docs.google.com/spreadsheets/d/1wUfNShlpjLR293VwtPcMe8LbeBh-QgytOFYeAdExomY/edit#gid=179782741">Here’s a link</a> to the tracker so you can follow along and make your own copy (it’s free).</p>
<p><a href="https://mtp2017stg.wpengine.com/wp-content/uploads/2017/07/ab-testing-sheet-1.png"><img loading="lazy" decoding="async" class="aligncenter size-full wp-image-11875" src="https://mtp2017stg.wpengine.com/wp-content/uploads/2017/07/ab-testing-sheet-1.png" alt="" width="1219" height="344" srcset="/wp-content/uploads/2017/07/ab-testing-sheet-1.png 1219w, /wp-content/uploads/2017/07/ab-testing-sheet-1-300x85.png 300w, /wp-content/uploads/2017/07/ab-testing-sheet-1-768x217.png 768w, /wp-content/uploads/2017/07/ab-testing-sheet-1-1024x289.png 1024w" sizes="auto, (max-width: 1219px) 100vw, 1219px" /></a></p>
<h3>Why you Need a Tracker</h3>
<p>As mentioned in the first post, if you plan to run a lot of A/B tests then you should think about using a document to store test information, which I call the tracker. There are several benefits that a tracker gives you:</p>
<ul>
<li><em>Forces you to think through the details of a test</em>. Given that each test follows a similar structure, the tracker acts like a checklist to ensure you haven’t overlooked important details. This prevents costly fixes and re-runs later on.</li>
<li><em>Ensures everyone is on the same page</em>. It’s easy for people to have different interpretations or recollections of what was agreed for an A/B test, but they’ll quickly identify these and converge if test information is listed in one place.</li>
<li><em>Gives visibility on live and upcoming tests</em>. Testing typically affects many teams in a company, therefore each team needs to be able to see at a glance which A/B tests are running or planned so they can adjust their plans.</li>
<li><em>Stores details of past tests so you can easily reference them</em>. It’s common to want to look back on old A/B tests to inform new ones, and the tracker prevents you from having to trawl through messages and project management tickets to do this.</li>
</ul>
<h3>Everyone Loves a Spreadsheet</h3>
<p>We use a Google spreadsheet for the tracker at <a href="http://www.peak.net/">Peak</a>. Why a spreadsheet? Because spreadsheets are easy to understand, update, share, filter, synchronise, and have different layers of permissions. They may not be exciting but they’re the perfect format for the job.</p>
<h3>The Parts of the Tracker</h3>
<p><a href="https://docs.google.com/spreadsheets/d/1wUfNShlpjLR293VwtPcMe8LbeBh-QgytOFYeAdExomY/edit#gid=179782741">Here’s a link</a> to the tracker that we use. It has five main parts, each of which is marked with a different coloured header.</p>
<p><img loading="lazy" decoding="async" class="aligncenter size-full wp-image-11874" src="https://mtp2017stg.wpengine.com/wp-content/uploads/2017/07/ab-testing-sheet-2.png" alt="" width="1335" height="431" srcset="/wp-content/uploads/2017/07/ab-testing-sheet-2.png 1335w, /wp-content/uploads/2017/07/ab-testing-sheet-2-300x97.png 300w, /wp-content/uploads/2017/07/ab-testing-sheet-2-768x248.png 768w, /wp-content/uploads/2017/07/ab-testing-sheet-2-1024x331.png 1024w" sizes="auto, (max-width: 1335px) 100vw, 1335px" /></p>
<ul>
<li><em>Test overview.</em> Records what the test is, where it’s happening inside (or outside) your product, why you’re doing it, when you’re doing it, and what the status is.</li>
<li><em>Segment info.</em> Describes the users in a test, including the countries they’re in, the languages they speak, and the devices they use</li>
</ul>
<p><img loading="lazy" decoding="async" class="aligncenter size-full wp-image-11873" src="https://mtp2017stg.wpengine.com/wp-content/uploads/2017/07/ab-testing-sheet-3.png" alt="" width="1335" height="418" srcset="/wp-content/uploads/2017/07/ab-testing-sheet-3.png 1335w, /wp-content/uploads/2017/07/ab-testing-sheet-3-300x94.png 300w, /wp-content/uploads/2017/07/ab-testing-sheet-3-768x240.png 768w, /wp-content/uploads/2017/07/ab-testing-sheet-3-1024x321.png 1024w" sizes="auto, (max-width: 1335px) 100vw, 1335px" /></p>
<ul>
<li><em>Variant info.</em> Indicates how you will divide these users into variants and what the experience will be like for each one.</li>
<li><em>KPIs</em>. Lists the metrics you will judge the success of the test on and the targets you hope to achieve if the test is successful.</li>
<li><em>Results and next steps</em>. A short summary of the test results, links to more detailed analysis, and the next steps.</li>
</ul>
<p>And that’s it! The tracker is unlikely to win an award anytime soon but we’ve found it to be a great way of staying on top of lots of tests.</p>
<p>If you found this post and the tracker useful, or have any questions or suggestions for improvements, I’d love to hear from you in the comments section below. Here’s the <a href="https://docs.google.com/spreadsheets/d/1wUfNShlpjLR293VwtPcMe8LbeBh-QgytOFYeAdExomY/edit#gid=179782741">link </a>to the tracker once more, which is free to copy and modify.</p>
<div class="g g-21"><div class="g-single a-75"><a class="gofollow" data-track="NzUsMjEsMSw2MA==" href="https://mtp2017stg.wpengine.com/product-management-training/public-workshops/"><img decoding="async" src="https://mtp2017stg.wpengine.com/wp-content/uploads/2021/10/General-Training-Level-up-FOOTER-scaled.jpg" /></a></div></div>

core/freeform

Data Driven Product Management

Data scaling for startups by Crystal Widjaja

Make Better Product Decisions by Using Quantitative and Qualitative Data, by Brian Tran

Testing Business Ideas, by David Bland

Developing a Data-Driven Culture at Dow Jones, By Helen Hewitt

When A/B Tests Aren’t Enough, By Scott Castle

Using Data to Build a Brand, by Abba Newberry

Socially Preferable by Nathan Kinch

They’re Just Fancy Averages by Ben Fields

Talking About Data Science by Jenessa Lancaster

Unlocking Data Science to Build the Future of Work by Mike Hyde

Tracking your A/B tests

ProductTank Columbus

ProductTank Tauranga

ProductTank Edinburgh

Prioritised Member AMA - Perfecting product roadmaps

ProductTank San Diego

Navigating difficult relationships panel

ProductTank Philadelphia

ProductTank Bucharest

ProductTank NoVa

Remote Training Workshops

ProductTank London

ProductTank Birmingham

ProductTank Oslo

ProductTank Astana

ProductTank NoVA

ProductTank Zürich

ProductTank Perth

ProductTank Tokyo

ProductTank Chicago

ProductTank Auckland

ProductTank Richmond

ProductTank Charlotte

ProductTank Reston

ProductTank Washington

ProductTank Sofia

ProductTank Raleigh

ProductTank Ljubljana

ProductTank Kraków

ProductTank Athens

ProductTank Barcelona

ProductTank Dublin

ProductTank Montréal

ProductTank Seattle

ProductTank Madison

ProductTank Singapore

ProductTank Portland

ProductTank Budapest

ProductTank Valencia

ProductTank Porto

ProductTank Los Angeles

ProductTank Austin

ProductTank Norwich

ProductTank Des Moines

ProductTank San Francisco

ProductTank Helsinki

ProductTank Reading

ProductTank Brighton

ProductTank Buenos Aires

ProductTank Bielefeld

ProductTank Charleston

ProductTank Bogotá

ProductTank Newcastle Upon Tyne

ProductTank Dallas

ProductTank Malmö

ProductTank Canberra

ProductTank Paris

ProductTank Manchester

#mtpcon SF+Americas 2022 Conference

#mtpEngage Hamburg 2022 Leadership Forum

#mtpEngage Hamburg 2022 Workshops

#mtpEngage Hamburg 2022 Conference

#mtpcon London+EMEA 2022 Conference

Tracking your A/B tests

Kevin Shanahan

Read more

Comments