post

Footer About

Footer Get Involved

Footer Links

Mind the Product

General

Product Unplugged Episode 3 with Pierce Healy

Product Unplugged Episode 2 with Lorilyn McCue

Product Strategy

Building Product Teams

Diversity and Inclusion

Digital Transformation

Artificial Intelligence (AI) & Machine Learning (ML)

Product Leadership

Product Management Career

Product Ethics

Minimum Viable Product

Enterprise Product Management

OKRs

Customer Research

Hypothesis and Testing

Business Strategy

User Research

👩‍🚀🚀   Live now for MTP members: <a href="https://mtp2017stg.wpengine.com/membership/#events" style="color:#FFFFFF;text-decoration:underline;">Driving Innovation AMA with Tendayi Viki</a>

Roadmaps: Break the rules without breaking the principles

Got questions about your product career?

Product roadmaps vs data – Jeremy Levy on The Product Experience

From engineering to product management at Google

Mind the Product joins Pendo

Emily brings nearly 15 years of product management experience to her role as Managing Director for Mind the Product. While she has worked across a variety of industries, she will always have a soft spot for the travel industry where she spent most of her early career building software for airlines and developing a leading consumer travel app, TripCase. Emily is passionate about the craft of product management and loves talking about new ways to make products people love.

Managing Director

Emily Tate

<p>At #mtpcon San Francisco, <a href="https://twitter.com/leisa" target="_blank" rel="noopener">Leisa Reichelt</a>, head of research and insights at Atlassian, took on our current approach to user research and how we take an evidence-based approach to doing completely the wrong thing.</p>
<p>Five years ago, the industry was focused on just getting people to do <em>any</em> user research. Product managers weren’t regularly talking to customers at all. These days, the drive for companies to be “customer obsessed” has brought about a new era where <em>everyone</em> is talking to customers. Designers, product managers, and researchers all think that doing using research is a primary function of their job. And while research is best when done as a team, how much time are we spending to grow our expertise in research so that we do it well?</p>
<p>While we should be out there doing research, we&#8217;re letting bad research pass for evidence, and we make decisions on data we shouldn’t be using for this purpose. We mess up research by trying to quantify things and make them seem more scientific than they really are.</p>
<h2>Perspective and Truth</h2>
<p><img loading="lazy" decoding="async" class="aligncenter size-full wp-image-13686" src="https://mtp2017stg.wpengine.com/wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-duck-and-shadow.jpg" alt="Leisa Reichelt shows how different a duck looks from its shadow" width="2048" height="1365" srcset="/wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-duck-and-shadow.jpg 2048w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-duck-and-shadow-300x200.jpg 300w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-duck-and-shadow-768x512.jpg 768w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-duck-and-shadow-1024x683.jpg 1024w" sizes="auto, (max-width: 2048px) 100vw, 2048px" /><br />
Looking at only parts of a picture can give a warped perspective, and we can see things that appear meaningful but which aren’t. We have to make sure we are getting a big enough picture in research to make a good decision. If you see the shadow of a duck, it can look very different from the duck itself. This happens in research as well. We are getting a narrow view of data and are making decisions based on the shadow. The shadow is real, but the duck is more real, and we have to broaden our context to get the full truth.</p>
<h2>Common Research Mistakes</h2>
<h3>Customer Interviews</h3>
<blockquote><p>People can make up an opinion about anything, and they’ll do so if asked. <cite>Jakob Nielsen</cite></p></blockquote>
<p>Customer interviews are a common method of user research that teams mess up. The answers you get in interviews depend on the questions you ask, and by asking the wrong questions, you can get really excited about data that doesn’t matter at all. We ask feature-based questions such as “What is the thing you do most often in Jira?” and get narrow answers. We should be asking questions like, “What are you doing most often when you come to use Jira?” With broad context, you learn about the “why”, hear gaps and opportunities that the product may be missing, and can eventually dig deeper to probe into the features themselves.<br />
<img loading="lazy" decoding="async" class="aligncenter size-full wp-image-13684" src="https://mtp2017stg.wpengine.com/wp-content/uploads/2018/08/Leisa-Reichelt-on-stage-at-mtpcon.jpg" alt="Leisa Reichelt speaks at mtpcon" width="2048" height="1365" srcset="/wp-content/uploads/2018/08/Leisa-Reichelt-on-stage-at-mtpcon.jpg 2048w, /wp-content/uploads/2018/08/Leisa-Reichelt-on-stage-at-mtpcon-300x200.jpg 300w, /wp-content/uploads/2018/08/Leisa-Reichelt-on-stage-at-mtpcon-768x512.jpg 768w, /wp-content/uploads/2018/08/Leisa-Reichelt-on-stage-at-mtpcon-1024x683.jpg 1024w" sizes="auto, (max-width: 2048px) 100vw, 2048px" /><br />
To do better customer interviews:</p>
<ul>
<li><strong>Start with wide context:</strong> don’t start with questions about your product</li>
<li><strong>Be user-centered, not product-centered:</strong> you should be able to use the same script for a competitor’s product</li>
<li><strong>Invest in analysis:</strong> get transcripts of the interviews, take the time to do affinity diagrams, and really get the full value from the research</li>
</ul>
<h3>Usability Testing</h3>
<p>We make decisions based on usability testing all the time. A popular method that has emerged is “Unmoderated Remote Usability Testing.” With this type of testing, you use software that allows you to build tests, post them, and let people complete usability tests from anywhere on their own. It allows you to get a lot of testing done without ever talking to a user directly, but it comes with a lot of pitfalls.<br />
<img loading="lazy" decoding="async" class="aligncenter size-full wp-image-13685" src="https://mtp2017stg.wpengine.com/wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-usability-testing.jpg" alt="Leisa Reichelt talks about unmoderated usability testing at mtpcon" width="2048" height="1365" srcset="/wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-usability-testing.jpg 2048w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-usability-testing-300x200.jpg 300w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-usability-testing-768x512.jpg 768w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-usability-testing-1024x683.jpg 1024w" sizes="auto, (max-width: 2048px) 100vw, 2048px" /><br />
With in-person usability tests, an experienced researcher is able to discern the difference in what a user <em>says</em> and what they <em>do.</em> They’re able to see body language that indicates when a user is frustrated, or see when a user only completed a task by randomly clicking around a screen. But with unmoderated usability testing, teams often simply download data rather than actually watch the videos of the tests; these things get turned into a simple metric of “task completed”, and quotes and ratings are pulled without the broader context.</p>
<p>To do better usability testing:</p>
<ul>
<li><strong>Make tests task not feature-based:</strong> test what people would really be doing, not the feature you wish they’d use</li>
<li><strong>Watch the videos in real time:</strong> don’t just download the data from unmoderated usability testing</li>
<li><strong>Look beyond task completion:</strong> it’s not OK that people eventually get to the right outcome &#8211; the quality of the experience along the way matters</li>
</ul>
<h3>Surveys</h3>
<blockquote><p>Faith in data grows in relation to your to your distance from the collection of it. <cite>Scott Berkun</cite></p></blockquote>
<p>Product managers love surveys. The further away you are from the point of collection of data, the more you trust it. But misleading questions and a lack of context can provide data that tells a very different story from the truth. Leisa gave an example from Atlassian Summit, where a user said in a dot-voting survey that they had great work-life balance. When she asked them about their answer, they said “I can work from home, so even though I work too much, I have flexibility.” The context gives a different perspective from the data point. Just because you can put it in a graph, doesn’t make it science. We need to be way more critical about what data we trust.<br />
<img loading="lazy" decoding="async" class="aligncenter size-full wp-image-13688" src="https://mtp2017stg.wpengine.com/wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-survey.jpg" alt="Leisa Reichelt shows a bad user survey at mtpcon" width="2048" height="1365" srcset="/wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-survey.jpg 2048w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-survey-300x200.jpg 300w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-survey-768x512.jpg 768w, /wp-content/uploads/2018/08/Leisa-Reichelt-mtpcon-survey-1024x683.jpg 1024w" sizes="auto, (max-width: 2048px) 100vw, 2048px" /><br />
To do better surveys:</p>
<ul>
<li><strong>Use qualitative techniques before quantitative:</strong> do interviews or contextual enquiry before you design a survey</li>
<li><strong>Triple test your survey:</strong> make sure people understand questions consistently and are able to answer accurately</li>
<li><strong>Use humans to analyse words:</strong> don’t rely on automated text analysis. Read the feedback.</li>
</ul>
<h2>Stop Quantifying Qualitative Research</h2>
<p>Why do we take these research methods and use them incorrectly? Leisa thinks the main reason is that we don’t have time. But if we claim to be “customer obsessed”, why are we trying to box research and empathy into the smallest amount of time possible? Take the time to do it right.</p>
<p>Leisa leaves us with the metric we should actually be aiming for. Rather than completing bad user research, just ask: <em>“In the last <strong>six weeks,</strong> have you personally observed a customer, using your product?”</em> If we’re going to make better products, the truth of our customers&#8217; experience is what we need. And truth can’t be automated or graphed. It takes time, and it takes humans.</p>
<div class="g g-21"><div class="g-single a-73"><a class="gofollow" data-track="NzMsMjEsMSw2MA==" href="https://community.mindtheproduct.com/c/general/"><img decoding="async" src="https://mtp2017stg.wpengine.com/wp-content/uploads/2021/10/Membership-Circle-banner-ad-scaled.jpg" /></a></div></div>

core/freeform

Prioritised Fireside Chat: Make better product decisions with JTBD

Connecting to your customers in a remote world by Anna Lee Anda

MTP Panel: Using discovery and research to influence key product decisions

Capturing Customer Context for Cross-Channel Experiences by Cheryl Platz

Here Be Dragons – Product and Discovery, by Randy Silver

Iterating Your Voice Product, by Lisa Vigar

Qualitative Intelligence by Mitchell Gillespie

Building Consumer-Grade Experiences for the Enterprise By Ciara Peter

Imaginary Research – How to Design More Human Stuff by Kate Nightingale

Using Research in Material Design by Elizabeth Churchill

Quantifying Qualitative Research by Leisa Reichelt

ProductTank Columbus

ProductTank Tauranga

ProductTank Edinburgh

Prioritised Member AMA - Perfecting product roadmaps

ProductTank San Diego

Navigating difficult relationships panel

ProductTank Philadelphia

ProductTank Bucharest

ProductTank NoVa

Remote Training Workshops

ProductTank London

ProductTank Birmingham

ProductTank Oslo

ProductTank Astana

ProductTank NoVA

ProductTank Tokyo

ProductTank Belgrade

ProductTank Madrid

ProductTank Leeds

ProductTank Zürich

ProductTank Perth

ProductTank Chicago

ProductTank Auckland

ProductTank Richmond

ProductTank Charlotte

ProductTank Reston

ProductTank Washington

ProductTank Sofia

ProductTank Raleigh

ProductTank Ljubljana

ProductTank Kraków

ProductTank Barcelona

ProductTank Dublin

ProductTank Montréal

ProductTank Bogotá

ProductTank Seattle

ProductTank Madison

ProductTank Singapore

ProductTank Portland

ProductTank Budapest

ProductTank Porto

ProductTank Austin

ProductTank Los Angeles

ProductTank Norwich

ProductTank Des Moines

ProductTank San Francisco

ProductTank Helsinki

ProductTank Reading

ProductTank Brighton

ProductTank Valencia

ProductTank Bielefeld

ProductTank Malmö

ProductTank Canberra

ProductTank Paris

ProductTank Manchester

#mtpcon SF+Americas 2022 Conference

#mtpEngage Hamburg 2022 Leadership Forum

#mtpEngage Hamburg 2022 Workshops

#mtpEngage Hamburg 2022 Conference

#mtpcon London+EMEA 2022 Conference