This post was first published by First Round's The Product Program.
As product managers, we constantly use data. We use it to align stakeholders with our roadmaps, track the efficacy of what we've built, and prioritize what we should build next. Yet it seems we could always be doing more—cutting a dataset in one more way or interviewing an additional customer. The stakes are high. As Eric Ries says in The Lean Startup, "The only way to win is to learn faster."
Anthony Schrauth, former Chief Product and Innovation Officer at Betterment, and Xiaodi Zhang, 1stdibs' Chief Product Officer, began our seventh session of the First Round Product Program with this quote—and both shared their tips on how to "learn faster." What follows is a recap of this session and the lessons that surfaced for the group.
Rule #76—No Excuses, Gather Implicit Data
While product managers understand the importance of data in the abstract, it often feels like reality gets in the way. We know we should measure everything, but at the same time, a feature is needed urgently or we don't have resourcing to test the value of a new idea. Anthony and Xiaodi stressed the value of finding small ways to measure implicit data—user behavior patterns observed through quantitative measures—in these situations.
For example, Xiaodi discussed how a product manager could prioritize adding a Facebook login feature with little information on how it would impact user acquisition. The product manager could set up a "Painted Door Test," building a button that offers users the option to sign up through Facebook but simply says the feature is temporarily down when anyone clicks on it. While such tests must be used sparingly given their effect on user experience, they are an easy way to determine if users care about a feature. They can also be replicated with a link to a simple landing page or sign-up button that adds a user to a waitlist.
Anthony had the opposite problem at Betterment, an automated investing service. His team had developed a feature they were confident would be a huge hit with users. It was an estimate of the taxes they would pay if they opted to change their portfolio. When the feature was complete, Anthony and his team used the opportunity to A/B test how the feature would influence users to not overreact to short-term market fluctuations. While the feature could have been released immediately, by taking the time to measure the effect on user behavior the team was able to apply the lessons from this testing to future efforts to encourage good investing on the platform.
Using Explicit Data for Illumination
As Anthony and Xiaodi switched to discussing explicit data—qualitative information clearly shared by customers—they aptly kicked things off with the Andrew Lang quote, "He uses statistics as a drunken man uses lamp posts—for support rather than illumination."
We've all likely been guilty of this drunken lamp post-leaning at some point, tempted to cherry-pick data or stretch what a customer said to validate our hypotheses. This is often exacerbated in customer interviews, where it's easy to lead a customer in a desired direction by asking leading questions or framing information a certain way. Our group discussed ways to check these instincts and set up customer conversations that truly illuminate:
Channel your inner Jane Goodall.
Product managers should visit customers where they use the product and observe how people use it. These visits are most effective when the conversation gets specific—ask how customers use the product in a particular situation and see if they'll keep a diary about their usage for a week. Any workarounds customers have developed for the product are particularly instructive.
Check your own biases at the door.
When companies expand into new markets or user groups, it's natural to assume that a value proposition that resonated with other users will resonate with this new group. When Betterment entered the B2B market of 401(k)s, they believed the most pressing need for HR managers was getting employees to save more. But after observing HR managers in action, the Betterment team realized that addressing the pain of manual data entry was the way to win the hearts of HR practitioners.
Even when doing our best to be a neutral observer, we often hear what we want to hear. Bringing team members from different functions to a user interview and discussing what was heard from each person's perspective can be quite helpful in truly understanding a user's needs.
Find the right users for your questions.
At B2B companies, product managers often find themselves engaging with employees from the C-suite to the mailroom and getting quite different feedback. It's important to determine who is relevant for the question you're trying to answer (e.g., who actually makes the purchasing decision or uses your product) and speak with them directly.
Ask these users the right questions.
When polling a group of users, it's important to find a meaningful metric that actually indicates your performance. Net Promoter Score (NPS), a commonly used metric that measures how likely a customer is to recommend a product to a friend, is not universally appropriate. For example, at Betterment, Anthony found that the odd fluctuations in customer NPS correlated perfectly with one thing: the performance of the stock market. Customers said they would refer people to Betterment when the market was doing well and felt less likely to refer people when it wasn't. This was not a reflection of the actual quality of the product or service. Anthony found that Customer Effort Score (a measure of whether a company made it "easy for me [to do something],") was a far better measure of how customers perceived the product.
No universal set of rules can cover how product managers should use data. But a determination to measure everything even when resource-constrained, and focus customer conversations while checking biases is certainly a good place to start. Figuring out how to expand this guidance for our particular company and product is part of the fun of being a product manager.
Quick shoutout to Amber and Camille for helping me with the final touches on this post. You can learn more about the First Round Product Program here. Stay tuned for more posts.
