Optimizing the Product Analytics Stack
One of the more interesting BPMA panel discussions last year focused on product analytics tools, process, and methodology. Panelists included Greg Achenbach, the VP Product at PredictAP, Joshua Herzig-Marx who is Head of Product at Brigit, and Douglyss Giuliana, the Director of Product at Alignable. The panel followed the typical format in which the main presentation consisted of host Chris Peterson, BPMA Co-Director of Programming, asking questions and the panelists offering their considered opinions.
Let me take a Q&A style approach to summarize the key insights for those who couldn’t attend.
Q #1: How do you think about coaching someone new to product management in the art of analytics, i.e., when to use analytics, when not, what are the right tools at the right time?
The panelists highlighted the need to be very clear on what is being measured and why. In effect, start with the end in mind. Asking why and being very clear on objectives was highly emphasized. It was recommended to start with business KPIs as the measurement goal, as a means of clarifying objectives and getting the attention and buy-in of the business-minded folks in the company.
It was noted that a user-oriented perspective is crucial. Analytics can and should be used to learn about users, their needs, and objectives. Discovering user needs and the problems they are trying to solve is a hallmark of product management, and product analytics should and can be used as a very powerful tool in the discovery process.
Q #2: What are the major categories of tools and typical use cases for those tools?
The consensus of the panelists is that product analytics tools fall into three categories:
- Tools that report on and enable you to visualize data in your database, like Looker and Tableau.
- Product analytics tools, like Google Analytics and Amplitude, that collect data on and analyze the behavior of users in the aggregate, tracking events like the site engagement rate, pages visited, and conversions.
- Tools like Hotjar and Fullstory that capture and display a full user session, providing the ability to drill down the series of interactions that comprise the user session.
It was noted that, while powerful, these tools can be even more powerful when used together. The example offered was that of a free trial of a new product. A review of the analytics reveal that conversions were not meeting expected or desired volumes. To answer the question “why,” the user session tools demonstrated that most users were exiting the conversion flow at a particular point, suggesting a design or information gathering flaw that needed attention.
It was also mentioned that analysis of non-user related data is important as well. For example, pricing/packaging data might be available in a backend database and the interrogation of this data with tools like Looker or Tableau, or even Python with its myriad packages and frameworks, in conjunction with front-end data analysis tools will provide the full picture.
Q #3: Where is the optimum place to start analysis and what tools are best?
All panelists agreed that none of the off-the-shelf tools can meet every requirement, so starting off by implementing a Customer Data Management (CDM) platform was recommended, and the consensus choice was Segment. A CDM platform provides a central repository for all analytics data as well as a means of creating custom event tracking to enhance the data analysis of the off-the-shelf tools. Once implemented, a platform like Segment provides product and business staff with plug-and-play capability that can accommodate multiple off-the-shelf tools without the need for engineering involvement.
Q #4: What tools are you using now?
Josh Herzig-Marx noted that his company uses Segment to capture events in one place. Next mentioned was Amplitude mainly for funnel tracking, providing for on the order of ¾ of product analytics. Heap and Mixpanel were also mentioned. Tools like Periscope and Looker can be used for direct database query.
Two of the panelists noted that they don’t typically track individual users, because even with hundreds of thousands of users, some are eager to speak, and they can just pick up the phone.
Doug Giuliana’s company also uses Segment to centralize custom coded events that can be funneled into whatever off-the-shelf analytics product they want. They also use Amplitude, finding it to be the most flexible off the out-of-the-box analytics tools. Doug offered the opinion that Amplitude is a lot more customizable than a Mixpanel. They use Logrocket, which is very similar to Fullstory, but provides logs that allow engineers to track behind-the-scenes events that are not evident through the UI.
Greg noted that his company has used Segment in the past but is not currently. He highly recommended the strategy of implementing an intermediary tool like Segment to code and capture event data and employs plug-and-play capability with other solutions. He recommended Looker as a reporting tool, Amplitude for product analytics, Fullstory for recording and watching user sessions, and ABTasty for A/B testing.
Q #5: How about A/B testing tools?
Feature flagging was a technique highly recommended specifically for product testing. In feature flagging, flags are built into the product which enable the availability of a particular feature or feature set for a targeted user cohort. It was noted that Google Firebase for mobile apps has a nice experiments feature that introduces a little bit of automation and control around feature flagging and gives product staff the ability to participate hands-on in the process.
Optimizely was identified as the gold standard is A/B testing tools. It was noted that Optimizely is flexible enough to accommodate marketing-oriented testing of landing pages and web content, which have slightly different requirements than product testing. It was also recommended for its powerful built-in analytics.
ABTasty was recommended for its marketing focus, simplicity, and ease-of-use.
The A/B testing tools are great, especially for front-end testing. Not requiring an engineer to change text on a button or a paragraph, or move an image provides a lot of control, freeing the engineers to concentrate on meatier tasks.
Q #6: What are common pitfalls and how to manage through them?
Not tracking the right events/data. Companies can architect products to track the data not aligned with KPIs, track data simply to say they are doing so, or not track data at all. Also, it is important to get analytics into the product or have third-party tools in place before launch to verify that user actions are consistent with stated preferences and intentions.
Not bringing engineering on board. Providing extensive event-tracking requirements to an engineering team as part of overall product requirements is no guarantee that something useful or even correct will result. It was strongly recommended to bring the organization, including engineering, up to speed on what, how and why of analytics tracking, and how those analytics relate to business objectives. Having done so can ultimately result in a predictable, reflexive ability of your engineering team to insert appropriately named events that ask and answer the right questions. Trying to do so after the fact can be a struggle.
Have the right tools in place, but don’t put those tools and the data they produce to good use. Often companies don’t invest enough time and resources in becoming expert in the use of analytics tools and technologies. It was recommended that organizations use these analytics tools to find answers to questions that haven’t been asked or perhaps anticipated. As has been emphasized repeatedly, It is important to use the tools to find answers to specific questions that align with business objectives, but these tools and technologies can also be used to discover things that are unexpected, potentially offering a competitive advantage.
