The following is an excerpt from my book on growth for SaaS companies, “Growth Pirate! An Entrepreneurs Guide To Achieving Data-Driven Growth In Your Startup”.
This excerpt is part of the introduction chapter that sets the foundation knowledge for the rest of the book to build upon. I’d recommend this post before diving into any other analytics articles as it’s a perfect primer for more advanced concepts and techniques.
In this post, you’ll cover:
- Working with Metrics effectively
- What makes a good metric
- Actionable vs. vanity metrics: How to spot the difference
- How to avoid Analysis Paralysis
- Segmentation and cohorts explained
- A/B Testing
Working With Metrics
Before we can explore how to use a startup metrics framework like Pirate Metrics effectively, we should cover some ground on the fundamentals of measuring data and turning this into useful information.
Working with analytics and metrics is a dangerous game, and it’s far too easy to drown under data, or allow a metric to walk your team off of a cliff.
I’ve sat in meetings and heard Marketing Directors justify their marketing budget based on their increase in Twitter followers. And listened to a trade exhibition sales guys try to convince me that a $2,600 booth is a good buy because on average, “exhibitors receive 300 business cards in their fish bowls”.
If you don’t understand the basics, metrics can lie to you. Or worse, people can manipulate metrics to lie to you.
If we want Pirate Metrics to work for us, we need to know how to identify good metrics and how to place data into our daily work without doing more harm than good.
What Makes A Good Metric
Not all metrics are created equal. To be effective, a metric needs to meet certain criteria. If it doesn’t hit these, then it’s pretty useless to your business.
- Accessible: If everyone in your team can’t understand what the metric means, then it’s useless. People need to be able to discuss it, remember it, apply it to their daily work. This doesn’t just mean having simple, human-readable names for metrics, but also a clearly understandable explanation of how that metric is calculated and what it means in the context of your product and users.
- Actionable: A metric is actionable if you can make a decision based on it’s result, and go and do something about it. The opposite of an actionable metric is a vanity metric.
- Comparable: If you can’t compare the metric to another date period, or to a different version of the product, then you have no frame of reference.
- Rates or Ratios: Counting absolute numbers can be useful, but monitoring the rate of change of that number, or the ratio of that number to another, is more valuable. Knowing the current speed of a car is OK, but knowing if it’s accelerating or braking is far more useful.
I’m not saying that metric that don’t meet this criteria are inaccurate or false, and I’m not saying it doesn’t tell us something about your business. It just doesn’t tell us anything useful for improving growth.
Actionable Vs. Vanity Metrics
There are two core types of metrics: those that can inform the decisions you make, and those that make you feel good. The former are the kinds we should be tracking, and the kinds we’ll discuss in this book. The latter are the kinds of metrics that serve no real business value, and yet very successful business products exist to provide exactly these…
That’s because, whether we like to admit it or not, we all like showing off. We have egos. Telling your buddies over cocktails that you had 1million app downloads last week is a cool number. Quoting to a journalist that you transferred 4 Petabytes of MP3’s last month is going to help your story.
Vanity metrics have a place in startups… and it’s when you’re boasting and trying to impress people who don’t really care about making improvements in your startup.
Important Note: Anecdotally, I’ve noticed many investors seem to ask for numbers that are total vanity. Whether this is inexperience on part of the investor, or if it’s because they need these vanity metrics to satiate their own stakeholders, is beyond my knowledge. All I know is, it never hurt to throw a few vanity metrics into the first few slides of a presentation to add an acceptable amount of ‘Wow Factor’.
But when you want data to drive you, to inform the decisions you make, to allow your company to become more intelligently managed, that’s when we need to engage with actionable metrics.
The easiest way to tell if a metric is actionable or not, is to ask yourself “What will I do differently based on the results of this metric?”. That’s a lot easier if the metric in question was only tracked because you had a specific question that you needed asking.
For example: “Which social network sends us the best traffic?” is a typical question from managers. So, you may decide to track “Total unique visitors coming from twitter”? Nope. Vanity. How about “Total paid signups coming from twitter traffic”? Getting better. But to be truly actionable, we should track “Signup conversion rate from twitter traffic” and compare that to “Signup conversion rate from Facebook”.
With these metrics, we could directly decide which platform to spend the most time on next month. Or which platform to increase ad buying.
As a cheat rule, if a metric is an absolute number, that is only ever going to increase, it’s almost definitely a vanity metric. If a metric is a conversion rate, comparable, and translates to specific business value then it’s probably actionable.
- Total signups = Vanity metric
- New signups this week = Actionable
- New signups this week via twitter = Actionable
- Weekly % growth of new signups this week via twitter = Actionable
For good form and guaranteed success, start with the question, then create the metric that best answers that question.
Possibly the most important change in making your metrics more accessible is to simply reduce the amount of metrics you’re monitoring at a time. When we’re flooded with data and information, we struggle to focus.
The average CEO in a small tech startup will receive around 100 emails a day. They will see around 50 news headlines (and that’s without checking Hacker News that day!), and have around 35 separate conversations with colleagues or customers.
That’s a huge amount of information coming in, just to handle the day-to-day operations. This isn’t even exclusive to the CEO – each member of the team will have similar experiences of information overload, even if the medium itself is different.
We need to keep our signal-to-noise ratio high so that we can take effective action on the data. Tracking 10 metrics might seem like a great idea, but really, how many experiments or changes in your product or marketing can you make today or even this week that will have an impact on those metrics? Chances are that on any single day, you’re making changes that are going to affect 1-3 core metrics.
So why have the others distracting you? When we have too many metrics, we try and take too much action. We consider too many decisions, strategies, problems.
If you’re still in doubt about the epidemic and effects that is analysis paralysis, observe an average marketer as they explore a Google Analytics dashboard. How many data points, percentages, numbers, charts etc. are thrown at you in a single browsing session? And you only came here to find out how your latest email campaign performed!
By sticking to a few metrics, and keeping them visually attractive and understandable, you’re far more likely to do anything with that information. The Growth Pirate method encourages focussing on a handful of metrics at any time, and changing these around regularly.
Segmentation and Cohorts
A segment of users is a group of people who share common attributes or properties. One segment might be ‘all visitors who are from United Kingdom’, and another segment could be ‘all users who came via a twitter ad, have created less than 3 projects and have invited 4 or more colleagues’.
The same person can belong to multiple segments. It’s useful to create segments with very fine granular detail, while also creating broader segments, for different purposes. You may have done this already with email marketing campaigns.
A cohort (often incorrectly used to mean a segment) is a segment over a period of time. For example, if we create our segment of ‘active users’ and then analyse those users over a timeline, starting from the date since they first signed up, we’d be analysing a cohort.
Cohort analysis is often done using a Cohort Matrix. For example, an ecommerce store might track ‘volume of purchases’ as a cohort analysis based on the date since the users first purchase. They could then view the exact same chart, broken down by users who are in the ‘United Kingdom’ segment vs. the ‘Canada’ segments.
Cohorts can be harder to grasp without a bit of practice and seeing them in the context of real world data:
The above cohort matrix shows an app that appears to be increasing it’s activation rate in the first month (increasing from 85% for first month to 95%) but also seems to be tackling it’s churn problem. If the above looks a little confusing at first, don’t worry I was exactly the same the first few times I saw these!
Cohorts compare behaviour over time, with a common start date. Segments allow us to group people based on properties, and to breakdown reports by those segments. A small semantic difference, but very important when we’re exploring metrics and communicating with the team.
People innovate, computers measure. As entrepreneurs, we’re capable of making huge leaps in ideas. “Instead of a faster horse, what about if we build an automobile with wheels and an engine?”. Machines however, are capable of optimising within the known confinements of the current scenario. “What makes a horse faster, feeding grass or feeding oats?”.
Entrepreneurship, gut instinct, innovation and qualitative research allows us to create a first prototype, but when we need to start optimising within that experiment, we use A/B testing. “Horses fed with grass (Group A) will be raced against horses fed with oats (Group B)”.
Ensuring the whole team understands the language and the fundamentals to metrics and analytics is an important part of implementing a company-wide Growth Pirate strategy. There’s nothing sexy about this stuff, it’s stock theory knowledge that we need before we can move onto much, much cooler stuff!
If you’re a developer who is trying to “look up” more often and start looking into company growth; or if you’re an outbound sales associate who is curious about the lower lead quality you’re receiving; or if you’re a marketer who wants to get a solid grounding in a modern approach to data-driven marketing… the above is all the knowledge you need to start learning how to become a Growth Pirate!