Six Ways Your Early Startup Data Can Be Misleading

Post written by

Alon Hillel-Tuch

Managing Partner at Stacked Capital, an early-stage technology fund focused on accelerators, incubators and pre-A ventures.

We’ve seen the headlines: "XYZ startup grows 60% per month" What?! Wow, how did they do that? While the percentage may be correct, it’s misleading. If you have four users, and next week you have seven, you grew by 75%! You only added three users. Fine, but what if each user pays $100,000 per month? Then we just added $300,000 -- that’s significant! You are right -- because the other number is big.

Early data can be misleading. It can cause startups to go down dark (and very expensive) rabbit holes, investors to misinterpret the health of a company and everyone to chase their own tails. Here are a few reasons why this is happening:

1. Faulty Polling

Having people that you know become early users creates atypical behavior. They tend to be more supportive, more accepting of issues and more willing to demonstrate good behavior. This can be useful for beta-testing, but it is often a poor indicator of typical user behavior. Startups showing early adoption that consists completely of friends and family doesn’t really prove use. It’s like saying you were able to sell your artwork for $500 without disclosing it was Auntie Sallie who bought it.

2. Low-Hanging Fruit

Early users may be easy converts, as they tend to be the prototype of a perfect user or a tech-hungry early adopter, but they’re often the odd person out. While they seem to prove people are using your platform, you’re just convincing people who don’t need convincing. Are they your real users?

3. Hands-On Everyone

Ever step into an empty restaurant? You end up with five waiters catering to every beck and call -- you get the equivalent to a personal chef and kitchen. It’s just not a realistic experience. You will be treated very differently when the restaurant is full and everyone needs to prioritize and handle probably more than they can. The same is true for your company. The experience given to early users is often not the same experience as you scale. If user feedback is based on the personal touch provided, see it as a friendly reminder to figure out whether you can and want to scale that experience.

4. Focusing On The Wrong User

Companies can have more than one user type and not even realize it. Facebook is an example of a multi-user base; their clients are the companies paying for ads, while their users (you) are the value-proposition to their clients. So, they need to look at not only growing their user-base, but also enhancing the value-proposition (you again). We sometimes forget the duality of certain models and lose focus. What we see more often than anything is a startup putting tons of "amazing partners" in their pitch deck. Often, it’s an affiliate relationship. The startup sends them business, and if it converts, they pay the startup. There is little risk for the partner. So, the startup is great at signing up for affiliate links. Do they have anyone to refer to these partners? Can they even attract the right type of users to refer? Who are they attracting now, and what do they do to keep growing the value-proposition? Remember, this is a business.

5. Drinking The Kool-Aid

All that really matters here is that you understand what you are looking at. Remember that middle school math class where you had to draw a best-fit line on a graph? With startups we sometimes see folks drawing a trend line through an uncorrelated data set to demonstrate that a trend exists, where there really isn’t one. And then they believe it themselves. This is mighty strong Kool-Aid.

6. Playing Statistician

Sometimes teams start measuring the analysis instead of their goals. I know I have been guilty of this in the past. Using bad data and no objectives causes you to continuously reinvent what you are doing as new data becomes available. You need to think about what the data is telling you and whether your behavior is accomplishing your set goals.

Let’s segment "time spent per user" to "high-touch" and "low-touch." How much support time was spent on low-touch users (and define what that means)? How many low-touch users exist? Do the same for high-touch users. Don’t ignore development costs. Often, startups spend significant development work personalizing technology for a specific user (100% high-touch). Custom work can be scalable under the right conditions, but usually, it isn’t. If the only reason you got a user is because you basically built software for them for free, you need to make a very compelling case for how you are going to be able to continue adopting new users that may not receive that level of white-glove service -- or how you will be able to continue to provide bespoke integrations. It can be a real trade-off.

There are a lot of factors contributing to your early data, and most have to do with collection and interpretation, so there may be observer bias. The information present is almost always valuable; it just may not indicate what you want it to indicate or validate what you want it to validate.

Turn the saying "cream of the crop" upside down, and you probably have it right. The cream is the fattiest and most desirable part of the milk, and it tends to rise to the top. It’s your early user -- they rise to the top and optically look like the best part of the milk batch. But you may know them, or they might be rabid fans of your mission statement (maybe the only ones). In startup land, what you want is below the cream -- the cream is a distraction. There is very little of it, and everyone chases it, so eventually, you run out. What do you do then? Get smart and recognize when you have a user that is cream and when you have one that is milk -- and which one you really want.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?

LinkedIn. Read Alon Hillel-Tuch's full execu...">Managing Partner at Stacked Capital, early stage technology fund focused on accelerators, incubators, and pre-A ventures. LinkedIn. Read Alon Hillel-Tuch's full execu...