Vanity Metrics 2.0

For most marketers, big data has failed to live up to its much-hyped expectations. Over the past few years, we've come to realize that more data isn't necessarily better data, and that drilling down to find the golden nuggets we really need is akin to the proverbial search for the needle in a haystack.

The problem with most conventional big data analytics is that they are essentially vanity metrics—numbers that make companies feel good but that offer no clear, actionable insight on what they can or should do with this information. Instead, they give us a false sense of security, allowing us to convince ourselves that having lots of data and big numbers translates into success.

However, large numbers often don't tell the whole story. For example, using site event counts as a measure of success offers little insight into real business performance, to see what's working and where problems may occur. Growth in certain events or site feature usage may give the appearance that business is booming, but, in fact, it may be the result of churn, or a growing yet less engaged user base. Perhaps engagement is plummeting as established users drop off and new users come on board. Or it could be that more people are using the product, but they're actually using it less frequently or for shorter periods of time—hardly the path to building critical lifetime customer value.

The issue is no longer how many users you have, but instead how engaged the ones you have are. All healthy businesses are growing week over week, but is your user base actually becoming more or less engaged? Are "power" users or "drive-by" users throwing off the numbers?

Get Over the Vanity—Get Real Insights

Digging deeper into the data reveals much greater insight that can help truly guide business decisions. Here are three key strategies to get your company beyond the "feel good" vanity metrics to uncover actionable data to drive critical marketing decision-making.

Forget mean. Look at median. Trying to determine what resonates with your typical (and ideal target) user based on aggregate data for average users is like trying to see the forest for the trees. There's just too much "stuff" that gets in the way. For any activity, it's not unusual to see a total count, number of users, and the average all going up, but the median—the proportion of users who actually conduct that activity—staying the same. While all of the conventional stats make this look like growth, in reality, people really aren't performing that activity any more than they were before. Case in point: One image/stock art e-tailer sees 21,000 active users view images about 600,000 times per week. On average, that amounts to 29 image views per user. However, the median shows that the typical user only views 12 images. In reality, power users are viewing proportionally more images, completely throwing off the perception of usage. Clearly, the median can reveal much insight into the actual site activity.

Conduct outlier analysis. Examining outliers—users and activities that fall outside the expected norm—can help identify usability issues and unusual (but good) behaviors. For example, if you observe an unusually high number of Event 1 combined with an unusually low number of Event 2, these outliers can make it easy to instantly understand what's going on. For instance, a photo sharing site identified a large group of users who were creating many albums but never renaming them, instead keeping the default name—an undesirable behavior for the site operator. This revelation enabled the product team to see that the interface for renaming the album was too subtle for some users. Thus, they were able to take action, leading to a change in the interface design and adding help content to show users how to change the album name.

Gather more data. Sure, big numbers are nice, but what if you could leverage them to get big insights as well? Sometimes that requires gathering a bit more data. For example, after launching a new feature, you discover that it's only being used by a tiny portion of your audience. But looking at the raw numbers, it's impossible to understand why. Reaching out to users with a survey or personal email to solicit their feedback and understand what appeals to them can help you understand what's working and what's not to improve the solution. Or, in the case of error events, understanding how often individual users trigger the error—and the steps leading up to the error event—can help you understand the root cause, devise a fix, and prioritize the technical team to implement the fix based on the proportion of users experiencing the problem.

Moving beyond the top-level vanity data to uncover real, actionable insights is the panacea for the big data challenge. Marketers must get past the "more is better" approach when it comes to metrics, user numbers, and growth. The ability to understand the drivers behind the numbers, and optimize tactics designed to fuel those triggers, is where the real value in big data lies. It's time to stop patting ourselves on the back over big stats and start driving real sales and revenue growth.

---

As a cofounder of Preact, Christopher Gooley builds data-driven tools to help companies better understand their customers' actions, enabling support and account management teams to proactively solve problems, increase engagement, and multiply per customer revenue. Previously, he worked at social video start-up DAVE.TV overseeing the development of branded video communities, including the architecture and development of its online software platform used by Disney, ABC, CBS, and other major media brands.

---

Related Articles

At Oracle OpenWorld Day Two, Speed of Big Data Processing Plays Starring Role

Data has no value unless you mine it quickly and effectively.

Choose a Winning Combination of Customer Experience Metrics

Improve customer loyalty with the right mix.

NICE Introduces Customer Engagement Analytics

The platform combines interaction analytics and transaction analytics for insights into the complete cross-channel customer journey.

Navigating the Data Universe

Let big data analytics serve as a guide star to business success.