Predicting Consumer Behavior: Do Retailers Know What You'll Buy?

In late 2017, I spoke with Dave Cherry (Executive Strategy Advisor with Cherry Advisory, LLC) and he shared the following breakdown of the key foundational elements of a single view of the customer:

Cherry Advisory, LLC

Today we are going to do a deeper dive into the most difficult, yet most valuable element in this foundation, behavioral patterns, and more specifically, predicting customer behavior.

Dave Cherry: Predicting consumer behavior accurately is the “holy grail” of both the single view of the customer and retail/consumer marketing. The benefits to the customer (increased personalization, contextually relevant promotions), marketing (improved targeting, optimized spend and touchpoints), finance (improved forecasting of sales, labor, and inventory), and customer experience/care (improved attraction, engagement and retention) are significant. The insights that can be gained by knowing what a customer is going to do with accuracy is exponentially more valuable than knowing what they previously did (transactional data) or their demographic profiles.

Drenik: So how are companies doing in terms of predicting consumer behavior accurately?

Cherry: In terms of aggregated (non-personalized), high level predictions, we are seeing some good predictions. Using data from Prosper Insights, NRF frequently publishes their outlook on total spend (e.g. Mother’s Day spending is expected to increase by x% this year) as well as category spend (e.g. flowers are predicted to be up y% and candy down z%). Many retailers are able to predict category/department, and sometimes choice level sales across their chain accurately. But while these are helpful in aggregate, it doesn’t help retailers make progress towards that holy grail – predicting a specific customer’s planned spend by category, and ultimately by attribute/choice.

Drenik: Can we get to that level of specificity?

Cherry: Yes, but of course with some work and an understanding of probability and confidence. Earlier this summer, at the CbusRetail R19 RetailReThought Conference, I facilitated a panel discussion on this topic with Megan Kvamme (CEO & Founder, Factgem), Dr. Martin Block (Professor Northwestern University) and Phil Rist (EVP Strategy, Prosper Insights & Analytics).

In that discussion, Dr. Block shared his research methodology and results around predicting cannabis usage among consumers. He used publicly available data (e.g. states where cannabis is legal) and aggregated survey data (e.g. how likely are you to use a cannabis product), to create a pruned decision tree. Through that, he found the most probabilistic profile of a potential cannabis user (a male with some college education, living in a state where marijuana is legal, with multiple children and a pet – specifically a bird).

Dr. Martin Block (Professor Northwestern University)

He also shared how he could vary any of those attributes and obtain a new probability of cannabis usage for that new profile with an associated confidence level.

Drenik: So then, it would seem easy for a cannabis retailer to just target individuals that fit those highly probable profiles, and watch conversion and revenues increase. But it’s not that easy, is it?

Cherry: Definitely not. In this example, most customer databases would only have gender and zip code. Few, if any, would also have the necessary attributes of number of children and pet ownership. So, it not so simple as to query your database for all existing customers that fit this profile.

Even if a retailer had this data, as Megan shared, it is likely in multiple legacy databases that are difficult to combine efficiently in a meaningful way. These legacy systems are not integrated and create operational inefficiencies and gaps in information. However, a tool like Factgem, as Megan shared in this video from the conference, can not only easily integrate disparate internal systems, but it can also pull in other relevant data from external sources such as weather or Prosper’s consumer insights.

Drenik: That certainly does help, as these data sources can now be combined by business users without deep technical expertise. This is progress, but is there more?

Cherry: Yes, there’s more. Even when available internal data on customers is linked to consumer insights, for example using Factgem and Prosper, retailers are still missing the most sizable potential market - those individuals that fit the profile but are not currently in their database. This is especially relevant in this example, as cannabis infused products are still new and few retailers have any historical transactional or demographic data geared towards this category.

Drenik: So how can a retailer fill these information gaps?

Cherry: First, the retailer has to identify the necessary data attributes that are missing. There are many data enrichment services available that can take even a single piece of identifying information (e.g. email) and append gender, zip, children, and other attributes. They also typically offer pre-defined personas that can be connected to a customer profile such as “big spender”, “fashionista”, or in this case “cannabis user”. While these services are helpful, they are not a panacea, as my experience has shown roughly 50% match rates on identification of the individual and with varying accuracy of the appended attributes. But we are getting closer.

I’m also excited about Prosper’s new Amazon Sagemaker models because they deliver something beyond the value that the data enrichment services provide – probabilities and confidence levels of specific behaviors linked to an identified individual. This allows a retailer to personalize a marketing message, a promotional offer and service to a specific individual, thereby increasing relevance, engagement and conversion.

Drenik: Are there any other obstacles would a retailer face in trying to accurately predict behavior?

Cherry: Absolutely, and it is the most difficult yet impactful attribute – context. In today’s digital, mobile economy, customers are constantly moving through different stages of the customer journey (which is no longer linear). A specific individual, with the same transactional and demographic attributes, can evoke different personas daily (and sometimes even within the same day). In one situation a customer may be seeking value from their favorite retailer, but looking for convenience or experience at another time.

Let’s take the potential cannabis user (male, with children and a bird) as an example. He may also be a working professional. So, targeting him on a Friday during his lunch hour or on Sunday morning when he is shopping with the kids, is not likely to yield the desired result, a conversion. Likewise, targeting him on a Saturday afternoon out with his friends would be both highly relevant and contextual – this is, when he’s more likely to come to the store, sample products, and make a purchase.

Many retailers talk about having the right product at the right price at the right time. The right time not only refers to having the inventory, but also establishing that connection with the customer within the right context. That is a challenge that many are still working to solve.

Drenik: And that sounds like a topic for another conversation. Thanks Dave.

 

I cover consumer-centric insights and analytics that provide executives with solutions needed to drive strategy. I am the CEO of Prosper Business Development where, for ...