Please rotate your screen


Understanding what consumers want without ever asking them

Black Swan’s Chief Strategy Officer Hugo Amos examines the power of social listening in brand strategy.

This article was published on PSFK and can be read here.

According to Harvard Business School, there are over 30,000 new consumer products launched every year and around 80% of them fail. Finding success in the consumer packaged goods market is a numbers game with the odds seemingly stacked against a brand’s favor.

Earlier this month Coca-Cola launched four new flavors of Diet Coke (Ginger Lime, Twisted Mango, Zesty Blood Orange and Feisty Cherry), along with a packaging makeover for their flagship Diet Coke flavor. Going by the Harvard Business School statistics and traditional methods, three of those new flavors will flop with consumers. But given the data-rich, always-on information age we live in, should this really still be the case?

Traditional methods of qualitative and quantitative consumer research include things like surveys, interviews, observations, focus groups, and these largely haven’t changed for the last 50 years—aside from perhaps using mobile phones to conduct surveys or webcams for interviews. Added to the mix in more recent years have been things like social media analysis and, in particular, social listening.

Social listening has become a standardized research practice for brands over the last few years, but its application to drive business intelligence for successful product innovation is limited in two main ways. Firstly, for social listening, brands need to know which keywords and topics they should be searching for in the first place. This means that they can only really track ‘known knowns.’ It also means they are the ones setting the parameters—what they think should be key terms, not necessarily those that the consumer is using in the ‘real world.’ Secondly, social listening is a rear-view perspective and looks only at what consumers have said in the past—problematic if you’re trying to find the next big thing.

For brands, the traditional research methods are familiar and comfortable but tend to be expensive, biased, outdated and incapable of informing them about what’s coming down the line and how to adapt to reap the benefits. Enter big data.

So much data is created across the internet every day. As a quick snapshot, per minute there are around:

455,000 tweets
4.1 million videos watched on YouTube
46,740 posts on Instagram
3.6 million Google searches
Obviously, this sounds hugely impressive and is partly what sparked the big data gold rush. Big data was heralded as the key to unlocking a wealth of key information that would have an enormous impact on a business’ bottom line. Many expected golden nuggets to just emerge. But it simultaneously caused many companies to feel more than a little overwhelmed after trying to embrace big data without proper guidance. Trying to analyze this overload of information quickly becomes a complicated mess of data that’s incredibly hard to understand, let alone pull insights from.

What’s more, data is only as good as the lens you put against it. Getting the right lens and clean data sets throughout are needed to take big data from spreadsheets full of unintelligible information to those very golden nuggets brands have been hunting for.

Social prediction is the evolution of social listening and uses consumer defined datasets, combined with smart data science and technology, to find early-emerging trends and associations that would be impossible for any analyst or team of analysts to spot alone.

Mark Twain famously said that “there’s no such thing as a new idea… we simply take a lot of old ideas and put them into a sort of mental kaleidoscope.”

In today’s world, we’ve reached a point where there’s so much information out there that, arguably, every product idea is already online somewhere. As a brand, if you want to know the next biggest flavor combination, the next must-have ingredient or desired ‘added benefit,’ it’s already being talked about and experimented with—it’s just a case of discovering these ‘unknown unknowns’ and knowing how to act on them. This is precisely where social prediction comes in. It allows brands to analyze the naturally occurring topics of conversations that 10 million people are talking about online, rather than what 10 people said in a focus group.

It’s fair to say, however, that information has never really been an issue for brands. They don’t tend to lack for insight on consumer trends, whether it’s the latest trend report, industry influencer blog, bespoke in-house research or sales data. A brand will already have an abundance of information at its fingertips. What is often lacking is knowing exactly which insight and trend to act on and prioritize in a robust, measurable, company-wide way.

Again this is where social prediction can help by accurately mapping trends by their level of maturity, predicting which ones will continue to sustain growth over the next year and beyond, and which ones will fade as quickly as they emerged. It helps brands take a scientific approach to accurately forecast what consumers will think and do next, and provides this valuable insight for decision makers in seconds rather than weeks and months.

We’ve long known that there is power in the crowd, but never before has it been as easy to harness it.

Hugo Amos is our Chief Strategy Officer and Co-Founder, using data science to create Black Swan events for clients, enabling a better understanding of their customers and new data-driven solutions.

If you’d like to know more about what we do at Black Swan, please get in touch via – we’d love to hear from you.