Our Insight and Innovation Director Nik Pearmine has written the following piece for Quirk’s Marketing Research Review about Big Social Data, and how it can be maximised for effective Market Research.

Social big data is creating a catalyst for change in insight

Never have we lived in an age where so many shared so much, nor has this information been so readily available. Over the last five years, big data has been touted as the solution to every problem brands and businesses face but while many analysts and marketers have implemented social data listening tools to track keywords and conversations, most are apathetic to its usefulness beyond minor, short-term marketing decisions like messaging iteration and content activation.

Scale and robustness of insight

Through artificial intelligence and machine learning technology, researchers can utilize data sets on an unprecedented scale. Combining large data sets with the right analysis techniques allows researchers to deliver more robust, quantitative, business-case-ready facts and figures.

Similar benefits are also apparent when considering the value of social data in terms of cost and speed. For the equivalent cost of recruiting 10 respondents for a focus group, a researcher can analyze 10 million naturally surfacing online conversations. High-value analysis can be achieved within days rather than months. It also offers the opportunity to design, test and learn by making it easy to conduct quick, smart pilots to sample data, report initial findings and then scale up to build out longer-term research projects.

Consumer trends in real-time

Advanced algorithms effectively ingest data from a myriad of social and public sources, then enrich and reassemble it to define correlations of significance. This opens the possibility to accurately and objectively plot and predict human behavior. Social prediction can help businesses anticipate and prioritize what ingredients, products or consumer trends are most likely to impact their brands over the next three to 24 months – important factors to innovation planning.

Naturally surfacing

One common criticism of research is the bias introduced by a researcher’s preconceived questions and discussion guides – they can only ask questions about what they already know. Social data, on the other hand, gives access to unprompted consumer conversation, using data science to aggregate and find naturally surfacing topics. This knowledge can then be used to inform and define the qualitative research program and identify questions researchers might not have known to ask.

The same is true of problems in the field of recruitment. Typically, after a briefing from the brand, a team of researchers creates a discussion guide from which they build a screener. Then a fieldwork team sources a small group of people from the general public who have opted to be included in marketing research and fit their criteria. At best, you’ll end up with individuals who roughly match the requirements of the brand but every stage relies on human judgment, allowing bias and subjectivity to creep in. Social data allows brands to objectively detect and engage influencers who have driven and participated in the subject content across one or more mediums. What’s more, by using a data-driven approach to identify key influencers, those influencers can then be recruited and engaged within a community, turning the data into a pipeline of innovation ideas.

Human behavior

Social data is driving a connection with real human behavior at a scale and speed that is not possible using other techniques. By using big social data, market researchers are delivering research programs based on the insight from millions of consumer conversations rather than a select few, placing robust, objectivity at the heart of insight and innovation.


The full article can be read here on Quirk’s Marketing Research Media.

Nik is Executive Insights & Innovation Director at Black Swan Data