Call to market researchers:

Move data to the center and surveys to the edge!

Photo by Ivan Bertolazzi from Pexels

Amazon is an amazing company and deserves attention if not admiration on how it creates, shapes, and manages markets -A reason why I consider the annual letter of its founder and CEO, Jeff Bezos, a must read for everyone, be he or she a seasoned professional or a graduate student seeking to learn from one of the most powerful companies in the world.

In his 2018 letter to shareholders, Jeff Bezos wrote that market research doesn’t help, which stung me as I spent many years in this industry, first managing projects then selling projects. Once the surprise passed, I admitted that Jeff Bezos’ statement has triggered a desire to analyze the market research industry. A competitive analysis makes sense in the light of the current digital transformation and business datafication taking place everywhere and in every company.

The current paper is a call to the industry professionals to tell them it is time to think of market research in a different way now that the lines that used to separate research companies, consulting firms, and data agencies are blurring. Nowadays, every data-centered company is a consulting and implementation company. This explains the current frenzy of mergers and acquisitions in the sector of digital and data services.

After explaining that no industry can escape disruptive innovation, the paper describes market research as-is and suggests how it should be. The proposal of change is based on lessons from how strategy consultants solve complex problems and how data agencies leverages client’s legacy systems before considering surveys.

Disruptive innovation

Disruptive innovation rarely sparks within incumbents. They ignore it and once it really starts to eat in their market, they positioned it as a cheaper alternative to their premium solution. By cheaper, they hope clients would interpret it as inferior. A big mistake as clients often give cheaper alternatives a try on small scale. If the alternative delivers on promise, they switch to it full scale. Otherwise, they throw it away with no regret because the lessons learned outweigh the costs incurred.

And once the alternative starts to bite further in the market, incumbents tend to follow the trend but in an awkward way. Instead of totally endorsing the alternative, they are inclined to continue positioning their offering at the center and the alternative at the edge of the solution as if the latter is optional. And they continue doing so without really admitting that a deep transformation is taking place. This creative destruction process may take years but it always ends by the entrants becoming market leaders, the incumbents becoming market followers.

It happened to telecoms, retail, music, media, advertising, and many other sectors. And it’s happening now to the market research industry, where the mainspring is still to survey citizens, consumers, or customers on their intentions, attitudes, preferences, or actions.

Market research

Until recently, market research incumbents did not pay too much attention to consulting firms and data agencies who viewed the client problem in a different way. Instead of selling to the client the need to collect more data (through questionnaire design, survey sampling, and data collection; things that they are not familiar with) they valued the data that the client already has on one hand and the data that is already available, on the other hand, for almost free on the Internet through APIs to the major social networks.

The consequences were dramatic to market research incumbents. Clients now allocate almost all of their marketing budget to upgrade their customer relationship management systems and enterprise resource planning systems, and to a lesser extend to their websites, mobile apps, and social media pages. That is what digital transformation and business datafication is all about. Only a small portion of the budget is allocated to the market research.

In a few years, the market research as we knew it, moved from the center to the edge of the marketing intelligence system of clients in retail, luxury, banking, and almost in all the business-to-consumer markets. The era of data collection as the king is coming to its end, as the focus has shifted from how we collect data to how to learn from data we already have. To take back its crown, the market research needs to change according to this fact.

The traditional process that consists of client brief, questionnaire design, survey sampling, data collection, data statistics, and findings presentation should be revamped to give more importance to the problem formulation and structuring, data audit and mining, problem-solving, and solution selling. The proposal to the client brief should not be anymore “we have a panel to survey for you” or “we will set-up a survey to collect data for you” but it should be “give us access to your CRM data to analyze”, “if need be, we will complement it with more data.

Data in the center, surveys at the edge

In the traditional market research, the process is composed of five steps:

• Step 1 defines the problem and research objectives. In this step, the client and research firm define the problem carefully and agree on the research objectives and constraints. In some cases, the client has already designed a questionnaire and considers it as part of the problem definition.
• Step 2 defines the research plan. In this step, the research firm develops a plan to gather data in terms of questionnaire design, survey sampling, survey method, and project timeline. In the majority of cases, the research firm has developed experience-based use cases they instantiate for each new client to reduce the cost per client.
• Step 3 collects data. This step is generally the most expensive and the most prone to errors. That is the reason why research firms are mistakenly considered by advertisers and marketers as data collectors. Either the research firm has a panel to use, execute the survey themselves, or hire other companies to carry out the survey, especially when the survey has taken place abroad.
• Step 4 analyzes data. In this step, researchers extract pertinent findings from the collected data and generalize them to an entire population of interest. The volume of data manipulated at this step is far from being big data, but the statistical models, algorithms, and tools are similar. People doing this work used to be called researchers are now called data analysts or data scientists.
• Step 5 presents the findings in the form of numbers and statistics. Often, clients are not fully satisfied either by the findings that are not aligned with what they expected or are overwhelmed by the quantity of data presented without actionable recommendations (hence the importance of solution-selling that consulting firms add at the end of their assignments).

To rejuvenate, market research needs less emphasis on survey data collection and more emphasis on strategic problem-solving as follows:

• Step 1 should be engaged as consulting firms do. Rather than just relying on the contact who sent the brief, consulting firms use TOSCA to state the problem:

Trouble: What makes the problem real and present?

Owner: Whose problem is this?

Success criteria: What will success look alike and when?

Constraints: What are the limits of resources and time-line?

Actors: Who has a say in the way the problem will be solved and what do they want?

Before starting anything, consulting firms try to understand the big picture to understand if the top-level goal is a transformation in client experience, internal operations, or business model. They also request to meet all the actors, those who initiated the project as well as those who will be impacted by the outputs of the project:

• Step 2 should not be engaged as a survey research planning but more as problem-solving planning where we define the kind of data we need and the order in which we should process it. Is it a media planning optimization problem? a direct marketing personalization problem? a sales’ forecasting problem? a customer segmentation and scoring problem? Each of these problems appeal for certain types of data, the majority of which is located into different information systems of the client.
• Step 3 should be enlarged to include data audit in which the research firm should mimic what data agencies do. First, consider data that is already in silos within the client, in particular data located in the customer relationship management systems. Then, if need be, complement it with open data. If no open data is relevant, then consider buying third-party data from GDPR-compliant data brokers. Finally, if no such data exists or is relevant, then consider a survey that will be more focused and therefore less expensive and less prone to errors. Proposals for surveys should be at the end and only if neither client data, nor open data or third-party data is available and workable. In other words, surveys should be at the edge and not anymore at the center of the project.
• Step 4 should include modern tools that not only have very sophisticated statistics features but informative visualization features to help both the data researcher and the survey manager. The final client quickly captures the most important insights that will lead to actionable recommendations. Lots of data mining tools are available free of charge on the Internet or as commercial products that include all the steps from data cleansing to data processing, data analytics, and finally data visualization. The majority of them do not necessitate advanced programming skills but analytical and synthesis skills.
• Step 5 should be enlarged to what consulting firms call solution-selling where the objectives are to help the client understand the situation. Rather than presenting slides with numbers and statistics, the survey manager should design a storyline and help the client to sell it to his or her management, and to the latter to do the same with regards to the top-level management. This step is of the highest importance if the results contradict what the client was expecting or recommend for a business an organizational transformation. Think of marketers who learn that their products should not be priced as premium products, advertisers who discover that their budget can be reduced to reach the same audience, and so on.

Wrap-up

For market research firms to win back the center of delivering insights and not being pushed to the edge of collecting data, they need to lower emphasis on data collection and raise emphasis on the larger problem at the origin of the request for proposals. To this end, they should learn from how consulting firms solve problems and sell solutions to the general management and how data agencies master data sourcing from internal databases and external suppliers. In short, now that clients want to leverage and value their data even if it resides in silos, surveys should be considered as the last resort and no more as the primary solution to client problems.

About the author and the paper

Strategy consultant and interim executive for complex technological transformations.

This paper benefits from strategy and implementation work done for top-tier consumer research firms, digital and data agencies, CPG/FMCG retailers, luxury and beauty brands, and a large mutual insurance.

Comments are welcome. Please, either post them on Medium or Linked, or email them to hlaasri at hbmjpartners dot com.

#marketresearch #innovation #strategy #businesstransformation #data