Concept testing survey questions that get past ‘I’d buy it’
Concept testing survey questions that get past polite answers: what to ask, monadic vs sequential monadic testing, and how to read a top-2-box score.
In short
Three points | 30 seconds- A 2007 study found that stated purchase intentions were more closely correlated with real purchases when people were asked about a specific brand or model, in a comparative mode and over a short time horizon. We build the first into our concept testing survey questions and bring in a comparison with a closing choice question; the third depends mostly on how close to launch you test.
- The same study, by Vicki Morwitz, Joel Steckel and Alok Gupta, found intentions were less correlated with purchase for new products than for existing ones, and new products are what concept tests are usually about.
- Coca-Cola changed its formula in 1985 after taste tests with nearly 200,000 consumers and brought the original back 79 days later, which in our view shows that a test can measure preference and still miss what a product means to people.
Concept testing survey questions decide what a concept test can tell you, and the most common one, “How likely are you to buy this?”, tells you less than its score suggests. This article goes through the questions worth asking, how to choose between monadic and sequential monadic testing, and how to read a top-2-box score before it reaches the board.
Picture your debrief. The new product has been described in a paragraph and an image and shown to a few hundred people who fit the target, and a healthy share ticked “definitely” or “probably would buy”. The chart is green, everyone relaxes, and a year later the product is on the shelf selling well below plan. Nobody lied in the survey: people gave a real reaction to a paragraph on a screen, and the plan needed to know what they’d buy in a shop.
What does a concept test measure?
A concept test puts a description, an image, a price or a prototype of something you haven’t launched yet in front of the people you’d sell it to, and records how they react. That reaction comes with nothing at stake, and it can still tell you a lot if the questions are built for it.
In 2007, Vicki Morwitz, Joel Steckel and Alok Gupta looked at what makes purchase intentions more or less closely related to actual purchasing, using data from a wide range of settings. They found intentions were more correlated with purchase in six conditions:1
- for existing products more than for new ones;
- for durable goods more than for non-durable goods;
- over short time horizons more than long ones;
- when people were asked about a specific brand or model rather than a product category;
- when purchase was measured as trial rather than as total market sales;
- when intentions were collected comparatively rather than monadically, one product at a time.
In our view the first condition matters most here, because concept tests are usually about new products, the side where intentions were less correlated with purchase. We read Steve Hoeffler’s research as pointing the same way for the newest ideas: people were more uncertain about how useful a really new product would be than an incremental one, and they coped by reasoning in ways that standard preference measurement, such as conjoint, doesn’t capture well.5
The study compared results across many settings, so it can’t promise that a better question gives a better forecast. It does suggest where the design of a concept test carries weight. How specific the question is and whether it’s comparative are choices you make when you write the survey. How long people wait between your survey and the chance to buy depends mostly on how close to launch you test. And the fifth condition tells you how to read the result: as a guide to trial more than to total sales.
Six conditions from a 2007 study by Vicki Morwitz, Joel Steckel and Alok Gupta in the International Journal of Forecasting. Purchase intentions were more correlated with actual purchases for existing products than for new ones, for durable goods than for non-durable goods, over short time horizons than long ones, when people were asked about a specific brand or model rather than a product category, when purchase was measured as trial rather than total market sales, and when intentions were collected comparatively rather than monadically, one product at a time. Two of the six are set by the product (new or existing, durable or not). Two are choices in how the question is written (a specific product, and a comparison with alternatives). For a test before launch, the time horizon depends mostly on how close to launch the test runs. The trial condition is about how the result is read, as a guide to trial more than to total sales, so the figure doesn’t count it. Concept tests are usually about new products, so they start on the less correlated side of the first condition. The study doesn’t combine the conditions into a score.
Set each line to match your study.
Source: Vicki G. Morwitz, Joel H. Steckel and Alok Gupta, “When do purchase intentions predict sales?”, International Journal of Forecasting, 2007. The six findings are from the study’s summary.
Which concept testing survey questions should you ask?
Here’s the set we’d start from for a new product, in this order. Treat them as concept testing examples to adapt to your category, rather than a template.
- Before showing anything: “When did you last [do what this product helps with]? What did you use?” Asked first, so the concept doesn’t colour the answer. It anchors the survey in what people do now and shows you the real competitor, which is often a habit, a spreadsheet or nothing at all.
- After showing the concept: “In your own words, what is this, and who is it for?” If people describe a different product from the one you meant, every score after it is measuring their version.
- “If this were available where you usually buy things like this, at this price, what are the chances that you would buy it in the first three months?” on a probability scale from 0 to 10, with odds at every point. Specific, because intentions tracked purchase more closely when people were asked about a specific model.1 Naming a near, fixed period is our own addition: it gives everyone the same time frame, though it doesn’t shorten the time between your test and the launch.
- “Which would you choose: [what they told you in question 1], this, or neither?” A comparative question, in the spirit of the 2007 finding that intentions collected comparatively were more correlated with purchase.1 It’s a different question from the ones the study looked at, but it brings a comparison into the survey. In our experience the “neither” option matters too: without it, people who wouldn’t buy either have to pick one, and the new product can look stronger than it is.
- “How different is this from what you’ve seen before?” A new idea that people see as one more version of what they have is a different problem from one they don’t understand.
- “What would stop you from buying it?” In our experience the barriers people name here, such as price, effort, trust or a partner who’d say no, are often more useful for the next version than the score.
Why a probability scale? The economist F. Thomas Juster compared questions about subjective purchase probability with the buying-intentions questions used in surveys of his time. In his data, purchase probabilities explained about twice as much of the cross-section variance in car purchase rates as buying intentions. Most of the purchases came from people who had said they didn’t intend to buy. The probability question split those people, and the ones who said they didn’t know, into groups that bought at clearly different rates.2 That was cars and household durables in the United States in the 1960s, so take it as a reason to try the scale and check how it works in your category. The 11-point scale he pioneered became known as the Juster scale. In a version used in later studies every point has a label and odds, from 0, “no chance, almost no chance (1 in 100)”, to 10, “certain, practically certain (99 in 100)”.3
Throughout, use balanced questions instead of agree/disagree statements. Jon Krosnick described how respondents faced with demanding questions can settle for a satisfactory answer, known as satisficing, and that can include agreeing with whatever the question asserts.4 “This product would make my mornings easier: agree or disagree?” invites that. “Would this product make your mornings easier, harder or no different?” doesn’t. And keep the survey short enough that people finish it with some attention left.
Monadic vs sequential monadic testing: which should you use?
This is one of the biggest choices in concept testing methods, and it mostly comes down to cost and contamination.
- Monadic testing: each person sees one concept. You compare concepts across groups that are randomly assigned and similar in profile. The reaction to each concept is uncontaminated by the others, but every concept needs its own sample, so testing four ideas needs roughly four times as many interviews as testing one.
- Sequential monadic testing: each person sees several concepts, one after another, and rates each. Because each person rates several, you need fewer respondents in total for the same number of ratings per concept. The first concept people see can become the yardstick for the rest, which is why the order is rotated between respondents and why it’s worth reading the scores from the first position on their own, which gives you a clean monadic read on a smaller base.
As we read the study’s summary, its comparative finding is about how the intention question is asked, one product on its own or among alternatives, more than about monadic or sequential monadic designs, although someone who has just seen three concepts is already comparing to some extent. Intentions collected one product at a time were less correlated with purchase than intentions collected comparatively.1 Much real buying is comparative, too, with people in front of a shelf or a page of results. So for a decision between ideas, we tend to use a monadic design for each concept’s score, because it isn’t coloured by the other concepts, and read it next to a closing comparative question against what people use now, which brings the survey closer to the comparative mode the study describes.
How do you read a top-2-box score?
A top-2-box score is the share of people who chose the top two answers on a purchase intent scale, usually “definitely would buy” and “probably would buy” on a five-point scale. If you use a 0 to 10 probability scale instead, read the average probability and the spread rather than squeezing it into a top-2-box share. Either way it’s a reaction to a description, so read it against something.
- Against your own history. A score means more next to the scores of earlier concepts and what those products did at launch. Linda Jamieson and Frank Bass, writing in 1989, noted that studies relating the two had often found substantial variation between stated intention and actual behaviour, and compared models that adjust stated intentions, using people’s views on things such as affordability, liking and availability, to predict trial of new products.6 In our experience the right adjustment depends on the category and market, so a rule of thumb from another category won’t transfer reliably.
- Across segments. A top-2-box score of 40% could hide 70% among people who already use something similar and 15% among everyone else, and most plans need some of the second group too.
- Next to the open answers. If the top two boxes are full and the “what would stop you” answers are still about price, even though the price was shown, it suggests people like the idea and are unsure the deal is worth it.
- When you check the score against later sales. Morwitz, Eric Johnson and David Schmittlein found that merely asking people once about their intent to buy increased their later purchase rate, and that the effect was smaller for people who already had experience with the product.7 If you follow up the same people to check what they bought, they may buy more than people you never asked, so compare them with a group you didn’t survey.
Why do concepts that test well still fail?
On 23 April 1985, Coca-Cola announced it was changing its formula, adopting one preferred in taste tests of nearly 200,000 consumers. By June the company’s consumer hotline was taking 1,500 calls a day, against 400 a day before the change, and the return of the original formula on 11 July closed a 79-day episode. The taste tests measured which drink people preferred. In the company’s own words, what they didn’t show was “the bond consumers felt with their Coca‑Cola.”8 New Coke changed an existing product rather than launching a new one, but we think the lesson about what a test leaves out carries over to concept tests.
This is where we’d bring an anthropological eye. In our view, a person filling in a survey answers as an evaluator, judging a description on its merits. The same person, a month after launch, is a parent on a Tuesday morning, with a routine, a budget, a partner’s opinion and a shelf full of things that already work well enough. A concept test removes that context to make the reaction measurable, and some of what decides a purchase can go with it.
You can put some of it back. Hoeffler tested ways of adding mental simulation (imagining using the product) and analogies to preference measurement for really new products, and found that some methods improved predictive accuracy while others made it worse,5 so how you ask people to imagine is worth testing too. For a care app for older people and everyone who looks after them, we didn’t stop at a description: older people, care home staff and social services tried the prototypes, and what they found went into the next version (the care app case). When the decision is big, a test with something at stake, such as a pre-order, a pilot or a landing page with a real price, shows behaviour that a survey can only ask about, as our guide to qual, quant and social listening explains.
This is what we do
For a decision about what to build, our concept and product testing work uses the three lenses of our H3™ method. Interviews and observation come first, to learn the words people use, what they do now and what the product would replace. The survey comes next, with concept testing survey questions built from that language: a monadic score on a probability scale and a closing comparative question. Social listening runs alongside, to hear how people talk about the problem when nobody is asking. We fuse the three, so the score comes with the reasons behind it. If you’re about to commission a test, a clear research brief can save weeks.
No survey can tell you for sure what people will buy, but better questions get you closer than a single “would you buy it?”
Sources
Where the figures and studies in this article come from.
- When do purchase intentions predict sales?Vicki G. Morwitz, Joel H. Steckel and Alok Gupta, International Journal of Forecasting, 2007
- Consumer Buying Intentions and Purchase Probability: An Experiment in Survey DesignF. Thomas Juster, Journal of the American Statistical Association, 1966
- A methodological investigation of the Juster scale: contextual requirements and mutually exclusive behavioursMathew Parackal, PhD thesis, Massey University, 2004
- Response strategies for coping with the cognitive demands of attitude measures in surveysJon A. Krosnick, Applied Cognitive Psychology, 1991
- Measuring Preferences for Really New ProductsSteve Hoeffler, Journal of Marketing Research, 2003
- Adjusting Stated Intention Measures to Predict Trial Purchase of New Products: A Comparison of Models and MethodsLinda F. Jamieson and Frank M. Bass, Journal of Marketing Research, 1989
- Does Measuring Intent Change Behavior?Vicki G. Morwitz, Eric J. Johnson and David C. Schmittlein, Journal of Consumer Research, 1993
- New Coke: The Most Memorable Marketing Blunder Ever?The Coca-Cola Company, company history page