Focus groups are the most abused form of research in marketing.
I would go so far as to say that more harm than good has been done with focus groups. Because they are relatively easy to set-up and conduct, people overlook their inherent limitations in order to get comfort from having done some form of research before making a risky decision. It allows people to check the box next to research. But in many cases, it would have been better to do no research at all than to have only done focus groups. Here are several of the factors that make focus groups a less than ideal research tool:
Selection Bias: Most research is built on the assumption that you are tapping into a random sample of your desired target market. Focus groups tend to have a persistent selection bias that negates that assumption. In our time-starved society, few people are willing to go out of their way and give 2-3 hours of their time to discuss something they don’t care that much about for $50. Those that do tend to either have time on their hands, are eager to share an opinion, or really want the money. In some cases, that may be fine. In most cases, it introduces a type of person whose motivations are different from the target audience you want to understand.
Sample Size: The biggest abuse of focus groups is treating them like a quantitative study. Too often, someone will conduct a series of focus groups, and form a conclusion about an idea based on the percentage of people who liked it. From a statistical perspective, this is unfounded. Five rounds of focus groups will involve about 20-25 people. Even if you assume a random sample of people, at this number the relative reliability of the results could be in the area of +/- 30%. So if 2/3rds of the people you talk to in focus groups give it a positive rating, it is equally likely that 2/3rds of your overall target actually dislikes the idea.
Rational Bias: Another well-known bias with focus groups is our desire to look smart. Said another way, we tend to make up rational reasons for why we like or don’t like things, even when we don’t have any. So when you sit people down in a room, walk them through an idea, and then ask them to comment, you are likely to get an artificial response. People don’t like to admit that they like or dislike things based on what may seem frivolous reasons to others. They won’t admit to wanting to look cool or be popular, even though we know that drives a lot of human behavior. So the answers you tend to get in focus groups downplay the emotional human elements that are central to our motivations.
Environmental Factors: There are a host of environmental factors around focus groups that color what you get out of them. For one, you are often exposing them to an idea outside of its context or in a less tangible form. For example, asking people if they think they’d notice a new package design after you’ve forced them to look at a picture of a package is not likely to get you a reaction anything close to glancing past a three-dimensional package in the middle of a supermarket aisle.
Group Dynamics: Most of us like to get along with others, so there is a hesitancy to disagree openly with another person in the group. Some people tend to be more comfortable talking than others. When you combine those factors together, you tend to get people who unintentionally dominate a group. More extroverted people comment first, and that sets a bench line for other people’s response. Good moderators can help draw out people, but they are fighting a basic human characteristic. So group reactions tend to move toward consensus rather than diversity.
Lack of a Hypothesis: Good research should have some ingoing hypothesis that you are trying to prove or disprove. For example, the hypothesis may be as simple as “current users prefer this new package design to the old one.” The format and questions in the research are then built around getting a solid read on that hypothesis. All too often in focus groups, this discipline falls by the wayside. People will just want to do some focus groups to get a general reaction to an idea. Without a hypothesis, people just look at the collection of comments and try to discern possible patterns in them after that fact. The problem with this is that focus groups always generate a number of positive and negative comments. So you can construct any number of theories to explain what may be a random pattern of responses.
So when is it a good idea to use focus groups? For the reasons cited above, they are not good at projecting how meaningful or universal a particular reaction might be. There are two situations when they are useful. One is as a starting point. If you are early in your learning process, and don’t even know what issues may be relevant, focus groups can help you get an initial lay of the land. The other situation is when you want to better understand the reasons behind a specific reaction. Suppose your team is working on a new interface design for a website. It’s a radical new design that you’re concerned might confuse novice users. A focus group would not necessarily give you an accurate sense of how many people found your interface confusing. But it could help you better understand what was causing the confusion for those users who were confused by it. In other words, if focus groups are not a reliable tool for finding the answer, they can be useful for understanding the reasons behind it.
The Ryan Report 