Ideal Questionnaire Length (and what your coffee says about you)
#WednesdayswithMDBInsight – This week Chris Bandak, EVP Market Research & Analytics, discusses questionnaire length in surveys and, of course, coffee.
Chris Bandak, EVP Market Research & Analytics
“May I have 45 minutes of your time?”
For a survey, probably not. So why should we expect the general public to oblige such a request? Which is basically what we would be doing if we designed surveys with a questionnaire that required 45 minutes to complete.
The science of market research has evolved a great deal over the years, with a mix of complicated wiz bang computations and sample sizes larger than some small countries. But let’s not be distracted from the primary goal - selecting the best approaches to maximize research effectiveness.
Even the most brilliant questionnaire can produce skewed and ambiguous results if the key fundamentals of market research are overlooked.
Questionnaire length has a significant impact on the success of any research project and needs to be considered very carefully during methodology design. I get asked about this often. Attempting to obtain answers about as many issues as possible produces a survey that lacks focus, clarity and efficiency. The net result is a questionnaire that reduces the study response rate and fatigues participants into providing careless answers.
To help illustrate this point, I produced a chart (see Figure 1) to provide an approximation of the percentage change in response rate relative to a 5-minute survey. It shows that a 15-minute survey reduces response rate by 41%, and that a 20-minute survey is likely to produce a response rate that is 59% lower than that of a 5-minute survey where all other factors are equal.Maybe you’re wondering why that matters. If the response rate goes down, does it really impact my project? Any decline in response rate is inversely proportional to the accuracy and ‘representativeness’ of the population we are sampling.
Take the following example, representing a classroom of 20 students and their grade distribution. The average grade is 74.5% among those 20 students.
Now let’s take a random sample of 10 of those students.
Now let’s assume that the market research study requires us to study 5 students and their class average. We have a choice of two methodologies to conduct the sampling:
- Interviewing 5 of the students from the random pull of 10, hence a 50% response rate.
- Interviewing 5 of the students from the total classroom of 21 students, hence a 25% response rate.
We can quickly recognize that the probability of getting a representative group of 5 students from the full classroom is far less likely than from the random pull. In other words, the decline in response rate significantly reduces the probability of a representative group of students. One might argue that we simply bump the sample size from 5 to 10 for the second methodology thus increasing the response rate to 50%. That would work for this tiny example however when dealing with populations in the millions and sample sizes in the thousands that solution becomes unrealistic.
This theory can also be applied to questionnaires that inhibit participation. When considering individuals in the population who might be hard to reach, have very limited time and simply don’t like surveys, asking for 20 minutes of their time could significantly reduce their likelihood of compliance. This raises an important issue, because these individuals could have very distinct attitudinal differences from others in your study and excluding them could result in skewed data.
This is not to say that all lengthy surveys will produce inaccurate results, but more to emphasize the need to analyze all parameters of a research project to ensure a balanced, efficient, and intelligent design aimed at providing the most accurate data available within a realistic budget.
Chris Bandak is Executive Vice-President Market Research & Analytics at MDB Insight. To learn more about Chris and the rest of our team, go to https://mdbinsight.com/team-bios/
Our Wednesdays with MDB Insight posts (#WednesdayswithMDBInsight) feature the thinkers and doers on our team sharing ideas and talking about what’s important to us as professionals. We have very diverse backgrounds and a range of interests to share with you. We hope you’re enjoying these posts and that you will join the conversation with us and let us know what’s on your mind mid-week.
Figure 1 source: Chris Bandak
Figure 2 source: Chris Bandak
Figure 3 source: Chris Bandak