A few months ago, I was writing a blog post about getting the information you need from customer surveys. As I was researching one section, there was a discussion to be had around in-person versus online surveys. That sent me down a rabbit hole of reading, watching, and talking to people who know far more about surveys than I do. Now I want to share that information with you.
Online surveys have continued to gain popularity since their inception. In the past couple of years, when it was less safe for most people to meet face-to-face, their use gained even more traction. But technology giveth, and technology taketh away. As we have continued to move toward more privacy online, survey answers have become somewhat less reliable.
That said, in-person surveys are not always the best answer either. Even with their faults, online surveys have advantages. These advantages help draw in more responses, make it easier for people to respond, and can often help us get a wider breadth of data.
In order to compare and contrast online versus in-person surveys we first need to look at where each of them excel and where they fall short.
The Draw of Online Surveys
The first area where online surveys shine is in our ability to reach such a wide audience with minimal effort. It’s easy to attach a survey link to a receipt email, or to spread our survey through social media. This convenience weighs heavily in the decision process for using online surveys, but as we will discuss later, it also raises potential problems.
Online surveys are also easy to format. There are loads of tools available for creating surveys, and many of them have great formatting and design tools built in. They offer a variety of ways to format questions and answers, allowing you flexibility on both ends of the discussion.
Online surveys also tend to be inexpensive. You might pay a subscription charge for a service that lets you set up your survey and collect responses. But those costs will pale compared to what some in-person surveys will require.
It’s often thought that online surveys can help people feel welcome to give more honest answers. In-person interviews, whether face to face or over video, can lead to social pressures. By allowing a bit of anonymity, online surveys can set a space for people to feel more free in their responses.
While there are many potential advantages to online surveys, one that stands out above many others is their accessibility. In-person surveys need people available to ask questions. Respondents can take an online survey whenever it is convenient for them. This accessibility also pours over into their design and development, where more flexibility can be allowed in timelines.
The Drawbacks of Online Surveys
Now that we’ve talked about the good stuff, we have to discuss the potential areas for concern. There aren’t a lot, but they can have serious implications. The first of these is because of the very nature of online surveys — there is nobody conducting the interview.
If you’ve written your questions well, chances are good that you’ll avoid problems related to having no interviewer. But sometimes a question isn’t as clear as we think it is. As the developer of a survey, it’s common to have a confirmation bias toward believing that everyone understands what you’re talking about. That’s not always the case, and with nobody around to answer clarification questions, it’s a potential problem area.
The proverbial elephant in the room is survey fraud. If you’re only sending out surveys to registered customers, you can avoid some of this potential. But surveys that get sent out into the wild — one big reason they’re so popular — are far more susceptible to manipulation than their in-person counterparts.
Sampling limits are also another important variable to consider. A 2020 study, published in the Indian Journal of Psychological Medicine gives an example.
“Imagine that we create a questionnaire to study how Covid-19 has affected the mental health of the population. We leave copies of our questionnaire on a park bench. These sheets of paper are blown by the wind to different destinations. Most people regard the questionnaire as litter. Some pick it up, look at it, and drop it. A few take the trouble to answer the questions and return the questionnaire. We analyze the data in the returned forms and conclude, perhaps, that Covid-19 has compromised the mental health of persons in our city or country.“
The Journal’s example explains that, since we can’t be sure who got the survey, confirmation bias could slip into the conclusions. The assumption is that the surveys that were returned are a representative sample of the overall population. But this (admittedly absurd) example shows us that surveys could have landed only to those people downwind, or the completed surveys could have come only from people who were literate.
It’s not impossible to avoid this bias. We could send surveys only to registered users, or require them to enter proprietary information in order to complete the survey. But every element that removes the process from being in-person opens the door for inequity and bias.
The Drawbacks of In-Person Surveys
If we are going to compare and contrast, then it’s important that we also take an unbiased look at in-person surveys. The first roadblock that people often run into is cost. But cost isn’t a simple problem.
When assessing whether to do an online or an in-person survey, there are a few elements that go into the cost:
- How much are we paying the people who develop the survey?
- Is this survey a good value for the time that they spend?
- What is the opportunity cost of producing and conducting the survey?
- Who will conduct the survey?
- How much will it cost us to have this person (or these people) away from their other work?
- Will you need to provide compensation for survey respondents?
While some of these apply to both online and in-person surveys, the gravity of each question is heavier when you’re designing a survey to be taken in person.
Another area of concern is choosing the right person to do the interview. Make this decision early, because you’re going to want to include this person in the question-writing process. It’s often best to have someone who is not directly related to the survey be the person who administers it. This helps to remove bias, which can cause respondent answers to include bias as well.
Since in-person surveys are a manual process, recording and interpreting the data is almost always manual as well. This is a factor that you’ll need to take into consideration with your cost estimates, and you need to be aware of the potential for mistakes in the data entry.
The final area to watch is how much time the in-person survey will take. If you need fast responses, in-person surveys are at a distinct disadvantage. It can take quite some time to reach a relevant sample size, and even in-person methods take more time than online forms.
The Draw of In-Person Surveys
For all the potential drawbacks, in-person surveys are still wildly popular. The impact of a trained, professional interviewer is hard to overstate. They can answer questions, clarify, and they can still move away if privacy is a concern for certain answers.
Beyond the advantages that a professional interviewer can bring, in-person surveys have a certain niche that online surveys can never match. If you want to see how someone interacts with your product, there’s no better way than in-person. They can touch it, smell it, taste it and engage senses in a way that online surveys don’t allow.
In-person surveys also allow the interviewer to control the environment. For example, if your company has brick and mortar stores, you can simulate a shopping environment. If it’s important that the scene be relaxed, or intense, or invoke some other feeling, in-person surveys allow for that level of customization.
In-person surveys can help you reach respondents that online surveys would miss. Language and literacy barriers, as well as digital divide concerns, become less of a factor with in-person surveys. It’s also more possible to verify that the people you are speaking to are actually the audience that you want to reach.
Body language is also a factor that in-person surveys have over their online counterparts. You can see how someone reacts, instead of only relying on their answer. Those reactions can then help to guide your next set of questions, or the reactions themselves could be valuable feedback.
Decisions, Decisions
No single blog post could ever fully encompass the pros and cons for each type of survey. That said, our goal with this writing was to help you make a more informed decision. By weighing the advantages, disadvantages, and opportunity costs of each type of survey, you should be able to find which one fits your needs best.
How have you used surveys, and what methods did you employ to get the responses that you need? Are there pros or cons that we should have mentioned but didn’t? We’d love to hear from you in the comments.
