As researchers, we care deeply about the quality of our results. It can be better to conduct no study at all than to conduct a fundamentally flawed study.
The core way to ensure high-quality surveys is to ask the right people the right questions in the right way. While Positly helps you ask the right people, this article should help you take care of the other two criteria and ensure that your studies produce meaningful results.
Asking good questions
As researchers, we need to understand the degree to which our framing and ordering of questions elicit a reaction or specific response from the participant. You can inadvertently write questions that influence the participant, causing them to answer a certain way. These influences can result in systematic biases and misleading results.
Here is (almost) everything you need to know to create a reliable survey with relevant and useful survey questions.
Determining question type
Before you dive into writing your questions, you need to carefully consider the types of questions you want to use for your study. Learn about each type so you can accurately identify which ones will be valuable to your research study.
Question types
Open-ended
Open-ended, or free text, survey questions allow the participant to think of and write their own response to a question. These are usually characterized by a blank box where participants can write whatever comes to mind. These responses are harder to analyze but offer insights to help the researcher hone their focus for subsequent studies.
Close-ended
Close-ended survey questions have a predetermined number of answers the participant can choose from. These questions are easy for the researcher to analyze and categorize to produce their findings. The following question types are all close-ended questions.
Multiple choice
Multiple choice survey questions offer a range of answers, allowing the participant to select the answer or answers that apply to them. These question types are easy for the participant to complete and create clean data that’s easy to analyze. These types of questions are also great in experiments (e.g., AB testing, picking things that people find familiar and categorizing responses generated from open text).
- Single-answer multiple-choice questions are great for demographic and behavioral questions where there can only be one answer.
- Multiple-answer multiple-choice questions can be utilized in situations where more than one answer applies to the participant.
Interval scale
Interval scale survey questions require participants to rate their sentiment on a numerical scale. These question types work great when analyzing psychographics, like satisfaction on a scale of 1-10, ranging from negative to positive. Interval scale survey questions allow you to assign numerical values to arbitrary responses, like recording an opinion.
Ordinal scale
Ordinal scale survey questions show how much the participant agrees or disagrees with a statement or question, in essence, by ranking questions from “not at all likely” to “highly likely” or something similar. Ordinal scales don’t have a mathematical difference; instead, they ask participants to make a choice, gauging their sentiment.
Pro tip: start open and end closed – from qualitative to quantitative
Qualitative
In the early stages of research, hypotheses are often vaguely defined and require a discovery process. This is when open-ended questions can provide the most significant value, helping you to hone in on key aspects of your research.
Open-ended questions require more effort on the participants’ part, so don’t bombard your participants with a study filled with them. Instead, please provide them with a select group of poignant questions that will help you identify the key areas of your research.
In iterative research, the responses to these open-ended survey questions can help guide you through the course of your research. Once you analyze enough free text responses during your pilot studies, you can convert these to multiple-choice options that can be studied quantitatively in a subsequent study.
Quantitative
Fast, iterative research enables you to form better hypotheses with some well-designed qualitative questions and then test those hypotheses quantitatively. When it comes time for quantitative testing, you’ll want to adjust your approach to ask close-ended questions that can be easily evaluated and analyzed to arrive at your findings.
Adjusting your questioning strategy over time is the key to producing high-quality responses that drive your research and maximize reliability. We recommend using a combination of carefully constructed scales and multiple-choice questions along with relevant open-ended questions throughout your research process to yield the best survey results.
Make the best use of your participants
Use optional questions strategically
Don’t be afraid to make questions optional to answer. It’s likely that participants won’t know the answers to all your questions or may not feel comfortable providing answers to sensitive questions.
Remember that single-select questions use mutually exclusive collectively exhaustive (MECE) segmentations to group information into categories that can only elicit one response from your participants.
If you feel there is a chance someone won’t fit into your MECE segmentations, you’ll want to make the response optional so they don’t answer randomly and skew your results.
Simplify questions for the participant
Make sure that the type of questions you use are easy for the participants to answer.
While open-ended questions (also known as free text or free response) have many benefits (gaining specific insight, generating hypotheses, clarifying responses), they are unfortunately more burdensome on participants and require more effort and time to answer – and to analyze!
Prevent participant dropout
Asking too many open-ended questions can lead to participants dropping out, losing interest, satisficing, or directly costing you money if you are paying for time.
Close-ended questions, however, can help to generate much more useful quantitative results, provide an easier comparison, and require less effort and time from participants.
This also saves you a lot of time in analysis and can improve the statistical significance and representativeness of your results.
Writing effective questions
Just like you can’t run a marathon without training, asking great questions takes practice. Over time, asking good questions will become second nature to you. Until then, here are some rules to keep in mind.
1. Remain neutral
When asking survey questions, it’s important that you do not put an opinion or prompt your participant to answer in a certain way (e.g., we think our app is very useful to people who need medical advice; do you think our app is useful?).
These leading questions can negate your results to the detriment of your research. When you do provide information, make sure that it is impartial and only relevant to the question that you are asking.
Example: Why do you love Positly so much?
This example implies that the participant enjoys using Positly and prompts them to answer in a certain way based on the wording of the question. The question isn’t objective, and the results will not be reliable.
2. Keep questions short and simple
Shorter questions are easier for participants to understand, yielding stronger and more educated answers and better results for your study. Every survey should follow the BRUSO model (Peterson, 2009). BRUSO stands for brief, relevant, unambiguous, specific, and objective. Your question should follow all of these rules to allow participants to understand and answer quickly and accurately.
3. Avoid using jargon
If you litter your question with confusing acronyms or highly technical language, you will confuse your participant. This makes it harder to get responses that are accurate and reliable. You risk making your participant feel unintelligent and inviting inaccurate or unreliable responses because the participant does not understand what you mean.
If you must include acronyms, make sure that they are clearly defined so the participant can understand exactly what you are asking.
4. Avoid asking too much
Similar to staying away from jargon, you should avoid double-barrelled questions as well. When you ask for more than one response to a single question, you will confuse your participants, and they will not know how to answer.
Example: Were you satisfied with the price and service you received?
This can be confusing because this example really has two questions in one. Instead, ask single-faceted questions and separate each topic into its own question.
For the example above, that would mean crafting one question about price and another about service. While that will add to the length of your survey, it will make it easier for the participants to understand and answer with accuracy.
5. Be exclusive
Wherever possible, in asking single-response close-ended questions, your answers should include all possible options without any overlap so each participant fits into a single category. Each participant should have a relevant response that they can decipher immediately and respond correctly.
These are called mutually exclusive, collectively exhaustive (MECE) questions. You can only select one answer, but there will definitely be a suitable answer for you.
Example:
Which caffeinated beverages do you drink most?
– Tea
– Coffee
– Cola
– Other caffeinated beverage
– I do not drink caffeinated beverages
6. Avoid bias
We all bring personal beliefs or understandings about how certain things work to our research. It’s essential that those beliefs or biases do not creep into the questions we ask.
Always strive to remain objective and exclude your own beliefs from your research so that you are generating reliable, verifiable results. If you are asking sensitive questions, leave them to the end of your survey to keep them from influencing the answers of the participants.
Ordering your questions and answers
Now that you’ve chosen and written great survey questions, you need to put them in order. The cadence of your survey could impact the results that you receive, and survey questions placed out of context should be avoided at all costs.
Here are a few things to keep in mind when selecting the order of your survey questions.
1. Take the funnel approach
The beginning of your survey should include broad, more general questions as a warm-up for your participants.
As the survey progresses, you should continue getting more specific, and the participants should be required to elaborate more on their answers. As a rule, your questions should go from broad to narrow, and the answers should go from small to large.
2. Randomize whenever possible
Avoid order bias completely by randomizing the order of questions and answers wherever you can.
In some cases, you may need to provide context first or build on questions over the course of the survey. However, there are many cases where the order does not matter.
When the order doesn’t matter, you can reduce bias (and improve significance) by ordering them at random.
3. Implement a progress bar
The participants’ time is valuable, and you should let them know where they are in the process to help keep them engaged and motivated.
Time estimates are even better, so the participants can know exactly where they are in the process and understand how much more time they need to focus on your survey.
Final remarks
When you choose to use Positly for your research, you’re taking out the headache of recruiting and managing high-quality participants for your study.
However, just because you have high-quality participants does not mean that you will automatically receive high-quality results. You need to carefully consider the design of your survey questions and ensure your objectivity.
You don’t want to miss important areas of discussion where your participants can add value to your overall research mission. They could even point out some valuable things that you haven’t even considered!
For the sake of everybody, we don’t want to waste our time, nor the participant’s time, due to poor survey design. If you follow the principles outlined here, you will be in a good position to maximize the quality of your study results.
