MBA Evidence-Based Innovation

When to Choose Open or Close-Ended Questions

Posted on Jun 7, 2016 10:00:00 AM

Open_Closed.pngDesigning an engaging questionnaire is one of the first steps in conducting a research study. To do so, researchers ask themselves what type of questions are the best to ask. The answer is that it depends – on what researchers are trying to learn from the survey, who they are recruiting to participate, and how much the respondents and researchers themselves know about the survey topic. The sections that follow explain open- and close-ended questions, offer some of their benefits and drawbacks, and provide recommendations on how to incorporate these question formats into business school stakeholder surveys.

Close-Ended Questions

The close-ended format gives survey respondents a finite number of ways in which to answer a question by offering a list of possible responses and establishing how many a survey respondent can choose. Single-select questions, those that require respondents to choose just one answer, are commonly used when researchers are asking about demographics, monetary values, or even categorizing finite periods of time – all of which require little to no “variety” to ensure an accurate response. 

The following question in our MBA Student Entry Survey provides a good example of the single-select question format:

  When did you first begin researching business schools? [RADIO BUTTONS]

  1. Less than 3 months prior to applying
  2. 3-6 months prior to applying
  3. 7-12 months prior to applying
  4. 13-24 months prior to applying
  5. More than 2 years prior to applying

Conversely, multi-select close-ended questions allow a respondent to choose more than one response to a question, but still from among a finite set of options. Our MBA Student Entry Survey also utilizes questions of this type, as shown below:

  In addition to this program, to what other programs did you seriously consider applying? Please    select all that apply. [CHECKBOXES]

  1. Full-time MBA (One-Year, residential)
  2. Full-time MBA (Two-Year, residential)
  3. Part-time MBA (Cohort/lockstep)
  4. Part-time MBA (Self-paced)
  5. Online/Distance MBA (Fully online)
  6. Online/Distance MBA (Hybrid/Blended)
  7. Executive MBA
  8. Specialized Masters program
  9. Other degree program
  10. Executive Education/Certificate non-degree program
  11. Did not consider other options

Close-ended questions are primarily beneficial in surveys because they are the most likely to ensure that data collected is easily compared between respondents, statistically significant, and representative of the larger population that a researcher is trying to measure. Close-ended responses allow for faster tabulation at the end of survey fielding, so they are also conducive to working with larger respondent sample sizes and a quicker turnaround on results. This question format may also be preferred among respondents who are concerned about the amount of time and effort it takes to participate in a survey because it allows for faster responses.

Despite their inherent efficiency, however, close-ended questions may confine the scope of research. Respondents are limited to choosing from among a pre-determined set of answers which may or may not reflect their true sentiment. When “off-the-cuff” responses are outside the provided options, some respondents may feel boxed in and forced to select what is for them the wrong answer. 

The format of a close-ended question also presents a challenge for researchers because it may assume that they know more about a survey topic than they actually do. Choosing the response options to include in a closed list requires that they know enough about a topic to choose the right possibilities. However, if their knowledge is limited, researchers may unintentionally omit necessary answer choices. In this case, open-ended questions are a viable alternative worth exploring. 

Open-Ended Questions

Unlike close-ended questions, open-ended questions do not limit a respondent to choosing from a predefined answer list. As the name implies, these questions are open for a respondent to answer however he/she chooses – within the confines of a text box. Percept Research typically uses the open-ended question format in our MBA Student Exit Survey when asking students to elaborate on the strengths and weaknesses of their program such as:

  What are the strongest aspects/features of this program? If no comments, please skip the question. [NOT MANDATORY] [PARAGRAPH TEXT BOX]

   What are the weakest aspects/features of this program? If no comments, please skip the question. [NOT MANDATORY] [PARAGRAPH TEXT BOX]

Open-ended questions expand the realm of possibilities in survey research. Since respondents are not limited to a predefined list of choices, how they answer a question can lead researchers to consider new ideas they may otherwise not consider. This also prevents researchers from having to know too much about the question topic prior to initiating a research project, allowing them to use a survey truly as a learning tool.

However, in order for researchers to optimize insights derived from an open-ended question, respondents have to be willing to put a lot into crafting their responses. Unlike a close-ended question that respondents can answer relatively quickly, open-ended questions require considerably more time and thought. They may be daunting to a respondent that does not want to invest a lot of time in completing a survey, increasing the frequency of gibberish responses and even non-response as a result. In this sense, researchers risk gleaning less insight from an open-ended question than one that is close-ended.

Even in the event that respondents do provide well-considered answers to open-ended questions, researchers must also invest considerable more time and effort in ensuring they can discern meaningful data. The work does not stop after simply reading survey respondents’ prose. Responses must then be coded into a finite set of categories – which researchers, often subjectively, must also create – so they can be quantified and analyzed. Cognizant of the amount of work involved in this task, Percept Research will code survey respondents’ open-ended (often referred to as “verbatim”) answers upon client request. Programs registering for one of our MBA Lifecycle studies can make this request using the Special Request Box during registration. 

purpose_of_open_ended_questions.jpg

Choosing Between Open- and Close-Ended Question Formats

In choosing a survey format, researchers must ask themselves a number of questions:

  • How long do we want our survey to be?
  • How much do we already know about our survey topic?
  • How much time do we think our respondents are willing to invest to participate?
  • How much time are we willing to invest in data coding and analysis?

The simple answer would be that organizations wanting to conduct a short and quick survey – perhaps on a topic about which they already have considerable knowledge – should focus on using close-ended questions. On the other hand, institutions that need to learn more about a topic and are willing to invest more of their own time as well as ask for more time from respondents are better suited to open-ended questions.

However, issues in survey research can rarely be addressed with simple answers. Percept Research suggests that well-crafted surveys contain a mix of question types. Respondents who can answer several questions quickly using a pre-defined response list and have the option of commenting on the most important topics within a survey through open-ended feedback are likely to be the most engaged in the survey process. As a result, institutions using the mixed approach should walk away with the most valuable data possible.

Another consideration is how question formats can be split across multiple studies to complement and inform each other. Organizations can, for example, conduct a short survey comprised mostly of open-ended questions among a small group of test subjects as a way to inform them about the types of issues and topic areas that should be addressed among a larger population in close-ended format. In a similar sense, a small focus group discussion can be used as a precursor to fielding a larger online survey. Doing so allows researchers a way to bridge what will mostly be qualitative feedback with more statistically significant quantitative data. Be sure to check out this related blog article: When Should Business Schools Engage Focus Groups or In-Depth Interviews?  

-------------------------------------------------------------------------------------------------------

Please leave a comment about your how your school utilizes internal or external surveys to improve your graduate management program. We would love to hear about your experience and your tips!

By-line:

Gretchen Grabowski, author for this article, is a Senior Project Manager at Percept Research. Gretchen welcomes your questions and comments. 

Empower Your Program

 

Topics: Qualitative Research, Survey Design, Open-Ended Questions