Developing a Questionnaire
Chapter 4
Types of Questions
•Open-ended
–high validity, low manipulative quality
•Closed-ended
–low validity, high manipulative quality
Open-ended
•An open-ended question is one in whichyou do not provide any standard answers tochoose from.
1.How old are you? ______ years.
2.What do you like best about your job?
Closed-ended
•A closed-ended question is one in whichyou provide the response categories, andthe respondent just chooses one:
What do you like best about your job?(a) The people(b) The diversity of skills you need to do it(c) The pay and/or benefits(d) Other:______________________________
Dichotomous Questions
•Dichotomous Question: a question that hastwo possible responses
–Could be
•Yes/No
•True/False
•Agree/Disagree
Questions based on Level of Measurement
•Use a nominal question to measure a variable
–Assign a number next to each response thathas no meaning; simply a placeholder.
•Use an ordinal question to measure a variable
–Rank order preferences
–More than 5 – 10 items is difficult
–Does not measure intensity
Interval Level
•Attempt to measure on an interval level
–Likert response scale: ask an opinion questionon a 1-to-5, 1-to-7, etc. bipolar scale
•Bipolar: has a neutral point and scale ends are atopposite positions of the opinion
–Semantic differential: an object is assessed bythe respondent on a set of bipolar adjective pairs
–Guttman scale: respondent checks each itemwith which they agree; constructed ascumulative, so if you agree to one, you probablyagree to all of the ones above it in the list
Filter/Contingency Questions
•To determine if a respondent is ‘qualified’ toanswer questions, might need a filter orcontingency question (also known as knowledge)
–Limit # of jumps
–If only two levels, use graphic to jump
–If you can't fit the response to a filter on asingle page, it's probably best to be send themto a page, rather than a question #
How many steps in the response scale?
•Statistical reliability of the data increasessharply with the number of scale steps upto about 7 steps
–After 7, it increases slowly, leveling offaround 11
–After 20, it decreases sharply
Should there be a middle category?
•Does it make sense to offer it?
•Should not be used as the “don’t know orno opinion” option.
–The middle option is usually placed betweenthe positive and negative responses.
–Sometimes it’s last in an interview.
Direct Magnitude Scaling
•Method of obtaining ratio-scaled data
–Idea is to give respondents an anchor point,and then ask them to answer questions relativeto that
•Example:
–Suppose you are interested in the severity ofcrimes.
•Begin by assigning a number to one crimeand then have respondents assign numbersto the others based upon a ratio.
Filtering "Don't Know"
•Standard format
–No "don't know" option is presented to the respondent,but is recorded if the respondent volunteers it.
•Quasi filter
–A "don't know" option is included among the possibleresponses.
•Full filter
–First the respondent is asked if they have an opinion.If yes, the question is asked.
Question Placement
•It's a good idea to put difficult, embarrassing orthreatening questions towards the end
–More likely to answer.
–If they get mad and quit, at least you've gotten most ofyour questions asked!
•Put related questions together to avoid giving theimpression of lack of meticulousness
•Watch out for questions that influence the answersto other questions.
Wording of Questions
•Direction of Statements
–Response bias
–Socially desirable
•Always and never
–Avoid this
–Better to phrase as ‘most’, ‘infrequently’
•Language
–Reflect educational level and reading ability
–Need for various languages
Frequency and Quantity
•Consider both frequency and quantity
–Consider number of times
–Consider duration of times
Mutually Exclusive and Exhaustive
•Mutually exclusive: not possible to selectmore than one category/value
•Exhaustive: providing all possiblecategories/values
Forced Choice
•Choose between 2 choices
–Might not be relevant
–Other choices exist (or at least possible)
–Lesser of two evils
Recalling Behavior
•Can be difficult to remember
•Ask questions that can be answered
•Choose time frames that are reasonable
•Pilot test for time frame issues
Response Bias
•Exaggerating the truth
•Socially desirable answers
•Consider using ‘trap’ questions
–Possibly fictional choice
Sensitive Items
•More comfortable answering in categories
–Minimize missing data
–Might loose statistical power
Evaluating Questions
•Pre-testing
•Cognitive interviewing
•Behavior coding
•Peer review
•Peer review has shown to be the bestmethod but it’s the least used.
Validity and Reliability Questions
•Evaluative strategies:
–Analysis of data to evaluate the strength of predictablerelationships among answers and with othercharacteristics of respondents.
–Comparisons of data from alternatively worded questionsasked of comparable samples.
–Comparison of answers against records.
–Measuring the consistency of answers of the samerespondents at two points in time.
Coding the Questionnaire
•Create a codebook: reference guide for thedata set
•Code: assigning a value to a responsecategory
–Often numeric code
–Pre-coding makes it easier
–Content analysis on open-ended items
–Yes/No often coded as present or not (0 or 1)
Missing Responses
•Why blank?
–Missed them
–Refusal to answer
–Didn’t feel it applied
–Didn’t know the answer
•To code or not
–Analyze the difference
–If know why, might consider
Piloting the Questionnaire
•Test it on yourself
–Possibly other experts
•Test on people similar to sample
–Don’t reuse (some exceptions)
•Discuss the survey with individuals
–During completion or After
Finding Respondents
•Best Methods of Selection
•Even with a good survey, poorly chosensample leads to poor results