FINAL REPORT: OUTLINE &OVERVIEW OF SURVEY ERRORS
Lu Ann Aday, Ph.D.
The University of Texas
School of Public Health
STEPS IN DESIGNING AND CONDUCTING HEALTH SURVEYS
TYPES OF SURVEY ERRORS
Systematic Error (bias): Difference(positive or negative) betweensurvey estimate and actualpopulation value
Variable (random) Error: Range orvariation (variance) in valuesof anestimate across observations (orcases)
EFFECTS OF SURVEY ERRORS ON SURVEY ESTIMATES
OVERVIEW OFSURVEY ERRORS
SURVEYPROCEDURE
SYSTEMATICERROR
VARIABLEERROR
Study Design
Poor internal validity
Poor external validity
Design specificationambiguity
Sample Design
Noncoverage bias
Weighting errors
Standard errors
Design effects
Data Collection
Unit nonresponse bias
Item nonresponse bias
Interviewer variability
Mode effects
Questionnaire Design
Under/over-reporting
Yea-saying
Order & context effects
Measurement
Low or poor validity
Low or poor reliability
Data Preparation
Imputation/Estimationerrors
Data coding, editing, ordata entry errors
Analysis Plan
Poor statistical conclusionvalidity
Low statistical precisionor power
SURVEY ERRORS:Study Design
Systematic
Poor internal validity. Thestudy design does notadequately and accuratelyaddress the study’shypotheses, particularlywith respect todemonstrating a causalrelationship between theindependent (predictor)and dependent (outcome)variables.
Poor external validity.Findings based on the studydesign cannot be widely oruniversally applied torelated populations orsubgroups.
Variable
Design specificationambiguity. Thestatement of the studyobjectives and relatedconcepts to be measuredin the survey are notclearly andunambiguously stated,particularly inrelationship to theunderlying study designand data analysis planfor the study.
SURVEY ERRORS:Sample Design
Systematic
Noncoverage bias. All unitsof the target population(e.g., households,individuals) are notincluded in the samplingframe (frame bias) or therespondent is not selectedrandomly (respondentselection bias).
Weighting errors.Respondents aredisproportionatelyrepresented in the surveysample by failing to weighteach of the cases by thedisproportionate probabilityof their falling into thesample (sampling fraction).
Variable
Standard errors. Thestandard error measuresrandom sampling variationin an estimate (e.g., meanor proportion) across allpossible random samples ofa certain size that couldtheoretically be drawn fromthe target population.
Design effects. The designeffect, computed as theratio of the variance of acomplex sample to that of asimple random sample,measures the increase inrandom sampling variationin an estimate due to thecomplex nature of a sampledesign.
SURVEY ERRORS:Data Collection
Systematic
Unit nonresponse bias.Selected units of the studysample (e.g., households,individuals) are notincluded in the final studydue to respondent refusalsor unavailability during thedata collection process.
Item nonresponse bias.Selected questions on thesurvey questionnaire arenot answered due torespondent refusals orinterviewer or respondenterrors or omissions duringthe data collection process.
Variable
Interviewer variability.Survey interviewers or datacollectors vary in how theyask or record answers tothe survey questions.
Mode effects. Theresponses to comparablequestions by respondentsvary across different datacollection methods (e.g.,personal interview,telephone interview, mailself-administeredquestionnaire, web survey,etc.). [Note: If theseeffects differ in a particulardirection across mode, theybecome systematic errors.]
SURVEY ERRORS:Questionnaire Design
Systematic
Under/over-reporting. Anestimate (e.g., mean orproportion) across samplesdiffers in a particular(negative or negative)direction from theunderlying actual (or true)population value for theestimate, i.e., is lower(underreporting) or higher(overreporting).
Yea-saying. Respondentstend to agree rather thandisagree with statementsas a whole (acquiescentresponse set) or with whatare perceived to be sociallydesirable responses (socialdesirability bias).
Variable
Order & context effects.Answers to selected surveyquestions vary dependingon whether they are askedbefore or after otherquestions and/or appear atthe beginning or the end ofthe survey questionnaire.
SURVEY ERRORS:Measurement
Systematic
Low or poor validity.Systematic departuresexist in answers to thecontent of a surveyquestion from themeaning of the conceptitself (content validity),a criterion for whatconstitutes an accurateanswer based onanother data source(criterion validity),and/or hypothesizedrelationships of theconcept being measuredwith other measures orconcepts (constructvalidity).
Variable
Low or poor reliability.Random variation existsin answers to a surveyquestion due to when itis asked (test-retestreliability), who asked it(inter-rater reliability),and/or that it is simplyone of a number ofquestions that couldhave been asked toobtain the information(internal consistencyreliability).
SURVEY ERRORS:Data Preparation
Systematic
Imputation/Estimationerrors. Procedures forassigning values to surveyquestions for whichanswers are not available(missing values due toitem nonresponse) eitherfrom data internal orexternal to the survey(imputation or estimation,respectively) introducesystematic errors (biases)in estimating or examiningrelationships betweenvariables.
Variable
Data coding, editing, ordata entry errors. Datacoding, editing, or dataentry personnel orprocedures introducerandom errors inproducing data filesbased on the surveyquestionnaires.
SURVEY ERRORS:Analysis Plan
Systematic
Poor statisticalconclusion validity. Theaccuracy of statisticalconclusions iscompromised due to theapplication of statisticalprocedures that do notmeet underlyingassumptions related tothe study design andobjectives, level ofmeasurement of studyvariables, and/or theunderlying populationdistribution.
Variable
Low statistical precisionor power. There areinsufficient cases in thestudy sample toestimate populationparameters with areasonable level ofprecision or to haveenough statistical powerto detect statistically(and substantively)significant relationshipsbetween variables ifthey do exist.