URBDP 591 A Lecture 16:Research Validity and Replication
Objectives
•Guidelines for Writing Final Paper
•Statistical Conclusion Validity
•Montecarlo Simulation/Randomization
•Evaluating Empirical Research
Guidelines for Writing Final Paper
Structure your paper around these key elements:
Statement of the Research Problem:
Research Question: What is the central research question that you are investigating?
Include in your introduction a brief statement describing the importance of the topic.
Plan of the paper:
Literature review:
Past Research:
Strengths and weaknesses of prior research you might discuss one or more of the following:
How existing research has overlooked or given inadequate attention to your topic.
How does your research improve upon the existing literature?
Conceptual Model and Research Hypothesis
Develop a conceptual model to describe your question and hypothesis.
Define important concepts and describe how you plan to operationalize important variables.
What do you expect to find? What are your expectations?
Research Methods:
The actual design or your proposed research. What is the overall research
method proposed:
a. Sampling: What is the target population? What sampling procedures will you use?
b. Research Design: What general type of research design is most appropriate for
your study?
c. Measurement: List all relevant variables - how are they related to the
concepts used in the conceptual/theoretical framework? Describe and
evaluate information on measurement reliability and validity.
d. Statistical Analysis: What specific statistical analysis techniques will be
used in your research?
Expected Results
Explain how your chosen strategy will answer your questions. Why the
methodology selected is most appropriate.
Limitation
Are any relevant shortcomings/limitations ? Discuss limitations of your
approach.
Reference
Complete list of references. Remember to add Page number for quotations.
Format for quotations
Ben-Akiva, M. and Bowman, J. L. (1998) Integration of an Activity-based Model System
and a Residential Location Model. Urban Studies 35, 1131-1153.
WRITING TIPS
Use simple and direct writing.
Be concise
Avoid jargon
Use active form
Spell check your paper
For additional information on writing see: The Elements
of Style, Fourth Edition by William Strunk Jr., and E.B. White.
Available also on-line.
ResearchQuestion
Hypothesis Formation
Based on Lit. Research.,propose some “newknowledge”
Research Design
Determine how toobtain the data totest the RH:
Data Collection
Carrying out theresearch design andgetting the data.
Data Analysis
Data collation andstatistical analysis
Hypothesis Testing
Based on designproperties and statisticalresults
Draw Conclusions
Decide how your “newknowledge” changes“what is known” abouta certain phenomenon
the “Research Loop”
Theory
Development
Applying the Research Loop
The “research loop” is applied over and over, in three ways…
•Testing a research hypothesis
–The first test of a research hypothesis -- using the “best”design you can
•Replication
–being sure your conclusions about a particular hypothesisare correct by repeating exactly the same research design
–the main purpose of replication is to acquire confidence inour methods, data and resulting conclusions
• Convergence
–testing variations of the research hypothesis: usingvariations of the research design (varying population,setting, task, measures and sometimes the data analyses
–the main purpose of convergence is to test the limits of the“generalizability” of our results
•what design/analysis changes lead to different results?
Conclusion Validity
Statistical Conclusion Validity
The extent to which an effect has occurredabove chance levels and how well theinvestigation can detect a difference (orrelationship) that actually exists.
Elements of Conclusion Validity:
Elements of Conclusion Validity:
•Sufficient Power.
•Reasonable evidence to find thatthat the cause and effect covary.
•Significant covariation.
Threats to Conclusion Validity
•Low statistical power
•Violated assumptions of statistical tests
•Fishing and the Error Rate problem
•Low reliability of measures
•Poor reliability of treatmentimplementation
Hypothesis testing
•Step 1: Set up hypothesis
–you should determine whether it is 1-tailed or 2-tailed test
•Step 2: Compute test statistics
•Step 3: Determine p-value of the test statistic
–for a pre-determined alpha, you can find the correspondingcritical limit
•Step 4: Draw conclusion
–reject H0 if p-value < alpha (ie greater than the critical limit)
–accept H0 if p-value > alpha (ie less than the critical limit)
A statistical hypothesis is a statementabout the parameters of a probabilitydistribution.
Null hypothesis Ho: 1 = 2
Alternative hypothesis Ha: 1 2
1
2
is the mean of
a distribution
The Four Components to aStatistical Conclusion
the number of units (e.g., people)accessible to study
the effect of the treatment orindependent variable relative tothe noise
the odds the observed result is dueto chance
the odds you’ll observe a treatmenteffect when it occurs
sample sizesample size
effect sizeeffect size
alpha levelalpha level
powerpower
Statistical significance vs practicalimportance
• Significance and importance
–A statistically significant result is one which we can be confident isreal and reproducible - not just the result of random variation.
–Whether the observed discrepancy (the effect size) is large enoughto have any practical importance is quite a different matter.
•If the sample size is too small...
–The test may not have sufficient power to establish whether adifference is significant, even though the difference may be largeenough to have substantial practical consequences if it is “real”.
•If the sample size is unnecessarily large...
–Differences may be established as statistically significant, and yetbe too small to have any practical consequences.
•The optimum sample size…
–is just large enough to detect differences of a size which theresearcher believes to be of practical importance. This firstlyinvolves a professional assessment of how large a difference isimportant, followed by a power analysis to determine the requiredsample size.
Randomization Testing: Method of testing hypotheses and
sometimes also determining confidence intervals for parameters.
Randomization tests determine the significance level of a test
statistic obtained from a set of data by comparing the statistic with a
randomization distribution.
What is a randomization distribution? It is a distribution of test
statistic values that is obtained by randomly reordering the
observed data values (without replacement) until all possible
permutations are obtained.
Most hypotheses of interest are alternatives to a null hypothesis
of randomness. A randomization test tells us how likely it is that
a certain pattern in the data arose by chance.
Randomization Tests
Original dataOriginal data
Randomly permuted dataRandomly permuted data
95%95%
5%5%
ThresholdThreshold
Test statistic under NullHypothesisTest statistic under NullHypothesis
ReplicateReplicate
Distribution of teststatistic Distribution of teststatistic
Hypothesis Testing Hypothesis Testing
Significance thresholds based on Permutation testSignificance thresholds based on Permutation test
The basic approach in a randomization test is to calculate a test
statistic from the observed data, and then randomly reshuffle
the data a large number of times, recalculating the test statistic
for each iteration.
Randomization Test
These statistics are used to generate a distribution of values, and the
observed value is compared to the distribution to see whether the observed
case is an event that was unlikely to have occurred through chance; that is,
if it is a tail value from the distribution.
Randomization Test
Randomization techniques allow the determination of the significance of an
observed test statistic by comparing it to the distribution of values obtained by
randomly reordering the data.
We ask: Is the observed value an unusually large or small value compared
to what might occur by chance?
This is typically done in one of two ways:
1) In some cases, random data values (e.g., spatial positions or other
measurements on individual subjects) are generated as random values.
2) In comparisons between groups, only the group memberships are randomized
while the same set of measurements are maintained.
These tests sometimes are referred to as permutation procedures (the
randomization is done by reordering the positions of elements in a array).
Randomization Techniques
How to do a randomization test:
1.Identify hypotheses (null and alternate).
2.Choose a test statistic.
3.Compute the test statistic for the original data.
4.Resample. Recompute the statistic for each of the ‘new
samples.’ When you stop resampling, you will have a
randomization distribution.
5. Accept or reject the null hypothesis by comparing the original
test statistic to the randomization distribution.
Randomization Test
Evaluating Empirical Research: A Synthesis
- Identify Key Variables and Design
Variables and their measurement scales
Research Hypotheses, Approaches, and Design
Measurement Reliability and Validity
- Evaluate Research Validity
Measurement Reliability and Statistics
Internal Validity
Equivalence of Groups
Control of External Variables
Measurement Validity
External Validity
Population Validity
Ecological Validity
- Issues of Interpretation