Simulation-basedinference beyond theintroductory course
Beth Chance
Department of Statistics
Cal Poly – San Luis Obispo
bchance@calpoly.edu
Background
Simulation-based inference has been advocated asa way to get students to think inferentially earlyand often in the introductory statistics course
Introduction to Statistical Investigation (Tintle et al)
Statistics: Unlocking the Power of Data (Lock et al)
Introductory Statistics with Randomization andSimulation (OpenIntro)
Statistical Thinking: A simulation approach tomodeling uncertainty (U of Minn)
Background
Students can use simulation-based inference(e.g., randomization tests, bootstrapping) to
Focus on the meaning of p-value and confidenceintervals at a conceptual level (“what couldhappen with my statistic by chance alone?”),
To explore their own questions (e.g, what aboutthe median instead of the mean)
To focus on the entire statistical process as a whole
Background
So what comes next?
Do we move to theory-based methods from thatpoint on?
Can we continue to leverage student understandingto ask even richer investigation questions?
What are some of the explorations available tostudents building on this approach?
Chi-square, ANOVA, Regression
Example – Dr. Spock’s Trial
Dr. Benjamin Spock was tried in 1968 forconspiracy to violate the Selective Services Act
His jury did not contain any women
Was his judge biased against women?
Judge 1
Judge 2
Judge 3
Judge 4
Judge 5
Judge 6
Judge 7
Total
Women on jury list
119
197
118
77
30
149
86
776
Men on jury list
235
533
287
149
81
403
511
2199
Total
354
730
405
226
111
552
597
2975
Proportions
.336
.270
.291
.341
.270
.270
.144
.336
Example – Dr. Spock’s Trial
Explore the data
How can we compare these 7 proportions?
Why don’t I just compare Judge 7 to 0.336?
Why don’t I just compare Judge 7 to Judge 4?
Why don’t I just run C(7,2) two-sample z-tests?
Judge 1
Judge 2
Judge 3
Judge 4
Judge 5
Judge 6
Judge 7
Total
Proportions
.336
.270
.291
.341
.270
.270
.144
.336
Example - Dr. Spock’s Trial
Explore the data
Can I run one test to consider the equality of all 7probabilities at once?
We ask students to develop their own statistic(formula) that measures how different these 7proportions are as a whole
Judge 1
Judge 2
Judge 3
Judge 4
Judge 5
Judge 6
Judge 7
Total
Proportions
.336
.270
.291
.341
.270
.270
.144
.336
Example – Dr. Spock’s Trial
Common initial choices of statistic
Largest – Smallest proportion (Max – Min)
Average proportion
Average difference in pairwise proportions
Statistic: Average difference in proportions Mean Absolute Difference (MAD)
Students can fairly quickly calculate by hand
Observed MAD (success = woman): 0.071
Does choiceof successmake adifference?
Example – Dr. Spock’s Trial
Consider the behavior of your statistic when thenull hypothesis is true
Do you expect it to be large or small? Close to zero?Positive or negative? What kinds of values will youconsider to be evidence against the null hypothesis?
Simulation:
Random assignment (shuffle the 2975 jurors amongthe 7 judges with same totals per judge)
Random sampling from binomial process withcommon probability of success say = 0.336
Example – Dr. Spock’s Trial
Strength of evidence: How often wouldwe get a MAD value of 0.071 by chancealone?
Why isn’t this distribution centered atzero?
Why isn’t this distribution symmetric?
What is the most important feature ofthis distribution to be learning about?
What conclusion can you draw from thisp-value?
Example – Dr. Spock’s Trial
What about the chi-square test statistic?
With a binary response:
Comparison to overall proportion rather thanpairwise differences
Larger values still evidence against null hypothesis
Can also examine individual “z-scores”
Directly see impact of sample size on size ofstatistic
Example – Dr. Spock’s Trial
What about the chi-square test statistic?
Observed statistic: 62.68
Simulation
Strength of evidence does
depend on choice of statistic
Some statistics have a nice
mathematical model
Under certain conditions
ANOVA? (Multiple means)
Can have the exact same conversation comparingmultiple means
Choice of statistic: Max – Min, MAD, F-statistic
If you can describe a formula for the statistic, youcan create its null distribution
Can help evaluate the effectiveness of a statistic(Is it monotonic with the evidence against thedata? Does it reflect sample size? Power…)
What are advantages to standardized statistics?
Example– Height vs. Foot length
Collect class data on heightand foot length
Moderate strong linearassociation with fun outliersto investigate
How accurately can I predictthe height from a footprint?
Advantages to usingfootprint over handprint?
Example – Height vs. Foot length
Simulation-based inference
Statistic: slope or correlation coefficient
Simulation: No association = Random assignment ofheights to foot lengths
Example – Height vs. Foot length
Lines meet at one point
“Bow-tie” pattern
Symmetric, centered atzero
SD approx. 0.338
Example – Height vs. Foot length
Can calculate the standardized statistic
estimate – null value
standard deviation
Theory-based regression analysis
Example – Height vs. Foot length
Simulation: No association = Randomassignment of heights to foot lengths
Theory-based inference: No linearassociation in the means of theconditional response distributions ateach x, but those conditionaldistributions are normally distributedwith a common variance 2
Example – Height vs. Foot length
Change the simulation: Sampling from finitepopulation (assuming some characteristics aboutthe population)
Example – Height vs. Foot length
Change the simulation: Sampling from finitepopulation (assuming some characteristics aboutthe population)
Example – Height vs. Foot length
Change the simulation: Sampling from finitepopulation (adjusting some characteristics aboutthe population)
Example – Height vs. Foot length
Change the simulation: Sampling from finitepopulation (adjusting some characteristics aboutthe population)
Big Ideas
Choice of simulation makes a difference
Especially the stronger the association
Are you pooling the results first
Student exploration, development of ideas
Students can explore factors impacting significanceand prediction accuracy (derive formulas)
Look at variance of predictors as a screening step(handprint vs. footprint)
In fact! Same issue with
comparing means and proportions
Random shuffling
Binomial sampling
Summary
Allow students to explore, develop intuition
Choice of statistic (e.g., German tank problem)
How simulate null model
Combine visual with calculation with conceptual
Predict and test
Technology needs to be complement not barrier
Focus on conceptual understanding of advancedtopics (e.g., algebra-based second course)
Simulate re-randomization within blocks
Questions?
bchance@calpoly.edu
Simulation-Based Inference blog:https://www.causeweb.org/sbi/