1
Arlene Ash
QMC - Third Tuesday
September 21, 2010
(as amended, Sept 23)
Analyzing Observational Data: Focuson Propensity ScoresAnalyzing Observational Data: Focuson Propensity Scores
2
The ProblemThe Problem
•Those with the intervention and those without havemarkedly different values for important measuredrisk factors &
•Outcome is related to the risk factors that areimbalanced between the groups &
•It is not clear how the risk factors and outcome arerelated
•Why may standard analyses be misleading?
3
True and Modeled RelationshipBetween Risk and OutcomeTrue and Modeled RelationshipBetween Risk and Outcome
4
Is Imbalance in Risk a Problem?Is Imbalance in Risk a Problem?
•If we correctly model the relationship betweenrisk factors and outcome, we correctly estimateeffect of the intervention
•With many risk factors, hard to know if therelationship between risk factors and outcomeis correctly modeled
•Propensity score - a way to reduce the effect ofimbalance in measured risk when models maybe inadequate
5
Propensity Score Method (Key Idea)Propensity Score Method (Key Idea)
•The propensity score (PS) for an observation isthe probability that the observation is “exposed”or “got the intervention”
•Use the PS model in pre-processing the data
–To draw a sub-sample where the exposed and non-exposed groups are fairly balanced on risk factors.Then
–Use standard techniques to analyze the sub-sample
6
Simple Propensity Score ApproachSimple Propensity Score Approach
•Estimate a model to predict the “probability ofintervention/exposure”
–This is “the propensity score”
•Divide the population into PS quintiles
•Create a subsample by taking equal numbers ofexposed and unexposed observations from each quintile
•Use a subsequent regression model to estimate theeffect of the intervention in the subsample
7
Propensity Score Sampling ExamplePropensity Score Sampling Example
PS Quintile
# Cases
# Controls
# Sampled
Lowest
12
81
24
2nd
30
67
60
Middle
44
38
76
4th
53
15
30
Highest
78
8
16
Total
217
209
206
8
Propensity Score Sampling Example:Treatments for Drug AbusersPropensity Score Sampling Example:Treatments for Drug Abusers
•Patients seeking substance abuse detoxification inBoston receive either
Residential detoxification
Lasts ~ one week + encouragement for post-detox treatment, or
Acupuncture
Acute (daily) detox + 3-6 months of maintenancewith acupuncture and motivational counseling
9
DataData
•From Boston’s publicly-funded substanceabuse treatment system
•All cases discharged from residential detox oracupuncture between 1/93 and 9/94
•Client classified (only once) as residential oracupuncture based on the modality of firstdischarge
10
OutcomeOutcome
•Is client re-admitted to detox within 6months? (Y/N)
•Study question: Are acupuncture clientsmore likely to be re-admitted thanresidential detox clients?
–Exposure = assigned to accupuncture
11
Client Characteristics Available AtTime Of AdmissionClient Characteristics Available AtTime Of Admission
•Gender
•Race/ethnicity
•Age
•Education
•Employment status
•Income
•Health insurance status
• Living situation
• Prior mental health
treatment
• Primary drug
• Substance abuse
treatment history
12
Residential Detox & Acupuncture Cases:% with Various CharacteristicsResidential Detox & Acupuncture Cases:% with Various Characteristics
Characteristic
Residential
(n = 6,907)
Acupuncture
(n = 1,104)
Gender: female
29
33
Race/ethnicity: black
46
46
Hispanic
12
10
White
41
43
Education: HS grad
56
59
College graduate
4
13
13
Employment: unemployed
86.8
43.2
Insurance: uninsured
65.4
52.3
Medicaid
28.2
21.2
Private insurance
3.0
15.4
Lives: with child
9.5
19.3
In shelter
30.3
2.9
Characteristic
Residential
(n = 6,907)
Acupuncture
(n = 1,104)
Characteristics of Residential Detox &Acupuncture Clients (2)Characteristics of Residential Detox &Acupuncture Clients (2)
14
Prior mental health treatment
12.3
27.8
Primary drug: alcohol
42.3
32.4
Cocaine
16.2
16.6
Crack
15.9
20.2
Heroin
24.6
19.0
Characteristic
Residential
(n = 6,907)
Acupuncture
(n = 1,104)
Characteristics of Residential Detox &Acupuncture Clients (3)Characteristics of Residential Detox &Acupuncture Clients (3)
15
Substance abuse admits in the last year
Residential detox: 0
1
2+
Short-term residential: 0
Long-term residential: 0
Outpatient: None
Acupuncture: None
56.7
20.2
23.1
76.2
80.5
80.6
95.9
81.0
12.1
7.0
94.8
93.5
54.3
90.1
Characteristic
Residential
(n = 6,907)
Acupuncture
(n = 1,104)
Characteristics of Residential Detox &Acupuncture Clients (4)Characteristics of Residential Detox &Acupuncture Clients (4)
16
Results Of Standard AnalysisResults Of Standard Analysis
Percentage of clients re-admitted to detox within 6 months
•Among 1,104 acupuncture cases, 18% re-admitted
•Among 6,907 residential detox cases, 36% re-admitted
•Raw odds ratio = 0.40
From a multivariable stepwise logistic regression model:
•Odds ratio for acupuncture:0.71 (CI = 0.53-0.95)
17
What’s the Worry? How Do WeAddress It?What’s the Worry? How Do WeAddress It?
•Given how different the two groups are, can we trust amodel to correctly estimate the effect of acupuncture?
•PS methods generalize (long-standing) matching-within-strata methods that work well with 1 or 2 predictors
•PS can address imbalances in many importantpredictors simultaneously
•Both traditional and PS matching allow for
–A pooled estimate (across all strata) or
–When N is large enough, stratum-specific estimates
18
Propensity Score ApplicationPropensity Score Application
•Use stepwise logistic regression to build a modelto predict whether a client “is exposed”(i.e.,receives acupuncture)
•Select sub-samples of exposed and non-exposedwith similar distributions of the “propensity score”(predicted probability of being exposed)
•Model (as before) on the sub-sample
19
Sampling ResultsSampling Results
•Able to match
740 who received acupuncture (out of 1,104)with
740 people who did not (out of 6,907)
•The risk factors in this subsample of 1480 aremuch more balanced between the two groups
20
Characteristic
Residential
Acupuncture
College graduate
Employed
Private Insurance
Lives with child or adult
Lives in shelter
Prior mental health Rx
7%
41%
9%
72%
5%
21%
(4%)
(13%)
(3%)
(55%)
(30%)
(12%)
7%
42%
6%
77%
4%
21%
(13%)
(57%)
(15%)
(76%)
(3%)
(28%)
Characteristics of Clients in Subsample(vs. Full Sample)Characteristics of Clients in Subsample(vs. Full Sample)
21
Comparing Standard and PropensityScore FindingsComparing Standard and PropensityScore Findings
From the multivariable model fit to all cases:
Odds Ratio for acupuncture: 0.71
95% Confidence Interval: 0.53-0.95
From multivariable model fit to more comparable sub-sample:
OR for acupuncture: 0.61
95% CI:0.39-0.94
22
SummarySummary
•In this case, results were similar - Why?
Original model was very good (C-statistic = 0.96)
•What we learned from the PS analysis:
–Could find a subset of (about 10% of) patientswho got residential detox who look very similarto those who got acupuncture
–Skeptics were more receptive to findings from thePS analysis
23
Which X’s Belong in the PS Model?Which X’s Belong in the PS Model?
The goal is to estimate the effect of exposure Eon outcome Y
•Confounders (Brookhart’s X1 variables)?
–Directly affect both E and Y
•Simple predictors (X2 s)?
–Affect Y but not E
•Simple selectors (X3 s)?
–Affect E but not Y
24
ExampleExample
The goal is to estimate the effect of
E = CABG surgery on
Y = 30-day mortality following admission for aheart attack
–Confounder (e.g., disease severity)
–Simple predictors (e.g., home support)
–Simple selectors, aka “instrumental variables”(e.g., random assignment)
Variable type
Directly affects
Belongs inwhich model
Outcome(Y)
Exposure(E)
PS
SubsequentRegression
X1
Confounder
1
1
Yes
Yes
X2
Predictor
1
0
?
Yes
X3
Selector
0
1
No
?
25
? = inclusion should neither harm nor help
DiscussionDiscussion
•The “pre-processing” that occurs when sub-sampling to create “PS-balanced”comparison groups protects against bias fromconfounding variables
•Putting selector variables in the PS model willhurt accuracy (by reducing the numbers ofgood matches) without making the groupsmore comparable
•Subsequent regression improves accuracy
26