1
Arlene Ash
QMC - Third Tuesday
September 21, 2010
Analyzing Observational Data: Focuson Propensity ScoresAnalyzing Observational Data: Focuson Propensity Scores
2
The ProblemThe Problem
•Those with the intervention and those without havemarkedly different values for important measuredrisk factors &
•Outcome is related to the risk factors that areimbalanced between the groups &
•It is not clear how the risk factors and outcome arerelated
•Why may standard analyses be misleading?
3
True and Modeled RelationshipBetween Risk and OutcomeTrue and Modeled RelationshipBetween Risk and Outcome
4
Is Imbalance in Risk a Problem?Is Imbalance in Risk a Problem?
•If we correctly model the relationship betweenrisk factors and outcome, we correctly estimateeffect of the intervention
•With many risk factors, hard to know if therelationship between risk factors and outcomeis correctly modeled
•Propensity score - a way to reduce the effect ofimbalance in measured risk when models maybe inadequate
5
Propensity Score Method(Key Idea)Propensity Score Method(Key Idea)
•Draw a sub-sample that is more balanced onrisk factors
•Use standard techniques to analyze the sub-sample
6
Typical Propensity Score ApproachTypical Propensity Score Approach
•Estimate a model to predict the “probability of receivingthe intervention”
– This is “the propensity score”
•Divide the full population into quintiles of the propensityscore
•Sample equal numbers of cases and controls from eachquintile
•Re-fit the model to estimate the effect of the interventionin the sampled cases
7
Propensity Score Sampling ExamplePropensity Score Sampling Example
PS Quintile
# Cases
# Controls
# Sampled
Lowest
12
81
24
2nd
30
67
60
Middle
44
38
76
4th
53
15
30
Highest
78
8
16
Total
217
209
206
8
Propensity Score Sampling Example:Treatments for Drug AbusersPropensity Score Sampling Example:Treatments for Drug Abusers
•Patients seeking substance abuse detoxification inBoston receive either
Residential detoxification
Lasts ~ one week + encouragement for post-detox treatment, or
Acupuncture
Acute (daily) detox + 3-6 months of maintenancewith acupuncture and motivational counseling
9
DataData
•From Boston’s publicly-funded substanceabuse treatment system
•All cases discharged from residential detox oracupuncture between 1/93 and 9/94
•Client classified (only once) as residential oracupuncture based on the modality of firstdischarge
10
OutcomeOutcome
•Is client re-admitted to detox within 6months? (Y/N)
•Study question: Are acupuncture clientsmore likely to be re-admitted thanresidential detox clients
11
Client Characteristics Available AtTime Of AdmissionClient Characteristics Available AtTime Of Admission
•Gender
•Race/ethnicity
•Age
•Education
•Employment status
•Income
•Health insurance status
• Living situation
• Prior mental health
treatment
• Primary drug
• Substance abuse
treatment history
12
Residential Detox & Acupuncture Cases:% with Various CharacteristicsResidential Detox & Acupuncture Cases:% with Various Characteristics
Characteristic
Residential
(n = 6,907)
Acupuncture
(n = 1,104)
Gender: female
29
33
Race/ethnicity: black
46
46
Hispanic
12
10
White
41
43
Education: HS grad
56
59
College graduate
4
13
13
Employment: unemployed
86.8
43.2
Insurance: uninsured
65.4
52.3
Medicaid
28.2
21.2
Private insurance
3.0
15.4
Lives: with child
9.5
19.3
In shelter
30.3
2.9
Characteristic
Residential
(n = 6,907)
Acupuncture
(n = 1,104)
Characteristics of Residential Detox &Acupuncture Clients (2)Characteristics of Residential Detox &Acupuncture Clients (2)
14
Prior mental health treatment
12.3
27.8
Primary drug: alcohol
42.3
32.4
Cocaine
16.2
16.6
Crack
15.9
20.2
Heroin
24.6
19.0
Characteristic
Residential
(n = 6,907)
Acupuncture
(n = 1,104)
Characteristics of Residential Detox &Acupuncture Clients (3)Characteristics of Residential Detox &Acupuncture Clients (3)
15
Substance abuse admits in the last year
Residential detox: 0
1
2+
Short-term residential: 0
Long-term residential: 0
Outpatient: None
Acupuncture: None
56.7
20.2
23.1
76.2
80.5
80.6
95.9
81.0
12.1
7.0
94.8
93.5
54.3
90.1
Characteristic
Residential
(n = 6,907)
Acupuncture
(n = 1,104)
Characteristics of Residential Detox &Acupuncture Clients (4)Characteristics of Residential Detox &Acupuncture Clients (4)
16
Results Of Standard AnalysisResults Of Standard Analysis
Percentage of clients re-admitted to detox within 6 months
•Among 1,104 acupuncture cases, 18% re-admitted
•Among 6,907 residential detox cases, 36% re-admitted
•Raw odds ratio = 0.40
From a multivariable stepwise logistic regression model:
•Odds ratio for acupuncture:0.71 (CI = 0.53-0.95)
17
ConcernConcern
•Given large differences in risk adjustors between thegroups and possibility of model mis-specification, shouldwe be concerned about the estimated effect ofacupuncture?
•Stratum-specific modeling has been used to addresssuch concerns historically
–Strata defined by a limited number of particularlyimportant risk adjustors
•Propensity scores, a generalization
–Used when there are many important predictors
18
Propensity Score ApplicationPropensity Score Application
•Use stepwise logistic regression to build a modelto predict whether a client “is a case”(i.e.,receives acupuncture)
•Select sub-samples of cases and controls thathave similar distribution of the “propensity score”(predicted probability of being a case)
•Model (as before) on the sub-sample
19
Sampling ResultsSampling Results
•Able to match 740 cases (out of the fullsample of 1,104 cases) with 740 comparablecontrols (out of the full sample of 6,907controls)
•Much more balance in terms of risk in thissub-sample
20
Characteristic
Residential
Acupuncture
College graduate
Employed
Private Insurance
Lives with child or adult
Lives in shelter
Prior mental health Rx
7%
41%
9%
72%
5%
21%
(4%)
(13%)
(3%)
(55%)
(30%)
(12%)
7%
42%
6%
77%
4%
21%
(13%)
(57%)
(15%)
(76%)
(3%)
(28%)
Characteristics of Clients in Subsample(Full Sample)Characteristics of Clients in Subsample(Full Sample)
21
Comparing Standard and PropensityScore FindingsComparing Standard and PropensityScore Findings
From the multivariable model fit to all cases:
Odds Ratio for acupuncture: 0.71
95% Confidence Interval: 0.53-0.95
From multivariable model fit to more comparable sub-sample:
OR for acupuncture: 0.61
95% CI:0.39-0.94
22
SummarySummary
•In this case, results were similar - Why?
Original model was very good (C-statistic = 0.96)
•What was learned from the propensity scoreanalysis:
–Could find a subset of controls (about 10%) wholook very similar to cases
–Found similar results in this subset, increasingthe credibility of the findings
23
Which Belong in the PS Model?Which Belong in the PS Model?
•Confounders (Brookhart’s X1 variables)?
•Simple predictors (X2 s)?
•Simple selectors (X3 s)?
•Let’s work together to fill in the following table
Variable type
Directly affects
Belongs inwhich model
Outcome(Y)
Exposure(E)
PS
Regression
X1
Confounder
1
1
?
?
X2
Predictor
1
0
?
?
X3
Selector
0
1
?
?
24