A speech about BoostingA speech about Boosting
Presenter: Roberto ValentiPresenter: Roberto Valenti
The Paper*The Paper*
*R.Schapire. The boosting approach to Machine Learning An Overview, 2001
I want YOU…I want YOU…
…TO UNDERSTAND…TO UNDERSTAND
OverviewOverview
•Introduction
•Adaboost
–How Does it work?
–Why does it work?
–Demo
–Extensions
–Performance & Applications
•Summary & Conclusions
•Questions
Introduction to BoostingIntroduction to Boosting
Let’s start
IntroductionIntroduction
•An example of Machine Learning:Spam classifier
•Highly accurate rule: difficult to find
•Inaccurate rule: ”BUY NOW”
•Introducing Boosting:
“An effective method of producing anaccurate prediction rule frominaccurate rules”
IntroductionIntroduction
•History of boosting:
–1989: Schapire
•First provable polynomial time boosting
–1990: Freund
•Much more efficient, but practical drawbacks
–1995: Freund & Schapire
•Adaboost: Focus of this Presentation
–…
IntroductionIntroduction
•The Boosting Approach
–Lots of Weak Classifiers
–One Strong Classifier
•Boosting key points:
–Give importance to misclassified data
–Find a way to combine weak classifiers ingeneral rule.
AdaboostAdaboost
How does it work?
Adaboost – How does it work?Adaboost – How does it work?
Adaboost – How does it work?Adaboost – How does it work?
Base Learner Job:
–Find a base Hypothesis:
–Minimize the error:
•Choose t
Adaboost – How does it work?Adaboost – How does it work?
AdaboostAdaboost
Why does it work?
Adaboost – Why does it work?Adaboost – Why does it work?
•Basic property: reduce the trainingerror
•On binary Distributions:
t
•Training error bounded by:
•Is at most e-2T->drops exponentially!
•Generalization Error bounded by:
–T= number of iterations
–m=sample size
–d= Vapnik-Chervonenkis dimension2
–Pr [.]= empirical probability
–Õ = Logarithmic and constant factors
•Overfitting in T!
Adaboost – Why does it work?Adaboost – Why does it work?
Adaboost – Why does it work?Adaboost – Why does it work?
•Margins of the training examples
margin(x,y)=
•Positive only if correctly classified by H
•Confidence in prediction:
•Qualitative Explanation of Effectiveness
–Not Quantitative.
Adaboost – Other ViewAdaboost – Other View
•Adaboost as a zero-sum Game
–Game matrix M
–Row Player: Adaboost
–Column Player: Base Learner
–Row player plays rows with distribution P
–Column player plays with distribution Q
–Expected Loss: PTMQ
•Play a Repeated game Matrix
Adaboost – Other ViewAdaboost – Other View
•Von Neumann’s minmax theorem:
•If exist a classifier with
•Then exist a combination of baseclassifiers with margin > 2
•Adaboost has potential of success
•Relations with Linear Programming andOnline Learning
AdaboostAdaboost
Demo
DemoDemo
AdaboostAdaboost
Extensions
Adaboost - ExtensionsAdaboost - Extensions
•History of Boosting:
–…
–1997: Freund & Schapire
•Adaboost.M1
–First Multiclass Generalization
–Fails if weak learner achieves less than 50%
•Adaboost.M2
–Creates a set of binary problems
–For x, better l1 or l2?
–1999: Schapire & Singer
•Adaboost.MH
–For x, better l1 or one of the others?
Adaboost - ExtensionsAdaboost - Extensions
–2001: Rochery, Schapire et al.
•Incorporating Human Knowledge
•Adaboost is data-driven
•Human Knowledge can compensatelack of data
•Human expert:
–Chose rule p mapping x to p(x) Є [0,1]
–Difficult!
–Simple rules should work..
Adaboost - ExtensionsAdaboost - Extensions
•To incorporate human knowledge
•Where
RE(p||q)=p ln(p/q)+(1-p) ln((1-p)/(1-q))
AdaboostAdaboost
Performance and Applications
Adaboost - Performance & ApplicationsAdaboost - Performance & Applications
Error Rates on Text categorizationError Rates on Text categorization
Reuters newswire articles
AP newswire headlines
Adaboost - Performance & ApplicationsAdaboost - Performance & Applications
Six Class Text Classification (TREC)Six Class Text Classification (TREC)
Training Error
Test Error
Adaboost - Performance & ApplicationsAdaboost - Performance & Applications
“How may I help you”
Spoken Language ClassificationSpoken Language Classification
“Help desk”
Adaboost - Performance & ApplicationsAdaboost - Performance & Applications
class, label1/weight1,label2/weight2
OCR: Outliers
Rounds:
12
25
4
Adaboost - ApplicationsAdaboost - Applications
•Text filtering
–Schapire, Singer, Singhal. Boosting and Rocchio appliedto text filtering.1998
•Routing
–Iyer, Lewis, Schapire, Singer, Singhal. Boosting fordocument routing.2000
•“Ranking” problems
–Freund, Iyer, Schapire, Singer. An efficientboostingalgorithm for combining preferences.1998
•Image retrieval
–Tieu, Viola. Boosting image retrieval.2000
•Medical diagnosis
–Merler, Furlanello, Larcher, Sboner. Tuning costsensitiveboosting and its application to melanoma diagnosis.2001
Adaboost - ApplicationsAdaboost - Applications
•Learning problems in natural languageprocessing
–Abney, Schapire, Singer. Boosting applied to tagging andPP attachment.1999
–Collins. Discriminative reranking for natural languageparsing.2000
–Escudero, Marquez, Rigau. Boosting applied to wordsense disambiguation.2000
–Haruno, Shirai, Ooyama. Using decision trees toconstruct a practical parser.1999
–Moreno, Logan, Raj. A boosting approach for confidencescoring.2001
–Walker, Rambow, Rogati. SPoT: A trainable sentenceplanner.2001
Summary and ConclusionsSummary and Conclusions
At last…
SummarySummary
•Boosting takes a weak learner andconverts it to a strong one
•Works by asymptotically minimizingthe training error
•Effectively maximizes the margin of thecombined hypothesis
•Adaboost is related to other manytopics
•It Works!
ConclusionsConclusions
•Adaboost advantages:
–Fast, simple and easy to program
–No parameter required
•Performance Dependency:
–(Skurichina, 2001) Boosting is only usefulfor large sample size.
–Choice of weak classifier
–Incorporation of classifier weights
–Data distribution
QuestionsQuestions
?
(don’t be mean)