AdaBoost

AdaBoost refers to a particular method of training a boosted classifier. A boosted classifier is a classifier of the form

$$F\_T(x)\; =\; \backslash sum\_\{t=1\}^T\; f\_t(x)$$

where each $f\_t$ is a weak learner that takes an object $x$ as input and returns a value indicating the class of the object. For example, in the two-class problem, the sign of the weak learner's output identifies the predicted object class and the absolute value gives the confidence in that classification.

Each weak learner produces an output hypothesis $h$ which fixes a prediction $h(x\_i)$ for each sample in the training set. At each iteration $t$, a weak learner is selected and assigned a coefficient $\backslash alpha\_t$ such that the total training error $E\_t$ of the resulting $t$-stage boosted classifier is minimized.

$$E\_t\; =\; \backslash sum\_i\; E[F\_\{t-1\}(x\_i)\; +\; \backslash alpha\_t\; h(x\_i)]$$

Here $F\_\{t-1\}(x)$ is the boosted classifier that has been built up to the previous stage of training and $f\_t(x)\; =\; \backslash alpha\_t\; h(x)$ is the weak learner that is being considered for addition to the final classifier.

At each iteration of the training process, a weight $w\_\{i,t\}$ is assigned to each sample in the training set equal to the current error $E(F\_\{t-1\}(x\_i))$ on that sample. These weights can be used in the training of the weak learner. For instance, decision trees can be grown which favor the splitting of sets of samples with large weights.

This derivation follows Rojas (2009):{{cite web |last1=Rojas |first1=Raúl |author1-link=Raúl Rojas |title=AdaBoost and the super bowl of classifiers a tutorial introduction to adaptive boosting. |url=http://www.inf.fu-berlin.de/inst/ag-ki/adaboost4.pdf |publisher=Freie University, Berlin |format=Tech. Rep. |date=2009}}

Suppose we have a data set $\backslash \{(x\_1,\; y\_1),\; \backslash ldots,\; (x\_N,\; y\_N)\backslash \}$ where each item $x\_i$ has an associated class $y\_i\; \backslash in\; \backslash \{-1,\; 1\backslash \}$, and a set of weak classifiers $\backslash \{k\_1,\; \backslash ldots,\; k\_L\backslash \}$ each of which outputs a classification $k\_j(x\_i)\; \backslash in\; \backslash \{-1,\; 1\backslash \}$ for each item. After the $(m-1)$-th iteration our boosted classifier is a linear combination of the weak classifiers of the form:

$$C\_\{(m-1)\}(x\_i)\; =\; \backslash alpha\_1k\_1(x\_i)\; +\; \backslash cdots\; +\; \backslash alpha\_\{m-1\}k\_\{m-1\}(x\_i),$$

where the class will be the sign of $C\_\{(m-1)\}(x\_i)$. At the $m$-th iteration we want to extend this to a better boosted classifier by adding another weak classifier $k\_m$, with another weight $\backslash alpha\_m$:

$$C\_\{m\}(x\_i)\; =\; C\_\{(m-1)\}(x\_i)\; +\; \backslash alpha\_m\; k\_m(x\_i)$$

So it remains to determine which weak classifier is the best choice for $k\_m$, and what its weight $\backslash alpha\_m$ should be. We define the total error $E$ of $C\_m$ as the sum of its exponential loss on each data point, given as follows:

$$E\; =\; \backslash sum\_\{i=1\}^N\; e^\{-y\_i\; C\_m(x\_i)\}\; =\; \backslash sum\_\{i=1\}^N\; e^\{-y\_i\; C\_\{(m-1)\}(x\_i)\}e^\{-y\_i\; \backslash alpha\_m\; k\_m(x\_i)\}$$

Letting $w\_i^\{(1)\}\; =\; 1$ and $w\_i^\{(m)\}\; =\; e^\{-y\_i\; C\_\{m-1\}(x\_i)\}$ for $m\; >\; 1$, we have:

$$E\; =\; \backslash sum\_\{i=1\}^N\; w\_i^\{(m)\}e^\{-y\_i\backslash alpha\_m\; k\_m(x\_i)\}$$

We can split this summation between those data points that are correctly classified by $k\_m$ (so $y\_i\; k\_m(x\_i)\; =\; 1$) and those that are misclassified (so $y\_i\; k\_m(x\_i)\; =\; -1$):

$$\backslash begin\{align\}$$

E &= \sum_{y_i = k_m(x_i)} w_i^{(m)}e^{-\alpha_m} + \sum_{y_i \neq k_m(x_i)} w_i^{(m)}e^{\alpha_m} \\

&= \sum_{i=1}^N w_i^{(m)}e^{-\alpha_m} + \sum_{y_i \neq k_m(x_i)} w_i^{(m)}\left(e^{\alpha_m} - e^{-\alpha_m}\right)

\end{align}

Since the only part of the right-hand side of this equation that depends on $k\_m$ is $\backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}$, we see that the $k\_m$ that minimizes $E$ is the one in the set $\backslash \{k\_1,\; \backslash ldots,\; k\_L\backslash \}$ that minimizes $\backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}$ [assuming that $\backslash alpha\_m\; >\; 0$], i.e. the weak classifier with the lowest weighted error (with weights $w\_i^\{(m)\}\; =\; e^\{-y\_i\; C\_\{m-1\}(x\_i)\}$).

To determine the desired weight $\backslash alpha\_m$ that minimizes $E$ with the $k\_m$ that we just determined, we differentiate:

$$\backslash frac\{d\; E\}\{d\; \backslash alpha\_m\}\; =\; \backslash frac\{d\; (\backslash sum\_\{y\_i\; =\; k\_m(x\_i)\}\; w\_i^\{(m)\}e^\{-\backslash alpha\_m\}\; +\; \backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}e^\{\backslash alpha\_m\})\; \}\{d\; \backslash alpha\_m\}$$

Luckily the minimum occurs when setting this to zero, then solving for $\backslash alpha\_m$ yields:

$$\backslash alpha\_m\; =\; \backslash frac\{1\}\{2\}\backslash ln\backslash left(\backslash frac\{\backslash sum\_\{y\_i\; =\; k\_m(x\_i)\}\; w\_i^\{(m)\}\}\{\backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}\}\backslash right)$$

{{Math proof|drop=hidden|

$$\backslash frac\{d\; E\}\{d\; \backslash alpha\_m\}\; =\; -\backslash sum\_\{y\_i\; =\; k\_m(x\_i)\}\; w\_i^\{(m)\}e^\{-\backslash alpha\_m\}\; +\; \backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}e^\{\backslash alpha\_m\}\; =\; 0$$

because $e^\{-\backslash alpha\_m\}$ does not depend on $i$

$$e^\{-\backslash alpha\_m\}\; \backslash sum\_\{y\_i\; =\; k\_m(x\_i)\}\; w\_i^\{(m)\}\; =\; e^\{\backslash alpha\_m\}\; \backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}$$

$$-\backslash alpha\_m\; +\; \backslash ln\backslash left(\; \backslash sum\_\{y\_i\; =\; k\_m(x\_i)\}\; w\_i^\{(m)\}\; \backslash right)\; =\; \backslash alpha\_m\; +\; \backslash ln\backslash left(\backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}\; \backslash right)$$

$$-2\backslash alpha\_m\; =\; \backslash ln\backslash left(\; \backslash dfrac\{\backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}\}\{\backslash sum\_\{y\_i\; =\; k\_m(x\_i)\}\; w\_i^\{(m)\}\}\; \backslash right)$$

$$\backslash alpha\_m\; =\; -\; \backslash dfrac\{1\}\{2\}\; \backslash ln\backslash left(\; \backslash dfrac\{\backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}\}\{\backslash sum\_\{y\_i\; =\; k\_m(x\_i)\}\; w\_i^\{(m)\}\}\; \backslash right)$$

$$\backslash alpha\_m\; =\; \backslash dfrac\{1\}\{2\}\; \backslash ln\backslash left(\; \backslash dfrac\{\backslash sum\_\{y\_i\; =\; k\_m(x\_i)\}\; w\_i^\{(m)\}\}\{\backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}\}\; \backslash right)$$

}}

We calculate the weighted error rate of the weak classifier to be $\backslash epsilon\_m\; =\; \backslash frac\{\backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}\}\{\backslash sum\_\{i=1\}^N\; w\_i^\{(m)\}\}$, so it follows that:

$$\backslash alpha\_m\; =\; \backslash frac\{1\}\{2\}\backslash ln\backslash left(\; \backslash frac\{1\; -\; \backslash epsilon\_m\}\{\backslash epsilon\_m\}\backslash right)$$

which is the negative logit function multiplied by 0.5. Due to the convexity of $E$ as a function of $\backslash alpha\_m$, this new expression for $\backslash alpha\_m$ gives the global minimum of the loss function.

Note: This derivation only applies when $k\_m(x\_i)\; \backslash in\; \backslash \{-1,\; 1\backslash \}$, though it can be a good starting guess in other cases, such as when the weak learner is biased ($k\_m(x)\; \backslash in\; \backslash \{a,b\backslash \},\; a\; \backslash neq\; -b$), has multiple leaves ($k\_m(x)\; \backslash in\; \backslash \{a,b,\backslash dots,n\backslash \}$) or is some other function $k\_m(x)\; \backslash in\; \backslash mathbb\{R\}$.

Thus we have derived the AdaBoost algorithm: At each iteration, choose the classifier $k\_m$, which minimizes the total weighted error $\backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}$, use this to calculate the error rate $\backslash epsilon\_m\; =\; \backslash frac\{\backslash sum\_\{y\_i\; \backslash neq\; k\_m(x\_i)\}\; w\_i^\{(m)\}\}\{\backslash sum\_\{i=1\}^N\; w\_i^\{(m)\}\}$, use this to calculate the weight $\backslash alpha\_m\; =\; \backslash frac\{1\}\{2\}\backslash ln\backslash left(\; \backslash frac\{1\; -\; \backslash epsilon\_m\}\{\backslash epsilon\_m\}\backslash right)$, and finally use this to improve the boosted classifier $C\_\{m-1\}$ to $C\_\{m\}\; =\; C\_\{(m-1)\}\; +\; \backslash alpha\_m\; k\_m$.

{{Refimprove section|date=May 2016}}

Boosting is a form of linear regression in which the features of each sample $x\_i$ are the outputs of some weak learner $h$ applied to $x\_i$.

While regression tries to fit $F(x)$ to $y(x)$ as precisely as possible without loss of generalization, typically using least square error $E(f)\; =\; (y(x)\; -\; f(x))^2$, whereas the AdaBoost error function $E(f)\; =\; e^\{-y(x)f(x)\}$ takes into account the fact that only the sign of the final result is used, thus $|F(x)|$ can be far larger than 1 without increasing error. However, the exponential increase in the error for sample $x\_i$ as $-y(x\_i)f(x\_i)$ increases, resulting in excessive weights being assigned to outliers.

One feature of the choice of exponential error function is that the error of the final additive model is the product of the error of each stage, that is, $e^\{\backslash sum\_i\; -y\_i\; f(x\_i)\}\; =\; \backslash prod\_i\; e^\{-y\_i\; f(x\_i)\}$. Thus it can be seen that the weight update in the AdaBoost algorithm is equivalent to recalculating the error on $F\_t(x)$ after each stage.

There is a lot of flexibility allowed in the choice of loss function. As long as the loss function is monotonic and continuously differentiable, the classifier is always driven toward purer solutions. Zhang (2004) provides a loss function based on least squares, a modified Huber loss function:

$$\backslash phi(y,f(x))\; =\; \backslash begin\{cases\}$$

-4y f(x) & \text{if }y f(x) < -1, \\

(y f(x) - 1)^2 &\text{if } -1 \le y f(x) \le 1. \\

0 &\text{if } y f(x) > 1

\end{cases}

This function is more well-behaved than LogitBoost for $f(x)$ close to 1 or -1, does not penalise ‘overconfident’ predictions ($y\; f(x)\; >\; 1$), unlike unmodified least squares, and only penalises samples misclassified with confidence greater than 1 linearly, as opposed to quadratically or exponentially, and is thus less susceptible to the effects of outliers.

{{Main article|Gradient boosting}}

Boosting can be seen as minimization of a convex loss function over a convex set of functions.{{cite journal |first=T. |last=Zhang |title=Statistical behavior and consistency of classification methods based on convex risk minimization |journal=Annals of Statistics |volume=32 |issue=1 |pages=56–85 |year=2004 |doi=10.1214/aos/1079120130 |jstor=3448494 |doi-access=free }} Specifically, the loss being minimized by AdaBoost is the exponential loss

$$\backslash sum\_i\; \backslash phi(i,y,f)\; =\; \backslash sum\_i\; e^\{-y\_i\; f(x\_i)\},$$

whereas LogitBoost performs logistic regression, minimizing

$$\backslash sum\_i\; \backslash phi(i,y,f)\; =\; \backslash sum\_i\; \backslash ln\backslash left(1+e^\{-y\_i\; f(x\_i)\}\backslash right).$$

In the gradient descent analogy, the output of the classifier for each training point is considered a point $\backslash left(F\_t(x\_1),\; \backslash dots,\; F\_t(x\_n)\backslash right)$ in n-dimensional space, where each axis corresponds to a training sample, each weak learner $h(x)$ corresponds to a vector of fixed orientation and length, and the goal is to reach the target point $(y\_1,\; \backslash dots,\; y\_n)$ (or any region where the value of loss function $E\_T(x\_1,\; \backslash dots,\; x\_n)$ is less than the value at that point), in the fewest steps. Thus AdaBoost algorithms perform either Cauchy (find $h(x)$ with the steepest gradient, choose $\backslash alpha$ to minimize test error) or Newton (choose some target point, find $\backslash alpha\; h(x)$ that brings $F\_t$ closest to that point) optimization of training error.

With:

• Samples $x\_1\; \backslash dots\; x\_n$
• Desired outputs $y\_1\; \backslash dots\; y\_n,\; y\; \backslash in\; \backslash \{-1,\; 1\backslash \}$
• Initial weights $w\_\{1,1\}\; \backslash dots\; w\_\{n,1\}$ set to $\backslash frac\{1\}\{n\}$
• Error function $E(f(x),\; y\_i)\; =\; e^\{-y\_i\; f(x\_i)\}$
• Weak learners $h\backslash colon\; x\; \backslash rightarrow\; \backslash \{-1,\; 1\backslash \}$

For $t$ in $1\; \backslash dots\; T$:

• Choose $h\_t(x)$:
• Find weak learner $h\_t(x)$ that minimizes $\backslash epsilon\_t$, the weighted sum error for misclassified points $\backslash epsilon\_t\; =\; \backslash sum^n\_\{\backslash stackrel\{i=1\}\{h\_t(x\_i)\backslash neq\; y\_i\}\}\; w\_\{i,t\}$
• Choose $\backslash alpha\_t\; =\; \backslash frac\{1\}\{2\}\; \backslash ln\; \backslash left(\backslash frac\{1-\backslash epsilon\_t\}\{\backslash epsilon\_t\}\backslash right)$
• Add to ensemble:
• $F\_t(x)\; =\; F\_\{t-1\}(x)\; +\; \backslash alpha\_t\; h\_t(x)$
• Update weights:
• $w\_\{i,t+1\}\; =\; w\_\{i,t\}\; e^\{-y\_i\; \backslash alpha\_t\; h\_t(x\_i)\}$ for $i$ in $1\; \backslash dots\; n$
• Renormalize $w\_\{i,t+1\}$ such that $\backslash sum\_i\; w\_\{i,t+1\}\; =\; 1$
• (Note: It can be shown that $\backslash frac\{\backslash sum\_\{h\_\{t\}(x\_i)\; =\; y\_i\}\; w\_\{i,t+1\}\}\{\backslash sum\_\{h\_\{t\}(x\_i)\; \backslash neq\; y\_i\}\; w\_\{i,t+1\}\}\; =\; \backslash frac\{\backslash sum\_\{h\_t(x\_i)\; =\; y\_i\}\; w\_\{i,t\}\}\{\backslash sum\_\{h\_t(x\_i)\; \backslash neq\; y\_i\}\; w\_\{i,t\}\}$ at every step, which can simplify the calculation of the new weights.)

The output of decision trees is a class probability estimate $p(x)\; =\; P(y=1\; |\; x)$, the probability that $x$ is in the positive class.{{cite journal | citeseerx = 10.1.1.51.9525 | title = Additive Logistic Regression: A Statistical View of Boosting | first1 = Jerome | last1 = Friedman | first2 = Trevor | last2 = Hastie | first3 = Robert | last3 = Tibshirani | journal = Annals of Statistics | year = 1998| volume = 28 | page = 2000 }} Friedman, Hastie and Tibshirani derive an analytical minimizer for $e^\{-y\backslash left(F\_\{t-1\}(x)+f\_t(p(x))\backslash right)\}$ for some fixed $p(x)$ (typically chosen using weighted least squares error):

:$f\_t(x)\; =\; \backslash frac\{1\}\{2\}\; \backslash ln\backslash left(\backslash frac\{x\}\{1-x\}\backslash right)$.

Thus, rather than multiplying the output of the entire tree by some fixed value, each leaf node is changed to output half the logit transform of its previous value.

{{Main article|LogitBoost}}

LogitBoost represents an application of established logistic regression techniques to the AdaBoost method. Rather than minimizing error with respect to y, weak learners are chosen to minimize the (weighted least-squares) error of $f\_t(x)$ with respect to

$$z\_t\; =\; \backslash frac\{y^*\; -\; p\_t(x)\}\{2\; p\_t(x)(1\; -\; p\_t(x))\},$$

where

$$p\_t(x)\; =\; \backslash frac\{e^\{F\_\{t-1\}(x)\}\}\{e^\{F\_\{t-1\}(x)\}+e^\{-F\_\{t-1\}(x)\}\},$$

$$w\_t\; =\; p\_t(x)(1\; -\; p\_t(x))$$

$$y^*\; =\; \backslash frac\{y+1\}\; 2.$$

That is $z\_t$ is the Newton–Raphson approximation of the minimizer of the log-likelihood error at stage $t$, and the weak learner $f\_t$ is chosen as the learner that best approximates $z\_t$ by weighted least squares.

As p approaches either 1 or 0, the value of $p\_t(x\_i)(1\; -\; p\_t(x\_i))$ becomes very small and the z term, which is large for misclassified samples, can become numerically unstable, due to machine precision rounding errors. This can be overcome by enforcing some limit on the absolute value of z and the minimum value of w

While previous boosting algorithms choose $f\_t$ greedily, minimizing the overall test error as much as possible at each step, GentleBoost features a bounded step size. $f\_t$ is chosen to minimize $\backslash sum\_i\; w\_\{t,i\}\; (y\_i-f\_t(x\_i))^2$, and no further coefficient is applied. Thus, in the case where a weak learner exhibits perfect classification performance, GentleBoost chooses $f\_t(x)\; =\; \backslash alpha\_t\; h\_t(x)$ exactly equal to $y$, while steepest descent algorithms try to set $\backslash alpha\_t\; =\; \backslash infty$. Empirical observations about the good performance of GentleBoost appear to back up Schapire and Singer's remark that allowing excessively large values of $\backslash alpha$ can lead to poor generalization performance.{{cite journal |last1=Schapire |first1=Robert |last2=Singer |first2=Yoram |year=1999 |title=Improved Boosting Algorithms Using Confidence-rated Predictions |pages=80–91 |citeseerx=10.1.1.33.4002}}{{cite web | url = http://www.site.uottawa.ca/~stan/csi5387/boost-tut-ppr.pdf | title = A Short Introduction to Boosting | last1 = Freund | last2 = Schapire | year = 1999 | postscript = : }}

A technique for speeding up processing of boosted classifiers, early termination refers to only testing each potential object with as many layers of the final classifier necessary to meet some confidence threshold, speeding up computation for cases where the class of the object can easily be determined. One such scheme is the object detection framework introduced by Viola and Jones:{{cite journal | citeseerx = 10.1.1.10.6807 | title = Rapid Object Detection Using a Boosted Cascade of Simple Features | first1 = Paul | last1 = Viola | first2 = Robert | last2 = Jones | year = 2001}} in an application with significantly more negative samples than positive, a cascade of separate boost classifiers is trained, the output of each stage biased such that some acceptably small fraction of positive samples is mislabeled as negative, and all samples marked as negative after each stage are discarded. If 50% of negative samples are filtered out by each stage, only a very small number of objects would pass through the entire classifier, reducing computation effort. This method has since been generalized, with a formula provided for choosing optimal thresholds at each stage to achieve some desired false positive and false negative rate.{{cite journal | title = Optimizing cascade classifiers. | last1 = McCane | first1 = Brendan | first2 = Kevin | last2 = Novins | first3 = Michael | last3 = Albert | year = 2005}}

In the field of statistics, where AdaBoost is more commonly applied to problems of moderate dimensionality, early stopping is used as a strategy to reduce overfitting.{{cite book|author1=Trevor Hastie |author2=Robert Tibshirani |author3=Jerome Friedman |title=The Elements of Statistical Learning: Data Mining, Inference, and Prediction|year=2009|publisher=Springer|location=New York|isbn=978-0-387-84858-7|url=http://statweb.stanford.edu/~tibs/ElemStatLearn/download.html|edition=2nd}} A validation set of samples is separated from the training set, performance of the classifier on the samples used for training is compared to performance on the validation samples, and training is terminated if performance on the validation sample is seen to decrease even as performance on the training set continues to improve.

For steepest descent versions of AdaBoost, where $\backslash alpha\_t$ is chosen at each layer t to minimize test error, the next layer added is said to be maximally independent of layer t:{{cite book | title = Adaboost with Totally Corrective Updates for Fast Face Detection | first1 = Jan | last1 = Šochman | first2 = Jiří | last2 = Matas | isbn=978-0-7695-2122-0 | year = 2004}} it is unlikely to choose a weak learner t+1 that is similar to learner t. However, there remains the possibility that t+1 produces similar information to some other earlier layer. Totally corrective algorithms, such as LPBoost, optimize the value of every coefficient after each step, such that new layers added are always maximally independent of every previous layer. This can be accomplished by backfitting, linear programming or some other method.

Pruning is the process of removing poorly performing weak classifiers to improve memory and execution-time cost of the boosted classifier. The simplest methods, which can be particularly effective in conjunction with totally corrective training, are weight- or margin-trimming: when the coefficient, or the contribution to the total test error, of some weak classifier falls below a certain threshold, that classifier is dropped. Margineantu & Dietterich{{cite journal | citeseerx = 10.1.1.38.7017 | title = Pruning Adaptive Boosting | first1 = Dragos | last1 = Margineantu | first2 = Thomas | last2 = Dietterich | year = 1997}} suggested an alternative criterion for trimming: weak classifiers should be selected such that the diversity of the ensemble is maximized. If two weak learners produce very similar outputs, efficiency can be improved by removing one of them and increasing the coefficient of the remaining weak learner.{{cite journal | title = On the Boosting Pruning Problem | first1 = Christino | last1 = Tamon | first2 = Jie | last2 = Xiang | year = 2000}}

• Bootstrap aggregating
• CoBoosting
• BrownBoost
• Gradient boosting
• {{slink|Multiplicative weight update method|AdaBoost algorithm}}

{{reflist}}

{{refbegin}}

• {{cite journal | citeseerx = 10.1.1.32.8918 | title = A decision-theoretic generalization of on-line learning and an application to boosting | journal = Journal of Computer and System Sciences | volume = 55 | pages = 119–139 | year = 1997 | postscript = : | doi = 10.1006/jcss.1997.1504 | last1 = Freund | first1 = Yoav | last2 = Schapire | first2 = Robert E }} original paper of Yoav Freund and Robert E.Schapire where AdaBoost is first introduced.
• {{cite journal | url = http://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/colt08.pdf | title = On the margin explanation of boosting algorithm. | first = Zhihua | last = Zhou | journal = In: Proceedings of the 21st Annual Conference on Learning Theory (COLT'08) | pages = 479–490 | year = 2008 }} On the margin explanation of boosting algorithm.
• {{cite journal | url = http://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/aij13marginbound.pdf | title = On the doubt about margin explanation of boosting | first = Zhihua | last = Zhou | journal = Artificial Intelligence | volume = 203 | issue = 2013 | pages = 1–18 | year = 2013 | bibcode = 2010arXiv1009.3613G | arxiv = 1009.3613 | doi = 10.1016/j.artint.2013.07.002 | s2cid = 2828847 }} On the doubt about margin explanation of boosting.

{{refend}}

{{DEFAULTSORT:AdaBoost}}

Category:Classification algorithms

Category:Ensemble learning