LogSumExp

The LogSumExp function domain is $\backslash R^n$, the real coordinate space, and its codomain is $\backslash R$, the real line.

It is an approximation to the maximum $\backslash max\_i\; x\_i$ with the following bounds

$$\backslash max\{\backslash \{x\_1,\; \backslash dots,\; x\_n\backslash \}\}\; \backslash leq\; \backslash mathrm\{LSE\}(x\_1,\; \backslash dots,\; x\_n)\; \backslash leq\; \backslash max\{\backslash \{x\_1,\; \backslash dots,\; x\_n\backslash \}\}\; +\; \backslash log(n).$$

The first inequality is strict unless $n\; =\; 1$. The second inequality is strict unless all arguments are equal.

(Proof: Let $m\; =\; \backslash max\_i\; x\_i$. Then $\backslash exp(m)\; \backslash leq\; \backslash sum\_\{i=1\}^n\; \backslash exp(x\_i)\; \backslash leq\; n\; \backslash exp(m)$. Applying the logarithm to the inequality gives the result.)

In addition, we can scale the function to make the bounds tighter. Consider the function $\backslash frac\; 1\; t\; \backslash mathrm\{LSE\}(tx\_1,\; \backslash dots,\; tx\_n)$. Then

(Proof: Replace each $x\_i$ with $tx\_i$ for some $t>0$ in the inequalities above, to give

and, since $t>0$

finally, dividing by $t$ gives the result.)

Also, if we multiply by a negative number instead, we of course find a comparison to the $\backslash min$ function:

The LogSumExp function is convex, and is strictly increasing everywhere in its domain.{{cite book|url=http://livebooklabs.com/keeppies/c5a5868ce26b8125|title=Optimization Models and Applications|first=Laurent|last=El Ghaoui|year=2017}} It is not strictly convex, since it is affine (linear plus a constant) on the diagonal and parallel lines:{{cite web|url=https://math.stackexchange.com/q/1189638 |title=convex analysis - About the strictly convexity of log-sum-exp function - Mathematics Stack Exchange|work=stackexchange.com}}

:$\backslash mathrm\{LSE\}(x\_1\; +\; c,\; \backslash dots,\; x\_n\; +\; c)\; =\backslash mathrm\{LSE\}(x\_1,\; \backslash dots,\; x\_n)\; +\; c.$

Other than this direction, it is strictly convex (the Hessian has rank {{tmath|n - 1}}), so for example restricting to a hyperplane that is transverse to the diagonal results in a strictly convex function. See $\backslash mathrm\{LSE\}\_0^+$, below.

Writing $\backslash mathbf\{x\}\; =\; (x\_1,\; \backslash dots,\; x\_n),$ the partial derivatives are:

$$\backslash frac\{\backslash partial\}\{\backslash partial\; x\_i\}\{\backslash mathrm\{LSE\}(\backslash mathbf\{x\})\}\; =$$

\frac{\exp x_i}{\sum_j \exp {x_j}},

which means the gradient of LogSumExp is the softmax function.

The convex conjugate of LogSumExp is the negative entropy.

The LSE function is often encountered when the usual arithmetic computations are performed on a logarithmic scale, as in log probability.{{Cite book|last=McElreath|first=Richard|url=http://worldcat.org/oclc/1107423386|title=Statistical Rethinking|oclc=1107423386}}

Similar to multiplication operations in linear-scale becoming simple additions in log-scale, an addition operation in

linear-scale becomes the LSE in log-scale:

$$\backslash mathrm\{LSE\}(\backslash log(x\_1),\; ...,\; \backslash log(x\_n))\; =\; \backslash log(x\_1\; +\; \backslash dots\; +\; x\_n)$$

A common purpose of using log-domain computations is to increase accuracy and avoid underflow and overflow problems

when very small or very large numbers are represented directly (i.e. in a linear domain) using limited-precision

floating point numbers.{{Cite web|title=Practical issues: Numeric stability.|url=https://cs231n.github.io/linear-classify/#softmax-classifier|website=CS231n Convolutional Neural Networks for Visual Recognition}}

Unfortunately, the use of LSE directly in this case can again cause overflow/underflow problems. Therefore, the

following equivalent must be used instead (especially when the accuracy of the above 'max' approximation is not sufficient).

$$\backslash mathrm\{LSE\}(x\_1,\; \backslash dots,\; x\_n)\; =\; x^*\; +\; \backslash log\backslash left(\; \backslash exp(x\_1-x^*)+\; \backslash cdots\; +\; \backslash exp(x\_n-x^*)\; \backslash right)$$

where $x^*\; =\; \backslash max\{\backslash \{x\_1,\; \backslash dots,\; x\_n\backslash \}\}$

Many math libraries such as IT++ provide a default routine of LSE and use this formula internally.

LSE is convex but not strictly convex.

We can define a strictly convex log-sum-exp type function{{cite arXiv

| first1 = Frank | last1 = Nielsen

| first2 = Gaetan | last2 = Hadjeres

| year = 2018

| title = Monte Carlo Information Geometry: The dually flat case

| class = cs.LG

| eprint = 1803.07225}} by adding an extra argument set to zero:

$$\backslash mathrm\{LSE\}\_0^+(x\_1,...,x\_n)\; =\; \backslash mathrm\{LSE\}(0,x\_1,...,x\_n)$$

This function is a proper Bregman generator (strictly convex and differentiable).

It is encountered in machine learning, for example, as the cumulant of the multinomial/binomial family.

In tropical analysis, this is the sum in the log semiring.

• Logarithmic mean
• Log semiring
• Smooth maximum
• Softmax function

{{Reflist}}

{{refbegin}}

{{refend}}

Category:Logarithms