Swish function

For β = 0, the function is linear: f(x) = x/2.

For β = 1, the function is the Sigmoid Linear Unit (SiLU).

With β → ∞, the function converges to ReLU.

Thus, the swish family smoothly interpolates between a linear function and the ReLU function.

Since $\backslash operatorname\{swish\}\_\backslash beta(x)\; =\; \backslash operatorname\{swish\}\_1(\backslash beta\; x)\; /\; \backslash beta$, all instances of swish have the same shape as the default $\backslash operatorname\{swish\}\_1$, zoomed by $\backslash beta$. One usually sets $\backslash beta\; >\; 0$. When $\backslash beta$ is trainable, this constraint can be enforced by $\backslash beta\; =\; e^b$, where $b$ is trainable.

$$\backslash operatorname\{swish\}\_1(x)\; =\; \backslash frac\; x2\; +\; \backslash frac\{x^2\}\{4\}-\backslash frac\{x^4\}\{48\}+\backslash frac\{x^6\}\{480\}+O\backslash left(x^8\backslash right)$$

$$\backslash begin\{aligned\}$$

\operatorname{swish}_1(x) &= \frac{x}{2} \tanh \left(\frac{x}{2}\right) + \frac{x}{2}\\

\operatorname{swish}_1(x) + \operatorname{swish}_{-1}(x) &= x \tanh \left(\frac{x}{2}\right) \\

\operatorname{swish}_1(x) - \operatorname{swish}_{-1}(x) &= x

\end{aligned}

Because $\backslash operatorname\{swish\}\_\backslash beta(x)\; =\; \backslash operatorname\{swish\}\_1(\backslash beta\; x)\; /\; \backslash beta$, it suffices to calculate its derivatives for the default case.$$\backslash operatorname\{swish\}\_1\text{'}(x)\; =\; \backslash frac\{x+\backslash sinh\; (x)\}\{4\; \backslash cosh\; ^2\backslash left(\backslash frac\{x\}\{2\}\backslash right)\}\; +\; \backslash frac\; 12$$so $\backslash operatorname\{swish\}\_1\text{'}(x)\; -\; \backslash frac\; 12$ is odd.$$\backslash operatorname\{swish\}\_1$$(x) = \frac{1- \frac x2 \tanh \left(\frac{x}{2}\right)}{2 \cosh ^2\left(\frac{x}{2}\right)} so $\backslash operatorname\{swish\}\_1$(x) is even.

SiLU was first proposed alongside the GELU in 2016, then again proposed in 2017 as the Sigmoid-weighted Linear Unit (SiL) in reinforcement learning. The SiLU/SiL was then again proposed as the SWISH over a year after its initial discovery, originally proposed without the learnable parameter β, so that β implicitly equaled 1. The swish paper was then updated to propose the activation with the learnable parameter β.

In 2017, after performing analysis on ImageNet data, researchers from Google indicated that using this function as an activation function in artificial neural networks improves the performance, compared to ReLU and sigmoid functions. It is believed that one reason for the improvement is that the swish function helps alleviate the vanishing gradient problem during backpropagation.

• Activation function
• Gating mechanism

{{reflist|refs=

{{cite arXiv |eprint = 1606.08415 |title = Gaussian Error Linear Units (GELUs) |last1 = Hendrycks |first1 = Dan |last2 = Gimpel |first2 = Kevin |year = 2016 |class = cs.LG}}

{{cite arXiv |first1=Stefan |last1=Elfwing |first2=Eiji |last2=Uchibe |first3=Kenji |last3=Doya |title=Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning |date=2017-11-02 |class=cs.LG |eprint=1702.03118v3}}

{{cite arXiv |title=Searching for Activation Functions |date=2017-10-27 |eprint=1710.05941v2 |last1=Ramachandran |first1=Prajit |last2=Zoph |first2=Barret |last3=Le |first3=Quoc V.|class=cs.NE }}

{{cite web |title=Swish as Neural Networks Activation Function |first=Sefik Ilkin |last=Serengil |series=Machine Learning, Math |date=2018-08-21 |url=https://sefiks.com/2018/08/21/swish-as-neural-networks-activation-function/ |access-date=2020-06-18 |url-status=live |archive-url=https://web.archive.org/web/20200618093348/https://sefiks.com/2018/08/21/swish-as-neural-networks-activation-function/ |archive-date=2020-06-18}}

}}{{Differentiable computing}}

Category:Functions and mappings

Category:Artificial neural networks