infomax

Infomax, or the principle of maximum information preservation, is an optimization principle for artificial neural networks and other information processing systems. It prescribes that a function that maps a set of input values $x$ to a set of output values $z(x)$ should be chosen or learned so as to maximize the average Shannon mutual information between $x$ and $z(x)$, subject to a set of specified constraints and/or noise processes. Infomax algorithms are learning algorithms that perform this optimization process. The principle was described by Linsker in 1988.{{cite journal |doi=10.1109/2.36 |author=Linsker R |title=Self-organization in a perceptual network |journal=IEEE Computer |volume=21 |issue=3 |pages=105–17 |year=1988 |s2cid=1527671 }} The objective function is called the InfoMax objective.

As the InfoMax objective is difficult to compute exactly, a related notion uses two models giving two outputs $z\_1(x),\; z\_2(x)$, and maximizes the mutual information between these. This contrastive InfoMax objective is a lower bound to the InfoMax objective.{{Cite journal |last1=Becker |first1=Suzanna |last2=Hinton |first2=Geoffrey E. |date=January 1992 |title=Self-organizing neural network that discovers surfaces in random-dot stereograms |url=https://www.nature.com/articles/355161a0 |journal=Nature |language=en |volume=355 |issue=6356 |pages=161–163 |doi=10.1038/355161a0 |pmid=1729650 |bibcode=1992Natur.355..161B |issn=1476-4687|url-access=subscription }}

Infomax, in its zero-noise limit, is related to the principle of redundancy reduction proposed for biological sensory processing by Horace Barlow in 1961,{{cite book |author=Barlow, H. |chapter=Possible principles underlying the transformations of sensory messages |editor=Rosenblith, W. |title=Sensory Communication |publisher=MIT Press |location=Cambridge MA |year=1961 |pages=217–234 }} and applied quantitatively to retinal processing by Atick and Redlich.{{cite journal |doi=10.1162/neco.1992.4.2.196 |author=Atick JJ, Redlich AN |title=What does the retina know about natural scenes? |journal=Neural Computation |volume=4 |pages=196–210 |year=1992 |issue=2 |s2cid=17515861 }}