interaction information

The conditional mutual information can be used to inductively define the interaction information for any finite number of variables as follows:

:$I(X\_1;\backslash ldots;X\_\{n+1\})\; =\; I(X\_1;\backslash ldots;X\_n)\; -\; I(X\_1;\backslash ldots;X\_n\backslash mid\; X\_\{n+1\}),$

where

:$I(X\_1;\backslash ldots;X\_n\; \backslash mid\; X\_\{n+1\})\; =\; \backslash mathbb\; E\_\{X\_\{n+1\}\}\backslash big(I(X\_1;\backslash ldots;X\_n)\; \backslash mid\; X\_\{n+1\}\backslash big).$

Some authors{{Cite journal|last=McGill|first=William J.|date=June 1954|title=Multivariate information transmission|url=http://dx.doi.org/10.1007/bf02289159|journal=Psychometrika|volume=19|issue=2|pages=97–116|doi=10.1007/bf02289159|s2cid=126431489 |issn=0033-3123|url-access=subscription}} define the interaction information differently, by swapping the two terms being subtracted in the preceding equation. This has the effect of reversing the sign for an odd number of variables.

For three variables $\backslash \{X,Y,Z\backslash \}$, the interaction information $I(X;Y;Z)$ is given by

:$$

I(X;Y;Z) = I(X;Y)-I(X;Y \mid Z)

where $I(X;Y)$ is the mutual information between variables $X$ and $Y$, and $I(X;Y\; \backslash mid\; Z)$ is the conditional mutual information between variables $X$ and $Y$ given $Z$. The interaction information is symmetric, so it does not matter which variable is conditioned on. This is easy to see when the interaction information is written in terms of entropy and joint entropy, as follows:

:$$

\begin{alignat}{3}

I(X;Y;Z) &= &&\; \bigl( H(X) + H(Y) + H(Z) \bigr) \\

& &&- \bigl( H(X,Y) + H(X,Z) + H(Y,Z) \bigr) \\

& &&+ H(X,Y,Z)

\end{alignat}

In general, for the set of variables $\backslash mathcal\{V\}=\backslash \{X\_\{1\},X\_\{2\},\backslash ldots\; ,X\_\{n\}\backslash \}$, the interaction information can be written in the following form (compare with Kirkwood approximation):

:$$

I(\mathcal{V}) = \sum_{\mathcal{T}\subseteq \mathcal{V}}(-1)^{\left\vert\mathcal{T}\right\vert-1}H(\mathcal{T})

For three variables, the interaction information measures the influence of a variable $Z$ on the amount of information shared between $X$ and $Y$. Because the term $I(X;Y\; \backslash mid\; Z)$ can be larger than $I(X;Y)$, the interaction information can be negative as well as positive. This will happen, for example, when $X$ and $Y$ are independent but not conditionally independent given $Z$. Positive interaction information indicates that variable $Z$ inhibits (i.e., accounts for or explains some of) the correlation between $X$ and $Y$, whereas negative interaction information indicates that variable $Z$ facilitates or enhances the correlation.

Interaction information is bounded. In the three variable case, it is bounded by{{Cite journal|last=Yeung|first=R.W.|date=May 1991|title=A new outlook on Shannon's information measures|url=http://dx.doi.org/10.1109/18.79902|journal=IEEE Transactions on Information Theory|volume=37|issue=3|pages=466–474|doi=10.1109/18.79902|issn=0018-9448|url-access=subscription}}

:$-\backslash min\; \backslash \{\; I(X;Y\; \backslash mid\; Z),\; I(Y;Z\; \backslash mid\; X),\; I(X;Z\; \backslash mid\; Y)\; \backslash \}\; \backslash leq\; I(X;Y;Z)\; \backslash leq\; \backslash min\; \backslash \{\; I(X;Y),\; I(Y;Z),\; I(X;Z)\; \backslash \}$

If three variables form a Markov chain $X\backslash to\; Y\; \backslash to\; Z$, then $I(X;Z\; \backslash mid\; Y)=0$, but $I(X;Z)\backslash ge\; 0$. Therefore

:$I(X;Y;Z)\; =\; I(X;Z)\; -\; I(X;Z\; \backslash mid\; Y)\; =\; I(X;Z)\; \backslash ge\; 0.$

Positive interaction information seems much more natural than negative interaction information in the sense that such explanatory effects are typical of common-cause structures. For example, clouds cause rain and also block the sun; therefore, the correlation between rain and darkness is partly accounted for by the presence of clouds, . The result is positive interaction information $I(\backslash text\{rain\};\backslash text\{dark\};\backslash text\{cloud\})$.

A car's engine can fail to start due to either a dead battery or a blocked fuel pump. Ordinarily, we assume that battery death and fuel pump blockage are independent events, $I(\backslash text\{blocked\; fuel\};\; \backslash text\{dead\; battery\})\; =\; 0$. But knowing that the car fails to start, if an inspection shows the battery to be in good health, we can conclude that the fuel pump must be blocked. Therefore $I(\backslash text\{blocked\; fuel\};\; \backslash text\{dead\; battery\}\; \backslash mid\; \backslash text\{engine\; fails\})\; >\; 0$, and the result is negative interaction information.

{{Confusing|section|date=June 2023}}

The possible negativity of interaction information can be the source of some confusion. Many authors have taken zero interaction information as a sign that three or more random variables do not interact, but this interpretation is wrong.{{Cite journal|last=Krippendorff|first=Klaus|date=August 2009|title=Information of interactions in complex systems|url=http://www.tandfonline.com/doi/abs/10.1080/03081070902993160|journal=International Journal of General Systems|language=en|volume=38|issue=6|pages=669–680|doi=10.1080/03081070902993160|s2cid=13923485 |issn=0308-1079}}

To see how difficult interpretation can be, consider a set of eight independent binary variables $\backslash \{X\_\{1\},X\_\{2\},X\_\{3\},X\_\{4\},X\_\{5\},X\_\{6\},X\_\{7\},X\_\{8\}\backslash \}$. Agglomerate these variables as follows:

:$$

\begin{align}

Y_{1} &=\{X_{1},X_{2},X_{3},X_{4},X_{5},X_{6},X_{7}\} \\

Y_{2} &=\{X_{4},X_{5},X_{6},X_{7}\} \\

Y_{3} &=\{X_{5},X_{6},X_{7},X_{8}\}

\end{align}

Because the $Y\_\{i\}$'s overlap each other (are redundant) on the three binary variables $\backslash \{X\_\{5\},X\_\{6\},X\_\{7\}\backslash \}$, we would expect the interaction information $I(Y\_\{1\};Y\_\{2\};Y\_\{3\})$ to equal $3$ bits, which it does. However, consider now the agglomerated variables

:$$

\begin{align}

Y_{1} &=\{X_{1},X_{2},X_{3},X_{4},X_{5},X_{6},X_{7}\} \\

Y_{2} &=\{X_{4},X_{5},X_{6},X_{7}\} \\

Y_{3} &=\{X_{5},X_{6},X_{7},X_{8}\} \\

Y_{4} &=\{X_{7},X_{8}\}

\end{align}

These are the same variables as before with the addition of $Y\_\{4\}=\backslash \{X\_\{7\},X\_\{8\}\backslash \}$. However, $I(Y\_\{1\};Y\_\{2\};Y\_\{3\};Y\_\{4\})$ in this case is actually equal to $+1$ bit, indicating less redundancy. This is correct in the sense that

:$$

\begin{align}

I(Y_{1};Y_{2};Y_{3};Y_{4}) &= I(Y_{1};Y_{2};Y_{3})-I(Y_{1};Y_{2};Y_{3}|Y_{4}) \\

&= 3-2 \\

&= 1

\end{align}

but it remains difficult to interpret.

• Jakulin and Bratko (2003b) provide a machine learning algorithm which uses interaction information.
• Killian, Kravitz and Gilson (2007) use mutual information expansion to extract entropy estimates from molecular simulations.{{Cite journal |last1=Killian |first1=Benjamin J. |last2=Yundenfreund Kravitz |first2=Joslyn |last3=Gilson |first3=Michael K. |date=2007-07-14 |title=Extraction of configurational entropy from molecular simulations via an expansion approximation |url=https://pubs.aip.org/aip/jcp/article/919602 |journal=The Journal of Chemical Physics |language=en |volume=127 |issue=2 |pages=024107 |doi=10.1063/1.2746329 |issn=0021-9606 |pmc=2707031 |pmid=17640119|bibcode=2007JChPh.127b4107K }}
• LeVine and Weinstein (2014) use interaction information and other N-body information measures to quantify allosteric couplings in molecular simulations.{{Cite arXiv |last1=LeVine |first1=Michael V. |last2=Perez-Aguilar |first2=Jose Manuel |last3=Weinstein |first3=Harel |date=2014-06-18 |title=N-body Information Theory (NbIT) Analysis of Rigid-Body Dynamics in Intracellular Loop 2 of the 5-HT2A Receptor |class=q-bio.BM |eprint=1406.4730 }}
• Moore et al. (2006), Chanda P, Zhang A, Brazeau D, Sucheston L, Freudenheim JL, Ambrosone C, Ramanathan M. (2007) and Chanda P, Sucheston L, Zhang A, Brazeau D, Freudenheim JL, Ambrosone C, Ramanathan M. (2008) demonstrate the use of interaction information for analyzing gene-gene and gene-environmental interactions associated with complex diseases.
• Pandey and Sarkar (2017) use interaction information in Cosmology to study the influence of large-scale environments on galaxy properties.
• A python package for computing all multivariate interaction or mutual informations, conditional mutual information, joint entropies, total correlations, information distance in a dataset of n variables is available .{{cite web|url=https://infotopo.readthedocs.io/en/latest/index.html|title=InfoTopo: Topological Information Data Analysis. Deep statistical unsupervised and supervised learning - File Exchange - Github|website=github.com/pierrebaudot/infotopopy/|access-date=26 September 2020}}

• Mutual information
• Total correlation
• Dual total correlation
• Partial Information Decomposition

{{Reflist}}

• {{cite journal | last1 = Baudot | first1 = P. | last2 = Bennequin | first2 = D. | year = 2015 | title = The homological nature of entropy | journal = Entropy | volume = 17 | issue = 5| pages = 1–66 | doi=10.3390/e17053253| bibcode = 2015Entrp..17.3253B | url = http://www.mis.mpg.de/preprints/2015/preprint2015_10.pdf | doi-access = free }}
• Bell, A J (2003), The co-information lattice [http://www.kecl.ntt.co.jp/icl/signal/ica2003/cdrom/data/0187.pdf]
• Fano, R M (1961), Transmission of Information: A Statistical Theory of Communications, MIT Press, Cambridge, MA.
• Garner W R (1962). Uncertainty and Structure as Psychological Concepts, JohnWiley & Sons, New York.
• {{cite journal | last1 = Han | first1 = T S | year = 1978 | title = Nonnegative entropy measures of multivariate symmetric correlations | journal = Information and Control | volume = 36 | issue = 2| pages = 133–156 | doi=10.1016/s0019-9958(78)90275-9| doi-access = free }}
• {{cite journal | last1 = Han | first1 = T S | year = 1980 | title = Multiple mutual information and multiple interactions in frequency data | journal = Information and Control | volume = 46 | pages = 26–45 | doi=10.1016/s0019-9958(80)90478-7| doi-access = free }}
• Hu Kuo Tin (1962), On the Amount of Information. Theory Probab. Appl.,7(4), 439-44. [https://web.archive.org/web/20150721210447/https://baudot.olympe.in/wp-content/uploads/2015/07/Hu-Kuo-Ting-196-On-the-amount-of-information-PRINCEPTS.pdf PDF]
• Jakulin A & Bratko I (2003a). Analyzing Attribute Dependencies, in N Lavra\quad{c}, D Gamberger, L Todorovski & H Blockeel, eds, Proceedings of the 7th European Conference on Principles and Practice of Knowledge Discovery in Databases, Springer, Cavtat-Dubrovnik, Croatia, pp. 229–240.
• Jakulin A & Bratko I (2003b). Quantifying and visualizing attribute interactions [https://arxiv.org/abs/cs/0308002v1].
• {{cite journal | last1 = Margolin | first1 = A | last2 = Wang | first2 = K | last3 = Califano | first3 = A | last4 = Nemenman | first4 = I | year = 2010 | title = Multivariate dependence and genetic networks inference | journal = IET Syst Biol | volume = 4 | issue = 6| pages = 428–440 | doi=10.1049/iet-syb.2010.0009| pmid = 21073241 | arxiv = 1001.1681 | s2cid = 14280921 }}
• {{cite journal | last1 = McGill | first1 = W J | year = 1954 | title = Multivariate information transmission | journal = Psychometrika | volume = 19 | issue = 2| pages = 97–116 | doi=10.1007/bf02289159| s2cid = 126431489 }}
• Moore JH, Gilbert JC, Tsai CT, Chiang FT, Holden T, Barney N, White BC (2006). A flexible computational framework for detecting, characterizing, and interpreting statistical patterns of epistasis in genetic studies of human disease susceptibility, Journal of Theoretical Biology 241, 252-261. [https://www.ncbi.nlm.nih.gov/pubmed/16457852?ordinalpos=19&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_DefaultReportPanel.Pubmed_RVDocSum]
• Nemenman I (2004). Information theory, multivariate dependence, and genetic network inference [https://arxiv.org/abs/q-bio.QM/0406015].
• Pearl, J (1988), Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, Morgan Kaufmann, San Mateo, CA.
• Tsujishita, T (1995), On triple mutual information, Advances in applied mathematics 16, 269-274.
• {{cite journal | last1 = Chanda | first1 = P | last2 = Zhang | first2 = A | last3 = Brazeau | first3 = D | last4 = Sucheston | first4 = L | last5 = Freudenheim | first5 = JL | last6 = Ambrosone | first6 = C | last7 = Ramanathan | first7 = M | year = 2007 | title = Information-theoretic metrics for visualizing gene-environment interactions | journal = American Journal of Human Genetics | volume = 81 | issue = 5| pages = 939–63 | doi = 10.1086/521878 | pmid = 17924337 | pmc=2265645}}
• {{cite journal | last1 = Chanda | first1 = P | last2 = Sucheston | first2 = L | last3 = Zhang | first3 = A | last4 = Brazeau | first4 = D | last5 = Freudenheim | first5 = JL | last6 = Ambrosone | first6 = C | last7 = Ramanathan | first7 = M | year = 2008 | title = AMBIENCE: a novel approach and efficient algorithm for identifying informative genetic and environmental associations with complex phenotypes | journal = Genetics | volume = 180 | issue = 2| pages = 1191–210 | doi = 10.1534/genetics.108.088542| pmid = 18780753| pmc=2567367}}
• {{cite journal | last1 = Killian | first1 = B J | last2 = Kravitz | first2 = J Y | last3 = Gilson | first3 = M K | year = 2007 | title = Extraction of configurational entropy from molecular simulations via an expansion approximation | journal = J. Chem. Phys. | volume = 127 | issue = 2| page = 024107 | doi=10.1063/1.2746329| pmid = 17640119 | pmc = 2707031 | bibcode = 2007JChPh.127b4107K }}
• LeVine MV, Weinstein H (2014), NbIT - A New Information Theory-Based Analysis of Allosteric Mechanisms Reveals Residues that Underlie Function in the Leucine Transporter LeuT. PLoS Computational Biology. [http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003603]
• {{cite journal | last1 = Pandey | first1 = Biswajit | last2 = Sarkar | first2 = Suman | year = 2017 | title = How much a galaxy knows about its large-scale environment?: An information theoretic perspective | journal = Monthly Notices of the Royal Astronomical Society Letters| volume = 467 | issue = 1| page = L6 | doi=10.1093/mnrasl/slw250| doi-access = free | arxiv = 1611.00283| bibcode = 2017MNRAS.467L...6P| s2cid = 119095496 }}
• https://www3.nd.edu/~jnl/ee80653/Fall2005/tutorials/sunil.pdf
• Yeung R W (1992). A new outlook on Shannon's information measures. in IEEE Transactions on Information Theory. [https://ieeexplore.ieee.org/document/79902]

Category:Information theory