partial information decomposition

Information theory can quantify the amount of information a single source variable $X\_1$ has about a target variable $Y$ via the mutual information $I(X\_1;Y)$. If we now consider a second source variable $X\_2$, classical information theory can only describe the mutual information of the joint variable $\backslash \{X\_1,X\_2\backslash \}$ with $Y$, given by $I(X\_1,X\_2;Y)$. In general however, it would be interesting to know how exactly the individual variables $X\_1$ and $X\_2$ and their interactions relate to $Y$.

Consider that we are given two source variables $X\_1,\; X\_2\; \backslash in\; \backslash \{0,1\backslash \}$ and a target variable $Y=XOR(X\_1,X\_2)$. In this case the total mutual information $I(X\_1,X\_2;Y)=1$, while the individual mutual information $I(X\_1;Y)=I(X\_2;Y)=0$. That is, there is synergistic information arising from the interaction of $X\_1,X\_2$ about $Y$, which cannot be easily captured with classical information theoretic quantities.

Partial information decomposition further decomposes the mutual information between the source variables $\backslash \{X\_1,X\_2\backslash \}$ with the target variable $Y$ as

$I(X\_1,X\_2;Y)=\backslash text\{Unq\}(X\_1;Y\; \backslash setminus\; X\_2)\; +\; \backslash text\{Unq\}(X\_2;Y\; \backslash setminus\; X\_1)\; +\; \backslash text\{Syn\}(X\_1,X\_2;Y)\; +\; \backslash text\{Red\}(X\_1,X\_2;Y)$

Here the individual information atoms are defined as

• $\backslash text\{Unq\}(X\_1;Y\; \backslash setminus\; X\_2)$ is the unique information that $X\_1$ has about $Y$, which is not in $X\_2$
• $\backslash text\{Syn\}(X\_1,X\_2;Y)$ is the synergistic information that is in the interaction of $X\_1$ and $X\_2$ about $Y$
• $\backslash text\{Red\}(X\_1,X\_2;Y)$ is the redundant information that is in both $X\_1$ or $X\_2$ about $Y$

There is, thus far, no universal agreement on how these terms should be defined, with different approaches that decompose information into redundant, unique, and synergistic components appearing in the literature.{{Cite journal | vauthors = Quax R, Har-Shemesh O, Sloot PM |date=February 2017 |title=Quantifying Synergistic Information Using Intermediate Stochastic Variables |journal=Entropy |language=en |volume=19 |issue=2 |pages=85 |doi=10.3390/e19020085 |issn=1099-4300|doi-access=free |arxiv=1602.01265 }}{{Cite journal | vauthors = Rosas FE, Mediano PA, Rassouli B, Barrett AB |date=2020-12-04 |title=An operational information decomposition via synergistic disclosure |journal=Journal of Physics A: Mathematical and Theoretical |volume=53 |issue=48 |pages=485001 |doi=10.1088/1751-8121/abb723 |arxiv=2001.10387 |bibcode=2020JPhA...53V5001R |s2cid=210932609 |issn=1751-8113}}{{cite journal | vauthors = Kolchinsky A | title = A Novel Approach to the Partial Information Decomposition | journal = Entropy | volume = 24 | issue = 3 | pages = 403 | date = March 2022 | pmid = 35327914 | doi = 10.3390/e24030403 | pmc = 8947370 | arxiv = 1908.08642 | bibcode = 2022Entrp..24..403K | doi-access = free }}

Despite the lack of universal agreement, partial information decomposition has been applied to diverse fields, including climatology,{{Cite journal | vauthors = Goodwell AE, Jiang P, Ruddell BL, Kumar P |date= February 2020 |title=Debates—Does Information Theory Provide a New Paradigm for Earth Science? Causality, Interaction, and Feedback |journal=Water Resources Research |language=en |volume=56 |issue=2 |doi=10.1029/2019WR024940 |bibcode= 2020WRR....5624940G |s2cid= 216201598 |issn=0043-1397|doi-access=free }} neuroscience{{cite journal | vauthors = Newman EL, Varley TF, Parakkattu VK, Sherrill SP, Beggs JM | title = Revealing the Dynamics of Neural Information Processing with Multivariate Information Decomposition | journal = Entropy | volume = 24 | issue = 7 | pages = 930 | date = July 2022 | pmid = 35885153 | doi = 10.3390/e24070930 | pmc = 9319160 | bibcode = 2022Entrp..24..930N | doi-access = free }}{{cite journal | vauthors = Luppi AI, Mediano PA, Rosas FE, Holland N, Fryer TD, O'Brien JT, Rowe JB, Menon DK, Bor D, Stamatakis EA | display-authors = 6 | title = A synergistic core for human brain evolution and cognition | journal = Nature Neuroscience | volume = 25 | issue = 6 | pages = 771–782 | date = June 2022 | pmid = 35618951 | doi = 10.1038/s41593-022-01070-0 | s2cid = 249096746 | pmc = 7614771 }}{{cite journal | vauthors = Wibral M, Priesemann V, Kay JW, Lizier JT, Phillips WA | title = Partial information decomposition as a unified approach to the specification of neural goal functions | journal = Brain and Cognition | volume = 112 | pages = 25–38 | date = March 2017 | pmid = 26475739 | doi = 10.1016/j.bandc.2015.09.004 | series = Perspectives on Human Probabilistic Inferences and the 'Bayesian Brain' | s2cid = 4394452 | doi-access = free | arxiv = 1510.00831 }} sociology,{{Cite journal | vauthors = Varley TF, Kaminski P |date= October 2022 |title=Untangling Synergistic Effects of Intersecting Social Identities with Partial Information Decomposition |journal=Entropy |language=en |volume=24 |issue=10 |pages=1387 |doi=10.3390/e24101387 |pmid= 37420406 |pmc= 9611752 |bibcode= 2022Entrp..24.1387V |issn=1099-4300|doi-access= free }} and machine learning{{Cite journal | vauthors = Tax TM, Mediano PA, Shanahan M |date= September 2017 |title=The Partial Information Decomposition of Generative Neural Network Models |journal=Entropy |language=en |volume=19 |issue=9 |pages=474 |doi=10.3390/e19090474 |bibcode= 2017Entrp..19..474T |issn=1099-4300|doi-access= free |hdl=10044/1/50586 |hdl-access=free }} Partial information decomposition has also been proposed as a possible foundation on which to build a mathematically robust definition of emergence in complex systems{{cite journal | vauthors = Mediano PA, Rosas FE, Luppi AI, Jensen HJ, Seth AK, Barrett AB, Carhart-Harris RL, Bor D | display-authors = 6 | title = Greater than the parts: a review of the information decomposition approach to causal emergence | journal = Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences | volume = 380 | issue = 2227 | pages = 20210246 | date = July 2022 | pmid = 35599558 | pmc = 9125226 | doi = 10.1098/rsta.2021.0246 }} and may be relevant to formal theories of consciousness.{{cite journal | vauthors = Luppi AI, Mediano PA, Rosas FE, Harrison DJ, Carhart-Harris RL, Bor D, Stamatakis EA | title = What it is like to be a bit: an integrated information decomposition account of emergent mental phenomena | journal = Neuroscience of Consciousness | volume = 2021 | issue = 2 | pages = niab027 | date = 2021 | pmid = 34804593 | pmc = 8600547 | doi = 10.1093/nc/niab027 }}

• Mutual information
• Total correlation
• Dual total correlation
• Interaction information

{{Reflist}}

Category:Information theory