Dual total correlation

Image:VennInfo3Var.svg of information theoretic measures for three variables x, y, and z. The dual total correlation is represented by the union of the three mutual informations and is shown in the diagram by the yellow, magenta, cyan, and gray regions.]]

For a set of n random variables $\backslash \{X\_1,\backslash ldots,\; X\_n\backslash \}$, the dual total correlation $D(X\_1,\backslash ldots,\; X\_n)$ is given by

:$D(X\_1,\backslash ldots,\; X\_n)\; =\; H\backslash left(\; X\_1,\; \backslash ldots,\; X\_n\; \backslash right)\; -\; \backslash sum\_\{i=1\}^n\; H\backslash left(\; X\_i\; \backslash mid\; X\_1,\; \backslash ldots,\; X\_\{i-1\},\; X\_\{i+1\},\; \backslash ldots,\; X\_n\; \backslash right)\; ,$

where $H(X\_\{1\},\backslash ldots,\; X\_\{n\})$ is the joint entropy of the variable set $\backslash \{X\_\{1\},\backslash ldots,\; X\_\{n\}\backslash \}$ and $H(X\_i\; \backslash mid\; \backslash cdots\; )$ is the conditional entropy of variable $X\_\{i\}$, given the rest.

The dual total correlation normalized between [0,1] is simply the dual total correlation divided by its maximum value $H(X\_\{1\},\; \backslash ldots,\; X\_\{n\})$,

:$ND(X\_1,\backslash ldots,\; X\_n)\; =\; \backslash frac\{D(X\_1,\backslash ldots,\; X\_n)\}\{H(X\_1,\backslash ldots,\; X\_n)\}\; .$

Dual total correlation is non-negative and bounded above by the joint entropy $H(X\_1,\; \backslash ldots,\; X\_n)$.

:$0\; \backslash leq\; D(X\_1,\; \backslash ldots,\; X\_n)\; \backslash leq\; H(X\_1,\; \backslash ldots,\; X\_n)\; .$

Secondly, Dual total correlation has a close relationship with total correlation, $C(X\_1,\; \backslash ldots,\; X\_n)$, and can be written in terms of differences between the total correlation of the whole, and all subsets of size $N-1$:{{cite journal |last1=Varley |first1=Thomas F. |last2=Pope |first2=Maria |last3=Faskowitz |first3=Joshua |last4=Sporns |first4=Olaf |title=Multivariate information theory uncovers synergistic subsystems of the human cerebral cortex |journal=Communications Biology |date=24 April 2023 |volume=6 |issue=1 |page=451 |doi=10.1038/s42003-023-04843-w|doi-access=free |pmid=37095282 |pmc=10125999 }}

:$D(\backslash textbf\{X\})\; =\; (N-1)C(\backslash textbf\{X\})\; -\; \backslash sum\_\{i=1\}^\{N\}\; C(\backslash textbf\{X\}^\{-i\})$

where $\backslash textbf\{X\}\; =\; \backslash \{X\_1,\backslash ldots,\; X\_n\backslash \}$ and $\backslash textbf\{X\}^\{-i\}\; =\; \backslash \{X\_1,\backslash ldots,\; X\_\{i-1\},\; X\_\{i+1\},\backslash ldots,\; X\_n\backslash \}$

Furthermore, the total correlation and dual total correlation are related by the following bounds:

:$\backslash frac\{C(X\_1,\; \backslash ldots,\; X\_n)\}\{n-1\}\; \backslash leq\; D(X\_1,\; \backslash ldots,\; X\_n)\; \backslash leq\; (n-1)\; \backslash ;\; C(X\_1,\; \backslash ldots,\; X\_n)\; .$

Finally, the difference between the total correlation and the dual total correlation defines a novel measure of higher-order information-sharing: the O-information:{{cite journal |last1=Rosas |first1=Fernando E. |last2=Mediano |first2=Pedro A. M. |last3=Gastpar |first3=Michael |last4=Jensen |first4=Henrik J. |title=Quantifying high-order interdependencies via multivariate extensions of the mutual information |journal=Physical Review E |date=13 September 2019 |volume=100 |issue=3 |page=032305 |doi=10.1103/PhysRevE.100.032305|pmid=31640038 |arxiv=1902.11239 |bibcode=2019PhRvE.100c2305R }}

:$\backslash Omega(\backslash textbf\{X\})\; =\; C(\backslash textbf\{X\})\; -\; D(\backslash textbf\{X\})$.

The O-information (first introduced as the "enigmatic information" by James and Crutchfield{{cite journal |last1=James |first1=Ryan G. |last2=Ellison |first2=Christopher J. |last3=Crutchfield |first3=James P. |title=Anatomy of a bit: Information in a time series observation |journal=Chaos: An Interdisciplinary Journal of Nonlinear Science |date=1 September 2011 |volume=21 |issue=3 |page=037109 |doi=10.1063/1.3637494|pmid=21974672 |arxiv=1105.2988 |bibcode=2011Chaos..21c7109J }} is a signed measure that quantifies the extent to which the information in a multivariate random variable is dominated by synergistic interactions (in which case ) or redundant interactions (in which case $\backslash Omega(\backslash textbf\{X\})\; >\; 0$.

Han (1978) originally defined the dual total correlation as,

: $$

\begin{align}

& D(X_1,\ldots, X_n) \\[10pt]

\equiv {} & \left[ \sum_{i=1}^n H(X_1, \ldots, X_{i-1}, X_{i+1}, \ldots, X_n ) \right] - (n-1) \; H(X_1, \ldots, X_n) \; .

\end{align}

However Abdallah and Plumbley (2010) showed its equivalence to the easier-to-understand form of the joint entropy minus the sum of conditional entropies via the following:

: $$

\begin{align}

& D(X_1,\ldots, X_n) \\[10pt]

\equiv {} & \left[ \sum_{i=1}^n H(X_1, \ldots, X_{i-1}, X_{i+1}, \ldots, X_n ) \right] - (n-1) \; H(X_1, \ldots, X_n) \\

= {} & \left[ \sum_{i=1}^n H(X_1, \ldots, X_{i-1}, X_{i+1}, \ldots, X_n ) \right] + (1-n) \; H(X_1, \ldots, X_n) \\

= {} & H(X_1, \ldots, X_n) + \left[ \sum_{i=1}^n H(X_1, \ldots, X_{i-1}, X_{i+1}, \ldots, X_n ) - H(X_1, \ldots, X_n) \right] \\

= {} & H\left( X_1, \ldots, X_n \right) - \sum_{i=1}^n H\left( X_i \mid X_1, \ldots, X_{i-1}, X_{i+1}, \ldots, X_n \right)\; .

\end{align}

• Interaction information
• Mutual information
• Total correlation

{{reflist}}

• {{cite journal |doi=10.1016/S0019-9958(78)91063-X|title=Polymatroidal dependence structure of a set of random variables|year=1978|last1=Fujishige|first1=Satoru|journal=Information and Control|volume=39|pages=55–72|doi-access=free}}
• {{cite journal |doi=10.1038/s42003-023-04843-w|title=Multivariate information theory uncovers synergistic subsystems of the human cerebral cortex|year=2023|last1=Varley|first1=Thomas|last2=Pope|first2=Maria|last3=Faskowitz|first3=Joshua|last4=Sporns|first4=Olaf|journal= Communications Biology|volume=6 |page=451 |doi-access=free|pmid=37095282 |pmc=10125999}}

Category:Information theory

Category:Probability theory

Category:Covariance and correlation