Canonical correlation#Solution

Given two column vectors $X\; =\; (x\_1,\; \backslash dots,\; x\_n)^T$ and $Y\; =\; (y\_1,\; \backslash dots,\; y\_m)^T$ of random variables with finite second moments, one may define the cross-covariance $\backslash Sigma\; \_\{XY\}\; =\; \backslash operatorname\{cov\}(X,\; Y)$ to be the $n\; \backslash times\; m$ matrix whose $(i,\; j)$ entry is the covariance $\backslash operatorname\{cov\}(x\_i,\; y\_j)$. In practice, we would estimate the covariance matrix based on sampled data from $X$ and $Y$ (i.e. from a pair of data matrices).

Canonical-correlation analysis seeks a sequence of vectors $a\_k$ ($a\_k\; \backslash in\backslash mathbb\; R^n$) and $b\_k$ ($b\_k\; \backslash in\backslash mathbb\; R^m$) such that the random variables $a\_k^T\; X$ and $b\_k^T\; Y$ maximize the correlation $\backslash rho\; =\; \backslash operatorname\{corr\}(a\_k^T\; X,\; b\_k^T\; Y)$. The (scalar) random variables $U\; =\; a\_1^T\; X$ and $V\; =\; b\_1^T\; Y$ are the first pair of canonical variables. Then one seeks vectors maximizing the same correlation subject to the constraint that they are to be uncorrelated with the first pair of canonical variables; this gives the second pair of canonical variables. This procedure may be continued up to $\backslash min\backslash \{m,n\backslash \}$ times.

: $$(a\_k,b\_k)\; =\; \backslash underset\{a,b\}\backslash operatorname\{argmax\}\; \backslash operatorname\{corr\}(a^T\; X,\; b^T\; Y)\; \backslash quad\backslash text\{\; subject\; to\; \}\; \backslash operatorname\{cov\}(a^T\; X,\; a\_j^T\; X)\; =\; \backslash operatorname\{cov\}(b^T\; Y,\; b\_j^T\; Y)\; =\; 0\; \backslash text\{\; for\; \}\; j=1,\; \backslash dots,\; k-1$$

The sets of vectors $a\_k,\; b\_k$ are called canonical directions or weight vectors or simply weights. The 'dual' sets of vectors $\backslash Sigma\_\{XX\}a\_k,\; \backslash Sigma\_\{YY\}\; b\_k$ are called canonical loading vectors or simply loadings; these are often more straightforward to interpret than the weights.{{Cite journal |last1=Gu |first1=Fei |last2=Wu |first2=Hao |date=2018-04-01 |title=Simultaneous canonical correlation analysis with invariant canonical loadings |url=https://doi.org/10.1007/s41237-017-0042-8 |journal=Behaviormetrika |language=en |volume=45 |issue=1 |pages=111–132 |doi=10.1007/s41237-017-0042-8 |issn=1349-6964}}

Let $\backslash Sigma\; \_\{XY\}$ be the cross-covariance matrix for any pair of (vector-shaped) random variables $X$ and $Y$. The target function to maximize is

:$$

\rho = \frac{a^T \Sigma _{XY} b}{\sqrt{a^T \Sigma _{XX} a} \sqrt{b^T \Sigma _{YY} b}}.

The first step is to define a change of basis and define

:$$

c = \Sigma _{XX} ^{1/2} a,

:$$

d = \Sigma _{YY} ^{1/2} b,

where $\backslash Sigma\_\{XX\}^\{1/2\}$ and $\backslash Sigma\_\{YY\}^\{1/2\}$ can be obtained from the eigen-decomposition (or by diagonalization):

:$$

\Sigma _{XX} ^{1/2} = V_X D_X^{1/2} V_X^\top,\qquad V_X D_X V_X^\top = \Sigma_{XX},

and

:$$

\Sigma _{YY} ^{1/2} = V_Y D_Y^{1/2} V_Y^\top,\qquad V_Y D_Y V_Y^\top = \Sigma_{YY}.

Thus

:$$

\rho = \frac{c^T \Sigma _{XX} ^{-1/2} \Sigma _{XY} \Sigma _{YY} ^{-1/2} d}{\sqrt{c^Tc} \sqrt{d^Td}}.

By the Cauchy–Schwarz inequality, ...can someone check the this, particularly the term to the right of "(d) leq"?

:$$

\left(c^T \Sigma _{XX} ^{-1/2} \Sigma _{XY} \Sigma _{YY} ^{-1/2} \right) (d) \leq \left(c^T \Sigma _{XX} ^{-1/2} \Sigma _{XY} \Sigma _{YY} ^{-1/2} \Sigma _{YY} ^{-1/2} \Sigma _{YX} \Sigma _{XX} ^{-1/2} c \right)^{1/2} \left(d^T d \right)^{1/2},

:$$

\rho \leq \frac{\left(c^T \Sigma _{XX}^{-1/2} \Sigma _{XY} \Sigma _{YY}^{-1} \Sigma _{YX} \Sigma_{XX}^{-1/2} c \right)^{1/2}}{\left(c^T c \right)^{1/2}}.

There is equality if the vectors $d$ and $\backslash Sigma\_\{YY\}^\{-1/2\}\; \backslash Sigma\_\{YX\}\; \backslash Sigma\_\{XX\}^\{-1/2\}\; c$ are collinear. In addition, the maximum of correlation is attained if $c$ is the eigenvector with the maximum eigenvalue for the matrix $\backslash Sigma\_\{XX\}^\{-1/2\}\; \backslash Sigma\_\{XY\}\; \backslash Sigma\_\{YY\}^\{-1\}\; \backslash Sigma\_\{YX\}\; \backslash Sigma\_\{XX\}^\{-1/2\}$ (see Rayleigh quotient). The subsequent pairs are found by using eigenvalues of decreasing magnitudes. Orthogonality is guaranteed by the symmetry of the correlation matrices.

Another way of viewing this computation is that $c$ and $d$ are the left and right singular vectors of the correlation matrix of X and Y corresponding to the highest singular value.

The solution is therefore:

• $c$ is an eigenvector of $\backslash Sigma\_\{XX\}^\{-1/2\}\; \backslash Sigma\_\{XY\}\; \backslash Sigma\_\{YY\}^\{-1\}\; \backslash Sigma\_\{YX\}\; \backslash Sigma\_\{XX\}^\{-1/2\}$
• $d$ is proportional to $\backslash Sigma\; \_\{YY\}^\{-1/2\}\; \backslash Sigma\_\{YX\}\; \backslash Sigma\_\{XX\}^\{-1/2\}\; c$

Reciprocally, there is also:

• $d$ is an eigenvector of $\backslash Sigma\_\{YY\}^\{-1/2\}\; \backslash Sigma\_\{YX\}\; \backslash Sigma\_\{XX\}^\{-1\}\; \backslash Sigma\_\{XY\}\; \backslash Sigma\_\{YY\}^\{-1/2\}$
• $c$ is proportional to $\backslash Sigma\_\{XX\}^\{-1/2\}\; \backslash Sigma\_\{XY\}\; \backslash Sigma\_\{YY\}^\{-1/2\}\; d$

Reversing the change of coordinates, we have that

• $a$ is an eigenvector of $\backslash Sigma\_\{XX\}^\{-1\}\; \backslash Sigma\_\{XY\}\; \backslash Sigma\_\{YY\}^\{-1\}\; \backslash Sigma\_\{YX\}$,
• $b$ is proportional to $\backslash Sigma\_\{YY\}^\{-1\}\; \backslash Sigma\_\{YX\}\; a;$
• $b$ is an eigenvector of $\backslash Sigma\; \_\{YY\}^\{-1\}\; \backslash Sigma\_\{YX\}\; \backslash Sigma\_\{XX\}^\{-1\}\; \backslash Sigma\_\{XY\},$
• $a$ is proportional to $\backslash Sigma\_\{XX\}^\{-1\}\; \backslash Sigma\_\{XY\}\; b$.

The canonical variables are defined by:

:$U\; =\; c^T\; \backslash Sigma\_\{XX\}^\{-1/2\}\; X\; =\; a^T\; X$

:$V\; =\; d^T\; \backslash Sigma\_\{YY\}^\{-1/2\}\; Y\; =\; b^T\; Y$

CCA can be computed using singular value decomposition on a correlation matrix.{{Cite journal | last1 = Hsu | first1 = D. | last2 = Kakade | first2 = S. M. | last3 = Zhang | first3 = T. | doi = 10.1016/j.jcss.2011.12.025 | title = A spectral algorithm for learning Hidden Markov Models | journal = Journal of Computer and System Sciences | volume = 78 | issue = 5 | pages = 1460 | year = 2012 | url = http://www.cs.mcgill.ca/~colt2009/papers/011.pdf| arxiv = 0811.4413| s2cid = 220740158 }} It is available as a function in{{Cite journal | last1 = Huang | first1 = S. Y. | last2 = Lee | first2 = M. H. | last3 = Hsiao | first3 = C. K. | doi = 10.1016/j.jspi.2008.10.011 | title = Nonlinear measures of association with kernel canonical correlation analysis and applications | journal = Journal of Statistical Planning and Inference | volume = 139 | issue = 7 | pages = 2162 | year = 2009 | url = http://www.stat.sinica.edu.tw/syhuang/papersdownload/KCCA-080906.pdf | access-date = 2015-09-04 | archive-date = 2017-03-13 | archive-url = https://web.archive.org/web/20170313203427/http://www.stat.sinica.edu.tw/syhuang/papersdownload/KCCA-080906.pdf | url-status = dead }}

• MATLAB as [http://www.mathworks.co.uk/help/stats/canoncorr.html canoncorr] ([https://sourceforge.net/p/octave/statistics/ci/default/tree/inst/canoncorr.m also] in Octave)
• R as the standard function [http://stat.ethz.ch/R-manual/R-devel/library/stats/html/cancor.html cancor] and several other packages, including [https://cran.r-project.org/package=candisc candisc], [https://cran.r-project.org/web/packages/CCA/index.html CCA] and [https://cran.r-project.org/web/packages/vegan/index.html vegan]. [https://cran.r-project.org/web/packages/CCP/index.html CCP] for statistical hypothesis testing in canonical correlation analysis.
• SAS as [https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/statug/statug_cancorr_toc.htm proc cancorr]
• Python in the library scikit-learn, as [http://scikit-learn.org/stable/modules/cross_decomposition.html cross decomposition] and in statsmodels, as [https://devdocs.io/statsmodels/generated/statsmodels.multivariate.cancorr.cancorr CanCorr]. The CCA-Zoo library {{Cite journal |last1=Chapman |first1=James |last2=Wang |first2=Hao-Ting |date=2021-12-18 |title=CCA-Zoo: A collection of Regularized, Deep Learning based, Kernel, and Probabilistic CCA methods in a scikit-learn style framework |journal=Journal of Open Source Software |language=en |volume=6 |issue=68 |pages=3823 |doi=10.21105/joss.03823 |issn=2475-9066|doi-access=free |bibcode=2021JOSS....6.3823C }} implements CCA extensions, such as probabilistic CCA, sparse CCA, multi-view CCA, and deep CCA.
• SPSS as macro CanCorr shipped with the main software
• Julia (programming language) in the [https://github.com/JuliaStats/MultivariateStats.jl MultivariateStats.jl] package.

CCA computation using singular value decomposition on a correlation matrix is related to the cosine of the angles between flats. The cosine function is ill-conditioned for small angles, leading to very inaccurate computation of highly correlated principal vectors in finite precision computer arithmetic. To fix this trouble, alternative algorithms{{Citation

| last1 = Knyazev

| first1 = A.V.

| last2 = Argentati

| first2 = M.E.

| title = Principal Angles between Subspaces in an A-Based Scalar Product: Algorithms and Perturbation Estimates

| journal = SIAM Journal on Scientific Computing

| volume = 23

| issue = 6

| pages = 2009–2041

| year = 2002

| doi = 10.1137/S1064827500377332

| bibcode = 2002SJSC...23.2008K

| citeseerx = 10.1.1.73.2914

}} are available in

• SciPy as [https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.subspace_angles.html linear-algebra function subspace_angles]
• MATLAB as [https://www.mathworks.com/matlabcentral/fileexchange/55-subspacea-m FileExchange function subspacea]

Each row can be tested for significance with the following method. Since the correlations are sorted, saying that row $i$ is zero implies all further correlations are also zero. If we have $p$ independent observations in a sample and $\backslash widehat\{\backslash rho\}\_i$ is the estimated correlation for $i\; =\; 1,\backslash dots,\; \backslash min\backslash \{m,n\backslash \}$. For the $i$th row, the test statistic is:

:$\backslash chi^2\; =\; -\; \backslash left(\; p\; -\; 1\; -\; \backslash frac\{1\}\{2\}(m\; +\; n\; +\; 1)\backslash right)\; \backslash ln\; \backslash prod\_\{j\; =\; i\}^\{\backslash min\backslash \{m,n\backslash \}\}\; (1\; -\; \backslash widehat\{\backslash rho\}\_j^2),$

which is asymptotically distributed as a chi-squared with $(m\; -\; i\; +\; 1)(n\; -\; i\; +\; 1)$ degrees of freedom for large $p$.{{Cite book

| author = Kanti V. Mardia, J. T. Kent and J. M. Bibby

| title = Multivariate Analysis

| year = 1979

| publisher = Academic Press

}} Since all the correlations from $\backslash min\backslash \{m,n\backslash \}$ to $p$ are logically zero (and estimated that way also) the product for the terms after this point is irrelevant.

Note that in the small sample size limit with then we are guaranteed that the top $m\; +\; n\; -\; p$ correlations will be identically 1 and hence the test is meaningless.Yang Song, Peter J. Schreier, David Ram´ırez, and Tanuj Hasija Canonical correlation analysis of high-dimensional data with very small sample support {{ArXiv|1604.02047}}

A typical use for canonical correlation in the experimental context is to take two sets of variables and see what is common among the two sets.{{cite book|last= Sieranoja|first=S.|author2=Sahidullah, Md| author3=Kinnunen, T.| author4= Komulainen, J.| author5= Hadid, A.|title=2018 IEEE 3rd International Conference on Signal and Image Processing (ICSIP) |chapter=Audiovisual Synchrony Detection with Optimized Audio Features |date=July 2018|pages=377–381 |doi=10.1109/SIPROCESS.2018.8600424 |isbn=978-1-5386-6396-7 |s2cid=51682024 |url=http://urn.fi/urn:nbn:fi-fe2020041415345 |chapter-url=http://cs.joensuu.fi/pages/tkinnu/webpage/pdf/audiovisual_synchrony_2018.pdf}} For example, in psychological testing, one could take two well established multidimensional personality tests such as the Minnesota Multiphasic Personality Inventory (MMPI-2) and the NEO. By seeing how the MMPI-2 factors relate to the NEO factors, one could gain insight into what dimensions were common between the tests and how much variance was shared. For example, one might find that an extraversion or neuroticism dimension accounted for a substantial amount of shared variance between the two tests.

One can also use canonical-correlation analysis to produce a model equation which relates two sets of variables, for example a set of performance measures and a set of explanatory variables, or a set of outputs and set of inputs. Constraint restrictions can be imposed on such a model to ensure it reflects theoretical requirements or intuitively obvious conditions. This type of model is known as a maximum correlation model.{{Cite journal | last1 = Tofallis | first1 = C. | title = Model Building with Multiple Dependent Variables and Constraints | doi = 10.1111/1467-9884.00195 | journal = Journal of the Royal Statistical Society, Series D | volume = 48 | issue = 3 | pages = 371–378 | year = 1999 | arxiv = 1109.0725| s2cid = 8942357 }}

Visualization of the results of canonical correlation is usually through bar plots of the coefficients of the two sets of variables for the pairs of canonical variates showing significant correlation. Some authors suggest that they are best visualized by plotting them as heliographs, a circular format with ray like bars, with each half representing the two sets of variables.{{Cite book | last1 = Degani | first1 = A. | last2 = Shafto | first2 = M. | last3 = Olson | first3 = L. | doi = 10.1007/11783183_11 | chapter = Canonical Correlation Analysis: Use of Composite Heliographs for Representing Multiple Patterns | title = Diagrammatic Representation and Inference | series = Lecture Notes in Computer Science | volume = 4045 | pages = 93 | year = 2006 | isbn = 978-3-540-35623-3 | chapter-url = http://ti.arc.nasa.gov/m/profile/adegani/Composite_Heliographs.pdf| citeseerx = 10.1.1.538.5217 }}

Let $X\; =\; x\_1$ with zero expected value, i.e., $\backslash operatorname\{E\}(X)=0$.

1. If $Y\; =\; X$, i.e., $X$ and $Y$ are perfectly correlated, then, e.g., $a=1$ and $b=1$, so that the first (and only in this example) pair of canonical variables is $U\; =\; X$ and $V\; =\; Y\; =X$.
2. If $Y\; =\; -X$, i.e., $X$ and $Y$ are perfectly anticorrelated, then, e.g., $a=1$ and $b=-1$, so that the first (and only in this example) pair of canonical variables is $U\; =\; X$ and $V\; =\; -Y\; =X$.

We notice that in both cases $U\; =V$, which illustrates that the canonical-correlation analysis treats correlated and anticorrelated variables similarly.

Assuming that $X\; =\; (x\_1,\; \backslash dots,\; x\_n)^T$ and $Y\; =\; (y\_1,\; \backslash dots,\; y\_m)^T$ have zero expected values, i.e., $\backslash operatorname\{E\}(X)=\backslash operatorname\{E\}(Y)=0$, their covariance matrices $\backslash Sigma\; \_\{XX\}\; =\backslash operatorname\{Cov\}(X,X)\; =\; \backslash operatorname\{E\}[X\; X^T]$ and $\backslash Sigma\; \_\{YY\}\; =\backslash operatorname\{Cov\}(Y,Y)\; =\; \backslash operatorname\{E\}[Y\; Y^T]$ can be viewed as Gram matrices in an inner product for the entries of $X$ and $Y$, correspondingly. In this interpretation, the random variables, entries $x\_i$ of $X$ and $y\_j$ of $Y$ are treated as elements of a vector space with an inner product given by the covariance $\backslash operatorname\{cov\}(x\_i,\; y\_j)$; see Covariance#Relationship to inner products.

The definition of the canonical variables $U$ and $V$ is then equivalent to the definition of principal vectors for the pair of subspaces spanned by the entries of $X$ and $Y$ with respect to this inner product. The canonical correlations $\backslash operatorname\{corr\}(U,V)$ is equal to the cosine of principal angles.

CCA can also be viewed as a special whitening transformation where the random vectors $X$ and $Y$ are simultaneously transformed in such a way that the cross-correlation between the whitened vectors $X^\{CCA\}$ and $Y^\{CCA\}$ is diagonal.{{cite journal | last1 = Jendoubi | first1 = T. | last2 = Strimmer | first2 = K. | title = A whitening approach to probabilistic canonical correlation analysis for omics data integration | journal = BMC Bioinformatics | volume = 20 | issue = 1 | pages = 15 | year = 2018 | arxiv = 1802.03490 | doi = 10.1186/s12859-018-2572-9 | pmid = 30626338 | pmc = 6327589 | doi-access = free }}

The canonical correlations are then interpreted as regression coefficients linking $X^\{CCA\}$ and $Y^\{CCA\}$ and may also be negative. The regression view of CCA also provides a way to construct a latent variable probabilistic generative model for CCA, with uncorrelated hidden variables representing shared and non-shared variability.{{cite journal |last1=Jendoubi |first1=Takoua |last2=Strimmer |first2=Korbinian |title=A whitening approach to probabilistic canonical correlation analysis for omics data integration |journal=BMC Bioinformatics |date=9 January 2019 |volume=20 |issue=1 |pages=15 |doi=10.1186/s12859-018-2572-9 |doi-access=free |pmid=30626338 |pmc=6327589 |language=en |issn=1471-2105}}

• Generalized canonical correlation
• RV coefficient
• Angles between flats
• Principal component analysis
• Linear discriminant analysis
• Regularized canonical correlation analysis
• Singular value decomposition
• Partial least squares regression

{{Reflist|30em}}

• [https://github.com/mhaghighat/dcaFuse Discriminant Correlation Analysis (DCA)]{{Cite journal |doi = 10.1109/TIFS.2016.2569061|title = Discriminant Correlation Analysis: Real-Time Feature Level Fusion for Multimodal Biometric Recognition|journal = IEEE Transactions on Information Forensics and Security|volume = 11|issue = 9|pages = 1984–1996|year = 2016|last1 = Haghighat|first1 = Mohammad|last2 = Abdel-Mottaleb|first2 = Mohamed|last3 = Alhalabi|first3 = Wadee| s2cid=15624506 |url = https://zenodo.org/record/889881}} (MATLAB)
• {{Cite journal | last1 = Hardoon | first1 = D. R. | last2 = Szedmak | first2 = S. | last3 = Shawe-Taylor | first3 = J. | doi = 10.1162/0899766042321814 | title = Canonical Correlation Analysis: An Overview with Application to Learning Methods | journal = Neural Computation | volume = 16 | issue = 12 | pages = 2639–2664 | year = 2004 | pmid = 15516276| citeseerx = 10.1.1.14.6452 | s2cid = 202473 }}
• [http://mpra.ub.uni-muenchen.de/12796/ A note on the ordinal canonical-correlation analysis of two sets of ranking scores] (Also provides a FORTRAN program)- in Journal of Quantitative Economics 7(2), 2009, pp. 173–199
• [http://ssrn.com/abstract=1331886 Representation-Constrained Canonical Correlation Analysis: A Hybridization of Canonical Correlation and Principal Component Analyses] (Also provides a FORTRAN program)- in Journal of Applied Economic Sciences 4(1), 2009, pp. 115–124

{{statistics|analysis|collapsed}}

Category:Covariance and correlation