Functional regression

Functional linear models (FLMs) are an extension of linear models (LMs). A linear model with scalar response $Y\backslash in\backslash mathbb\{R\}$ and scalar covariates $X\backslash in\backslash mathbb\{R\}^p$ can be written as

{{NumBlk|::|$$Y\; =\; \backslash beta\_0\; +\; \backslash langle\; X,\backslash beta\backslash rangle\; +\; \backslash varepsilon,$$|{{EquationRef|1}}}}

where $\backslash langle\backslash cdot,\backslash cdot\backslash rangle$ denotes the inner product in Euclidean space, $\backslash beta\_0\backslash in\backslash mathbb\{R\}$ and $\backslash beta\backslash in\backslash mathbb\{R\}^p$ denote the regression coefficients, and $\backslash varepsilon$ is a random error with mean zero and finite variance. FLMs can be divided into two types based on the responses.

Functional linear models with scalar responses can be obtained by replacing the scalar covariates $X$ and the coefficient vector $\backslash beta$ in model ({{EquationNote|1}}) by a centered functional covariate $X^c(\backslash cdot)\; =\; X(\backslash cdot)\; -\; \backslash mathbb\{E\}(X(\backslash cdot))$ and a coefficient function $\backslash beta\; =\; \backslash beta(\backslash cdot)$ with domain $\backslash mathcal\{T\}$, respectively, and replacing the inner product in Euclidean space by that in Hilbert space $L^2$,

{{NumBlk|::|$$Y\; =\; \backslash beta\_0\; +\; \backslash langle\; X^c,\; \backslash beta\backslash rangle\; +\backslash varepsilon\; =\; \backslash beta\_0\; +\; \backslash int\_\backslash mathcal\{T\}\; X^c(t)\backslash beta(t)\backslash ,dt\; +\; \backslash varepsilon,$$|{{EquationRef|2}}}}

where $\backslash langle\; \backslash cdot,\; \backslash cdot\; \backslash rangle$ here denotes the inner product in $L^2$. One approach to estimating $\backslash beta\_0$ and $\backslash beta(\backslash cdot)$ is to expand the centered covariate $X^c(\backslash cdot)$ and the coefficient function $\backslash beta(\backslash cdot)$ in the same functional basis, for example, B-spline basis or the eigenbasis used in the Karhunen–Loève expansion. Suppose $\backslash \{\backslash phi\_k\backslash \}\_\{k=1\}^\backslash infty$ is an orthonormal basis of $L^2$. Expanding $X^c$ and $\backslash beta$ in this basis, $X^c(\backslash cdot)\; =\; \backslash sum\_\{k=1\}^\backslash infty\; x\_k\; \backslash phi\_k(\backslash cdot)$, $\backslash beta(\backslash cdot)\; =\; \backslash sum\_\{k=1\}^\backslash infty\; \backslash beta\_k\; \backslash phi\_k(\backslash cdot)$, model ({{EquationNote|2}}) becomes

$$Y\; =\; \backslash beta\_0\; +\; \backslash sum\_\{k=1\}^\backslash infty\; \backslash beta\_k\; x\_k\; +\backslash varepsilon.$$

For implementation, regularization is needed and can be done through truncation, $L^2$ penalization or $L^1$ penalization.{{cite journal|doi=10.1146/annurev-statistics-010814-020413|title=Functional Regression|year=2015|last1=Morris|first1=Jeffrey S.|journal=Annual Review of Statistics and Its Application|volume=2|issue=1|pages=321–359|arxiv=1406.4068|bibcode=2015AnRSA...2..321M|s2cid=18637009}} In addition, a reproducing kernel Hilbert space (RKHS) approach can also be used to estimate $\backslash beta\_0$ and $\backslash beta(\backslash cdot)$ in model ({{EquationNote|2}})Yuan and Cai (2010). "A reproducing kernel Hilbert space approach to functional linear regression". The Annals of Statistics. 38 (6):3412–3444. doi:[http://doi.org/10.1214/09-AOS772 10.1214/09-AOS772].

Adding multiple functional and scalar covariates, model ({{EquationNote|2}}) can be extended to

{{NumBlk|::|$$Y\; =\; \backslash sum\_\{k=1\}^q\; Z\_k\backslash alpha\_k\; +\; \backslash sum\_\{j=1\}^p\; \backslash int\_\{\backslash mathcal\{T\}\_j\}\; X\_j^c(t)\; \backslash beta\_j(t)\; \backslash ,dt\; +\; \backslash varepsilon,$$|{{EquationRef|3}}}}

where $Z\_1,\backslash ldots,Z\_q$ are scalar covariates with $Z\_1=1$, $\backslash alpha\_1,\backslash ldots,\backslash alpha\_q$ are regression coefficients for $Z\_1,\backslash ldots,Z\_q$, respectively, $X^c\_j$ is a centered functional covariate given by $X\_j^c(\backslash cdot)\; =\; X\_j(\backslash cdot)\; -\; \backslash mathbb\{E\}(X\_j(\backslash cdot))$, $\backslash beta\_j$ is regression coefficient function for $X\_j^c(\backslash cdot)$, and $\backslash mathcal\{T\}\_j$ is the domain of $X\_j$ and $\backslash beta\_j$, for $j=1,\backslash ldots,p$. However, due to the parametric component $\backslash alpha$, the estimation methods for model ({{EquationNote|2}}) cannot be used in this case{{cite journal|doi=10.1146/annurev-statistics-041715-033624|title=Functional Data Analysis|year=2016|last1=Wang|first1=Jane-Ling|last2=Chiou|first2=Jeng-Min|last3=Müller|first3=Hans-Georg|journal=Annual Review of Statistics and Its Application|volume=3|issue=1|pages=257–295|bibcode=2016AnRSA...3..257W|url=https://zenodo.org/record/895750|doi-access=free}} and alternative estimation methods for model ({{EquationNote|3}}) are available.{{Cite journal |last=Kong |first=Dehan |last2=Xue |first2=Kaijie |last3=Yao |first3=Fang |last4=Zhang |first4=Hao H. |date= |title=Partially functional linear regression in high dimensions |url=https://academic.oup.com/biomet/article-lookup/doi/10.1093/biomet/asv062 |journal=Biometrika |language=en |volume=103 |issue=1 |pages=147–159 |doi=10.1093/biomet/asv062 |issn=0006-3444}}{{Cite journal |last=Hu |first=Z. |date=2004-06-01 |title=Profile-kernel versus backfitting in the partially linear models for longitudinal/clustered data |url=https://academic.oup.com/biomet/article-lookup/doi/10.1093/biomet/91.2.251 |journal=Biometrika |language=en |volume=91 |issue=2 |pages=251–262 |doi=10.1093/biomet/91.2.251 |issn=0006-3444|url-access=subscription }}

For a functional response $Y(\backslash cdot)$ with domain $\backslash mathcal\{T\}$ and a functional covariate $X(\backslash cdot)$ with domain $\backslash mathcal\{S\}$, two FLMs regressing $Y(\backslash cdot)$ on $X(\backslash cdot)$ have been considered.Ramsay and Silverman (2005). Functional data analysis, 2nd ed., New York: Springer, {{ISBN|0-387-40080-X}}. One of these two models is of the form

{{NumBlk|::|$$Y(t)\; =\; \backslash beta\_0(t)\; +\; \backslash int\_\{\backslash mathcal\{S\}\}\; \backslash beta(s,t)\; X^c(s)\backslash ,ds\; +\; \backslash varepsilon(t),\backslash \; \backslash text\{for\}\backslash \; t\backslash in\backslash mathcal\{T\},$$|{{EquationRef|4}}}}

where $X^c(\backslash cdot)\; =\; X(\backslash cdot)\; -\; \backslash mathbb\{E\}(X(\backslash cdot))$ is still the centered functional covariate, $\backslash beta\_0(\backslash cdot)$ and $\backslash beta(\backslash cdot,\backslash cdot)$ are coefficient functions, and $\backslash varepsilon(\backslash cdot)$ is usually assumed to be a random process with mean zero and finite variance. In this case, at any given time $t\backslash in\backslash mathcal\{T\}$, the value of $Y$, i.e., $Y(t)$, depends on the entire trajectory of $X$. Model ({{EquationNote|4}}), for any given time $t$, is an extension of multivariate linear regression with the inner product in Euclidean space replaced by that in $L^2$. An estimating equation motivated by multivariate linear regression is

$$r\_\{XY\}\; =\; R\_\{XX\}\backslash beta,\; \backslash text\{\; for\; \}\; \backslash beta\backslash in\; L^2(\backslash mathcal\{S\}\backslash times\backslash mathcal\{S\}),$$

where $r\_\{XY\}(s,t)\; =\; \backslash text\{cov\}(X(s),Y(t))$, $R\_\{XX\}:\; L^2(\backslash mathcal\{S\}\backslash times\backslash mathcal\{S\})\; \backslash rightarrow\; L^2(\backslash mathcal\{S\}\backslash times\backslash mathcal\{T\})$ is defined as $(R\_\{XX\}\backslash beta)(s,t)\; =\; \backslash int\_\backslash mathcal\{S\}\; r\_\{XX\}(s,w)\backslash beta(w,t)dw$ with $r\_\{XX\}(s,w)\; =\; \backslash text\{cov\}(X(s),X(w))$ for $s,w\backslash in\backslash mathcal\{S\}$. Regularization is needed and can be done through truncation, $L^2$ penalization or $L^1$ penalization. Various estimation methods for model ({{EquationNote|4}}) are available.{{Cite journal |last=Ramsay |first=J. O. |last2=Dalzell |first2=C. J. |date=1991 |title=Some Tools for Functional Data Analysis |url=https://www.jstor.org/stable/2345586 |journal=Journal of the Royal Statistical Society. Series B (Methodological) |volume=53 |issue=3 |pages=539–572 |issn=0035-9246}}{{Cite journal |last=Yao |first=Fang |last2=Müller |first2=Hans-Georg |last3=Wang |first3=Jane-Ling |date= |title=Functional linear regression analysis for longitudinal data |url=https://projecteuclid.org/journals/annals-of-statistics/volume-33/issue-6/Functional-linear-regression-analysis-for-longitudinal-data/10.1214/009053605000000660.full |journal=The Annals of Statistics |volume=33 |issue=6 |pages=2873–2903 |doi=10.1214/009053605000000660 |issn=0090-5364|arxiv=math/0603132 }}

When $X$ and $Y$ are concurrently observed, i.e., $\backslash mathcal\{S\}=\backslash mathcal\{T\}$,{{Cite journal |last=Grenander |first=Ulf |date= |title=Stochastic processes and statistical inference |url=https://projecteuclid.org/journals/arkiv-for-matematik/volume-1/issue-3/Stochastic-processes-and-statistical-inference/10.1007/BF02590638.full |journal=Arkiv för Matematik |volume=1 |issue=3 |pages=195–277 |doi=10.1007/BF02590638 |issn=0004-2080}} it is reasonable to consider a historical functional linear model, where the current value of $Y$ only depends on the history of $X$, i.e., $\backslash beta(s,t)=0$ for $s>t$ in model ({{EquationNote|4}}).{{Cite journal |last=Malfait |first=Nicole |last2=Ramsay |first2=James O. |date=2003 |title=The historical functional linear model |url=https://onlinelibrary.wiley.com/doi/10.2307/3316063 |journal=Canadian Journal of Statistics |language=en |volume=31 |issue=2 |pages=115–128 |doi=10.2307/3316063 |issn=1708-945X|url-access=subscription }} A simpler version of the historical functional linear model is the functional concurrent model (see below).

Adding multiple functional covariates, model ({{EquationNote|4}}) can be extended to

{{NumBlk|::|$$Y(t)\; =\; \backslash beta\_0(t)\; +\; \backslash sum\_\{j=1\}^p\backslash int\_\{\backslash mathcal\{S\}\_j\}\; \backslash beta\_j(s,t)\; X^c\_j(s)\backslash ,ds\; +\; \backslash varepsilon(t),\backslash \; \backslash text\{for\}\backslash \; t\backslash in\backslash mathcal\{T\},$$|{{EquationRef|5}}}}

where for $j=1,\backslash ldots,p$, $X\_j^c(\backslash cdot)=X\_j(\backslash cdot)\; -\; \backslash mathbb\{E\}(X\_j(\backslash cdot))$ is a centered functional covariate with domain $\backslash mathcal\{S\}\_j$, and $\backslash beta\_j(\backslash cdot,\backslash cdot)$ is the corresponding coefficient function with the same domain, respectively. In particular, taking $X\_j(\backslash cdot)$ as a constant function yields a special case of model ({{EquationNote|5}})

$$Y(t)\; =\; \backslash sum\_\{j=1\}^p\; X\_j\; \backslash beta\_j(t)\; +\; \backslash varepsilon(t),\backslash \; \backslash text\{for\}\backslash \; t\backslash in\backslash mathcal\{T\},$$

which is a FLM with functional responses and scalar covariates.

Assuming that $\backslash mathcal\{S\}\; =\; \backslash mathcal\{T\}$, another model, known as the functional concurrent model, sometimes also referred to as the varying-coefficient model, is of the form

{{NumBlk|::|$$Y(t)\; =\; \backslash alpha\_0(t)\; +\; \backslash alpha(t)X(t)+\backslash varepsilon(t),\backslash \; \backslash text\{for\}\backslash \; t\backslash in\backslash mathcal\{T\},$$|{{EquationRef|6}}}}

where $\backslash alpha\_0$ and $\backslash alpha$ are coefficient functions. Note that model ({{EquationNote|6}}) assumes the value of $Y$ at time $t$, i.e., $Y(t)$, only depends on that of $X$ at the same time, i.e., $X(t)$. Various estimation methods can be applied to model ({{EquationNote|6}}).{{Cite journal |last=Fan |first=Jianqing |last2=Zhang |first2=Wenyang |date= |title=Statistical estimation in varying coefficient models |url=https://projecteuclid.org/journals/annals-of-statistics/volume-27/issue-5/Statistical-estimation-in-varying-coefficient-models/10.1214/aos/1017939139.full |journal=The Annals of Statistics |volume=27 |issue=5 |pages=1491–1518 |doi=10.1214/aos/1017939139 |issn=0090-5364}}{{Cite journal |last=Huang |first=Jianhua Z. |last2=Wu |first2=Colin O. |last3=Zhou |first3=Lan |date=2004 |title=Polynomial Spline Estimation and Inference for Varying Coefficient Models with Longitudinal Data |url=https://www.jstor.org/stable/24307415 |journal=Statistica Sinica |volume=14 |issue=3 |pages=763–788 |issn=1017-0405}}{{Cite journal |last=Şentürk |first=Damla |last2=Müller |first2=Hans-Georg |date=2010-09-01 |title=Functional Varying Coefficient Models for Longitudinal Data |url=https://www.tandfonline.com/doi/abs/10.1198/jasa.2010.tm09228 |journal=Journal of the American Statistical Association |doi=10.1198/jasa.2010.tm09228 |issn=0162-1459}}

Adding multiple functional covariates, model ({{EquationNote|6}}) can also be extended to

$$Y(t)\; =\; \backslash alpha\_0(t)\; +\; \backslash sum\_\{j=1\}^p\backslash alpha\_j(t)X\_j(t)+\backslash varepsilon(t),\backslash \; \backslash text\{for\}\backslash \; t\backslash in\backslash mathcal\{T\},$$

where $X\_1,\backslash ldots,X\_p$ are multiple functional covariates with domain $\backslash mathcal\{T\}$ and $\backslash alpha\_0,\backslash alpha\_1,\backslash ldots,\backslash alpha\_p$ are the coefficient functions with the same domain.

Functional polynomial models are an extension of the FLMs with scalar responses, analogous to extending linear regression to polynomial regression. For a scalar response $Y$ and a functional covariate $X(\backslash cdot)$ with domain $\backslash mathcal\{T\}$, the simplest example of functional polynomial models is functional quadratic regression{{Cite journal |last=Yao |first=F. |last2=Muller |first2=H.-G. |date=2010-03-01 |title=Functional quadratic regression |url=https://academic.oup.com/biomet/article-lookup/doi/10.1093/biomet/asp069 |journal=Biometrika |language=en |volume=97 |issue=1 |pages=49–64 |doi=10.1093/biomet/asp069 |issn=0006-3444|url-access=subscription }}

$$Y\; =\; \backslash alpha\; +\; \backslash int\_\backslash mathcal\{T\}\backslash beta(t)X^c(t)\backslash ,dt\; +\; \backslash int\_\backslash mathcal\{T\}\; \backslash int\_\backslash mathcal\{T\}\; \backslash gamma(s,t)\; X^c(s)X^c(t)\; \backslash ,ds\backslash ,dt\; +\; \backslash varepsilon,$$

where $X^c(\backslash cdot)\; =\; X(\backslash cdot)\; -\; \backslash mathbb\{E\}(X(\backslash cdot))$ is the centered functional covariate, $\backslash alpha$ is a scalar coefficient, $\backslash beta(\backslash cdot)$ and $\backslash gamma(\backslash cdot,\backslash cdot)$ are coefficient functions with domains $\backslash mathcal\{T\}$ and $\backslash mathcal\{T\}\backslash times\backslash mathcal\{T\}$, respectively, and $\backslash varepsilon$ is a random error with mean zero and finite variance. By analogy to FLMs with scalar responses, estimation of functional polynomial models can be obtained through expanding both the centered covariate $X^c$ and the coefficient functions $\backslash beta$ and $\backslash gamma$ in an orthonormal basis.

A functional multiple index model is given by

$$Y\; =\; g\backslash left(\backslash int\_\{\backslash mathcal\{T\}\}\; X^c(t)\; \backslash beta\_1(t)\backslash ,dt,\; \backslash ldots,\; \backslash int\_\{\backslash mathcal\{T\}\}\; X^c(t)\; \backslash beta\_p(t)\backslash ,dt\; \backslash right)\; +\; \backslash varepsilon.$$

Taking $p=1$ yields a functional single index model. However, for $p>1$, this model is problematic due to curse of dimensionality. With $p>1$ and relatively small sample sizes, the estimator given by this model often has large variance.{{Cite journal |last=Chen |first=Dong |last2=Hall |first2=Peter |last3=Müller |first3=Hans-Georg |date= |title=Single and multiple index functional regression models with nonparametric link |url=https://projecteuclid.org/journals/annals-of-statistics/volume-39/issue-3/Single-and-multiple-index-functional-regression-models-with-nonparametric-link/10.1214/11-AOS882.full |journal=The Annals of Statistics |volume=39 |issue=3 |pages=1720–1747 |doi=10.1214/11-AOS882 |issn=0090-5364|arxiv=1211.5018 }} An alternative $p$-component functional multiple index model can be expressed as

$$Y\; =\; g\_1\backslash left(\backslash int\_\{\backslash mathcal\{T\}\}\; X^c(t)\; \backslash beta\_1(t)\backslash ,dt\backslash right)+\; \backslash cdots+\; g\_p\backslash left(\backslash int\_\{\backslash mathcal\{T\}\}\; X^c(t)\; \backslash beta\_p(t)\backslash ,dt\; \backslash right)\; +\; \backslash varepsilon.$$

Estimation methods for functional single and multiple index models are available.{{Cite journal |last=Jiang |first=Ci-Ren |last2=Wang |first2=Jane-Ling |date= |title=Functional single index models for longitudinal data |url=https://projecteuclid.org/journals/annals-of-statistics/volume-39/issue-1/Functional-single-index-models-for-longitudinal-data/10.1214/10-AOS845.full |journal=The Annals of Statistics |volume=39 |issue=1 |pages=362–388 |doi=10.1214/10-AOS845 |issn=0090-5364|arxiv=1103.1726 }}

Given an expansion of a functional covariate $X$ with domain $\backslash mathcal\{T\}$ in an orthonormal basis $\backslash \{\backslash phi\_k\backslash \}\_\{k=1\}^\backslash infty$: $X(t)\; =\; \backslash sum\_\{k=1\}^\backslash infty\; x\_k\; \backslash phi\_k(t)$, a functional linear model with scalar responses shown in model ({{EquationNote|2}}) can be written as

$$\backslash mathbb\{E\}(Y|X)=\backslash mathbb\{E\}(Y)\; +\; \backslash sum\_\{k=1\}^\backslash infty\; \backslash beta\_k\; x\_k.$$

One form of FAMs is obtained by replacing the linear function of $x\_k$, i.e., $\backslash beta\_k\; x\_k$, by a general smooth function $f\_k$,

$$\backslash mathbb\{E\}(Y|X)=\backslash mathbb\{E\}(Y)\; +\; \backslash sum\_\{k=1\}^\backslash infty\; f\_k(x\_k),$$

where $f\_k$ satisfies $\backslash mathbb\{E\}(f\_k(x\_k))=0$ for $k\backslash in\backslash mathbb\{N\}$.{{Cite journal |last=Müller |first=Hans-Georg |last2=Yao |first2=Fang |date=2008-12-01 |title=Functional Additive Models |url=https://www.tandfonline.com/doi/abs/10.1198/016214508000000751 |journal=Journal of the American Statistical Association |doi=10.1198/016214508000000751 |issn=0162-1459|url-access=subscription }} Another form of FAMs consists of a sequence of time-additive models:

$$\backslash mathbb\{E\}(Y|X(t\_1),\backslash ldots,X(t\_p))=\backslash sum\_\{j=1\}^p\; f\_j(X(t\_j)),$$

where $\backslash \{t\_1,\backslash ldots,t\_p\backslash \}$ is a dense grid on $\backslash mathcal\{T\}$ with increasing size $p\backslash in\backslash mathbb\{N\}$, and $f\_j(x)\; =\; g(t\_j,x)$ with $g$ a smooth function, for $j=1,\backslash ldots,p${{Cite journal |last=Fan |first=Yingying |last2=James |first2=Gareth M. |last3=Radchenko |first3=Peter |date= |title=Functional additive regression |url=https://projecteuclid.org/journals/annals-of-statistics/volume-43/issue-5/Functional-additive-regression/10.1214/15-AOS1346.full |journal=The Annals of Statistics |volume=43 |issue=5 |pages=2296–2325 |doi=10.1214/15-AOS1346 |issn=0090-5364|arxiv=1510.04064 }}

A direct extension of FLMs with scalar responses shown in model ({{EquationNote|2}}) is to add a link function to create a generalized functional linear model (GFLM) by analogy to extending linear regression to generalized linear regression (GLM), of which the three components are:

1. Linear predictor $\backslash eta\; =\; \backslash beta\_0\; +\; \backslash int\_\{\backslash mathcal\{T\}\}\; X^c(t)\backslash beta(t)\backslash ,dt$;
2. Variance function $\backslash text\{Var\}(Y|X)\; =\; V(\backslash mu)$, where $\backslash mu\; =\; \backslash mathbb\{E\}(Y|X)$ is the conditional mean;
3. Link function $g$ connecting the conditional mean and the linear predictor through $\backslash mu=g(\backslash eta)$.

• Functional data analysis
• Functional principal component analysis
• Karhunen–Loève theorem
• Generalized functional linear model
• Stochastic processes

Category:Regression analysis