Least-squares function approximation

{{See also|Fourier series|Generalized Fourier series}}

A generalization to approximation of a data set is the approximation of a function by a sum of other functions, usually an orthogonal set:

{{cite book |title=Applied analysis |author=Cornelius Lanczos |pages =212–213 |isbn=0-486-65656-X |publisher=Dover Publications |year=1988 |edition=Reprint of 1956 Prentice–Hall |url=https://books.google.com/books?id=6E85hExIqHYC&pg=PA212}}

:$f(x)\; \backslash approx\; f\_n\; (x)\; =\; a\_1\; \backslash phi\; \_1\; (x)\; +\; a\_2\; \backslash phi\; \_2(x)\; +\; \backslash cdots\; +\; a\_n\; \backslash phi\; \_n\; (x),\; \backslash$

with the set of functions {$\backslash \; \backslash phi\; \_j\; (x)$} an orthonormal set over the interval of interest, {{nowrap|say [a, b]}}: see also Fejér's theorem. The coefficients {$\backslash \; a\_j$} are selected to make the magnitude of the difference ||{{nowrap|f − f_n}}||² as small as possible. For example, the magnitude, or norm, of a function {{nowrap|g (x )}} over the {{nowrap|interval [a, b]}} can be defined by:

{{cite book |title=Fourier analysis and its application |page =69 |chapter=Equation 3.14 |author=Gerald B Folland |chapter-url=https://books.google.com/books?id=ix2iCQ-o9x4C&pg=PA69 |isbn=978-0-8218-4790-9 |publisher=American Mathematical Society Bookstore |year=2009 |edition=Reprint of Wadsworth and Brooks/Cole 1992}}

:$\backslash |g\backslash |\; =\; \backslash left(\backslash int\_a^b\; g^*(x)g(x)\; \backslash ,\; dx\; \backslash right)^\{1/2\}$

where the ‘*’ denotes complex conjugate in the case of complex functions. The extension of Pythagoras' theorem in this manner leads to function spaces and the notion of Lebesgue measure, an idea of “space” more general than the original basis of Euclidean geometry. The {{nowrap|{ $\backslash phi\_j\; (x)\backslash$ } }} satisfy orthonormality relations:

{{cite book |title=Fourier Analysis and Its Applications|page =69 |first1=Gerald B | last1= Folland|url=https://books.google.com/books?id=ix2iCQ-o9x4C&pg=PA69 |isbn=978-0-8218-4790-9 |year=2009 |publisher=American Mathematical Society}}

:$\backslash int\_a^b\; \backslash phi\; \_i^*\; (x)\backslash phi\; \_j\; (x)\; \backslash ,\; dx\; =\backslash delta\_\{ij\},$

where δ_ij is the Kronecker delta. Substituting function {{nowrap|f_n}} into these equations then leads to

the n-dimensional Pythagorean theorem:

{{cite book |title=Statistical methods: the geometric approach |author= David J. Saville, Graham R. Wood |chapter=§2.5 Sum of squares |page=30 |chapter-url=https://books.google.com/books?id=8ummgMVRev0C&pg=PA30 |isbn=0-387-97517-9 |year=1991 |edition=3rd |publisher=Springer}}

:$\backslash |f\_n\backslash |^2\; =\; |a\_1|^2\; +\; |a\_2|^2\; +\; \backslash cdots\; +\; |a\_n|^2.\; \backslash ,$

The coefficients {a_j} making {{nowrap begin}}||f − f_n||²{{nowrap end}} as small as possible are found to be:

:$a\_j\; =\; \backslash int\_a^b\; \backslash phi\; \_j^*\; (x)f\; (x)\; \backslash ,\; dx.$

The generalization of the n-dimensional Pythagorean theorem to infinite-dimensional  real inner product spaces is known as Parseval's identity or Parseval's equation.

{{cite book |title=cited work |page =77 |chapter=Equation 3.22 |author=Gerald B Folland |chapter-url=https://books.google.com/books?id=ix2iCQ-o9x4C&pg=PA77 |isbn=978-0-8218-4790-9 |date=2009-01-13 |publisher =American Mathematical Soc. }}

Particular examples of such a representation of a function are the Fourier series and the generalized Fourier series.

It follows that one can find a "best" approximation of another function by minimizing the area between two functions, a continuous function $f$ on $[a,b]$ and a function $g\backslash in\; W$ where $W$ is a subspace of $C[a,b]$:

:$\backslash text\{Area\}\; =\; \backslash int\_a^b\; \backslash left\backslash vert\; f(x)\; -\; g(x)\backslash right\backslash vert\; \backslash ,\; dx,$

all within the subspace $W$. Due to the frequent difficulty of evaluating integrands involving absolute value, one can instead define

:$\backslash int\_a^b\; [\; f(x)\; -\; g(x)\; ]\; ^2\backslash ,\; dx$

as an adequate criterion for obtaining the least squares approximation, function $g$, of $f$ with respect to the inner product space $W$.

As such, $\backslash lVert\; f-g\; \backslash rVert\; ^2$ or, equivalently, $\backslash lVert\; f-g\; \backslash rVert$, can thus be written in vector form:

:$\backslash int\_a^b\; [\; f(x)-g(x)\; ]^2\backslash ,\; dx\; =\; \backslash left\backslash langle\; f-g\; ,\; f-g\backslash right\backslash rangle\; =\; \backslash lVert\; f-g\backslash rVert^2.$

In other words, the least squares approximation of $f$ is the function $g\backslash in\; \backslash text\{\; subspace\; \}\; W$ closest to $f$ in terms of the inner product $\backslash left\; \backslash langle\; f,g\; \backslash right\; \backslash rangle$. Furthermore, this can be applied with a theorem:

:Let $f$ be continuous on $[\; a,b\; ]$, and let $W$ be a finite-dimensional subspace of $C[a,b]$. The least squares approximating function of $f$ with respect to $W$ is given by

::$g\; =\; \backslash left\; \backslash langle\; f,\backslash vec\; w\_1\; \backslash right\; \backslash rangle\; \backslash vec\; w\_1\; +\; \backslash left\; \backslash langle\; f,\backslash vec\; w\_2\; \backslash right\; \backslash rangle\; \backslash vec\; w\_2\; +\; \backslash cdots\; +\; \backslash left\; \backslash langle\; f,\backslash vec\; w\_n\; \backslash right\; \backslash rangle\; \backslash vec\; w\_n,$

:where $B\; =\; \backslash \{\backslash vec\; w\_1\; ,\; \backslash vec\; w\_2\; ,\; \backslash dots\; ,\; \backslash vec\; w\_n\; \backslash \}$ is an orthonormal basis for $W$.

{{reflist}}

Category:Least squares

Category:Approximation theory