Hilbert projection theorem

Some intuition for the theorem can be obtained by considering the first order condition of the optimization problem.

Consider a finite dimensional real Hilbert space $H$ with a subspace $C$ and a point $x.$ If $m\; \backslash in\; C$ is a {{em|minimizer}} or {{em|minimum point}} of the function $N\; :\; C\; \backslash to\; \backslash R$ defined by $N(c)\; :=\; \backslash |c\; -\; x\backslash |$ (which is the same as the minimum point of $c\; \backslash mapsto\; \backslash |c\; -\; x\backslash |^2$), then derivative must be zero at $m.$

In matrix derivative notation{{cite web|last1=Petersen|first1=Kaare|title=The Matrix Cookbook|url=https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf|access-date=9 January 2021}}

$$\backslash begin\{aligned\}$$

\partial \lVert x - c \rVert^2 &= \partial \langle c - x, c - x \rangle \\

&= 2 \langle c - x, \partial c\rangle

\end{aligned}

Since $\backslash partial\; c$ is a vector in $C$ that represents an arbitrary tangent direction, it follows that $m\; -\; x$ must be orthogonal to every vector in $C.$

{{math theorem|name=Hilbert projection theorem|math_statement=For every vector $x$ in a Hilbert space $H$ and every nonempty closed convex $C\; \backslash subseteq\; H,$ there exists a unique vector $m\; \backslash in\; C$ for which $\backslash lVert\; x\; -\; m\; \backslash rVert$ is equal to $\backslash delta\; :=\; \backslash inf\_\{c\; \backslash in\; C\}\; \backslash |x\; -\; c\backslash |.$

If the closed subset $C$ is also a vector subspace of $H$ then this minimizer $m$ is the unique element in $C$ such that $x\; -\; m$ is orthogonal to $C.$

}}

{{math proof|title=Proof that a minimum point $y$ exists|drop=hidden|proof=

Let $\backslash delta\; :=\; \backslash inf\_\{c\; \backslash in\; C\}\; \backslash |x\; -\; c\backslash |$ be the distance between $x$ and $C,$ $\backslash left(c\_n\backslash right)\_\{n=1\}^\{\backslash infty\}$ a sequence in $C$ such that the distance squared between $x$ and $c\_n$ is less than or equal to $\backslash delta^2\; +\; 1/n.$ Let $n$ and $m$ be two integers, then the following equalities are true:

$$\backslash left\backslash |c\_n\; -\; c\_m\backslash right\backslash |^2\; =\; \backslash left\backslash |c\_n\; -\; x\backslash right\backslash |^2\; +\; \backslash left\backslash |c\_m\; -\; x\backslash right\backslash |^2\; -\; 2\; \backslash left\backslash langle\; c\_n\; -\; x\; \backslash ,\; ,\; \backslash ,\; c\_m\; -\; x\backslash right\backslash rangle$$

and

$$4\; \backslash left\backslash |\backslash frac\{c\_n\; +\; c\_m\}2\; -\; x\backslash right\backslash |^2\; =\; \backslash left\backslash |c\_n\; -\; x\backslash right\backslash |^2\; +\; \backslash left\backslash |c\_m\; -\; x\backslash right\backslash |^2\; +\; 2\; \backslash left\backslash langle\; c\_n\; -\; x\; \backslash ,\; ,\; \backslash ,\; c\_m\; -\; x\backslash right\backslash rangle$$

Therefore

$$\backslash left\backslash |c\_n\; -\; c\_m\backslash right\backslash |^2\; =\; 2\; \backslash left\backslash |c\_n\; -\; x\backslash right\backslash |^2\; +\; 2\backslash left\backslash |c\_m\; -\; x\backslash right\backslash |^2\; -\; 4\backslash left\backslash |\backslash frac\{c\_n\; +\; c\_m\}2\; -\; x\backslash right\backslash |^2$$

(This equation is the same as the formula $a^2\; =\; 2\; b^2\; +\; 2\; c^2\; -\; 4\; M\_a^2$ for the length $M\_a$ of a median in a triangle with sides of length $a,\; b,$ and $c,$ where specifically, the triangle's vertices are $x,\; c\_m,\; c\_n$).

By giving an upper bound to the first two terms of the equality and by noticing that the midpoint of $c\_n$ and $c\_m$ belong to $C$ and has therefore a distance greater than or equal to $\backslash delta$ from $x,$ it follows that:

$$\backslash |c\_n\; -\; c\_m\backslash |^2\; \backslash ;\; \backslash leq\; \backslash ;\; 2\backslash left(\backslash delta^2\; +\; \backslash frac\{1\}\{n\}\backslash right)\; +\; 2\backslash left(\backslash delta^2\; +\; \backslash frac\{1\}\{m\}\backslash right)\; -\; 4\backslash delta^2\; =\; 2\backslash left(\backslash frac\{1\}\{n\}\; +\; \backslash frac\{1\}\{m\}\backslash right)$$

The last inequality proves that $\backslash left(c\_n\backslash right)\_\{n=1\}^\{\backslash infty\}$ is a Cauchy sequence. Since $C$ is complete, the sequence is therefore convergent to a point $m\; \backslash in\; C,$ whose distance from $x$ is minimal.

$\backslash blacksquare$

}}

{{math proof|title=Proof that $m$ is unique|drop=hidden|proof=

Let $m\_1$ and $m\_2$ be two minimum points. Then:

$$\backslash |m\_2\; -\; m\_1\backslash |^2\; =\; 2\backslash |m\_1\; -\; x\backslash |^2\; +\; 2\backslash |m\_2\; -\; x\backslash |^2\; -\; 4\; \backslash left\backslash |\backslash frac\{m\_1\; +\; m\_2\}2\; -\; x\backslash right\backslash |^2$$

Since $\backslash frac\{m\_1\; +\; m\_2\}2$ belongs to $C,$ we have $\backslash left\backslash |\backslash frac\{m\_1\; +\; m\_2\}\; 2\; -\; x\backslash right\backslash |^2\; \backslash geq\; \backslash delta^2$ and therefore

$$\backslash |m\_2\; -\; m\_1\backslash |^2\; \backslash leq\; 2\; \backslash delta^2\; +\; 2\; \backslash delta^2\; -\; 4\; \backslash delta^2\; =\; 0.$$

Hence $m\_1\; =\; m\_2,$ which proves uniqueness.

$\backslash blacksquare$

}}

{{math proof|title=Proof of characterization of minimum point when $C$ is a closed vector subspace|drop=hidden|proof=

Assume that $C$ is a closed vector subspace of $H.$ It must be shown the minimizer $m$ is the unique element in $C$ such that $\backslash langle\; m\; -\; x,\; c\; \backslash rangle\; =\; 0$ for every $c\; \backslash in\; C.$

Proof that the condition is sufficient:

Let $z\; \backslash in\; C$ be such that $\backslash langle\; z\; -\; x,\; c\; \backslash rangle\; =\; 0$ for all $c\; \backslash in\; C.$

If $c\; \backslash in\; C$ then $c\; -\; z\; \backslash in\; C$ and so

$$\backslash |c-x\backslash |^2\; =\; \backslash |(z-x)\; +\; (c-z)\backslash |^2\; =\; \backslash |z-x\backslash |^2\; +\; \backslash |c-z\backslash |^2\; +\; 2\; \backslash langle\; z-x,\; c-z\; \backslash rangle\; =\; \backslash |z-x\backslash |^2\; +\; \backslash |c-z\backslash |^2$$

which implies that $\backslash |z-x\backslash |^2\; \backslash leq\; \backslash |c-x\backslash |^2.$

Because $c\; \backslash in\; C$ was arbitrary, this proves that $\backslash |z-x\backslash |\; =\; \backslash inf\_\{c\; \backslash in\; C\}\; \backslash |c\; -\; x\backslash |$ and so $z$ is a minimum point.

Proof that the condition is necessary:

Let $m\; \backslash in\; C$ be the minimum point. Let $c\; \backslash in\; C$ and $t\; \backslash in\; \backslash R.$

Because $m\; +\; t\; c\; \backslash in\; C,$ the minimality of $m$ guarantees that $\backslash |m-x\backslash |\; \backslash leq\; \backslash |(m\; +\; t\; c)\; -\; x\backslash |.$ Thus

$$\backslash |(m\; +\; t\; c)\; -\; x\backslash |^2\; -\; \backslash |m-x\backslash |^2\; =\; 2t\backslash langle\; m-x,\; c\backslash rangle\; +\; t^2\; \backslash |c\backslash |^2$$

is always non-negative and $\backslash langle\; m-x,\; c\backslash rangle$ must be a real number.

If $\backslash langle\; m\; -\; x,\; c\backslash rangle\; \backslash neq\; 0$ then the map $f(t)\; :=\; 2t\backslash langle\; m\; -\; x,\; c\backslash rangle\; +\; t^2\; \backslash |c\backslash |^2$ has a minimum at $t\_0\; :=\; -\; \backslash frac\{\backslash langle\; m\; -\; x,\; c\backslash rangle\}\{\backslash |c\backslash |^2\}$ and moreover, which is a contradiction.

Thus $\backslash langle\; m\; -\; x,\; c\backslash rangle\; =\; 0.$

$\backslash blacksquare$

}}

It suffices to prove the theorem in the case of $x\; =\; 0$ because the general case follows from the statement below by replacing $C$ with $C\; -\; x.$

{{math theorem|name=Hilbert projection theorem (case $x\; =\; 0$){{sfn|Rudin|1991|pp=306–309}}|math_statement=For every nonempty closed convex subset $C\; \backslash subseteq\; H$ of a Hilbert space $H,$ there exists a unique vector $m\; \backslash in\; C$ such that $\backslash inf\_\{c\; \backslash in\; C\}\; \backslash |\; c\; \backslash |\; =\; \backslash |\; m\; \backslash |.$

Furthermore, letting $d\; :=\; \backslash inf\_\{c\; \backslash in\; C\}\; \backslash |\; c\; \backslash |,$ if $\backslash left(c\_n\backslash right)\_\{n=1\}^\{\backslash infty\}$ is {{em|any}} sequence in $C$ such that $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; \backslash left\backslash |c\_n\backslash right\backslash |\; =\; d$ in $\backslash R$Because the norm $\backslash |\; \backslash cdot\; \backslash |\; :\; H\; \backslash to\; \backslash R$ is continuous, if $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; x\_n$ converges in $H$ then necessarily $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; \backslash left\backslash |x\_n\backslash right\backslash |$ converges in $\backslash R.$ But in general, the converse is not guaranteed. However, under this theorem's hypotheses, knowing that $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; \backslash left\backslash |c\_n\backslash right\backslash |\; =\; d$ in $\backslash R$ {{em|is sufficient}} to conclude that $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; c\_n$ converges in $H.$ then $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; c\_n\; =\; m$ in $H.$

}}

{{math proof|drop=hidden|proof=

Let $C$ be as described in this theorem and let

$$d\; :=\; \backslash inf\_\{c\; \backslash in\; C\}\; \backslash |\; c\; \backslash |.$$

This theorem will follow from the following lemmas.

{{math theorem|name=Lemma 1|math_statement=If $c\_\{\backslash bull\}\; :=\; \backslash left(c\_n\backslash right)\_\{n=1\}^\{\backslash infty\}$ is {{em|any}} sequence in $C$ such that $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; \backslash left\backslash |c\_n\backslash right\backslash |\; =\; d$ in $\backslash R$ then there exists some $c\; \backslash in\; C$ such that $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; c\_n\; =\; c$ in $H.$ Furthermore, $\backslash |c\backslash |\; =\; d.$

}}

{{collapse top|title=Proof of Lemma 1|left=true}}

File:Parallelogram law.svg

Because $C$ is convex, if $m,\; n\; \backslash in\; \backslash N$ then $\backslash frac\{1\}\{2\}\backslash left(c\_m\; +\; c\_n\backslash right)\; \backslash in\; C$ so that by definition of the infimum, $d\; \backslash leq\; \backslash left\backslash |\; \backslash frac\{1\}\{2\}\backslash left(c\_m\; +\; c\_n\backslash right)\; \backslash right\backslash |,$ which implies that $4d^2\; \backslash leq\; \backslash left\backslash |c\_m\; +\; c\_n\backslash right\backslash |^2.$

By the parallelogram law,

$$\backslash left\backslash |c\_m\; +\; c\_n\backslash right\backslash |^2\; +\; \backslash left\backslash |c\_m\; -\; c\_n\backslash right\backslash |^2\; =\; 2\; \backslash left\backslash |c\_m\backslash right\backslash |^2\; +\; 2\; \backslash left\backslash |c\_n\backslash right\backslash |^2$$

where $4d^2\; \backslash leq\; \backslash left\backslash |c\_m\; +\; c\_n\backslash right\backslash |^2$ now implies

$$4\; d^2\; +\; \backslash left\backslash |c\_m\; -\; c\_n\backslash right\backslash |^2\; ~\backslash leq~\; 2\; \backslash left\backslash |c\_m\backslash right\backslash |^2\; +\; 2\; \backslash left\backslash |c\_n\backslash right\backslash |^2$$

and so

$$\backslash begin\{alignat\}\{4\}$$

\left\|c_m - c_n\right\|^2

~\leq~ 2 \left\|c_m\right\|^2 + 2 \left\|c_n\right\|^2 - 4 d^2

\end{alignat}

The assumption $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; \backslash left\backslash |c\_n\backslash right\backslash |\; =\; d$ implies that the right hand side (RHS) of the above inequality can be made arbitrary close to $0$ by making $m$ and $n$ sufficiently large.Explicitly, this means that given any $\backslash epsilon\; >\; 0$ there exists some integer $N\; >\; 0$ such that "the quantity" is $\backslash ,\backslash leq\; \backslash epsilon$ whenever $m,\; n\; \backslash geq\; N.$ Here, "the quantity" refers to the inequality's right hand side $2\; \backslash left\backslash |c\_m\backslash right\backslash |^2\; +\; 2\; \backslash left\backslash |c\_n\backslash right\backslash |^2\; -\; 4\; d^2$ and later in the proof, "the quantity" will also refer to $\backslash left\backslash |c\_m\; -\; c\_n\backslash right\backslash |^2$ and then $\backslash left\backslash |c\_m\; -\; c\_n\backslash right\backslash |.$ By definition of "Cauchy sequence," $\backslash left(c\_n\backslash right)\_\{n=1\}^\{\backslash infty\}$ is Cauchy in $H$ if and only if "the quantity" $\backslash left\backslash |c\_m\; -\; c\_n\backslash right\backslash |$ satisfies this aforementioned condition. The same must consequently also be true of the inequality's left hand side $\backslash left\backslash |c\_m\; -\; c\_n\backslash right\backslash |^2$ and thus also of $\backslash left\backslash |c\_m\; -\; c\_n\backslash right\backslash |,$ which proves that $\backslash left(c\_n\backslash right)\_\{n=1\}^\{\backslash infty\}$ is a Cauchy sequence in $H.$

Since $H$ is complete, there exists some $c\; \backslash in\; H$ such that $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; c\_n\; =\; c$ in $H.$

Because every $c\_n$ belongs to $C,$ which is a closed subset of $H,$ their limit $c$ must also belongs to this closed subset, which proves that $c\; \backslash in\; C.$

Since the norm $\backslash |\; \backslash ,\backslash cdot\backslash ,\; \backslash |\; :\; H\; \backslash to\; \backslash R$ is a continuous function, $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; c\_n\; =\; c$ in $H$ implies that $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; \backslash left\backslash |c\_n\backslash right\backslash |\; =\; \backslash |c\backslash |$ in $\backslash R.$ But $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; \backslash left\backslash |c\_n\backslash right\backslash |\; =\; d$ also holds (by assumption) so that $\backslash |c\backslash |\; =\; d$ (because limits in $\backslash R$ are unique).

$\backslash blacksquare$

{{collapse bottom}}

{{math theorem|name=Lemma 2|math_statement=A sequence $\backslash left(c\_n\backslash right)\_\{n=1\}^\{\backslash infty\}$ satisfying the hypotheses of Lemma 1 exists.

}}

{{collapse top|title=Proof of Lemma 2|left=true}}

The existence of the sequence follows from the definition of the infimum, as is now shown.

The set $S\; :=\; \backslash \{\; \backslash |\; c\; \backslash |\; :\; c\; \backslash in\; C\; \backslash \}$ is a non-empty subset of non-negative real numbers and $d\; :=\; \backslash inf\_\{c\; \backslash in\; C\}\; \backslash |\; c\; \backslash |\; =\; \backslash inf\; S.$

Let $n\; \backslash geq\; 1$ be an integer.

Because there exists some $s\_n\; \backslash in\; S$ such that

Since $s\_n\; \backslash in\; S,$ $d\; =\; \backslash inf\; S\; \backslash leq\; s\_n$ holds (by definition of the infimum). Thus and now the squeeze theorem implies that $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; s\_n\; =\; d$ in $\backslash R.$ (This first part of the proof works for any non-empty subset of $S\; \backslash subseteq\; \backslash R$ for which $d\; :=\; \backslash inf\_\{s\; \backslash in\; S\}\; s$ is finite).

For every $n\; \backslash in\; \backslash N,$ the fact that $s\_n\; \backslash in\; S\; =\; \backslash \{\; \backslash |\; c\; \backslash |\; :\; c\; \backslash in\; C\; \backslash \}$ means that there exists some $c\_n\; \backslash in\; C$ such that $s\_n\; =\; \backslash left\backslash |\; c\_n\; \backslash right\backslash |.$

The convergence $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; s\_n\; =\; d$ in $\backslash R$ thus becomes $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; \backslash left\backslash |c\_n\backslash right\backslash |\; =\; d$ in $\backslash R.$

$\backslash blacksquare$

{{collapse bottom}}

Lemma 2 and Lemma 1 together prove that there exists some $c\; \backslash in\; C$ such that $\backslash |c\backslash |\; =\; d.$

Lemma 1 can be used to prove uniqueness as follows.

Suppose $b\; \backslash in\; C$ is such that $\backslash |b\backslash |\; =\; d$ and denote the sequence $$b,\; c,\; b,\; c,\; b,\; c,\; \backslash ldots$$ by $\backslash left(c\_n\backslash right)\_\{n=1\}^\{\backslash infty\}$ so that the subsequence $\backslash left(c\_\{2n\}\backslash right)\_\{n=1\}^\{\backslash infty\}$ of even indices is the constant sequence $c,\; c,\; c,\; \backslash ldots$ while the subsequence $\backslash left(c\_\{2n\; -\; 1\}\backslash right)\_\{n=1\}^\{\backslash infty\}$ of odd indices is the constant sequence $b,\; b,\; b,\; \backslash ldots.$

Because $\backslash left\backslash |c\_n\backslash right\backslash |\; =\; d$ for every $n\; \backslash in\; \backslash N,$ $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; \backslash left\backslash |c\_n\backslash right\backslash |\; =\; \backslash lim\_\{n\; \backslash to\; \backslash infty\}\; d\; =\; d$ in $\backslash R,$ which shows that the sequence $\backslash left(c\_n\backslash right)\_\{n=1\}^\{\backslash infty\}$ satisfies the hypotheses of Lemma 1.

Lemma 1 guarantees the existence of some $x\; \backslash in\; C$ such that $\backslash lim\_\{n\; \backslash to\; \backslash infty\}\; c\_n\; =\; x$ in $H.$

Because $\backslash left(c\_n\backslash right)\_\{n=1\}^\{\backslash infty\}$ converges to $x,$ so do all of its subsequences.

In particular, the subsequence $c,\; c,\; c,\; \backslash ldots$ converges to $x,$ which implies that $x\; =\; c$ (because limits in $H$ are unique and this constant subsequence also converges to $c$). Similarly, $x\; =\; b$ because the subsequence $b,\; b,\; b,\; \backslash ldots$ converges to both $x$ and $b.$ Thus $b\; =\; c,$ which proves the theorem. $\backslash blacksquare$

}}

{{math theorem|name=Proposition|math_statement=If $C$ is a closed vector subspace of a Hilbert space $H$ thenTechnically, $H\; =\; K\; \backslash oplus\; K^\{\backslash bot\}$ means that the addition map $K\; \backslash times\; K^\{\backslash bot\}\; \backslash to\; H$ defined by $(k,\; p)\; \backslash mapsto\; k\; +\; p$ is a surjective linear isomorphism and homeomorphism. See the article on complemented subspaces for more details.

$$H\; =\; C\; \backslash oplus\; C^\{\backslash bot\}.$$

}}

{{collapse top|title=Proof{{sfn|Rudin|1991|pp=307−309}}|left=true}}

{{em|Proof that $C\; \backslash cap\; C^\{\backslash bot\}\; =\; \backslash \{\; 0\; \backslash \}$}}:

If $c\; \backslash in\; C\; \backslash cap\; C^\{\backslash bot\}$ then

$0\; =\; \backslash langle\; \backslash ,c,\; \backslash ,c\backslash ,\; \backslash rangle\; =\; \backslash |c\backslash |^2,$

which implies $c\; =\; 0.$

$\backslash blacksquare$

{{hr|1}}

{{em|Proof that $C^\{\backslash bot\}$ is a closed vector subspace of $H$}}:

Let $P\; :=\; \backslash prod\_\{c\; \backslash in\; C\}\; \backslash mathbb\{F\}$ where $\backslash mathbb\{F\}$ is the underlying scalar field of $H$ and define

$$\backslash begin\{alignat\}\{4\}$$

L : \,& H && \to \,&& P \\

& h && \mapsto\,&& \left(\langle \,h, \,c\, \rangle\right)_{c \in C} \\

\end{alignat}

which is continuous and linear because this is true of each of its coordinates $h\; \backslash mapsto\; \backslash langle\; h,\; c\; \backslash rangle.$

The set $C^\{\backslash bot\}\; =\; L^\{-1\}(0)\; =\; L^\{-1\}\backslash left(\backslash \{\; 0\; \backslash \}\backslash right)$ is closed in $H$ because $\backslash \{\; 0\; \backslash \}$ is closed in $P$ and $L\; :\; H\; \backslash to\; P$ is continuous.

The kernel of any linear map is a vector subspace of its domain, which is why $C^\{\backslash bot\}\; =\; \backslash ker\; L$ is a vector subspace of $H.$

$\backslash blacksquare$

{{hr|1}}

{{em|Proof that $C\; +\; C^\{\backslash bot\}\; =\; H$}}:

Let $x\; \backslash in\; H.$

The Hilbert projection theorem guarantees the existence of a unique $m\; \backslash in\; C$ such that $\backslash |x\; -\; m\backslash |\; \backslash leq\; \backslash |x\; -\; c\backslash |\; \backslash text\{\; for\; all\; \}\; c\; \backslash in\; C$ (or equivalently, for all $x\; -\; c\; \backslash in\; x\; -\; C$).

Let $p\; :=\; x\; -\; m$ so that $x\; =\; m\; +\; p\; \backslash in\; C\; +\; p$ and it remains to show that $p\; \backslash in\; C^\{\backslash bot\}.$

The inequality above can be rewritten as:

$$\backslash |p\backslash |\; \backslash leq\; \backslash |z\backslash |\; \backslash quad\; \backslash text\{\; for\; all\; \}\; z\; \backslash in\; x\; -\; C.$$

Because $m\; \backslash in\; C$ and $C$ is a vector space, $m\; +\; C\; =\; C$ and $C\; =\; -\; C,$ which implies that $x\; -\; C\; =\; x\; +\; C\; =\; p\; +\; m\; +\; C\; =\; p\; +\; C.$

The previous inequality thus becomes

$$\backslash |p\backslash |\; \backslash leq\; \backslash |z\backslash |\; \backslash quad\; \backslash text\{\; for\; all\; \}\; z\; \backslash in\; p\; +\; C.$$

or equivalently,

$$\backslash |p\backslash |\; \backslash leq\; \backslash |p\; +\; c\backslash |\; \backslash quad\; \backslash text\{\; for\; all\; \}\; c\; \backslash in\; C.$$

But this last statement is true if and only if $\backslash langle\; \backslash ,p,\; c\backslash ,\; \backslash rangle\; =\; 0$ every $c\; \backslash in\; C.$ Thus $p\; \backslash in\; C^\{\backslash bot\}.$

$\backslash blacksquare$

{{collapse bottom}}

Expression as a global minimum

The statement and conclusion of the Hilbert projection theorem can be expressed in terms of global minimums of the followings functions. Their notation will also be used to simplify certain statements.

Given a non-empty subset $C\; \backslash subseteq\; H$ and some $x\; \backslash in\; H,$ define a function

$$d\_\{C,x\}\; :\; C\; \backslash to\; [0,\; \backslash infty)\; \backslash quad\; \backslash text\{\; by\; \}\; c\; \backslash mapsto\; \backslash |x\; -\; c\backslash |.$$

A {{em|global minimum point}} of $d\_\{C,x\},$ if one exists, is any point $m$ in $\backslash ,\backslash operatorname\{domain\}\; d\_\{C,x\}\; =\; C\backslash ,$ such that

$$d\_\{C,x\}(m)\; \backslash ,\backslash leq\backslash ,\; d\_\{C,x\}(c)\; \backslash quad\; \backslash text\{\; for\; all\; \}\; c\; \backslash in\; C,$$

in which case $d\_\{C,x\}(m)\; =\; \backslash |m\; -\; x\backslash |$ is equal to the {{em|global minimum value}} of the function $d\_\{C,\; x\},$ which is:

$$\backslash inf\_\{c\; \backslash in\; C\}\; d\_\{C,x\}(c)\; =\; \backslash inf\_\{c\; \backslash in\; C\}\; \backslash |x\; -\; c\backslash |.$$

Effects of translations and scalings

When this global minimum point $m$ exists and is unique then denote it by $\backslash min(C,\; x);$ explicitly, the defining properties of $\backslash min(C,\; x)$ (if it exists) are:

$$\backslash min(C,\; x)\; \backslash in\; C\; \backslash quad\; \backslash text\; \{\; and\; \}\; \backslash quad\; \backslash left\backslash |x\; -\; \backslash min(C,\; x)\backslash right\backslash |\; \backslash leq\; \backslash |x\; -\; c\backslash |\; \backslash quad\; \backslash text\{\; for\; all\; \}\; c\; \backslash in\; C.$$

The Hilbert projection theorem guarantees that this unique minimum point exists whenever $C$ is a non-empty closed and convex subset of a Hilbert space.

However, such a minimum point can also exist in non-convex or non-closed subsets as well; for instance, just as long is $C$ is non-empty, if $x\; \backslash in\; C$ then $\backslash min(C,\; x)\; =\; x.$

If $C\; \backslash subseteq\; H$ is a non-empty subset, $s$ is any scalar, and $x,\; x\_0\; \backslash in\; H$ are any vectors then

$$\backslash ,\backslash min\backslash left(s\; C\; +\; x\_0,\; s\; x\; +\; x\_0\backslash right)\; =\; s\; \backslash min(C,\; x)\; +\; x\_0$$

which implies:

$$\backslash begin\{alignat\}\{6\}$$

\min&(s C, s x) &&= s &&\min(C, x) \\

\min&(- C, - x) &&= - &&\min(C, x) \\

\end{alignat}

$$\backslash begin\{alignat\}\{6\}$$

\min\left(C + x_0, x + x_0\right) &= \min(C, x) + x_0 \\

\min\left(C - x_0, x - x_0\right) &= \min(C, x) - x_0 \\

\end{alignat}

$$\backslash begin\{alignat\}\{6\}$$

\min&(C, - x) {} &&= \min(C + x, 0) - x \\

\min&(C, 0) \;+\; x\;\;\;\; &&= \min(C + x, x) \\

\min&(C - x, 0) {} &&= \min(C, x) - x \\

\end{alignat}

Examples

The following counter-example demonstrates a continuous linear isomorphism $A\; :\; H\; \backslash to\; H$ for which $\backslash ,\backslash min(A(C),\; A(x))\; \backslash neq\; A(\backslash min(C,\; x)).$

Endow $H\; :=\; \backslash R^2$ with the dot product, let $x\_0\; :=\; (0,\; 1),$ and for every real $s\; \backslash in\; \backslash R,$ let $L\_s\; :=\; \backslash \{\; (x,\; s\; x)\; :\; x\; \backslash in\; \backslash R\; \backslash \}$ be the line of slope $s$ through the origin, where it is readily verified that $\backslash min\backslash left(L\_s,\; x\_0\backslash right)\; =\; \backslash frac\{s\}\{1+s^2\}(1,\; s).$

Pick a real number $r\; \backslash neq\; 0$ and define $A\; :\; \backslash R^2\; \backslash to\; \backslash R^2$ by $A(x,\; y)\; :=\; (r\; x,\; y)$ (so this map scales the $x-$coordinate by $r$ while leaving the $y-$coordinate unchanged).

Then $A\; :\; \backslash R^2\; \backslash to\; \backslash R^2$ is an invertible continuous linear operator that satisfies $A\backslash left(L\_s\backslash right)\; =\; L\_\{s/r\}$ and $A\backslash left(x\_0\backslash right)\; =\; x\_0,$

so that $\backslash ,\backslash min\backslash left(A\backslash left(L\_s\backslash right),\; A\backslash left(x\_0\backslash right)\backslash right)\; =\; \backslash frac\{s\}\{r^2\; +\; s^2\}\; (1,\; s)$ and $A\backslash left(\backslash min\backslash left(L\_s,\; x\_0\backslash right)\backslash right)\; =\; \backslash frac\{s\}\{1\; +\; s^2\}\; \backslash left(r,\; s\backslash right).$

Consequently, if $C\; :=\; L\_s$ with $s\; \backslash neq\; 0$ and if $(r,\; s)\; \backslash neq\; (\backslash pm\; 1,\; 1)$ then $\backslash ,\backslash min(A(C),\; A\backslash left(x\_0\backslash right))\; \backslash neq\; A\backslash left(\backslash min\backslash left(C,\; x\_0\backslash right)\backslash right).$

• {{annotated link|Orthogonal complement}}
• {{annotated link|Orthogonal projection}}
• {{annotated link|Orthogonality principle}}
• {{annotated link|Riesz representation theorem}}

{{reflist|group=note}}

{{reflist}}

• {{cite book|last=Rudin|first=Walter|authorlink=Walter Rudin|title=Real and Complex Analysis|edition=Third|year=1987}}
• {{Rudin Walter Functional Analysis|edition=2}}

{{Functional analysis}}

{{Hilbert space}}

{{DEFAULTSORT:Hilbert Projection Theorem}}

Category:Convex analysis

Category:Theorems in functional analysis