Convex optimization#linear

A convex optimization problem is defined by two ingredients:{{cite book |last1=Hiriart-Urruty |first1=Jean-Baptiste |url=https://books.google.com/books?id=Gdl4Jc3RVjcC&q=lemarechal+convex+analysis+and+minimization |title=Convex analysis and minimization algorithms: Fundamentals |last2=Lemaréchal |first2=Claude |year=1996 |isbn=9783540568506 |page=291|publisher=Springer }}{{cite book |last1=Ben-Tal |first1=Aharon |url=https://books.google.com/books?id=M3MqpEJ3jzQC&q=Lectures+on+Modern+Convex+Optimization:+Analysis,+Algorithms, |title=Lectures on modern convex optimization: analysis, algorithms, and engineering applications |last2=Nemirovskiĭ |first2=Arkadiĭ Semenovich |year=2001 |isbn=9780898714913 |pages=335–336}}

• The objective function, which is a real-valued convex function of n variables, $f\; :\backslash mathcal\; D\; \backslash subseteq\; \backslash mathbb\{R\}^n\; \backslash to\; \backslash mathbb\{R\}$;
• The feasible set, which is a convex subset $C\backslash subseteq\; \backslash mathbb\{R\}^n$.

The goal of the problem is to find some $\backslash mathbf\{x^\backslash ast\}\; \backslash in\; C$ attaining

:$\backslash inf\; \backslash \{\; f(\backslash mathbf\{x\})\; :\; \backslash mathbf\{x\}\; \backslash in\; C\; \backslash \}$.

In general, there are three options regarding the existence of a solution:{{Cite book |last1=Boyd |first1=Stephen |url=https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf |title=Convex Optimization |last2=Vandenberghe |first2=Lieven |publisher=Cambridge University Press |year=2004 |isbn=978-0-521-83378-3 |access-date=12 Apr 2021}}{{Rp|location=chpt.4}}

• If such a point x* exists, it is referred to as an optimal point or solution; the set of all optimal points is called the optimal set; and the problem is called solvable.
• If $f$ is unbounded below over $C$, or the infimum is not attained, then the optimization problem is said to be unbounded.
• Otherwise, if $C$ is the empty set, then the problem is said to be infeasible.

A convex optimization problem is in standard form if it is written as

:$\backslash begin\{align\}$

&\underset{\mathbf{x}}{\operatorname{minimize}}& & f(\mathbf{x}) \\

&\operatorname{subject\ to}

& &g_i(\mathbf{x}) \leq 0, \quad i = 1, \dots, m \\

&&&h_i(\mathbf{x}) = 0, \quad i = 1, \dots, p,

\end{align}

where:{{Rp|location=chpt.4}}

• $\backslash mathbf\{x\}\; \backslash in\; \backslash mathbb\{R\}^n$ is the vector of optimization variables;
• The objective function $f:\; \backslash mathcal\; D\; \backslash subseteq\; \backslash mathbb\{R\}^n\; \backslash to\; \backslash mathbb\{R\}$ is a convex function;
• The inequality constraint functions $g\_i\; :\; \backslash mathbb\{R\}^n\; \backslash to\; \backslash mathbb\{R\}$, $i=1,\; \backslash ldots,\; m$, are convex functions;
• The equality constraint functions $h\_i\; :\; \backslash mathbb\{R\}^n\; \backslash to\; \backslash mathbb\{R\}$, $i=1,\; \backslash ldots,\; p$, are affine transformations, that is, of the form: $h\_i(\backslash mathbf\{x\})\; =\; \backslash mathbf\{a\_i\}\backslash cdot\; \backslash mathbf\{x\}\; -\; b\_i$, where $\backslash mathbf\{a\_i\}$ is a vector and $b\_i$ is a scalar.

The feasible set $C$ of the optimization problem consists of all points $\backslash mathbf\{x\}\; \backslash in\; \backslash mathcal\{D\}$ satisfying the inequality and the equality constraints. This set is convex because $\backslash mathcal\{D\}$ is convex, the sublevel sets of convex functions are convex, affine sets are convex, and the intersection of convex sets is convex.{{Rp|location=chpt.2}}

Many optimization problems can be equivalently formulated in this standard form. For example, the problem of maximizing a concave function $f$ can be re-formulated equivalently as the problem of minimizing the convex function $-f$. The problem of maximizing a concave function over a convex set is commonly called a convex optimization problem.{{cite web | url=https://www.solver.com/convex-optimization | title=Optimization Problem Types - Convex Optimization | date=9 January 2011 }}

In the standard form it is possible to assume, without loss of generality, that the objective function f is a linear function. This is because any program with a general objective can be transformed into a program with a linear objective by adding a single variable t and a single constraint, as follows:{{Cite book |last=Arkadi Nemirovsky |url=https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=8c3cb6395a35cb504019f87f447d65cb6cf1cdf0 |title=Interior point polynomial-time methods in convex programming |year=2004}}{{Rp|location=1.4}}

:$\backslash begin\{align\}$

&\underset{\mathbf{x},t}{\operatorname{minimize}}& & t \\

&\operatorname{subject\ to}

& &f(\mathbf{x}) - t \leq 0 \\

&& &g_i(\mathbf{x}) \leq 0, \quad i = 1, \dots, m \\

&&&h_i(\mathbf{x}) = 0, \quad i = 1, \dots, p,

\end{align}

Every convex program can be presented in a conic form, which means minimizing a linear objective over the intersection of an affine plane and a convex cone:{{Rp|location=5.1}}

:$\backslash begin\{align\}$

&\underset{\mathbf{x}}{\operatorname{minimize}}& & c^T x \\

&\operatorname{subject\ to}

& &x \in (b+L)\cap K

\end{align}

where K is a closed pointed convex cone, L is a linear subspace of Rⁿ, and b is a vector in Rⁿ. A linear program in standard form is the special case in which K is the nonnegative orthant of Rⁿ.

It is possible to convert a convex program in standard form, to a convex program with no equality constraints.{{Rp|page=132}} Denote the equality constraints h_i(x)=0 as Ax=b, where A has n columns. If Ax=b is infeasible, then of course the original problem is infeasible. Otherwise, it has some solution x₀ , and the set of all solutions can be presented as: Fz+x₀, where z is in R^k, k=n-rank(A), and F is an n-by-k matrix. Substituting x = Fz+x₀ in the original problem gives:

$\backslash begin\{align\}$

&\underset{\mathbf{x}}{\operatorname{minimize}}& & f(\mathbf{F \mathbf{z} + \mathbf{x}_0}) \\

&\operatorname{subject\ to}

& &g_i(\mathbf{F \mathbf{z} + \mathbf{x}_0}) \leq 0, \quad i = 1, \dots, m \\

\end{align}

The following problem classes are all convex optimization problems, or can be reduced to convex optimization problems via simple transformations:{{Rp|location=chpt.4}}

{{cite journal |last1=Agrawal |first1=Akshay |last2=Verschueren |first2=Robin |last3=Diamond |first3=Steven |last4=Boyd |first4=Stephen |year=2018 |title=A rewriting system for convex optimization problems |url=https://web.stanford.edu/~boyd/papers/pdf/cvxpy_rewriting.pdf |journal=Control and Decision |volume=5 |issue=1 |pages=42–60 |arxiv=1709.04494 |doi=10.1080/23307706.2017.1397554 |s2cid=67856259}}

File:Hierarchy compact convex.png, QP: quadratic programming, SOCP second-order cone program, SDP: semidefinite programming, CP: conic optimization.)]]

• Linear programming problems are the simplest convex programs. In LP, the objective and constraint functions are all linear.
• Quadratic programming are the next-simplest. In QP, the constraints are all linear, but the objective may be a convex quadratic function.
• Second order cone programming are more general.
• Semidefinite programming are more general.
• Conic optimization are even more general - see figure to the right,

Other special cases include;

• Least squares
• Quadratic minimization with convex quadratic constraints
• Geometric programming
• Entropy maximization with appropriate constraints.

The following are useful properties of convex optimization problems:{{cite journal | author = Rockafellar, R. Tyrrell | title = Lagrange multipliers and optimality | journal = SIAM Review | volume = 35 | issue = 2 | year = 1993 | pages = 183–238 |url = http://web.williams.edu/Mathematics/sjmiller/public_html/105Sp10/handouts/Rockafellar_LagrangeMultAndOptimality.pdf | doi=10.1137/1035044| citeseerx = 10.1.1.161.7209}}{{Rp|location=chpt.4}}

• every local minimum is a global minimum;
• the optimal set is convex;
• if the objective function is strictly convex, then the problem has at most one optimal point.

These results are used by the theory of convex minimization along with geometric notions from functional analysis (in Hilbert spaces) such as the Hilbert projection theorem, the separating hyperplane theorem, and Farkas' lemma.{{citation needed|date=July 2022}}

The convex programs easiest to solve are the unconstrained problems, or the problems with only equality constraints. As the equality constraints are all linear, they can be eliminated with linear algebra and integrated into the objective, thus converting an equality-constrained problem into an unconstrained one.

In the class of unconstrained (or equality-constrained) problems, the simplest ones are those in which the objective is quadratic. For these problems, the KKT conditions (which are necessary for optimality) are all linear, so they can be solved analytically.{{Rp|location=chpt.11}}

For unconstrained (or equality-constrained) problems with a general convex objective that is twice-differentiable, Newton's method can be used. It can be seen as reducing a general unconstrained convex problem, to a sequence of quadratic problems.{{Rp|location=chpt.11}}Newton's method can be combined with line search for an appropriate step size, and it can be mathematically proven to converge quickly.

Other efficient algorithms for unconstrained minimization are gradient descent (a special case of steepest descent).

The more challenging problems are those with inequality constraints. A common way to solve them is to reduce them to unconstrained problems by adding a barrier function, enforcing the inequality constraints, to the objective function. Such methods are called interior point methods.{{Rp|location=chpt.11}}They have to be initialized by finding a feasible interior point using by so-called phase I methods, which either find a feasible point or show that none exist. Phase I methods generally consist of reducing the search in question to a simpler convex optimization problem.{{Rp|location=chpt.11}}

Convex optimization problems can also be solved by the following contemporary methods:For methods for convex minimization, see the volumes by Hiriart-Urruty and Lemaréchal (bundle) and the textbooks by Ruszczyński, Bertsekas, and

Boyd and Vandenberghe (interior point).

• Bundle methods (Wolfe, Lemaréchal, Kiwiel), and
• Subgradient projection methods (Polyak),
• Interior-point methods, which make use of self-concordant barrier functions {{cite book |title=Interior-Point Polynomial Algorithms in Convex Programming |last1= Nesterov |first1=Yurii |first2=Nemirovskii |last2=Arkadii |year=1995 |publisher=Society for Industrial and Applied Mathematics |isbn=978-0898715156 }} and self-regular barrier functions.{{cite journal|last1=Peng|first1=Jiming|last2=Roos|first2=Cornelis|last3=Terlaky|first3=Tamás|title=Self-regular functions and new search directions for linear and semidefinite optimization|journal=Mathematical Programming|volume=93|issue=1|year=2002|pages=129–171|issn=0025-5610|doi=10.1007/s101070200296|s2cid=28882966}}
• Cutting-plane methods
• Ellipsoid method
• Subgradient method
• Dual subgradients and the drift-plus-penalty method

Subgradient methods can be implemented simply and so are widely used.{{Cite journal |date=2006 |title=Numerical Optimization |url=https://link.springer.com/book/10.1007/978-0-387-40065-5 |journal=Springer Series in Operations Research and Financial Engineering |language=en |doi=10.1007/978-0-387-40065-5|isbn=978-0-387-30303-1 |url-access=subscription }} Dual subgradient methods are subgradient methods applied to a dual problem. The drift-plus-penalty method is similar to the dual subgradient method, but takes a time average of the primal variables.{{Citation needed|date=April 2021}}

{{Main|Lagrange multiplier}}

{{Unreferenced section|date=April 2021}}

Consider a convex minimization problem given in standard form by a cost function $f(x)$ and inequality constraints $g\_i(x)\backslash leq\; 0$ for $1\; \backslash leq\; i\; \backslash leq\; m$. Then the domain $\backslash mathcal\{X\}$ is:

:$\backslash mathcal\{X\}\; =\; \backslash left\backslash \{x\backslash in\; X\; \backslash vert\; g\_1(x),\; \backslash ldots,\; g\_m(x)\backslash leq\; 0\backslash right\backslash \}.$

The Lagrangian function for the problem is{{cite book |first1=Brian |last1=Beavis |first2=Ian M. |last2=Dobbs |chapter=Static Optimization |title=Optimization and Stability Theory for Economic Analysis |location=New York |publisher=Cambridge University Press |year=1990 |isbn=0-521-33605-8 |page=40 |chapter-url=https://books.google.com/books?id=L7HMACFgnXMC&pg=PA40 }}

:$L(x,\backslash lambda\_\{0\},\backslash lambda\_1,\; \backslash ldots\; ,\backslash lambda\_\{m\})=\backslash lambda\_\{0\}\; f(x)\; +\; \backslash lambda\_\{1\}\; g\_\{1\}\; (x)+\backslash cdots\; +\; \backslash lambda\_\{m\}\; g\_\{m\}\; (x).$

For each point $x$ in $X$ that minimizes $f$ over $X$, there exist real numbers $\backslash lambda\_\{0\},\backslash lambda\_1,\; \backslash ldots,\; \backslash lambda\_\{m\},$ called Lagrange multipliers, that satisfy these conditions simultaneously:

1. $x$ minimizes $L(y,\backslash lambda\_\{0\},\backslash lambda\_\{1\},\backslash ldots\; ,\backslash lambda\_\{m\})$ over all $y\; \backslash in\; X,$
2. $\backslash lambda\_\{0\},\backslash lambda\_\{1\},\backslash ldots\; ,\backslash lambda\_\{m\}\; \backslash geq\; 0,$ with at least one $\backslash lambda\_\{k\}\; >\; 0,$
3. $\backslash lambda\_\{1\}g\_\{1\}(x)=\backslash cdots\; =\; \backslash lambda\_\{m\}g\_\{m\}(x)\; =\; 0$ (complementary slackness).

If there exists a "strictly feasible point", that is, a point $z$ satisfying

:

then the statement above can be strengthened to require that $\backslash lambda\_\{0\}=1$.

Conversely, if some $x$ in $X$ satisfies (1)–(3) for scalars $\backslash lambda\_\{0\},\backslash ldots,\backslash lambda\_\{m\}$ with $\backslash lambda\_\{0\}=1$ then $x$ is certain to minimize $f$ over $X$.

There is a large software ecosystem for convex optimization. This ecosystem has two main categories: solvers on the one hand and modeling tools (or interfaces) on the other hand.

Solvers implement the algorithms themselves and are usually written in C. They require users to specify optimization problems in very specific formats which may not be natural from a modeling perspective. Modeling tools are separate pieces of software that let the user specify an optimization in higher-level syntax. They manage all transformations to and from the user's high-level model and the solver's input/output format.

Below are two tables. The first shows shows modelling tools (such as CVXPY and JuMP.jl) and the second solvers (such as SCS and MOSEK). They are by no means exhaustive.

Convex optimization can be used to model problems in a wide range of disciplines, such as automatic control systems, estimation and signal processing, communications and networks, electronic circuit design,{{Rp|location=|page=17}} data analysis and modeling, finance, statistics (optimal experimental design),Christensen/Klarbring, chpt. 4. and structural optimization, where the approximation concept has proven to be efficient.Schmit, L.A.; Fleury, C. 1980: Structural synthesis by combining approximation concepts and dual methods. J. Amer. Inst. Aeronaut. Astronaut 18, 1252-1260 Convex optimization can be used to model problems in the following fields:

• Portfolio optimization.{{Cite web |last1=Boyd |first1=Stephen |last2=Diamond |first2=Stephen |last3=Zhang |first3=Junzi |last4=Agrawal |first4=Akshay |title=Convex Optimization Applications |url=https://web.stanford.edu/~boyd/papers/pdf/cvx_applications.pdf |url-status=live |archive-url=https://web.archive.org/web/20151001185038/http://web.stanford.edu/~boyd/papers/pdf/cvx_applications.pdf |archive-date=2015-10-01 |access-date=12 Apr 2021}}
• Worst-case risk analysis.
• Optimal advertising.
• Variations of statistical regression (including regularization and quantile regression).
• Model fitting (particularly multiclass classification{{Cite web |last=Malick |first=Jérôme |date=2011-09-28 |title=Convex optimization: applications, formulations, relaxations |url=https://www-ljk.imag.fr//membres/Jerome.Malick/Talks/11-INRIA.pdf |url-status=live |archive-url=https://web.archive.org/web/20210412044738/https://www-ljk.imag.fr//membres/Jerome.Malick/Talks/11-INRIA.pdf |archive-date=2021-04-12 |access-date=12 Apr 2021}}).
• Electricity generation optimization.
• Combinatorial optimization.
• Non-probabilistic modelling of uncertainty.Ben Haim Y. and Elishakoff I., Convex Models of Uncertainty in Applied Mechanics, Elsevier Science Publishers, Amsterdam, 1990
• Localization using wireless signals Ahmad Bazzi, Dirk TM Slock, and Lisa Meilhac. "Online angle of arrival estimation in the presence of mutual coupling." 2016 IEEE Statistical Signal Processing Workshop (SSP). IEEE, 2016.

Extensions of convex optimization include the optimization of biconvex, pseudo-convex, and quasiconvex functions. Extensions of the theory of convex analysis and iterative methods for approximately solving non-convex minimization problems occur in the field of generalized convexity, also known as abstract convex analysis.{{Citation needed|date=April 2021}}

• Duality
• Karush–Kuhn–Tucker conditions
• Optimization problem
• Proximal gradient method
• Algorithmic problems on convex sets

• {{cite book

|last1=Bertsekas

|first1=Dimitri P.

|last2=Nedic

|first2=Angelia

|last3 = Ozdaglar

|first3 = Asuman

| title = Convex Analysis and Optimization

| publisher = Athena Scientific

| year = 2003

| location = Belmont, MA.

| isbn = 978-1-886529-45-8

}}

• {{cite book

| last = Bertsekas

| first = Dimitri P.

| author-link = Dimitri P. Bertsekas

| title = Convex Optimization Theory

| publisher = Athena Scientific

| year = 2009

| location = Belmont, MA.

| isbn = 978-1-886529-31-1

}}

• {{cite book

| last = Bertsekas

| first = Dimitri P.

| author-link = Dimitri P. Bertsekas

| title = Convex Optimization Algorithms

| publisher = Athena Scientific

| year = 2015

| location = Belmont, MA.

| isbn = 978-1-886529-28-1

}}

• {{Cite book|last1=Borwein|first1=Jonathan|url=https://carma.newcastle.edu.au/resources/jon/Preprints/Books/CaNo2/cano2f.pdf|title=Convex Analysis and Nonlinear Optimization: Theory and Examples, Second Edition|last2=Lewis|first2=Adrian|publisher=Springer|year=2000|access-date=12 Apr 2021}}
• {{cite book

| author1 = Christensen, Peter W.

| author2 = Anders Klarbring

| title = An introduction to structural optimization

| volume = 153

| publisher = Springer Science & Business Media

| year = 2008

| url = https://books.google.com/books?id=80IeN__MYI8C

| isbn = 9781402086663

}}

• Hiriart-Urruty, Jean-Baptiste, and Lemaréchal, Claude. (2004). Fundamentals of Convex analysis. Berlin: Springer.
• {{cite book|last1=Hiriart-Urruty|first1=Jean-Baptiste|last2=Lemaréchal|first2=Claude|title=Convex analysis and minimization algorithms, Volume I: Fundamentals|series=Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]|volume=305|publisher=Springer-Verlag|location=Berlin|year=1993|pages=xviii+417|isbn=978-3-540-56850-6|mr=1261420|author-link2=Claude Lemaréchal}}
• {{cite book|last1=Hiriart-Urruty|first1=Jean-Baptiste|last2=Lemaréchal|first2=Claude|title=Convex analysis and minimization algorithms, Volume II: Advanced theory and bundle methods|series=Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]|volume=306|publisher=Springer-Verlag|location=Berlin|year=1993|pages=xviii+346|isbn=978-3-540-56852-0|mr=1295240|author-link2=Claude Lemaréchal}}
• {{cite book|first=Krzysztof C.|last=Kiwiel|title=Methods of Descent for Nondifferentiable Optimization|url=https://archive.org/details/methodsofdescent0000kiwi|url-access=registration|year=1985|publisher=Springer-Verlag|location= New York|series=Lecture Notes in Mathematics|isbn=978-3-540-15642-0 }}
• {{cite book|last=Lemaréchal|first=Claude|s2cid=9048698|chapter=Lagrangian relaxation|pages=112–156|title=Computational combinatorial optimization: Papers from the Spring School held in Schloß Dagstuhl, May 15–19, 2000|editor=Michael Jünger and Denis Naddef|series=Lecture Notes in Computer Science|volume=2241|publisher=Springer-Verlag|location=Berlin|year=2001|isbn=978-3-540-42877-0|mr=1900016|doi=10.1007/3-540-45586-8_4|author-link=Claude Lemaréchal}}
• {{cite book |last1=Nesterov|first1=Yurii|last2=Nemirovskii|first2=Arkadii|year=1994|title=Interior Point Polynomial Methods in Convex Programming|publisher=SIAM

}}

• Nesterov, Yurii. (2004). [https://books.google.com/books?id=2-ElBQAAQBAJ&dq=%22Introductory+Lectures+on+Convex+Optimization%22&pg=PA1 Introductory Lectures on Convex Optimization], Kluwer Academic Publishers
• {{cite book

| last = Rockafellar

| first = R. T.

| author-link = R. Tyrrell Rockafellar

| title = Convex analysis

| publisher = Princeton University Press

| year = 1970

| location = Princeton

}}

• {{cite book

| last = Ruszczyński

| first = Andrzej

| author-link = Andrzej Piotr Ruszczyński

| title = Nonlinear Optimization

| publisher = Princeton University Press

| year = 2006

}}

• Schmit, L.A.; Fleury, C. 1980: Structural synthesis by combining approximation concepts and dual methods. J. Amer. Inst. Aeronaut. Astronaut 18, 1252-1260

{{Commons category}}

• [https://web.stanford.edu/class/ee364a/ EE364a: Convex Optimization I] and [https://web.stanford.edu/class/ee364b/ EE364b: Convex Optimization II], Stanford course homepages
• [https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-253-convex-analysis-and-optimization-spring-2012/lecture-notes/ 6.253: Convex Analysis and Optimization], an MIT OCW course homepage
• Brian Borchers, [https://web.archive.org/web/20170918180026/http://infohost.nmt.edu/~borchers/presentation.pdf An overview of software for convex optimization]
• [https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf Convex Optimization Book by Lieven Vandenberghe and Stephen P. Boyd]

{{Optimization algorithms|convex}}

{{Convex analysis and variational analysis}}

Category:Convex analysis

Category:Mathematical optimization