variable elimination

Enabling a key reduction in algorithmic complexity, a factor $f$, also known as a potential, of variables $V$ is a relation between each instantiation of $v$ of variables $f$ to a non-negative number, commonly denoted as $f(x)$.{{Cite book|title=Modeling and Reasoning with Bayesian Networks|last=Darwiche|first=Adnan|date=2009-01-01|isbn=9780511811357|doi=10.1017/cbo9780511811357}} A factor does not necessarily have a set interpretation. One may perform operations on factors of different representations such as a probability distribution or conditional distribution. Joint distributions often become too large to handle as the complexity of this operation is exponential. Thus variable elimination becomes more feasible when computing factorized entities.

Algorithm 1, called sum-out (SO), or marginalization, eliminates a single variable $v$ from a set $\backslash phi$ of factors,Koller, D., Friedman, N.: Probabilistic Graphical Models: Principles and Techniques. MIT Press, Cambridge, MA (2009) and returns the resulting set of factors. The algorithm collect-relevant simply returns those factors in $\backslash phi$ involving variable $v$.

Algorithm 1 sum-out($v$,$\backslash phi$)

:$\backslash Phi$ = collect factors relevant to $v$

:$\backslash Psi$ = the product of all factors in $\backslash Phi$

:$\backslash tau\; =\; \backslash sum\_\{v\}\; \backslash Psi$

return $(\backslash phi\; -\; \backslash Phi)\; \backslash cup\; \backslash \{\backslash tau\backslash \}$

Example

Here we have a joint probability distribution. A variable, $v$ can be summed out between a set of instantiations where the set $V-v$ at minimum must agree over the remaining variables. The value of $v$ is irrelevant when it is the variable to be summed out.

After eliminating $V\_1$, its reference is excluded and we are left with a distribution only over the remaining variables and the sum of each instantiation.

The resulting distribution which follows the sum-out operation only helps to answer queries that do not mention $V\_1$. Also worthy to note, the summing-out operation is commutative.

Computing a product between multiple factors results in a factor compatible with a single instantiation in each factor.

Algorithm 2 mult-factors($v$,$\backslash phi$)

:$Z$ = Union of all variables between product of factors $f\_1(X\_1),...,f\_m(X\_m)$

:$f$ = a factor over $f$ where $f$ for all $f$

:For each instantiation $z$

::For 1 to $m$

:::$x\_1=$ instantiation of variables $X\_1$ consistent with $z$

:::$f(z)\; =\; f(z)f\_i(x\_i)$

:return $f$

Factor multiplication is not only commutative but also associative.

The most common query type is in the form $p(X|E\; =\; e)$ where $X$ and $E$ are disjoint subsets of $U$, and $E$ is observed taking value $e$. A basic algorithm to computing p(X|E = e) is called variable elimination (VE), first put forth in.

Taken from, this algorithm computes $p(X|E\; =\; e)$ from a discrete Bayesian network B. VE calls SO to eliminate variables one by one. More specifically, in Algorithm 2, $\backslash phi$ is the set C of conditional probability tables (henceforth "CPTs") for B, $X$ is a list of query variables, $E$ is a list of observed variables, $e$ is the corresponding list of observed values, and $\backslash sigma$ is an elimination ordering for variables $U\; -\; XE$, where $XE$ denotes $X\; \backslash cup\; E$.

Variable Elimination Algorithm VE($\backslash phi,\; X,\; E,\; e,\; \backslash sigma$)

:Multiply factors with appropriate CPTs while σ is not empty

:Remove the first variable $v$ from $\backslash sigma$

:$\backslash phi$ = sum-out$(v,\backslash phi)$

:$p(X,\; E\; =\; e)$ = the product of all factors $\backslash Psi\; \backslash in\; \backslash phi$

return $p(X,E\; =\; e)/\; \backslash sum\_\{X\}\; p(X,E\; =\; e)$

Finding the optimal order in which to eliminate variables is an NP-hard problem. As such there are heuristics one may follow to better optimize performance by order:

1. Minimum Degree: Eliminate the variable which results in constructing the smallest factor possible.
2. Minimum Fill: By constructing an undirected graph showing variable relations expressed by all CPTs, eliminate the variable which would result in the least edges to be added post elimination.

{{Reflist}}

Category:Graphical models