Markov renewal process

File:Marked point process.png

In the context of a jump process that takes states in a state space $\backslash mathrm\{S\}$, consider the set of random variables $(X\_n,T\_n)$, where $T\_n$ represents the jump times and $X\_n$ represents the associated states in the sequence of states (see Figure). Let the sequence of inter-arrival times $\backslash tau\_n=T\_n-T\_\{n-1\}$. In order for the sequence $(X\_n,T\_n)$ to be considered a Markov renewal process the following condition should hold:

$$

\begin{align}

& \Pr(\tau_{n+1}\le t, X_{n+1}=j\mid(X_0, T_0), (X_1, T_1),\ldots, (X_n=i, T_n)) \\[5pt]

= {} & \Pr(\tau_{n+1}\le t, X_{n+1}=j\mid X_n=i)\, \forall n \ge1,t\ge0, i,j \in \mathrm{S}

\end{align}

1. Let $X\_n$ and $T\_n$ be as defined in the previous statement. Defining a new stochastic process $Y\_t:=X\_n$ for $t\; \backslash in\; [T\_n,T\_\{n+1\})$, then the process $Y\_t$ is called a semi-Markov process as it happens in a continuous-time Markov chain. The process is Markovian only at the specified jump instants, justifying the name semi-Markov.{{cite book|last=Medhi|first=J.|title=Stochastic processes|year=1982|publisher=Wiley & Sons|location=New York|isbn=978-0-470-27000-4}}{{cite book|last=Ross|first=Sheldon M.|title=Stochastic processes.|year=1999|publisher=Routledge.|location=New York [u.a.]|isbn=978-0-471-12062-9|edition=2nd}}{{cite book|last=Barbu|first=Vlad Stefan|title=Semi-Markov chains and hidden semi-Markov models toward applications: their use in reliability and DNA analysis|year=2008|publisher=Springer|location=New York|isbn=978-0-387-73171-1|author2=Limnios, Nikolaos}} (See also: hidden semi-Markov model.)
2. A semi-Markov process (defined in the above bullet point) in which all the holding times are exponentially distributed is called a continuous-time Markov chain. In other words, if the inter-arrival times are exponentially distributed and if the waiting time in a state and the next state reached are independent, we have a continuous-time Markov chain.
3. :$\backslash begin\{align\}$

& \Pr(\tau_{n+1}\le t, X_{n+1}=j\mid(X_0, T_0), (X_1, T_1),\ldots, (X_n=i, T_n)) \\[3pt]

= {} & \Pr(\tau_{n+1}\le t, X_{n+1}=j\mid X_n=i) \\[3pt]

= {} & \Pr(X_{n+1}=j\mid X_n=i)(1-e^{-\lambda_i t}), \text{ for all } n \ge1,t\ge0, i,j \in \mathrm{S}, i \ne j

\end{align}

1. The sequence $X\_n$ in the Markov renewal process is a discrete-time Markov chain. In other words, if the time variables are ignored in the Markov renewal process equation, we end up with a discrete-time Markov chain.
2. :$\backslash Pr(X\_\{n+1\}=j\backslash mid\; X\_0,\; X\_1,\; \backslash ldots,\; X\_n=i)=\backslash Pr(X\_\{n+1\}=j\backslash mid\; X\_n=i)\backslash ,\; \backslash forall\; n\; \backslash ge1,\; i,j\; \backslash in\; \backslash mathrm\{S\}$
3. If the sequence of $\backslash tau$s is independent and identically distributed, and if their distribution does not depend on the state $X\_n$, then the process is a renewal. So, if the states are ignored and we have a chain of iid times, then we have a renewal process.
4. :$\backslash Pr(\backslash tau\_\{n+1\}\backslash le\; t\backslash mid\; T\_0,\; T\_1,\; \backslash ldots,\; T\_n)=\backslash Pr(\backslash tau\_\{n+1\}\backslash le\; t)\backslash ,\; \backslash forall\; n\; \backslash ge1,\; \backslash forall\; t\backslash ge0$

• Markov process
• Renewal theory
• Variable-order Markov model
• Hidden semi-Markov model

{{more footnotes|date=July 2012}}

{{Reflist}}

Category:Markov processes