Markov renewal on-off processes, also known as alternating renewal processes or telegraph processes, arise in applications in a variety of fields. Classic alternating renewal processes were first studied in [2, 12] and [11] with applications to animal ethology, maintenance of electronic equipment, and communication engineering, respectively. Recently, the telegraph process and their variations are employed in various fields of applications such as pricing in mathematical finance and insurance (see, e.g., [9, 6–8, 16]), modelling the propagation of a damped wave in physics (see, e.g., [4, 10]), and inventory and storage models in engineering [18]. In certain applications, the off-state of an on-off process has multiple types, which raises new questions about its properties such as the distribution of the occupation time in specific off-state type marginally or multiple off-state types jointly. Our interest in on-off processes with multiple off-states is motivated by our recent work in animal movement modeling, where a predator can have different nonmoving states such as resting or handling a kill [15, 14]. The occupation time in the handling state is important for ecologists to understand the behavior of predators.

To fix ideas, consider a server that has two different types of failures, each requiring a different time and cost to repair. One basic question is: if the server is on at time 0, what is the distribution of the time spent on fixing the type one failures by time $t>0$? We model the server state process with the following Markov renewal process. The process starts in state 0 (the on-state) and spends there a random holding time according to a given absolutely continuous distribution. When the first holding time is over, we flip an asymmetric coin to decide which type of failure (off-state 1 or off-state 2) comes next. The holding times in the off-states are also absolutely continuous, generally with different distributions. Once the second holding time is over the process returns to the on-state (state 0) and then we repeat the construction process.

The occupation time of a specific type of the off-state is our focus. For the on-state, the distribution of the occupation time is known from the results on the telegraph process [13, 5, 17, 19]. Indeed, if we collapse the two off-states into one, then the resulting process is a regular on-off process (or alternating renewal process) whose off-state holding time distribution is a mixture of the two original off-state holding time distributions. If the different types of repair require different resources, however, we want to know the distribution of the occupation time in a particular type of off-state. For that task, the results on classical telegraph processes are not directly applicable. Moreover, assume that the cost for being in an off-state is proportional to its occupation time and different for different off-states. If one wants to know the distribution of the total cost for repairs of all kinds by a fixed time, then we need the joint distribution of the two types of off-state occupation times.

Our problem is related to but different from some recent works. An extension of the telegraph process to a process with three states is studied in [1], where within a renewal cycle all three states are visited in a given deterministic order. In our case, however, we have an on-off process with two off-states; only two states (the on-state and an off-state) are visited in each renewal cycle, but the off-states is chosen randomly. Another related work is [3], where there are two states, but at each random epoch, the new state is determined by the outcome of a random trial, and, as a result, the process can stay in the same state.

The key idea to study the occupation time of a specific off-state is to exploit a certain periodicity of our Markov renewal process. In fact, when analyzing any Markov renewal process, it is always convenient to do the conditioning on returning to a certain state. For the on-off process with two off-states, state 0 has an additional nice property. The numbers of steps between two consecutive visits of state 0 is not random, and it is always equal to 2. We derive the marginal distribution of the occupation time of a specific off-state first, and then we modify our derivation to obtain the joint distribution of the two off-state occupation times. Of course, one can get the marginal distribution by integrating the joint one. Our approach, however, is easier to follow.

This article is arranged as follows. In Section 2, the formal construction of on-off process with two off-states and the definition of its occupation time are provided. The marginal and joint distribution of off-states occupation time are derived in Section 3 and 4. In Section 5, a special case with holding time modeled by Lévy distribution are discussed alongside with an application of server maintenance cost. Additional technical discussion is provided in Section 6.

Suppose that we are given the following collection of independent sequences of nonnegative random variables:

Now, we present our construction of the on-off process with two off-states, $\{X(t),t\ge 0\}$, with state-space $\{0,1,2\}$.

That is, the process starts in state 0 and stays in this state ${U_{1}}$ time units. Then it switches to state 1 or 2, which is decided by random variable ${\xi _{1}}$. Depending on the value of ${\xi _{1}}$, it stays in state 1 or 2 for ${S_{1}}$ or ${L_{1}}$ time units, respectively. Then the process switches back to state 0. The first renewal cycle is finished, and then the procedure is repeated again. A realization of the on-off process is given in Figure 1.

Note also that ${T_{n}}\to \infty $ with probability 1, because it is bounded from below by a sum of n iid positive random variables. This means that process $X(t)$ is well-defined for all $t>0$.

If all the holding times are exponentially distributed (with rates ${\lambda _{i}}$, $i=0,1,2$), then $X(t)$ is a continuous time Markov chain with the state space $\{0,1,2\}$ and initial distribution $\Pr (X(0)=0)=1$. Its transition rate matrix is given by where again ${p_{1}},{p_{2}},{\lambda _{0}},{\lambda _{1}},{\lambda _{2}}>0$ and ${p_{1}}+{p_{2}}=1$.

The occupation times in state 0, state 1, and state 2 are, respectively, The corresponding defective marginal densities of $S(t)$ and $L(t)$ are denoted as where $t\ge 0$, $0<s<t$, $j=0,1,2$. The defective joint two-dimensional density of $S(t)$ and $L(t)$ for $u,v>0$, $u+v<t$ is denoted as where $j=0,1,2$. The densities are defective in the following sense: since both occupation times $S(t)$ and $L(t)$ have an atom at 0, all three sums and are less than 1.

To derive the distribution of occupation time we will need some auxiliary random variables. Let $N(t)$ be the number of cycles (or returns to the on-state) by time t. Formally, for $n\ge 0$, Finally, let ${D_{k}}={\xi _{k}}{S_{k}}+(1-{\xi _{k}}){L_{k}}$, $k\ge 1$. These random variables are associated with the off-state holding time of the regular on-off process when two off-states are combined.

Our formulas include convolutions of different distributions. We will use the following notation. If we are given a cdf $G(\cdot )$ and its pdf $g(\cdot )$, then ${G^{(n)}}(\cdot )$ denotes n-fold convolution of $G(\cdot )$, and ${g^{(n)}}(\cdot )$ denotes the n-fold convolution of $g(\cdot )$. If we are given two cdfs $G(\cdot )$ and $H(\cdot )$ with pdfs $g(\cdot )$ and $h(\cdot )$, then $G\ast H(\cdot )$ denotes the convolution of cdfs $G(\cdot )$ and $H(\cdot )$, and $g\ast h(\cdot )$ denotes the convolution of pdfs $g(\cdot )$ and $h(\cdot )$.

We also use the following conventions. Any summation over the empty set is 0. Zero-fold convolution ${G^{(0)}}(\cdot )$ is the cdf of a random variable that is equal to 0 with probability 1. Finally, ${g^{(k)}}\ast {h^{(0)}}(\cdot )={g^{(k)}}(\cdot )$ for any $k\ge 1$.

Theorem 1 gives us the distribution of occupation time in state 1. Since both off-states enter our story in a completely symmetric way, the formulas for the occupation time in state 2 can be obtained by interchanging state 1 and 2 in Theorem 1.

As mentioned in the introduction, if we are interested in the distribution of the total cost, then we need the joint distribution of off-state occupation times. More specifically, assume that the total cost $C(t)$ is a linear function of occupation times, that is, where ${\alpha _{0}}$, ${\alpha _{1}}$, and ${\alpha _{2}}$ are positive constants representing the maintenance cost per unit of time for state 0, 1, and 2, respctively. Then the distribution of $C(t)$ is fully determined by the joint distribution of $S(t)$ and $L(t)$. Note that we do not need the joint distribution of all three occupation times, because $U(t)=t-S(t)-L(t)$.

Let us also note here that the cost function process $\{C(t),t\ge 0\}$ is similar to what is know in the literature as integrated telegraph process. Indeed, $C(t)$ is a linearly increasing process. It goes up with velocity ${\alpha _{0}}$, if $X(t)$ is in state 0. When the process $X(t)$ switches to one of two off-states, it goes up either with velocity ${\alpha _{0}}$ or ${\alpha _{2}}$ (depending on an off-state).

Since both $S(t)$ and $L(t)$ have atoms at 0, for every value of $X(t)$ we have four cases: (1) both occupation times are 0; (2) $S(t)=0$, $L(t)>0$; (3) $S(t)>0$, $L(t)=0$; and (4) both $S(t)$ and $L(t)$ are strictly greater than 0. In total, we have 12 formulas. Some of them are trivial. For instance, event $\{S(t)=0,X(t)=1\}$ has probability 0, therefore, the corresponding defective one-dimensional density of $L(t)$ is also 0. Moreover, since there is a certain symmetry between $S(t)$ and $L(t)$, some formulas can be found by interchanging state 1 and 2. That is why in the following theorem we have only 5 formulas.

One can verify now that, for instance, Using symmetry between $S(t)$ and $L(t)$, we also can get that

In this section we consider a special case when all the holding times have the Lévy distribution (with location parameter 0 and possibly different scale parameters). The Lévy distribution with scale parameter ${c^{2}}$ has pdf and cdf The Lévy distribution is heavy-tailed with infinite expectation. The median is given by $0.5{c^{2}}{\left({\mathtt{erfc}^{-1}}(0.5)\right)^{-2}}\approx 2.198112{c^{2}}$, where $\mathtt{erfc}$ is the complementary error function:

The parametrization that we use is a bit unusual, but it allows us to shorten our notation for convolutions. More specifically, as a member of the family of stable distributions with mobility parameter 1/2, the Lévy distribution is closed under the operation of convolution in the following way:

Let ${c_{U}^{2}}$, ${c_{S}^{2}}$, and ${c_{L}^{2}}$ be the scale parameters for states 0, 1 and 2, respectively. Then the formulas from Theorem 1 and Theorem 2 do not involve any convolutions. For instance, in this case we have that The expectations and variances of the off-times is given by Note that the discrete component of the occupation times is not used for these calculations, because the atoms are at 0. The covariance of $S(t)$ and $L(t)$ is given by

As an example we consider a server with median holding time 7 days, 0.5 hour, and 4 hours for the on-state, the short off-state, and the long off-state, respectively. Using one day as the time unit, the model parameters are: ${c_{U}}\approx 1.785$, ${c_{S}}\approx 0.097$, and ${c_{L}}\approx 0.275$. Assume also that the total repair cost is given by that is, long repairs are twice costlier than the short ones.

Figure 2 presents defective marginal densities ${p_{Sj}}(s,t)$ and ${p_{Lj}}(s,t)$ when $t=30$ days and ${p_{1}}=.9$ (that is, the less serious breakdowns occur 9 times more often). The marginal densities for both occupation times are severely defected when $X(30)=0$ (the process is in the on-state). As we mentioned above $S(30)$ and $L(30)$ have atoms at $s=0$:

Figure 3 shows the contour plot of defective joint densities ${p_{SLj}}(u,v,t)$ with the same parameter setup, where the negative association of the two occupation times is obvious. We also ran a simulation study to numerically confirm the correctness of Theorem 1 and Theorem 2. A total $1,000,000$ realizations of the above on-off process were generated and the empirical results are consistent with the theoretical densities (not shown).

Table 1 summarizes the expectation and variance of the two types of off-state occupation times and the associated repair cost for $t\in \{30,60\}$ days and ${p_{1}}\in \{0.70,0.75,\dots ,0.95,0.99\}$. As ${p_{1}}$ increases, the expectation and the variance goes up for the less serious breakdown time but goes down for the more serious breakdown time. The total repair cost has lower expectation and variance for smaller values of ${p_{1}}$. The correlation of the two off-state occupation times is negative, with a magnitude decreasing as ${p_{1}}$ increases. When t is doubled, the expectation of the occupation times and the total repair cost is slightly more than doubled, which is because the process always starts from the on-state with a random holding time.

The marginal distribution of an off-state occupation time could be obtained by collapsing the on-state with the other off-state and using results on the telegraph process. Nonetheless, this approach would lead to more complicated formulas because the holding time of the new collapsed state is distributed as an infinite mixture of convolutions. Further, this technique cannot be employed if we need the joint distribution of the two off-state occupation times.

The total repair cost process $C(t)$ also can be viewed as a telegraph process governed by the on-off process with multiple off-states. That is, $C(t)$ is the time t position of a particle that moves with speed ${\alpha _{j}}$ whenever $X(t)=j$, $j=0,1,2$. The formulas for the joint distribution of $X(t)$ and occupation times can be useful for parameter estimation when process $C(t)$ is discretely observed. In particular, if $X(t)$ is also Markov (that is, all the holding times are exponential), then the efficient likelihood estimation is possible with help of tools for hidden Markov model [14].