4
$\begingroup$

The conditional expected time remaining for an event to occur seems to grow with waiting time. This seems either wrong or like some sort of paradox.

Let's take the Lomax distribution as an example. Assume that we are waiting for an event and the waiting time for events is distributed as

$$f(x)=\frac{a}{b}\left[1+\frac{x}{b}\right]^{-(a+1)}$$

with $a>2$ and $b>0$. Also assume that we have observed at time $T$ that the event has not yet occurred. The conditional density for the waiting time becomes

$$f(x\mid X\geq T)=\frac{f(x)}{f(x\geq T)} = \frac{f(x)}{1-f(x< T)} = \frac{\frac{a}{b} \left[1+\frac{x}{b}\right]^{-(a+1)}}{\left[1+\frac{T}{b}\right]^{-a}}$$

The expected value of the waiting time given we have observed the event has not occurred at time $T$ is

\begin{align} & E[X\mid x\geq T]=\int_T^\infty xg(x\mid x\geq T) \\[8pt] = {} & \left[1+\frac{T}{b}\right]^a \int_T^\infty \frac{a}{b} x \left[1+\frac{x}{b}\right]^{-(a+1)} \, dx \end{align}

The integral can be evaluated using integration by parts.

\begin{align} & \int_T^\infty \frac{a}{b} x\left[1+\frac{x}{b}\right]^{-(a+1)} \, dx \\[8pt] = {} & \left.-x\left(1+\frac{x}{b} \right)^{-a} \right\vert_T^\infty + \left. \frac{b}{-a+1} \left(1+\frac{x}{b}\right)^{-a+1} \right\vert_T^\infty \\[8pt] = {} & T\left(1+\frac{T}{b}\right)^{-a}-\frac{b}{-a+1} \left(1+\frac{T}{b}\right)^{-a+1} \end{align}

I have used the following manipulation to evaluate the expression at $\infty$. \begin{align} & x\left(1+\frac{x}{b}\right)^{-a} = x\left(\frac{b+x}{b} \right)^{-a} \\[8pt] = {} & x\left(\frac{b}{b+x}\right)^a = b^a \frac{x}{(b+x)^a}<b^a \frac{x}{x^a} \\[8pt] = {} & b^a \frac{1}{x^{a-1}} \end{align}

It should be clear that this goes to zero as $x\rightarrow\infty$ for $a>2$ and $b>0$.

Then finally the expected value is

$$\left[1+\frac{T}{b}\right]^a \left[T\left(1+\frac{T}{b}\right)^{-a} - \frac{b}{-a+1} \left(1+\frac{T}{b}\right)^{-a+1}\right] = T+\frac{b}{a-1} \left[1+\frac{T}{b} \right]^a$$

If we care about the waiting time after the observation as opposed to the total waiting time that is

$$\frac{b}{a-1}\left[1+\frac{T}{b}\right]^a$$

So the expected additional waiting time after we take note the event has not occurred yet grows, at minimum, quadratically ($a>2$) with the time of our observation. This seems very counter intuitive to me. Typically if you've waited a while for an event to occur and it hasn't yet, you assume it will occur soon.

I'm wondering if I've missed something or messed something up? Maybe this is the result of the Lomax distribution having a heavy tail? Is there an intuitive explanation for this result?

This same behavior seems to apply to a conditional Weibull distribution as well (I seem to recall seeing this during numerical simulation a few years ago). Given this is the case, why are the Lomax and Weibull distributions used in fields such as actuarial sciences and reliability modeling where I'd imagine conditional waiting time is important, and such a paradox would hinder the business goal (I think)?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ The longer you wait, the more likely it becomes that the event is a rare one: what is counter-intuitive about that? Regardless, when there appears to be a conflict between your intuition and your model, then you shouldn't just question the intuition: maybe the model's not the right one for the phenomenon you are reasoning about. $\endgroup$ Commented Sep 9, 2024 at 13:35
  • $\begingroup$ This is true for the exponential distribution as well. However, conditional expected time remaining for that distribution is constant which is an intuitive result to me. (stats.stackexchange.com/questions/48496/…) It sounds like you are saying the math is right and the polynomial growth is a result of assumed distribution? $\endgroup$ Commented Sep 9, 2024 at 14:20
  • 6
    $\begingroup$ Math is math--it means nothing by itself. When you assume a waiting time distribution as a model for a phenomenon, then you have either to accept its mathematical consequences or conclude that distribution is not an accurate model. $\endgroup$ Commented Sep 9, 2024 at 14:31
  • $\begingroup$ What is needed in order to address this question are some intuitive examples of natural events about which one would normally become less certain of their happening soon if one has waited a long time for them to happen. $\endgroup$ Commented Feb 25 at 17:38
  • 1
    $\begingroup$ I think the counterintuitive part comes from ppl (myself included :) ) have a harder time to think in hazard rates / addl waiting time and ergodicity/expected time vs actual passage of time @kjetilbhalvorsen Your Facebook post example is intuitive; if I consider the textbook example of bus waiting time, I also find it counterintuitive given that the actual time the bus arrives is not changing just because I have been waiting for it ( or better to whuber's point: the wrong model assumption). Easy to make more counterintuitive examples ( like health recovery) $\endgroup$ Commented Mar 6 at 14:33

1 Answer 1

Reset to default
2
+50
$\begingroup$

It might be simpler to think about this in terms of the hazard function (aka failure rate):

The hazard function $h$ is defined as the event rate at time $t$, conditional on survival at time $t$.

Then the Weibull distribution illustrates the issues well. From this NIST web page, the hazard function (in that page's parameterization) for the standard distribution is:

$$ h(x) = \gamma x ^{(\gamma -1)}$$

With $\gamma=1$, you have a standard exponential distribution with constant hazard (and constant conditional waiting time). A comment indicates that you find that "intuitive."

If $\gamma<1$, then the hazard (failure rate) decreases with time. Wikipedia gives this example:

This happens if there is significant "infant mortality", or defective items failing early and the failure rate decreasing over time as the defective items are weeded out of the population.

That seems to be the situation described in your question.

If $\gamma >1$, then the hazard increases with time. Wikipedia continues with the following example:

This happens if there is an "aging" process, or parts that are more likely to fail as time goes on.

I now find that situation distressingly intuitive.

In general, how the "conditional remaining time to event" changes with time depends on the specifics of the distribution of event times.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The hazard function for the Lomax distribution does indeed decrease with time. $\endgroup$ Commented Mar 4 at 15:37

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.