All forecasts are wrong, but some are also inadequate

Forecasting is difficult, partly because all models are wrong (or because we are wrong about models). And while we usually don’t know—with certainty anyway—what’s wrong, we often can know what’s not quite right.

Specifically, we may not know the best model that can generate the most accurate forecast—not until we tried them all, so never, really. But we can know which of the models generates an inadequate forecast. Such are forecasts that are either biased or inefficient or both. A forecast that doesn’t suffer from these deficiencies is said to be optimal (or adequate, or rational).

An optimal forecast, in the world of Mincer and Zarnowitz, implies that if we were to regress the realization of a variable on its forecast, the intercept coefficient (alpha) should be zero, and the slope coefficient (beta) should be one, in the following equation:

\(y_{t+h}=\alpha+\beta \hat{y}_{t+h|t}+\varepsilon_{t+h}\)

Here, the left-hand side variable is the realization of the variable (observed ex-post), and the right-hand side variable is the forecast of this variable made in period t (using available information at that time) for period t+h, where h is the horizon of a forecast. The last term is an independent and identically distributed (iid) error, which accounts for our inability to make a perfect forecast. That’s not a problem, at least not from the standpoint of generating an optimal forecast.

A problem is when either alpha is different from zero or beta is different from one (or both). That would suggest the inadequacy of a forecast.

Intuitively, a way to think of a forecast is it being an optimal forecast that may be “measured” with an error. That is:

\(\hat{y}_{t+h|t}=\hat{y}_{t+h|t}^{\ast}+\upsilon_{t+h|t}\)

Here, the right-hand side of the equation consists of the optimal forecast and measurement error (denoted by upsilon). If upsilon is zero for all time periods in consideration, then the forecast is the optimal forecast. Otherwise, “Houston, we have a problem!”

Perhaps the simplest case is when the mean of the error is some value that is different from zero, but the variance of the error is zero. That is, a forecast is an optimal forecast plus some constant. Such a forecast consistently underestimates or overestimates the optimal forecast, depending on the sign of this constant, which makes it biased but efficient—its variance is equal to that of the optimal forecast.

A less simple, but more interesting case is when the error has some variance. In such an instance, variations of measurement error are possible.

Let’s begin with classical measurement error. As an example, consider a time series that is best approximated by a random walk model:

\(y_t=y_{t-1}+\varepsilon_t,~~\varepsilon_t\sim iid(0,\sigma^2)\)

We know, the optimal forecast for the next realization of the variable is its current realization. Suppose, instead, we used a lagged realization of the variable as a forecast.

This will not impose bias, because the difference between the optimal forecast and our forecast is merely a zero-mean iid shock (or the sum of such shocks if we use a more distant lag as the forecast).

This will impose inefficiency, however. The variance of the forecast will be larger than that of the optimal forecast. Specifically, suppose we use lag k realization as the forecast. In that case, the variance will be “inflated” by the amount equivalent to the k-times the variance of the error term in the random walk model. In so doing, also, we will inevitably impose the attenuation bias in the Mincer-Zarnowitz regression.

The narrative doesn’t change much—if at all—if the error were to be nonclassical. For example, consider a case where the time series follows a mean-reverting process (to keep things simple, let’s assume that the unconditional mean of the series is zero). Suppose we ignored this and assumed the series follow a random walk process. If we generate forecasts following this assumption, we will, again, face a problem of inefficiency.

Specifically, our forecast will overestimate the optimal forecast in the positive range of the time series—resulting in positive measurement error; it will underestimate the optimal forecast in the negative range of the series—resulting in negative measurement error. As a result, the measurement error will be correlated with the forecast, which will constitute its nonclassical nature. The practical implication of this will be the same as before—an inefficient forecast.

In Mincer-Zarnowitz regression, this will result in a slope coefficient that is different from unity. It’s more fun, and intuitively more appealing, to illustrate this via a graph:

We can see, the regression line doesn’t overlap with the 45-degree dashed line that goes through the origin of the coordinate space. If it did, it would imply a one-to-one relationship between the observed variable and its forecast—a feature of an optimal forecast. As it stands, we have evidence of an inefficient forecast. The regression line goes through the intersection of the mean realization (horizontal dashed line) and the mean forecast (vertical dashed line), however. This implies the unbiasedness of the forecast.

Examples presented here illustrated two special cases of an inadequate forecast. A forecast, of course, can be simultaneously biased and inefficient as well. In any case, in a world where all models are wrong, the least we can do is weed out those that are just not good enough. The tests of forecast adequacy (or rationality) in the Mincer-Zarnowitz regression setting, or variations of it, allow us to do so.

This R code was used to generate the figure.