1 Introduction
Not having a time series at the desired frequency is a common problem for researchers and analysts. For example, instead of quarterly sales, they only have annual sales. Instead of a daily stock market index, they only have a weekly index. While there is no way to fully make up for the missing data, there are useful workarounds: with the help of one or more high frequency indicator series, the low frequency series may be disaggregated into a high frequency series. For example, quarterly exports could help disaggregating annual sales, and a foreign stock market index could help disaggregating the stock market index at home.
Even when there is no high frequency indicator series, one still may want to disaggregate a low frequency series. While the accuracy of the resulting high frequency series will be low, it may still be worth doing so. For example, estimating a vector-autoregressive model requires all variables to have the same frequency. Having one bad high frequency series could still be preferable to the switch to a lower frequency.
The package tempdisagg (Sax and Steiner 2013) implements the following standard methods for temporal disaggregation: Denton, Denton-Cholette, Chow-Lin, Fernandez and Litterman. On the one hand, Denton (Denton 1971) and Denton-Cholette (e.g. Dagum and Cholette 2006) are primarily concerned with movement preservation, generating a series that is similar to the indicator series whether or not the indicator is correlated with the low frequency series. Alternatively, these methods can disaggregate a series without an indicator. On the other hand, Chow-Lin, Fernandez and Litterman use one or several indicators and perform a regression on the low frequency series. Chow-Lin (Chow and Lin 1971) is suited for stationary or cointegrated series, while Fernandez (Fernández 1981) and Litterman (Litterman 1983) deal with non-cointegrated series.
All disaggregation methods ensure that either the sum, the average, the first or the last value of the resulting high frequency series is consistent with the low frequency series. They can deal with situations where the high frequency is an integer multiple of the low frequency (e.g. years to quarters, weeks to days), but not with irregular frequencies (e.g. weeks to months).
Temporal disaggregation methods are widely used in official statistics. For example, in France, Italy and other European countries, quarterly figures of Gross Domestic Product (GDP) are computed using disaggregation methods. Outside of R, there are several software packages to perform temporal disaggregation: Ecotrim by Barcellan et al. (2003); a Matlab extension by Quilis (2012); and a RATS extension by Doan (2008). An overview of the capabilities of the different software programs is given in Table 1.1
The first section discusses the standard methods for temporal disaggregation and summarizes them in a unifying framework. Section 2 discusses the working and implementation of the tempdisagg package. Section 3 presents an illustrative example.
2 A framework for disaggregation
The aim of temporal disaggregation is to find an unknown high frequency series \(y\), whose sums, averages, first or last values are consistent with a known low frequency series \(y_l\) (The subscript \(l\) denotes low frequency variables). In order to estimate \(y\), one or more other high frequency indicator variables can be used. We collect these high frequency series in a matrix \(X\). For the ease of exposition and without loss of generality, the terms annual and quarterly will be used instead of low frequency and high frequency hereafter.
| Methods | Ecotrim | Matlab add-on | RATS add-on | tempdisagg |
|---|---|---|---|---|
| Chow-Lin (max. log) | erroneous | full | full | full |
| Chow-Lin (min. RSS) | full | full | no | full |
| Fernández | full | full | full | full |
| Litterman (max. log) | erroneous | full | full | full |
| Litterman (min. RSS) | full | full | no | full |
| Denton-Cholette | partial | erroneous | no | full |
| Denton | no | partial | no | full |
| Contemporaneous constraint | partial | partial | no | no |
The diversity of temporal disaggregation methods can be narrowed by putting the methods in a two-step framework: First, a preliminary quarterly series \(p\) has to be determined; second, the differences between the annual values of the preliminary series and the annual values of the observed series have to be distributed among the preliminary quarterly series. The sum of the preliminary quarterly series and the distributed annual residuals yields the final estimation of the quarterly series, \(\hat{y}\). Formally, \[\hat{y} = p + D u_l \,. \label{eq:decomp} \tag{1}\] \(D\) is a \(n \times n_l\) distribution matrix, with \(n\) and \(n_l\) denoting the number of quarterly and annual observations, respectively. \(u_l\) is a vector of length \(n_l\) and contains the differences between the annualized values of \(p\) and the actual annual values, \(y_l\): \[u_l \equiv \ y_l - C p \,. \label{eq:residuals} \tag{2}\] Multiplying the \(n_l \times n\) conversion matrix, \(C\), with a quarterly series performs annualization. With two years and eight quarters, and annual values representing the sum of the quarterly values (e.g. GDP), the conversion matrix, \(C\), is constructed the following way:2 \[C = \begin{bmatrix} 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 1 & 1 & 1 & 1\\ \end{bmatrix} \,.\]
Equation (1) constitutes a unifying framework for all disaggregation methods. The methods differ in how they determine the preliminary series, \(p\), and the distribution matrix, \(D\). Table 2 summarizes the differences in the calculation of \(p\) and \(D\). We will discuss them in turn.
Preliminary series
The methods of Denton and Denton-Cholette use a single indicator as their preliminary series: \[p = X \,,\] where \(X\) is a \(n \times 1\) matrix. As a special case, a constant (e.g. a series consisting of only 1s in each quarter) can be embodied as an indicator, allowing for temporal disaggregation without high frequency indicator series.
The regression-based methods Chow-Lin, Fernandez and Litterman perform a Generalized Least Squares Regression (GLS) of the annual values, \(y_l\), on the annualized quarterly indicator series, \(CX\). In this case, \(X\) represents a \(n \times m\) matrix, where \(m\) denotes the number of indicators (including a possible constant). For a given variance-covariance matrix, \(\Sigma\), the GLS estimator, \(\hat{\beta}\), is calculated in the standard way (the estimation of \(\Sigma\) is discussed below): \[\hat{\beta}(\Sigma) = \left[ X' C' ( C \Sigma C' )^{-1} C X \right]^{-1} X' C' ( C \Sigma C' )^{-1} y_l \,. \label{eq:GLS} \tag{3}\] The critical assumption of the regression-based methods is that the linear relationship between the annual series \(CX\) and \(y_l\) also holds between the quarterly series \(X\) and \(y\). Thus, the preliminary series is calculated as the fitted values of the GLS regression: \[p = \hat{\beta}X \,. \label{eq:fitted} \tag{4}\]
Distribution matrix
With the exception of Denton-Cholette, the distribution matrix of all temporal disaggregation methods is a function of the variance-covariance matrix, \(\Sigma\): \[D = \Sigma \, C' (C\,\Sigma \,C')^{-1} \,. \label{eq:DStandard} \tag{5}\]
The Denton methods minimize the squared absolute or relative deviations from a (differenced) indicator series, where the parameter \(h\) defines the degree of differencing. For the additive Denton methods and for \(h=0\), the sum of the squared absolute deviations between the indicator and the final series is minimized. For \(h=1\), the deviations of first differences are minimized, for \(h=2\), the deviations of the differences of the first differences, and so forth. For the proportional Denton methods, deviations are measured in relative terms.
For the additive Denton method with \(h=1\), the variance-covariance matrix has the following structure: \[\Sigma_{\,\mathrm{D}} = (\Delta'\Delta)^{-1}= \begin{bmatrix} 1 & 1 & \cdots & 1\\ 1 & 2 & \cdots & 2\\ \vdots & \vdots & \ddots & \vdots \\ 1 & 2 & \cdots & n \end{bmatrix} \,, \label{eq:SDenton} \tag{6}\] where \(\Delta\) is a \(n \times n\) difference matrix with 1 on its main diagonal, \(-1\) on its first subdiagonal and \(0\) elsewhere. For \(h=2\), \(\Delta'\Delta\) is multiplied by \(\Delta'\) from the left and \(\Delta\) from the right side. For \(h=0\), it is the identity matrix of size \(n\).
Denton-Cholette is a modification of the original approach and removes the spurious transient movement at the beginning of the resulting series. While generally preferable, the calculation of the distribution matrix, \(D_{\mathrm{D\,C}}\), does not fit into the simple framework (see Dagum and Cholette 2006 136, for an extensive description).
Chow-Lin assumes that the quarterly residuals follow an autoregressive process of order 1 (AR1), i.e., \(u_t = \rho u_{t-1} + \epsilon_t\), where \(\epsilon\) is \(\mathrm{WN}(0, \sigma_\epsilon)\) (with \(\mathrm{WN}\) denoting White Noise) and \(\left|\rho\right| < 1\). The resulting covariance matrix has the following form: \[\Sigma_{\,\mathrm{CL}}(\rho) =\frac{\sigma^{2}_{\epsilon}}{1 - \rho^{2}} \cdot \begin{bmatrix} 1 & \rho & \cdots & \rho^{n-1}\\ \rho & 1 & \cdots & \rho^{n-2}\\ \vdots & \vdots & \ddots & \vdots \\ \rho^{n-1} & \rho^{n-2} & \cdots & 1 \end{bmatrix} \,.\] The estimation of \(\Sigma_{\,\mathrm{CL}}\) thus requires the estimation of an AR1 parameter \(\rho\), which will be discussed in the next section. The variance, \(\sigma^{2}_{\epsilon}\), cancels out and does not affect the calculation of neither \(D\) nor \(\hat{\beta}\).
| Methods | \(p\) | \(D\) | \(\Sigma\) |
|---|---|---|---|
denton |
\(X\) | \(\Sigma \, C' (C \, \Sigma \, C')^{-1}\) | \(\Sigma_{\,\mathrm{D}}\) |
denton-cholette |
\(X\) | \(D_{\mathrm{D\,C}}\) | |
chow-lin-maxlog, chow-lin-minrss-ecotrim, |
\(\hat{\beta}X\) | \(\Sigma \, C' (C \, \Sigma \, C')^{-1}\) | \(\Sigma_{\,\mathrm{CL}}(\rho)\) |
chow-lin-minrss-quilis |
|||
litterman-maxlog, litterman-minrss |
\(\hat{\beta}X\) | \(\Sigma \, C' (C \, \Sigma \, C')^{-1}\) | \(\Sigma_{\,\mathrm{L}}(\rho)\) |
fernandez |
\(\hat{\beta}X\) | \(\Sigma \, C' (C \, \Sigma \, C')^{-1}\) | \(\Sigma_{\,\mathrm{L}}(0)\) |
The remaining methods deal with cases when the quarterly indicators and the annual series are not cointegrated. Fernandez and Litterman assume that the quarterly residuals follow a non-stationary process, i.e. \(u_t = u_{t-1} + v_t\), where \(v\) is an AR1 \((v_t = \rho v_{t-1} + \epsilon_t\), where \(\epsilon\) is \(\mathrm{WN}(0, \sigma_\epsilon))\). Fernandez is a special case of Litterman, where \(\rho = 0\), and, therefore, \(u\) follows a random walk. The variance-covariance matrix can be calculated as follows: \[\Sigma_{\mathrm{L}}(\rho) = \sigma^{2}_{\epsilon} \left[\Delta'H(\rho)'H(\rho)\Delta\right]^{-1} \,,\] where \(\Delta\) is the same \(n \times n\) difference matrix as in the Denton case; \(H(\rho)\) is a \(n \times n\) matrix with 1 on its main diagonal, \(-\rho\) on its first subdiagonal and \(0\) elsewhere. For the special case of Fernandez, with \(\rho = 0\), the resulting covariance matrix has the following form: \[\Sigma_{\,\mathrm{L}}(0) = \sigma^{2}_{\epsilon} \cdot (\Delta'\Delta)^{-1} = \sigma^{2}_{\epsilon} \cdot \Sigma_D \,.\]
Estimating the autoregressive parameter
There are several ways to estimate the autoregressive parameter \(\rho\) in the Chow-Lin and Litterman methods. An iterative procedure has been proposed by Chow and Lin (1971). It infers the parameter from the observed autocorrelation of the low frequency residuals, \(u_l\).
In a different approach, Bournay and Laroque (1979 23) suggest the maximization of the likelihood of the GLS-regression: \[L( \rho, \sigma^2_{\epsilon}, \beta ) = \frac{\exp \left[ - \frac{1}{2} u_{l}' \, \left( C \, \Sigma \, C' \right)^{-1} \, u_{l} \right]} {\left( 2 \pi \right)^{n_{l} / 2} \cdot \left[\det \left( C \, \Sigma \, C' \right) \right]^{1/2} } \,,\] where \(u_{l}\) is given by Eq. (2) and (4). \(\hat{\beta}\) turns out to be the GLS estimator from Eq. (3). The maximum likelihood estimator of the autoregressive parameter, \(\hat{\rho}\), is a consistent estimator of the true value, thus it has been chosen as the default estimator. However, in some cases, \(\hat{\rho}\) turns out to be negative even if the true \(\rho\) is positive. Thus, by default, tempdisagg constrains the optimization space for \(\rho\) to positive values.
A final approach is the minimization of the weighted residual sum of squares, as it has been suggested by Barbone et al. (1981):
\[RSS(\rho, \sigma^2_{\epsilon}, \beta) = u_{l}' \left( C \, \Sigma \, C' \right)^{-1} u_{l} \,.\] Contrary to the maximum likelihood approach, \(\sigma^2_{\epsilon}\) does not cancel out. The results are thus sensitive to the specification of \(\Sigma\), with different implementations leading to different but inconsistent estimations of \(\rho\).
3 The tempdisagg package
The selection of a temporal disaggregation model is similar to the
selection of a linear regression model. Thus, td, the main function of
the package, closely mirrors the working of the lm function (package
stats), including taking advantage of the formula interface.3
td(formula, conversion = "sum", to = "quarterly", method = "chow-lin-maxlog",
truncated.rho = 0, fixed.rho = 0.5, criterion = "proportional", h = 1,
start = NULL, end = NULL, ...)The left hand side of the formula denotes the low frequency series, the
right hand side the indicators. If no indicator is specified, the right
hand side must be set equal to 1. The variables can be entered as time
series objects of class "ts" or as standard vectors or matrices. If
entered as "ts" objects, the resulting series will be "ts" objects
as well.
The conversion argument indicates whether the low frequency values are
sums, averages, first or last values of the high frequency values
("sum" (default), "average", "first" or "last", respectively).
The method argument indicates the method of temporal disaggregation,
as shown in Table 2 (see ?td for a complete listing
of methods). The to argument indicates the high frequency destination
as a character string ("quarterly" (default) or "monthly") or as a
scalar (e.g. 2, 7, for year-semester or week-day conversion). It is
only required if no indicator series is specified (Denton methods), or
if standard vectors are used instead of time series objects. Finally,
you can set an optional start or end date. This is identical to
pre-processing the input series with window.
td returns an object of class "td". The function predict computes
the disaggregated high frequency series, \(\hat{y}\). If the high
frequency indicator series are longer than the low frequency series, the
resulting series will be extrapolated.
The implementation of tempdisagg follows the same notation and modular
structure as the exposure in the previous section. Internally, td uses
the optimize function (package stats) to solve the one-dimensional
optimization problem at the core of the Chow-Lin and Litterman methods.
For GLS estimation, td uses an efficient and nummerically stable
algorithm that is based on the qr-decomposition (Paige 1979).
4 An example
Suppose we have an annual series and want to create quarterly values
that sum up to the annual values. Panel 1 of Fig. 1
depicts annual sales of the pharmaceutical and chemical industry in
Switzerland, sales.a, from which we want to create a quarterly series.
The following example demonstrates the basic use of tempdisagg. It can
also be run by demo(tempdisagg).
The most simple method is "denton-cholette" without an indicator
series. It performs a simple interpolation that meets the temporal
additivity constraint. In R, this can be done the following way:
> library(tempdisagg)
> data(swisspharma)
> m1 <- td(sales.a ~ 1, to = "quarterly", method = "denton-cholette")
> predict(m1)td produces an object of class "td". The formula, sales.a ~ 1,
indicates that our low frequency variable, sales.a, will be
disaggregated with a constant, 1 (see ?formula for the handling of
the intercept in the formula interface). The resulting quarterly values
of sales can be extracted with the predict function. As there is no
additional information on quarterly movements, the resulting series is
very smooth (Panel 2 of Fig. 1).
While this purely mathematical approach is easy to perform and does not
need any other data series, the economic value of the resulting series
may be limited. There might be a related quarterly series that follows a
similar movement than sales. For example, we may use quarterly exports
of pharmaceutical and chemical products, exports.q (Panel 3 of
Fig. 1):
> m2 <- td(sales.a ~ 0 + exports.q, method = "denton-cholette")Because we cannot use more than one indicator with the
"denton-cholette" (or "denton") method, the intercept must be
specified as missing in the formula (0). Contrary to the first
example, the to argument is redundant, because the destination
frequency can be interfered from the time series properties of
exports.q. Applying the predict function to the resulting model
leads to a much more interesting series, as shown in Panel 4 of
Fig. 1. As the indicator series is longer than the
annual series, there is an extrapolation period, in which quarterly
sales are forecasted.
With an indicator, the "denton-cholette" method simply transfers the
movement of the indicator to the resulting series. Even if in fact there
were no correlation between the two series, there would be a strong
similarity between the indicator and the resulting series. In contrast,
regression based methods transfer the movement only if the indicator
series and the resulting series are actually correlated on the annual
level. For example, a Chow-Lin regression of the same problem as above
can be performed the following way:
> m3 <- td(sales.a ~ exports.q)As "chow-lin-maxlog" is the default method, it does not need to be
specified. Like with the corresponding lm method, summary produces
an overview of the regression:
> summary(m3)
Call:
td(formula = sales.a ~ exports.q)
Residuals:
Min 1Q Median 3Q Max
-77.892 -7.711 -4.628 9.647 36.448
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.241e+01 1.493e+00 8.311 1.06e-09 ***
exports.q 1.339e-02 1.672e-04 80.111 < 2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
'chow-lin-maxlog' disaggregation with 'sum' conversion
36 low-freq. obs. converted to 146 high-freq. obs.
Adjusted R-squared: 0.9946 AR1-Parameter: 0 (truncated)There is indeed a strong correlation between exports and sales, as it
has been assumed in the "denton-cholette" example above. The
coefficient of exports.q is highly significant, and the very high
adjusted \(R^2\) points to a strong relationship between the two
variables. The coefficients are the result of a GLS regression between
the annual series. The AR1 parameter, \(\rho\), was estimated to be
negative; in order to avoid the undesirable side-effects of a negative
\(\rho\), it has been truncated to 0 (This feature can be turned off).
Again, with the predict function, we can extract the resulting
quarterly series of sales (Panel 5 of Fig. 1). Like all
regression based methods, "chow-lin-maxlog" can also be used with more
than one indicator series:
> m4 <- td(formula = sales.a ~ exports.q + imports.q)In our example, we actually know the true data on quarterly sales, so we can compare the estimated values to the true values. With an indicator series, both the Denton method and Chow-Lin produce a series that is close to the true series (Panel 6 of Fig. 1). This is, of course, due to fact that in this example, exports are a good indicator for sales. If the indicator is less close to the series of interest, the resulting series will be less close to the true series.
5 Summary
tempdisagg implements the standard methods for temporal disaggregation. It offers a way to disaggregate a low frequency time series into a higher frequency series, while either the sum, the average, the first or the last value of the resulting high frequency series is consistent with the low frequency series. Temporal disaggregation can be performed with or without the help of one or more high frequency indicators. If good indicators are at hand, the resulting series may be close to the true series.
CRAN packages used
CRAN Task Views implied by cited packages
Note
This article is converted from a Legacy LaTeX article using the texor package. The pdf version is the official version. To report a problem with the html, refer to CONTRIBUTE on the R Journal homepage.