lspartition: Partitioning-Based Least Squares Regression

Nonparametric partitioning-based least squares regression is an important tool in empirical work. Common examples include regressions based on splines, wavelets, and piecewise polynomials. This article discusses the main methodological and numerical features of the R software package lspartition, which implements modern estimation and inference results for partitioning-based least squares (series) regression estimation. This article discusses the main methodological and numerical features of the R software package lspartition, which implements results for partitioning-based least squares (series) regression estimation and inference from Cattaneo and Farrell (2013) and Cattaneo, Farrell, and Feng (2019). These results cover the multivariate regression function as well as its derivatives. First, the package provides data-driven methods to choose the number of partition knots optimally, according to integrated mean squared error, yielding optimal point estimation. Second, robust bias correction is implemented to combine this point estimator with valid inference. Third, the package provides estimates and inference for the unknown function both pointwise and uniformly in the conditioning variables. In particular, valid confidence bands are provided. Finally, an extension to two-sample analysis is developed, which can be used in treatment-control comparisons and related problems


Introduction
Nonparametric partitioning-based least squares regression estimation is an important method for estimating conditional expectation functions in statistics, economics, and other disciplines. These methods first partition the support of covariates and then construct a set of local basis functions on top of the partition to approximate the unknown regression function or its derivatives. Empirically popular basis functions include splines, compactly supported wavelets, and piecewise polynomials. For textbook reviews on classical and modern nonparametric regression methodology see, among others, Fan and Gijbels (1996), Györfi, Kohler, Krzyżak, and Walk (2002), Ruppert, Wand, and Carroll (2009), and Harezlak, Ruppert, and Wand (2018). For a review on partitioning-based approximations in nonparametrics and machine learning see Zhang and Singer (2010) and references therein.
This article gives a detailed discussion of the software package lspartition, available for R, which implements partitioning-based least squares regression estimation and inference. This package offers several features which improve on existing tools, leveraging the recent results of Cattaneo and Farrell (2013) and Cattaneo, Farrell, and Feng (2019), and delivering data-driven methods to easily implement partitioning-based estimation and inference, including optimal tuning parameter choices and uniform inference results such as confidence bands. We cover splines, compactly supported wavelets, and piecewise polynomials, in a unified way, encompassing prior methods and routines previously unavailable without manual coding by researchers. Piecewise polynomials generally differ from splines and wavelets in that they do not enforce global smoothness over the partition, but in the special cases of zero-degree bases on a tensor-product partition, the three basis choices (i.e., zero-degree spline, Haar wavelet, and piecewise constant) are equivalent.
The first contribution offered by lspartition is a data-driven choice of the number of partitioning knots that is optimal in an integrated mean squared error (IMSE) sense. A major hurdle to practical implementation of any nonparametric estimator is tuning parameter choice, and by offering several feasible IMSE-optimal methods for splines, compactly supported wavelets, and piecewise polynomials, lspartition provides practitioners with tools to overcome this important implementation issue.
However, point estimation optimal tuning parameter choices yield invalid inference in general, and the IMSE-optimal choice is no exception. The second contribution of lspartition is the inclusion of robust bias correction methods, which allow for inference based on optimal point estimator. lspartition implements the three methods studied by Cattaneo, Farrell, and Feng (2019), which are based on novel bias expansions therein. Both the bias and variance quantities are kept in pre-asymptotic form, yielding better bias correction and standard errors robust to conditional heteroskedasticity of unknown form. Collectively, this style of robust bias correction has been proven to yield improved inference in other nonparametric contexts Farrell, 2018, 2019).
The third main contribution is valid inference, both pointwise and uniformly in the support of the conditioning variables. When robust bias correction is employed, this inference is valid for the IMSEoptimal point estimator, allowing the researcher to combine an optimal partition for point estimation and a "faithful" measure of uncertainty (i.e., one that uses the same nonparametric estimation choices, here captured by the partition). In particular, lspartition delivers valid confidence bands that cover the entire regression function and its derivatives. These data-driven confidence bands are constructed by approximating the distribution of t-statistic processes, using either a plug-in approach or a bootstrap approach. Importantly, the construction of confidence bands does not employ (asymptotic) extreme value theory, but instead uses the strong approximation results of Cattaneo, Farrell, and Feng (2019), which perform substantially better in samples of moderate size.
Last but not least, the package also offers a convenient function to implement estimation and inference for linear combinations of regression estimators of different groups with all the features mentioned above. This function can be used to analyze conditional treatment effects in random control trials in particular, or for two-sample comparisons more generally. For example, a common question in applications is whether two groups have the same "trend" in a regression function, and this is often answered in a restricted way by testing a single interaction term in a (parametric) linear model. In contrast, lspartition delivers a valid measure of this difference nonparametrically and uniformly over the support of the conditioning variables, greatly increasing its flexibility in applications.
All of these contributions are fully implemented for splines, wavelets, and piecewise polynomials through the following four functions included in the package lspartition: • lsprobust(): This function implements estimation and inference for partitioning-based least squares regression. It takes the partitioning scheme as given, and constructs point and variance estimators, bias correction, conventional and robust bias-corrected confidence intervals, and simulation-based conventional and robust bias-corrected uniform inference measures (e.g., confidence bands). Three approximation bases are provided: B-splines, Cohen-Daubechies-Vial wavelets, and piecewise polynomials. When the partitioning scheme is not specified, the companion function lspkselect() is used to select a tensor-product partition in a fully data-driven fashion.
• lspkselect(): This function implements data-driven procedures to select the number of knots for partitioning-based least squares regression. It allows for evenly-spaced and quantile-spaced knot placements, and computes the corresponding IMSE-optimal choices. Two selectors are provided: rule of thumb (ROT) and direct plug-in (DPI) rule.
• lsplincom(): This function implements estimation and robust inference procedures for linear combinations of regression estimators of multiple groups based on lsprobust(). Given a userspecified linear combination, it offers all the estimation and inference methods available in the functions lsprobust() and lspkselect().
• lsprobust.plot(): This function builds on ggplot2 (Wickham and Chang, 2016), and is used as a wrapper for plotting results. It plots regression function curves, robust bias-corrected confidence intervals and uniform confidence bands, among other possibilities.
The paper continues as follows. Section 2.2 describes the basic setup including a brief introduction to partitioning-based least squares regression and the empirical example to be used throughout to illustrate features of lspartition. Section 2.3 discusses data-driven IMSE-optimal selection of the number of knots and gives implementation details. Estimation and inference implementation is covered in Section 2.4, including bias correction methods. The last section concludes. We defer to Cattaneo, Farrell, and Feng (2019, CFF hereafter) for complete theoretical and technical details. Statements below are sometimes specific versions of a general case therein.

Setup
We assume that {(y i , x i ) : 1 ≤ i ≤ n} is an observed random sample of a scalar outcome y i and a d-vector of covariates x i ∈ X ⊂ R d . The object of interest is the regression function µ(x) = E[y i |x i = x] or its derivative, the latter denoted by ∂ q µ(x) = ∂ [q] µ(x)/∂x Estimation and inference is based on least squares regression of y i on set of basis functions of x i which are themselves built on top of a partition of the support X . A partition, denoted by ∆ = {δ l ⊂ X : 1 ≤ l ≤ κ}, is a collection of κ disjoint open sets such that the closure of their union is X . For a partition, a set of basis functions, each of order m and denoted by p(x), is constructed so that each individual function (i.e., each element of the vector p(x)) is nonzero on a fixed number of contiguous δ l . lspartition allows for three such bases: piecewise polynomials, B-splines, and Cohen-Daubechies-Vial wavelets (Cohen, Daubechies, and Vial, 1993). For the first two bases, the order m of the basis can be any positive integer, and any derivative of µ up to total order (m − 1) can be estimated employing such a basis. For wavelets, the current version allows for m ≤ 4 (i.e., up to cubic wavelets), and q = (0, . . . , 0). The package takes m = 2 (linear basis) as default. To fix ideas, consider d = 1 with piecewise polynomials. Each δ l is an interval and p(x) collects all the indicator functions 1{x ∈ δ l }, 1 ≤ l ≤ κ.
Once the basis p(x) is constructed, the final estimator of ∂ q µ(x) is where [q] < m. When q = 0, we write µ(·) = ∂ 0 µ(·) for simplicity. The approximation power of such estimators increases with the granularity of the partition ∆ and the order m. We take the latter as fixed in practice. The most popular structure of ∆ in applications is a tensor-product form, which partitions each covariate marginally into intervals and then sets ∆ to be the set of all tensor (Cartesian) products of these intervals (CFF consider more general cases). For this type of partition, the user must choose the number and placement of the partitioning knots in each dimension. lspartition allows for three knot placement types: user-specified, evenly-spaced, and quantile-spaced. In the first case, the user has complete freedom to choose both the number and positions of knots for each dimension. In the latter two cases, the knot placement scheme is pre-specified, and hence only the number of subintervals for each dimension needs to be chosen.
We denote the number of knots in the d dimensions of the regressor x i by κ = (κ 1 , . . . , κ d ) ∈ Z d + , which can be either specified by users or selected by data-driven procedures (see Section 2.3 below). Moreover, for wavelet bases, motivated by the standard multi-resolution analysis, we provide an option J for the regression command lsprobust(), which indicates the resolution level of a wavelet basis. This gives κ = 2 J , = 1, . . . d, for a resolution J (see Chui, 2016, for a review). In any case, the tuning parameter to be chosen is κ = κ 1 × · · · × κ d . In the next section we choose κ to minimize the IMSE of the estimator (1).

Package and data
We will showcase the main aspects of lspartition using a running empirical example. The package is available in R and can be installed as follows: > install.packages("lspartition", dependencies = TRUE) > library(lspartition) The data we use come from Capital Bikeshare, and is available at http://archive.ics.uci.edu/ ml/datasets/Bike+Sharing+Dataset/. For the first 19 days of each month of 2011 and 2012 we observe the outcome count, the total number of rentals and the covariates atemp, the "feels-like" temperature in Celsius, and workingday, a binary indicator for working days (versus weekends and holidays). The data is summarized as follows.
> data <-read.csv("bikesharing.csv", header = TRUE) > summary ( We will investigate nonparametrically the relationship between temperature and number of rentals and compare the two groups defined by the type of days: The sample code that follows will use this designation of y, x, and g.

Partitioning scheme selection
We will now briefly describe the IMSE expansion and its use in tuning parameter selection. To differentiate the original point estimator of (1) and the post-bias-correction estimators, we will add a subscript "0" to the original estimator: ∂ q µ 0 (x). The three bias corrections discussed below will add corresponding subscripts of 1, 2, and 3. We first discuss the bias and variance of ∂ q µ 0 (x), and then use these for minimizing the IMSE. Throughout, ≈ denotes that the approximation holds for large sample in probability and E n [·] denotes the sample average over 1 ≤ i ≤ n. To simplify notation, we may write the estimator as Again, note the subscript "0"; the bias-corrected estimators are of the same form (see below).

Bias and variance
The bias expansion for the ∂ q µ 0 (x) is: B m,q (·) is the leading approximation error in the L ∞ -norm and the second term is the accompanying error from the linear projection of B m,0 (·) onto the space spanned by the basis functions. The form of each of these is complex, and depends on the basis, but what is crucial for the present purposes is that the form is known and the only unknown elements are derivatives of order m, ∂ u µ(x), [u] = m. CFF derive exact expressions for splines, wavelets, and piecewise polynomials. Both bias terms will, in general, contribute to the same order, and both will matter in finite samples. However, the second term in (3) will be higher order if the bases are carefully constructed so that B m,0 (·) is orthogonal to p(·) in L 2 with respect to the Lebesgue measure. lspartition allows users to choose whether the projection of the leading error is used in partitioning scheme selection, as well as estimation and inference.
The conditional variance is straightforward from least squares algebra, and takes the familiar sandwich form: where Only σ 2 (x i ) is unknown here, and will be replaced by a residual-based estimator. In particular lspartition allows for the standard Heteroskedasticity-Consistent (HC) class of estimators via the options hc0, hc1, hc2, hc3. See Long and Ervin (2000) for a review in the context of least squares regression.

Integrated mean squared error
In general, for a weighting function w(x), CFF derive the following (conditional) IMSE expansion: where the n-varying quantities V κ,q and B κ,q correspond to a fixed-n approximation to the variance and squared bias, respectively.
Under regularity conditions on the partition and basis used, CFF derive explicit leading constants in this expansion. lspartition implements IMSE-minimization for the common simple case where ∆ is a tensor product of marginally formed intervals where the same number of intervals are used for each dimension. Specifically, To selectκ IMSE , or equivalently κ IMSE , assume that the partitioning knots {0 = t ,0 < t ,1 < · · · < t ,κ−1 < t ,κ = 1} are generated as quantiles of some marginal distributions G (·), = 1, . . . , d, that is, where Then, the IMSE-optimal choice for q = 0 is where x is a ceiling operator that outputs the smallest integer that is no less than x and B G,0 is a (squared) bias term that may depend on the marginals G and, as before, is entirely known up to m th order derivatives:

Implementation details
Two popular choices of partitioning schemes are evenly-spaced partitions (ktype="uni"), which sets G (·) to be the uniform distribution over the support of the data, and quantile-spaced partitions (ktype="qua"), which sets G (·) to be the empirical distribution function of each covariate. The package lspartition implements both partitioning schemes, and for each case offers two IMSE-optimal tuning parameter selection procedures: rule of thumb (imse-rot) and direct plug-in (imse-dpi) choices. We close this section with a brief description of the implementation details and an illustration using real data.

Rule-of-Thumb Choice
The rule-of-thumb choice is based on the special case of q = 0. Let the weighting function w(x) be the density of x i . The implementation steps are summarized in the following: • Bias constant. The unknown quantities in the bias constants are: ∂ u µ(·), u ∈ Λ m , which is estimated by a global polynomial regression of degree (m + 2); and the density of x i , which is either assumed to be uniform or estimated by a trimmed-from-below Gaussian reference model (controlled by the option rotnorm).
• Variance constant. The unknown quantities in the variance constants are: the conditional variance which is estimated by global polynomial regressions of degree (m + 2); and the density of x i , which is either assumed to be uniform or estimated by a trimmed-from-below Gaussian reference model.
• Rule-of-thumbκ rot . Using the above results, a simple rule-of-thumb choice ofκ iŝ where B G,0 and V 0 are the estimates of bias and variance constants respectively. While this choice ofκ is obtained under strong parametric assumptions, it still exhibits the correct convergence rate (κ rot n 1 2m+d ).
The command lspkselect() implements the rule-of-thumb selection (kselect="imse-rot"). For example, we focus on a subsample of bike rentals during working days (g==1), and then the selected number of knots are reported in the following: > summary(lspkselect(y, x, kselect = "imse-rot", subset = (g == + 1))) Call: lspkselect In this example, for the point estimator based on an evenly-spaced partition, the rule-of-thumb estimate of the IMSE-optimal number of knots is k = 5, and for the derivative estimators used in bias correction for later inference, the rule-of-thumb choice is k.bc = 9.

Direct Plug-in Choice
Assuming that the weighting w(x) is equal to the density of x i , the package lspartition implements a direct-plug-in (DPI) procedure summarized by the following steps.
• Preliminary choice ofκ: Implement the rule-of-thumb procedure to obtainκ rot .
• Preliminary regression. Given the user-specified basis, knot placement scheme, and rule-ofthumb choiceκ rot , implement a partitioning-based regression of order (m + 1) to estimate all necessary order-m derivatives; denote these by ∂ u µ pre (·), [u] = m.
• Bias constant. Construct an estimate B m,q (·) of the leading error B m,q (·) by replacing ∂ u µ(·) by ∂ u µ pre (·). B m,0 (·) can be obtained similarly. Then, use the pre-asymptotic version of the conditional bias to estimate the bias constant: As mentioned before, for the three bases considered in the package lspartition, the second term in the conditional bias is of smaller order under some additional conditions. We employ this property to simplify the estimate of bias constant for wavelets. For splines and piecewise polynomials, however, users may specify whether the projection of the leading error is taken into account in the selection procedure (see option proj).
• Variance constant. Implement a partitioning-based series regression of order m withκ =κ rot , and then use the pre-asymptotic version of the conditional variance to estimate the variance constant. Specifically, let , and w i is the weighting scheme used to construct different HC variance estimators.
• Direct plug-inκ. Collecting all these results, a direct plug-in choice ofκ iŝ The following shows the results of the direct plug-in procedure based on the real data: > summary(lspkselect(y, x, kselect = "imse-dpi", subset = (g == + 1))) Call: lspkselect The direct plug-in procedure gives more partitioning knots than the rule-of-thumb, leading to a finer partition. For point estimation,κ dpi = 8 knots are suggested, while for bias correction purpose, it selectsκ dpi = 10 knots to estimate derivatives in the leading bias. Quantile-spaced knot placement is obtained by adding ktype = "qua".

Estimation and inference
This section reviews and illustrates the estimation and inference procedures implemented. A crucial ingredient is the bias correction that allows for valid inference after tuning parameter selection.

Point estimation and bias correction
The estimator ∂ q µ 0 (x) is IMSE-optimal from a point estimation perspective when implemented using the choice κ IMSE to form ∆, but conventional inference methods based on this resulting point estimator will be invalid. More precisely, the ratio of bias to standard error in the t-statistic is nonnegligible, requiring either ad-hoc undersmoothing or some form of bias correction. In addition to the (uncorrected) point estimate in (1), the package lspartition implements the three bias correction options derived by CFF for valid (pointwise and uniform) inference. All these strategies resort to a higher-order basis,p(x), of orderm > m. The partition∆ wherep(x) is built on may be different from ∆ but need not be. These approaches allow researchers to combine an optimal point estimate ∂ q µ 0 (x) based on the IMSE-optimal κ IMSE with inference based on the same tuning parameter and partitioning scheme choices.
• Approach 1: Higher-order-basis bias correction. Usep(x) to construct a higher-order least squares estimator ∂ q µ 1 (x) which takes exactly the same form as ∂ q µ 0 (x) but has less bias. If we substitute y i and ∂ q µ 1 (x) for µ(x i ) and ∂ q µ(x) in (2) respectively and subtract this estimated bias from ∂ q µ 0 (x), the resulting "bias-corrected" estimator is equivalent to ∂ q µ 1 (x). This option is called by bc="bc1".
• Approach 2: Least squares bias correction. Construct ∂ q µ 1 (x) and substitute it for ∂ q µ(x) in (2), but replace µ(x i ) byμ 1 (x i ) rather than y i . The least squares bias-corrected estimator ∂ q µ 2 (x) is obtained by subtracting this estimated bias from ∂ q µ 0 (x). The supplement to CFF discusses in detail how this approach relates to higher-order-basis bias correction and when they are equivalent. This option is called by bc="bc2".
The optimal (uncorrected) point estimator (j = 0) and the three bias-corrected estimators (j = 1, 2, 3) can be written in a unified form: These estimators only differ in γ q,j (·) and Π j (·), which depend in different ways on p(x) andp(x). See CFF for exact formulas.

Pointwise inference
Pointwise inference relies on a Gaussian approximation for the t-statistics: where Ω j (x)/n = γ q,j (x) Σ j γ q,j (x)/n is an estimator of the conditional variance of ∂ q µ j (·), and denotes convergence in distribution.
and the w i 's are additional weights leading to various HC variance estimators. Then nominal 100(1 − α)-percent symmetric confidence intervals are where Φ u is the u th quantile of the standard normal distribution.
For conventional confidence intervals (j = 0), the (asymptotically) correct coverage relies on undersmoothing (κ κ IMSE ) that renders the bias negligible relative to the standard error in large samples. Though straightforward in theory, it is difficult to implement in a principled way. In comparison, given the IMSE-optimal tuning parameter, all three bias-corrected estimators (j = 1, 2, 3) have only higher-order bias, and thus the corresponding confidence intervals based on these estimators will have asymptotically correct coverage. Importantly, the Studentization quantity Ω j (x)/n also captures the additional variability introduced by bias correction.

Uniform inference
To obtain uniform inference (over the support of x), CFF establish Gaussian approximations for the whole t-statistic processes, and propose several sampling-based approximations which are easy to implement in practice. To be concrete, for each j = 0, 1, 2, 3, there exists a Gaussian process Z j (·) such that where γ q,j (·) and Ω j (·) are population counterparts of γ q,j (·) and Ω j (·), N K j is a K j -dimensional standard normal random vector, and K j is the length of Π j , and is proportional to κ. The notation ≈ d means that the two processes are asymptotically equal in distribution in the following sense: in a sufficiently rich probability space, we have identical copies of T j (·) and Z j (·) whose difference converges in probability to zero uniformly.
The Gaussian stochastic process Z j (·) is not feasible in practice because it involves unknown population quantities. Thus, the package lspartition offers two options for implementation: plug-in or bootstrap.
• Plug-in. Replace all unknowns in Z j (·) by some consistent estimators: CFF show that Z j (·) delivers a valid distributional approximation to T j (·). In practice one may obtain many simulated realizations of Z j (·) by sampling from the K j -dimensional standard normal distribution conditional on the data. This option is called by uni.method="pl".
• Bootstrap. Construct a bootstrapped version of the approximation process (conditional on the data): is an i.i.d sequence of bounded random variables with zero mean and unit variance. CFF show that this bootstrapped process also approximates Z j (·) conditional on the data. Thus one can implement bootstrapping by sampling from the distribution of ω i given the data. In the package lspartition, the ω i 's are taken to be Rademacher variables, and this option is called by uni.method="wb".
Importantly, these strong approximations apply to the whole t-statistic processes, and thus can be used to implement general inference procedures based on transformations of T j (·). The main regression command lsprobust() will output the the following quantities for uniform analyses upon setting uni.out=TRUE: • t.num.pl,t.num.wb1,t.num.wb2: The numerators of approximation processes except the "simulated components", which are evaluated at a set of pre-specified grid points K. Suppose that K contains L grid points. Then for the plug-in method, the numerator, stored in t.num.pl, For wild bootstrap, the numerator is separated to t.num.wb1 and t.num.wb2, which are γ q,j (x) /n : x ∈ K and (Π j (x 1 ), . . . , Π(x n )) respectively.
• t.denom: The denominator of approximation processes, i.e., Ω j (x)/n : x ∈ K , stored in a vector of length L.
• res: Residuals from the specified bias-corrected regression (needed for bootstrap-based approximation).

Linear combinations
The package lspartition also includes a function lsplincom(), which implements estimation and inference for a linear combination of regression functions of different subgroups. To be concrete, consider a random trial with G groups. Let µ(x; g) be the conditional expectation function (CEF) for group g, g = 1, . . . , G. The parameter of interest is θ(x) = ∑ G g=1 r g ∂ q µ(x; g), i.e., a linear combination of CEFs (or derivatives thereof) for different groups. To fix ideas, consider the most common application, To implement estimation and inference for θ(x), lsplincom() first calls lsprobust() to obtain a point estimate ∂ q µ 0 (x; g) and all other objects for each group. The tuning parameter for each group can be selected by the data-driven procedures above. Then the point estimate of θ(x) is The standard error of θ j (x) can be obtained simply by taking the appropriate linear combination of standard errors for each ∂ q µ j (x; g) and their estimated covariances. Robust confidence intervals can be similarly constructed as in (5). lsplincom() also allows users to construct confidence bands for θ(·). Specifically, it requests lsprobust() to output the numerators (t.num.pl for "plug-in", or t.num.wb1 and t.num.wb2 for "bootstrap") and denominators (t.denom) of the feasible approximation processes Z j (·) or z * (·). Let U j (·; g) and v j (·; g) denote the numerator and denominator from group g based on bias correction approach j, g = 1, . . . , G and j = 1, 2, 3. The approximation process for the t-statistic process based on θ j (x) is Z j,θ (·) = ∑ G g=1 r g U j (·; g)N g,K j,g ∑ G g=1 r 2 g v j,g (·) 2 where {N g,K j,g } G g=1 is a collection of independent standard normal vectors, and K j,g indicates the dimension of N g,K j,g . As discussed before, the dimensionality of these normal vectors depends on the particular bias correction approach and may vary across groups since the selected number of knots may be different across groups. The bootstrap approximation process z * j,θ (·) can be constructed similarly.
Given these processes, inference is implemented by sampling from G standard normal vectors ("plug-in" method) or G groups of Rademacher vectors given the data. Then critical values used to construct 100(1 − α) confidence bands for θ(·) are estimated similarly by 100(1 − α) empirical quantiles of sup x∈X | Z j,θ (x)| or sup x∈X | z * j,θ (x)|. As an illustration, we compare the number of rentals during working days and other time periods (weekends and holidays) based on linear splines and plug-in bias correction. To begin with, we first estimate the conditional mean function for each group using the command lsprobust().