Methodology extending nonparametric goodness-of-fit tests to discrete null distributions has existed for several decades. However, modern statistical software has generally failed to provide this methodology to users. We offer a revision of R’s ks.test()
function and a new cvm.test()
function that fill this need in the R language for two of the most popular nonparametric goodness-of-fit tests. This paper describes these contributions and provides examples of their usage. Particular attention is given to various numerical issues that arise in their implementation.
Goodness-of-fit tests are used to assess whether data are consistent
with a hypothesized null distribution. The
In their most basic forms, these nonparametric goodness-of-fit tests are
intended for continuous hypothesized distributions, but they have also
been adapted for discrete distributions. Unfortunately, most modern
statistical software packages and programming environments have failed
to incorporate these discrete versions. As a result, researchers would
typically rely upon the
This paper presents a revision of R’s ks.test()
function and a new
cvm.test()
function to fill this void for researchers and
practitioners in the R environment. This work was motivated by the need
for such goodness-of-fit testing in a study of Olympic figure skating
scoring (Emerson and Arnold 2011). We first present overviews of the theory
and general implementation of the discrete Kolmogorov-Smirnov and
Cramér-von Mises tests. We discuss the particular implementation of the
tests in R and provide examples. We conclude with a short discussion,
including the state of existing continuous and two-sample Cramér-von
Mises testing in R.
The most popular nonparametric goodness-of-fit test is the
Kolmogorov-Smirnov test. Given the cumulative distribution function
When
The extension of the Kolmogorov-Smirnov test to non-continuous null
distributions is not straightforward. The formula of the test statistic
The implementation of the discrete Kolmogorov-Smirnov test involves two
steps. First, the particular test statistic is calculated (corresponding
to the desired one-sided or two-sided test). Then, the
The form of the test statistic is the same as in the continuous case; it
would seem that no additional work would be required for the
implementation, but this is not the case. Consider two non-decreasing
functions
Computing the maximum over these
If it is known that
where the discontinuities in
Having obtained the test statistic, the ks.test()
and in the papers of Conover and Gleser.
For larger sample sizes (or when requested for smaller sample sizes),
the classical Kolmogorov-Smirnov test is used and is known to produce
conservative p-values for discrete distributions; the revised
ks.test()
supports estimation of p-values via simulation if desired.
While the Kolmogorov-Smirnov test may be the most popular of the
nonparametric goodness-of-fit tests, Cramér-von Mises tests have been
shown to be more powerful against a large class of alternatives
hypotheses. The original test was developed by Harald Cramér and Richard
von Mises (Cramér 1928; Mises 1928) and further adapted by
(Anderson and Darling 1952), and (Watson 1961). The original test statistic,
As with the original Kolmogorov-Smirnov test statistic, these all have
test statistic null distributions which are independent of the
hypothesized continuous models. The
It has been shown that these tests can be more powerful than Kolmogorov-Smirnov tests to certain deviations from the hypothesized distribution. They all involve integration over the whole range of data, rather than use of a supremum, so they are best-suited for situations where the true alternative distribution deviates a little over the whole support rather than having large deviations over a small section of the support. (Stephens 1974) offers a comprehensive analysis of the relative powers of these tests.
Generalizations of the Cramér-von Mises tests to discrete distributions were developed in (Choulakian et al. 1994). As with the Kolmogorov-Smirnov test, the forms of the test statistics are unchanged, and the null distributions of the test statistics are again hypothesis-dependent. (Choulakian et al. 1994) does not offer finite-sample results, but rather shows that the asymptotic distributions of the test statistics under the null hypothesis each involve consideration of a weighted sum of independent chi-squared variables (with the weights depending on the particular null distribution).
Calculation of the three test statistics is done using the matrix algebra given by (Choulakian et al. 1994). The only notable difficulty in the implementation of the discrete form of the tests involves calculating the percentiles of the weighted sum of chi-squares,
where
for continuous functions
There is no analytic solution to the integral in ((6)), so
the integration is accomplished numerically. This seems fine in most
situations we considered, but numerical issues appear in the regime of
large test statistics
We resolve this problem by using a combination of two conservative approximations to avoid the numerical instability. First, consider the following inequality:
The values for the weighted sum can be bounded using a simple transformation and a chi-squared distribution of a higher degree of freedom. Second, consider the Markov inequality:
where the bound can be minimized over
The original formulation, numerical integration of ((6)), is
preferable for most
Functions ks.test()
and cvm.test()
are provided for convenience in
package dgof, available on
CRAN. Function ks.test()
offers a revision of R’s Kolmogorov-Smirnov
function ks.test()
from recommended package stats; cvm.test()
is a
new function for Cramér-von Mises tests.
The revised ks.test()
function supports one-sample tests for discrete
null distributions by allowing the second argument, y
, to be an
empirical cumulative distribution function (an R function with class
"ecdf"
) or an object of class "stepfun"
specifying a discrete
distribution. As in the original version of ks.test()
, the presence of
ties in the data (the first argument, x
) generates a warning unless
y
describes a discrete distribution. If the sample size is less than
or equal to 30, or when exact=TRUE
, exact exact = FALSE
(or when exact
is unspecified and the sample size is greater than 30) the classical
Kolmogorov-Smirnov null distribution of the test statistic is used and
resulting simulate.p.value=TRUE
option.
The function cvm.test()
is similar in design to ks.test()
. Its first
two arguments specify the data and null distribution; the only extra
option, type
, specifies the variant of the Cramér-von Mises test:
a numerical vector of data values.
an ecdf
or step-function (stepfun
) for specifying the null model
the variant of the Cramér-von Mises test; W2
is the default and
most common method, U2
is for cyclical data, and A2
is the
Anderson-Darling alternative.
As with ks.test()
, cvm.test()
returns an object of class "htest"
.
Consider a toy example with observed data of length 2 (specifically, the
values 0 and 1) and a hypothesized null distribution that places equal
probability on the values 0 and 1. With the current ks.test()
function
in R (which, admittedly, doesn’t claim to handle discrete
distributions), the reported
> stats::ks.test(c(0, 1), ecdf(c(0, 1)))
One-sample Kolmogorov-Smirnov test
data: c(0, 1)
D = 0.5, p-value = 0.5
alternative hypothesis: two-sided
Instead, the value of ks.test()
fixes this
problem when the user provides a discrete distribution:
> library(dgof)
> dgof::ks.test(c(0, 1), ecdf(c(0, 1)))
One-sample Kolmogorov-Smirnov test
data: c(0, 1)
D = 0, p-value = 1
alternative hypothesis: two-sided
Next, we simulate a sample of size 25 from the discrete uniform
distribution on the integers ks.test()
implementation. The first is the default two-sided
test, where the exact
> set.seed(1)
> x <- sample(1:10, 25, replace = TRUE)
> x
[1] 3 4 6 10 3 9 10 7 7 1 3 2 7
[14] 4 8 5 8 10 4 8 10 3 7 2 3
> dgof::ks.test(x, ecdf(1:10))
One-sample Kolmogorov-Smirnov test
data: x
D = 0.08, p-value = 0.9354
alternative hypothesis: two-sided
Next, we conduct the default one-sided test, where Conover’s method
provides the exact
> dgof::ks.test(x, ecdf(1:10),
+ alternative = "g")
One-sample Kolmogorov-Smirnov test
data: x
D^+ = 0.04, p-value = 0.7731
alternative hypothesis:
the CDF of x lies above the null hypothesis
In contrast, the option exact=FALSE
results in the
> dgof::ks.test(x, ecdf(1:10),
+ alternative = "g", exact = FALSE)
One-sample Kolmogorov-Smirnov test
data: x
D^+ = 0.04, p-value = 0.9231
alternative hypothesis:
the CDF of x lies above the null hypothesis
The
> dgof::ks.test(x, ecdf(1:10),
+ alternative = "g",
+ simulate.p.value = TRUE, B = 10000)
One-sample Kolmogorov-Smirnov test
data: x
D^+ = 0.04, p-value = 0.7717
alternative hypothesis:
the CDF of x lies above the null hypothesis
A different toy example shows the dangers of using R’s existing
ks.test()
function with discrete data:
> dgof::ks.test(rep(1, 3), ecdf(1:3))
One-sample Kolmogorov-Smirnov test
data: rep(1, 3)
D = 0.6667, p-value = 0.07407
alternative hypothesis: two-sided
If, instead, either exact=FALSE
is used with the new ks.test()
function, or if the original stats::ks.test()
is used, the reported
We demonstrate the Cramér-von Mises tests with the same simulated data.
> cvm.test(x, ecdf(1:10))
Cramer-von Mises - W2
data: x
W2 = 0.057, p-value = 0.8114
alternative hypothesis: Two.sided
> cvm.test(x, ecdf(1:10), type = "A2")
Cramer-von Mises - A2
data: x
A2 = 0.3969, p-value = 0.75
alternative hypothesis: Two.sided
We conclude with a toy cyclical example showing that the test is invariant to cyclic reordering of the support.
> set.seed(1)
> y <- sample(1:4, 20, replace = TRUE)
> cvm.test(y, ecdf(1:4), type = 'U2')
Cramer-von Mises - U2
data: y
U2 = 0.0094, p-value = 0.945
alternative hypothesis: Two.sided
> z <- y%%4 + 1
> cvm.test(z, ecdf(1:4), type = 'U2')
Cramer-von Mises - U2
data: z
U2 = 0.0094, p-value = 0.945
alternative hypothesis: Two.sided
In contrast, the Kolmogorov-Smirnov or the standard Cramér-von Mises
tests produce different results after such a reordering. For example,
the default Cramér-von Mises test yields y
and z
, respectively.
This paper presents the implementation of several nonparametric
goodness-of-fit tests for discrete null distributions. In some cases the
ks.test()
, no
simulations may be necessary necessary for these methods; they were
generally developed during an era when extensive simulations may have
been prohibitively expensive or time-consuming. However, this does raise
the possibility that alternative tests relying upon modern computational
abilities could provide even greater power in certain situations, a
possible avenue for future work.
In the continuous setting, both of the Kolmogorov-Smirnov and the Cramér-von Mises tests have two-sample analogues. When data are observed from two processes or sampled from two populations, the hypothesis tested is whether they came from the same (unspecified) distribution. With the discrete case, however, the null distribution of the test statistic depends on the underlying probability model, as discussed by (Walsh 1963). Such an extension would require the specification of a null distribution, which generally goes unstated in two-sample goodness-of-fit tests. We note that (Dufour and Farhat 2001) explored two-sample goodness-of-fit tests for discrete distributions using a permutation test approach.
Further generalizations of goodness-of-fit tests for discrete distributions are described in the extended study of (Wet and Venter 1994). There are existing R packages for certain type of Cramér-von Mises goodness-of-fit tests for continuous distributions. Functions implemented in package nortest (Gross 2006) focus on the composite hypothesis of normality, while package ADGofTest (Bellosta 2009) provides the Anderson-Darling variant of the test for general continuous distributions. Packages CvM2SL1Test (Xiao and Y. Cui 2009a) and CvM2SL2Test (Xiao and Y. Cui 2009b) provide two-sample Cramér-von Mises tests with continuous distributions. Package cramer (Franz 2006) offers a multivariate Cramér test for the two-sample problem. Finally, we note that the discrete goodness-of-fit tests discussed in this paper do not allow the estimation of parameters of the hypothesized null distributions (see (Lockhart et al. 2007) for example).
We are grateful to Le-Minh Ho for his implementation of the algorithm described in Niederhausen (1981) for calculating rectangular probabilities of uniform order statistics. We also appreciate the feedback from two reviewers of this paper as well as from a reviewer of (Emerson and Arnold 2011).
dgof, nortest, ADGofTest, CvM2SL1Test, CvM2SL2Test, cramer
This article is converted from a Legacy LaTeX article using the texor package. The pdf version is the official version. To report a problem with the html, refer to CONTRIBUTE on the R Journal homepage.
Text and figures are licensed under Creative Commons Attribution CC BY 4.0. The figures that have been reused from other sources don't fall under this license and can be recognized by a note in their caption: "Figure from ...".
For attribution, please cite this work as
Arnold & Emerson, "Nonparametric Goodness-of-Fit Tests for Discrete Null Distributions", The R Journal, 2011
BibTeX citation
@article{RJ-2011-016, author = {Arnold, Taylor B. and Emerson, John W.}, title = {Nonparametric Goodness-of-Fit Tests for Discrete Null Distributions}, journal = {The R Journal}, year = {2011}, note = {https://rjournal.github.io/}, volume = {3}, issue = {2}, issn = {2073-4859}, pages = {34-39} }