Two-sided Exact Tests and Matching Confidence Intervals for Discrete Data

There is an inherent relationship between two-sided hypothesis tests and confidence intervals. A series of two-sided hypothesis tests may be inverted to obtain the matching 100(1-α)% confidence interval defined as the smallest interval that contains all point null parameter values that would not be rejected at the α level. Unfortunately, for discrete data there are several different ways of defining two-sided exact tests and the most commonly used two-sided exact tests are defined one way, while the most commonly used exact confidence intervals are inversions of tests defined another way. This can lead to inconsistencies where the exact test rejects but the exact confidence interval contains the null parameter value. The packages exactci and exact2x2 provide several exact tests with the matching confidence intervals avoiding these inconsistencies as much as possible. Examples are given for binomial and Poisson parameters and both paired and unpaired 2 × 2 tables. Applied statisticians are increasingly being encouraged to report confidence intervals (CI) and parameter estimates along with p-values from hypothesis tests. The htest class of the stats package is ideally suited to these kinds of analyses, because all the related statistics may be presented when the results are printed. For exact two-sided tests applied to discrete data, a test-CI inconsistency may occur: the p-value may indicate a significant result at level α while the associated 100(1-α)% confidence interval may cover the null value of the parameter. Ideally, we would like to present a unified report (Hirji, 2006), whereby the p-value and the confidence interval match as much as possible.

Michael P. Fay
2010-6-01

CRAN packages used

exactci, exact2x2, exactci, exact2x2, exactci, exact2x2, PropCIs, rateratio.test, coin, perm

CRAN Task Views implied by cited packages

ClinicalTrials, Survival

Reuse

Text and figures are licensed under Creative Commons Attribution CC BY 4.0. The figures that have been reused from other sources don't fall under this license and can be recognized by a note in their caption: "Figure from ...".

Citation

For attribution, please cite this work as

Fay, "Two-sided Exact Tests and Matching Confidence Intervals for Discrete Data", The R Journal, 2010

BibTeX citation

@article{RJ-2010-008,
  author = {Fay, Michael P.},
  title = {Two-sided Exact Tests and Matching Confidence Intervals for Discrete Data},
  journal = {The R Journal},
  year = {2010},
  note = {https://doi.org/10.32614/RJ-2010-008},
  doi = {10.32614/RJ-2010-008},
  volume = {2},
  issue = {1},
  issn = {2073-4859},
  pages = {53-58}
}