revengc : An R package to reverse engineer summarized data

Decoupled (e.g. separate averages) and censored (e.g. > 100 species) variables are continually reported by many well-established organizations, such as the World Health Organization (WHO), Centers for Disease Control and Prevention (CDC), and World Bank. The challenge therefore is to infer what the original data could have been given summarized information. We present an R package that reverse engineers censored and/or decoupled data with two main functions. The cnbinom.pars() function estimates the average and dispersion parameter of a censored univariate frequency table. The rec() function reverse engineers summarized data into an uncensored bivariate table of probabilities.


Introduction
The revengc R package was originally developed to help model building occupancy [1].Household size and area of residential structures are typically found in any given national census.If a census revealed the raw data or provided a full uncensored contingency table (household size × area), computing interior density as people per area would be straightforward.However, household size and area are often reported as decoupled variables (separate univariate frequency tables, average values, or a combination of the two).Furthermore, if a contingency table is provided, it typically left (<, ≤), right (>, ≥, +), and interval (−) censored.This summarized information is problematic for numerous reasons.How can a people per area ratio be calculated when no affiliation between the variables exist?If a census reports a household size average of 5.3, then how many houses are there with 1 person, 2 people, . . ., 10 people?If a census reports that there are 100 houses in an area of 26−50 square meters, then how many houses are in 26, 27, . . ., 50 square meters?
A tool that approximates negative binomial parameters from a censored univariate frequency table as well as estimates interior cells of a contingency table governed by negative binomial and/or Poisson marginals can also be useful for other areas ranging from demographic and epidemiological data to ecological inference problems.For example, population and community ecologist could unpack censored organism counts or average life expectancy values.Moreover, other summarized examples include the average number of births, the number of new disease cases, the number of mutations in a gene or average mutation rate, etc.We attempt to accommodate for various application of count data by offering five scenarios that can be reverse engineered: 1. cnbinom.pars() -An univariate frequency table estimates an average and dispersion parameter 2. rec() -Decoupled averages estimates an uncensored contingency table of probabilities 3. rec() -Decoupled frequency tables estimates an uncensored contingency table of probabilities 4. rec() -An average and frequency table estimates an uncensored contingency table of probabilities

rec() -A censored contingency table estimates an uncensored contingency table of probabilities
This paper proceeds with our reverse engineering methodology for the two main functions, cnbinom.pars()and rec().We provide an in-depth analysis of how we implemented both the negative binomial and Poisson distribution as well as the truncdist [2,3] and mipfp R package [4,5].Since the revengc package has specific input requirements for both cnbinom.pars()and rec(), we continue with an explanation of how to format the input tables properly.We then provide coded examples that implements revengc on national census data (household size and area) and end with concluding remarks.

Methodology: cnbinom.pars()
The methodology for the cnbinom.pars()function is relatively straightforward.To estimate an average µ and dispersion r parameter, a censored frequency table is fit to a negative binomial distribution using a maximum log-likelihood function customized to handle left (<, ≤), right (>, ≥, +), and interval (−) censored data.To show an example, first recall the negative binomial distribution P (X = x | µ, r) parameterized as a distribution of the number of failures X before the r th success in independent trials (1).With success probability p in each trail, r ≥ 0 and 0 ≤ p ≤ 1 [6].
Now consider an arbitrary censored frequency table x that has a combination of left censored (x < c), interval censored (a ≤ x ≤ b), and right censored (x > d) data (i.e.a, b, c, and d represent the censoring limits).The optimal µ and r parameter for x maximizes its custom log-likelihood function (2).

Negative Binomial and Poisson Distribution
When only an average is provided, we assume the average and variance are equal and rely on a Poisson distribution (i.e. the probability of observing x events in a given interval is given by Equation 3).We understand that there are many cases where data has more variation than what is indicated by the Poisson distribution (e.g.overdispersion).However, with limited data, the Poisson distribution is implemented due to its convenient property of having only one parameter, λ = average.For the cases with more data (e.g.univariate frequency table(s) or censored contingency table), we account for dispersion by relying on the more flexible negative binomial distribution (1).Hence, in these cases, the cnbinom.pars()function estimates the optimal average µ and dispersion r parameters.

The truncdist R package
With the negative binomial (µ and r) and/or Poisson (λ) parameters, rec() calculates truncated distributions to represent uncensored row (Xlowerbound:Xupperbound) and column (Ylowerbound:Yupperbound) margins.Calculations use the truncdist R package, and to provide a reference, Equation 4gives the probability density function of a truncated X distribution over the interval (a,b] (i.e. the negative binomial and/or Poisson probability density function is represented by g(•) and their corresponding cumulative distribution function is denoted by G(•)).Note, truncated distributions are very practical in this context because the distributions (margins) are restricted to a desired row and column length.
The (a, b] interval needed for both X (row of contingency table) and Y (column of contingency table) can be selected intuitively or with a brute force method.If rec() outputs a final contingency table with higher probabilities near the edge(s) of the table, then it would make sense to increase the range of the bound(s).For both variables, this would just involve making the lower bound less, making the upper bound more, or doing a combination of the two.The opposite holds true as well.If the final contingency table in rec() has very low probabilities near the edge(s) of the table, the range of the particular bound(s) should be decreased.

3.4
The mipfp R package rec() utilizing the mipfp R package to calculate cross tabulation probability estimates, and mipfp requires fixed marginals, a seed estimation method, and a seed matrix.The row and column marginals are uncensored truncated distributions (see the truncdist R package section) while opportunities for sensitivity analysis are presented with the seed estimation method and seed matrix.For example, mipfp offers four seed estimation methods (Table 1); the default method in rec() is the iterative proportional fitting procedure.Although the algorithms vary, they all adjust cell proportions p xy in a X × Y contingency table to known marginal probabilities, π x+ and π +y (i.e.all interior cell estimates π xy are subject to marginal constraints (5)).For an expanded explanation of these methods, please refer to [7] and [8].[4].
The seed matrix input can be arbitrary, but rec() provides reasonable defaults.For the decoupled cases (two averages, two tables, or a combination of a table and average), the absence of additional information makes it difficult to say much about the joint distribution.Therefore, rec() assumes independence between the variables, which is equivalent in making the X × Y seed a matrix of ones; rec() converts the matrix of ones to probabilities.When a censored contingency table is provided, independence does not have to be assumed and the interior cells can be weighted.rec() creates the default seed matrix by first repeating probability cells, which correspond to the censored contingency table, for the newly created and compatible uncensored cross tabulations.Just as in the decoupled cases, the cell values in this matrix are also changed to probabilities.To see an example of the default seed for a censored contingency table, see the Worked exampled section.where censoredtable is a frequency table (censored and/or uncensored).A data.frame and matrix are acceptable classes.See the Data entry section for formatting.The output is a list consisting of an estimated average µ and dispersion parameter r.
The output is a list containing an uncensored contingency table of probabilities (rows range from Xlowerbound:Xupperbound and the columns range from Ylowerbound:Yupperbound) as well as the row and column parameters used in making the margins for the mipfp R package.

Data entry
The Note, less than or equal to (≤ and LE) is not equivalent to less than (< and L) and greater than or equal to (≥, +, and GE) is not equivalent to greater than (> and G).revengc also uses closed intervals.Table 2 shows three different examples that all give the same cnbinom.pars()output.
The censored contingency table for rec() has a similar format.The censored symbols should follow the requirements listed above.The table's class can be a data.frameor a matrix.The column names should be the Y category values.The first column should be the X category values and the row names can be arbitrary.The inside of the table are X × Y frequencies or probabilities; the tabulations must be non-negative if the seed.estimation.method= "ipfp" or strictly positive if the seed.estimation.method is "ml", "lsq", or "chi2".The row X and column Y marginal totals need to be placed in this table.The top left, top right, and bottom left corners of the table should be NA or blank.The bottom right corner can be a total cross tabulation sum value, NA, or blank.

Formatting tables in R
The code below shows how to format these tables properly in R.
cnbinom.pars()The cnbinom.pars()function has the following format: cnbinom.pars(censoredtable) input tables are formatted to accommodate most open source data.The univariate frequency table used in cnbinom.pars()and/or rec() needs to be a data.frameor matrix class with two columns and n rows.The categories must be in the first column with the frequencies or probabilities in the second column.Row names should never be placed in this table (the default row names should always be 1:n).Column names can be any character string.The only symbols accepted for censored data are listed below.

Table 2
is a formatted example.

Table 2 .
Examples of correctly formatted univariate tables.
[10]l Living Standards Survey[9]provides both a censored table and average for urban household size.We use the censored table to show that the cnbinom.pars()functioncalculatesaclose approximation to the provided average household size (4.4 people).Note, there is overdispersion in the data.In 2010, the Population Census Data -Statistics Indonesia provided over 60 censored contingency tables containing household member size by floor area of dwelling unit (square meter)[10].The tables are separated by province, urban, and rural.Here we use the rural Aceh Province table to show the multiple coding steps and functions implemented inside rec().This allows the user to see a methodology workflow in code form.The final uncensored household size by area estimated probability table, which implemented the "ipfp" seed estimation method and default seed matrix, has rows ranging from 1 (Xlowerbound) to 15 was designed to reverse engineer summarized and decoupled variables with two main functions: cnbinom.pars()and rec().Relying on a negative binomial distribution, cnbinom.pars()approximates the average and dispersion parameter of a censored univariate frequency table.rec() fills in missing interior cell values from observed aggregated data (e.g.decoupled average(s) and/or censored frequency table(s) or a censored contingency table A # table of row marginals provides average and dispersion for x row.marginal.table<-row.marginal(contingencytable.csv)x <-cnbinom.pars(row.marginal.table)# table of column marginals provides average and dispersion for y column.marginal.table<-column.marginal(contingencytable.csv)y <-cnbinom.pars(column.marginal.table)revengc