GsymPoint: An R Package to Estimate the Generalized Symmetry Point, an Optimal Cut-off Point for Binary Classification in Continuous Diagnostic Tests

In clinical practice, it is very useful to select an optimal cutpoint in the scale of a continuous biomarker or diagnostic test for classifying individuals as healthy or diseased. Several methods for choosing optimal cutpoints have been presented in the literature, depending on the ultimate goal. One of these methods, the generalized symmetry point, recently introduced, generalizes the symmetry point by incorporating the misclassification costs. Two statistical approaches have been proposed in the literature for estimating this optimal cutpoint and its associated sensitivity and specificity measures, a parametric method based on the generalized pivotal quantity and a nonparametric method based on empirical likelihood. In this paper, we introduce GsymPoint, an R package that implements these methods in a user-friendly environment, allowing the end-user to calculate the generalized symmetry point depending on the levels of certain categorical covariates. The practical use of this package is illustrated using three real biomedical datasets.


Introduction
The classification of cases and controls is a common task in several fields. For example, it is conducted in the atmospheric sciences (rainy/non rainy day), finance (good/not good payer), sociology (good/not good citizen), industry (product of good/poor quality), computing science (spam/nonspam) or health sciences (healthy/diseased), among others. However, in this paper we will focus on the latter example, where we will be interested on the classification of individuals as healthy or diseased using a continuous biomarker or diagnostic test that will be based on a cutoff point or discrimination value c. If we suppose, without loss of generality, that high test values are associated with the disease under study, individuals with a diagnostic test value equal to or higher than c are classified as diseased (positive test) and individuals with a value lower than c are classified as healthy (negative test). The test can classify a diseased patient incorrectly, that is, a false negative decision, or detect a patient as diseased when his (or her) true status is healthy (a false positive misclassification). So, for each cutoff point c, it is necessary to quantify these errors to check the validity of the diagnostic test in clinical practice. In a similar way, the test can correctly classify a diseased patient (a true positive decision) or a healthy patient (a true negative decision). For each cutpoint c, the sensitivity and specificity measures of the accuracy of the diagnostic test can be defined from these correct decisions. The sensitivity (Se) is the probability of a true positive decision, that is, the probability of correctly classifying a diseased patient (true positive rate) and the specificity (Sp) is the probability of a true negative decision (true negative rate), that is, the probability of correctly classifying a healthy individual. Similarly, the probabilities of the incorrect classifications are defined as 1 − Se (false negative rate) and 1 − Sp (false positive rate).
Considering Se versus 1 − Sp for all possible values of c, a curve is obtained. This curve is called the Receiver Operating Characteristic (ROC) curve (Metz, 1978;Swets and Pickett, 1982) and it is a graphical global measure of the diagnostic accuracy of a continuous diagnostic test, independent of the cutpoint and the disease prevalence. In addition, it serves as a guide when selecting optimal cutoffs and as a measure of the overlapping of the test values between healthy and diseased populations. Numerical indexes defined from ROC curves are often used to summarize the accuracy of a diagnostic test. For instance, the area under the curve (AUC) (Bamber, 1975;Swets, 1979) is the most commonly used and it takes values between 0.5 (the AUC value of an uninformative test, the same as a random prediction) and 1 (the AUC value of a perfect test that classifies correctly all individuals, either healthy or diseased). The AUC is equal to the Mann-Whitney U statistic for comparing two distributions. Both of them can be interpreted as the probability that the diagnostic value of a randomly chosen diseased individual will be higher than the diagnostic value of a randomly chosen healthy one (Hanley and McNeil, 1982).
A key question in clinical practice is to find a cutpoint that "best" discriminates between patients with and without the disease. However, one cannot talk in absolute terms of a "best choice". This is the reason why several criteria for selecting optimal cutpoints have been proposed in the literature depending on the ultimate goal of such selection (see Youden, 1950;Feinstein, 1975;Metz, 1978;Schäfer, 1989;Vermont et al., 1991;Greiner, 1995;Pepe, 2003, for example). One of the best-known methods is based on selecting the cutpoint that provides the same value for the sensitivity and specificity. This point is known as the equivalence or symmetry point (Greiner, 1995;Defreitas et al., 2004;Adlhoch et al., 2011). Graphically, it corresponds with the operating point on the ROC curve that intersects the perpendicular to the positive diagonal line, that is, y = 1 − x, where x is the false positive rate. The symmetry point can also be seen as the point that maximizes simultaneously both types of correct classifications (Riddle and Stratford, 1999;Gallop et al., 2003), that is, it corresponds to the probability of correctly classifying any subject, whether it is healthy or diseased (Jiménez-Valverde, 2012, 2014. Additionally, the incorporation of costs for the misclassification rates in the estimation of optimal cutpoints is crucial for evaluating not only the test accuracy but also its clinical efficacy, although this aspect is not taken into account most of the times. So, an interesting generalization of the equivalence or symmetry point, c S , that takes into account the costs associated to the false positive and false negative misclassifications, C FP and C FN , respectively, is the generalized equivalence point or generalized symmetry point, c GS , that satisfies the following equation: where ρ = C FP C FN is the relative loss (cost) of a false-positive decision as compared with a false-negative decision (see López-Ratón et al., 2016, for more details). Similarly to the symmetry point, this costbased generalization is obtained by intersecting the ROC curve and the line y = 1 − ρx, where x is the false positive rate. Obviously, when ρ = 1 in Equation 1, the generalized symmetry point yields the traditional symmetry point. The reader can see some medical examples, that have taken into account the misclassification costs in their ROC analysis, in the review conducted by Cantor et al. (1999) where the Cost/Benefit (C/B) ratio is discussed (C/B= 1 ρ ). Additionally, Subtil and Rabilloud (2015) include some common values for the C/B ratio (C/B = 2, 5, 10, 100). High values of C/B ratio mean that it is considered more harmful not to treat a diseased individual than to treat a healthy one.
Two statistical approaches have been recently introduced in the literature (López-Ratón et al., 2016) to obtain point estimates and confidence intervals for the generalized symmetry point and its associated sensitivity and specificity measures, a parametric method based on the Generalized Pivotal Quantity (GPQ) under the assumption of normality (Weerahandi, 1993(Weerahandi, , 1995Lai et al., 2012), and a nonparametric method based on the Empirical Likelihood (EL) methodology without any parametric assumptions (Thomas and Grunkemeier, 1975;Molanes-López and Letón, 2011).
The availability of software for estimating optimal cutpoints in a user-friendly environment is very important and necessary for facilitating, mainly to the biomedical staff, the selection of optimal cutpoints in clinical practice. There are several packages in R to carry out this task, such as PresenceAbsence (Freeman and Moisen, 2008), DiagnosisMed (Brasil, 2010), pROC (Robin et al., 2011) and OptimalCutpoints . However, these packages only consider the classical non-parametric empirical method for estimating optimal cutpoints and accuracy measures, that is, none of them take into account recent methodology introduced in ROC analysis such as the GPQ and EL approaches above-mentioned (Molanes-López and Letón, 2011;Lai et al., 2012).
In this paper we present GsymPoint, a package written in R for estimating the generalized symmetry point (López-Ratón et al., 2017), which is freely available from the Comprehensive R Archive Network (CRAN) at http://CRAN.R-project.org/package=GsymPoint. Specifically, this package enables end-users to obtain point estimates and 100(1 − α)% confidence intervals (with α the signification level) for the cost-based generalization of the symmetry point, c GS , and its associated sensitivity and specificity measures, Se(c GS ) and Sp(c GS ), by means of the two statistical approaches pointed out above, the GPQ and EL approaches, that turn out to be more efficient than the empirical one. Therefore, we take into account in the estimation process when normality can be assumed, and we consider one of the most widely used criteria for selecting optimal cutpoints in clinical practice, the criterion that generalizes the method where the sensitivity and specificity indexes are the same, taking into account the misclassification costs, a very important issue when selecting the optimal cutpoint in a specific clinical setting. In addition, since the test accuracy can be influenced by specific characteristics such as the patient's age or gender, or the disease severity (Pepe, 2003), the GsymPoint package allows the computation of the generalized symmetry point for each level of a given categorical covariate. On one hand, the numerical output of GsymPoint includes the generalized symmetry point and its associated sensitivity and specificity indexes with their corresponding 100(1 − α)% confidence intervals. On the other hand, based on the graphical interpretation of the generalized symmetry point, the graphical output shows the empirical ROC curve and the line y = 1 − ρx.
The rest of paper is organized as follows. In Section 2.2, we briefly review the two methods included in our GsymPoint package for obtaining point estimates and confidence intervals for the generalized symmetry point and its sensitivity and specificity measures. In Section 2.3, we describe the general use of this package, describing the most important functions. In Section 2.4, we illustrate the practical application of the package using three real biomedical datasets on melanoma, prostate cancer, and coronary artery disease. Finally, in Section 2.5 we conclude with a discussion and some interesting future extensions.

Generalized symmetry point estimating methods
In this section we briefly explain the two methods included in the GsymPoint package for estimating and constructing confidence intervals for the generalized symmetry point c GS and its associated accuracy measures Sp(c GS ) and Se(c GS ). To make inference on these parameters of interest, we consider two independent samples of i.i.d. (independently and identically distributed) observations, Y 0k 0 n 0 k 0 =1 and Y 1k 1 n 1 k 1 =1 , taken from the healthy and diseased populations, Y 0 and Y 1 , respectively, with sample sizes n 0 and n 1 .

Generalized pivotal quantity method
Generalized confidence intervals refer to a parametric methodology based on the normality assumption, first introduced by Weerahandi (1993Weerahandi ( , 1995 and recently applied in the context of diagnostic studies to the Youden index by Lai et al. (2012) and Zhou and Qin (2013), and to the generalized symmetry point by López-Ratón et al. (2016).
Assuming that the diagnostic test in healthy and diseased populations Y i follows a normal distribution with mean µ i and standard deviation σ i , for i = 0, 1, it follows that the ROC curve has an explicit expression: where a = µ 1 −µ 0 and Φ denotes the standard normal cumulative distribution function (cdf).
Therefore, under the normality assumption, using if necessary a monotone transformation of Box-Cox type to achieve normality, it follows from Equations 1-2 that the generalized symmetry point c GS can be estimated from the following equation: For computing the GPQ-based confidence intervals of c GS , Sp(c GS ) and Se(c GS ), López-Ratón et al. (2016) follow the same reasoning as in Lai et al. (2012), based on considering their corresponding generalized pivotal quantities, that is, substituting a and b into Equation 3 with their generalized pivotal values. For instance, if R c GS denotes the generalized pivotal quantity for estimating c GS , and R c GS ,α denotes the αth quantile of the distribution of R c GS , then the corresponding 100(1 − α)% confidence interval for c GS based on its generalized pivotal quantity is given by the percentile method, that is, by (R c GS ,α/2 , R c GS ,1−α/2 ).

Empirical likelihood method
The empirical likelihood method was firstly introduced by Thomas and Grunkemeier (1975) that proposed the construction of EL-based confidence intervals for the Kaplan-Meier estimator. Nowadays, this methodology is an active area of research in several fields due to the good properties presented by EL-based confidence intervals and regions (see, for example, Molanes-López et al., 2009, among others). Moreover, this methodology has the advantages of easy implementation and not requiring any particular parametric assumption. In the recent literature, Molanes-López and Letón (2011) proposed a bootstrap-based EL approach to make inference on the Youden index and its associated optimal cutpoint, and López-Ratón et al. (2016) applied these same bootstrap-based EL ideas for estimating and constructing confidence intervals for the generalized symmetry point and its corresponding specificity and sensitivity measures. The key point in both works is that the optimal cutpoint of int erest can be seen as specific quantiles of the two populations involved.
As López-Ratón et al. (2016) mention, c GS can be seen as the Sp(c GS )-th quantile of the healthy population and the ρ(1 − Sp(c GS ))-th quantile of the diseased population. Considering that the value of Sp(c GS ) is known in advance, they derive the following log-likelihood function to make inference on the generalized symmetry point: where, for i = 0, 1,F i,g i is a kernel-type estimate of the cdf F i given bŷ with K(x) = y −∞ K(z)dz, K a kernel function and g i the corresponding smoothing parameter, for i = 0, 1. Therefore, assuming that Sp(c GS ) is known in advance, a nonparametric point estimate of c GS could be found by minimizing (c) over c, and a confidence interval for it could be obtained based on the usual χ 2 limiting distribution of (c). However, taking into account the fact that Sp(c GS ) is unknown, López-Ratón et al. (2016) propose to estimate this parameter first by means of a kernelbased method and then obtain a non-parametric point estimate of c GS by minimizing the previous log-likelihood function, (c), but where now the unknown parameter Sp(c GS ) is replaced by its estimate. Besides, they introduce a bootstrap-based EL approach that reproduces this procedure for each pair of resamples taken independently from the two original samples and construct percentile based confidence intervals from all the EL-based estimates previously calculated for those bootstrap resamples.

The GsymPoint package
In this section we present GsymPoint, a package written in R for estimating the generalized symmetry point (López-Ratón et al., 2017), which is freely available from the Comprehensive R Archive Network (CRAN) at http://CRAN.R-project.org/package=GsymPoint. This package enables end-users to obtain point estimates and 100(1 − α)% confidence intervals using two recent methodologies introduced in ROC analysis (Molanes-López and Letón, 2011;Lai et al., 2012) for the generalized symmetry point and its corresponding sensitivity and specificity indexes. Specifically, the two estimating methods presented in the previous section, the generalized pivotal quantity method (Weerahandi, 1993(Weerahandi, , 1995Lai et al., 2012;Zhou and Qin, 2013) and the empirical likelihood method (Thomas and Grunkemeier, 1975;Molanes-López and Letón, 2011) have been incorporated in a clear and user-friendly way for the end-user. The GsymPoint package only requires a data-entry file where each column indicates a variable and each row indicates an individual or patient. This dataset must, at least, contain the variable with the diagnostic test values, the variable that indicates the true disease status (diseased or healthy) and the variable with the levels of a categorical covariate of interest in case the optimal cutpoint has to be computed for each level of that covariate. The numerical output of GsymPoint package includes the generalized symmetry point and its corresponding sensitivity and specificity indexes with their associated 100(1 − α)% confidence intervals. In basis on the graphical interpretation of the generalized symmetry point, the graphical output shows the intersection point between the empirical ROC curve and the line y = 1 − ρx, that is, an empirical estimate of the operating point corresponding to the generalized symmetry point. Table 1 provides a summary of the most important functions include d in the package.

The gsym.point() function
The main function of the package is the gsym.point() function, which uses the selected method(s) (GPQ, EL or auto, where this last option automatically selects the most appropriate method based on the normality assumption) to obtain (parametric and/or nonparametric) confidence intervals and point estimates for c GS , Sp(c GS ) and Se(c GS ), and creates an object of class "gsym.point". The code to use the gsym.point() function is as follows: control.gsym.point() Used to set several parameters that control the estimation process of the optimal cutpoint. print() Print method for "gsym.point" class objects.
plot() Plot method for "gsym.point" class objects that shows in the same graph the empirical ROC curve and the line y = 1 − ρx. The intersection point between them is an empirical estimate of the operating point associated to the optimal cutpoint given by the generalized symmetry point. The methods argument is a character vector to select the estimation method(s) to be used. The possible options are: "GPQ","EL", c("GPQ","EL"), c("EL","GPQ") or "auto".
The data argument is the data frame containing all the needed variables: the diagnostic marker, the true disease status and, when necessary, the categorical covariate; the marker and status arguments are character strings with the names of the diagnostic test and the variable that distinguishes healthy from diseased individuals, respectively. The value codifying healthy individuals in this last variable status is indicated in the tag.healthy argument.
The categorical.cov argument is a character string with the name of the categorical covariate according to which c GS is automatically computed for each of its levels. By default it is NULL, that is, no categorical covariate is considered in the analysis.
The control argument indicates the output of the control.gsym.point() function, which controls the whole calculation process. This function will be explained in detail in the following subsection.
The CFN and CFP arguments are the misclassification costs of false negative and false positive classifications, respectively. The default value is 1 for both, that is, no misclassification costs are taking into account by default.
The confidence.level argument is the numerical value of the confidence level 1 − α considered for the construction of confidence intervals. By default it is equal to 0.95.
The trace argument is a logical value that shows information on the calculation progress if TRUE. By default it is FALSE.
The seed argument is a logical value, such that if TRUE, a seed is fixed for generating the trials when computing the confidence intervals, allowing the reproducibility of the results at any other time. The default value is FALSE.
The value.seed argument is the numerical value for the fixed seed if seed is TRUE. By default it is equal to 3.
The verbose argument is a logical value that allows to show extra information on progress of running. By default it is TRUE.
Some of these arguments, methods, data, marker, status and tag.healthy, are essential and must be specified in the call to the gsym.point() function because, otherwise, an error is produced. The other arguments, categorical.cov, control, confidence.level, trace, seed and value.seed, are optional and, if they are not specified explicitly in the call, the values by default are taken.

The control.gsym.point() function
It should be noted that there are some extra arguments, specific to each estimation method. We considered to include all of them in the control argument, which is a list of control values specified by calling to the control.gsym.point() function, designed to replace the default values yielded by the control.gsym.point() function. The arguments of the control.gsym.point() function are presented in Table 2.

B
The number of simulations in the empirical likelihood ("EL") method. The default value is 499 based on Carpenter and Bithell (2000). c_sampling The constant needed for resampling in the empirical likelihood ("EL") method. The default value is 0.25. c_F The constant needed for estimating the distribution in the empirical likelihood ("EL") method. The default value is 0.25. c_ELq The constant needed for estimating the empirical likelihood function in the empirical likelihood ("EL") method. The default value is 0.25. c_R The constant needed for estimating the ROC curve in the empirical likelihood ("EL") method. The default value is 0.25.

I
The number of replicates in the generalized pivotal quantity ("GPQ") method. The default value is 2500.

The summary.gsym.point() function
Numerical results are printed on the screen, and the output yielded by the summary.gsym.point() function or the summary() method always includes: • The matched call to the main function gsym.point().
• The estimated value of the area under the ROC curve (AUC) based on the Mann-Whitney U statistic.
• Information related to the generalized symmetry point: -The method(s) (EL and/or GPQ) used for estimating c GS , Sp(c GS ) and Se(c GS ).
-The point estimates and confidence intervals for c GS , Sp(c GS ) and Se(c GS ).
Apart from the matched call, that it is presented only once at the beginning, all the other information will be shown for each level of the categorical covariate if this is specified in the call, that is, if the categorical.cov argument in the gsym.point() function is not NULL. The call to this function is as follows: where the object argument is a "gsym.point" class object as produced by the gsym.point() function and the ellipsis ... indicate further arguments passed to or from other methods. None are used in this method.

The plot.gsym.point() function
The graphical output of the GsymPoint package is yielded by the plot.gsym.point() function or by the plot() method. This function plots the empirical ROC curve and the line y = 1 − ρx. The call to this function is as follows: plot(x, legend = TRUE, ...) where the x argument is a "gsym.point" class object as produced by the gsym.point() function, the argument legend is a logical value for including the AUC value in the legend when it is TRUE (value by default) and the ellipsis ... indicate further arguments passed to the plot.default() method.

Practical application of GsymPoint package
This section illustrates the use of the R-based GsymPoint package by means of three real biomedical datasets on melanoma, prostate cancer, and coronary artery disease.

Melanoma dataset
Dermatologists use a clinical scoring scheme without dermoscope (CSS) or a dermoscopic scoring scheme (DSS) to clinically evaluate lesions on the skin on the basis of several visible features such as asymmetry, border irregularity, colouration and size. We have considered a dataset on 72 patients with suspicious lesions of being a melanoma (Venkatraman and Begg, 1996). Taking into account that 21 melanomas were detected through biopsies, our objective here is to evaluate the capacity of the CSS as a potential non-invasive diagnostic marker for discriminating between melanomas and non-melanomas using the generalized symmetry point as the binary classification threshold. In the following, we illustrate the use of the GsymPoint package for estimating that optimal threshold value. The first step consists on attaching the GsymPoint package and the melanoma dataset as follows: > library("GsymPoint") > data("melanoma") With the following instruction, we can get some basic summary statistics of the variables included in the melanoma dataset: Max. :1.0000 To estimate the generalized symmetry point of the CSS marker, we need to call the gsym.point() function. For instance, as specified below.
> melanoma.cutpoint1 <-gsym.point(methods = "GPQ", data = melanoma, + marker = "X", status = "group", tag.healthy = 0, + categorical.cov = NULL, CFN = 2, CFP = 1, + control = control.gsym.point(), confidence.level = 0.95, + trace = FALSE, seed = TRUE, value.seed = 3, verbose = TRUE) In this call, we have considered that a false negative decision is 2 times more serious than a false positive decision, and so we have set CFN = 2 and CFP = 1 for the misclassification costs. Besides, we have considered the GPQ method of estimation (methods = "GPQ") because, according to the Shapiro-Wilk normality test, the CSS values can be assumed normally distributed in both melanoma and non-melanoma groups, and under this assumption the GPQ method is more adequate than the EL method in this case.
The melanoma.cutpoint1 object produced by this call is a list that consists of the following components: "GPQ", "methods", "call", and "data", as can be checked below with the names command: > names(melanoma.cutpoint1) [1] "GPQ" "methods" "call" "data" The last three components, "methods", "call" and "data" are, respectively, a character vector with the value of the argument methods used in the call, the matched call, and the data frame used in the analysis. The first component, "GPQ", contains the results provided by the GPQ method regarding the estimation of the generalized symmetry point. Below, we detail these results: > names(melanoma.cutpoint1$GPQ) [1] "Global" > names(melanoma.cutpoint1$GPQ$Global) [1] "optimal.result" "AUC" "rho" [4] "pvalue.healthy" "pvalue.diseased" The "optimal.result" component is a list with the point estimates and 100(1 − α)% confidence intervals of the generalized symmetry point and its associated sensitivity and specificity accuracy measures, "AUC" is the numerical value of the area under the ROC curve, "rho" is the numerical value of the costs ratio ρ = C FP C FN , "pvalue.healthy" is the p-value of the Shapiro-Wilk normality test used to check the normality assumption of the marker in the healthy population, and "pvalue.diseased" is the p-value of the Shapiro-Wilk normality test used to check the normality assumption in the diseased population.

Shapiro-Wilk test p-values
In this case, as can be seen above, the output provided by the summary.gsym.point() function shows: 1. An informative message indicating that the marker can be considered normally distributed in both groups, according to the Shapiro-Wilk normality test.
2. The Shapiro-Wilk test p-values indicating normality in both groups.
3. The AUC value and information related to the generalized symmetry point, that is, the point estimates and the GPQ based 100(1 − α)% confidence intervals for the generalized symmetry point and its corresponding sensitivity and specificity indexes. By default, confidence intervals are computed with a default confidence level, 1 − α, equal to 95%, although a different value can be set in the confidence.level argument of the main gsym.point() function.
So far, we have considered the GPQ method that, under normality assumptions, turns out to be more appropriate than the EL method. As you can see below, if we now replace the GPQ method by the EL method in the call, the program shows in the first place an informative message on the normality assumption and the better appropriateness of the GPQ method.

Shapiro-Wilk test p-values
Group 0 Group 1 Original marker 0.4719 0.9084 By means of the summary() function, we show below the results obtained with the EL method. In this case, after reproducing the call used to create the melanoma.cutpoint2 object, the same informative message is shown regarding the better appropriateness of the GPQ method to this dataset.

Shapiro-Wilk test p-values
The R Journal Vol. 9/1, June 2017 ISSN 2073-4859 Since both estimating methods are adequate for this dataset, we could specify them simultaneously in the call to the gsym.point() function as follows.

Shapiro-Wilk test p-values
Group 0 Group 1 Original marker 0.4719 0.9084 With the option methods = "auto" the program selects in this case the GPQ method based on the normality assumption satisfied for this dataset.
The graphical output of any of the three objects previously created, can be obtained by means of the plot.gsym.point() function, which can be abbreviated by the plot() method.  For instance, the call below > plot(melanoma.cutpoint1) The R Journal Vol. 9/1, June 2017 ISSN 2073-4859 shows the plot of the empirical Receiver Operating Characteristic (ROC) curve, the line y = 1 − ρx, and the intersection point between them, that is, an empirical estimate of the operating point associated to the generalized symmetry point of the CSS marker for discriminating between patients with and without melanoma. Although this is the default output (see Figure 1), the end-user can add specific graphic parameters, such as color, legend, etc.

Prostate cancer dataset
We have considered here the dataset on prostate cancer analyzed by Le (2006). In order to design an appropriate treatment strategy for a patient with prostate cancer, it is important to know first whether cancer has spread or not to the neighboring lymph nodes. Although a laparoscopic surgery could confirm the true status of the patient, it is interesting to find a non-invasive diagnostic method to predict whether nodal involvement is present or not. In this dataset, 20 patients (of the total of 55 patients) had nodal involvement, and the level of acid phosphatase in blood serum (APBS) (×100) was considered as the diagnostic marker for predicting nodal involvement. We illustrate below how to apply the GsymPoint package for estimating the generalized symmetry point of the APBS marker that will be used for discriminating between individuals with and without nodal involvement.
As shown below, after loading the Gsympoint package and the prostate cancer dataset, we use the gsym.point() function to estimate the generalized symmetry point of the APBS marker and its associated specificity and sensitivity indexes. > library("GsymPoint") > data("prostate") > prostate.cutpoint1 <-gsym.point(methods = "GPQ", data = prostate, + marker = "marker", status = "status", tag.healthy = 0, + categorical.cov = NULL, CFN = 10, CFP = 1, + control = control.gsym.point(I = 1500), confidence.level = 0.95, + trace = FALSE, seed = TRUE, value.seed = 3, verbose = TRUE) Since cancer is a very serious disease which can cause death, a FN result in this biomedical example is much more harmful than a FP result. Specifically, we have considered that a false negative classification is exactly 10 times more serious than a false positive classification (CFN = 10, CFP = 1). Since the Shapiro-Wilk normality test indicated that both groups could be assumed normally distributed (after a Box-Cox transformation of the data), the GPQ method is more adequate than the EL method in this case. For the sake of illustration, we have changed the default value of the number of replicates associated to the GPQ method by setting control = control.gsym.point(I = 1500) in the call to the gsym.point() function. The default value for this parameter is I = 2500.

Shapiro-Wilk test p-values
The R Journal Vol. 9/1, June 2017 ISSN 2073-4859 In this case, the numerical output obtained with the summary.gsym.point() function shows: 1. An informative message indicating that the original data can not be assumed normally distributed in both groups, but the Box-Cox transformed data can be considered normally distributed in both groups, according to the Shapiro-Wilk normality test. As can be checked with the command below, the prostate.cutpoint1 object yields a list with the following components: "GPQ", "methods", "call", and "data".
> prostate.cutpoint1$GPQ$Global$pvalue.healthy [1] 3.276498e-07 7. "pvalue.diseased" is the numerical value of the p-value obtained by the Shapiro-Wilk normality test for checking the normality assumption of the marker in the diseased population. Similarly to the previous example, if we now consider the EL method instead of the GPQ method for estimating the generalized symmetry point and its accuracy measures, an informative message is shown by the package, advising the user that the GPQ method is more suitable for this dataset due to the fact that the Box-Cox transformed marker can be assumed normally distributed in both groups. > prostate.cutpoint2 <-gsym.point(methods = "EL", data = prostate, + marker = "marker", status = "status", tag.healthy = 0, + categorical.cov = NULL, CFN = 10, CFP = 1, + control = control.gsym.point(B = 999), confidence.level = 0.95, + trace = FALSE, seed = TRUE, value.seed = 3, verbose = TRUE) According to the Shapiro-Wilk normality test, the marker can not be considered normally distributed in both groups. However, after transforming the marker using the Box-Cox transformation estimate, the Shapiro-Wilk normality test indicates that the transformed marker can be considered normally distributed in both groups. Therefore the GPQ method would be more suitable for this dataset.
Box-Cox lambda estimate = -1.2494 The results obtained with the EL method are the following.

Coronary artery disease dataset
We now consider a study conducted on 141 patients (96 with coronary lesions and 45 with non-stenotic coronaries) admitted to the cardiology department of a teaching hospital in Galicia (Northwest Spain) for evaluating chest pain or cardiovascular disease, where the leukocyte elastase determination was investigated as a potential clinical marker for the diagnosis of coronary artery disease (Amaro et al., 1995). Since in this biomedical example there is available information regarding the gender of the patient (female or male), we will illustrate the practical application of the GsymPoint package to these data taking into account the covariate gender, that is, the generalized symmetry point will be computed for each gender in the scale of the elastase concentration to diagnose coronary artery disease (CAD). From here on, we will refer to this dataset as elastase.
First of all, we need to load the GsymPoint package and the corresponding elastase dataset: > library("GsymPoint") > data("elastase") Now, for computing the generalized symmetry point in the elastase scale taking into account the categorical covariate gender, we simply have to include categorical.cov = "gender" in the call below.
> elastase.gender.cutpoint1 <-gsym.point(methods = "GPQ", + data = elastase, marker = "elas", status = "status", + tag.healthy = 0, categorical.cov = "gender", CFN = 10, CFP = 1, + control = control.gsym.point(), confidence.level = 0.95, + trace = FALSE, seed = TRUE, value.seed = 3, verbose = TRUE) In this case we are interested in having a high sensitivity, that is, a low number of false negatives. Therefore, the same values as in the previous prostate cancer dataset were considered for the misclassification costs, CFN = 10 and CFP = 1, that is, a false negative result is regarded as 10 times more serious than a false positive one. Since the elastase concentration in females and males follow the Box-Cox family in both CAD and non-CAD groups according to the Shapiro-Wilk normality test, the GPQ method of estimation is more adequate than the EL method in this case.
The elastase.gender.cutpoint1 object produced by the previous call is a list with the following components: > names(elastase.gender.cutpoint1) [1] "GPQ" "methods" "levels.cat" "call" [5] "data" Similarly to the previous datasets, the "methods" component is a character vector with the value of the argument methods used in the call, "call" is the matched call and "data" is the data frame used in the analysis. However, now there is an extra component, "levels.cat", a character vector indicating the levels of the categorical covariate ("Female" and "Male" in this case). Besides, the first component, "GPQ", that includes the results associated with the GPQ method is itself a two-component list with "Female" and "Male" as can be seen below.
> names(elastase.gender.cutpoint1$GPQ) [1] "Female" "Male" For each level of the categorical covariate, you get a list with nine components. For instance, for the subgroup of males, elastase.gender.cutpoint1$GPQ$Male contains the following components: > names(elastase.gender.cutpoint1$GPQ$Male) [1] "optimal.result" "AUC" [3] "rho" "lambda" [5] "normality.transformed" "pvalue.healthy" [7] "pvalue.diseased" "pvalue.healthy.transformed" [9] "pvalue.diseased.transformed" The "optimal.result" component of elastase.gender.cutpoint1$GPQ$Male is a list with the point estimates and 100(1 − α)% confidence intervals of the generalized symmetry point in the scale of the elastase concentration and its associated sensitivity and specificity accuracy measures in the ROC space corresponding to the subgroup of individuals that are males. A summary of the numerical results is shown bellow. In this case, the results obtained are shown for each level of the categorical covariate gender, that is, for females and males. However, similarly to previous examples, the output shows first the AUC value and then information related to the generalized symmetry point (point estimates and 100(1 − α)% confidence intervals obtained for each method of estimation specified in the main gsym.point() function). By default, confidence intervals are computed for a confidence level 1-α of 0.95, but this value can be changed directly in the confidence.level argument of the gsym.point() function.

Shapiro-Wilk test p-values
Female : According to the Shapiro-Wilk normality test, the marker can be considered normally distributed in both groups. Therefore the GPQ method would be more suitable for this dataset.

Male :
According to the Shapiro-Wilk normality test, the marker can not be considered normally distributed in both groups. However, after transforming the marker using the Box-Cox transformation estimate, the Shapiro-Wilk normality test indicates that the transformed marker can be considered normally distributed in both groups. Therefore the GPQ method would be more suitable for this dataset. The graphical results can be obtained by means of the plot.gsym.point() function or merely the plot() method. For instance, the following command > plot(elastase.gender.cutpoint1) yields the graphical output shown in Figure 3, where the empirical ROC curve of the elastase concentration is represented separately for females and males, together with the line y = 1 − ρx and the intersection point associated to the generalized symmetry point for detecting CAD. This is the output that appears by default for females and males, respectively. However, as usual, the end-user can change several graphical parameters, such as legends, colors, etc.

Discussion
The selection of optimal cutpoints in the scale of a quantitative diagnostic test or biomarker that can help in the diagnosis of a disease is an important issue in the clinical sciences. Depending on the main objective of such selection, several criteria of optimality have been proposed in the literature to carry out this task. One of the most popular in clinical practice is based on selecting the cutoff that provides the same sensitivity and specificity, known in the literature as the equivalence or symmetry point (Greiner, 1995;Defreitas et al., 2004;Adlhoch et al., 2011). However, this cutpoint is not valid in scenarios where the severity of misclassifying a diseased patient is not the same as the severity of misclassifying a healthy patient, which is normally the case in practice. For instance, in cancer disease, a false negative decision is in general more serious than a false positive decision. Hence, when selecting an optimal cutpoint it is very important to take into account the costs of the different incorrect diagnostic decisions. For this reason, the generalized symmetry point, a generalization of the symmetry point that incorporates the costs of the misclassifications is very adequate and useful in practice (López-Ratón et al., 2016). Although there are several R packages that implement different criteria for selecting the optimal cutoff point such as PresenceAbsence (Freeman and Moisen, 2008), DiagnosisMed (Brasil, 2010), pROC (Robin et al., 2011) and OptimalCutpoints , up to our knowledge, none of them includes the criterion based on the generalized symmetry point nor recent estimation approaches such as the GPQ and EL methods that provide more efficient estimates than the empirical ones (Molanes-López and Letón, 2011;Lai et al., 2012). In order to avoid that the use of the generalized symmetry point is limited in the clinical practice by the lack of software that implements all necessary computations to estimate it, we have developed the Gsympoint package, a user-friendly R package that fills this gap. As it has been illustrated in this paper, the GsymPoint package allows the possibility that the generalized symmetry point is straightforwardly estimated for each level of a certain categorical covariate that represents an individual characteristic such as gender, age or disease severity, and that may influence the discrimination capacity of the diagnostic test (Pepe, 2003). Possible interesting extensions of the GsymPoint package could be taken into account. For instance, the implementation of this same methodology to estimate other accuracy measures such as predictive values or diagnostic likelihood ratios, the incorporation of more efficient methods for estimating the generalized symmetry point and its accuracy measures, and the extension of this methodology to other more complex scenarios where, for instance, the diagnostic test is subject to the measurement of error, there is presence of partial disease verification (see Alonzo, 2014, and references therein), the disease status evolves over time and the disease onset time is subject to censoring (Rota et al., 2015) or there are continuous covariates available that may affect the diagnostic capacity of the biomarker and that should be taken into consideration.