Working with Multilabel Datasets in R : The mldr Package

Most classification algorithms deal with datasets which have a set of input features, the variables to be used as predictors, and only one output class, the variable to be predicted. However, in late years many scenarios in which the classifier has to work with several outputs have come to life. Automatic labeling of text documents, image annotation or protein classification are among them. Multilabel datasets are the product of these new needs, and they have many specific traits. The mldr package allows the user to load datasets of this kind, obtain their characteristics, produce specialized plots, and manipulate them. The goal is to provide the exploratory tools needed to analyze multilabel datasets, as well as the transformation and manipulation functions that will make possible to apply binary and multiclass classification models to this data or the development of new multilabel classifiers. Thanks to its integrated user interface, the exploratory functions will be available even to non-specialized R users.


Introduction
Pattern classification is an important task nowadays and is in use everywhere, from our e-mail client, which is able to separate spam from legit messages, to credit institutions, that rely on it to detect fraud and grant or deny loans.All these cases operate with binary datasets, since a message is either spam or legit, and multiclass datasets, the loan is safe, medium, risky or highly risky, for instance.In both cases the user expects only one output.
The huge growth of the amount of information stored in late years on the web, such as blog posts, pictures taken from cameras and phones, videos hosted on YouTube, and messages on social networks, has led to more complex classification work.A blog post can be classified into several non-exclusive categories, for instance news, economy and politics simultaneously.A picture can be assigned a set of labels, such as landscape, sky and forest.A video can be labeled into several music genres at once, etc.All of these are examples of problems in need of multilabel classification.
Binary and multiclass datasets can be managed in R by using data frames.Usually the last attribute (column of the "data.frame") is the output class, which might contain only TRUE/FALSE values or values belonging to a finite set (a factor).Multilabel datasets (MLDs) can also be stored in an R "data.frame", but an additional structure to know which attributes are output labels is needed.Moreover, this kind of datasets have many specific characteristics that do not exist in the traditional ones.The average number of labels per instance, the imbalance ratio for each label, the number of labelsets (sets of labels assigned to each row) and their frequencies, and the level of concurrence among imbalanced labels are some of the traits that differentiate an MLD from the others.
Until now, most of the software to work with MLDs has been written in Java.The two best known frameworks are MULAN (Tsoumakas et al., 2011) and MEKA (Read and Reutemann, 2012).Both implementations rely on WEKA (Hall et al., 2009), which offers a large variety of binary and multiclass classifiers, as well as functions needed to deal with ARFF (Attribute-Relation File Format) files.Most of the existing MLDs are stored in ARFF format.MULAN and MEKA provide the specialized tools needed to deal with multilabel ARFFs, and the infrastructure to build multilabel classifiers (MLCs).Although R can access WEKA functionality through the RWeka (Hornik et al., 2009) package, handling MLDs is far from an easy task in R.This has been the main motivation behind the development of the mldr package.To the best of our knowledge, mldr is the first R package aimed to ease the work with multilabel data.
The mldr package aims to provide the user with the functions needed to perform exploratory analysis of MLDs, determining their main traits both statistically and visually.Moreover, it also brings the proper tools to manipulate this kind of datasets, including the application of the most common transformation methods, BR (Binary Relevance) and LP (Label Powerset), that will be described in the following section.These would be the foundation for processing MLDs with traditional classifiers, as well as for developing new multilabel algorithms.
The mldr package does not depend on the RWeka package, and it is not linked to MULAN nor MEKA.It has been designed to allow reading both MULAN and MEKA MLDs, but without any external dependencies.In fact, it would be possible to load MLDs stored in other file formats, as well as creating them from scratch.When loaded, MLDs are wrapped in an S3 type object with class "mldr", which allows for the use of methods.The object will contain the data in the MLD and also a large set of measures obtained from it.The functions provided by the package ease the access to this information, produce some specific plots, and make possible the manipulation of its content.A web-based graphical user interface, developed using the shiny (Chang et al., 2015) package, puts the exploratory analysis tools of the mldr package at the fingertips of all users, even those who have little experience using R.
In the following section the foundations related to MLDs and MLC will be briefly introduced.After that, the structure of the mldr package, and the operations it provides will be explained.Finally, the user interface provided by mldr to ease exploratory analysis tasks over MLDs will be shown.All code displayed in this paper is available in a vignette, accessible by entering vignette("mldr",package = "mldr").

Working with multilabel datasets
MLDs are generated from text documents (Klimt and Yang, 2004), sets of images (Duygulu et al., 2002), music collections, and protein attributes (Diplaris et al., 2005), among other sources.For each sample a set of features (input attributes) is collected, and a set of labels (the output labelset) is assigned.Usually there are several hundreds or even thousands of attributes, and it is not rare that a MLD has more labels than features.Some MLDs have only a few labels per instance, while others have dozens of them.In some MLDs the number of label combinations (labelsets) is quite short, whereas in others it can be very large.Most MLDs are imbalanced, which means that some labels are very frequent while others are scarcely represented.The labels in an MLD can be correlated or not.Moreover, frequent labels and rare labels can appear together in the same instances.
As can be seen, a lot of different scenarios can be found depending on the MLD characteristics.This is the reason why several specific measures have been designed to assess MLD traits (Tsoumakas et al., 2010), since they can have a serious impact on the MLC's performance.The following two subsections introduce several of these measures and some of the approaches pursued to face multilabel classification.

Multilabel dataset traits
The most common characterization measures for MLDs can be grouped into four categories, as depicted in Figure 1.
The most basic information that can be obtained from an MLD is the number of instances, attributes and labels.For any MLD containing |D| instances, any instance D i , i ∈ {1.. |D|} will be the union of a set of attributes and a set of labels where f is the number of input features and X j is the space of possible values for the j-th attribute, j ∈ {1.. f }.L being the full set of labels used in D, Y i could be any subset of items in L. Therefore, theoretically the number of potential labelsets could be 2 |L| .In practice this number tends to be limited by |D|.
Each instance D i has an associated labelset, whose length (number of active labels) can be in the range {0..|L|}.The average number of active labels per instance is the most basic measure of any MLD, usually known as Card (standing for cardinality).It is calculated as shown in Equation 1. Dividing this measure by the number of labels in L, as shown in Equation 2, results in a dimension-less measure, The R Journal Vol.7/2, December 2015 ISSN 2073-4859 known as Dens (standing for label density).
Most multilabel datasets are imbalanced, meaning that some of the labels are very frequent whereas others are quite rare.The level of imbalance of a label can be measured by the imbalance ratio, IRLbl, defined in Equation 3. To know how much imbalance there is in D, the MeanIR measure (Charte et al., 2015) is calculated as the mean imbalance ratio among all labels, as shown in Equation 4. In order to know the significance of this last measure, the standard CV (Coefficient of Variation, Equation 5) can be used.
The number of different labelsets, as well as the amount of them being unique labelsets (i.e., appearing only once in D), give us an idea on how sparsely the labels are distributed.The labelsets by themselves indicate how the labels in L are related.A very frequent labelset implies that the labels in it tend to appear jointly in D. The SCUMBLE measure, introduced in Charte et al. (2014) and shown in Equation 7, is used to assess the concurrence level among frequent and infrequent labels.
Besides the aforementioned insights, there are some other interesting traits that can be indirectly obtained from the previous measures, such as the ratio between input features and output labels, the maximum IRLbl, or the coefficient of variation in the imbalance of levels, among others.
Although the raw numbers given by these calculations describe the nature of any multilabel dataset to a good level, in general a visualization of its characteristics is desirable to ease its interpretation by researchers.
The information obtained from the previous measures depicts the characteristics of the dataset.These insights, along with other factors such as the loss function used by the classifier, help in choosing the most proper algorithm to learn from it and, in the future, make predictions on new data.Traditional classification models, such as trees and support vector machines, are designed to give only one output as result.Multilabel classification can mainly be faced through two different approaches discussed in the following.

Multilabel classification
• Algorithm adaptation: The goal is to modify existing algorithms taking into account the multilabel nature of the samples, for instance hosting more than one class in the leaves of a tree instead of only one.• Problem transformation: This approach transforms the original data to make it suitable to traditional classification algorithms, then combines the obtained predictions to build the labelsets given as output result.
Although several transformation methods have been defined in the specialized literature, there are two among them that stand out because they are the foundation for many others: The R Journal Vol.7/2, December 2015 ISSN 2073-4859 • Binary Relevance (BR): Introduced by Godbole and Sarawagi (2004) as an adaptation of OVA (One-vs-All) to the multilabel scenario, this method transforms the original multilabel dataset into several binary datasets, as many as there are different labels.In this way any binary classifier can be used by joining the individual predictions to generate the final output.
• Label Powerset (LP): Introduced by Boutell et al. (2004), this method transforms the multilabel dataset into a multiclass dataset by using the labelset of each instance as class identifier.Any multiclass classifier can be used, transforming back the predicted class into a labelset.
BR and LP have been used not only as a direct technique to implement multilabel classifiers, but also as a base method to build more sophisticated algorithms.Several ensembles of binary classifiers relying on BR have been proposed, such as CC (Classifier Chains) or ECC (Ensemble of Classifier Chains), both by Read et al. (2011).The same is applicable to the LP transformation, the foundation of ensemble multilabel classifiers such as RAkEL (Random k-Labelsets for Multi-Label Classification, Tsoumakas and Vlahavas 2007) and EPS (Ensemble of Pruned Sets, Read et al. 2008).
For the readers interested in more details, a recent review on multilabel classification has been published by Zhang and Zhou (2014).

The mldr package
R is among the most used tools when it comes to performing data mining tasks, including binary and multiclass classification.However, the work with MLDs in R is not as easy as it is with classic datasets.This is the main motivation behind the development of the mldr package, whose goals and functionality are described in this section.

Main goals of the mldr package
When we planned the development of this package, our main objective was to ease the exploration of MLDs in R.This included loading existing MLDs in different formats, as well as obtaining from them all the available information.These functions should be accessible to everyone, even to users not used to the R command line but to GUIs (Graphical User Interfaces) such as those provided by packages Rcmdr (aka R Commander, Fox 2005) or rattle (Williams, 2011).
At the same time, we aimed to include the tools needed to manipulate the MLDs, to apply filters and transformations, as well as to create MLDs from scratch.This functionality, directed to more experienced R users, opens the doors to implement other algorithms on top of mldr, for instance preprocessing methods or multilabel classifiers.

Installing and loading the mldr package
The mldr package is available from the Comprehensive R Archive Network (CRAN), therefore it can be installed as any other package, by simply typing: > install.packages("mldr")mldr depends on three R packages: XML (Lang and the CRAN Team, 2015), circlize (Gu et al., 2014) and shiny.The first one allows reading XML (eXtensible Markup Language) files, the second one is used to generate a specific type of plot (described below), and the third one is the base of its user interface.
Older releases of mldr, as well as the development version, are available at http://github.com/ fcharte/mldr.It is possible to install the development version using the install_github() function from devtools (Wickham and Chang, 2015).
Once installed, the package has to be loaded before it can be used.This can be done through the library() or require() functions, as usual.After loading the package three sample MLDs will be available: birds, emotions and genbase.These are contained in the 'birds.rda','emotions.rda'and 'genbase.rda'files, which are lazily loaded along with the package.
The mldr package uses its own internal representation for MLDs, which are assigned the "mldr" class.Inside an "mldr" object, such as the previous mentioned emotions or birds, both the data in the MLD and all the information obtained from this data can be found.

Loading and creating MLDs
Besides the three sample MLDs included in the package, the mldr() function allows to load any MLD stored in MULAN or MEKA file formats.Assuming that the files 'corel5k.arff' and 'corel5k.xml', which The R Journal Vol.7/2, December 2015 ISSN 2073-4859 hold the Corel5k (Duygulu et al., 2002) MLD in MULAN format, are in the current directory, the loading is done as follows: > corel5k <-mldr("corel5k") If the XML file is not available, it is possible to indicate just the number of labels in the MLD instead.In this case, the function assumes that the labels are at the end of the list of features.For instance: > corel5k <-mldr("corel5k", label_amount = 374) Loading an MLD in MEKA file format is equally easy.In this case there is no XML file with label information used, but a special header inside the ARFF file, a fact that will be indicated to mldr() with the use_xml argument: > imdb <-mldr("imdb", use_xml = FALSE) In all cases the result, as long as the MLD can be correctly loaded and parsed, will be a new "mldr" object ready to use.
If the MLD we are interested in is not in MULAN or MEKA format, firstly it will have to be loaded into a "data.frame",for instance using functions such as read.csv(),read.table() or a more specialized reader, and secondly this "data.frame"and an integer vector stating the indices of the labels inside it are given to the mldr_from_dataframe() function.This is a general function for creating an "mldr" object from any "data.frame",so it can also be used to generate new MLDs on the fly, as shown in the following example: > df <-data.frame(matrix(rnorm(1000),ncol = 10)) > df$Label1 <-c(sample(c(0,1), 100, replace = TRUE)) > df$Label2 <-c(sample(c(0,1), 100, replace = TRUE)) > mymldr <-mldr_from_dataframe(df, labelIndices = c(11, 12), name = "testMLDR") This will assign to mymldr an MLD, named testMLDR, with 10 input attributes and 2 labels.

Obtaining information from an MLD
After loading any MLD, a quick summary of its main characteristics can be obtained by means of the usual summary() The same is applicable for labelsets and attributes, by means of the labelsets and attributes elements of the class.
To access the MLD content, attributes and label values, the print() function can be used, as well as the dataset element of the "mldr" object.

Plotting functions
Exploratory analysis of MLDs can be tedious, since most of them have thousands of attributes and hundreds of labels.The mldr package provides a plot() function specific for dealing with "mldr" objects, allowing the generation of several specific types of plots.The first argument given to plot() must be an "mldr" object, while the second one specifies the type of plot to be produced.

> plot(emotions, type = "LH")
There are seven different types of plots available: three histograms showing relations between instances and labels, two bar plots with similar purpose, a circular plot indicating types of attributes and a concurrence plot for labels.All of them are shown in Figure 2, generated by the following code: > layout(matrix(c(1, 1, 7, 1, 1, 4, 5, 5, 4, 2, 6, 3), 4, 3, byrow = TRUE)) > plot(emotions, type = c("LC", "LH", "LSH", "LB", "LSB", "CH", "AT")) The concurrence plot is the default one, with type "LC", and responds to the need of exploring interactions among labels, and specifically between majority and minority ones.This plot has a circular shape, with the circumference partitioned into several disjoint arcs representing labels.Each arc has length proportional to the number of instances where the label is present.These arcs are in turn divided into bands that join two of them, showing the relation between the corresponding labels.The width of each band indicates the strength of the relation, since it is proportional to the number of instances in which both labels appear simultaneously.In this manner, a concurrence plot can show whether imbalanced labels appear frequently together, a situation which could limit the possible improvement of a preprocessing technique (Charte et al., 2014).
Since drawing interactions among a lot of labels can produce a confusing result, this last type of plot accepts more arguments: labelCount, which accepts an integer that will be used to generate the plot with that number of labels chosen at random; and labelIndices, which allows to indicate exactly the indices of the labels to be displayed in the plot.For example, in order to plot the first ten labels of genbase: > plot(genbase, labelIndices = genbase$labels$index[1:10]) The label histogram (type "LH") relates labels and instances in a way that shows how wellrepresented labels are in general.The X axis represents the number of instances and the Y axis the amount of labels.This means that if a large number of labels are appearing in very few instances, all data will be concentrated on the left side of the plot.On the contrary, if labels are generally present in many instances, data will tend to accumulate on the right side.This plot shows imbalance of labels when there is data accumulated on both sides of the plot, which implies that many labels are underrepresented, and a large amount are overrepresented as well.
The labelset histogram (named "LSH") is similar to the former.However, instead of representing the number of instances in which each label appears, it shows the amount of labelsets.This indicates quantitatively whether labelsets repeat consistently or not among instances.
The label and labelset bar plots display exactly the number of instances for each one of the labels and labelsets, respectively.Their codes are "LB" for the label bar plot and "LSB" for the labelset one.
The cardinality histogram (type "CH") represents the amount of labels instances have in general.Therefore data accumulating on the right side of the plot indicates that instances do have a notable amount of labels, whereas data concentrating on the left side shows the opposite situation.
The attribute types plot (named "AT") is a pie chart displaying the number of labels, numeric attributes and finite set (character) attributes, thus showing the proportions between these types of attributes to ease the understanding of the amount of input information and that of output data.
The R Journal Vol.7/2, December 2015 ISSN 2073-4859 Additionally, plot() accepts coloring arguments, col and color.function.The former can be used on all plot types except for the label concurrence plot, and must be a vector of colors.The latter is only used on the label concurrence plot and accepts a coloring function, such as rainbow or heat.colors,as can be seen in the following example: > plot(emotions, type = "LC", color.function= heat.colors)> plot(emotions, type = "LB", col = terrain.colors(emotions$measures$num.labels))

Transforming and filtering functions
Manipulation of datasets is a crucial task in multilabel classification.Since transformation is one of the main approaches to tackle the problem, both BR and LP transformations are implemented in package mldr.They can be obtained using the mldr_transform function, which accepts an "mldr" object as first argument, the type of transformation, "BR" or "LP", as second, and an optional vector of label The R Journal Vol.7/2, December 2015 ISSN 2073-4859 indices to be included in the transformation as last argument: > emotionsbr <-mldr_transform(emotions, type = "BR") > emotionslp <-mldr_transform(emotions, type = "LP", emotions$labels$index[1:4]) The BR transformation will return a list of "data.frame"objects, each one of them using one of the labels as class, whereas the LP transformation will return a single "data.frame"representing a multiclass dataset using each labelset as a class.Both of these transformations can be directly used in order to apply binary and multiclass classification algorithms, or even implement new ones.

instances [1] 222
Combined with the joining operator, +, this enables users to implement new preprocessing techniques that modify information in the MLD in order to improve classification results.For example, the following would be an implementation of an algorithm disabling majority labels on instances with highly imbalanced labels: > mldbase <-mld[.SCUMBLE <= mld$measures$scumble] > # Samples with coocurrence of highly imbalanced labels > mldhigh <-mld[.SCUMBLE > mld$measures$scumble] > majIndexes <-mld$labels[mld$labels$IRLbl < mld$measures$meanIR, "index"] > # Deactivate majority labels > mldhigh$dataset[, majIndexes] <-0 > mldbase + mldhigh # Join the instances without changes with the filtered ones In this last example, the first two commands filter the MLD, separating instances with their SCUMBLE lower than the mean and those with it higher.Then, the third line obtains the indices of the labels with lower IRLbl than the mean, thus these are the majority labels of the dataset.Finally, these labels are set to 0 in the instances with high SCUMBLE, and then the two partitions are joined again.
Lastly, another useful feature included in the mldr package is the MLD comparison with the == operator.This indicates whether both MLDs in comparison share the same structure, which would mean they have the same attributes, and these would have the same type.

Assessing multilabel predictive performance
Assuming that a set of predictions has been obtained for a MLD, e.g., through a set of binary classifiers, a multiclass classifier or any other algorithm, the next step would be to evaluate the classification performance.In the literature there exist more than 20 metrics for this task, and some of them are quite complex to calculate.The mldr package provides the mldr_evaluate function to accomplish this task, supplying both example based and label based metrics.
Multilabel evaluation metrics are grouped into two main categories: example based and label based metrics.Example based metrics are computed individually for each instance, then averaged to obtain the final value.Label based metrics are computed per label, instead of per instance.There are two approaches called micro-averaging and macro-averaging (described below).The output of the classifier can be a bipartition (i.e., a set of 0s and 1s denoting the predicted labels) or a ranking (i.e., a set of real values denoting the relevance of each label).For this reason, there are bipartition based and ranking based evaluation metrics for each one of the two previous categories.
D being the MLD, L the full set of labels used in D, Y i the subset of predicted labels for the i-th instance, and Z i the true subset of labels, the example/bipartition based metrics returned by mldr_evaluate are the following: • Accuracy: It is defined (see Equation 8) as the proportion of correctly predicted labels with respect to the total number of labels for each instance.
• Precision: This metric is computed as indicated in Equation 9, giving as result the ratio of relevant labels predicted by the classifier.
• Recall: It is a metric (see Equation 10) commonly used along with the previous one, measuring the proportion of predicted labels which are relevant.
• F-Measure: As can be seen in Equation 11, this metric is the harmonic mean between Precision (see Equation 9) and Recall (see Equation 10), providing a balanced assessment between precision and sensitivity.
• Hamming Loss: It is the most common evaluation metric in the multilabel literature, computed (see Equation 12) as the symmetric difference between predicted and true labels and divided by the total number of labels in the MLD.
• Subset Accuracy: This metric is also known as 0/1 Subset Accuracy and Classification Accuracy, and it is the most strict evaluation metric.The expr operator (see Equation 13) returns 1 when expr is true and 0 otherwise.In this case its value is 1 only if the predicted set of labels equals the true one.
Let rank (x i , y) be a function returning the position of y, a certain label, in the x i instance.The example/ranking based evaluation metrics returned by the mldr_evaluate function are the following ones: • Average Precision: This metric (see Equation 14) computes the proportion of labels ranked ahead of a certain relevant label.The goal is to establish how many positions have to be The R Journal Vol.7/2, December 2015 ISSN 2073-4859 traversed until this label is found.
|{y ∈ Y i : rank (x i , y ) ≤ rank (x i , y)}| rank (x i , y) .( 14) • Coverage: Defined as indicated in Equation 15, this metric calculates the extent to which it is necessary to go up in the ranking to cover all relevant labels.
• One Error: It is a metric (see Equation 16) which determines how many times the best ranked label given by the classifier is not part of the true label set of the instance.
• Ranking Loss: This metric (see Equation 17) compares each pair of labels in L, computing how many times a relevant label (member of the true labelset) appears ranked lower than a non-relevant label.In the equation, Y i denotes L\Y i .

RankingLoss
Regarding the label based metrics, there are two different ways to aggregate the values of the labels.The macro-averaging approach (see Equation 18) computes the metric independently for each label and then averages the obtained values to get the final measure.On the contrary, the micro-averaging approach (see Equation 19) first aggregates the counters for all the labels and then computes the metric only once.In the following equations TP, FP, TN and FN stand for True Positives, False Positives, True Negatives and False Negatives, respectively.
All the bipartition based metrics, such as Precision, Recall or FMeasure, can be computed as label based measures following these two approaches.In this category, there are as well as some ranking based metrics, such as MacroAUC (see Equation 20) and MicroAUC (see Equation 21).

MacroAUC
When the partition of the MLD for which the predictions have been obtained, and the predictions themselves are given to the mldr_evaluate function, a list of 20 measures is returned.For instance: > # Get the true labels in emotions > predictions <-as.matrix(emotions$dataset[,emotions$labels$index]) > # and introduce some noise > predictions[sample(1:593, 100), sample(1:6, 100, replace = TRUE)] <-+ sample(0:1, 100, replace = TRUE) > # then evaluate the predictive performance > res <-mldr_evaluate(emotions, predictions) > str(res) The R Journal Vol.7/2, December 2015 ISSN 2073-4859 The GUI will initially display the Main section, as shown in Figure 4.It contains options able to select an MLD from those available, and to load a new one by uploading its ARFF and XML files onto the application.On the right side, several plots are stacked.These show the amount of attributes of each type (numeric, character or label), the amount of labels per instance, the amount of instances corresponding to labels and the number of instances related to labelsets.Each plot can be saved as an image on the file system.Right below these graphics, some tables containing basic measures are shown.The first one lists generic measures related to the entire MLD, and is followed by measures specific to labels, such as Card or Dens.The last table shows a summary of measures for labelsets.
The Labels section contains a table enumerating each label of the MLD with its relevant details and measures: its index in the attribute list, its count and frequency, its IRLbl and its SCUMBLE.Labels in this table can be reordered using the headers, and filtered by the Search field.Furthermore, if the list is longer than the number specified in the Show field, it will be split into several pages.The data shown in all tables can be exported to files in several formats.On the right side, a plot shows the amount of instances that have each label.This is an interactive plot, and allows the range of labels to be manipulated.
Since relations between labels can determine the behavior of new data, studying labelsets is important in multilabel classification.Thus, the section named Labelsets provides information about them, listing each labelset along with its count.This list can be filtered and split into pages as well, and is accompanied by a bar plot showing the count of instances per labelset.
In order to obtain statistical measures about input attributes, the Attributes section organizes all of them into a paged table, displaying their type and some data or measures according to it.If the attribute is numeric, then there will be a table containing its minimum and maximum values, its quartiles and its mean.On the contrary, if the attribute takes values from a finite set, each possible value will be shown along with its count in the MLD.
Lastly, concurrence among labels is provenly a factor to take into account when applying preprocessing techniques to MLDs.For this reason, the Concurrence section attempts to create an easy way of visualizing concurrence among labels (see Figure 5), with a label concurrence plot displaying the selected labels in the left-side table and their coocurrences represented by bands in the circle.By default, the ten labels with highest SCUMBLE are selected.The user is able to select and deselect other labels by clicking their corresponding row on the table.

Summary
In this paper the mldr package, aimed to provide exploratory analysis and manipulation tools for MLDs, has been introduced.The functions supplied by this package allow both loading existing MLDs and generating new ones.Several characterization measures and specific plots can be obtained for any MLD, and the content of an MLD can be extracted, filtered and joined, producing new MLDs.Any MLD can be transformed into a set of binary datasets or a multiclass dataset by means of the The R Journal Vol.7/2, December 2015 ISSN 2073-4859 In its current version, package mldr is a strong base to develop any preprocessing method for MLDs, as has been shown.The development of the mldr package will continue in the near future by including the tools needed to implement and evaluate multilabel classifiers.With this foundation, we aim to encourage other developers to incorporate their own algorithms into mldr, as we will do in forthcoming releases.

Figure 2 :
Figure 2: Plots generated by mldr's plot() function.The type of plot is indicated at the top-left corner of each rectangle.

Figure 4 :
Figure 4: Main page of the shiny based graphical user interface.

Figure 5 :
Figure 5: The plots can be customized and saved.
function, as shown below:Any of these measures can be individually obtained through the measures element of the "mldr" class, like this: Full information about the labels in the MLD, including the number of times they appear, their IRLbl and SCUMBLE measures, can be retrieved by using the labels element of the "mldr" class: