Tools for Analyzing R Code the Tidy Way

With the current emphasis on reproducibility and replicability, there is an increasing need to examine how data analyses are conducted. In order to analyze the between researcher variability in data analysis choices as well as the aspects within the data analysis pipeline that contribute to the variability in results, we have created two R packages: matahari and tidycode. These packages build on methods created for natural language processing; rather than allowing for the processing of natural language, we focus on R code as the substrate of interest. The matahari package facilitates the logging of everything that is typed in the R console or in an R script in a tidy data frame. The tidycode package contains tools to allow for analyzing R calls in a tidy manner. We demonstrate the utility of these packages as well as walk through two examples.

Lucy D’Agostino McGowan (Wake Forest University) , Sean Kross (UC San Diego) , Jeffrey Leek (Johns Hopkins Bloomberg School of Public Health)
2020-09-10

Introduction

With the current emphasis on reproducibility and replicability, there is an increasing need to examine how data analyses are conducted (Goecks et al. 2010; Peng 2011; Ioannidis et al. 2014; McNutt 2014; Miguel et al. 2014; Richard 2014; Leek and Peng 2015; Nosek et al. 2015; Sidi and Harel 2018). In order to accurately replicate a result, the exact methods used for data analysis need to be recorded, including the specific analytic steps taken as well as the software utilized (Waltemath and Wolkenhauer 2016). Studies across multiple disciplines have examined the global set of possible data analyses that can be conducted on a specific data set (Silberzhan et al. 2018). While we are able to define this global set, very little is known about the actual variation that exists between researchers. For example, it is possible that the true range of data analysis choices is realistically a much more narrow set than the global sets that are presented. There is a breadth of excellent research and experiments examining how people read visual information (Buja et al. 2009; Majumder et al. 2013; Wickham et al. 2015; Loy et al. 2016, 2017), for example the Experiments on Visual Inference detailed here: (http://mamajumder.github.io/html/experiments.html), but not how they actually make analysis choices, specifically analysis coding choices. In addition to not knowing about the “data analysis choice” variability between researchers, we also do not know which portions of the data analysis pipeline result in the most variability in the ultimate research result. We seek to build tools to analyze these two aspects of data analysis:

  1. The between researcher variability in data analysis choices

  2. The aspects within the data analysis pipeline that contribute to the variability in results

Specifically, we have designed a framework to conduct such analyses and created two R packages that allow for the study of data analysis code conducted in R. In addition to answering these crucial questions for broad research fields, we see these tools having additional concrete use cases. These tools will facilitate data science and statistics pedagogy, allowing researchers and instructors to investigate how students are conducting data analyses in the classroom. Alternatively, a researcher could use these tools to examine how collaborators have conducted a data analysis. Finally, these tools could be used in a meta-manner to explore how current software and tools in R are being utilized.

Tidy principles

We specifically employ tidy principles in our proposed packages. Tidy refers to an implementation strategy propagated by Hadley Wickham and implemented by the Tidyverse team at RStudio (Wickham and Grolemund 2016) Here, by tidy we mean our packages adhere to the following principles:

  1. Our functions follow the principles outlined in R packages  (Wickham 2015) as well as the tidyverse style guide (Wickham 2019).

  2. Our output data sets are tidy, as in:

By implementing these tidy principles, and thus outputting tidy data frames, we allow for data manipulation and analysis to be conducted using a specific set of tools, such as those included in the tidyverse meta package (Wickham et al. 2019).

Ultimately, we create a mechanism to utilize methods created for natural language processing; here the substrate is code rather than natural language. We model our tools to emulate the tidytext package (Silge and Robinson 2016, 2017); instead of analyzing tokens of text, we are analyzing tokens of code.

We present two packages, matahari, a package for logging everything that is typed in the R console or in an R script, and tidycode, a package with tools to allow for analyzing R calls in a tidy manner. In this paper, we first explain how these packages work. We then demonstrate two examples, one that analyzes data collected from an online experiment, and one that analyzes “old” data via previously created R scripts.

Methods

We have created two R packages, matahari and tidycode. The former is a way to log R code, the latter allows the user to analyze R calls on the function-level in a tidy manner. Figure 1 is a flowchart of the process described in more detail below. This flowchart is adapted from Figure 2.1 in Text Mining with R: A Tidy Approach (Silge and Robinson 2017).

graphic without alt text
Figure 1: A flowchart of a typical analysis that uses matahari and tidycode to analyze and classify R code.

We demonstrate how to create these tidy data frames of R code and then emulate the data analysis workflow similar to that put forth in the tidy text literature.

Terminology

In this paper, we refer to R “expressions” or “calls” as well as R “functions” and “arguments”. An R call is a combination of an R function with arguments. For example, the following is an R call (Example 1).

library(tidycode)

Example 1. R call, library

Another example of an R call is the following piped chain of functions from the dplyr package (Example 2).

starwars %>%
  select(height, mass)

Example 2. Piped R call

Specifically, we know something is a call in R if is.call() is TRUE.

quote(starwars %>%
  select(height, mass)) %>%
  is.call()
#> [1] TRUE

Calls in R are made up of a function or name of a function, and arguments. For example, the call library(tidycode) from Example 1 is comprised of the function library() and the argument tidycode. Example 2 is a bit more complicated. The piped code can be rewritten, as seen in Example 3.

`%>%`(starwars, select(height, mass))

Example 3. Rewritten piped R call

From this example, it is easier to see that the function for this R call is %>% with two arguments, starwars and select(height, mass). Notice that one of these arguments is an R call itself, select(height, mass).

matahari

matahari is a simple package for logging R code in a tidy manner. It can be installed from CRAN using the following code.

install.packages("matahari")

There are three ways to use the matahari package:

  1. Record R code as it is typed and output a tidy data frame of the contents

  2. Input a character string of R code and output a tidy data frame of the contents

  3. Input an R file containing R code and output a tidy data frame of the contents

In the following sections, we will split these into two categories, tidy logging from the R console (1) and tidy logging from an R script (2 and 3).

Tidy logging from the R console

In order to begin logging from the R console, the dance_start() function is used. Logging is paused using dance_stop() and the log can be viewed using dance_tbl(). For example, the following code will result in the subsequent tidy data frame.

library(matahari)
dance_start()
1 + 2
"here is some text"
sum(1:10)
dance_stop()
dance_tbl()
#> # A tibble: 6 x 6
#>   expr        value             path       contents   selection dt                  
#>   <list>      <list>            <list>     <list>     <list>    <dttm>             
#> 1 <languag... <S3: sessionIn... <lgl [1... <lgl [1... <lgl [1]> 2018-09-11 22:22:12
#> 2 <languag... <lgl [1]>         <lgl [1... <lgl [1... <lgl [1]> 2018-09-11 22:22:12
#> 3 <languag... <lgl [1]>         <lgl [1... <lgl [1... <lgl [1]> 2018-09-11 22:22:12
#> 4 <chr [1]>   <lgl [1]>         <lgl [1... <lgl [1... <lgl [1]> 2018-09-11 22:22:12
#> 5 <languag... <lgl [1]>         <lgl [1... <lgl [1... <lgl [1]> 2018-09-11 22:22:12
#> 6 <languag... <S3: sessionIn... <lgl [1... <lgl [1... <lgl [1]> 2018-09-11 22:22:12

Example 4. Logging R code from the R console using matahari

The resulting tidy data frame consists of 6 columns: expr, the R call that was run, value, the value that was output, path, if the code was run within RStudio, this will be the path to the file in focus, contents, the file contents of the RStudio editor tab in focus, selection, the text that is highlighted in the RStudio editor tab in focus, and dt, the date and time the expression was run. By default, value, path, contents and selection will not be logged unless the argument is set to TRUE in the dance_start() function. For example, if the analyst wanted the output data frame to include the values computed, they would input dance_start(value = TRUE).

In this particular data frame, there are 6 rows. The first and final rows report the R session information at the time when dance_start() was initiated (row 1) and when dance_stop() was run (row 6). The second row holds the R call dance_start(), the first command run in the R console, was run; the third row holds 1 + 2, the fourth holds here is some text, and the fifth holds sum(1:10).

dance_tbl()[["expr"]]
#> [[1]]
#> sessionInfo()
#> 
#> [[2]]
#> dance_start()
#> 
#> [[3]]
#> 1 + 2
#> 
#> [[4]]
#> [1] "here is some text"
#> 
#> [[5]]
#> sum(1:10)
#> 
#> [[6]]
#> sessionInfo()

These functions work by saving an invisible data frame called .dance that is referenced by dance_tbl(). Each time dance_start() is subsequently run after dance_stop(), new rows of data are added to this data frame. This invisible data frame exists in a new environment created by the matahari package. We can remove this data frame by running dance_remove().

This data frame can be manipulated using common R techniques. Below, we rerun the same code as above, this time saving the values that are computed in the R console by using the value = TRUE parameter.

dance_start(value = TRUE)
1 + 2
"here is some text"
sum(1:10)
dance_stop()
tbl <- dance_tbl()

As an example of the type of data wrangling that this tidy format allows for, using dplyr and purrr, we can manipulate this to only examine expressions that result in numeric values.

library(dplyr)
library(purrr)

t_numeric <- tbl %>%
  mutate(
    numeric_output = map_lgl(value, is.numeric)
  ) %>%
  filter(numeric_output)

t_numeric
#> # A tibble: 3 x 7
#>   expr       value     path      contents  selection dt                  numeric_output
#>   <list>     <list>    <list>    <list>    <list>    <dttm>              <lgl>         
#> 1 <language> <int [1]> <lgl [1]> <lgl [1]> <lgl [1]> 2019-04-29 22:39:05 TRUE          
#> 2 <language> <dbl [1]> <lgl [1]> <lgl [1]> <lgl [1]> 2019-04-29 22:39:05 TRUE          
#> 3 <language> <int [1]> <lgl [1]> <lgl [1]> <lgl [1]> 2019-04-29 22:39:05 TRUE 

Here, three rows are output, since we have filtered to only calls with numeric output:

  1. The dance_start() call (this defaults to have a numeric value of

  2. The 1 + 2 call, resulting in a value of 3

  3. The sum(1:10), resulting in a value of 55

Tidy logging from an R script

In addition to allowing for the logging of everything typed in the R console, the matahari package also allows for the logging of pre-created R scripts. This can be done using the dance_recital() function, which allows for either a .R file or a character string of R calls as the input. For example, if we have a code file called sample_code.R, we can run dance_recital("sample_code.R") to create a tidy data frame. Alternatively, we can enter code directly as a string of text, such as dance_recital("1 + 2") to create the tidy data frame. Below illustrates this functionality.

code_file <- system.file("test", "sample_code.R", package = "matahari")
dance_recital(code_file)
#> # A tibble: 7 x 6
#>   expr       value     error             output    warnings  messages 
#>   <list>     <list>    <list>            <list>    <list>    <list>   
#> 1 <language> <dbl [1]> <NULL>            <chr [1]> <chr [0]> <chr [0]>
#> 2 <chr [1]>  <chr [1]> <NULL>            <chr [1]> <chr [0]> <chr [0]>
#> 3 <language> <dbl [1]> <NULL>            <chr [1]> <chr [0]> <chr [0]>
#> 4 <language> <NULL>    <S3: simpleError> <NULL>    <NULL>    <NULL>   
#> 5 <language> <chr [1]> <NULL>            <chr [1]> <chr [1]> <chr [0]>
#> 6 <language> <NULL>    <NULL>            <chr [1]> <chr [0]> <chr [1]>
#> 7 <language> <NULL>    <NULL>            <chr [1]> <chr [0]> <chr [0]>

Example 5. R call, Logging code