The R Journal: accepted article

This article will be copy edited and may be changed before publication.

The bdpar Package: Big Data Pipelining Architecture for R PDF download
Miguel Ferreiro-Díaz, Tomás R. Cotos-Yáñez, José R. Méndez and David Ruano-Ordás

Abstract In the last years, big data has become a useful paradigm for taking advantage of multiple sources to find relevant knowledge in real domains (such as the design of personalized marketing campaigns or helping to palliate the effects of several mortal diseases). Big data programming tools and methods have evolved over time from a MapReduce to a pipeline-based archetype. Concretely the use of pipelining schemes has become the most reliable way of processing and analysing large amounts of data. To this end, this work introduces bdpar, a new highly customizable pipeline-based framework (using the OOP paradigm provided by R6 package) able to execute multiple pre-processing tasks over heterogeneous data sources. Moreover, to increase the flexibility and performance, bdpar provides helpful features such as (i) the definition of a novel object-based pipe operator (%>|%), (ii) the ability to easily design and deploy new (and customized) input data parsers, tasks and pipelines, (iii) only-once execution which avoids the execution of previously processed information (instances) guaranteeing that only new both input data and pipelines are executed, (iv) the capability to perform serial or parallel operations according to the user needs, (v) the inclusion of a debugging mechanism which allows users to check the status of each instance (and find possible errors) throughout the process.

Received: 2020-04-30; online 2021-07-16
CRAN packages: R6, repo, drake, magrittr, rtweet, tuber, tm
CRAN Task Views implied by cited CRAN packages: HighPerformanceComputing, ReproducibleResearch, WebTechnologies, NaturalLanguageProcessing

CC BY 4.0
This article is licensed under a Creative Commons Attribution 4.0 International license.

  author = {Miguel Ferreiro-Díaz and Tomás R. Cotos-Yáñez and José R.
          Méndez and David Ruano-Ordás},
  title = {{The bdpar Package: Big Data Pipelining Architecture for R}},
  year = {2021},
  journal = {{The R Journal}},
  doi = {10.32614/RJ-2021-065},
  url = {}