Author

Affiliation

1 Literature Review: Reproducibility in Computational Research Using R

1.1 Introduction to the Reproducibility Crisis

The scientific community has increasingly recognized the critical importance of reproducibility in research, leading to what has been termed the “reproducibility crisis” (Baker, 2016). This crisis affects multiple disciplines, but is particularly acute in computational research where software environments, package dependencies, and operating system differences can significantly impact results.

Peng (2011) introduced a framework for reproducible research in computational science, arguing that standard scientific publication practices were insufficient for computational analyses. He proposed a spectrum of reproducibility, from “publication only” to “full replication,” advocating for code and data sharing as minimal requirements. Building upon this foundation, Stodden et al. (2016) surveyed practices across scientific fields, finding that while most researchers valued reproducibility, implementation of reproducible workflows remained challenging due to technical barriers.

1.2 Challenges in R-Based Reproducibility

The R ecosystem presents specific challenges for reproducibility. As identified by Marwick et al. (2018), R analysis code frequently fails to execute correctly across different computing environments due to:

Package version inconsistencies: R packages evolve rapidly, with functions and arguments changing over time
System dependencies: Many R packages rely on external libraries and system-level requirements
R version differences: Core functionality can vary between R versions
Operating system variations: Platform-specific behaviors can affect computational results

These challenges are compounded in collaborative research where team members may use different computing setups. Kitzes et al. (2017) presented case studies highlighting how seemingly minor environment differences could lead to significant variations in analytical outcomes.

1.3 Package Management Solutions for R

1.3.1 The renv Approach

Package management has evolved as a solution to dependency challenges within R. CRAN itself offers limited built-in versioning, necessitating additional tools. The renv package, developed by Ushey (2022), has emerged as a robust solution for creating reproducible R environments. renv addresses reproducibility at the package level by:

Creating project-specific library paths
Recording exact package versions in a lockfile
Facilitating restoration of specific package environments
Supporting transparent version control of dependencies

Prior to renv, similar functionality was offered by packrat (Ushey et al., 2018), which pioneered the isolation of project dependencies but had limitations in efficiency and usability that renv subsequently addressed. As observed by Peikert and Brandmaier (2021), renv represents an important advancement in making dependency management more accessible to R users.

Ooms (2013) earlier highlighted the importance of addressing package dependencies, proposing semantic versioning practices for R packages. This work informed the development of dependency management tools like renv, illustrating the progression of the field.

1.3.2 Alternative R Package Management Approaches

While renv has gained prominence, other approaches exist. The checkpoint package (Microsoft Corporation, 2022) takes a temporal approach, installing packages as they existed on CRAN at a specific date. This “time machine” approach provides a different model of reproducibility, focusing on temporal consistency rather than project-specific dependency tracking.

Another approach is represented by the miniCRAN package (de Vries & Meys, 2019), which allows creating local, customized CRAN-like repositories with specific package versions. This has proven especially useful in air-gapped or enterprise environments where internet access may be limited.

1.4 Containerization as a Solution for System-Level Reproducibility

1.4.1 Docker and Containerization

While package management addresses R-specific dependencies, containerization addresses system-level reproducibility. Boettiger (2015) introduced Docker as a solution for reproducible research, highlighting its ability to create self-contained computing environments that include not just the R environment but also the underlying operating system, libraries, and configurations.

Nüst et al. (2020) evaluated different containerization approaches for reproducible research, comparing Docker, Singularity, and other technologies. They found Docker to be particularly well-suited for reproducible R workflows due to its widespread adoption, ease of use, and comprehensive isolation properties.

1.4.2 The Rocker Project

The Rocker Project (Boettiger & Eddelbuettel, 2017) has been instrumental in making Docker accessible to the R community by providing standardized Docker images for R with different configurations. These pre-built images serve as reliable base environments for reproducible research, addressing a key barrier to adoption.

Eddelbuettel and Boettiger (2018) further documented how the Rocker Project supports different use cases, from basic R installations to complex data science stacks including RStudio, Shiny, and common data science packages. This work has made containerization more accessible to R users without extensive systems administration knowledge.

1.5 Integrating Package Management and Containerization

The integration of package management (renv) with containerization (Docker) represents a comprehensive approach to reproducibility. Peikert et al. (2021) proposed such an integrated workflow, demonstrating how combining these tools provides both package-level and system-level reproducibility.

Leveraging this combined approach, Rodríguez-Sánchez et al. (2016) developed workflows for ecological research that maintain reproducibility across collaborators and over time. Similarly, Wiebels and Moreau (2021) demonstrated the effectiveness of this integration for neuroimaging research, where computational environments are particularly complex.

1.6 Testing and Validation in Reproducible Workflows

Recent literature emphasizes the importance of integrating testing into reproducible workflows. Wickham (2011) introduced the testthat package, providing a framework for unit testing in R that has become a standard. Building on this, Parker (2017) argued that testing is an essential component of reproducibility, as it provides validation that code performs as expected across environments.

Bryan (2018) connected testing practices to broader workflow considerations, demonstrating how testing fits into version control and project organization to create robust, reproducible research. The integration of continuous integration services with containerized environments has further formalized testing in reproducible workflows (Ushey & Allaire, 2020).

1.7 Practical Implementations in Different Research Domains

The application of these reproducibility tools varies across domains:

In genomics, Kulkarni et al. (2018) demonstrated how Docker and package management tools enable complex bioinformatics pipelines to be shared and reused. Similarly, Grüning et al. (2018) showed how Bioconductor packages could be effectively versioned and containerized.

In social sciences, Healy (2018) advocated for reproducible workflows using R, emphasizing how containerization prevents the “works on my machine” problem when sharing analyses with colleagues.

For statistical education, Cetinkaya-Rundel and Rundel (2018) demonstrated how Docker containers provide consistent environments for teaching statistics with R, ensuring that all students have identical computational setups.

1.8 Adoption Challenges and Future Directions

Despite the technical solutions available, adoption remains challenging. Stodden et al. (2018) identified several barriers to adoption of reproducible practices, including time constraints, training limitations, and institutional factors. McNutt (2014) emphasized the need for journals and institutions to incentivize reproducibility practices through policy changes.

Looking forward, Brinckman et al. (2019) highlighted the need for better integration between reproducibility tools and research workflows, making reproducibility “built-in” rather than an additional step. The development of graphical interfaces and simplified workflows, such as those proposed by Perkel (2020), may help address adoption barriers.

Beaulieu-Jones and Greene (2017) suggested that the future of reproducible research will increasingly rely on automated systems that can validate reproducibility without manual intervention, an approach that has gained traction with the growth of continuous integration practices in research computing.

1.9 Conclusion

The literature reveals a clear progression in addressing reproducibility challenges in R, from early recognition of the problem to increasingly sophisticated technical solutions. The combination of package management (renv) and containerization (Docker) represents the current state-of-the-art for achieving comprehensive reproducibility.

Future research and development efforts should focus on reducing barriers to adoption, integrating reproducibility practices into mainstream research workflows, and developing automated validation systems. The ultimate goal remains making reproducibility the default state for computational research rather than an exceptional achievement.

2 Bibliography

Allaire, J., Xie, Y., McPherson, J., Luraschi, J., Ushey, K., Atkins, A., Wickham, H., Cheng, J., Chang, W., & Iannone, R. (2022). rmarkdown: Dynamic Documents for R. R package version 2.14. https://rmarkdown.rstudio.com

Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 452–454. https://doi.org/10.1038/533452a

Beaulieu-Jones, B. K., & Greene, C. S. (2017). Reproducibility of computational workflows is automated using continuous analysis. Nature Biotechnology, 35(4), 342–346. https://doi.org/10.1038/nbt.3780

Boettiger, C. (2015). An introduction to Docker for reproducible research. ACM SIGOPS Operating Systems Review, 49(1), 71–79. https://doi.org/10.1145/2723872.2723882

Boettiger, C., & Eddelbuettel, D. (2017). An Introduction to Rocker: Docker Containers for R. The R Journal, 9(2), 527–536. https://doi.org/10.32614/RJ-2017-065

Brinckman, A., Chard, K., Gaffney, N., Hategan, M., Jones, M. B., Kowalik, K., Kulasekaran, S., Ludäscher, B., Mecum, B. D., Nabrzyski, J., Stodden, V., Taylor, I. J., Turk, M. J., & Turner, K. (2019). Computing environments for reproducibility: Capturing the “Whole Tale”. Future Generation Computer Systems, 94, 854–867. https://doi.org/10.1016/j.future.2017.12.029

Bryan, J. (2018). Excuse me, do you have a moment to talk about version control? The American Statistician, 72(1), 20–27. https://doi.org/10.1080/00031305.2017.1399928

Cetinkaya-Rundel, M., & Rundel, C. (2018). Infrastructure and tools for teaching computing throughout the statistical curriculum. The American Statistician, 72(1), 58–65. https://doi.org/10.1080/00031305.2017.1397549

de Vries, A., & Meys, J. (2019). miniCRAN: Create a Mini Version of CRAN Containing Only Selected Packages. R package version 0.2.15. https://CRAN.R-project.org/package=miniCRAN

Eddelbuettel, D., & Boettiger, C. (2018). The Rocker Project: Promoting the Use of Docker in the R Community. In D. Cook, A. Unwin, & R. Smith (Eds.), Software Infrastructure for Statistical Computing.

Gruning, B., Chilton, J., Koster, J., Dale, R., Soranzo, N., van den Beek, M., Goecks, J., Backofen, R., Nekrutenko, A., & Taylor, J. (2018). Practical computational reproducibility in the life sciences. Cell Systems, 6(6), 631–635. https://doi.org/10.1016/j.cels.2018.03.014

Healy, K. (2018). The plain person’s guide to plain text social science. In Data Visualization for Social Science: A practical introduction with R and ggplot2. Princeton University Press.

Horst, A. M., Hill, A. P., & Gorman, K. B. (2022). palmerpenguins: Palmer Archipelago (Antarctica) penguin data. R Journal. https://doi.org/10.32614/RJ-2022-020

Kitzes, J., Turek, D., & Deniz, F. (Eds.). (2017). The Practice of Reproducible Research: Case Studies and Lessons from the Data-Intensive Sciences. University of California Press.

Kulkarni, N., Alessandrì, L., Panero, R., Arigoni, M., Olivero, M., Ferrero, G., Cordero, F., Beccuti, M., & Calogero, R. A. (2018). Reproducible bioinformatics project: a community for reproducible bioinformatics analysis pipelines. BMC Bioinformatics, 19(10), 349. https://doi.org/10.1186/s12859-018-2296-x

Marwick, B., Boettiger, C., & Mullen, L. (2018). Packaging data analytical work reproducibly using R (and friends). The American Statistician, 72(1), 80–88. https://doi.org/10.1080/00031305.2017.1375986

McNutt, M. (2014). Reproducibility. Science, 343(6168), 229. https://doi.org/10.1126/science.1250475

Microsoft Corporation. (2022). checkpoint: Install Packages from Snapshots on the Checkpoint Server for Reproducibility. R package version 1.0.1. https://CRAN.R-project.org/package=checkpoint

Nüst, D., Sochat, V., Marwick, B., Eglen, S. J., Head, T., Hirst, T., & Evans, B. D. (2020). Ten simple rules for writing Dockerfiles for reproducible data science. PLOS Computational Biology, 16(11), e1008316. https://doi.org/10.1371/journal.pcbi.1008316

Ooms, J. (2013). Possible directions for improving dependency versioning in R. R Journal, 5(1), 197–206. https://doi.org/10.32614/RJ-2013-019

Parker, H. (2017). Opinionated analysis development. PeerJ Preprints, 5, e3210v1. https://doi.org/10.7287/peerj.preprints.3210v1

Peikert, A., & Brandmaier, A. M. (2021). A reproducible data analysis workflow with R Markdown, Git, Make, and Docker. Quantitative and Computational Methods in Behavioral Sciences, 1, e3763. https://doi.org/10.5964/qcmb.3763

Peikert, A., van Lissa, C. J., & Brandmaier, A. M. (2021). Reproducible research in R: A tutorial on how to do the same thing more than once. Psych, 3(4), 836–867. https://doi.org/10.3390/psych3040053

Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226–1227. https://doi.org/10.1126/science.1213847

Perkel, J. M. (2020). Challenge to scientists: does your ten-year-old code still run? Nature, 584(7822), 656–658. https://doi.org/10.1038/d41586-020-02462-7

Rodríguez-Sánchez, F., Pérez-Luque, A. J., Bartomeus, I., & Varela, S. (2016). Ciencia reproducible: qué, por qué, cómo. Ecosistemas, 25(2), 83–92. https://doi.org/10.7818/ECOS.2016.25-2.11

Stodden, V., McNutt, M., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P., & Taufer, M. (2016). Enhancing reproducibility for computational methods. Science, 354(6317), 1240–1241. https://doi.org/10.1126/science.aah6168

Stodden, V., Seiler, J., & Ma, Z. (2018). An empirical analysis of journal policy effectiveness for computational reproducibility. Proceedings of the National Academy of Sciences, 115(11), 2584–2589. https://doi.org/10.1073/pnas.1708290115

The Rocker Project. (2023). Docker containers for the R environment. https://rocker-project.org/

Ushey, K. (2022). renv: Project Environments. R package version 0.15.5. https://CRAN.R-project.org/package=renv

Ushey, K., & Allaire, J. (2020). Continuous Integration for R Packages. RStudio Blog. https://blog.rstudio.com/2020/01/15/continuous-integration-for-r-packages/

Ushey, K., McPherson, J., Cheng, J., Atkins, A., & Allaire, J. (2018). packrat: A Dependency Management System for Projects and their R Package Dependencies. R package version 0.5.0. https://CRAN.R-project.org/package=packrat

Wickham, H. (2011). testthat: Get started with testing. The R Journal, 3(1), 5–10. https://doi.org/10.32614/RJ-2011-002

Wiebels, K., & Moreau, D. (2021). Leveraging containers for reproducible psychological research. Advances in Methods and Practices in Psychological Science, 4(3), 25152459211017853. https://doi.org/10.1177/25152459211017853

Xie, Y. (2021). knitr: A General-Purpose Package for Dynamic Report Generation in R. R package version 1.37. https://yihui.org/knitr/

Reuse

CC BY 4.0

Citation

BibTeX citation:

@online{(ryy)_glenn_thomas,
  author = {(Ryy) Glenn Thomas, Ronald},
  url = {https://focusonr.org/posts/share_R_code_via_docker/lit_review.html},
  langid = {en}
}

For attribution, please cite this work as:

(Ryy) Glenn Thomas, Ronald. n.d. https://focusonr.org/posts/share_R_code_via_docker/lit_review.html.

# Literature Review: Reproducibility in Computational Research Using R ## Introduction to the Reproducibility Crisis The scientific community has increasingly recognized the critical importance of reproducibility in research, leading to what has been termed the "reproducibility crisis" (Baker, 2016). This crisis affects multiple disciplines, but is particularly acute in computational research where software environments, package dependencies, and operating system differences can significantly impact results. Peng (2011) introduced a framework for reproducible research in computational science, arguing that standard scientific publication practices were insufficient for computational analyses. He proposed a spectrum of reproducibility, from "publication only" to "full replication," advocating for code and data sharing as minimal requirements. Building upon this foundation, Stodden et al. (2016) surveyed practices across scientific fields, finding that while most researchers valued reproducibility, implementation of reproducible workflows remained challenging due to technical barriers. ## Challenges in R-Based Reproducibility The R ecosystem presents specific challenges for reproducibility. As identified by Marwick et al. (2018), R analysis code frequently fails to execute correctly across different computing environments due to: 1. **Package version inconsistencies**: R packages evolve rapidly, with functions and arguments changing over time 2. **System dependencies**: Many R packages rely on external libraries and system-level requirements 3. **R version differences**: Core functionality can vary between R versions 4. **Operating system variations**: Platform-specific behaviors can affect computational results These challenges are compounded in collaborative research where team members may use different computing setups. Kitzes et al. (2017) presented case studies highlighting how seemingly minor environment differences could lead to significant variations in analytical outcomes. ## Package Management Solutions for R ### The renv Approach Package management has evolved as a solution to dependency challenges within R. CRAN itself offers limited built-in versioning, necessitating additional tools. The renv package, developed by Ushey (2022), has emerged as a robust solution for creating reproducible R environments. renv addresses reproducibility at the package level by: - Creating project-specific library paths - Recording exact package versions in a lockfile - Facilitating restoration of specific package environments - Supporting transparent version control of dependencies Prior to renv, similar functionality was offered by packrat (Ushey et al., 2018), which pioneered the isolation of project dependencies but had limitations in efficiency and usability that renv subsequently addressed. As observed by Peikert and Brandmaier (2021), renv represents an important advancement in making dependency management more accessible to R users. Ooms (2013) earlier highlighted the importance of addressing package dependencies, proposing semantic versioning practices for R packages. This work informed the development of dependency management tools like renv, illustrating the progression of the field. ### Alternative R Package Management Approaches While renv has gained prominence, other approaches exist. The checkpoint package (Microsoft Corporation, 2022) takes a temporal approach, installing packages as they existed on CRAN at a specific date. This "time machine" approach provides a different model of reproducibility, focusing on temporal consistency rather than project-specific dependency tracking. Another approach is represented by the miniCRAN package (de Vries & Meys, 2019), which allows creating local, customized CRAN-like repositories with specific package versions. This has proven especially useful in air-gapped or enterprise environments where internet access may be limited. ## Containerization as a Solution for System-Level Reproducibility ### Docker and Containerization While package management addresses R-specific dependencies, containerization addresses system-level reproducibility. Boettiger (2015) introduced Docker as a solution for reproducible research, highlighting its ability to create self-contained computing environments that include not just the R environment but also the underlying operating system, libraries, and configurations. Nüst et al. (2020) evaluated different containerization approaches for reproducible research, comparing Docker, Singularity, and other technologies. They found Docker to be particularly well-suited for reproducible R workflows due to its widespread adoption, ease of use, and comprehensive isolation properties. ### The Rocker Project The Rocker Project (Boettiger & Eddelbuettel, 2017) has been instrumental in making Docker accessible to the R community by providing standardized Docker images for R with different configurations. These pre-built images serve as reliable base environments for reproducible research, addressing a key barrier to adoption. Eddelbuettel and Boettiger (2018) further documented how the Rocker Project supports different use cases, from basic R installations to complex data science stacks including RStudio, Shiny, and common data science packages. This work has made containerization more accessible to R users without extensive systems administration knowledge. ## Integrating Package Management and Containerization The integration of package management (renv) with containerization (Docker) represents a comprehensive approach to reproducibility. Peikert et al. (2021) proposed such an integrated workflow, demonstrating how combining these tools provides both package-level and system-level reproducibility. Leveraging this combined approach, Rodríguez-Sánchez et al. (2016) developed workflows for ecological research that maintain reproducibility across collaborators and over time. Similarly, Wiebels and Moreau (2021) demonstrated the effectiveness of this integration for neuroimaging research, where computational environments are particularly complex. ## Testing and Validation in Reproducible Workflows Recent literature emphasizes the importance of integrating testing into reproducible workflows. Wickham (2011) introduced the testthat package, providing a framework for unit testing in R that has become a standard. Building on this, Parker (2017) argued that testing is an essential component of reproducibility, as it provides validation that code performs as expected across environments. Bryan (2018) connected testing practices to broader workflow considerations, demonstrating how testing fits into version control and project organization to create robust, reproducible research. The integration of continuous integration services with containerized environments has further formalized testing in reproducible workflows (Ushey & Allaire, 2020). ## Practical Implementations in Different Research Domains The application of these reproducibility tools varies across domains: In **genomics**, Kulkarni et al. (2018) demonstrated how Docker and package management tools enable complex bioinformatics pipelines to be shared and reused. Similarly, Grüning et al. (2018) showed how Bioconductor packages could be effectively versioned and containerized. In **social sciences**, Healy (2018) advocated for reproducible workflows using R, emphasizing how containerization prevents the "works on my machine" problem when sharing analyses with colleagues. For **statistical education**, Cetinkaya-Rundel and Rundel (2018) demonstrated how Docker containers provide consistent environments for teaching statistics with R, ensuring that all students have identical computational setups. ## Adoption Challenges and Future Directions Despite the technical solutions available, adoption remains challenging. Stodden et al. (2018) identified several barriers to adoption of reproducible practices, including time constraints, training limitations, and institutional factors. McNutt (2014) emphasized the need for journals and institutions to incentivize reproducibility practices through policy changes. Looking forward, Brinckman et al. (2019) highlighted the need for better integration between reproducibility tools and research workflows, making reproducibility "built-in" rather than an additional step. The development of graphical interfaces and simplified workflows, such as those proposed by Perkel (2020), may help address adoption barriers. Beaulieu-Jones and Greene (2017) suggested that the future of reproducible research will increasingly rely on automated systems that can validate reproducibility without manual intervention, an approach that has gained traction with the growth of continuous integration practices in research computing. ## Conclusion The literature reveals a clear progression in addressing reproducibility challenges in R, from early recognition of the problem to increasingly sophisticated technical solutions. The combination of package management (renv) and containerization (Docker) represents the current state-of-the-art for achieving comprehensive reproducibility. Future research and development efforts should focus on reducing barriers to adoption, integrating reproducibility practices into mainstream research workflows, and developing automated validation systems. The ultimate goal remains making reproducibility the default state for computational research rather than an exceptional achievement. # Bibliography Allaire, J., Xie, Y., McPherson, J., Luraschi, J., Ushey, K., Atkins, A., Wickham, H., Cheng, J., Chang, W., & Iannone, R. (2022). rmarkdown: Dynamic Documents for R. R package version 2.14. https://rmarkdown.rstudio.com Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 452–454. https://doi.org/10.1038/533452a Beaulieu-Jones, B. K., & Greene, C. S. (2017). Reproducibility of computational workflows is automated using continuous analysis. Nature Biotechnology, 35(4), 342–346. https://doi.org/10.1038/nbt.3780 Boettiger, C. (2015). An introduction to Docker for reproducible research. ACM SIGOPS Operating Systems Review, 49(1), 71–79. https://doi.org/10.1145/2723872.2723882 Boettiger, C., & Eddelbuettel, D. (2017). An Introduction to Rocker: Docker Containers for R. The R Journal, 9(2), 527–536. https://doi.org/10.32614/RJ-2017-065 Brinckman, A., Chard, K., Gaffney, N., Hategan, M., Jones, M. B., Kowalik, K., Kulasekaran, S., Ludäscher, B., Mecum, B. D., Nabrzyski, J., Stodden, V., Taylor, I. J., Turk, M. J., & Turner, K. (2019). Computing environments for reproducibility: Capturing the "Whole Tale". Future Generation Computer Systems, 94, 854–867. https://doi.org/10.1016/j.future.2017.12.029 Bryan, J. (2018). Excuse me, do you have a moment to talk about version control? The American Statistician, 72(1), 20–27. https://doi.org/10.1080/00031305.2017.1399928 Cetinkaya-Rundel, M., & Rundel, C. (2018). Infrastructure and tools for teaching computing throughout the statistical curriculum. The American Statistician, 72(1), 58–65. https://doi.org/10.1080/00031305.2017.1397549 de Vries, A., & Meys, J. (2019). miniCRAN: Create a Mini Version of CRAN Containing Only Selected Packages. R package version 0.2.15. https://CRAN.R-project.org/package=miniCRAN Eddelbuettel, D., & Boettiger, C. (2018). The Rocker Project: Promoting the Use of Docker in the R Community. In D. Cook, A. Unwin, & R. Smith (Eds.), Software Infrastructure for Statistical Computing. Gruning, B., Chilton, J., Koster, J., Dale, R., Soranzo, N., van den Beek, M., Goecks, J., Backofen, R., Nekrutenko, A., & Taylor, J. (2018). Practical computational reproducibility in the life sciences. Cell Systems, 6(6), 631–635. https://doi.org/10.1016/j.cels.2018.03.014 Healy, K. (2018). The plain person's guide to plain text social science. In Data Visualization for Social Science: A practical introduction with R and ggplot2. Princeton University Press. Horst, A. M., Hill, A. P., & Gorman, K. B. (2022). palmerpenguins: Palmer Archipelago (Antarctica) penguin data. R Journal. https://doi.org/10.32614/RJ-2022-020 Kitzes, J., Turek, D., & Deniz, F. (Eds.). (2017). The Practice of Reproducible Research: Case Studies and Lessons from the Data-Intensive Sciences. University of California Press. Kulkarni, N., Alessandrì, L., Panero, R., Arigoni, M., Olivero, M., Ferrero, G., Cordero, F., Beccuti, M., & Calogero, R. A. (2018). Reproducible bioinformatics project: a community for reproducible bioinformatics analysis pipelines. BMC Bioinformatics, 19(10), 349. https://doi.org/10.1186/s12859-018-2296-x Marwick, B., Boettiger, C., & Mullen, L. (2018). Packaging data analytical work reproducibly using R (and friends). The American Statistician, 72(1), 80–88. https://doi.org/10.1080/00031305.2017.1375986 McNutt, M. (2014). Reproducibility. Science, 343(6168), 229. https://doi.org/10.1126/science.1250475 Microsoft Corporation. (2022). checkpoint: Install Packages from Snapshots on the Checkpoint Server for Reproducibility. R package version 1.0.1. https://CRAN.R-project.org/package=checkpoint Nüst, D., Sochat, V., Marwick, B., Eglen, S. J., Head, T., Hirst, T., & Evans, B. D. (2020). Ten simple rules for writing Dockerfiles for reproducible data science. PLOS Computational Biology, 16(11), e1008316. https://doi.org/10.1371/journal.pcbi.1008316 Ooms, J. (2013). Possible directions for improving dependency versioning in R. R Journal, 5(1), 197–206. https://doi.org/10.32614/RJ-2013-019 Parker, H. (2017). Opinionated analysis development. PeerJ Preprints, 5, e3210v1. https://doi.org/10.7287/peerj.preprints.3210v1 Peikert, A., & Brandmaier, A. M. (2021). A reproducible data analysis workflow with R Markdown, Git, Make, and Docker. Quantitative and Computational Methods in Behavioral Sciences, 1, e3763. https://doi.org/10.5964/qcmb.3763 Peikert, A., van Lissa, C. J., & Brandmaier, A. M. (2021). Reproducible research in R: A tutorial on how to do the same thing more than once. Psych, 3(4), 836–867. https://doi.org/10.3390/psych3040053 Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226–1227. https://doi.org/10.1126/science.1213847 Perkel, J. M. (2020). Challenge to scientists: does your ten-year-old code still run? Nature, 584(7822), 656–658. https://doi.org/10.1038/d41586-020-02462-7 Rodríguez-Sánchez, F., Pérez-Luque, A. J., Bartomeus, I., & Varela, S. (2016). Ciencia reproducible: qué, por qué, cómo. Ecosistemas, 25(2), 83–92. https://doi.org/10.7818/ECOS.2016.25-2.11 Stodden, V., McNutt, M., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M. A., Ioannidis, J. P., & Taufer, M. (2016). Enhancing reproducibility for computational methods. Science, 354(6317), 1240–1241. https://doi.org/10.1126/science.aah6168 Stodden, V., Seiler, J., & Ma, Z. (2018). An empirical analysis of journal policy effectiveness for computational reproducibility. Proceedings of the National Academy of Sciences, 115(11), 2584–2589. https://doi.org/10.1073/pnas.1708290115 The Rocker Project. (2023). Docker containers for the R environment. https://rocker-project.org/ Ushey, K. (2022). renv: Project Environments. R package version 0.15.5. https://CRAN.R-project.org/package=renv Ushey, K., & Allaire, J. (2020). Continuous Integration for R Packages. RStudio Blog. https://blog.rstudio.com/2020/01/15/continuous-integration-for-r-packages/ Ushey, K., McPherson, J., Cheng, J., Atkins, A., & Allaire, J. (2018). packrat: A Dependency Management System for Projects and their R Package Dependencies. R package version 0.5.0. https://CRAN.R-project.org/package=packrat Wickham, H. (2011). testthat: Get started with testing. The R Journal, 3(1), 5–10. https://doi.org/10.32614/RJ-2011-002 Wiebels, K., & Moreau, D. (2021). Leveraging containers for reproducible psychological research. Advances in Methods and Practices in Psychological Science, 4(3), 25152459211017853. https://doi.org/10.1177/25152459211017853 Xie, Y. (2021). knitr: A General-Purpose Package for Dynamic Report Generation in R. R package version 1.37. https://yihui.org/knitr/