mirai

みらい

( 未来 )

Minimalist Async Evaluation Framework for R

Lightweight parallel code execution and distributed computing.

mirai() returns a ‘mirai’ object immediately. Designed for simplicity, a ‘mirai’ evaluates an R expression asynchronously, on local or network resources, resolving automatically upon completion.

State of the art networking and concurrency via nanonext offers reliable and efficient scheduling over fast inter-process communications or TCP/IP secured by TLS.

mirai パッケージを試してみたところ、かなり速くて驚きました

Installation

Install the latest release from CRAN:

install.packages("mirai")

Or the development version from R-universe:

install.packages("mirai", repos = "https://shikokuchuo.r-universe.dev")

Quick Start

Use mirai() to evaluate an expression asynchronously in a separate, clean R process.

A ‘mirai’ object is returned immediately.

library(mirai)

m <- mirai(
  {
    res <- rnorm(x) + y ^ 2
    res / rev(res)
  },
  x = 10,
  y = runif(1)
)

m
#> < mirai | $data >

Above, all specified name = value pairs are passed through to the ‘mirai’.

The ‘mirai’ yields an ‘unresolved’ logical NA whilst the async operation is ongoing.

m$data
#> 'unresolved' logi NA

To check whether a mirai has resolved:

unresolved(m)
#> [1] FALSE

Upon completion, the ‘mirai’ resolves automatically to the evaluated result.

m$data
#>  [1] -0.5767602  1.6767432  2.5189824  1.8092596  5.9579094  0.1678441
#>  [7]  0.5527123  0.3969857  0.5963942 -1.7338229

Alternatively, explicitly call and wait for the result using call_mirai().

call_mirai(m)$data
#>  [1] -0.5767602  1.6767432  2.5189824  1.8092596  5.9579094  0.1678441
#>  [7]  0.5527123  0.3969857  0.5963942 -1.7338229

Daemons

Daemons are persistent background processes created to receive ‘mirai’ requests.

They may be deployed for:

Local parallel processing; or

Remote network distributed computing.

Launchers allow daemons to be started both on the local machine and across the network via SSH etc.

Secure TLS connections can be automatically-configured on-the-fly for remote daemon connections.

Refer to the {mirai} vignette for full package functionality. This may be accessed within R by:

vignette("mirai", package = "mirai")

Integrations

The following core integrations are documented, with usage examples in the linked vignettes:

  Provides an alternative communications backend for R, implementing a low-level feature request by R-Core at R Project Sprint 2023. ‘miraiCluster’ may also be used with foreach, which is supported via doParallel.

  Implements the next generation of completely event-driven, non-polling promises. ‘mirai’ may be used interchageably with ‘promises’, including with the promise pipe %...>%.

  Asynchronous parallel / distributed backend, supporting the next level of responsiveness and scalability for Shiny. Launches ExtendedTasks, or plugs directly into the reactive framework for advanced uses.

  Asynchronous parallel / distributed backend, capable of scaling Plumber applications in production usage.

  Allows queries using the Apache Arrow format to be handled seamlessly over ADBC database connections hosted in daemon processes.

  Allows Torch tensors and complex objects such as models and optimizers to be used seamlessly across parallel processes.

Powering Crew and Targets High Performance Computing

  Targets, a Make-like pipeline tool for statistics and data science, has integrated and adopted crew as its default high-performance computing backend.

  Crew is a distributed worker-launcher extending mirai to different distributed computing platforms, from traditional clusters to cloud services.

  crew.cluster enables mirai-based workflows on traditional high-performance computing clusters using LFS, PBS/TORQUE, SGE and SLURM.

  crew.aws.batch extends mirai to cloud computing using AWS Batch.

Thanks

We would like to thank in particular:

Will Landau for being instrumental in shaping development of the package, from initiating the original request for persistent daemons, through to orchestrating robustness testing for the high performance computing requirements of crew and targets.

Joe Cheng for optimising the promises method to make mirai work seamlessly within Shiny, and prototyping non-polling promises, which is implemented across nanonext and mirai.

Luke Tierney of R Core, for discussion on L’Ecuyer-CMRG streams to ensure statistical independence in parallel processing, and making it possible for mirai to be the first ‘alternative communications backend for R’.

Henrik Bengtsson for valuable insights leading to the interface accepting broader usage patterns.

Daniel Falbel for discussion around an efficient solution to serialization and transmission of torch tensors.

Kirill Müller for discussion on using ‘daemons’ to host Arrow database connections.

  for funding work on the TLS implementation in nanonext, used to provide secure connections in mirai.

Links

◈ mirai R package: https://shikokuchuo.net/mirai/

mirai is listed in CRAN Task View:
- High Performance Computing: https://cran.r-project.org/view=HighPerformanceComputing

◈ nanonext R package: https://shikokuchuo.net/nanonext/

NNG website: https://nng.nanomsg.org/

–

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.