Overview

Relevant source files

Purpose and Scope

This document introduces the mirai package at a high level, explaining its purpose as a minimalist async evaluation framework for R, its role in the R ecosystem, and key architectural concepts. For detailed information on specific topics, refer to:

Architecture and design principles: Architecture and Design Philosophy
Technology foundation: Technology Stack
Practical usage: Getting Started
Core functionality details: Core Functionality

Sources: DESCRIPTION1-24 README.md1-40 man/mirai-package.Rd1-18

What is mirai

mirai is a minimalist async evaluation framework for R that enables asynchronous parallel and distributed computing. The name "mirai" (ミライ, 未来) means "future" in Japanese, reflecting its core abstraction: R expressions that will be evaluated in the future.

The package evaluates R expressions asynchronously in parallel processes, either locally on the same machine or distributed across networks. It is built on modern networking and concurrency primitives provided by the nanonext package (R bindings) and the NNG (Nanomsg Next Generation) C library.

Core Design Principles:

Inverted Topology: Daemons dial into the host rather than host pushing to workers, enabling dynamic scaling without connection management overhead
Event-Driven Promises: Non-blocking asynchronous operations that resolve via condition variables, eliminating polling overhead
Explicit Dependencies: Clear evaluation model with no hidden state - all dependencies must be explicitly passed via .args or ... parameters
FIFO Scheduling: Optional dispatcher provides optimal load balancing through first-in-first-out task assignment

The inverted topology is central to mirai's scalability: daemons can connect and disconnect from anywhere (local, SSH, HPC clusters, cloud) without the host managing individual connections.

Sources: DESCRIPTION13-23 README.md20-42 README.Rmd28-56 README.md74-108

Key Components and Code Entities

Core Components:

Component	Function	File Location
Task Creation	`mirai()`	R/mirai.R
Parallel Mapping	`mirai_map()`	R/map.R
Worker Management	`daemons()`	R/daemons.R
Background Worker	`daemon()`	R/daemon.R
Task Dispatcher	`dispatcher()`	R/dispatcher.R
Global Setup	`everywhere()`	R/everywhere.R
Status Monitoring	`status()`, `info()`	R/status.R

Sources: README.md41-72 NEWS.md11-28

System Architecture

High-Level Architecture: Inverted Topology

Key Architectural Patterns:

Pattern	Implementation	File Reference
Inverted Topology	Daemons dial into host/dispatcher using `req` sockets	R/daemon.R105-150
Event-Driven Resolution	`mirai` objects resolve via condition variables (non-polling)	R/mirai.R50-120
FIFO Scheduling	Dispatcher assigns tasks to first available daemon	R/dispatcher.R80-200
Profile Registry	Compute profiles stored in `..` environment	R/daemons.R45-90
Socket Patterns	`rep`/`req` for host-dispatcher, `poly`/`req` for dispatcher-daemon	R/dispatcher.R50-80

With Dispatcher (daemons(n, dispatcher = TRUE)):

Host creates rep socket that listens at a URL
Dispatcher process creates poly socket that listens at a daemon URL
Daemons dial into dispatcher's poly socket with req sockets
Tasks are queued and assigned FIFO to available daemons
Results flow back through dispatcher to host

Without Dispatcher (daemons(n, dispatcher = FALSE)):

Host directly creates listening socket
Daemons dial into host with round-robin assignment
No queuing, direct connection between host and each daemon

Sources: R/daemons.R45-250 R/dispatcher.R1-200 R/daemon.R1-250 NEWS.md199-233

Communication Flow and NNG Socket Types

NNG Socket Patterns Used:

Pattern	Socket Type	Usage	Created By
Reply	`nng_rep0`	Host/dispatcher listens for requests	`nanonext::rep_socket()`
Request	`nng_req0`	Daemons send requests and receive replies	`nanonext::req_socket()`
Polyamorous	`nng_rep0_poly`	Dispatcher handles multiple daemon connections	`nanonext::rep_socket(poly = TRUE)`

The req/rep pattern provides automatic connection management and message routing. When a daemon dials into a listening socket, NNG automatically establishes the bidirectional channel.

Sources: R/mirai.R1-250 R/daemon.R105-200 man/mirai-package.Rd19-28

Communication and Transport

Platform-Specific Transports

Transport Layer Details:

Platform	URL Scheme	Example	Latency	Implementation
Linux	`abstract://`	`abstract://mirai-8a3f2c`	~1-10 μs	Abstract namespace Unix sockets (no filesystem)
MacOS/POSIX	`ipc://`	`ipc:///tmp/mirai-8a3f2c`	~1-10 μs	Unix domain sockets via `/tmp` directory
Windows	`ipc://`	`ipc:///./pipe/mirai-8a3f2c`	~10-100 μs	Named pipes via `\\.\pipe\` namespace
Remote TCP	`tcp://`	`tcp://192.168.1.100:5555`	~100-1000 μs	Standard TCP/IP over network
Secure TLS	`tls+tcp://`	`tls+tcp://192.168.1.100:5555`	~200-2000 μs	TLS 1.3 with auto-generated certificates
WebSocket	`ws://`	`ws://192.168.1.100:5555`	~100-1000 μs	WebSocket protocol (legacy support)

URL Construction Functions:

Function	Purpose	Returns	Example
`local_url()`	Generate random local IPC URL	Character string	`abstract://mirai-8a3f2c` (Linux)
`local_url(tcp = TRUE)`	Generate random local TCP URL	Character string	`tcp://127.0.0.1:33945`
`host_url()`	Get all network interface IPs	Named character vector	`c(eth0 = "tcp://192.168.1.100:0")`
`host_url(ws = TRUE)`	Get WebSocket URLs	Named character vector	`c(eth0 = "ws://192.168.1.100:0")`

The :0 port notation means NNG will auto-select an available port and report it back.

Sources: R/launchers.R1-250 man/mirai-package.Rd19-28 NEWS.md467-482 NEWS.md150-156

TLS Configuration and Security

Zero-Configuration TLS (automatic):

Custom TLS Certificates (manual):

The tls argument at daemons() and tlscert argument at daemon() accept either a length-2 character vector of file paths or a raw vector containing certificate data.

Sources: NEWS.md467-482 NEWS.md79-82 R/daemons.R100-150 R/daemon.R50-100

Role in the R Ecosystem

mirai serves as foundational infrastructure for asynchronous, parallel, and distributed computing in R:

Key Integrations:

R Base (parallel): Official alternative communications backend (R >= 4.4) via miraiCluster objects implementing the cluster API
Promises: Event-driven promises through as.promise() methods, enabling non-blocking reactive programming in Shiny and plumber2
Serialization: Custom serialization framework via serial_config() and register_serial() enables transfer of reference objects (torch tensors, Arrow tables, Polars DataFrames) across processes
Data Science: Powers parallel processing in purrr, tidymodels, and targets (via the crew package)
Observability: OpenTelemetry integration for distributed tracing when the otel package is available

Sources: README.md109-146 README.Rmd108-129 NEWS.md42-68

Core Capabilities Summary

Task Evaluation

mirai(): Create asynchronous tasks that evaluate R expressions in parallel processes
mirai_map(): Apply functions over collections in parallel with progress tracking and error handling
everywhere(): Broadcast expressions to all daemons for environment setup

Daemon Management

daemons(): Configure persistent daemon pools with dispatcher or non-dispatcher modes
daemon(): Individual daemon process for task execution
dispatcher(): Optional FIFO scheduler and connection manager

Deployment

Local: launch_local() spawns daemons on the same machine using IPC
SSH: ssh_config() and launch_remote() deploy via SSH with tunneling support
HPC: cluster_config() integrates with Slurm, SGE, Torque/PBS, LSF resource managers
Generic: remote_config() provides templating for custom deployment commands

Resource Isolation

Compute Profiles: Named daemon sets managed via .compute parameter, stored in .. environment
Scoping: with_daemons() for temporary profile switching, local_daemons() for frame-based scope

Error Handling

miraiError: Evaluation errors with preserved stack traces and condition metadata
miraiInterrupt: User interruptions
errorValue: Timeout (5), connection reset (19), cancellation (20)
Detection: is_mirai_error(), is_mirai_interrupt(), is_error_value()

Monitoring

status(): Detailed connection and task statistics matrix
info(): Succinct information for programmatic use
unresolved(): Check if mirai has completed

Performance

Scales to millions of tasks across thousands of connections
Zero-latency event-driven architecture (non-polling)
In-memory queuing with no file system storage requirement
1,000x efficiency improvement over traditional approaches

Sources: README.md25-108 DESCRIPTION13-23 NEWS.md1-233