How do you create a progress bar when using the "foreach()" function in R?

Edit: After an update to the doSNOW package it has become quite simple to display a nice progress bar when using %dopar% and it works on Linux, Windows and OS X

doSNOW now officially supports progress bars via the .options.snow argument.

library(doSNOW) cl <- makeCluster(2) registerDoSNOW(cl) iterations <- 100 pb <- txtProgressBar(max = iterations, style = 3) progress <- function(n) setTxtProgressBar(pb, n) opts <- list(progress = progress) result <- foreach(i = 1:iterations, .combine = rbind, .options.snow = opts) %dopar% { s <- summary(rnorm(1e6))[3] return(s) } close(pb) stopCluster(cl) 

Yet another way of tracking progress, if you keep in mind the total number of iterations, is to set .verbose = T as this will print to the console which iterations have been finished.

Previous solution for Linux and OS X

On Ubuntu 14.04 (64 bit) and OS X (El Capitan) the progress bar is displayed even when using %dopar% if in the makeCluster function oufile = "" is set. It does not seem to work under Windows. From the help on makeCluster:

outfile: Where to direct the stdout and stderr connection output from the workers. "" indicates no redirection (which may only be useful for workers on the local machine). Defaults to ‘/dev/null’ (‘nul:’ on Windows).

Example code:

library(foreach) library(doSNOW) cl <- makeCluster(4, outfile="") # number of cores. Notice 'outfile' registerDoSNOW(cl) iterations <- 100 pb <- txtProgressBar(min = 1, max = iterations, style = 3) result <- foreach(i = 1:iterations, .combine = rbind) %dopar% { s <- summary(rnorm(1e6))[3] setTxtProgressBar(pb, i) return(s) } close(pb) stopCluster(cl) 

This is what the progress bar looks like. It looks a little odd since a new bar is printed for every progression of the bar and because a worker may lag a bit which causes the progress bar to go back and forth occasionally.

You can also get this to work with the progress package.

# loading parallel and doSNOW package and creating cluster ---------------- library(parallel) library(doSNOW) numCores<-detectCores() cl <- makeCluster(numCores) registerDoSNOW(cl) # progress bar ------------------------------------------------------------ library(progress) iterations <- 100 # used for the foreach loop pb <- progress_bar$new( format = "letter = :letter [:bar] :elapsed | eta: :eta", total = iterations, # 100 width = 60) progress_letter <- rep(LETTERS[1:10], 10) # token reported in progress bar # allowing progress bar to be used in foreach ----------------------------- progress <- function(n){ pb$tick(tokens = list(letter = progress_letter[n])) } opts <- list(progress = progress) # foreach loop ------------------------------------------------------------ library(foreach) foreach(i = 1:iterations, .combine = rbind, .options.snow = opts) %dopar% { summary(rnorm(1e6))[3] } stopCluster(cl) 

This code is a modified version of the doRedis example, and will make a progress bar even when using %dopar% with a parallel backend:

#Load Libraries library(foreach) library(utils) library(iterators) library(doParallel) library(snow) #Choose number of iterations n <- 1000 #Progress combine function f <- function(){ pb <- txtProgressBar(min=1, max=n-1,style=3) count <- 0 function(...) { count <<- count + length(list(...)) - 1 setTxtProgressBar(pb,count) Sys.sleep(0.01) flush.console() c(...) } } #Start a cluster cl <- makeCluster(4, type='SOCK') registerDoParallel(cl) # Run the loop in parallel k <- foreach(i = icount(n), .final=sum, .combine=f()) %dopar% { log2(i) } head(k) #Stop the cluster stopCluster(cl) 

You have to know the number of iterations and the combination function ahead of time.

This is now possible with the parallel package. Tested with R 3.2.3 on OSX 10.11, running inside RStudio, using a "PSOCK"-type cluster.

library(doParallel) # default cluster type on my machine is "PSOCK", YMMV with other types cl <- parallel::makeCluster(4, outfile = "") registerDoParallel(cl) n <- 10000 pb <- txtProgressBar(0, n, style = 2) invisible(foreach(i = icount(n)) %dopar% { setTxtProgressBar(pb, i) }) stopCluster(cl) 

Strangely, it only displays correctly with style = 3.

You save the start time with Sys.time() before the loop. Loop over rows or columns or something which you know the total of. Then, inside the loop you can calculate the time ran so far (see difftime), percentage complete, speed and estimated time left. Each process can print those progress lines with the message function. You'll get an output something like

1/1000 complete @ 1 items/s, ETA: 00:00:45 2/1000 complete @ 1 items/s, ETA: 00:00:44 

Obviously the looping order will greatly affect how well this works. Don't know about foreach but with multicore's mclapply you'd get good results using mc.preschedule=FALSE, which means that items are allocated to processes one-by-one in order as previous items complete.

This code implements a progress bar tracking a parallelized foreach loop using the doMC backend, and using the excellent progress package in R. It assumes that all cores, specified by numCores, do an approximately equal amount of work.

library(foreach) library(doMC) library(progress) iterations <- 100 numCores <- 8 registerDoMC(cores=numCores) pbTracker <- function(pb,i,numCores) { if (i %% numCores == 0) { pb$tick() } } pb <- progress_bar$new( format <- " progress [:bar] :percent eta: :eta", total <- iterations / numCores, clear = FALSE, width= 60) output = foreach(i=1:iterations) %dopar% { pbTracker(pb,i,numCores) Sys.sleep(1/20) }