Is it possible to create a text log within a foreach() %dopar% call in R?

Question

I want to log certain events to a .txt file in my R script. Part of the script is parallelized. Is it possible to log events with cat() or similar inside a parallelized function?

This does not work.

 chunks <- list(1:3, 4:6) foreach(i = 1:cores) %:% foreach(x = chunks[[i]]) %dopar% { cat("Working on chunk, ", i, "number ", x, "...\n\n") }

This works, but it is system specific (not ideal)

foreach(i = 1:cores, .packages = "glue") %:% foreach(x = chunks[[i]]) %dopar% { system(glue("echo 'Working on chunk {i}, number {x}' >> output.txt")) }

Example output of the second (working) code block:

Working on chunk 1, number 1 Working on chunk 1, number 2 Working on chunk 1, number 3 Working on chunk 2, number 4 Working on chunk 2, number 6 Working on chunk 2, number 5

Does this answer your question? No standard output received inside foreach loop — F. Privé
– F. Privé, Commented Apr 17, 2020 at 20:06
@F.Privé this looks really great and is very closed, but it has unintended consequences. For example, when loading packages for each thread (.packages = c("package", "names") I get all the package loading text for each thread which is really excessive! Any tips? — Brigadeiro
– Brigadeiro, Commented Apr 17, 2020 at 23:17

Akshit · Accepted Answer · 2020-04-17 21:07:01Z

You can use doSNOW package in R. You can specify the file where you want the output logs too. Hope this works for you.

library(doSNOW) cl <- makeCluster(2, outfile = "abc.out") registerDoSNOW(cl) chunks <- list(1:3, 4:6) foreach(i = 1:2) %:% foreach(x = chunks[[i]]) %dopar% { cat("Working on chunk, ", i, "number ", x, "...\n\n") print("ongoing") # your statements } stopCluster(cl)

Do you really need the "outfile = "abc.out" part? can you just write cl <- makeCluster(2) instead?

Collectives™ on Stack Overflow

Is it possible to create a text log within a foreach() %dopar% call in R?

1 Answer 1

1 Comment

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Linked

Related