r - nested %dopar% loop between lists

Question

I need to perform a nested %dopar% loop between two lists in R.

I have the loop working with a non-parallelised code, as follows:

main_lst = rep(list(list()), 10) # create main list where loop's results will be stored lst_1 = rep(list(list()), 25) # create list no. 1 for (i in 1:length(lst_1)) { lst_1[[i]] = data.frame(x = seq(1:30), y = rnorm(30)) } lst_2 = rep(list(list()), 10) # create list no. 2 for (i in 1:length(lst_2)) { lst_2[[i]] = data.frame(x = seq(16:30), z = rnorm(15)) } #### Do the for loop (non parallelised) for (h in 1:length(main_lst)) { for (i in 1:length(lst_1)) { main_lst[[h]][[i]] = merge(lst_1[[i]], lst_2[[h]][,c(1:2)], by = 'x') } }

Any suggestion on how I can parallelise the above for loop? Shall I try lapply (or parlapply) instead?

Here what I tried but it does not work:

### Run in Parallel library(foreach) library(doParallel) #setup parallel backend to use many processors cores=detectCores() cl = makeCluster(cores[1]-1) registerDoParallel(cl) main_lst = foreach(h=1:length(main_lst)) %:% { foreach(i=1:length(lst_1)) %dopar% { main_lst[[h]][[i]] = merge(lst_1[[i]], lst_2[[h]][,c(1:2)], by = 'x') } } #stop cluster stopCluster(cl)

Error in foreach(h = 1:main_lst) %:% { : "%:%" was passed an illegal right operand

Steve Lee · Accepted Answer · 2019-05-09 12:50:52Z

I edited your just a few (deleted { and } , change the h iteration limit and allocate main_lst)

main_lst = foreach(h=1:10) %:% foreach(i=1:length(lst_1)) %dopar% { merge(lst_1[[i]], lst_2[[h]][,c(1:2)], by = 'x') }

The result of foreach statement will be gathered by list automatically(Unless you set specific type like .combine = rbind.)

So you don't have to allocate it!!

thanks. but if I do h=1:length(main_lst) or h=1:10 it is the same :)

Collectives™ on Stack Overflow

r - nested %dopar% loop between lists

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related