37

I have tried running the following code on a Unix machine with 20 CPU, using R foreach, parallel, doParallel, and party packages (my objective is to have the party / varimp function working on several CPUs in parallel):

parallel_compute_varimp <- function (object, mincriterion = 0, conditional = FALSE, threshold = 0.2, nperm = 1, OOB = TRUE, pre1.0_0 = conditional) { response <- object@responses input <- object@data@get("input") xnames <- colnames(input) inp <- initVariableFrame(input, trafo = NULL) y <- object@responses@variables[[1]] error <- function(x, oob) mean((levels(y)[sapply(x, which.max)] != y)[oob]) w <- object@initweights perror <- matrix(0, nrow = nperm * length(object@ensemble), ncol = length(xnames)) colnames(perror) <- xnames data = foreach(b = 1:length(object@ensemble), .packages = c("party","stats"), .combine = rbind) %dopar% { try({ tree <- object@ensemble[[b]] oob <- object@weights[[b]] == 0 p <- .Call("R_predict", tree, inp, mincriterion, -1L, PACKAGE = "party") eoob <- error(p, oob) for (j in unique(varIDs(tree))) { for (per in 1:nperm) { if (conditional || pre1.0_0) { tmp <- inp ccl <- create_cond_list(conditional, threshold, xnames[j], input) if (is.null(ccl)) { perm <- sample(which(oob)) } else { perm <- conditional_perm(ccl, xnames, input, tree, oob) } tmp@variables[[j]][which(oob)] <- tmp@variables[[j]][perm] p <- .Call("R_predict", tree, tmp, mincriterion, -1L, PACKAGE = "party") } else { p <- .Call("R_predict", tree, inp, mincriterion, as.integer(j), PACKAGE = "party") } perror[b, j] <- (error(p, oob) - eoob) } } ######## # return data to the %dopar% loop data variable perror[b, ] ######## }) # END OF TRY } # END OF LOOP WITH PARALLEL COMPUTING perror = data perror <- as.data.frame(perror) return(MeanDecreaseAccuracy = colMeans(perror)) } environment(parallel_compute_varimp) <- asNamespace('party') cl <- makeCluster(detectCores()) registerDoParallel(cl, cores = detectCores()) <...> system.time(data.cforest.varimp <- parallel_compute_varimp(data.cforest, conditional = TRUE))

but I am getting an error:

> system.time(data.cforest.varimp <- parallel_compute_varimp(data.cforest, conditional = TRUE)) Error in unserialize(socklist[[n]]) : error reading from connection Timing stopped at: 58.302 13.197 709.307

The code was working with a smaller dataset on 4 CPUs.

I am running out of ideas. Can someone suggest a way to reach my objective of running party package varimp function on parallel CPUs?

Improve this question
2
  • 2
    did you figure out this issue? Commented Sep 30, 2014 at 0:48
  • 1
    I don't think so. I forgot all about it anyway :) Commented Sep 30, 2014 at 7:55

1 Answer 1

Reset to default
48

The error:

Error in unserialize(socklist[[n]]) : error reading from connection

means that the master process got an error when calling unserialize to read from the socket connection to one of the workers. That probably means that the corresponding worker died, thus dropping its end of the socket connection. Unfortunately, it may have died for any number of reasons, many of which are very system specific.

You can usually figure out why the worker died by using the makeCluster "outfile" option so that the error message generated by the worker isn't thrown away. I usually recommend using outfile="" as described in this answer. Note that the "outfile" option works the same in both the snow and parallel packages.

You could also verify that your foreach loop works correctly when executed sequentially by registering the sequential backend:

registerDoSEQ()

If you're lucky, the foreach loop will fail when executed sequentially, since it's usually easier to figure out what is going wrong.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.