Skip to main content
1 of 4
galdre
  • 2.3k
  • 18
  • 32

Consume a Lazy Sequence in Parallel

I have a problem in which I am searching for particular combinations of 12 integers, where each integer is drawn from a range r. There are some really easy ways to prune the search space, and there are also some parts of the search that are more computationally intensive. My naive approach was to use (for [a r, b r, ..., k r, l r :when (and (stuff) (more-stuff))]) to perform the entire search.

But this executes the search in a single thread. I'd like to move the more computationally intensive part of the search to separate threads, so my slightly less naive approach was to put the easy pruning in the for statement, and to make a search function that did the harder work. Then: (filter search (for [..... :when (prune)])). There's a filter function in the reducers library, but that won't work on lazy seqs.

So, what is a good way to consume a lazy sequence in parallel? My last naive approach would be something like sticking the sequence in an atom:

(defn search-gen [lazys] (let [s (atom [nil lazys])] (fn [] (first (swap! s (fn [[_ s]] [(first s) (rest s)])))))) 

Then I could have a thread pool of six or so using this function to search the space.

Question: I have the uneasy feeling that I'm making this harder than it needs to be. Is there an easier method to consume a lazy sequence in parallel? Finally, is my entire approach fundamentally flawed? Is there a better way, perhaps one that doesn't require lazy sequences?

galdre
  • 2.3k
  • 18
  • 32