I have a problem in which I am searching for numbers with certain properties in a very large search space (possibly infinite, but definitely too large for the whole space to fit in memory). So I want a lazy sequence, which I filter. My naive approach was to use list comprehension (for) for the entire search, but this executes the search in a single thread. There are some really easy ways to prune the search space, and there are also some parts of the search that are more computationally intensive.

My slightly less naive approach was to put the easy pruning in the for expression, and to make a search function that did the harder work. Then: (filter search (for [..... :when (prune)])). There's a filter function in the reducers library, but that won't work on lazy seqs. I can't convert from a lazy seq, because of the memory constraint.

So, what is a good way to filter a lazy sequence in parallel? My last naive approach would be something like sticking the sequence in an atom:

(defn accessor-gen [lazys] (let [s (atom [nil lazys])] (fn [] (first (swap! s (fn [[_ s]] [(first s) (rest s)])))))) 

Then I could have a thread pool of six or so using this function to search the space.

Question: I have the uneasy feeling that I'm making this harder than it needs to be. Also, I'm concerned about contention on the atom. Is there a more straightforward method to consume a lazy sequence in parallel? Finally, is my entire approach fundamentally flawed? Is there a better way, perhaps one that doesn't require lazy sequences?

I have a problem in which I am searching for numbers with certain properties in a very large search space (possibly infinite, but definitely too large for the whole space to fit in memory). So I want a lazy sequence, which I filter. My naive approach was to use list comprehension (for) for the entire search, but this executes the search in a single thread. There are some really easy ways to prune the search space, and there are also some parts of the search that are more computationally intensive.

My slightly less naive approach was to put the easy pruning in the for expression, and to make a search function that did the harder work. Then: (filter search (for [..... :when (prune)])). There's a filter function in the reducers library, but that won't work on lazy seqs. I can't convert from a lazy seq, because of the memory constraint.

So, what is a good way to filter a lazy sequence in parallel? My last naive approach would be something like sticking the sequence in an atom:

(defn accessor-gen [lazys] (let [s (atom [nil lazys])] (fn [] (first (swap! s (fn [[_ s]] [(first s) (rest s)])))))) 

Then I could have a thread pool of six or so using this function to search the space.

Question: I have the uneasy feeling that I'm making this harder than it needs to be. Also, I'm concerned about contention on the atom. Is there a more straightforward method to consume a lazy sequence in parallel? Finally, is my entire approach fundamentally flawed? Is there a better way, perhaps one that doesn't require lazy sequences?

I have a problem in which I am searching for particular combinations of 12 integers, where each integer is drawn from a range r. There are some really easy ways to prune the search space, and there are also some parts of the search that are more computationally intensive. My naive approach was to use (for [a r, b r, ..., k r, l r :when (and (stuff) (more-stuff))]) to perform the entire search.

But this executes the search in a single thread. I'd like to move the more computationally intensive part of the search to separate threads, so my slightly less naive approach was to put the easy pruning in the for statement, and to make a search function that did the harder work. Then: (filter search (for [..... :when (prune)])). There's a filter function in the reducers library, but that won't work on lazy seqs.

So, what is a good way to consume a lazy sequence in parallel? My last naive approach would be something like sticking the sequence in an atom:

(defn accessor-gen [lazys] (let [s (atom [nil lazys])] (fn [] (first (swap! s (fn [[_ s]] [(first s) (rest s)])))))) 

Then I could have a thread pool of six or so using this function to search the space.

Question: I have the uneasy feeling that I'm making this harder than it needs to be. Also, I'm concerned about contention on the atom. Is there a more straightforward method to consume a lazy sequence in parallel? Finally, is my entire approach fundamentally flawed? Is there a better way, perhaps one that doesn't require lazy sequences?

I have a problem in which I am searching for numbers with certain properties in a very large search space (possibly infinite, but definitely too large for the whole space to fit in memory). So I want a lazy sequence, which I filter. My naive approach was to use list comprehension (for) for the entire search, but this executes the search in a single thread. There are some really easy ways to prune the search space, and there are also some parts of the search that are more computationally intensive.

My slightly less naive approach was to put the easy pruning in the for expression, and to make a search function that did the harder work. Then: (filter search (for [..... :when (prune)])). There's a filter function in the reducers library, but that won't work on lazy seqs. I can't convert from a lazy seq, because of the memory constraint.

So, what is a good way to filter a lazy sequence in parallel? My last naive approach would be something like sticking the sequence in an atom:

(defn accessor-gen [lazys] (let [s (atom [nil lazys])] (fn [] (first (swap! s (fn [[_ s]] [(first s) (rest s)])))))) 

Then I could have a thread pool of six or so using this function to search the space.

Question: I have the uneasy feeling that I'm making this harder than it needs to be. Also, I'm concerned about contention on the atom. Is there a more straightforward method to consume a lazy sequence in parallel? Finally, is my entire approach fundamentally flawed? Is there a better way, perhaps one that doesn't require lazy sequences?

I have a problem in which I am searching for particular combinations of 12 integers, where each integer is drawn from a range r. There are some really easy ways to prune the search space, and there are also some parts of the search that are more computationally intensive. My naive approach was to use (for [a r, b r, ..., k r, l r :when (and (stuff) (more-stuff))]) to perform the entire search.

But this executes the search in a single thread. I'd like to move the more computationally intensive part of the search to separate threads, so my slightly less naive approach was to put the easy pruning in the for statement, and to make a search function that did the harder work. Then: (filter search (for [..... :when (prune)])). There's a filter function in the reducers library, but that won't work on lazy seqs.

So, what is a good way to consume a lazy sequence in parallel? My last naive approach would be something like sticking the sequence in an atom:

(defn search-gen [lazys] (let [s (atom [nil lazys])] (fn [] (first (swap! s (fn [[_ s]] [(first s) (rest s)])))))) 

Then I could have a thread pool of six or so using this function to search the space.

Question: I have the uneasy feeling that I'm making this harder than it needs to be. Is there an easier method to consume a lazy sequence in parallel? Finally, is my entire approach fundamentally flawed? Is there a better way, perhaps one that doesn't require lazy sequences?

I have a problem in which I am searching for particular combinations of 12 integers, where each integer is drawn from a range r. There are some really easy ways to prune the search space, and there are also some parts of the search that are more computationally intensive. My naive approach was to use (for [a r, b r, ..., k r, l r :when (and (stuff) (more-stuff))]) to perform the entire search.

But this executes the search in a single thread. I'd like to move the more computationally intensive part of the search to separate threads, so my slightly less naive approach was to put the easy pruning in the for statement, and to make a search function that did the harder work. Then: (filter search (for [..... :when (prune)])). There's a filter function in the reducers library, but that won't work on lazy seqs.

So, what is a good way to consume a lazy sequence in parallel? My last naive approach would be something like sticking the sequence in an atom:

(defn accessor-gen [lazys] (let [s (atom [nil lazys])] (fn [] (first (swap! s (fn [[_ s]] [(first s) (rest s)])))))) 

Then I could have a thread pool of six or so using this function to search the space.

Question: I have the uneasy feeling that I'm making this harder than it needs to be. Also, I'm concerned about contention on the atom. Is there a more straightforward method to consume a lazy sequence in parallel? Finally, is my entire approach fundamentally flawed? Is there a better way, perhaps one that doesn't require lazy sequences?