Looping multiple arrays

Question

I have an array containing multiple arrays (the number may vary) of significant dimensions (an array may contain up to 150 objects).

With the array, I need to find a combination (one element for each sub-array) that matches a condition.

Due to the dimensions, I tried to use Enumerator::Lazy as follows

catch :match do array[0].product(*array[1..-1]).lazy.each do |combination| throw :match if ConditionMatcher.match(combination) end end

However, I realize when I call each the enumerator is evaluated and it performs very slowly. I have tried to replace each with methods included in Enumerator::Lazy such as take_while

 array[0].product(*array[1..-1]).lazy.take_while do |combination| return false if ConditionMatcher.match(combination) end

But also, in this case, the product is evaluated with low performance.

For better performance, even id I don't really like it, I'm thinking to replace product with a nested each loop. Something like

catch :match do array[0].each do |first| array[1].each do |second| array[2].each do |third| throw :match if ConditionMatcher.match([first, second, third]) end end end end

Due to the fact that the number of sub-arrays changes from time to time. I'm not sure how to implement it.

Moreover, is there a better way to loop through all the sub-arrays without loading the entire set of combinations?

Update 1 -

Each sub-array contains an ActiveRecord::Relation on a polymorphic association. Therefore, each element of each combination responds to the same 2 methods (start_time and end_time) each returning an instance of Time.

The matcher checks if all the objects in the combination don't have overlapping times.

Can you give some example of the inputs and the expected arguments to the ConditionMatcher.match? — spickermann
– spickermann, Commented Feb 21, 2020 at 8:37
When the data is coming from the database did you consider implementing it in SQL? Usually, it is much faster to run a complex SQL query than to load all records into memory and iterate over all records in Ruby. — spickermann
– spickermann, Commented Feb 21, 2020 at 8:52
Yes, I did and ideally, I'd love to move all the logic to SQL. However, the relations are not a simple where but they take into consideration multiple factors (and multiple tables) and I have no idea neither where to start from in SQL. So, for now, I think to complete it in Ruby and then, with time, move the logic to SQL. — Sig
– Sig, Commented Feb 21, 2020 at 8:58

Stefan · Accepted Answer · 2020-02-21 09:58:01Z

The problem is that Array#product already returns a huge array containing all combinations. With 3 sub-arrays containing 150 items each, it returns a 150 × 150 × 150 = 3,375,000 element array. Calling lazy on that array won't speed up anything.

To make product calculate the Cartesian product lazily (i.e. one combination after the other), you simply have to (directly) pass a block to it:

first, *others = array first.product(*others) do |combination| # ... end

Collectives™ on Stack Overflow

Looping multiple arrays

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related