Using an iterator as an argument to a function multiple times from one vector

Question

I'm trying to write some Rust code to decode GPS data from an SDR receiver. I'm reading samples in from a file and converting the binary data to a series of complex numbers, which is a time-consuming process. However, there are times when I want to stream samples in without keeping them in memory (e.g. one very large file processed only one way or samples directly from the receiver) and other times when I want to keep the whole data set in memory (e.g. one small file processed in multiple different ways) to avoid repeating the work of parsing the binary file.

Therefore, I want to write functions or structs with iterators to be as general as possible, but I know they aren't sized, so I need to put them in a Box. I would have expected something like this to work.

This is the simplest example I could come up with to demonstrate the same basic problem.

fn sum_squares_plus(iter: Box<Iterator<Item = usize>>, x: usize) -> usize { let mut ans: usize = 0; for i in iter { ans += i * i; } ans + x } fn main() { // Pretend this is an expensive operation that I don't want to repeat five times let small_data: Vec<usize> = (0..10).collect(); for x in 0..5 { // Want to iterate over immutable references to the elements of small_data let iterbox: Box<Iterator<Item = usize>> = Box::new(small_data.iter()); println!("{}: {}", x, sum_squares_plus(iterbox, x)); } // 0..100 is more than 0..10 and I'm only using it once, // so I want to 'stream' it instead of storing it all in memory let x = 55; println!("{}: {}", x, sum_squares_plus(Box::new(0..100), x)); }

I've tried several different variants of this, but none seem to work. In this particular case, I'm getting

error[E0271]: type mismatch resolving `<std::slice::Iter<'_, usize> as std::iter::Iterator>::Item == usize` --> src/main.rs:15:52 | 15 | let iterbox: Box<Iterator<Item = usize>> = Box::new(small_data.iter()); | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected reference, found usize | = note: expected type `&usize` found type `usize` = note: required for the cast to the object type `dyn std::iter::Iterator<Item = usize>`

I'm not worried about concurrency and I'd be happy to just get it working sequentially on a single thread, but a concurrent solution would be a nice bonus.

Why are you passing a Box<T> and not just &T in your functions? Do you want multiple references to your iterators? If you want to pass a flow of data between threads / tasks, did you consider std::collections::VecDeque which can be fed from one end and drained from another? I'm just trying to understand your design considerations. — 9000
– 9000, Commented Jun 5, 2019 at 20:05
I'm using a Box<T> because it implements Sized and I want to be able to use it in structs. In the first case, I want to have one master copy of the data in a Vec and create iterators over immutable references to the elements. I don't want to use VecDeque because I don't want to drain it on the receiving end. I also don't want to just pass immutable references to the Vec because I also want to be able to stream samples into the consumer without storing them in memory. — John Stanford
– John Stanford, Commented Jun 5, 2019 at 20:22

Shepmaster · Accepted Answer · 2019-06-05 23:50:22Z

The current error you're running into is here:

let iterbox:Box<Iterator<Item = usize>> = Box::new(small_data.iter());

You're declaring that you want an iterator that returns usize items, but small_data.iter() is an iterator that returns references to usize items (&usize). That why you get the error "expected reference, found usize". usize is a small type that's cloneable so you can simply use the .cloned() iterator adapter to provide an iterator that actually returns a usize.

let iterbox: Box<Iterator<Item = usize>> = Box::new(small_data.iter().cloned());

Once you're past that hurdle, the next problem is that the iterator returned over small_data contains a reference to the small_data. Since sum_squares_plus is defined to accept a Box<Iterator<Item = usize>>, it's implied in that signature that the Iterator trait object within the box has a 'static lifetime. The iterator you're providing does not because it borrows small_data. To fix that you need to adjust the sum_squares_plus definition to

fn sum_squares_plus<'a>(iter: Box<Iterator<Item = usize> + 'a>, x: usize) -> usize

Note the 'a lifetime annotations. The code should then compile, but unless there's some constraints other than what's clearly defined here, a more idiomatic and efficient approach would be to avoid using trait objects and the associated allocations. The below code should work using static dispatch without any trait objects.

fn sum_squares_plus<I: Iterator<Item = usize>>(iter: I, x: usize) -> usize { let mut ans: usize = 0; for i in iter { ans += i * i; } ans + x } fn main() { // Pretend this is an expensive operation that I don't want to repeat five times let small_data: Vec<usize> = (0..10).collect(); for x in 0..5 { println!("{}: {}", x, sum_squares_plus(small_data.iter().cloned(), x)); } // 0..100 is more than 0..10 and I'm only using it once, // so I want to 'stream' it instead of storing it all in memory let x = 55; println!("{}: {}", x, sum_squares_plus(Box::new(0..100), x)); }

Collectives™ on Stack Overflow

Using an iterator as an argument to a function multiple times from one vector

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related