Iterableiterator

[toobaz]

If the approach is not overkill, it can be used also in UTF8Recoder and UnicodeReader from pandas/common/io.py. Closes #12153.

[jreback]

see #9496

[jreback]

move to pandas/io/common.py, inherit from object, call this BaseIterable, raise AbstractMethodError for __next__

[jreback]

make a parent class of Boto and UTFReader as well in`io/common.py``

[jreback]

you can also incorporate elements of TableIterator in io/pytables.py (and make this a sub-class)

[toobaz]

BaseIterable or BaseIterator? True, it makes an iterable out of an iterator, but most iterables will not be iterators.

[max-sixty]

inherit from object?

[toobaz]

TableIterator in io/pytables.py is still a bit too cryptic to me, but I can try to take a look at it next week.

[toobaz]

Each chunk has an index restarting from 0 (also in master). Isn't this a bug?

[jreback]

No, that shouldn't be the case

In [46]: s = DataFrame({'A' : range(10)}).to_csv(None) In [47]: s Out[47]: ',A\n0,0\n1,1\n2,2\n3,3\n4,4\n5,5\n6,6\n7,7\n8,8\n9,9\n' In [48]: pd.read_csv(StringIO(s),index_col=0) Out[48]: A 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 In [49]: list(pd.read_csv(StringIO(s),index_col=0,chunksize=4)) Out[49]: [ A 0 0 1 1 2 2 3 3, A 4 4 5 5 6 6 7 7, A 8 8 9 9] 

[toobaz]

I mean when the index is not loaded:

In [7]: list(pd.read_csv(StringIO(s), chunksize=4)) Out[7]: [ Unnamed: 0 A 0 0 0 1 1 1 2 2 2 3 3 3, Unnamed: 0 A 0 4 4 1 5 5 2 6 6 3 7 7, Unnamed: 0 A 0 8 8 1 9 9] 

Wouldn't we prefer a consistent index?

[jreback]

hmm yes I agree

you'd have to keep state in the in the iterators I think though -
I do this is HDFStore iirc

but might be a bit non trivial

if u think u can fix easily - go ahead
otherwise create an issue

[toobaz]

No, I'd like to do it but it probably won't be immediate: #12185 .

[jreback]

ok on doing HDFStore iterator in another issue.

[jreback]

AbstractMethodError(self)

[toobaz]

I removed edits to BotoFileLikeReader because as far as I can judge the failing tests were due to it (through its ancestor key.Key) already being an iterable, and knowing nothing about this format/service it's hard for me to debug.

[jreback]

can you add tests for read_stata & read_sas

[toobaz]

yep, done

[jreback]

thanks @toobaz

I have marked off csv,stat,sas, remaining are hdf & msgpack which are in #9496