29213: Dataframe Constructor from List of List and non-iterables

[matttan90]

• closes BUG: pd.Dataframe.from_dict when first value is a list #29213
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff

---

What

Whilst the original issue is from the factory method DataFrame.from_dict(d, orient='index'), the main issue is the order of elements within the list in the main constructor:

For example:

pd.DataFrame([[1, 2, 3], 4]) # doesn't work # TypeError: object of type 'int' has no len() In [2]: pd.DataFrame([4, [1, 2, 3]]) # works, creates a 1D DataFrame Out[2]: 0 0 4 1 [1, 2, 3] 

Current Constructor Logic on List Argument

• Current logic looks at the first element, and infers that all other elements are iterables as well.
  • if all elements in the list are iterables, it generates a 2D DataFrame.
  • if any element from index-1 onwards is a non-iterable, it doesn't have a len() method and fails.

Proposed Solution

• Based on the first element, try to infer that all elements are iterables as well.
  • If not all subsequent elements are iterables, then return 1D DataFrame. This would be the same behaviour as it would have been if the first element in the list is non-iterable.
  • This should not have performance degradation as there is noneed to check if all elements in the list are iterables.

Note

• This does not solve the issue whereby we have iterables of different types (such as lists and strings...)

In [2]: pd.DataFrame([[1, 2, 3], 'foobar']) Out[2]: 0 1 2 3 4 5 0 1 2 3 None None None 1 f o o b a r 

[jreback]

where is the TypeError actualy being raised? we dont' want to have very wide try/excepts they tend to hide errors.

[matttan90]

Its actually raised here. At that point, it is already inferred that it is a list of lists (which is not always true).

Debugging trace:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../../core/frame.py:465: in __init__ arrays, columns = to_arrays(data, columns, dtype=dtype) ../../core/internals/construction.py:452: in to_arrays return _list_to_arrays(data, columns, coerce_float=coerce_float, dtype=dtype) ../../core/internals/construction.py:484: in _list_to_arrays content = list(lib.to_object_array(data).T) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ??? E TypeError: object of type 'int' has no len() 

[matttan90]

We can also try to catch that specific exception with something like this?

except TypeError as e: if not str(e).endswith("has no len()"): raise else: .... 

[jreback]

I don't want a try/except here at all, this just hides errors. you can do something more narrow in _list_to_arrays

[matttan90]

Okay thanks for the feedback! I'll have a look

[jreback]

I don't want a try/except here at all, this just hides errors. you can do something more narrow in _list_to_arrays

[jreback]

can you merge master and update

[WillAyd]

@matttan90 can you address comments?

[jreback]

can you merge master and will look again

[matttan90]

Sorry. My bad for disappearing. Will think again on how I can implement a fix on this..

[WillAyd]

Closing as I think stale, but @matttan90 ping if you want to pick back up and can merge master / address comments