Skip to content

Openpyxl engine for reading excel files #25092

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 87 commits into from
Jun 28, 2019
Merged

Openpyxl engine for reading excel files #25092

Changes from 1 commit
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
e29b4c0
prepare testing reading excel files with multiple engines
tdamsma Feb 2, 2019
e0199a8
add openpyxl tests
tdamsma Feb 2, 2019
ce4eb01
implement first version of openpyxl reader
tdamsma Feb 2, 2019
b25877e
pep8 issues
tdamsma Feb 2, 2019
821fa4d
suppress openpyxl warnings
tdamsma Feb 2, 2019
4694668
add code for all edge cases that are tested for. Unfortunately got pr…
tdamsma Feb 7, 2019
712f1ef
formatting
tdamsma Feb 7, 2019
1d49a0e
Merge commit '683c7b55f5195fdf4f524239066cbf6f1301f0e7' into openpyxl…
tdamsma Feb 7, 2019
1473c0e
improve docstring
tdamsma Feb 7, 2019
6e8ffba
also test openpyxl reader for .xlsm files
tdamsma Feb 7, 2019
d57dfc1
explicitly use 64bit floats and ints
tdamsma Feb 7, 2019
e984f6b
Merge commit '6359bbc4c9ce6dd05bc8b422641cda74871cde43' into openpyxl…
tdamsma Feb 11, 2019
44f7af2
formatting
tdamsma Feb 11, 2019
98d3865
skip TestOpenpyxlReader when openpyxl is not installed
tdamsma Feb 11, 2019
d0188ba
Attempt to generalize _XlrdReader __init__ and move it to _BaseExcelR…
tdamsma Feb 12, 2019
205d52b
Merge commit 'f4568fd76e864d8aee3d23f5a81302262d6e0dcb' into openpyxl…
tdamsma Feb 20, 2019
7b550bf
register openpyxl writer engine, fix imports
tdamsma Feb 26, 2019
875de8d
import type_error explicitly
tdamsma Feb 26, 2019
12ad6d8
Merge branch 'master' into openpyxl-reader
tdamsma Mar 11, 2019
dfd6a36
Merge branch 'master' into openpyxl-reader
tdamsma Mar 19, 2019
fef7233
Merge branch 'master' into openpyxl-reader
tdamsma Apr 20, 2019
eaafd5f
get rid of some py2 compatibility legacy
tdamsma Apr 21, 2019
8d2db02
Merge branch 'master' into openpyxl-reader
tdamsma Apr 22, 2019
13e7793
fix some type chcking
tdamsma Apr 22, 2019
b053cce
linting
tdamsma Apr 22, 2019
fe4dd73
see if this works on linux
tdamsma Apr 22, 2019
64e5f2d
run isort on _openpyxl.py
tdamsma Apr 22, 2019
99b2cad
Merge remote-tracking branch 'upstream/master' into openpyxl-reader
tdamsma Apr 23, 2019
ce5ac05
Merge remote-tracking branch 'upstream/master' into openpyxl-reader
tdamsma Apr 23, 2019
c7895ea
Merge remote-tracking branch 'pandas/master' into openpyxl-reader
tdamsma Apr 27, 2019
2ca9368
refactor handling of sheet_name keyword
tdamsma Apr 27, 2019
5fb1aef
extract code to parse a single sheet to a method
tdamsma Apr 27, 2019
537dd0c
extract handling of header keywords
tdamsma Apr 27, 2019
44cddc5
extract handling of convert_float keyword to method
tdamsma Apr 27, 2019
e4c8f23
extract handling of index_col to method
tdamsma Apr 27, 2019
daff364
extract handling of usecols keyword to method
tdamsma Apr 27, 2019
1224918
remove redundant code
tdamsma Apr 27, 2019
1bfc030
Merge remote-tracking branch 'upstream/master' into excel-read-shared…
tdamsma Apr 28, 2019
747311e
Merge branch 'master' into excel-read-shared-init-to-baseclass
tdamsma Apr 28, 2019
a77a4c7
implement suggestions @WillAyd
tdamsma Apr 29, 2019
ddcaad8
Merge remote-tracking branch 'upstream/master' into excel-read-shared…
tdamsma Apr 29, 2019
757235d
Merge branch 'excel-read-shared-init-to-baseclass' into openpyxl-reader
tdamsma Apr 29, 2019
cdd627f
remove _engine keyword altogether
tdamsma Apr 29, 2019
0b58109
Merge branch 'excel-read-shared-init-to-baseclass' into openpyxl-reader
tdamsma Apr 29, 2019
45f21f8
Clean up __init__
tdamsma Apr 29, 2019
e97d029
Implement work around for Linux py35_compat import error
tdamsma Apr 29, 2019
1edae5e
fix regression for reading s3 files
tdamsma Apr 30, 2019
a69e104
Merge branch 'excel-read-shared-init-to-baseclass' into openpyxl-reader
tdamsma Apr 30, 2019
f5f40e4
expand code highlighting the weirdness of a failing/skipped test.
tdamsma Apr 30, 2019
22e24bb
remove _engine keyword altogether
tdamsma Apr 29, 2019
903b188
fix regression for reading s3 files
tdamsma Apr 30, 2019
1b3ae99
Merge branch 'excel-read-shared-init-to-baseclass' into openpyxl-reader
tdamsma Apr 30, 2019
02e19a8
Merge remote-tracking branch 'upstream/master' into openpyxl-reader
tdamsma Apr 30, 2019
3e18f97
Merge remote-tracking branch 'upstream/master' into openpyxl-reader
tdamsma Apr 30, 2019
d11956c
remove accidental commit
tdamsma May 1, 2019
61d7a3f
ditch some code
tdamsma May 1, 2019
13d41b2
Merge remote-tracking branch 'upstream/master' into openpyxl-reader
tdamsma Jun 10, 2019
97c85f5
remove skips for openpyxl for tests that should pass
tdamsma Jun 11, 2019
614d972
Add `by_blocks=True` to failing `assert_frame_equal` tests, as per @W…
tdamsma Jun 13, 2019
d87d9c0
Merge remote-tracking branch 'upstream/master' into openpyxl-reader
WillAyd Jun 27, 2019
7348b0c
Updated import machinery
WillAyd Jun 27, 2019
c1a1792
Cleaned up nan replacement
WillAyd Jun 27, 2019
d72ca5a
Simplified introspection
WillAyd Jun 27, 2019
0bba345
Used common renaming method
WillAyd Jun 27, 2019
8dd8bf6
Reverted some test changes
WillAyd Jun 27, 2019
eaaa680
Reset yield statement
WillAyd Jun 27, 2019
6bf5183
Better missing label handling
WillAyd Jun 27, 2019
a06bf9b
Aligned implementation with base
WillAyd Jun 27, 2019
f43e90f
Fix bool handling
WillAyd Jun 27, 2019
8fabe0a
Fixed 0 handling
WillAyd Jun 27, 2019
0ff5ce3
Aligned float handling with xlrd
WillAyd Jun 27, 2019
fb73692
xfailed overflow test
WillAyd Jun 27, 2019
17b1d73
lint and isort fixup
WillAyd Jun 27, 2019
3d248ed
Removed by_blocks
WillAyd Jun 27, 2019
c369fd8
Revert "Reverted some test changes"
tdamsma Jun 28, 2019
70b15a4
use readonly mode. Should be more performant and also this ignores Me…
tdamsma Jun 28, 2019
a3a3bca
formatting issues
tdamsma Jun 28, 2019
fcd43f0
handle datetime cells explicitly for openpyxl < 2.5.0 compatibility
tdamsma Jun 28, 2019
d9c1fa6
type fixup
WillAyd Jun 28, 2019
3c239a4
whatsnew
WillAyd Jun 28, 2019
4a25a5a
Removed np.nan from Scalar
WillAyd Jun 28, 2019
6258e59
revert test_reader changes again. Not needed anymore because of using…
tdamsma Jun 28, 2019
00f34b1
more types and whitespace cleanup
WillAyd Jun 28, 2019
a1fba90
Added config for excel reader. Not sure how to test this
tdamsma Jun 28, 2019
88ee325
whatsnew
WillAyd Jun 28, 2019
837ce26
Merge remote-tracking branch 'upstream/master' into openpyxl-reader
WillAyd Jun 28, 2019
dddc8c5
Regenerated test1 files
WillAyd Jun 28, 2019
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
formatting
  • Loading branch information
@tdamsma
tdamsma committed Feb 11, 2019
commit 44f7af28839d788ab4185cd02ef85fed1718998f
2 changes: 1 addition & 1 deletion pandas/tests/io/test_excel.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ def ignore_xlrd_time_clock_warning():
@contextlib.contextmanager
Copy link
Member

@WillAyd WillAyd May 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make this a fixture instead?
Copy link
Contributor Author

@tdamsma tdamsma May 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I followed the example of ignore_xlrd_time_clock_warning, shall I turn that into a fixture as well? Also, ignore_xlrd_time_clock_warning and ignore_openpyxl_unknown_extension_warning are always used together, shall I combine them?
Copy link
Member

@WillAyd WillAyd May 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd leave the xlrd one alone; that would make a nice follow up to consolidate
def ignore_openpyxl_unknown_extension_warning():
Copy link
Member

@WillAyd WillAyd Feb 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need this?
Copy link
Contributor Author

@tdamsma tdamsma Feb 12, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Openpyxl does not support all elements that are in xlsx sheets, and raises warnings for when these are found. See also https://bitbucket.org/openpyxl/openpyxl/issues/537/userwarning-unknown-extension-is-not and https://stackoverflow.com/questions/34322231/python-2-7-openpyxl-userwarning.

This code suppresses that warning as the tests are making assertions about which warnings should be raised.
Copy link
Member

@WillAyd WillAyd Feb 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need this? I think from the cleanup PR you had we should have done away with unknown extensions, no?
Copy link
Contributor Author

@tdamsma tdamsma Feb 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unknown extensions refer to excel internals like Conditional Formatting. That is and excel extension that openpyxl does not support. Perhaps I can implement a more generic warnings filter as a fixture
Copy link
Member

@WillAyd WillAyd May 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know which files were actually causing this? If conditional formatting is what is causing this I'm hesitant to blanket add this as I would think issues would be limited to a handful of files at most
"""
Context manager to ignore warnings raised by openpyxl library, regarding
Context manager to ignore warnings raised by openpyxl library, regarding
the use of unsupported extensions within the xlsx files.
"""

Expand Down