BUG: parser_source as a filename with multibyte characters in Windows(non utf-8 filesystem)

[hshimizu77]

fopen() in Windows doesn't accept utf-8 encoded filename with multibyte characters, so need to convert it to filesystem encoding.
Set 'utf-8' as default in case sys.getfilesystemencoding() return None.
sys.getfilesystemencoding() will return 'mbcs' in Windows, and will 'utf-8' or user setting in other systems.

[jreback]

thanks

can u write a test that fails without this change and works with it ?

[jreback]

@hshimizu77 tests?

[hshimizu77]

I have added a test but I'm wondering it's right place or not.

[jreback]

you can just do:

source = source.encode(sys.getfilesystemencoding() or 'utf-8')

[jreback]

that's the correct place

[jreback]

@TomAugspurger can you test this on OSX?

@hshimizu77 ok...let me test on windows; I think we may put this in w/o the test (but don't change the test ATM).

as I don't think pandas will install on some systems because of the encoding of the filename (in theory it should always work, but don't want to bork older systems).

[TomAugspurger]

The test errors

(pandas-dev) ~/E/p/l/p/s/p/p/i/tests (hshimizu77-bug-fix=) nosetests test_parsers:ParserTests.test_read_csv_example_in_windows_filesystem E ====================================================================== ERROR: test_parsers.ParserTests.test_read_csv_example_in_windows_filesystem ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/admin/Envs/pandas-dev/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/Users/admin/Envs/pandas-dev/lib/python2.7/site-packages/pandas/pandas/io/tests/test_parsers.py", line 1827, in test_read_csv_example_in_windows_filesystem self.read_csv(path) File "/Users/admin/Envs/pandas-dev/lib/python2.7/site-packages/pandas/pandas/io/tests/test_parsers.py", line 55, in read_csv raise NotImplementedError NotImplementedError ---------------------------------------------------------------------- Ran 1 test in 0.067s FAILED (errors=1) 

Seems to work from the interpreter though:

In [1]: from test_parsers import * In [2]: path = tm.get_data_path(u'日本語ファイル名テスト_read_csv_in_win_filesystem.csv') In [3]: pd.read_csv(path) Out[3]: A B C 0 100 200 300 1 aaa bbb ccc [2 rows x 3 columns]

[TomAugspurger]

I think you want read_csv not self.read_csv

[jreback]

@hshimizu77

for this

path = tm.get_data_path(u'日本語ファイル名テスト_read_csv_in_win_filesystem.csv') 

use u(.....) instead of the a literal u (this is translated by the compat module as needed for py2/3

[jreback]

merged via 79df67a

thanks!