Python Check for unicode files

Question

I'm very new to python and I have a mix of both ansi and unicode (utf-16-le) text based files in a series of directories. I've got some code which reads the text files okay until it hits a unicode file which at the mo, I've written in the code to skip. . I'm wondering if there's anyway I can get python to run a

with codecs.open

type of thing when it hits a unicode file as part of one prog? With my currrent level of python experience, the only way I could see of doing this is to write two separate progs; one to process the ANSI stuff and one for the Unicode.

Thanks in advance for any help you can provide

thkang · Accepted Answer · 2013-05-14 09:27:29Z

use unicode by default(which is a good programming discipline) and switch to ansi only if necessary.

import codecs def opener(filename): try: f = codecs.open(filename, encoding='utf-8') except UnicodeError: f = open(filename) return f

Jordan Jambazov · Accepted Answer · 2013-05-14 09:25:57Z

0

Just open all files using UTF-8.

f = codecs.open(file_name, "r", "utf-8")

answered May 14, 2013 at 9:25

Jordan Jambazov

3,6301 gold badge23 silver badges41 bronze badges

1 Comment

user2377057 Over a year ago

Thanks. I've tried that and I get a utf8 can't decode byte 0xac in position 26449:

Collectives™ on Stack Overflow

Python Check for unicode files

2 Answers 2

Comments

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Related