UnicodeDecodeError during encode?

I think you're confusing Unicode strings and Unicode encodings (like UTF-8).

os.walk(".") returns the filenames (and directory names etc.) as strings that are encoded in the current codepage. It will silently remove characters that are not present in your current codepage (see this question for a striking example).

Therefore, if your file/directory names contain characters outside of your encoding's range, then you definitely need to use a Unicode string to specify the starting directory, for example by calling os.walk(u"."). Then you don't need to (and shouldn't) call unicode() on the results any longer, because they already are Unicode strings.

If you don't do this, you first need to decode the filenames (as in mystring.decode("cp850")) which will give you a Unicode string:

>>> "\xab".decode("cp850") u'\xbd' 

Then you can encode that into UTF-8 or any other encoding.

>>> _.encode("utf-8") '\xc2\xbd' 

If you're still confused why unicode("\xab") throws a decoding error, maybe the following explanation helps:

"\xab" is an encoded string. Python has no way of knowing which encoding that is, but before you can convert it to Unicode, it needs to be decoded first. Without any specification from you, unicode() assumes that it is encoded in ASCII, and when it tries to decode it under this assumption, it fails because \xab isn't part of ASCII. So either you need to find out which encoding is being used by your filesystem and call unicode("\xab", encoding="cp850") or whatever, or start with Unicode strings in the first place.

As I understand it your issue is that os.walk(unicode_path) fails to decode some filenames to Unicode. This problem is fixed in Python 3.1+ (see PEP 383: Non-decodable Bytes in System Character Interfaces):

File names, environment variables, and command line arguments are defined as being character data in POSIX; the C APIs however allow passing arbitrary bytes - whether these conform to a certain encoding or not. This PEP proposes a means of dealing with such irregularities by embedding the bytes in character strings in such a way that allows recreation of the original byte string.

Windows provides Unicode API to access filesystem so there shouldn't be this problem.

Python 2.7 (utf-8 filesystem on Linux):

>>> import os >>> list(os.walk(".")) [('.', [], ['\xc3('])] >>> list(os.walk(u".")) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/os.py", line 284, in walk if isdir(join(top, name)): File "/usr/lib/python2.7/posixpath.py", line 71, in join path += '/' + b UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: \ ordinal not in range(128) 

Python 3.3:

>>> import os >>> list(os.walk(b'.')) [(b'.', [], [b'\xc3('])] >>> list(os.walk(u'.')) [('.', [], ['\udcc3('])] 

Your str2uni() function tries (it introduces ambiguous names) to solve the same issue as "surrogateescape" error handler on Python 3. Use bytestrings for filenames on Python 2 if you are expecting filenames that can't be decoded using sys.getfilesystemencoding().

'\xab' 

Is a byte, number 171.

u'\xab' 

Is a character, U+00AB Left-pointing double angle quotation mark («).

u'\xab' is a short-hand way of saying u'\u00ab'. It's not the same (not even the same datatype) as the byte '\xab'; it would probably have been clearer to always use the \u syntax in Unicode string literals IMO, but it's too late to fix that now.

To go from bytes to characters is known as a decode operation. To go from characters to bytes is known as an encode operation. For either direction, you need to know which encoding is used to map between the two.

>>> unicode('\xab') UnicodeDecodeError 

unicode is a character string, so there is an implicit decode operation when you pass bytes to the unicode() constructor. If you don't tell it which encoding you want you get the default encoding which is often ascii. ASCII doesn't have a meaning for byte 171 so you get an error.

>>> unicode(u'\xab') u'\xab' 

Since u'\xab' (or u'\u00ab') is already a character string, there is no implicit conversion in passing it to the unicode() constructor - you get an unchanged copy.

res = u'' for ch in val: res += unichr(ord(ch)) return res 

The encoding that maps each input byte to the Unicode character with the same ordinal value is ISO-8859-1. Consequently you could replace this loop with just:

return unicode(val, 'iso-8859-1') 

(However note that if Windows is in the mix, then the encoding you want is probably not that one but the somewhat-similar windows-1252.)

One really horrible hack that occurs is to write our own str2uni() method

This isn't generally a good idea. UnicodeErrors are Python telling you you've misunderstood something about string types; ignoring that error instead of fixing it at source means you're more likely to hide subtle failures that will bite you later.

filename = unicode(fname) 

So this would be better replaced with: filename = unicode(fname, 'iso-8859-1') if you know your filesystem is using ISO-8859-1 filenames. If your system locales are set up correctly then it should be possible to find out the encoding your filesystem is using, and go straight to that:

filename = unicode(fname, sys.getfilesystemencoding()) 

Though actually if it is set up correctly, you can skip all the encode/decode fuss by asking Python to treat filesystem paths as native Unicode instead of byte strings. You do that by passing a Unicode character string into the os filename interfaces:

for _,_,files in os.walk(u'/path/to/folder'): # note u'' string for fname in files: filename = fname # nothing more to do! 

PS. The character in 3″ Floppy should really be U+2033 Double Prime, but there is no encoding for that in ISO-8859-1. Better in the long term to use UTF-8 filesystem encoding so you can include any character.