Confusion about python unicode

Question

I have writen a python file in windows7 by sublime text, there are some Chinese characters in the file, so when I run it, the characters become unrecognizable (the same occurs in cmd and git bash):

# -*- coding: utf-8 -*- str = "测试" print str arr = [] arr.append(str) print arr

the result is:

娴嬭瘯 ['\xe6\xb5\x8b\xe8\xaf\x95']

How can I solve this problem? what is the cause of this problem? and the arr print result shouldn't be unicode like \uXXX?

By the way, without the # -*- coding: utf-8 -*- I can't even run it:

$ python test.py File "test.py", line 2 SyntaxError: Non-ASCII character '\xe6' in file test.py on line 2, but no encodi ng declared; see http://www.python.org/peps/pep-0263.html for details

I just googled the statement， why can't the code run without it?

Since unicode handling is different in 3, I'll add Python 2.7 as a tag. — Codie CodeMonkey
– Codie CodeMonkey, Commented Aug 4, 2013 at 11:08
The lack of a \uXXX is because it isn't an unicode string, but a byte sequence. Prepend a u to "测试" for an unicode string. — Wessie
– Wessie, Commented Aug 4, 2013 at 11:13

Ashwini Chaudhary · Accepted Answer · 2013-08-04 11:19:28Z

# -*- coding: utf-8 -*- is needed to specify the encoding used in file.

You're getting ['\xe6\xb5\x8b\xe8\xaf\x95'] as output because your string is a byte string not a unicode string, add a u prefix to the string to convert it to a unicode string.

>>> strs = u"测试" >>> lis = [strs] >>> print lis [u'\u6d4b\u8bd5'] >>> print lis[0] 测试

Tim Pietzcker · Accepted Answer · 2013-08-04 11:18:06Z

You're seeing the UTF-8-encoded version of your string (which you shouldn't name str, by the way). By adding the # -*- coding: utf-8 -*- line at the start of your script, you're telling Python that that's the encoding your script is using. Are you sure that it is in fact using that encoding?

If that's not the case (check your editor!) or if your terminal window (where you're printing the string) happens to be using a different encoding, you'll get gibberish (or errors if the encoded string can't be interpreted in that encoding).

Only if you decode your (byte)string, you'll get a Unicode object.

So first you need to know your terminal's character encoding. Then you should be converting all strings to Unicode as soon as possible and manipulate only Unicode objects in your program until it's time to output them - at which point you need to encode them to the correct encoding.

For example

# -*- coding: utf-8 -*- s = u"测试" s = s + u"娴嬭瘯" print s.encode("somecodepage")

Collectives™ on Stack Overflow

Confusion about python unicode

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related