26

I'm getting a 

UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in position 34: ordinal not in range(128)

on a string stored in 'a.desc' below as it contains the '£' character. It's stored in the underlying Google App Engine datastore as a unicode string so that's fine. The cStringIO.StringIO.writelines function is trying seemingly trying to encode it in ascii format:

result.writelines(['blahblah',a.desc,'blahblahblah'])

How do I instruct it to treat the encoding as unicode if that's the correct phrasing?

app engine runs on python 2.5

Improve this question

4 Answers 4

Reset to default
39

You can wrap the StringIO object in a codecs.StreamReaderWriter object to automatically encode and decode unicode.

Like this:

import cStringIO, codecs buffer = cStringIO.StringIO() codecinfo = codecs.lookup("utf8") wrapper = codecs.StreamReaderWriter(buffer, codecinfo.streamreader, codecinfo.streamwriter) wrapper.writelines([u"list of", u"unicode strings"])

buffer will be filled with utf-8 encoded bytes.

If I understand your case correctly, you will only need to write, so you could also do:

import cStringIO, codecs buffer = cStringIO.StringIO() wrapper = codecs.getwriter("utf8")(buffer)
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Also, the file-like object returned by cStringIO.StringIO() doesn't work in the with statement, but the wrapper returned by codecs.StreamReaderWriter() does!
This sounds similar to stackoverflow.com/q/45101658/562769 - do you know the answer to my question?
23

StringIO documentation:

Unlike the memory files implemented by the StringIO module, those provided by [cStringIO] are not able to accept Unicode strings that cannot be encoded as plain ASCII strings.

If possible, use StringIO instead of cStringIO.

Improve this answer

6 Comments

I switched (cStringIO is meant to be better performance-wise) and it didn't throw an error but did print '£' instead of just '£'. Why is 'Â' showing up now?
'£' is the Windows-1252 decoding of 0xc2 0xa3 which is the UTF-8 encoding of u'£'. Maybe your terminal, app, or wherever you're seeing that is configured for Windows-1252 instead of UTF-8.
hmm. Essentially I'm looking at a web server response through Chrome browser. Would that be the issue?
In Chrome you can set the encoding under which the page will be interpreted-- Page menu -> Encoding. Select "Unicode (UTF-8)" and see if that fixes it...
Nope. ISO-8859-1 will behave the same as Windows-1252 in that regard. You probably want to explicitly set the UTF-8 encoding in your page headers so that browsers don't have to guess the encoding. (Unless, of course, something else in your app is already generating output in a non-UTF-8 encoding.)
|
5

You can also encode your string as utf-8 manually before adding it to the StringIO

for val in rows: if isinstance(val, unicode): val = val.encode('utf-8') result.writelines(rows)
Improve this answer

1 Comment

use isinstance instead of type is X
0

Python 2.6 introduced the io module and you should consider using io.StringIO(), "An in-memory stream for unicode text."

In older python versions this is not optimized (pure Python), in later versions this has been optimized to (fast) C code.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.