27

I'm trying to pass big strings of random html through regular expressions and my Python 2.6 script is choking on this:

UnicodeEncodeError: 'ascii' codec can't encode character

I traced it back to a trademark superscript on the end of this word: Protection™ -- and I expect to encounter others like it in the future.

Is there a module to process non-ascii characters? or, what is the best way to handle/escape non-ascii stuff in python?

Thanks! Full error:

E ====================================================================== ERROR: test_untitled (__main__.Untitled) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Python26\Test2.py", line 26, in test_untitled ofile.write(Whois + '\n') UnicodeEncodeError: 'ascii' codec can't encode character u'\u2122' in position 1005: ordinal not in range(128)

Full Script:

from selenium import selenium import unittest, time, re, csv, logging class Untitled(unittest.TestCase): def setUp(self): self.verificationErrors = [] self.selenium = selenium("localhost", 4444, "*firefox", "http://www.BaseDomain.com/") self.selenium.start() self.selenium.set_timeout("90000") def test_untitled(self): sel = self.selenium spamReader = csv.reader(open('SubDomainList.csv', 'rb')) for row in spamReader: sel.open(row[0]) time.sleep(10) Test = sel.get_text("//html/body/div/table/tbody/tr/td/form/div/table/tbody/tr[7]/td") Test = Test.replace(",","") Test = Test.replace("\n", "") ofile = open('TestOut.csv', 'ab') ofile.write(Test + '\n') ofile.close() def tearDown(self): self.selenium.stop() self.assertEqual([], self.verificationErrors) if __name__ == "__main__": unittest.main()
Improve this question
3
  • Please post python version, and the traceback that is part of the exception. Commented Oct 31, 2009 at 0:09
  • 1
    Which version of Python are you working with? Python's Unicode support has evolved greatly in the last few versions. Commented Oct 31, 2009 at 0:10
  • Here is the version: Python 2.6 Thanks! Commented Oct 31, 2009 at 0:17

4 Answers 4

Reset to default
32

You're trying to convert unicode to ascii in "strict" mode:

>>> help(str.encode) Help on method_descriptor: encode(...) S.encode([encoding[,errors]]) -> object Encodes S using the codec registered for encoding. encoding defaults to the default encoding. errors may be given to set a different error handling scheme. Default is 'strict' meaning that encoding errors raise a UnicodeEncodeError. Other possible values are 'ignore', 'replace' and 'xmlcharrefreplace' as well as any other name registered with codecs.register_error that is able to handle UnicodeEncodeErrors.

You probably want something like one of the following: 

s = u'Protection™' print s.encode('ascii', 'ignore') # removes the ™ print s.encode('ascii', 'replace') # replaces with ? print s.encode('ascii','xmlcharrefreplace') # turn into xml entities print s.encode('ascii', 'strict') # throw UnicodeEncodeErrors
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

thanks for the effort -- I updated my question and will try to make it work with your info. -KBF1
22

You're trying to pass a bytestring to something, but it's impossible (from the scarcity of info you provide) to tell what you're trying to pass it to. You start with a Unicode string that cannot be encoded as ASCII (the default codec), so, you'll have to encode by some different codec (or transliterate it, as @R.Pate suggests) -- but it's impossible for use to say what codec you should use, because we don't know what you're passing the bytestring and therefore don't know what that unknown subsystem is going to be able to accept and process correctly in terms of codecs.

In such total darkness as you leave us in, utf-8 is a reasonable blind guess (since it's a codec that can represent any Unicode string exactly as a bytestring, and it's the standard codec for many purposes, such as XML) -- but it can't be any more than a blind guess, until and unless you're going to tell us more about what you're trying to pass that bytestring to, and for what purposes.

Passing thestring.encode('utf-8') rather than bare thestring will definitely avoid the particular error you're seeing right now, but it may result in peculiar displays (or whatever it is you're trying to do with that bytestring!) unless the recipient is ready, willing and able to accept utf-8 encoding (and how could WE know, having absolutely zero idea about what the recipient could possibly be?!-)

Improve this answer

3 Comments

updated info as per your notes and I will start looking into how to use utf-8 now -- thanks!
So, now we know your error comes while writing to a file - moving to utf-8 will surely fix THAT... but when is the file read back again and how is it processed then? We're still totally in the dark about the real purpose of your unicode -> bytestring conversion!-)
Full script provided = general advice is also welcome. thanks!
1

The "best" way always depends on your requirements; so, what are yours? Is ignoring non-ASCII appropriate? Should you replace ™ with "(tm)"? (Which looks fancy for this example, but quickly breaks down for other codepoints—but it may be just what you want.) Could the exception be exactly what you need; now you just need to handle it in some way?

Only you can really answer this question.

Improve this answer

Comments

0

First of all, try installing translations for English language (or any other if needed):

sudo apt-get install language-pack-en

which provides translation data updates for all supported packages (including Python).

And make sure you use the right encoding in your code.

For example:

open(foo, encoding='utf-8')

Then double check your system configuration like value of LANG or configuration of locale (/etc/default/locale) and don't forget to re-login your session.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.