How to replace unicode characters in string with something else python?

Question

I have a string that I got from reading a HTML webpage with bullets that have a symbol like "•" because of the bulleted list. Note that the text is an HTML source from a webpage using Python 2.7's urllib2.read(webaddress).

I know the unicode character for the bullet character as U+2022, but how do I actually replace that unicode character with something else?

I tried doing str.replace("•", "something")

but it does not appear to work... how do I do this?

What is the type of the string, and which version of Python are you using? — Fred Foo
– Fred Foo, Commented Oct 26, 2012 at 20:14
I'm sorry, I'm not going to download a webpage using urllib2 now. What is the type? str or unicode? — Fred Foo
– Fred Foo, Commented Oct 26, 2012 at 20:16
Have you tried u.encode('ascii', 'replace') and then replacing '?' ? — r_31415
– r_31415, Commented Oct 26, 2012 at 20:16
if your python code contains utf-8 characters, you should use the 'magic comment' # coding=utf8 in the first or the second line of your code. — Kinjal Dixit
– Kinjal Dixit, Commented Oct 15, 2013 at 12:09

Fred Foo · Accepted Answer · 2012-10-26 20:23:18Z

85

Decode the string to Unicode. Assuming it's UTF-8-encoded:
```
str.decode("utf-8") 
```
Call the replace method and be sure to pass it a Unicode string as its first argument:
```
str.decode("utf-8").replace(u"\u2022", "*") 
```

Encode back to UTF-8, if needed:

str.decode("utf-8").replace(u"\u2022", "*").encode("utf-8")

(Fortunately, Python 3 puts a stop to this mess. Step 3 should really only be performed just prior to I/O. Also, mind you that calling a string str shadows the built-in type str.)

answered Oct 26, 2012 at 20:23

Fred Foo

365k80 gold badges765 silver badges852 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Kristada673 Over a year ago

I get this error: TypeError: Can't convert 'bytes' object to str implicitly

cryanbhu Over a year ago

could you please elaborate in what way "Python 3 puts a stop to this mess"? How would I do this in Python 3 then?

Dimitri Kopriwa Over a year ago

I get: .decode("utf-8") -> AttributeError: 'str' object has no attribute 'decode'. Did you mean: 'encode'?

shadowbq Over a year ago

in python3 drop the .decode("utf-8") and the .replace(u"\u2022", "*") should work. See question stackoverflow.com/q/15335052/1569557

RParadox · Accepted Answer · 2012-10-26 20:36:15Z

16

Encode string as unicode.

>>> special = u"\u2022" >>> abc = u'ABC•def' >>> abc.replace(special,'X') u'ABCXdef'

edited Oct 26, 2012 at 20:36

answered Oct 26, 2012 at 20:24

RParadox

6,9414 gold badges26 silver badges35 bronze badges

2 Comments

Rolando Over a year ago

what is "special"? I get a nameError: name 'special' is not defined.

igauravsehrawat Over a year ago

@Rolando Notice 'u' has been prefixed in string, that makes it unicode string.

Rahul Kumar Gupta · Accepted Answer · 2021-09-10 01:14:55Z

Try this one.

you will get the output in a normal string

str.encode().decode('unicode-escape')

and after that, you can perform any replacement.

str.replace('•','something')

This proves useful when the \u sequence is actually present as is in the source string.

David · Accepted Answer · 2012-10-26 20:29:46Z

3

import re regex = re.compile("u'2022'",re.UNICODE) newstring = re.sub(regex, something, yourstring, <optional flags>)

edited Oct 26, 2012 at 20:29

answered Oct 26, 2012 at 20:16

David

6,5912 gold badges27 silver badges22 bronze badges

5 Comments

Rolando Over a year ago

It is not an asterisk, it is a bullet (circle shape)

Rolando Over a year ago

When trying: re.sub(u'2022', varcontainingstring, ''), it makes the string empty with nothing in it.

David Over a year ago

@Damascusi fixed - try it now

Teodor Anton Over a year ago

@NullUserException Why is it a bad idea to use a regex to replace fixed strings?

NullUserException Over a year ago

@AntonTeodor Regex is less efficient than a simple string search and replace. It will work though

Ayushman Verma · Accepted Answer · 2022-06-20 23:23:43Z

str1 = "This is Python\u500cPool"

Encode the string to ASCII and replace all the utf-8 characters with '?'.

str1 = str1.encode("ascii", "replace")

Decode the byte stream to string.

str1 = str1.decode(encoding="utf-8", errors="ignore")

Replace the question mark with the desired character.

str1 = str1.replace("?"," ")

Sahil Mahajan Mj · Accepted Answer · 2018-11-19 12:15:51Z

Funny the answer is hidden in among the answers.

str.replace("•", "something")

would work if you use the right semantics.

str.replace(u"\u2022","something")

works wonders ;) , thnx to RParadox for the hint.

Khánh Pluto · Accepted Answer · 2021-10-01 07:40:28Z

If you want to remove all \u character. Code below for you

def replace_unicode_character(self, content: str): content = content.encode('utf-8') if "\\x80" in str(content): count_unicode = 0 i = 0 while i < len(content): if "\\x" in str(content[i:i + 1]): if count_unicode % 3 == 0: content = content[:i] + b'\x80\x80\x80' + content[i + 3:] i += 2 count_unicode += 1 i += 1 content = content.replace(b'\x80\x80\x80', b'') return content.decode('utf-8')

Collectives™ on Stack Overflow

How to replace unicode characters in string with something else python?

7 Answers 7

4 Comments

2 Comments

1 Comment

5 Comments

Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

4 Comments

2 Comments

1 Comment

5 Comments

Comments

Comments

Comments

Linked

Related