i have a string "Mikael Håfström" which contains some special characters how do i remove this using python?
3
- 1Is your string a unicode string? Do you want to remove the characters or rather replace by "standard" characters?Sven Marnach– Sven Marnach2011-03-10 10:54:19 +00:00Commented Mar 10, 2011 at 10:54
- 8Every character is special in its own way.Ignacio Vazquez-Abrams– Ignacio Vazquez-Abrams2011-03-10 10:59:50 +00:00Commented Mar 10, 2011 at 10:59
- 1Related: What is the best way to remove accents in a python unicode string?Sven Marnach– Sven Marnach2011-03-10 10:59:56 +00:00Commented Mar 10, 2011 at 10:59
Add a comment |
3 Answers
You can use the unicodedata module to normalize unicode strings and encode them in their ASCII form like so:
>>> import unicodedata >>> source = u'Mikael Håfström' >>> unicodedata.normalize('NFKD', source).encode('ascii', 'ignore') 'Mikael Hafstrom' One notable exception is that the letters 'đ' and 'Đ' are not recognized by Python and they do not get encoded to 'd', so they will simply be omitted from the result. That's a voiced alveolo-palatal affricate present in the latin alphabet of some SEE languages, so it may or may not immediately concern you based on your audience or whether or not your providing full support for the Latin-1 character set. I currently have Python 2.6.5 (Mar 19 2010) running locally and the issue is present, though I'm sure it may have been resolved with newer releases.
4 Comments
John Machin
I can't imagine why you call those two IPA letters a "notable" exception. BTW: s/alveolar-palatal/dental/. There are several letters and ligatures in Latin-1 which don't normalise to ASCII. Maybe you are confusing those two with Eth, used in Icelandic, Faroese, and Elfdalian. In any case the NFKD gimmick needs augmentation with a built-in list of exceptions. See my answer.
Filip Dupanović
For South Slavic languages, which I use, I did notice that there was no support for 'Dj', 'dj', 'Đ' or 'đ', hence the notable exception ;)! Truth be told, I don't have any experience with northern Germanic languages, and this is a rather specialized topic, so I'm a bit grizzled that there are 9 exceptions listed on effbot. Has any of this been corrected to date?
John Machin
Dupanovic: "issue"? "support"? "corrected"? You appear to mis-understand the purpose of the NFKD normalisation -- it's to decompose Unicode characters, if possible, not for the purpose of smashing them into ASCII. For some it's not possible; they don't decompose. No correction is required. All of the
unicodedata functions get their data directly from tables provided by unicode.org. There is no "issue".Filip Dupanović
I've finally noticed my slip where I said normalized, instead of encoded, that seems to have brought much confusion. Thanks John, your affable didactics were much appreciated!
See this effbot article (includes code). It makes reasonable transliterations into ASCII characters where possible. It is possible to extend the built-in conversion table to handle many other characters (e.g. those used in Eastern European languages) that don't have a canonical decomposition.
