Stripping non printable characters from a string in python

As far as I know, the most pythonic/efficient method would be:

import string filtered_string = filter(lambda x: x in string.printable, myStr) 

Note: string.printable - String of ASCII characters which are considered printable.

Here is example for unicode:

def remove_non_printable(value: str) -> str: return ''.join(i for i in value if i.isprintable()) 

You could try setting up a filter using the unicodedata.category() function:

import unicodedata printable = {'Lu', 'Ll'} def filter_non_printable(str): return ''.join(c for c in str if unicodedata.category(c) in printable) 

See Table 4-9 on page 175 in the Unicode database character properties for the available categories

The following will work with Unicode input and is rather fast...

import sys # build a table mapping all non-printable characters to None NOPRINT_TRANS_TABLE = { i: None for i in range(0, sys.maxunicode + 1) if not chr(i).isprintable() } def make_printable(s): """Replace non-printable characters in a string.""" # the translate method on str removes characters # that map to None from the string return s.translate(NOPRINT_TRANS_TABLE) assert make_printable('Café') == 'Café' assert make_printable('\x00\x11Hello') == 'Hello' assert make_printable('') == '' 

My own testing suggests this approach is faster than functions that iterate over the string and return a result using str.join.

In Python 3,

def filter_nonprintable(text): import itertools # Use characters of control category nonprintable = itertools.chain(range(0x00,0x20),range(0x7f,0xa0)) # Use translate to remove all non-printable characters return text.translate({character:None for character in nonprintable}) 

See this StackOverflow post on removing punctuation for how .translate() compares to regex & .replace()

The ranges can be generated via nonprintable = (ord(c) for c in (chr(i) for i in range(sys.maxunicode)) if unicodedata.category(c)=='Cc') using the Unicode character database categories as shown by @Ants Aasma.

This function uses list comprehensions and str.join, so it runs in linear time instead of O(n^2):

from curses.ascii import isprint def printable(input): return ''.join(char for char in input if isprint(char)) 

Yet another option in python 3:

re.sub(f'[^{re.escape(string.printable)}]', '', my_string) 

Based on @Ber's answer, I suggest removing only control characters as defined in the Unicode character database categories:

import unicodedata def filter_non_printable(s): return ''.join(c for c in s if not unicodedata.category(c).startswith('C')) 

An elegant pythonic solution to stripping 'non printable' characters from a string in python is to use the isprintable() string method together with a generator expression or list comprehension depending on the use case ie. size of the string:

 ''.join(c for c in my_string if c.isprintable()) 

str.isprintable() Return True if all characters in the string are printable or the string is empty, False otherwise. Nonprintable characters are those characters defined in the Unicode character database as “Other” or “Separator”, excepting the ASCII space (0x20) which is considered printable. (Note that printable characters in this context are those which should not be escaped when repr() is invoked on a string. It has no bearing on the handling of strings written to sys.stdout or sys.stderr.)

The best I've come up with now is (thanks to the python-izers above)

def filter_non_printable(str): return ''.join([c for c in str if ord(c) > 31 or ord(c) == 9]) 

This is the only way I've found out that works with Unicode characters/strings

Any better options?

In Python there's no POSIX regex classes

There are when using the regex library: https://pypi.org/project/regex/

It is well maintained and supports Unicode regex, Posix regex and many more. The usage (method signatures) is very similar to Python's re.

From the documentation:

[[:alpha:]]; [[:^alpha:]]

POSIX character classes are supported. These are normally treated as an alternative form of \p{...}.

(I'm not affiliated, just a user.)

The one below performs faster than the others above. Take a look

''.join([x if x in string.printable else '' for x in Str]) 

Adapted from answers by Ants Aasma and shawnrad:

nonprintable = set(map(chr, list(range(0,32)) + list(range(127,160)))) ord_dict = {ord(character):None for character in nonprintable} def filter_nonprintable(text): return text.translate(ord_dict) #use str = "this is my string" str = filter_nonprintable(str) print(str) 

tested on Python 3.7.7

To remove 'whitespace',

import re t = """ \n\t<p>&nbsp;</p>\n\t<p>&nbsp;</p>\n\t<p>&nbsp;</p>\n\t<p>&nbsp;</p>\n\t<p> """ pat = re.compile(r'[\t\n]') print(pat.sub("", t)) 

1. Error description Run the copied and pasted python code report:

Python invalid non-printable character U+00A0

2. The cause of the error The space in the copied code is not the same as the format in Python;

3. Solution Delete the space and re-enter the space. For example, the red part in the above picture is an abnormal space. Delete and re-enter the space to run;

Source : Python invalid non-printable character U+00A0

I used this:

import sys import unicodedata # the test string has embedded characters, \u2069 \u2068 test_string = """"ABC⁩.⁨ 6", "}""" nonprintable = list((ord(c) for c in (chr(i) for i in range(sys.maxunicode)) if unicodedata.category(c) in ['Cc','Cf'])) translate_dict = {character: None for character in nonprintable} print("Before translate, using repr()", repr(test_string)) print("After translate, using repr()", repr(test_string.translate(translate_dict)))