23

I want to remove all types of escape sequences from a list of strings. How can I do this? input:

['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray']

output:

['william', 'short', 'twitter', 'video', 'guy', 'ray']

http://docs.python.org/reference/lexical_analysis.html#string-literals

Improve this question
1
  • 2
    The final string object does not carry any information on wether the string literal that contrcuted it contained an escape sequence. If you can't even tell if there are any, how would you "remove" them? Commented Nov 13, 2011 at 22:30

6 Answers 6

Reset to default
41

If you want to strip out some characters you don't like, you can use the translate function to strip them out:

>>> s="\x01\x02\x10\x13\x20\x21hello world" >>> print(s) !hello world >>> s '\x01\x02\x10\x13 !hello world' >>> escapes = ''.join([chr(char) for char in range(1, 32)]) >>> t = s.translate(None, escapes) >>> t ' !hello world'

This will strip out all these control characters:

 001 1 01 SOH (start of heading) 002 2 02 STX (start of text) 003 3 03 ETX (end of text) 004 4 04 EOT (end of transmission) 005 5 05 ENQ (enquiry) 006 6 06 ACK (acknowledge) 007 7 07 BEL '\a' (bell) 010 8 08 BS '\b' (backspace) 011 9 09 HT '\t' (horizontal tab) 012 10 0A LF '\n' (new line) 013 11 0B VT '\v' (vertical tab) 014 12 0C FF '\f' (form feed) 015 13 0D CR '\r' (carriage ret) 016 14 0E SO (shift out) 017 15 0F SI (shift in) 020 16 10 DLE (data link escape) 021 17 11 DC1 (device control 1) 022 18 12 DC2 (device control 2) 023 19 13 DC3 (device control 3) 024 20 14 DC4 (device control 4) 025 21 15 NAK (negative ack.) 026 22 16 SYN (synchronous idle) 027 23 17 ETB (end of trans. blk) 030 24 18 CAN (cancel) 031 25 19 EM (end of medium) 032 26 1A SUB (substitute) 033 27 1B ESC (escape) 034 28 1C FS (file separator) 035 29 1D GS (group separator) 036 30 1E RS (record separator) 037 31 1F US (unit separator)

For Python newer than 3.1, the sequence is different:

>>> s="\x01\x02\x10\x13\x20\x21hello world" >>> print(s) !hello world >>> s '\x01\x02\x10\x13 !hello world' >>> escapes = ''.join([chr(char) for char in range(1, 32)]) >>> translator = str.maketrans('', '', escapes) >>> t = s.translate(translator) >>> t ' !hello world'
Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Sorry, that loop just makes me cringe. escapes = ''.join([chr(char) for char in range(1, 32)]) s.translate(None, escapes)
@AdamGriffiths, that's a nice change. Thanks.
I like this answer rather than the first one. it works well and very flexible.
This answer is a bit outdated. In Python 3.1 and above you would need to add these two lines after the declaration of escapes. translator = str.maketrans('', '', escapes) and then t = s.translate(translator)
@yvihs, thanks for the fix! I've edited my answer for the newer pythons.
19

Something like this?

>>> from ast import literal_eval >>> s = r'Hello,\nworld!' >>> print(literal_eval("'%s'" % s)) Hello, world!

Edit: ok, that's not what you want. What you want can't be done in general, because, as @Sven Marnach explained, strings don't actually contain escape sequences. Those are just notation in string literals.

You can filter all strings with non-ASCII characters from your list with

def is_ascii(s): try: s.decode('ascii') return True except UnicodeDecodeError: return False [s for s in ['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray'] if is_ascii(s)]
Improve this answer

Comments

4

You could filter out "words" that are not alphanumeric using a list comprehension and str.isalnum():

>>> l = ['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray'] >>> [word for word in l if word.isalnum()] ['william', 'short', 'twitter', 'video', 'guy', 'ray']

If you wish to filter out numbers, too, use str.isalpha() instead:

>>> l = ['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray', '456'] >>> [word for word in l if word.isalpha()] ['william', 'short', 'twitter', 'video', 'guy', 'ray']
Improve this answer

1 Comment

This is a good answer for many applications, although it should be noted that other non-alphanumerical characters like whitespaces will also suffer the axe.
4

The following worked for me in Python 3.6:

word = '\t\t\t\t\t\ntest\t msg' filter = ''.join([chr(i) for i in range(1, 32)]) word.translate(str.maketrans('', '', filter))

Output:

'test msg'

Source here

Improve this answer

1 Comment

Worked great for me!
2

This cannot be done, at least at the broad scope you are asking. As others have mentioned, runtime python doesn't know the difference between the something with escape sequences, and something without.

Example:

print ('\x61' == 'a')

prints True. So there's no way to find the difference between these two strings, unless you try some static analysis of your python script.

Improve this answer

Comments

0

I had similar issues while converting from hexadimal to String.This is what finally worked in python Example

list_l = ['william', 'short', '\x80', 'twitter', '\xaa', '\xe2', 'video', 'guy', 'ray'] decode_data=[] for l in list_l: data =l.decode('ascii', 'ignore') if data != "": decode_data.append(data) # output :[u'william', u'short', u'twitter', u'video', u'guy', u'ray']
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.