Try the following code. Warning, it is only a proof of concept. When the text contains also characters written as non-escape sequences, the replacement must be done the more complicated way (I will show later when wanted). See the comments below.
import binascii s1 = '\\xd0\\xb1' print('s1 =', repr(s1), '=', list(s1)) # list() to emphasize what are the characters s2 = s1.replace('\\x', '') print('s2 =', repr(s2)) b = binascii.unhexlify(s2) print('b =', repr(b), '=', list(b)) s3 = b.decode('utf8') print('s3 =', ascii(s3)) with open('output.txt', 'w', encoding='utf-8') as f: f.write(s3)
It prints on concole:
c:\__Python\user\so20210201>py a.py s1 = '\\xd0\\xb1' = ['\\', 'x', 'd', '0', '\\', 'x', 'b', '1'] s2 = 'd0b1' b = b'\xd0\xb1' = [208, 177] s3 = '\u0431'
And it writes the character to the output.txt file.
The problem is that the question combines both unicode escaping and escaping binary values. In other words, the unicode string can contain some sequence that represents binary value somehow; however, you cannot force that binary value into the unicode string directly, because any unicode character is actually an abstract integer, and the integer can be represented in many ways (as a sequence of bytes).
If the unicode string contained escape sequences like \\n, it could be done differently, using the 'unicode_escape' prescription for bytes.decode(). However, in this case, you need both decoding from ascii escape sequences and then from utf-8.
Update: Here is a function for converting your kind of strings with other ascii characters (i.e. not only the escape sequences). The function use the finite automaton -- may look too complex at first (actually it is only verbose).
def userDecode(s): status = 0 lst = [] # result as list of bytes as ints xx = None # variable for one byte escape conversion for c in s: # unicode character print(status, ' c ==', c) ## just for debugging if status == 0: if c == '\\': status = 1 # escape sequence for a byte starts else: lst.append(ord(c)) # convert to integer elif status == 1: # x expected assert(c == 'x') status = 2 elif status == 2: # first nibble expected xx = c status = 3 elif status == 3: # second nibble expected xx += c lst.append(int(xx, 16)) # this is a hex representation of int status = 0 # Construct the bytes from the ordinal values in the list, and decode # it as UTF-8 string. return bytes(lst).decode('utf-8') if __name__ == '__main__': s = userDecode('\\xd0\\xb1whatever') print(ascii(s)) # cannot be displayed on console that does not support unicode with open('output.txt', 'w', encoding='utf-8') as f: f.write(s)
Look also inside the generated file. Remove the debug print. It displays the following on the console:
c:\__Python\user\so20210201>b.py 0 c == \ 1 c == x 2 c == d 3 c == 0 0 c == \ 1 c == x 2 c == b 3 c == 1 0 c == w 0 c == h 0 c == a 0 c == t 0 c == e 0 c == v 0 c == e 0 c == r '\u0431whatever'
