2

I know that bytes.decode gives a string and string.encode gives the bytes, but only if the correct encoding is used.

Suppose I have a bytes object encoded using gb18030
If I try to decode it using big5:

>>name = '深入 damon' >>b1 = name.encode('gb18030') >>> b1.decode('big5') UnicodeDecodeError: 'big5' codec can't decode byte 0xc8 in position 2: illegal multibyte sequence

Is there some way the encoding can be found from a bytes object?
I couldn't find any useful api in this regard in python3docs.

Improve this question
1
  • 1
    There isn't. If there was a general way of finding it, the decode function would not need its argument. The best you can get is educated guesses. The encoding is not part of the information contained in the bytes (unless you know are dealing with self-describing data, like HTML)), it is a property external to it. Commented May 31, 2013 at 10:47

2 Answers 2

Reset to default
7

You can use the chardet package. Read this tutorial.

If you are using Ubuntu:

sudo apt-get install python3-chardet

If you are using pip:

pip install chardet2
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

thanks @kev, I ran into the universaldetector import error ,but then your link to the dip3 case study helped..much obliged
4

Since you've entered it from the console, the encoding will be sys.stdin.encoding

>>> name = '深入 damon' >>> import sys >>> sys.stdin.encoding 'UTF-8' >>> b1 = name.decode(sys.stdin.encoding) >>> b1 u'\u6df1\u5165 damon' >>> b1.encode(sys.stdin.encoding) '\xe6\xb7\xb1\xe5\x85\xa5 damon' >>> print b1.encode(sys.stdin.encoding) 深入 damon
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.