I have a C++ program which reads text files. Currently I'm using C's fopen() to read and then fgetc() to read the next character. I typedef'd a "file character", which is actually an int (and I can change it to long without problems, obviously).
Now the program can read UTF-7 and UTF-8 text files, but what if I use UTF-16 or UTF-32 text files? Is there a way to infer the file encoding and then read the file properly? Even passing to C++'s istream's wouldn't be a problem.
While you cannot definitely infer, in practice, you might try-and-fail based on a list of encodings.
Additionally, utf files are permitted (but not required) to store a byte order mark: https://en.wikipedia.org/wiki/Byte_order_mark . If you have it, you are lucky, as that's different amongst the encodings.
There's no way to figure it out reliably for an arbitrary byte stream. You can open the same way a binary executable file, which is not encoded in any of the mentioned encodings.
What you can do is try to guess. Treat the file as binary and read the the first 10k bytes or something like that, then compare the distribution of byte values to some canonical models you've built and see which one is the closest, and go with that one.
To build such a model you can take some texts (either stuff you already have or copy from wikipedia some articles) encode them with the various encodings and run the same algorithm to build the distribution. Average the results and use that as the canonical models for comparison. Works best when you tend to have the same kind of text (i.e. if you build the models with plain English text it might be difficult to classify documents using non-ascii characters).
If your files have a byte order mark, it helps a lot.
