0

I've got a big problem with encoding. The code I'm using should work but it doesn't!

Here is the code:

FileStream fs = new FileStream(saveFile, FileMode.Create, FileAccess.Write, FileShare.None); System.IO.StreamWriter objWriter; objWriter = new System.IO.StreamWriter(fs , Encoding.Unicode); string textLine; if (System.IO.File.Exists(readFile) == true) { System.IO.StreamReader objReader; objReader = new System.IO.StreamReader(readFile, Encoding.Unicode); do { textLine = objReader.ReadLine(); if (textLine.IndexOf(searchString) != -1) { tempString = textLine; position1 = textLine.IndexOf(searchString); tempString = textLine.Substring(position1); if (tempString.IndexOf("(") != -1) { position2 = tempString.IndexOf("("); //MessageBox.Show(tempString.Length.ToString()); tempString = tempString.Substring(0, position2); } } objWriter.WriteLine(textLine); } while (objReader.Peek() != -1); } objWriter.Close(); MessageBox.Show(tempString); MessageBox.Show("Done!");

I have to read a file that has mixed English characters and some Cyrillic characters, but after reading and processing the file, when I try to save the file to a new location all the cyrilic symbols are "?" or some other unknown symbol. I tried every possible encoding and it does not work!

Improve this question
8
  • msdn.microsoft.com/en-us/library/… 'The UnicodeEncoding object that is returned by this property may not have the appropriate behavior for your application. It uses replacement fallback to replace each string that it cannot encode and each byte that it cannot decode with a question mark ("?") character. ' Commented Oct 28, 2011 at 9:29
  • 1
    Are you sure that the input file you are reading is encoded with Unicode? Commented Oct 28, 2011 at 9:58
  • No i`m not sure but i have tested all others encodings as well and the result is the same. Commented Oct 28, 2011 at 10:07
  • Did you really try all 140 of them? Post a screenshot of a hex viewer that's displaying the file content. Commented Oct 28, 2011 at 10:16
  • 1
    The text is gibrish... this is it: asdaддасдасдасд фсд фд гфд гд фг фх гфх гй хйкй ;л ер вер в р ев дфг д фгд фг дгд фг Commented Oct 28, 2011 at 10:49

2 Answers 2

Reset to default
4

From the example you posted it seems that the file doesn't have a BOM and yet it contains cyrillic characters. Without a BOM the StreamReader cannot guess the correct encoding. So you could assume Windows-1251 encoding since the file contains cyrillic characters (according to the HEX dump you have shown in the comments section).

So here's what you may try:

using (var reader = new StreamReader("input.txt", Encoding.GetEncoding("Windows-1251"))) using (var writer = new StreamWriter("output.txt", false, Encoding.UTF8)) { string line; while ((line = reader.ReadLine()) != null) { // reading the input file line by line ... // perform the parsing and write to the UTF-8 output encoded file writer.WriteLine(line); } }
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

If you unsure of the encoding of your input file, don't specifiy it, let the StreamReader implementation inspect.

I suspect that your source file is not Unicode but rather using your local Windows Encoding.

Make a completely new file and don't specifiy any encoding in the reader.

objReader = new System.IO.StreamReader(readFile);
Improve this answer

3 Comments

I did that at the begining, but since it did not worked that is why i tryed with encoding and i did tested every possible standart encoding and every possible cyrylic encoding and the result is the same. And i have no idea what do.
Perhaps your source file has a BOM but the content is not UTF ?!? What encoding is the source file? If you make a completely new file with notepad and don't specify any encoding it should work.
FileStream never guesses at an encoding, it only works with bytes.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.