wostream fails to output wstring

Question

I am using Visual Studio C++ 2008 (Express). When I run the below code, the wostream (both std::wcout, and std::wfstream) stops outputting at the first non-ASCII character (in this case Chinese) encountered. Plain ASCII characters print fine. However, in the debugger, I can see that the wstrings are in fact properly populated with Chinese characters, and the output << ... is in fact getting executed.

The project settings in the Visual Studio solution are set to "Use Unicode Character Set". Why is std::wostream failing to output Unicode characters outside of the ASCII range?

void PrintTable(const std::vector<std::vector<std::wstring>> &table, std::wostream& output) { for (unsigned int i=0; i < table.size(); ++i) { for (unsigned int j=0; j < table[i].size(); ++j) { output << table[i][j] << L"\t"; } //output << std::endl; } } void TestUnicodeSingleTableChinesePronouns() { FileProcessor p("SingleTableChinesePronouns.docx"); FileProcessor::iterator fileIterator; std::wofstream myFile("data.bin", std::ios::out | std::ios::binary); for(fileIterator = p.begin(); fileIterator != p.end(); ++fileIterator) { PrintTable(*fileIterator, myFile); PrintTable(*fileIterator, std::wcout); std::cout<<std::endl<<"---------------------------------------"<<std::endl; } myFile.flush(); myFile.close(); }

related questions: stackoverflow.com/questions/2492077/… stackoverflow.com/questions/388490/… — Nathan
– Nathan, Commented Jul 25, 2012 at 4:27

bames53 · Accepted Answer · 2015-09-28 21:13:50Z

By default the locale that std::wcout and std::wofstream use for certain operations is the "C" locale, which is not required to support non-ascii characters (or any character outside C++'s basic character set). Change the locale to one that supports the characters you want to use.

The simplest thing to do on Windows is unfortunately to use legacy codepages, however you really should avoid that. Legacy codepages are bad news. Instead you should use Unicode, whether UTF-8, UTF-16, or whatever. Also you'll have to work around Windows' unfortunate console model that makes writing to the console very different from writing to other kinds of output streams. You might need to find or write your own output buffer that specifically handles the console (or maybe file a bug asking Microsoft to fix it).

Here's an example of console output:

#include <Windows.h> #include <streambuf> #include <iostream> class Console_streambuf : public std::basic_streambuf<wchar_t> { HANDLE m_out; public: Console_streambuf(HANDLE out) : m_out(out) {} virtual int_type overflow(int_type c = traits_type::eof()) { wchar_t wc = c; DWORD numberOfCharsWritten; BOOL res = WriteConsoleW(m_out, &wc, 1, &numberOfCharsWritten, NULL); (void)res; return 1; } }; int main() { Console_streambuf out(GetStdHandle(STD_OUTPUT_HANDLE)); auto old_buf = std::wcout.rdbuf(&out); std::wcout << L"привет, 猫咪!\n"; std::wcout.rdbuf(old_buf); // replace old buffer so that destruction can happen correctly. FIXME: use RAII to do this in an exception safe manner. }

You can do UTF-8 output to a file like this (although I'm not sure VS2008 supports codecvt_utf8_utf16):

#include <codecvt> #include <fstream> int main() { std::wofstream myFile("data.bin", std::ios::out | std::ios::binary); myFile.imbue(std::locale(myFile.getloc(),new std::codecvt_utf8_utf16<wchar_t>)); myFile << L"привет, 猫咪!"; }

fyi, VS2008 does not support codecvt_utf8_utf16. However, this was enough to get me going, and I would like to move to a newer version anyway.
Won't new std::codecvt_utf8_utf16<wchar_t>() cause memory leak?
@Zotyi The locale library's design does not follow the usual rules and requires code that would be bad practice in other contexts.
@bames53 , so this means the newly allocated std::codecvt_utf8_utf16 will be deleted by the std lib?

bames53 · Accepted Answer · 2012-07-23 18:00:24Z

0

Include the following header file

#include <locale>

at the start of main, add the following line.

std::locale::global(std::locale("chinese"));

This helps to set the proper locale.

edited Jul 23, 2012 at 18:00

bames53

88.7k15 gold badges191 silver badges255 bronze badges

answered Jul 23, 2012 at 14:19

PermanentGuest

5,3713 gold badges29 silver badges39 bronze badges

2 Comments

Nathan Over a year ago

I specifically intend to avoid locales, because this program needs to handle multiple languages simultaneously. Hence the attempt to use unicode.

Nathan Over a year ago

And, in fact, using the suggested advice only changes it so that the console prints a ? for the first couple non-ascii characters, after which it again prints nothing.

Collectives™ on Stack Overflow

wostream fails to output wstring

2 Answers 2

5 Comments

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

2 Comments

Linked

Related