2

I'd like to convert a ASCII char* to wchar_t* in C++ on Linux without using mbstowcs(). On iOS and Windows, this works perfectly. On Android, however, mbstowcs seems to convert things quite literally, one-to-one. Even using different variations of setlocale(), I've been unable to successfully convert.

I might end up with just manually converting it on Android by copying 1 byte, and filling the rest with zeroes. But is this proper for ASCII? Are the first 255 characters of UTF-32/Unicode the same as the ASCII (ISO 8859-1/ISO Latin-1) character set?

Improve this question
5
  • 2
    ASCII is only 7 bit, not 8 bit, so it doesn't define 255 characters. Commented Aug 7, 2013 at 15:46
  • 2
    Are the first 255 characters of UTF-32/Unicode the same as the ASCII character set? No, since ASCII only defines the first 128 characters. But they are the same as the first 128 Unicode characters. Commented Aug 7, 2013 at 15:46
  • My understanding was that ISO 8859-1/ISO Latin-1 covers the ASCII characters from 128 to 255. So, I guess I'm specifically asking, are all the characters of ISO 8859-1/ISO Latin-1 map directly to the first 255 characters of UTF-32/Unicode? Commented Aug 7, 2013 at 15:53
  • 2
    Wikipedia: "The first 256 code points were made identical to the content of ISO-8859-1". Commented Aug 7, 2013 at 15:56
  • boost::nowide. Header only, copy and enjoy. It is recommended by utf8everywhere.org Commented Aug 7, 2013 at 18:54

3 Answers 3

Reset to default
3

To make thinks a bit clearer :

  • ASCII is a character encoding using values from 0..127 to encode a single character.
  • Latin-1 is another character set, that extends ASCII by using the values from 128..255 to encode its own characters.

Indeed most architecture byte is 8 bits, so there are still 128 values available when storing ASCII characters in byte. Several different character set were thus designed to extend ASCII for values from 128..255. Happy accident, the one referred as Latin-1 was used for the first 256 code points in Unicode (as pointed by BoBTFish). So if you have on one hand string of chars that you know is encoded using Latin-1, you can just assign each value to a wchar_t (which will ensure a correct "zero filling" with regard to endianness on your architecture), and it will be a valid wstring of unicode code points corresponding to the same characters. Then, the consumer of your wstring has to interpret its content as unicode code points.

Also, as soon as you cannot guarantee the encoding of the original string is Latin-1, you will run into problems. (eg, UTF-8 encoding is not mapping byte-per-byte to Latin-1).

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

If you don't mind taking an STL dependency and using string and wstring instead of raw char * and wchar_t * pointers, you can use a function like the following to perform string conversions:

template<typename TARGET, typename SOURCE> TARGET convertString(const SOURCE &s) { TARGET result; result.assign(s.begin(), s.end()); return result; }

Use this as follows:

#include <string> #include <iostream> using namespace std; int main() { wstring wstr(L"HELLO WORLD"); string str(convertString<string, wstring>(wstr)); cout << str << endl; return 0; }

This performs a character-by-character conversion and is platform-independent. This has been tested on Windows using GCC 4.7.3 and Visual C++ 2012 as well as on Linux using GCC 4.7.3.

Improve this answer

6 Comments

I am using STL for string/wstring - so that isn't the issue. But is there any benefit to that over something where I just set the char/wchar_t to each other directly with casting? It seems for my purposes, converting to/fromt wstring/string just to convert and bring it back, seems like it'd be more overhead than raw writing of data.
@PhoenixX_2: There might be a little overhead but it's more likely to be correct than an implementation written from scratch. By using features of the standard library you take advantage of the fact that it's been tested by hundreds of thousands of developers over many years. Character conversions are difficult and error-prone. The STL guys are smarter than me so I'd trust their implementation over mine any day!
@PhoenixX_2 The advantage is that the iterator based assign() will copy each element of the string over to the target type. Going from char* to wchar_t* will convert your single byte characters to double byte characters. Casting will not yield a string that you can actually print and would likely cause other problems with functions that rely on NULL termination. If you are just using the values as opaque blobs and don't need to treat them as strings, casting might work.
@RichardBook That's a fair assessment!
@sbaker The implementation would be setting the chars to wchar_ts, so they would indeed be treated properly without concerns of NULL terminating or what have you.
|
0

The following code can be shortened using std::wstring_convert:

#include <string> #include <locale> std::string convert(std::wstring str, std::locale loc = std::locale(), std::mbstate_t state = std::mbstate_t()) { const wchar_t* a; char *b; std::string res; res.resize(str.size()); auto bytes = std::use_facet<std::codecvt<wchar_t, char, std::mbstate_t>>(loc) .out(state, &str[0], &str[str.size()], a, &res[0], &res[res.size()], b); return res; } int main() { std::wstring a = L"abcdef"; std::string b = convert(a); }

Demo

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.