Force strings to UTF-8 from any encoding

Question

In my rails app I'm working with RSS feeds from all around the world, and some feeds have links that are not in UTF-8. The original feed links are out of my control, and in order to use them in other parts of the app, they need to be in UTF-8.

How can I detect encoding and convert to UTF-8?

To detect an encoding, you need to parse the accompanying meta information of the documents, i.e. HTTP headers or <meta> tags. — deceze
– deceze ♦, Commented Oct 18, 2012 at 6:40

Dylan Pierce · Accepted Answer · 2019-12-16 17:03:34Z

71

Ruby 1.9

"Forcing" an encoding is easy, however it won't convert the characters just change the encoding:

str = str.force_encoding('UTF-8') str.encoding.name # => 'UTF-8'

If you want to perform a conversion, use encode:

begin str.encode("UTF-8") rescue Encoding::UndefinedConversionError # ... end

I would definitely read the following post for more information:
http://graysoftinc.com/character-encodings/ruby-19s-string

edited Dec 16, 2019 at 17:03

Dylan Pierce

4,6883 gold badges41 silver badges49 bronze badges

answered Oct 18, 2012 at 6:39

kwarrick

6,2602 gold badges29 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Hackeron Over a year ago

Doesn't work: whois = whois.force_encoding("UTF-8") \n whois.encoding.name => "UTF-8" \n whois.scan(/role:\s+(.+)/i) -- Throws: ArgumentError: invalid byte sequence in UTF-8

kwarrick Over a year ago

As stated, force_encoding does not convert the characters and certainly cannot magically interpret invalid UTF-8 byte sequences.

Joseworks Over a year ago

Current syntax for Ruby 2.2.0 and above is: str.force_encoding(Encoding::UTF_8) Encoding

kjw0188 · Accepted Answer · 2023-06-20 23:26:50Z

This will ensure you have the correct encoding and won't error out because it replaces any invalid or undefined character with a blank string.

This will ensure no matter what, that you have a valid UTF-8 string

str.encode(Encoding.find('UTF-8'), {invalid: :replace, undef: :replace, replace: ''})

For Ruby 3.0+:

str.encode(Encoding.find('UTF-8'), invalid: :replace, undef: :replace, replace: '')

This will raise no implicit conversion of Hash into String on modern ruby (probably after 3.0) Use str.encode(Encoding.find('UTF-8'), invalid: :replace, undef: :replace, replace: '')
thx i have this problem, your solution solve problem with ruby 3+ ;)

ar31an · Accepted Answer · 2020-06-12 09:00:36Z

Only this solution worked for me:

string.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')

Note the binary argument.

Dipak Panchal · Accepted Answer · 2012-10-18 05:56:45Z

Iconv

require 'iconv' i = Iconv.new('UTF-8','LATIN1') a_with_hat = i.iconv("\xc2")

Summary: the iconv gem does all the work of converting encodings. Make sure it's installed with:

gem install iconv

Now, you need to know what encoding your string is currently in as Ruby 1.8 treats Strings as an array of bytes (with no intrinsic encoding.) For example, say your string was in latin1 and you wanted to convert it to utf-8

require 'iconv' string_in_utf8_encoding = Iconv.conv("UTF8", "LATIN1", string_in_latin1_encoding)

Thanks for the answer, but in my case the source data is inconsistent and I don't really have a reliable way to preempt encodings
Iconv should not be used anymore. (deprecated) stackoverflow.com/questions/8148762/…

Collectives™ on Stack Overflow

Force strings to UTF-8 from any encoding

4 Answers 4

3 Comments

2 Comments

Comments

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

2 Comments

Comments

2 Comments

Linked

Related