627

What is the fastest, easiest tool or method to convert text files between character sets?

Specifically, I need to convert from UTF-8 to ISO-8859-15 and vice versa.

Everything goes: one-liners in your favorite scripting language, command-line tools or other utilities for OS, web sites, etc.

Best solutions so far:

On Linux/UNIX/OS X/cygwin:

  • Gnu iconv suggested by Troels Arvin is best used as a filter. It seems to be universally available. Example:

     $ iconv -f UTF-8 -t ISO-8859-15 in.txt > out.txt

    As pointed out by Ben, there is an online converter using iconv.

  • recode (manual) suggested by Cheekysoft will convert one or several files in-place. Example:

     $ recode UTF8..ISO-8859-15 in.txt

    This one uses shorter aliases:

     $ recode utf8..l9 in.txt

    Recode also supports surfaces which can be used to convert between different line ending types and encodings:

    Convert newlines from LF (Unix) to CR-LF (DOS):

     $ recode ../CR-LF in.txt

    Base64 encode file:

     $ recode ../Base64 in.txt

    You can also combine them.

    Convert a Base64 encoded UTF8 file with Unix line endings to Base64 encoded Latin 1 file with Dos line endings:

     $ recode utf8/Base64..l1/CR-LF/Base64 file.txt

On Windows with Powershell (Jay Bazuzi):

  • PS C:\> gc -en utf8 in.txt | Out-File -en ascii out.txt

(No ISO-8859-15 support though; it says that supported charsets are unicode, utf7, utf8, utf32, ascii, bigendianunicode, default, and oem.)

Edit

Do you mean iso-8859-1 support? Using "String" does this e.g. for vice versa

gc -en string in.txt | Out-File -en utf8 out.txt

Note: The possible enumeration values are "Unknown, String, Unicode, Byte, BigEndianUnicode, UTF8, UTF7, Ascii".

Improve this question
9
  • 1
    I tried gc -en Ascii readme.html | Out-File -en UTF8 readme.html but it converts the file to utf-8 but then it's empty! Notepad++ says the file is Ansi-format but reading up as I understand it that's not even a valid charset?? uk.answers.yahoo.com/question/index?qid=20100927014115AAiRExF Commented Sep 13, 2013 at 12:24
  • 3
    Just come across this looking for an answer to a related question - great summary! Just thought it was worth adding that recode will act as a filter as well if you don't pass it any filenames, e.g.: recode utf8..l9 < in.txt > out.txt Commented Mar 6, 2014 at 11:05
  • 1
    iconv.com/iconv.htm seems to be dead for me? (timeout) Commented May 12, 2014 at 6:51
  • 3
    If you use enca, you do not need to specify the input encoding. It is often enough just to specify the language: enca -L ru -x utf8 FILE.TXT. Commented Jul 31, 2015 at 19:04
  • 1
    Actually, iconv worked much better as an in-place converter instead of a filter. Converting a file with more than 2 million lines using iconv -f UTF-32 -t UTF-8 input.csv > output.csv saved only about seven hundred thousand lines, only a third. Using the in-place version iconv -f UTF-32 -t UTF-8 file.csv converted successfully all 2 million plus lines. Commented May 19, 2016 at 23:04

21 Answers 21

Reset to default
308
+150

Stand-alone utility approach

iconv -f ISO-8859-1 -t UTF-8 in.txt > out.txt 
-f ENCODING the encoding of the input -t ENCODING the encoding of the output

You don't have to specify either of these arguments. They will default to your current locale, which is usually UTF-8.

Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

For anyone else who's getting tripped up by the non-dash versions being unavailable, it looks like OSX (and possibly all BSD) versions of iconv don't support the non-dash aliases for the various UTF-* encodings. iconv -l | grep UTF will tell you all the UTF-related encodings that your copy of iconv does support.
Don't know the encoding of your input file? Use chardet in.txt to generate a best guess. The result can be used as ENCODING in iconv -f ENCODING.
Prevent exit at invalid characters (avoiding illegal input sequence at position messages), and replace "weird" characters with "similar" characters: iconv -c -f UTF-8 -t ISO-8859-1//TRANSLIT in.txt > out.txt.
FWIW the file command reported my source as UTF-16 Little Endian; running iconv -f UTF-16 -t UTF-8... transformed it incorrectly to ASCII, i had to explicitly specify iconv -f UTF-16LE... to output UTF-8
@jimmymcheung: Plain 7-bit ASCII is by definition the exact same as Unicode. So it's expected.
|
115

Try VIM

If you have vim you can use this:

Not tested for every encoding.

The cool part about this is that you don't have to know the source encoding

vim +"set nobomb | set fenc=utf8 | x" filename.txt

Be aware that this command modify directly the file

Explanation part!

  1. + : Used by vim to directly enter command when opening a file. Usualy used to open a file at a specific line: vim +14 file.txt
  2. | : Separator of multiple commands (like ; in bash)
  3. set nobomb : no utf-8 BOM
  4. set fenc=utf8 : Set new encoding to utf-8 doc link
  5. x : Save and close file
  6. filename.txt : path to the file
  7. " : qotes are here because of pipes. (otherwise bash will use them as bash pipe)
Improve this answer

5 Comments

Quite cool, but somewhat slow. Is there a way to change this to convert a number of files at once (thus saving on vim's initialization costs)?
np, additionaly you can view the bom if you use vim -b or head file.txt|cat -e
for example: find -regextype posix-extended -type f -regex ".*\.(h|cpp|rc|fx|cs|props|xaml)" -exec vim +'set nobomb | set fenc=utf8 | x' {} \;
I used this to convert the encoding of CSV files and was really excited when I saw the charset had indeed changed. Unfortunately, when I went to load the file into MySQL, it had a different number of columns than what it previously had before running the vim command. Wonder if it would be possible to just open the file, convert the encoding, and save/close the file while leaving all other file content the same?
many ways: 1 - Use @Gabriel's command , 2 - Shell expansion vim +'set nobomb | set fenc=utf8 | x' *.yaml (e.g.), 3 - A loop for f in a.txt b.txt; do vim +'set nobomb | set fenc=utf8 | x' "${f}"; done (none of theses has been tested)
40

Under Linux you can use the very powerful recode command to try and convert between the different charsets as well as any line ending issues. recode -l will show you all of the formats and encodings that the tool can convert between. It is likely to be a VERY long list.

Improve this answer

2 Comments

How do you convert to LF? There is /CR and /CR-LF but no /LF
@AaronFranke: as recode internally works with LF, simply use / (instead of /CR) after the ... this should do the trick as long as you don't use encodings which imply different end of lines and don't use the strict mode (-s). for more information have a look at the section "Representation for end of lines" of the info-file.
24 
Get-Content -Encoding UTF8 FILE-UTF8.TXT | Out-File -Encoding UTF7 FILE-UTF7.TXT

The shortest version, if you can assume that the input BOM is correct:

gc FILE.TXT | Out-File -en utf7 file-utf7.txt
Improve this answer

6 Comments

Here's a shorter version that works better. gc .\file-utf8.txt | sc -en utf7 .\file-utf7.txt
@LarryBattle: How does Set-Content work better than Out-File?
...oh. I guess they're nearly the same thing. I had trouble running your example because I was assuming that both versions were using the same file-utf8.txt file for input since they both had the same output file as file-utf7.txt.
This would be really great, except that it doesn't support UTF16. It supports UTF32, but not UTF16! I wouldn't need to convert files, except that a lot of Microsoft software (f.e. SQL server bcp) insists on UTF16 - and then their utility won't convert to it. Interesting to say the least.
I tried gc -en Ascii readme.html | Out-File -en UTF8 readme.html but it converts the file to utf-8 but then it's empty! Notepad++ says the file is Ansi-format but reading up as I understand it that's not even a valid charset?? uk.answers.yahoo.com/question/index?qid=20100927014115AAiRExF
|
24

iconv(1)

iconv -f FROM-ENCODING -t TO-ENCODING file.txt

Also there are iconv-based tools in many languages.

Improve this answer

1 Comment

What about auto-detecting the original encoding?
19

Try iconv Bash function

I've put this into .bashrc:

utf8() { iconv -f ISO-8859-1 -t UTF-8 $1 > $1.tmp rm $1 mv $1.tmp $1 }

..to be able to convert files like so:

utf8 MyClass.java
Improve this answer

4 Comments

it's better style to use tmp=$(mktmp) to create a temporary file. Also, the line with rm is redundant.
can you complete this function with auto detect input format?
beware, this function deletes the input file without verifying that the iconv call succeeded.
This changes the contents of the text file. I ran this on a UTF-8 with BOM expecting to get out a UTF-8 without BOM file, but it prepended  at the start of the file.
19

Try Notepad++

On Windows I was able to use Notepad++ to do the conversion from ISO-8859-1 to UTF-8. Click "Encoding" and then "Convert to UTF-8".

Improve this answer

Comments

17

Oneliner using find, with automatic character set detection

The character encoding of all matching text files gets detected automatically and all matching text files are converted to utf-8 encoding:

$ find . -type f -iname *.txt -exec sh -c 'iconv -f $(file -bi "$1" |sed -e "s/.*[ ]charset=//") -t utf-8 -o converted "$1" && mv converted "$1"' -- {} \;

To perform these steps, a sub shell sh is used with -exec, running a one-liner with the -c flag, and passing the filename as the positional argument "$1" with -- {}. In between, the utf-8 output file is temporarily named converted.

Whereby file -bi means:

  • -b, --brief Do not prepend filenames to output lines (brief mode).

  • -i, --mime Causes the file command to output mime type strings rather than the more traditional human readable ones. Thus it may say for example text/plain; charset=us-ascii rather than ASCII text. The sed command cuts this to only us-ascii as is required by iconv.

The find command is very useful for such file management automation. Click here for more find galore.

Improve this answer

7 Comments

I had to adapt this solution a bit to work on Mac OS X, at least at my version. find . -type f -iname *.txt -exec sh -c 'iconv -f $(file -b --mime-encoding "$1" | awk "{print toupper(\$0)}") -t UTF-8 > converted "$1" && mv converted "$1"' -- {} \;
Your code worked on Windows 7 with MinGW-w64 (latest version) too. Thanks for sharing it!
@rmuller The sed command is there on purpose, enabling the automatic detection of character encoding. I have expanded the answer to explain this now. It would be courteous with regards to the readership to delete any remaining irrelevant comments. Thank you.
@SergeStroobandt Maybe i was not clear enough. My point is when you use "file -b --mime-encoding" instead of "file -bi" there is no need for filtering the result with sed. This command already returns the file encoding only. So in your example "us-ascii"
This doesn't actually seem to do anything for me on Linux. I saved a file as UTF-8 with BOM and expected it to convert to UTF-8 without BOM and it didn't.
|
8

Assuming, you don't know the input encoding and still wish to automate most of the conversion, I concluded this one liner from summing up previous answers.

iconv -f $(chardetect input.text | awk '{print $2}') -t utf-8 -o output.text
Improve this answer

1 Comment

Is there any Windows alternative method about this method? Thanks.
6

DOS/Windows: use Code page

chcp 65001>NUL type ascii.txt > unicode.txt

Command chcp can be used to change the code page. Code page 65001 is Microsoft name for UTF-8. After setting code page, the output generated by following commands will be of code page set.

Improve this answer

Comments

5

Try EncodingChecker

EncodingChecker on github

File Encoding Checker is a GUI tool that allows you to validate the text encoding of one or more files. The tool can display the encoding for all selected files, or only the files that do not have the encodings you specify.

File Encoding Checker requires .NET 4 or above to run.

For encoding detection, File Encoding Checker uses the UtfUnknown Charset Detector library. UTF-16 text files without byte-order-mark (BOM) can be detected by heuristics.

enter image description here

Improve this answer

2 Comments

Very nice tool, it can also convert the detected encoding to the user specified encoding.
How do you install it? I didn't find any instructions.
3

PHP iconv()

iconv("UTF-8", "ISO-8859-15", $input);

Improve this answer

1 Comment

This statement works great when converting strings, but not for files.
2

Visual Studio Code

  1. Open your file in Visual Studio Code
  2. Reopen with Encoding: In the bottom status bar, to the right, you should see your current file encoding (eg "UTF-8"). Click this and select "Reopen with Encoding".
  3. Select the correct encoding of the file (eg: ISO 8859-2).
  4. Confirm that your content is displaying as expected.
  5. Save with Encoding: The bottom status bar should now display your new encoding format (eg: ISO 8859-2). Click this and choose "Save with Encoding" and select UTF-8 (or whatever new encoding you want).

NOTE: THIS WILL OVERWRITE YOUR ORGINIAL FILE. MAKE A BACKUP FIRST.

Improve this answer

Comments

1

to write properties file (Java) normally I use this in linux (mint and ubuntu distributions):

$ native2ascii filename.properties

For example:

$ cat test.properties first=Execução número um second=Execução número dois $ native2ascii test.properties first=Execu\u00e7\u00e3o n\u00famero um second=Execu\u00e7\u00e3o n\u00famero dois

PS: I writed Execution number one/two in portugues to force special characters.

In my case, in first execution I received this message:

$ native2ascii teste.txt The program 'native2ascii' can be found in the following packages: * gcj-5-jdk * openjdk-8-jdk-headless * gcj-4.8-jdk * gcj-4.9-jdk Try: sudo apt install <selected package>

When I installed the first option (gcj-5-jdk) the problem was finished.

I hope this help someone.

Improve this answer

Comments

1

With ruby:

ruby -e "File.write('output.txt', File.read('input.txt').encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: ''))"

Source: https://robots.thoughtbot.com/fight-back-utf-8-invalid-byte-sequences

Improve this answer

Comments

1

Use this Python script: https://github.com/goerz/convert_encoding.py Works on any platform. Requires Python 2.7.

Improve this answer

Comments

1

Simply change encoding of loaded file in IntelliJ IDEA IDE, on the right of status bar (bottom), where current charset is indicated. It prompts to Reload or Convert, use Convert. Make sure you backed up original file in advance.

Improve this answer

Comments

1

In powershell:

function Recode($InCharset, $InFile, $OutCharset, $OutFile) { # Read input file in the source encoding $Encoding = [System.Text.Encoding]::GetEncoding($InCharset) $Text = [System.IO.File]::ReadAllText($InFile, $Encoding) # Write output file in the destination encoding $Encoding = [System.Text.Encoding]::GetEncoding($OutCharset) [System.IO.File]::WriteAllText($OutFile, $Text, $Encoding) } Recode Windows-1252 "$pwd\in.txt" utf8 "$pwd\out.txt"

For a list of supported encoding names:

https://learn.microsoft.com/en-us/dotnet/api/system.text.encoding

Improve this answer

Comments

1

There is also a web tool to convert file encoding: https://webtool.cloud/change-file-encoding

It supports wide range of encodings, including some rare ones, like IBM code page 37.

Improve this answer

Comments

0

My favorite tool for this is Jedit (a java based text editor) which has two very convenient features :

  • One which enables the user to reload a text with a different encoding (and, as such, to control visually the result)
  • Another one which enables the user to explicitly choose the encoding (and end of line char) before saving
Improve this answer

Comments

0

If macOS GUI applications are your bread and butter, SubEthaEdit is the text editor I usually go to for encoding-wrangling — its "conversion preview" allows you to see all invalid characters in the output encoding, and fix/remove them.

And it's open-source now, so yay for them 😉.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.