Unicode in Perl not working

Question

I have some text files which I am trying to transform with a Perl script on Windows. The text files look normal in Notepad+, but all the regexes in my script were failing to match. Then I noticed that when I open the text files in NotePad+, the status bar says "UCS-2 Little Endia" (sic). I am assuming this corresponds to the encoding UCS-2LE. So I created "readFile" and "writeFile" subs in Perl, like so:

use PerlIO::encoding; my $enc = ':encoding(UCS-2LE)'; sub readFile { my ($fName) = @_; open my $f, "<$enc", $fName or die "can't read $fName\n"; local $/; my $txt = <$f>; close $f; return $txt; } sub writeFile { my ($fName, $txt) = @_; open my $f, ">$enc", $fName or die "can't write $fName\n"; print $f $txt; close $f; } my $fName = 'someFile.txt'; my $txt = readFile $fName; # ... transform $txt using s/// ... writeFile $fName, $txt;

Now the regexes match (although less often than I expect), but the output contains long strings of Asian-looking characters interspersed with longs strings of the correct text. Is my code wrong? Or perhaps Notepad+ is wrong about the encoding? How should I proceed?

Provide some test data, an additional hexdump of it is a good idea. Show the complete code that causes the problem. If you do not enable us to reproduce the problem, we can only speculate. — daxim
– daxim, Commented Jul 22, 2010 at 7:40
@daxim: the code here is reasonably complete, the only reason I can't check it is because I don't have the notepad+ editor to compare with. At a guess it's a BOM problem. — user181548
– user181548, Commented Jul 22, 2010 at 8:00

Community · Accepted Answer · 2017-05-23 12:30:48Z

OK, I figured it out. The problem was being caused by a disconnect between the encoding translation done by the "encoding..." parameter of the "open" call and the default CRLF translation done by Perl on Windows. What appeared to be happening was that LF was being translated to CRLF on output after the encoding had already been done, which threw off the "parity" of the 16-bit encoding for the following line. Once the next line was reached, the "parity" got put back. That would explain the "long strings of Asian-looking characters interspersed with longs strings of the correct text"... every other line was being messed up.

To correct it, I took out the encoding parameter in my "open" call and added a "binmode" call, as follows:

open my $f, $fName or die "can't read $fName\n"; binmode $f, ':raw:encoding(UCS-2LE)';

binmode apparently has a concept of "layered" I/O handling that is somewhat complicated.

One thing I can't figure out is how to get my CRLF translation back. If I leave out :raw or add :crlf, the "parity" problem returns. I've tried re-ordering as well and can't get it to work.

(I added this as a separate question: CRLF translation with Unicode in Perl)

user181548 · Accepted Answer · 2010-07-22 00:32:15Z

I don't have the Notepad+ editor to check but it may be a BOM problem with your output encoding not containing a BOM.

http://perldoc.perl.org/Encode/Unicode.html#Size%2c-Endianness%2c-and-BOM

Maybe you need to encode $txt using a byte order mark as described above.

Collectives™ on Stack Overflow

Unicode in Perl not working

2 Answers 2

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Linked

Related