Return to Answer

edited body

Source Link

edited Nov 17, 2016 at 14:30

Stéphane Chazelas

586.2k
96
1.1k
1.7k

Same issue here. Needed to remove BOM (UTF-16LE) from a file. Ended up using "tr" with octal codes for the 0xFF 0xFE:

$ cat old.csv
ÿþ"SET01"|"0000001"|"2016-11-15"|""|"0"|""|""|"Data01"

$ echo "ibase=16\nobae=8\nFF" | bc
377
$ echo "ibase=16\nobae=8\nFE" | bc
376

$ cat old.csv | tr -d "\377\376"
"SET01"|"0000001"|"2016-11-15"|""|"0"|""|""|"Data01"

$ cat old.csv ÿþ"SET01"|"0000001"|"2016-11-15"|""|"0"|""|""|"Data01" $ echo "ibase=16\nobae=8\nFF" | bc 377 $ echo "ibase=16\nobae=8\nFE" | bc 376 $ cat old.csv | tr -d "\377\376" "SET01"|"0000001"|"2016-11-15"|""|"0"|""|""|"Data01"

Same issue here. Needed to remove BOM (UTF-16LE) from a file. Ended up using "tr" with octal codes for the 0xFF 0xFE:

$ cat old.csv
ÿþ"SET01"|"0000001"|"2016-11-15"|""|"0"|""|""|"Data01"

$ echo "ibase=16\nobae=8\nFF" | bc
377
$ echo "ibase=16\nobae=8\nFE" | bc
376

$ cat old.csv | tr -d "\377\376"
"SET01"|"0000001"|"2016-11-15"|""|"0"|""|""|"Data01"

Same issue here. Needed to remove BOM (UTF-16LE) from a file. Ended up using "tr" with octal codes for the 0xFF 0xFE:

$ cat old.csv ÿþ"SET01"|"0000001"|"2016-11-15"|""|"0"|""|""|"Data01" $ echo "ibase=16\nobae=8\nFF" | bc 377 $ echo "ibase=16\nobae=8\nFE" | bc 376 $ cat old.csv | tr -d "\377\376" "SET01"|"0000001"|"2016-11-15"|""|"0"|""|""|"Data01"