1

I am trying to remove "^@" from multiple text files using Unix platform. I have already found this solution, but it does not work for my case. I also used sed -i -e 's/^@//g' testfile.txt and dos2unix testfile.txt. sample data are put here.

Any suggestion would be appreciated.

Improve this question
11
  • Can you post your sample input and expected output? Commented Oct 29, 2018 at 12:36
  • I do not know about literal circumflexes and ats. Can you please clarify it? I redo dos2unix but no success. you can also the file in this link. Commented Oct 29, 2018 at 12:37
  • dropbox.com/s/73a6muje9ew3dej/test.txt?dl=0 Commented Oct 29, 2018 at 12:37
  • 2
    @SeyedOmidNabavi You can convert the file to UTF-8 with something like iconv or recode: iconv -f UTF-16 -t UTF-8 < test.txt or recode utf16le..utf8 < test.txt Commented Oct 29, 2018 at 12:47
  • 1
    Possible duplicate of Using iconv to convert from UTF-16LE to UTF-8 Commented Oct 29, 2018 at 12:50

1 Answer 1

Reset to default
2

The ^@ that you're seeing isn't a literal string. It's an escape code for a NUL (character value 0). If you want to remove them all:

tr -d '\0' <test.txt >newfile.txt

To help diagnose this sort of thing, the od (octal dump) utility is handy. I ran this on the test file you linked, to confirm that they were NULs:

$ od -c test.txt | head 0000000 \0 A \0 i \0 r \0 Q \0 u \0 a \0 l \0 i 0000020 \0 t \0 y \0 S \0 t \0 a \0 t \0 i \0 o 0000040 \0 n \0 E \0 o \0 I \0 C \0 o \0 d \0 e 0000060 \0 \n \0 D \0 E \0 H \0 E \0 0 \0 4 \0 4 * 0000400 \0 \n \0 D \0 E \0 H \0 E \0 0 \0 4 \0 0000420 4 \0 \n \0 D \0 E \0 H \0 E \0 0 \0 4 \0 * 0422160 4 \0 \n \n 0422164
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Although this solved the immediate problem, it looks like the data was UTF-16, and you should probably use a real conversion tool (see comments on your question).
The file is in UTF-16. Removing the nulls will screw up any non-ASCII characters.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.