Removing duplicate rows in vi?

Try this:

:%s/^\(.*\)\(\n\1\)\+$/\1/ 

It searches for any line immediately followed by one or more copies of itself, and replaces it with a single copy.

Make a copy of your file though before you try it. It's untested.

From command line just do:

sort file | uniq > file.new 

awk '!x[$0]++' yourfile.txt if you want to preserve the order (i.e., sorting is not acceptable). In order to invoke it from vim, :! can be used.

I would combine two of the answers above:

go to head of file sort the whole file remove duplicate entries with uniq 1G !Gsort 1G !Guniq 

If you were interested in seeing how many duplicate lines were removed, use control-G before and after to check on the number of lines present in your buffer.

g/^\(.*\)$\n\1/d 

Works for me on Windows. Lines must be sorted first though.

If you don't want to sort/uniq the entire file, you can select the lines you want to make uniq in visual mode and then simply: :sort u.

Select the lines in visual-line mode (Shift+v), then :!uniq. That'll only catch duplicates which come one after another.

Regarding how Uniq can be implemented in VimL, search for Uniq in a plugin I'm maintaining. You'll see various ways to implement it that were given on Vim mailing-list.

Otherwise, :sort u is indeed the way to go.

An alternative method that does not use vi/vim (for very large files), is from the Linux command line use sort and uniq:

sort {file-name} | uniq -u 

From here this will remove adjacent and non-adjacent duplicates without sorting:

:%!awk '\!a[$0]++' 

This technically uses something outside of vim, but is called from within vim (and therefore only works in linux which has awk).

To do this entirely from within vim you can do this using a macro and the norm command to execute it on every line. On linux, this was fast, but on windows it took an oddly long time. Disabling plugins using vim -u NONE seemed to help somewhat.

qa # create macro in register 'a' y$ # yank the current line :.+1,$g/<ctrl-r>0/d # from the next line to the end of file, delete any pattern that matches q # end of macro :%norm! @a # apply macro on every line in file. 

Note this doesn't remove empty lines so performing

:g/^$/d 

to remove any blank spaces may be useful.

:%s/^\(.*\)\(\n\1\)\+$/\1/gec 

or

:%s/^\(.*\)\(\n\1\)\+$/\1/ge 

this is my answer for you ,it can remove multiple duplicate lines and only keep one not remove !

I would use !}uniq, but that only works if there are no blank lines.

For every line in a file use: :1,$!uniq.

This version only removes repeated lines that are contigous. I mean, only deletes consecutive repeated lines. Using the given map the function does note mess up with blank lines. But if change the REGEX to match start of line ^ it will also remove duplicated blank lines.

" function to delete duplicate lines function! DelDuplicatedLines() while getline(".") == getline(line(".") - 1) exec 'norm! ddk' endwhile while getline(".") == getline(line(".") + 1) exec 'norm! dd' endwhile endfunction nnoremap <Leader>d :g/./call DelDuplicatedLines()<CR> 

This command got me a buffer without any duplicate lines without sorting, and it shouldn't be very hard to research why it works or how it could work better:

:%!python3.11 -c 'exec("import fileinput\nLINES = []\nfor line in fileinput.input():\n line = line.splitlines()[0]\n if line not in LINES:\n print(line)\n LINES.append(line)\n")' 

This worked for me for both .csv and .txt

awk '!seen[$0]++' <filename> > <newFileName>

Explanation: The first part of the command prints unique rows and the second part i.e. after the middle arrow is to save the output of the first part.

awk '!seen[$0]++' <filename>

>

<newFileName>