Remove Duplicate Lines from Text using Java

If you feed the lines into a LinkedHashSet, it ignores the repeated ones, since it's a set, but preserves the order, since it's linked. If you just want to know whether you've seena given line before, feed them into a simple Set as you go on, and ignore those which the Set already contains/contained.

It can be easy to remove duplicate line from text or File using new java Stream API. Stream support different aggregate feature like sort,distinct and work with different java's existing data structures and their methods. Following example can use to remove duplicate or sort the content in File using Stream API

package removeword; import java.io.IOException; import java.nio.file.Files; import java.nio.file.OpenOption; import java.nio.file.Path; import java.nio.file.Paths; import java.util.Arrays; import java.util.Scanner; import java.util.stream.Stream; import static java.nio.file.StandardOpenOption.*; import static java.util.stream.Collectors.joining; public class Java8UniqueWords { public static void main(String[] args) throws IOException { Path sourcePath = Paths.get("C:/Users/source.txt"); Path changedPath = Paths.get("C:/Users/removedDouplicate_file.txt"); try (final Stream<String> lines = Files.lines(sourcePath ) // .map(line -> line.toLowerCase()) /*optional to use existing string methods*/ .distinct() // .sorted()) /*aggregrate function to sort disctincted line*/ { final String uniqueWords = lines.collect(joining("\n")); System.out.println("Final Output:" + uniqueWords); Files.write(changedPath , uniqueWords.getBytes(),WRITE, TRUNCATE_EXISTING); } } } 

Read the text file using a BufferedReader and store it in a LinkedHashSet. Print it back out.

Here's an example:

public class DuplicateRemover { public String stripDuplicates(String aHunk) { StringBuilder result = new StringBuilder(); Set<String> uniqueLines = new LinkedHashSet<String>(); String[] chunks = aHunk.split("\n"); uniqueLines.addAll(Arrays.asList(chunks)); for (String chunk : uniqueLines) { result.append(chunk).append("\n"); } return result.toString(); } } 

Here's some unit tests to verify ( ignore my evil copy-paste ;) ):

import org.junit.Test; import static org.junit.Assert.*; public class DuplicateRemoverTest { @Test public void removesDuplicateLines() { String input = "a\nb\nc\nb\nd\n"; String expected = "a\nb\nc\nd\n"; DuplicateRemover remover = new DuplicateRemover(); String actual = remover.stripDuplicates(input); assertEquals(expected, actual); } @Test public void removesDuplicateLinesUnalphabetized() { String input = "z\nb\nc\nb\nz\n"; String expected = "z\nb\nc\n"; DuplicateRemover remover = new DuplicateRemover(); String actual = remover.stripDuplicates(input); assertEquals(expected, actual); } } 

Here's another solution. Let's just use UNIX!

cat MyFile.java | uniq > MyFile.java 

Edit: Oh wait, I re-read the topic. Is this a legal solution since I managed to be language agnostic?

For better/optimum performance, it's wise to use Java 8's API features viz. Streams & Method references with LinkedHashSet for Collection as below:

import java.io.IOException; import java.io.PrintWriter; import java.nio.file.Files; import java.nio.file.Paths; import java.util.LinkedHashSet; import java.util.stream.Collectors; public class UniqueOperation { private static PrintWriter pw; enter code here public static void main(String[] args) throws IOException { pw = new PrintWriter("abc.txt"); for(String p : Files.newBufferedReader(Paths.get("C:/Users/as00465129/Desktop/FrontEndUdemyLinks.txt")). lines(). collect(Collectors.toCollection(LinkedHashSet::new))) pw.println(p); pw.flush(); pw.close(); System.out.println("File operation performed successfully"); } 

here I'm using a hashset to store seen lines

Scanner scan;//input Set<String> lines = new HashSet<String>(); StringBuilder strb = new StringBuilder(); while(scan.hasNextLine()){ String line = scan.nextLine(); if(lines.add(line)) strb.append(line); }