How to remove duplicate words from a text file using java

Question

I have a input file like

names.txt

 Maria Derek Erica Livia Jack Anita Kendall Maria Livia Derek Jamie Jack Thomson Erica

I want to output like. Removing duplicates words from the name

output.txt

 Maria Derek Erica Livia Jack Anita Kendall Jamie Thomson

I already tried to read file separated by whitespace and then add them into ArrayList then I lost what to do next to generate output.

import java.util.Scanner; import java.io.*; import java.util.ArrayList; public class duplicate { public static void main(String[] args) throws IOException { ArrayList<String> list = new ArrayList<String>(); File file = new File("weather.txt"); Scanner input = new Scanner(file); String prev = input.next(); int count = 0; while (input.hasNext()) { String next = input.next(); System.out.println(next); set.add(next); count = count + 1; if(prev.equals(next)) { System.out.println("Match found: -" + prev); } prev = next; } System.out.println(list); System.out.println("Word count: " + count); } }

Your input vs your output in the example you give is inconsistent. If there are duplicates, which one should be retained? The first one, like with "Maria", or the last one, like with "Erica"? What should happen to a line that only contains a duplicate name? I feel like you need to have your desired functionality be more well-defined here before trying to address the mechanics of how to accomplish it. — Jordan
– Jordan, Commented May 30, 2019 at 15:28

Qingfei Yuan · Accepted Answer · 2019-05-30 15:36:09Z

3

If order doesn't matter, just use Set. Set will filter out duplicate element automatically. Then just print out that set. You will be fine.

Set<String> list = new HashSet<String>();

edited May 30, 2019 at 15:36

answered May 30, 2019 at 15:29

Qingfei Yuan

1,2121 gold badge9 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Atik Rahman Over a year ago

I tried but I failed to use set. Is there any version requirements for set ?

Tyulpan Tyulpan · Accepted Answer · 2019-05-30 16:05:28Z

You can even save the order creating set as:

Set<String> uniqueNames = new LinkedHashSet<>();

Then you can just add the elements to your set:

uniqueNames.add(next);

P.S. · Accepted Answer · 2019-05-30 16:05:15Z

import java.util.Scanner; import java.io.*; import java.util.HashSet; public class Duplicate { public static void main(String[] args) throws IOException { HashSet<String> lines = new HashSet<String>(); File file = new File("weather.txt"); Scanner input = new Scanner(file); int count = 0; while (input.hasNext()) { String next = input.next(); System.out.println(next); lines.add(next); count++; } input.close(); System.out.println(lines); System.out.println("Word count: " + count); System.out.println("Unique word count: " + lines.size()); } }

jeremye · Accepted Answer · 2019-05-31 01:36:30Z

You are going to want to make use of a Set or HashSet in order to track duplicate names, and an ArrayList in order to store your ultimate lines to be written to your output file.

Since we are making changes on a line-by-line basis, we need to 1) read the file line-by-line, 2) split the line up to process the individual names, 3) edit the line or create a new resulting line as required, and 4) then store the edited line to be output at the end:

import java.util.*; import java.io.*; public class Duplicate { public static void main(String[] args) throws IOException { HashSet<String> uniqueNames = new HashSet<String>(); ArrayList<String> outputLines = new ArrayList<String>(); Scanner input = new Scanner(new File("input.txt")); while (input.hasNextLine()) { // Split a line into an array of names. String[] names = input.nextLine().split(" "); String edited = ""; for (int i = 0; i < names.length; i++) { // If the name is already in the set, remove it. if (uniqueNames.add(names[i])) { edited += names[i] + " "; } } edited = edited.strip(); // remove excess whitespace // Add the final line to our output array. if (!edited.equals("")) { outputLines.add(edited); } } // Write the output array to a file. String outputFn = "output.txt"; BufferedWriter output = new BufferedWriter(new FileWriter(outputFn)); output.write(String.join("\n", outputLines)); output.close(); System.out.println("File '" + outputFn + "' created!"); } }

I leave it as an exercise if you want to add other features, like counting total words or unique words, since the question seemed to be mostly about the removing duplicates behavior.

1) you've made the solution, we are helping here, not solving; 2) your solution is bad
@TyulpanTyulpan I'm open to the possibility of both being true. Let's tackle them one at a time: what do you think is bad about my solution, and how would you suggest to improve it?
Do you know this is insane practice to use classes instead of interfaces for objects creation? Do you know this task can be solved in a few lines instead of yours 30 using Java 8?
@TyulpanTyulpan I would love to see a solution in only a few lines, that sounds really cool!

Collectives™ on Stack Overflow

How to remove duplicate words from a text file using java

4 Answers 4

1 Comment

Comments

Comments

4 Comments

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

Comments

Comments

4 Comments

Related