0 
static void Main(string[] args) { string TheDataFile = ""; string ErrorMsg = ""; string lngTransDate = ""; ProcessDataFile ProcessTheDataFile = new ProcessDataFile(); string TheFile = "S:\\MIS\\Provider NPI file\\Processed\\npidata_20050523-20161009.csv"; string[] lines = File.ReadAllLines(TheFile, Encoding.UTF8);//Read all lines to an array Console.WriteLine(lines.Length.ToString()); Console.ReadLine(); }

This throws an error because the file is very large (has 6 million lines). Is there a way to handle large files and count the # of lines?

Improve this question
4
  • Read it line by line. See this example. Commented Jul 19, 2017 at 15:29
  • If you just want to get the line count, stream it and loop through line by line to get the count. That way you aren't holding the whole thing in memory. Commented Jul 19, 2017 at 15:29
  • Maybe this post can help Commented Jul 19, 2017 at 15:30
  • BTW, you don't need the .ToString() on lines.Length... Console.WriteLine will handle integers just fine. Commented Jul 19, 2017 at 15:44

3 Answers 3

Reset to default
4

Use a StreamReader:

string TheFile = "S:\\MIS\\Provider NPI file\\Processed\\npidata_20050523-20161009.csv"; int count = 0; using (System.IO.StreamReader sr = new System.IO.StreamReader(TheFile)) { while (sr.ReadLine() != null) count++; }
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

On a 1.6GB data file, this method has used half the memory my way did.
2

You need to do a lazy evaluation of the file so it isn't loaded into memory entirelly.

Helper method

public static class ToolsEx { public static IEnumerable<string> ReadAsLines(this string filename) { using (var streamReader = new StreamReader(filename)) while (!streamReader.EndOfStream) yield return streamReader.ReadLine(); } }

Usage 

var lineCount = "yourfile.txt".ReadAsLines().Count();
Improve this answer

5 Comments

Personally I would call the method EnumerateLines so it follows the Directory.GetFiles()/Directory.EnumerateFiles() pattern. Also not make it a extension method of a string, doing "yourfile.txt".ReadAsLines() looks weird to me.
That's your opinion... and I disagree with it :) The intent is to read the file. It's also easier for non-programmers to understand what Read versus Enumerate. (In my opinion)
I would agree with @ScottChamberlain Not all strings are files. That's the weird part.
If it makes you feel better then use it like this...var lineCount = ToolsEx.ReadAsLines(filename:"yourfile.txt").Count()
I'm with @ScottChamberlain regarding string extension method. Also there is no need of such helper method at all since it replicates File.ReadLines
2

According to this already accepted answer, this should do it.

using System; using System.IO; namespace CountLinesInFiles_45194927 { class Program { static void Main(string[] args) { int counter = 0; foreach (var line in File.ReadLines("c:\\Path\\To\\File.whatever")) { counter++; } Console.WriteLine(counter); Console.ReadLine(); } } }
Improve this answer

1 Comment

@ScottChamberlain, Using a 1.6GB datafile, the VSHost process, using OP's way, used over 5GB of memory. My way did not go over 14mb. Not to mention that OP's way also took longer to process.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.