How to get the line count of a large file cheaply in Python

One line, faster than the for loop of the OP (although not the fastest) and very concise:

num_lines = sum(1 for _ in open('myfile.txt')) 

You can also boost the speed (and robustness) by using rbU mode and include it in a with block to close the file:

with open("myfile.txt", "rbU") as f: num_lines = sum(1 for _ in f) 

Note: The U in rbU mode is deprecated since Python 3.3 and above, so iwe should use rb instead of rbU (and it has been removed in Python 3.11).

I believe that a memory mapped file will be the fastest solution. I tried four functions: the function posted by the OP (opcount); a simple iteration over the lines in the file (simplecount); readline with a memory-mapped filed (mmap) (mapcount); and the buffer read solution offered by Mykola Kharechko (bufcount).

I ran each function five times, and calculated the average run-time for a 1.2 million-line text file.

Windows XP, Python 2.5, 2 GB RAM, 2 GHz AMD processor

Here are my results:

mapcount : 0.465599966049 simplecount : 0.756399965286 bufcount : 0.546800041199 opcount : 0.718600034714 

Numbers for Python 2.6:

mapcount : 0.471799945831 simplecount : 0.634400033951 bufcount : 0.468800067902 opcount : 0.602999973297 

So the buffer read strategy seems to be the fastest for Windows/Python 2.6

Here is the code:

from __future__ import with_statement import time import mmap import random from collections import defaultdict def mapcount(filename): with open(filename, "r+") as f: buf = mmap.mmap(f.fileno(), 0) lines = 0 readline = buf.readline while readline(): lines += 1 return lines def simplecount(filename): lines = 0 for line in open(filename): lines += 1 return lines def bufcount(filename): f = open(filename) lines = 0 buf_size = 1024 * 1024 read_f = f.read # loop optimization buf = read_f(buf_size) while buf: lines += buf.count('\n') buf = read_f(buf_size) return lines def opcount(fname): with open(fname) as f: for i, l in enumerate(f): pass return i + 1 counts = defaultdict(list) for i in range(5): for func in [mapcount, simplecount, bufcount, opcount]: start_time = time.time() assert func("big_file.txt") == 1209138 counts[func].append(time.time() - start_time) for key, vals in counts.items(): print key.__name__, ":", sum(vals) / float(len(vals)) 

All of these solutions ignore one way to make this run considerably faster, namely by using the unbuffered (raw) interface, using bytearrays, and doing your own buffering. (This only applies in Python 3. In Python 2, the raw interface may or may not be used by default, but in Python 3, you'll default into Unicode.)

Using a modified version of the timing tool, I believe the following code is faster (and marginally more Pythonic) than any of the solutions offered:

def rawcount(filename): f = open(filename, 'rb') lines = 0 buf_size = 1024 * 1024 read_f = f.raw.read buf = read_f(buf_size) while buf: lines += buf.count(b'\n') buf = read_f(buf_size) return lines 

Using a separate generator function, this runs a smidge faster:

def _make_gen(reader): b = reader(1024 * 1024) while b: yield b b = reader(1024*1024) def rawgencount(filename): f = open(filename, 'rb') f_gen = _make_gen(f.raw.read) return sum(buf.count(b'\n') for buf in f_gen) 

This can be done completely with generators expressions in-line using itertools, but it gets pretty weird looking:

from itertools import (takewhile, repeat) def rawincount(filename): f = open(filename, 'rb') bufgen = takewhile(lambda x: x, (f.raw.read(1024*1024) for _ in repeat(None))) return sum(buf.count(b'\n') for buf in bufgen) 

Here are my timings:

function average, s min, s ratio rawincount 0.0043 0.0041 1.00 rawgencount 0.0044 0.0042 1.01 rawcount 0.0048 0.0045 1.09 bufcount 0.008 0.0068 1.64 wccount 0.01 0.0097 2.35 itercount 0.014 0.014 3.41 opcount 0.02 0.02 4.83 kylecount 0.021 0.021 5.05 simplecount 0.022 0.022 5.25 mapcount 0.037 0.031 7.46 

You could execute a subprocess and run wc -l filename

import subprocess def file_len(fname): p = subprocess.Popen(['wc', '-l', fname], stdout=subprocess.PIPE, stderr=subprocess.PIPE) result, err = p.communicate() if p.returncode != 0: raise IOError(err) return int(result.strip().split()[0]) 

After a perfplot analysis, one has to recommend the buffered read solution

def buf_count_newlines_gen(fname): def _make_gen(reader): while True: b = reader(2 ** 16) if not b: break yield b with open(fname, "rb") as f: count = sum(buf.count(b"\n") for buf in _make_gen(f.raw.read)) return count 

It's fast and memory-efficient. Most other solutions are about 20 times slower.

Code to reproduce the plot:

import mmap import subprocess from functools import partial import perfplot def setup(n): fname = "t.txt" with open(fname, "w") as f: for i in range(n): f.write(str(i) + "\n") return fname def for_enumerate(fname): i = 0 with open(fname) as f: for i, _ in enumerate(f): pass return i + 1 def sum1(fname): return sum(1 for _ in open(fname)) def mmap_count(fname): with open(fname, "r+") as f: buf = mmap.mmap(f.fileno(), 0) lines = 0 while buf.readline(): lines += 1 return lines def for_open(fname): lines = 0 for _ in open(fname): lines += 1 return lines def buf_count_newlines(fname): lines = 0 buf_size = 2 ** 16 with open(fname) as f: buf = f.read(buf_size) while buf: lines += buf.count("\n") buf = f.read(buf_size) return lines def buf_count_newlines_gen(fname): def _make_gen(reader): b = reader(2 ** 16) while b: yield b b = reader(2 ** 16) with open(fname, "rb") as f: count = sum(buf.count(b"\n") for buf in _make_gen(f.raw.read)) return count def wc_l(fname): return int(subprocess.check_output(["wc", "-l", fname]).split()[0]) def sum_partial(fname): with open(fname) as f: count = sum(x.count("\n") for x in iter(partial(f.read, 2 ** 16), "")) return count def read_count(fname): return open(fname).read().count("\n") b = perfplot.bench( setup=setup, kernels=[ for_enumerate, sum1, mmap_count, for_open, wc_l, buf_count_newlines, buf_count_newlines_gen, sum_partial, read_count, ], n_range=[2 ** k for k in range(27)], xlabel="num lines", ) b.save("out.png") b.show() 

A one-line Bash solution similar to this answer, using the modern subprocess.check_output function:

def line_count(filename): return int(subprocess.check_output(['wc', '-l', filename]).split()[0]) 

Here is a Python program to use the multiprocessing library to distribute the line counting across machines/cores. My test improves counting a 20 million line file from 26 seconds to 7 seconds using an 8-core Windows 64-bit server. Note: not using memory mapping makes things much slower.

import multiprocessing, sys, time, os, mmap import logging, logging.handlers def init_logger(pid): console_format = 'P{0} %(levelname)s %(message)s'.format(pid) logger = logging.getLogger() # New logger at root level logger.setLevel(logging.INFO) logger.handlers.append(logging.StreamHandler()) logger.handlers[0].setFormatter(logging.Formatter(console_format, '%d/%m/%y %H:%M:%S')) def getFileLineCount(queues, pid, processes, file1): init_logger(pid) logging.info('start') physical_file = open(file1, "r") # mmap.mmap(fileno, length[, tagname[, access[, offset]]] m1 = mmap.mmap(physical_file.fileno(), 0, access=mmap.ACCESS_READ) # Work out file size to divide up line counting fSize = os.stat(file1).st_size chunk = (fSize / processes) + 1 lines = 0 # Get where I start and stop _seedStart = chunk * (pid) _seekEnd = chunk * (pid+1) seekStart = int(_seedStart) seekEnd = int(_seekEnd) if seekEnd < int(_seekEnd + 1): seekEnd += 1 if _seedStart < int(seekStart + 1): seekStart += 1 if seekEnd > fSize: seekEnd = fSize # Find where to start if pid > 0: m1.seek(seekStart) # Read next line l1 = m1.readline() # Need to use readline with memory mapped files seekStart = m1.tell() # Tell previous rank my seek start to make their seek end if pid > 0: queues[pid-1].put(seekStart) if pid < processes-1: seekEnd = queues[pid].get() m1.seek(seekStart) l1 = m1.readline() while len(l1) > 0: lines += 1 l1 = m1.readline() if m1.tell() > seekEnd or len(l1) == 0: break logging.info('done') # Add up the results if pid == 0: for p in range(1, processes): lines += queues[0].get() queues[0].put(lines) # The total lines counted else: queues[0].put(lines) m1.close() physical_file.close() if __name__ == '__main__': init_logger('main') if len(sys.argv) > 1: file_name = sys.argv[1] else: logging.fatal('parameters required: file-name [processes]') exit() t = time.time() processes = multiprocessing.cpu_count() if len(sys.argv) > 2: processes = int(sys.argv[2]) queues = [] # A queue for each process for pid in range(processes): queues.append(multiprocessing.Queue()) jobs = [] prev_pipe = 0 for pid in range(processes): p = multiprocessing.Process(target = getFileLineCount, args=(queues, pid, processes, file_name,)) p.start() jobs.append(p) jobs[0].join() # Wait for counting to finish lines = queues[0].get() logging.info('finished {} Lines:{}'.format( time.time() - t, lines)) 

I would use Python's file object method readlines, as follows:

with open(input_file) as foo: lines = len(foo.readlines()) 

This opens the file, creates a list of lines in the file, counts the length of the list, saves that to a variable and closes the file again.

This is the fastest thing I have found using pure Python.

You can use whatever amount of memory you want by setting buffer, though 2**16 appears to be a sweet spot on my computer.

from functools import partial buffer=2**16 with open(myfile) as f: print sum(x.count('\n') for x in iter(partial(f.read,buffer), '')) 

I found the answer here Why is reading lines from stdin much slower in C++ than Python? and tweaked it just a tiny bit. It’s a very good read to understand how to count lines quickly, though wc -l is still about 75% faster than anything else.

def file_len(full_path): """ Count number of lines in a file.""" f = open(full_path) nr_of_lines = sum(1 for line in f) f.close() return nr_of_lines 

Here is what I use, and it seems pretty clean:

import subprocess def count_file_lines(file_path): """ Counts the number of lines in a file using wc utility. :param file_path: path to file :return: int, no of lines """ num = subprocess.check_output(['wc', '-l', file_path]) num = num.split(' ') return int(num[0]) 

This is marginally faster than using pure Python, but at the cost of memory usage. Subprocess will fork a new process with the same memory footprint as the parent process while it executes your command.

One line solution:

import os os.system("wc -l filename") 

My snippet:

>>> os.system('wc -l *.txt') 

Output:

0 bar.txt 1000 command.txt 3 test_file.txt 1003 total 

Kyle's answer

num_lines = sum(1 for line in open('my_file.txt')) 

is probably best. An alternative for this is:

num_lines = len(open('my_file.txt').read().splitlines()) 

Here is the comparison of performance of both:

In [20]: timeit sum(1 for line in open('Charts.ipynb')) 100000 loops, best of 3: 9.79 µs per loop In [21]: timeit len(open('Charts.ipynb').read().splitlines()) 100000 loops, best of 3: 12 µs per loop 

I got a small (4-8%) improvement with this version which reuses a constant buffer, so it should avoid any memory or GC overhead:

lines = 0 buffer = bytearray(2048) with open(filename) as f: while f.readinto(buffer) > 0: lines += buffer.count('\n') 

You can play around with the buffer size and maybe see a little improvement.

As for me this variant will be the fastest:

#!/usr/bin/env python def main(): f = open('filename') lines = 0 buf_size = 1024 * 1024 read_f = f.read # loop optimization buf = read_f(buf_size) while buf: lines += buf.count('\n') buf = read_f(buf_size) print lines if __name__ == '__main__': main() 

reasons: buffering faster than reading line by line and string.count is also very fast

This code is shorter and clearer. It's probably the best way:

num_lines = open('yourfile.ext').read().count('\n') 

Just to complete the methods in previous answers, I tried a variant with the fileinput module:

import fileinput as fi def filecount(fname): for line in fi.input(fname): pass return fi.lineno() 

And passed a 60-million-lines file to all the stated methods in previous answers:

mapcount: 6.13 simplecount: 4.59 opcount: 4.43 filecount: 43.3 bufcount: 0.171 

It's a little surprise to me that fileinput is that bad and scales far worse than all the other methods...

I have modified the buffer case like this:

def CountLines(filename): f = open(filename) try: lines = 1 buf_size = 1024 * 1024 read_f = f.read # loop optimization buf = read_f(buf_size) # Empty file if not buf: return 0 while buf: lines += buf.count('\n') buf = read_f(buf_size) return lines finally: f.close() 

Now also empty files and the last line (without \n) are counted.

There are already so many answers with great timing comparison, but I believe they are just looking at number of lines to measure performance (e.g., the great graph from Nico Schlömer).

To be accurate while measuring performance, we should look at:

• the number of lines
• the average size of the lines
• ... the resulting total size of the file (which might impact memory)

First of all, the function of the OP (with a for) and the function sum(1 for line in f) are not performing that well...

Good contenders are with mmap or buffer.

To summarize: based on my analysis (Python 3.9 on Windows with SSD):

1. For big files with relatively short lines (within 100 characters): use function with a buffer buf_count_newlines_gen

   def buf_count_newlines_gen(fname: str) -> int: """Count the number of lines in a file""" def _make_gen(reader): b = reader(1024 * 1024) while b: yield b b = reader(1024 * 1024) with open(fname, "rb") as f: count = sum(buf.count(b"\n") for buf in _make_gen(f.raw.read)) return count 

2. For files with potentially longer lines (up to 2000 characters), disregarding the number of lines: use function with mmap: count_nb_lines_mmap

   def count_nb_lines_mmap(file: Path) -> int: """Count the number of lines in a file""" with open(file, mode="rb") as f: mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) nb_lines = 0 while mm.readline(): nb_lines += 1 mm.close() return nb_lines 

3. For a short code with very good performance (especially for files of size up to medium size):

   def itercount(filename: str) -> int: """Count the number of lines in a file""" with open(filename, 'rb') as f: return sum(1 for _ in f) 

Here is a summary of the different metrics (average time with timeit on 7 runs with 10 loops each):

Note: I have also compared count_nb_lines_mmap and buf_count_newlines_gen on a file of 8 GB, with 9.7 million lines of more than 800 characters. We got an average of 5.39 seconds for buf_count_newlines_gen vs. 4.2 seconds for count_nb_lines_mmap, so this latter function seems indeed better for files with longer lines.

Here is the code I have used:

import mmap from pathlib import Path def count_nb_lines_blocks(file: Path) -> int: """Count the number of lines in a file""" def blocks(files, size=65536): while True: b = files.read(size) if not b: break yield b with open(file, encoding="utf-8", errors="ignore") as f: return sum(bl.count("\n") for bl in blocks(f)) def count_nb_lines_mmap(file: Path) -> int: """Count the number of lines in a file""" with open(file, mode="rb") as f: mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) nb_lines = 0 while mm.readline(): nb_lines += 1 mm.close() return nb_lines def count_nb_lines_sum(file: Path) -> int: """Count the number of lines in a file""" with open(file, "r", encoding="utf-8", errors="ignore") as f: return sum(1 for line in f) def count_nb_lines_for(file: Path) -> int: """Count the number of lines in a file""" i = 0 with open(file) as f: for i, _ in enumerate(f, start=1): pass return i def buf_count_newlines_gen(fname: str) -> int: """Count the number of lines in a file""" def _make_gen(reader): b = reader(1024 * 1024) while b: yield b b = reader(1024 * 1024) with open(fname, "rb") as f: count = sum(buf.count(b"\n") for buf in _make_gen(f.raw.read)) return count def itercount(filename: str) -> int: """Count the number of lines in a file""" with open(filename, 'rbU') as f: return sum(1 for _ in f) files = [small_file, big_file, small_file_shorter, big_file_shorter, small_file_shorter_sim_size, big_file_shorter_sim_size] for file in files: print(f"File: {file.name} (size: {file.stat().st_size / 1024 ** 2:.2f} MB)") for func in [ count_nb_lines_blocks, count_nb_lines_mmap, count_nb_lines_sum, count_nb_lines_for, buf_count_newlines_gen, itercount, ]: result = func(file) time = Timer(lambda: func(file)).repeat(7, 10) print(f" * {func.__name__}: {result} lines in {mean(time) / 10 * 1000:.3f} ms") print() 

File: small_file.ndjson (size: 1.16 MB) * count_nb_lines_blocks: 915 lines in 1.718 ms * count_nb_lines_mmap: 915 lines in 0.582 ms * count_nb_lines_sum: 915 lines in 1.993 ms * count_nb_lines_for: 915 lines in 3.876 ms * buf_count_newlines_gen: 915 lines in 1.032 ms * itercount: 915 lines in 0.817 ms File: big_file.ndjson (size: 317.99 MB) * count_nb_lines_blocks: 389000 lines in 415.393 ms * count_nb_lines_mmap: 389000 lines in 185.461 ms * count_nb_lines_sum: 389000 lines in 485.370 ms * count_nb_lines_for: 389000 lines in 967.075 ms * buf_count_newlines_gen: 389000 lines in 213.458 ms * itercount: 389000 lines in 223.120 ms File: small_file__shorter.ndjson (size: 0.04 MB) * count_nb_lines_blocks: 915 lines in 0.183 ms * count_nb_lines_mmap: 915 lines in 0.185 ms * count_nb_lines_sum: 915 lines in 0.251 ms * count_nb_lines_for: 915 lines in 0.244 ms * buf_count_newlines_gen: 915 lines in 0.665 ms * itercount: 915 lines in 0.135 ms File: big_file__shorter.ndjson (size: 17.42 MB) * count_nb_lines_blocks: 389000 lines in 36.799 ms * count_nb_lines_mmap: 389000 lines in 44.801 ms * count_nb_lines_sum: 389000 lines in 59.068 ms * count_nb_lines_for: 389000 lines in 81.387 ms * buf_count_newlines_gen: 389000 lines in 15.620 ms * itercount: 389000 lines in 31.292 ms File: small_file__shorter_sim_size.ndjson (size: 1.21 MB) * count_nb_lines_blocks: 36457 lines in 1.920 ms * count_nb_lines_mmap: 36457 lines in 2.615 ms * count_nb_lines_sum: 36457 lines in 3.993 ms * count_nb_lines_for: 36457 lines in 6.011 ms * buf_count_newlines_gen: 36457 lines in 1.363 ms * itercount: 36457 lines in 2.147 ms File: big_file__shorter_sim_size.ndjson (size: 328.19 MB) * count_nb_lines_blocks: 9834248 lines in 517.920 ms * count_nb_lines_mmap: 9834248 lines in 691.637 ms * count_nb_lines_sum: 9834248 lines in 1109.669 ms * count_nb_lines_for: 9834248 lines in 1683.859 ms * buf_count_newlines_gen: 9834248 lines in 318.939 ms * itercount: 9834248 lines in 628.760 ms 

If one wants to get the line count cheaply in Python in Linux, I recommend this method:

import os print os.popen("wc -l file_path").readline().split()[0] 

file_path can be both abstract file path or relative path. Hope this may help.

This is a meta-comment on some of the other answers.

1. The line-reading and buffered \n-counting techniques won't return the same answer for every file, because some text files have no newline at the end of the last line. You can work around this by checking the last byte of the last nonempty buffer and adding 1 if it's not b'\n'.

2. In Python 3, opening the file in text mode and in binary mode can yield different results, because text mode by default recognizes CR, LF, and CRLF as line endings (converting them all to '\n'), while in binary mode only LF and CRLF will be counted if you count b'\n'. This applies whether you read by lines or into a fixed-size buffer. The classic Mac OS used CR as a line ending; I don't know how common those files are these days.

3. The buffer-reading approach uses a bounded amount of RAM independent of file size, while the line-reading approach could read the entire file into RAM at once in the worst case (especially if the file uses CR line endings). In the worst case it may use substantially more RAM than the file size, because of overhead from dynamic resizing of the line buffer and (if you opened in text mode) Unicode decoding and storage.

4. You can improve the memory usage, and probably the speed, of the buffered approach by pre-allocating a bytearray and using readinto instead of read. One of the existing answers (with few votes) does this, but it's buggy (it double-counts some bytes).

5. The top buffer-reading answer uses a large buffer (1 MiB). Using a smaller buffer can actually be faster because of OS readahead. If you read 32K or 64K at a time, the OS will probably start reading the next 32K/64K into the cache before you ask for it, and each trip to the kernel will return almost immediately. If you read 1 MiB at a time, the OS is unlikely to speculatively read a whole megabyte. It may preread a smaller amount but you will still spend a significant amount of time sitting in the kernel waiting for the disk to return the rest of the data.

There are a lot of answers already, but unfortunately most of them are just tiny economies on a barely optimizable problem...

I worked on several projects where line count was the core function of the software, and working as fast as possible with a huge number of files was of paramount importance.

The main bottleneck with line count is I/O access, as you need to read each line in order to detect the line return character, there is simply no way around. The second potential bottleneck is memory management: the more you load at once, the faster you can process, but this bottleneck is negligible compared to the first.

Hence, there are three major ways to reduce the processing time of a line count function, apart from tiny optimizations such as disabling GC collection and other micro-managing tricks:

1. Hardware solution: the major and most obvious way is non-programmatic: buy a very fast SSD/flash hard drive. By far, this is how you can get the biggest speed boosts.

2. Data preprocessing and lines parallelization: if you generate or can modify how the files you process are generated, or if it's acceptable that you can preprocess them. First convert the line return to Unix style (\n) as this will save 1 character compared to Windows (not a big save, but it's an easy gain), and secondly and most importantly, you can potentially write lines of fixed length. If you need variable length, you can pad smaller lines if the length variability is not that big. This way, you can calculate instantly the number of lines from the total file size, which is much faster to access. Also, by having fixed length lines, not only can you generally pre-allocate memory which will speed up processing, but also you can process lines in parallel! Of course, parallelization works better with a flash/SSD disk that has much faster random access I/O than HDDs.. Often, the best solution to a problem is to preprocess it so that it better fits your end purpose.

3. Disks parallelization + hardware solution: if you can buy multiple hard disks (and if possible SSD flash disks), then you can even go beyond the speed of one disk by leveraging parallelization, by storing your files in a balanced way (easiest is to balance by total size) among disks, and then read in parallel from all those disks. Then, you can expect to get a multiplier boost in proportion with the number of disks you have. If buying multiple disks is not an option for you, then parallelization likely won't help (except if your disk has multiple reading headers like some professional-grade disks, but even then the disk's internal cache memory and PCB circuitry will likely be a bottleneck and prevent you from fully using all heads in parallel, plus you have to devise a specific code for this hard drive you'll use because you need to know the exact cluster mapping so that you store your files on clusters under different heads, and so that you can read them with different heads after). Indeed, it's commonly known that sequential reading is almost always faster than random reading, and parallelization on a single disk will have a performance more similar to random reading than sequential reading (you can test your hard drive speed in both aspects using CrystalDiskMark for example).

If none of those are an option, then you can only rely on micromanaging tricks to improve by a few percents the speed of your line counting function, but don't expect anything really significant. Rather, you can expect the time you'll spend tweaking will be disproportionate compared to the returns in speed improvement you'll see.

print open('file.txt', 'r').read().count("\n") + 1 

Using Numba

We can use Numba to JIT (Just in time) compile our function to machine code. def numbacountparallel(fname) runs 2.8x faster than def file_len(fname) from the question.

Notes:

The OS had already cached the file to memory before the benchmarks were run as I don't see much disk activity on my PC. The time would be much slower when reading the file for the first time making the time advantage of using Numba insignificant.

The JIT compilation takes extra time the first time the function is called.

This would be useful if we were doing more than just counting lines.

Cython is another option.

Conclusion

As counting lines will be I/O bound, use the def file_len(fname) from the question unless you want to do more than just count lines.

import timeit from numba import jit, prange import numpy as np from itertools import (takewhile,repeat) FILE = '../data/us_confirmed.csv' # 40.6MB, 371755 line file CR = ord('\n') # Copied from the question above. Used as a benchmark def file_len(fname): with open(fname) as f: for i, l in enumerate(f): pass return i + 1 # Copied from another answer. Used as a benchmark def rawincount(filename): f = open(filename, 'rb') bufgen = takewhile(lambda x: x, (f.read(1024*1024*10) for _ in repeat(None))) return sum( buf.count(b'\n') for buf in bufgen ) # Single thread @jit(nopython=True) def numbacountsingle_chunk(bs): c = 0 for i in range(len(bs)): if bs[i] == CR: c += 1 return c def numbacountsingle(filename): f = open(filename, "rb") total = 0 while True: chunk = f.read(1024*1024*10) lines = numbacountsingle_chunk(chunk) total += lines if not chunk: break return total # Multi thread @jit(nopython=True, parallel=True) def numbacountparallel_chunk(bs): c = 0 for i in prange(len(bs)): if bs[i] == CR: c += 1 return c def numbacountparallel(filename): f = open(filename, "rb") total = 0 while True: chunk = f.read(1024*1024*10) lines = numbacountparallel_chunk(np.frombuffer(chunk, dtype=np.uint8)) total += lines if not chunk: break return total print('numbacountparallel') print(numbacountparallel(FILE)) # This allows Numba to compile and cache the function without adding to the time. print(timeit.Timer(lambda: numbacountparallel(FILE)).timeit(number=100)) print('\nnumbacountsingle') print(numbacountsingle(FILE)) print(timeit.Timer(lambda: numbacountsingle(FILE)).timeit(number=100)) print('\nfile_len') print(file_len(FILE)) print(timeit.Timer(lambda: rawincount(FILE)).timeit(number=100)) print('\nrawincount') print(rawincount(FILE)) print(timeit.Timer(lambda: rawincount(FILE)).timeit(number=100)) 

Time in seconds for 100 calls to each function

numbacountparallel 371755 2.8007332000000003 numbacountsingle 371755 3.1508585999999994 file_len 371755 6.7945494 rawincount 371755 6.815438 

Simple methods:

1. Method 1

   >>> f = len(open("myfile.txt").readlines()) >>> f 

   Output:

   430 

2. Method 2

   >>> f = open("myfile.txt").read().count('\n') >>> f 

   Output:

   430 

3. Method 3

   num_lines = len(list(open('myfile.txt'))) 

def count_text_file_lines(path): with open(path, 'rt') as file: line_count = sum(1 for _line in file) return line_count 

An alternative for big files is using xreadlines():

count = 0 for line in open(thefilepath).xreadlines( ): count += 1 

For Python 3 please see: What substitutes xreadlines() in Python 3?

The result of opening a file is an iterator, which can be converted to a sequence, which has a length:

with open(filename) as f: return len(list(f)) 

This is more concise than your explicit loop, and avoids the enumerate.

This could work:

import fileinput import sys counter = 0 for line in fileinput.input([sys.argv[1]]): counter += 1 fileinput.close() print counter