How to use to find files recursively?

Question

I would like to list all files recursively in a directory. I currently have a directory structure like this:

src/main.c
src/dir/file1.c
src/another-dir/file2.c
src/another-dir/nested/files/file3.c

I've tried to do the following:

from glob import glob glob(os.path.join('src','*.c'))

But this will only get be files directly in the src subfolder, e.g. I get main.c but I will not get file1.c, file2.c etc.

from glob import glob glob(os.path.join('src','*.c')) glob(os.path.join('src','*','*.c')) glob(os.path.join('src','*','*','*.c')) glob(os.path.join('src','*','*','*','*.c'))

But this is obviously limited and clunky, how can I do this properly?

doesn't glob('src/**/*.c') work in this case?

Likith Reddy
– Likith Reddy

2021-09-13 08:06:31 +00:00
Commented Sep 13, 2021 at 8:06 — Likith Reddy
– Likith Reddy, Commented Sep 13, 2021 at 8:06

Johan Dahlin · Accepted Answer · 2023-05-31 11:22:41Z

There are a couple of ways:

pathlib.Path().rglob()

Use pathlib.Path().rglob() from the pathlib module, which was introduced in Python 3.5.

from pathlib import Path for path in Path('src').rglob('*.c'): print(path.name)

glob.glob()

If you don't want to use pathlib, use glob.glob():

from glob import glob for filename in glob('src/**/*.c', recursive=True): print(filename)

For cases where matching files beginning with a dot (.); like files in the current directory or hidden files on Unix based system, use the os.walk() solution below.

os.walk()

For older Python versions, use os.walk() to recursively walk a directory and fnmatch.filter() to match against a simple expression:

import fnmatch import os matches = [] for root, dirnames, filenames in os.walk('src'): for filename in fnmatch.filter(filenames, '*.c'): matches.append(os.path.join(root, filename))

This version should also be faster depending on how many files you have, as the pathlib module has a bit of overhead over os.walk().

For Python older than 2.2 there is os.path.walk() which is a little more fiddly to use than os.walk()
@gnibbler I know that is an old comment, but my comment is just to let people know that os.path.walk() is deprecated and has been removed in Python 3.
@DevC that might work in the specific case asked in this question, but it's easy to imagine someone that wants to do use it with queries such as 'a*.c' etc, so I think it's worth keeping the current somewhat slow answer.
For what it's worth, in my case finding 10,000+ files with glob was much slower than with os.walk, so I went with the latter solution for that reason.
For python 3.4, pathlib.Path('src').glob('**/*.c') should work.

Pedro Lobito · Accepted Answer · 2023-09-22 10:01:27Z

For python >= 3.5 you can use **, recursive=True, i.e.:

import glob for f in glob.glob('/path/**/*.c', recursive=True): print(f)

If recursive is True (default False), the pattern ** will match any files and zero or more directories and subdirectories. If the pattern is followed by an os.sep, only directories and subdirectories match.

Python 3 Demo

This works better than pathlib.Path('./path/').glob('*/') because it also so in folder with size of 0
recursive is also set to False by default in Python 3.8.*.

Johan Dahlin · Accepted Answer · 2011-11-23 11:00:57Z

Similar to other solutions, but using fnmatch.fnmatch instead of glob, since os.walk already listed the filenames:

import os, fnmatch def find_files(directory, pattern): for root, dirs, files in os.walk(directory): for basename in files: if fnmatch.fnmatch(basename, pattern): filename = os.path.join(root, basename) yield filename for filename in find_files('src', '*.c'): print 'Found C source:', filename

Also, using a generator alows you to process each file as it is found, instead of finding all the files and then processing them.

Matt · Accepted Answer · 2015-07-14 19:40:54Z

93

I've modified the glob module to support ** for recursive globbing, e.g:

>>> import glob2 >>> all_header_files = glob2.glob('src/**/*.c')

https://github.com/miracle2k/python-glob2/

Useful when you want to provide your users with the ability to use the ** syntax, and thus os.walk() alone is not good enough.

edited Jul 14, 2015 at 19:40

Matt

75.4k26 gold badges156 silver badges181 bronze badges

answered Jun 26, 2011 at 14:14

miracle2k

32.6k21 gold badges68 silver badges67 bronze badges

4 Comments

ArtOfWarfare Over a year ago

Can we make this stop after it finds the first match? Maybe make it possible to use it as a generator rather than having it return a list of every possible result? Also, is this a DFS or a BFS? I'd much prefer a BFS, I think, so that files which are near the root are found first. +1 for making this module and providing it on GitHub/pip.

ArtOfWarfare Over a year ago

The ** syntax was added to the official glob module in Python 3.5.

cs95 Over a year ago

@ArtOfWarfare Alright, fine. This is still useful for < 3.5.

winklerrr Over a year ago

To activate recursive globbing using ** with the official glob module, do: glob(path, recursive=True)

taleinat · Accepted Answer · 2015-08-19 09:13:25Z

79

Starting with Python 3.4, one can use the glob() method of one of the Path classes in the new pathlib module, which supports ** wildcards. For example:

from pathlib import Path for file_path in Path('src').glob('**/*.c'): print(file_path) # do whatever you need with these files

Update: Starting with Python 3.5, the same syntax is also supported by glob.glob().

edited Aug 19, 2015 at 9:13

answered Nov 11, 2014 at 16:08

taleinat

8,7311 gold badge36 silver badges47 bronze badges

3 Comments

taleinat Over a year ago

Indeed, and it will be in Python 3.5. It was supposed to already be so in Python 3.4, but was omitted by mistake.

taleinat Over a year ago

This syntax is now supported by glob.glob() as of Python 3.5.

pjgranahan Over a year ago

Note that you can also use pathlib.PurePath.relative_to in combination to get relative paths. See my answer here for more context.

Sebastian Mach · Accepted Answer · 2014-12-09 10:02:02Z

import os import fnmatch def recursive_glob(treeroot, pattern): results = [] for base, dirs, files in os.walk(treeroot): goodfiles = fnmatch.filter(files, pattern) results.extend(os.path.join(base, f) for f in goodfiles) return results

fnmatch gives you exactly the same patterns as glob, so this is really an excellent replacement for glob.glob with very close semantics. An iterative version (e.g. a generator), IOW a replacement for glob.iglob, is a trivial adaptation (just yield the intermediate results as you go, instead of extending a single results list to return at the end).

What do you think about using recursive_glob(pattern, treeroot='.') as I suggested in my edit? This way, it can be called for example as recursive_glob('*.txt') and intuitively match the syntax of glob.
@ChrisRedford, I see it as a pretty minor issue either way. As it stands now, it matches the "files then pattern" argument order of fnmatch.filter, which is roughly as useful as the possibility of matching single-argument glob.glob.

Geoff Reedy · Accepted Answer · 2010-02-02 18:24:48Z

You'll want to use os.walk to collect filenames that match your criteria. For example:

import os cfiles = [] for root, dirs, files in os.walk('src'): for file in files: if file.endswith('.c'): cfiles.append(os.path.join(root, file))

blueyed · Accepted Answer · 2014-09-10 17:38:57Z

Here's a solution with nested list comprehensions, os.walk and simple suffix matching instead of glob:

import os cfiles = [os.path.join(root, filename) for root, dirnames, filenames in os.walk('src') for filename in filenames if filename.endswith('.c')]

It can be compressed to a one-liner:

import os;cfiles=[os.path.join(r,f) for r,d,fs in os.walk('src') for f in fs if f.endswith('.c')]

or generalized as a function:

import os def recursive_glob(rootdir='.', suffix=''): return [os.path.join(looproot, filename) for looproot, _, filenames in os.walk(rootdir) for filename in filenames if filename.endswith(suffix)] cfiles = recursive_glob('src', '.c')

If you do need full glob style patterns, you can follow Alex's and Bruno's example and use fnmatch:

import fnmatch import os def recursive_glob(rootdir='.', pattern='*'): return [os.path.join(looproot, filename) for looproot, _, filenames in os.walk(rootdir) for filename in filenames if fnmatch.fnmatch(filename, pattern)] cfiles = recursive_glob('src', '*.c')

pylang · Accepted Answer · 2019-10-22 12:06:31Z

Consider pathlib.rglob().

This is like calling Path.glob() with "**/" added in front of the given relative pattern:

import pathlib for p in pathlib.Path("src").rglob("*.c"): print(p)

See also @taleinat's related post here and a similar post elsewhere.

Milovan Tomašević · Accepted Answer · 2020-08-03 05:10:01Z

import os, glob for each in glob.glob('path/**/*.c', recursive=True): print(f'Name with path: {each} \nName without path: {os.path.basename(each)}')

glob.glob('*.c') :matches all files ending in .c in current directory
glob.glob('*/*.c') :same as 1
glob.glob('**/*.c') :matches all files ending in .c in the immediate subdirectories only, but not in the current directory
glob.glob('*.c',recursive=True) :same as 1
glob.glob('*/*.c',recursive=True) :same as 3
glob.glob('**/*.c',recursive=True) :matches all files ending in .c in the current directory and in all subdirectories

Daniel · Accepted Answer · 2020-09-07 15:49:11Z

In case this may interest anyone, I've profiled the top three proposed methods. I have about ~500K files in the globbed folder (in total), and 2K files that match the desired pattern.

here's the (very basic) code

import glob import json import fnmatch import os from pathlib import Path from time import time def find_files_iglob(): return glob.iglob("./data/**/data.json", recursive=True) def find_files_oswalk(): for root, dirnames, filenames in os.walk('data'): for filename in fnmatch.filter(filenames, 'data.json'): yield os.path.join(root, filename) def find_files_rglob(): return Path('data').rglob('data.json') t0 = time() for f in find_files_oswalk(): pass t1 = time() for f in find_files_rglob(): pass t2 = time() for f in find_files_iglob(): pass t3 = time() print(t1-t0, t2-t1, t3-t2)

And the results I got were:
os_walk: ~3.6sec
rglob ~14.5sec
iglob: ~16.9sec

The platform: Ubuntu 16.04, x86_64 (core i7),

Thank you for the benchmark. I ran this on 10k files with Python 3.9.12 and the rankings are the same as in this benchmark (os.walk is fastest), although the difference is not as extreme as it is in your example.

mmmmmm · Accepted Answer · 2013-01-05 10:59:28Z

Recently I had to recover my pictures with the extension .jpg. I ran photorec and recovered 4579 directories 2.2 million files within, having tremendous variety of extensions.With the script below I was able to select 50133 files havin .jpg extension within minutes:

#!/usr/binenv python2.7 import glob import shutil import os src_dir = "/home/mustafa/Masaüstü/yedek" dst_dir = "/home/mustafa/Genel/media" for mediafile in glob.iglob(os.path.join(src_dir, "*", "*.jpg")): #"*" is for subdirectory shutil.copy(mediafile, dst_dir)

daveoncode · Accepted Answer · 2012-07-28 22:09:23Z

based on other answers this is my current working implementation, which retrieves nested xml files in a root directory:

files = [] for root, dirnames, filenames in os.walk(myDir): files.extend(glob.glob(root + "/*.xml"))

I'm really having fun with python :)

Sami · Accepted Answer · 2019-11-21 08:55:15Z

For python 3.5 and later

import glob #file_names_array = glob.glob('path/*.c', recursive=True) #above works for files directly at path/ as guided by NeStack #updated version file_names_array = glob.glob('path/**/*.c', recursive=True)

further you might need

for full_path_in_src in file_names_array: print (full_path_in_src ) # be like 'abc/xyz.c' #Full system path of this would be like => 'path till src/abc/xyz.c'

Your first line of code doesn't work for looking into subdirectories. But if you just expand it by /** it works for me, like that: file_names_array = glob.glob('src/**/*.c', recursive=True)

Andrew Alcock · Accepted Answer · 2012-05-15 09:14:13Z

Johan and Bruno provide excellent solutions on the minimal requirement as stated. I have just released Formic which implements Ant FileSet and Globs which can handle this and more complicated scenarios. An implementation of your requirement is:

import formic fileset = formic.FileSet(include="/src/**/*.c") for file_name in fileset.qualified_files(): print file_name

Formic appears to be abandoned?! And it does not support Python 3 (bitbucket.org/aviser/formic/issue/12/support-python-3)

chris-piekarski · Accepted Answer · 2011-12-28 18:07:03Z

Another way to do it using just the glob module. Just seed the rglob method with a starting base directory and a pattern to match and it will return a list of matching file names.

import glob import os def _getDirs(base): return [x for x in glob.iglob(os.path.join( base, '*')) if os.path.isdir(x) ] def rglob(base, pattern): list = [] list.extend(glob.glob(os.path.join(base,pattern))) dirs = _getDirs(base) if len(dirs): for d in dirs: list.extend(rglob(os.path.join(base,d), pattern)) return list

xtofl · Accepted Answer · 2013-06-24 10:41:03Z

Or with a list comprehension:

 >>> base = r"c:\User\xtofl" >>> binfiles = [ os.path.join(base,f) for base, _, files in os.walk(root) for f in files if f.endswith(".jpg") ]

gerrit · Accepted Answer · 2020-10-08 13:49:14Z

If the files are on a remote file system or inside an archive, you can use an implementation of the fsspec AbstractFileSystem class. For example, to list all the files in a zipfile:

from fsspec.implementations.zip import ZipFileSystem fs = ZipFileSystem("/tmp/test.zip") fs.glob("/**") # equivalent: fs.find("/")

or to list all the files in a publicly available S3 bucket:

from s3fs import S3FileSystem fs_s3 = S3FileSystem(anon=True) fs_s3.glob("noaa-goes16/ABI-L1b-RadF/2020/045/**") # or use fs_s3.find

you can also use it for a local filesystem, which may be interesting if your implementation should be filesystem-agnostic:

from fsspec.implementations.local import LocalFileSystem fs = LocalFileSystem() fs.glob("/tmp/test/**")

Other implementations include Google Cloud, Github, SFTP/SSH, Dropbox, and Azure. For details, see the fsspec API documentation.

Shaurya Gupta · Accepted Answer · 2013-07-27 18:12:29Z

Just made this.. it will print files and directory in hierarchical way

But I didn't used fnmatch or walk

#!/usr/bin/python import os,glob,sys def dirlist(path, c = 1): for i in glob.glob(os.path.join(path, "*")): if os.path.isfile(i): filepath, filename = os.path.split(i) print '----' *c + filename elif os.path.isdir(i): dirname = os.path.basename(i) print '----' *c + dirname c+=1 dirlist(i,c) c-=1 path = os.path.normpath(sys.argv[1]) print(os.path.basename(path)) dirlist(path)

hipertracker · Accepted Answer · 2013-08-02 16:01:34Z

That one uses fnmatch or regular expression:

import fnmatch, os def filepaths(directory, pattern): for root, dirs, files in os.walk(directory): for basename in files: try: matched = pattern.match(basename) except AttributeError: matched = fnmatch.fnmatch(basename, pattern) if matched: yield os.path.join(root, basename) # usage if __name__ == '__main__': from pprint import pprint as pp import re path = r'/Users/hipertracker/app/myapp' pp([x for x in filepaths(path, re.compile(r'.*\.py$'))]) pp([x for x in filepaths(path, '*.py')])

f0xdx · Accepted Answer · 2015-12-05 17:42:56Z

In addition to the suggested answers, you can do this with some lazy generation and list comprehension magic:

import os, glob, itertools results = itertools.chain.from_iterable(glob.iglob(os.path.join(root,'*.c')) for root, dirs, files in os.walk('src')) for f in results: print(f)

Besides fitting in one line and avoiding unnecessary lists in memory, this also has the nice side effect, that you can use it in a way similar to the ** operator, e.g., you could use os.path.join(root, 'some/path/*.c') in order to get all .c files in all sub directories of src that have this structure.

Sanjay Bharwani · Accepted Answer · 2020-04-03 10:10:40Z

This is a working code on Python 2.7. As part of my devops work, I was required to write a script which would move the config files marked with live-appName.properties to appName.properties. There could be other extension files as well like live-appName.xml.

Below is a working code for this, which finds the files in the given directories (nested level) and then renames (moves) it to the required filename

def flipProperties(searchDir): print "Flipping properties to point to live DB" for root, dirnames, filenames in os.walk(searchDir): for filename in fnmatch.filter(filenames, 'live-*.*'): targetFileName = os.path.join(root, filename.split("live-")[1]) print "File "+ os.path.join(root, filename) + "will be moved to " + targetFileName shutil.move(os.path.join(root, filename), targetFileName)

This function is called from a main script

flipProperties(searchDir)

Hope this helps someone struggling with similar issues.

flowfree · Accepted Answer · 2013-06-03 01:29:07Z

Simplified version of Johan Dahlin's answer, without fnmatch.

import os matches = [] for root, dirnames, filenames in os.walk('src'): matches += [os.path.join(root, f) for f in filenames if f[-2:] == '.c']

sackpower · Accepted Answer · 2014-08-18 17:50:33Z

Here is my solution using list comprehension to search for multiple file extensions recursively in a directory and all subdirectories:

import os, glob def _globrec(path, *exts): """ Glob recursively a directory and all subdirectories for multiple file extensions Note: Glob is case-insensitive, i. e. for '\*.jpg' you will get files ending with .jpg and .JPG Parameters ---------- path : str A directory name exts : tuple File extensions to glob for Returns ------- files : list list of files matching extensions in exts in path and subfolders """ dirs = [a[0] for a in os.walk(path)] f_filter = [d+e for d in dirs for e in exts] return [f for files in [glob.iglob(files) for files in f_filter] for f in files] my_pictures = _globrec(r'C:\Temp', '\*.jpg','\*.bmp','\*.png','\*.gif') for f in my_pictures: print f

yoyo · Accepted Answer · 2015-06-30 15:39:36Z

Here is a solution that will match the pattern against the full path and not just the base filename.

It uses fnmatch.translate to convert a glob-style pattern into a regular expression, which is then matched against the full path of each file found while walking the directory.

re.IGNORECASE is optional, but desirable on Windows since the file system itself is not case-sensitive. (I didn't bother compiling the regex because docs indicate it should be cached internally.)

import fnmatch import os import re def findfiles(dir, pattern): patternregex = fnmatch.translate(pattern) for root, dirs, files in os.walk(dir): for basename in files: filename = os.path.join(root, basename) if re.search(patternregex, filename, re.IGNORECASE): yield filename

a breadth-first search would be more efficient: split the pattern by slashes, call os.listdir on each level, and only descend when the dirname matches. then the glob pattern ** would allow recursive descending to find the next path segment. im pretty sure this already exists somewhere...

user1903204 · Accepted Answer · 2014-01-27 19:03:53Z

import sys, os, glob dir_list = ["c:\\books\\heap"] while len(dir_list) > 0: cur_dir = dir_list[0] del dir_list[0] list_of_files = glob.glob(cur_dir+'\\*') for book in list_of_files: if os.path.isfile(book): print(book) else: dir_list.append(book)

ihightower · Accepted Answer · 2014-11-15 13:39:49Z

I modified the top answer in this posting.. and recently created this script which will loop through all files in a given directory (searchdir) and the sub-directories under it... and prints filename, rootdir, modified/creation date, and size.

Hope this helps someone... and they can walk the directory and get fileinfo.

import time import fnmatch import os def fileinfo(file): filename = os.path.basename(file) rootdir = os.path.dirname(file) lastmod = time.ctime(os.path.getmtime(file)) creation = time.ctime(os.path.getctime(file)) filesize = os.path.getsize(file) print "%s**\t%s\t%s\t%s\t%s" % (rootdir, filename, lastmod, creation, filesize) searchdir = r'D:\Your\Directory\Root' matches = [] for root, dirnames, filenames in os.walk(searchdir): ## for filename in fnmatch.filter(filenames, '*.c'): for filename in filenames: ## matches.append(os.path.join(root, filename)) ##print matches fileinfo(os.path.join(root, filename))

Roman · Accepted Answer · 2017-06-23 10:20:37Z

-1

I needed a solution for python 2.x that works fast on large directories.
I endet up with this:

import subprocess foundfiles= subprocess.check_output("ls src/*.c src/**/*.c", shell=True) for foundfile in foundfiles.splitlines(): print foundfile

Note that you might need some exception handling in case ls doesn't find any matching file.

answered Jun 23, 2017 at 10:20

Roman

8571 gold badge10 silver badges18 bronze badges

3 Comments

Roman Over a year ago

I just realized that ls src/**/*.c only works if globstar option is enabled (shopt -s globstar) - see this answer for details.

tripleee Over a year ago

A subprocess is never a good solution if you want to go fast, and ls in scripts is definitely something to avoid.

Roman Over a year ago

Ok, I didn't know about this. It works for me - and takes less than a second (instead of more than 30 seconds...)

Collectives™ on Stack Overflow

How to use to find files recursively?

28 Answers 28

14 Comments

3 Comments

Comments

4 Comments

3 Comments

2 Comments

Comments

Comments

Comments

Comments

1 Comment

Comments

Comments

1 Comment

1 Comment

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments

1 Comment

Comments

Comments

3 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

28 Answers 28

14 Comments

3 Comments

Comments

4 Comments

3 Comments

2 Comments

Comments

Comments

Comments

Comments

1 Comment

Comments

Comments

1 Comment

1 Comment

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments

1 Comment

Comments

Comments

3 Comments

Linked

Related