1

I need to delete a large amount of files from a list with their names (I need to delete 2450 files from a total of 10015 files). But the code I'm using, though it works, is too time-consuming to do the job. It's definitely not optimized to get the job done faster. Anyone have a better idea to deal with this problem?

os.chdir(directoryPath) for filename in os.listdir('D:\Python\Jupyter\IP_Project\DataBase'): for pattern in ['ISIC_0024396*', 'ISIC_0024630*', 'ISIC_0024672*', 'ISIC_0024700*', 'ISIC_0024771*', 'ISIC_0024834*', 'ISIC_0024869*', 'ISIC_0024918*', 'ISIC_0024962*', 'ISIC_0024998*', 'ISIC_0025005*', 'ISIC_0025040*', 'ISIC_0025046*', 'ISIC_0025064*', 'ISIC_0025073*', 'ISIC_0025112*', 'ISIC_0025152*', 'ISIC_0025168*', 'ISIC_0025170*', 'ISIC_0025193*', 'ISIC_0025208*', 'ISIC_0025231*', 'ISIC_0025297*', 'ISIC_0025322*', 'ISIC_0034319*', 'ISIC_0034320*']: if fnmatch.fnmatch(filename, pattern): os.remove(filename)

Note: I decreases the number of file names in the above code to better exemplify my idea. But as I said above, these are 2450 filenames.

Thanks for tips !

Improve this question
6
  • 3
    Perhaps use bash instead? And just run rm ISIC_0024396*, rm <etc> for each wildcard? This is if I'm understanding your question correctly and the format of your files Commented Dec 5, 2018 at 13:56
  • 1
    Why don't you simply handover the pattern to an external shell call? Should be way faster. Commented Dec 5, 2018 at 13:57
  • Really is this slow? It should be quite fast. If you're on a linux system, then use bash as TerryA says, or at least use os.system('rm <filename>'). os.remove is cross-platform, which may slow it down. Commented Dec 5, 2018 at 13:57
  • you could also speed things up by breaking once you've matched a pattern, as it stands you're testing every filename against every pattern so as a rough measure that would probably half the time taken to run. Commented Dec 5, 2018 at 14:37
  • @TerryA, The files are images of skin lesions. Total of 10015 images (.jpg), however there are repetitions of images by injury. Some lesions have more than one picture. And I need to have a dataset with just one image per lesion. In this way, it extracts a list with the name of all the images that are repeated and would like to erase them (images to be deleted have a total of 2545 files, with 7470 still remaining in the directory). My initial idea would be to automate the task a bit, because I might have to do it again. Would your idea be to delete all images directly? Even if it's 2545 files? Commented Dec 5, 2018 at 15:22

2 Answers 2

Reset to default
1

as stated in my comment above you are currently continuing to check for further matches after you find the first one, even though in this instance you won't and as the file has been removed it's not useful to, by breaking after a match:

os.chdir(directoryPath) for filename in os.listdir('D:\Python\Jupyter\IP_Project\DataBase'): for pattern in ['ISIC_0024396*', 'ISIC_0024630*', 'ISIC_0024672*', 'ISIC_0024700*', 'ISIC_0024771*', 'ISIC_0024834*', 'ISIC_0024869*', 'ISIC_0024918*', 'ISIC_0024962*', 'ISIC_0024998*', 'ISIC_0025005*', 'ISIC_0025040*', 'ISIC_0025046*', 'ISIC_0025064*', 'ISIC_0025073*', 'ISIC_0025112*', 'ISIC_0025152*', 'ISIC_0025168*', 'ISIC_0025170*', 'ISIC_0025193*', 'ISIC_0025208*', 'ISIC_0025231*', 'ISIC_0025297*', 'ISIC_0025322*', 'ISIC_0034319*', 'ISIC_0034320*']: if fnmatch.fnmatch(filename, pattern): os.remove(filename) break # break now that we matched and move on to next file

you in theory reduce the time taken to process all of them by roughly half (assuming an even distribution of names per pattern)

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

Thanks guys for the tips. For my problem specifically I can solve it in a much simpler way by removing the files directly as you indicated.

os.chdir(directory_path) for filename in repeated_images: os.remove(filename)
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.