Get files without looking at a specific directory or looking two levels down

I am trying to enumerate file path under a directory. This directory has lots of sub directories that even though I am looking for a specific file, the enumeration is taking a lot of time. So trying to optimize it.

Here is the structure:

MAIN -> This is where I start looking

 string path = "something main here"; Directory.EnumerateFiles(path , "grouptest.log", SearchOption.AllDirectories); 

Now within the main directory, I know where all grouptest.log exists.

main -> sampleDir1 -> textDir1 -> StoryDir -> grouptest.log -> textDir2 -> StoryDir -> grouptest.log sampleDir2 -> textDir1 -> StoryDir -> grouptest.log 

It always exists two levels down. Under Main\SampleDirX\textDirX\grouptest.log. And it doesnt exist under StoryDir at all, so we dont have to look here.

I didnt find any way to exclude a pattern from the EnumerateFiles() or GetFiles() and there is no way to use regex within the search pattern.

Approaches tried:

1. Direct enumeration of grouptest.log.

2. Get subdirectories of main with

   Directory.GetDirectories(path, "*", SearchOption.AllDirectories).Where(dir => !Path.GetFullPath(dir).Contains(@"\StoryDir")).ToList(); 

And then read for each subDir

Directory.EnumerateFiles(subDir , "grouptest.log", SearchOption.TopDirectoryOnly); 

Still taking a long time.

How can I specify I want to exclude the directory to read before I enumerate files?

EnumerateFiles and running where clause on the file paths is still taking long time. I want to be able to exclude the StoryDir paths in the first place or just look two levels down.