Skip other paths under target if target declares sources

[pyrtsa]

Motivation:

Some large Swift packages like Firebase make Xcode and SPM very slow at resolving package graph. It seems to be largely due to (their dependencies) containing large amounts of unrelated files which SPM checks each anyway.

Quoting my post on Swift Forums,

Tracing swift-package with the File Activity instruments further reveals that SPM is indeed hitting a lot of files under Firebase and its dependencies. The biggest of those libraries consist of over 20,000 files, most of them (excluded code, tests and test data) irrelevant to the Swift package. TargetSourcesBuilder starts by finding all files reachable under the target's directory. But that becomes a problem if a package is making deliberate use of .target(... path: exclude: sources: ).

Modifications:

Don't include every file under targetPath for those targets which declare their own sources explicitly. Instead, only traverse the file and directory paths included in declaredSources.

Result:

If this change is right to do (it breaks some potentially obsolete unit tests!), it could provide up to a further 4.5x speedup, evaluating show-dependencies for me in just 2.324s/0.767s/0.561s.

This change currently breaks some seemingly obsolete unit tests, which assert a package structure foreign to today's conventions. Should this PR move forward, I need assistance from maintainers resolving what to do with those tests and which new test cases to add.

[tomerd]

thanks so much for this PR @pyrtsa

@neonich @abertelrud @MaxDesiatov ptal

[MaxDesiatov]

This makes sense to me overall, but I'm not sure if it has any disruptive impact since I have little knowledge of the PackageLoading module. I'd assume failing tests need to be audited to understand the full impact and make a decision based on that.

[neonichu]

This change currently breaks some seemingly obsolete unit tests, which assert a package structure foreign to today's conventions. Should this PR move forward, I need assistance from maintainers resolving what to do with those tests and which new test cases to add.

We should tie the behavior to the tools version, then the tests and any packages relying on the old behavior can continue to function. Please use .vNext initially until we know the concrete version we'll land this in.

[abertelrud]

Gave this a try locally, and the tests that break are:

• testXcodeSpecificResourcesAreNotIncludedByDefault
• testUnhandledResources
• testDocCFilesDoNotCauseWarningOutsideXCBuild

This is because the lists of other and ignored files are no longer being populated. I think TargetSourceBuilder would need to be reworked a bit to support this. I am not sure whether anything needs to be adjusted in the manifest: there's a bit of an ambiguity now because sources means both "source code files" and also "here are where the target files are, resources as well as source files" depending on whether directories or plain files are specified.

So this change can't be included as-is, but it's something to keep in mind when reworking TargetSourceBuilder.

[neonichu]

Making this a draft since it won't be possible to proceed with it as-is.