[rfc] introduce artifact bundle wrapper

[tomerd]

motivation: allow download of remote zipfile that could be wrapped as an artifact bundle

changes:

• looks for new binary artifact type "artifactbundlewrapper"
• download and extract the zipfile specified in the url of the wrapper and treat it as an artifact bundle

TODO:

• cleanup code
• add test

rdar://106353640

[tomerd]

poc to discuss with @FranzBusch @MaxDesiatov @neonichu

[tomerd]

actual change starts here, rest os formatting :p

[tomerd]

end of actual change

[tomerd]

the new wrapper file format will be a JSON containing these fields

[tomerd]

may need to make it more complicated with array of variants so you can have path -> supported triple all in one zip

[FranzBusch]

If I understand this correctly we would have a zip file that just contains a JSON file. That JSON file contains the following

{ "schemaVersion": "1.0", "variants": [ { “url”: “<url/to/artifact.zip>”, “checksum”: “<checksum of the zip>”, "supportedTriples": [ "<triple1>", ... ] }, ] } 

Is my assumption correct here?

[tomerd]

almost

{ "schemaVersion": "1.0", "variants": [ { "url": <url/to/artifact.zip>, "checksum": <checksum of the zip>, "supportedTriples": [<triple>, <triple>], "path": <realtive path inside the zip file where the tools are located> }, ] } 

the idea here is that this would work similar to the bundle index file. the initial request get a JSON file (not zip) with the format like ^^, then SwiftPM will find the best variant based on triple and download the zip file and set it up using the path

[neonichu]

If I understand correctly, the purpose of this is to reference an existing ZIP of some kind that was produced without knowledge of SwiftPM. This implies to me that the structure can be arbitrary and we need a way to declare where the actual binary is located. Also implies we need to support the case where each binary has its own ZIP as well as the case where different platform binaries are in one ZIP.

[FranzBusch]

If I understand correctly, the purpose of this is to reference an existing ZIP of some kind that was produced without knowledge of SwiftPM. This implies to me that the structure can be arbitrary and we need a way to declare where the actual binary is located. Also implies we need to support the case where each binary has its own ZIP as well as the case where different platform binaries are in one ZIP.

Agreed that this is important. Taking protoc as an example they have zips uploaded to the GH releases. The layout of a zip looks like this

protoc-22.2-osx-aarch_64.zip/ ├─ bin/ ├─ protoc ├─ include/ 

[MaxDesiatov]

+1 on separate archives for each triple being common enough. Maybe we need a new schema version to be able to express this.

[tomerd]

yes: #6298 (comment)

[neonichu]

How about putting the wrapper stuff into the manifest instead? The idea of the artifact bundle was that the package author and the producer of the artifacts can be separate, but in case of the wrapper, presumably everything belongs together so it seems weird to have a JSON file that essentially describes content that belongs in the package manifest.

[MaxDesiatov]

How about putting the wrapper stuff into the manifest instead? The idea of the artifact bundle was that the package author and the producer of the artifacts can be separate, but in case of the wrapper, presumably everything belongs together so it seems weird to have a JSON file that essentially describes content that belongs in the package manifest.

I thought about this too, although the amount of stuff you have to specify in the manifest is not insignificant, especially if you need to support multiple triples.

Additionally, isn't the end goal to have such wrappers published on the registry as separate packages? That would make it easier for a given package to know that it always interacts with a single registry that it (supposedly) trusts. If if a vulnerability is discovered in a wrapped binary, the registry could mark such wrapper is vulnerable so that all packages using it will know about it. I don't think that's possible if every package has to hardcode URLs for these binaries in its own manifest. From this perspective indirection could be an intentional feature, not an inconvenience.

Update: Disregard the paragraph above, I see what you mean. If a bundle wrapper has to have its own package manifest anyway to become a standalone package, a separate JSON provides even more indirection than I can see benefits from.

[neonichu]

Additionally, isn't the end goal to have such wrappers published on the registry as separate packages?

If that's the goal, I wonder if the reverse question applies, should there even be a package manifest for this or should it be possible to publish a repo with just the artifact wrapper in it?

There is definitely the use case of vending a bunch of things where the binary would just be one of them, but there's also the case where the binary is supposed to be self-contained and versioned. At that point, it does seem very hacky to have to modify to interconnected files.

[tomerd]

How about putting the wrapper stuff into the manifest instead? The idea of the artifact bundle was that the package author and the producer of the artifacts can be separate, but in case of the wrapper, presumably everything belongs together so it seems weird to have a JSON file that essentially describes content that belongs in the package manifest.

interesting idea. I can experiment with that that may look like

[MaxDesiatov]

There is definitely the use case of vending a bunch of things where the binary would just be one of them, but there's also the case where the binary is supposed to be self-contained and versioned. At that point, it does seem very hacky to have to modify to interconnected files.

I've also been thinking today that calling them "binaries" may be wrong. What if it's a Python or a shell script? I don't have a good name for it though, we already have executableTarget that means executable Swift binaries specifically.

I could easily imagine people hosting some resources (fonts, shaders, textures etc) on a CDN in such bundles if they don't want them committed to their git repositories.

[neonichu]

What if it's a Python or a shell script?

Hopefully, it's not 😅 I think we would need some way to declare that they are, otherwise things would just break if there's no installed Python or the PATH doesn't contain the specific one you want (as it is often the case in Xcode).

[FranzBusch]

I think having just a repo with the repo in it would be very interesting.
Could we then also point to local wrappers?

So in the case of protoc we would create a swift-protoc repo that has both the wrapper and a Package.swift in there?

[MaxDesiatov]

What if it's a Python or a shell script?

Hopefully, it's not 😅 I think we would need some way to declare that they are, otherwise things would just break if there's no installed Python or the PATH doesn't contain the specific one you want (as it is often the case in Xcode).

This still means shell scripts could work. Nothing prevents someone from sticking a URL to a shell script there instead of a binary and distributing it that way. SwiftPM wouldn't even know, unless we're willing to explicitly check for it somehow.

My point is that there's probably not much sense in making that distinction, it's either just an executable (a binary or a script), or arbitrary resource file altogether if we think that's acceptable too?

[neonichu]

This still means shell scripts could work. Nothing prevents someone from sticking a URL to a shell script there instead of a binary and distributing it that way. SwiftPM wouldn't even know, unless we're willing to explicitly check for it somehow.

Right, the name binaryTarget comes from the original feature where it could only be an actual binary. That was repurposed for artifact bundles at some point and probably missed that one could vend a shell script as well which might mostly work.

I wonder if we do want to check for it explicitly since even a shell script probably wouldn't work out of the box on Windows etc. Even if we don't, I think there's still a difference between something accidentally working and the API naming reflecting that it is explicitly supported.

[MaxDesiatov]

I wonder if we do want to check for it explicitly since even a shell script probably wouldn't work out of the box on Windows etc.

In that case Windows wouldn't be specified as a supported triple.

I think there's still a difference between something accidentally working and the API naming reflecting that it is explicitly supported.

I totally agree with that. I didn't mean to block this PR or argue about existing naming. Just wanted to raise a possibility of different use cases that we could think of in background, in case someone comes with a feature request asking whether they can include scripts in their artifact bundles. The feature seems general enough for us to support that, seemingly with little effort. And keeping these use cases in mind could impact future design considerations.

[tomerd]

how do we see the repository / package.swift idea work?

a repository with package.swift + checked in binaries with a bunch of unstructured directories that include the tool binaries? and package.swift gaining new syntax to declare local tools in a relative path to the package root?

[MaxDesiatov]

IDK if committing binaries to git is a good idea in general, wouldn't people prefer publishing binaries as attachments to GitHub releases? Or do we want to support both binaries in git and attachments to releases (i.e. arbitrary URLs with checksums) for the sake of completeness?

[tomerd]

right, so this is why we had the URLs initially, but then that creates more indirection and weaker guarantees around versioning

[FranzBusch]

So for the protoc I don't want to check in artifacts into git. We could already do this right now. I want to point at the release zips in the proto repo

[tomerd]

so what would be in the repository in this case? just a Package.swift? feels a bit weird and not clear to me the advantage over a JSON file?

[neonichu]

so what would be in the repository in this case? just a Package.swift? feels a bit weird and not clear to me the advantage over a JSON file?

That's how a binary-only package with remote binaries works currently, right? It can point to a remove artifact bundle and have no files beyond a Package.swift.

I think what would be weird to me is if we make those types of packages grow two files, so I think they should either contain a JSON file and we somehow build a package model from it automatically or we declare everything in the manifest.

[tomerd]

right, so what would the target here be? a binaryTarget with special attributes? or something new altogether?

[FranzBusch]

So in my mind the repo would look something like this

// Package.swift let package = Package( name: "swift-protoc", products: [ .executable( name: "protoc", targets: ["protoc"] ), ], targets: [ .binaryTarget( name: "protoc", path: "protoc.json" // We probably want to use a different extension here to identify these files properly ) ) // proto.json { "schemaVersion": "1.0", "variants": [ { "url": <url/to/artifact.zip>, "checksum": <checksum of the zip>, "supportedTriples": [<triple>, <triple>], "path": <realtive path inside the zip file where the tools are located> }, ] } 

At that point it is definitely debatable if we want to fold the information into the manifest itself but I don't have strong opinions about this.

[tomerd]

@neonichu ?

[tomerd]

some like this?

.binaryTarget( name: "protoc", url: <url/to/artifact.zip>, checksum: <checksum of the zip>, variants: [ .init( "supportedTriples": [<triple>, <triple>], "path": <realtive path inside the zip file where the tools are located> ) ] )