Skip to content

Conversation

@slarse
Copy link
Contributor

@slarse slarse commented Nov 30, 2025

🧢 Changes

This PR does two (highly related) things.

  1. Adds Ubuntu 22.04 builds for x86_64 and ARM64 (side note: are the Linux ARM64 builds actually used??)
  2. Upgrades the core Linux dependencies, most notably libwebkit2gtk-4.1

I've been fiddling around with this in my fork to try stuff out in the actual runners, and produced the Ubuntu 22.04 and Ubuntu 24.04 builds found here.

Note: Don't mind the fact that the artifacts are 500 MB each, I was veeery lazy and just added the entire target/release/bundle directory as an artifact. In other words, the artifacts contain way more stuff than they need to.

The Ubuntu 24.04 AppImage appears to be working very well on my Arch Linux system under Wayland.

I was hoping the Ubuntu 22.04 AppImage would also work, but I'm having some very strange linking issues with it when running on my Arch machine (see below). So, I do believe we need to build for both older and newer systems, independently.

Possible linker error when running Ubuntu 22.04 AppImage on Arch Linux

I run into this error when trying to work with remotes.

command: fetch_from_remotes params: {"projectId":"ff91a7a9-1b3d-4b71-abee-6254bc1b7731","action":"modal"}) backend error: git command exited with non-zero exit code 128: ["fetch", "--quiet", "--prune", "upstream", "+refs/heads/*:refs/remotes/upstream/*"] STDOUT: STDERR: git: /tmp/.mount_GitButGDbenp/usr/lib/libpcre2-8.so.0: no version information available (required by git) /usr/lib/git-core/git: /tmp/.mount_GitButGDbenp/usr/lib/libpcre2-8.so.0: no version information available (required by /usr/lib/git-core/git) /usr/lib/git-core/git-remote-https: symbol lookup error: /usr/lib/libcurl.so.4: undefined symbol: nghttp2_option_set_no_rfc9113_leading_and_trailing_ws_validation fatal: remote helper 'https' aborted session

The strangeness is that nghttp2_option_set_no_rfc9113_leading_and_trailing_ws_validation is absolutely defined in /usr/lib/libcurl.so.4, so there has to be some linker issue. This error does not occur when using the Ubuntu 24.04 AppImage.

☕️ Reasoning

See this comment for a pretty comprehensive justification for the addition of the Ubuntu 22.04 build.

As for the dependency upgrades, they are well overdue. The WebkitGTK build that's currently used is years old and pretty buggy. Upgrading the dependencies fixes the freeze issue described in #9955 on my Arch Linux machine, and probably touches many of the other issues reported on the Linux bundles, the AppImage in particular.

🎫 Affected issues

Fixes: #9955 (The dependency upgrades in the Ubuntu 24.04 image seems to eliminate this problem)

Also improves upon #7671 with the Ubuntu 22.04 build, which links to GLIBC 2.35 instead of 2.38 (as is the case in the Ubuntu 24.04 build). This will let the AppImage run on older distros.

Very likely touches upon more of the Linux-specific issues, but I have not had the time to evaluate more of them.

TODO

On 1/3 systems I've tested on (another Arch Linux+Wayland machine), I'm getting Could not create default EGL display: EGL_BAD_PARAMETER. Aborting .... Which is strange. I want to figure that out before proceeding here.

Testing matrix

Machine AppImage Status Workaround
Arch Linux + Wayland (mesa) Ubuntu 22.04 ❌ EGL_BAD_PARAMETER #5282 (comment)
Arch Linux + Wayland (mesa) Ubuntu 24.04 ❌ EGL_BAD_PARAMETER #5282 (comment)
Arch Linux + Wayland (Nvidia) Ubuntu 22.04 ⚠️ Cannot work with remotes
Arch Linux + Wayland (Nvidia) Ubuntu 24.04 ✔️ No issues detected
Ubuntu 22.04 + X11 (mesa) Ubuntu 22.04 ✔️ No issues detected
Ubuntu 22.04 + X11 (mesa) Ubuntu 24.04 ❌ Cannot launch due to GLIBC mismatch

Important caveat: The Ubuntu 22.04 build may not last

I don't know for how long the Ubuntu 22.04 build will keep on building, as the WebKit project dropped official support for it on 2025-04-25. Which is odd as Debian Bookworm (the Ubuntu 22.04 base distro) has support until June 2026. libwebkit2gtk-4.1 is however still an officially Ubuntu-maintained package, and the current upstream version 2.50.1 was released in October. I think that's sufficient recent activity to make it worthwhile to start running the build.

@vercel
Copy link

vercel bot commented Nov 30, 2025

@slarse is attempting to deploy a commit to the GitButler Team on Vercel.

A member of the Team first needs to authorize it.

Comment on lines +192 to +197
'libwebkit2gtk-4.1-0=2.50.*' \
'libwebkit2gtk-4.1-dev=2.50.*' \
'libjavascriptcoregtk-4.1-0=2.50.*' \
'libjavascriptcoregtk-4.1-dev=2.50.*' \
'gir1.2-javascriptcoregtk-4.1=2.50.*' \
'gir1.2-webkit2-4.1=2.50.*';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe locking down on the minor version is a good enough tradeoff. That way, we get patches as well as Debian/Ubuntu-specific changes automatically, while minimizing risk that we get a breaking update.

Minor version bumps in libwebkit does seem to introduce drastic changes, so I think it's a good idea to perform such updates under more controlled forms.

if: runner.os == 'Linux'
run: |
sudo apt update;
sudo apt upgrade;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The base image for the runner is most often outdated, I think it makes sense to ensure that we get the latest and greatest of everything that's installed for whichever distro we're building for.

Comment on lines +105 to +106
- platform: ubuntu-22.04-arm # [linux, ARM64]
target: ''
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As noted in OP, I'm really unsure if this build is desirable.

@slarse slarse marked this pull request as draft December 1, 2025 07:14
@Byron
Copy link
Collaborator

Byron commented Dec 1, 2025

Thanks so much, this PR could be very impactful to the linux users out there!

Without much ado, I am creating a new nightly from it and invite you to test out the official builds, as downloaded from the main site.

Something I am wondering about is if the additional ubuntu 22 images will confuse the website. And even if not, I think we will need an upgrade so these builds can be differentiated.

(side note: are the Linux ARM64 builds actually used??)

They are downloadable, but… I am sure I am missing something.


And it turns out that the build is failing: https://github.com/gitbutlerapp/gitbutler/actions/runs/19814434277/job/56762812224 .

My feeling is that it might re-use caches from other ubuntu versions as the shared key only contains the target, and maybe that causes trouble. The failed ubuntu 22 job used v0-rust-app-release-build-nightly-Linux-x64-341760e0-379a590c, and the working ubuntu 24 job used v0-rust-app-release-build-nightly-Linux-x64-341760e0-379a590c as well. If nothing else, it's probably racy when writing back. But is this really the cause of the failure? I don't know. But I do know that the weirdest errors happen if the Cargo cache is borked, something that regularly happens even when switching compiler versions. And maybe that's the real issue here.

 --> crates/but-claude/src/notifications.rs:3:5 | 3 | use notify_rust::Notification; | ^^^^^^^^^^^ can't find crate Compiling gitbutler-edit-mode v0.0.0 (/home/runner/work/gitbutler/gitbutler/crates/gitbutler-edit-mode) For more information about this error, try `rustc --explain E0463`. error: could not compile `but-claude` (lib) due to 1 previous error warning: build failed, waiting for other jobs to finish... failed to build app: failed to build app Error [tauri_cli_node] failed to build app: failed to build app  ELIFECYCLE  Command failed with exit code 1. 

Let me try that.

That way we can be sure that differences in compiler versions or other subleties won't throw the build off course. Cargo/rustc definitely is buggy in that regard.
@Byron
Copy link
Collaborator

Byron commented Dec 1, 2025

Ok, here is the new publish: https://github.com/gitbutlerapp/gitbutler/actions/runs/19814866150, and if it passes and the website doesn't choke, this PR can definitely be merged when you think it's acceptable.

The website could be updated in a separate step then.

@Byron
Copy link
Collaborator

Byron commented Dec 1, 2025

The build went through this time, but I am a bit anxious that the download links aren't correct. The filenames aren't qualified enough to differentiate the different linux platforms, and I am afraid that the Ubuntu 22/24 builds are randomly swapped or something.

Could you validate the linux bundles available on the website?

Thanks again.

@slarse
Copy link
Contributor Author

slarse commented Dec 1, 2025

@Byron So sorry, I think you caught this just before I made it a draft as I encountered the EGL_BAD_PARAMETER problem on one machine. See the testing matrix I edited into the PR body. I think it's an upstream issue in Tauri, it's reported all over the place. From some very cursory investigation it appears to be a MESA issue, because I don't encounter it when using a Nvidia GPU with proprietary drivers.

Should be noted that I don't encounter the issue on Ubuntu 22.04 using the Ubuntu 22.04 AppImage, even though that machine is running Intel with MESA drivers, but that may be due to having older drivers. Unsure.

But I need to put in some more work here and do some more compatibility testing. I really want an AppImage that works well across all my Arch machines.

The build went through this time, but I am a bit anxious that the download links aren't correct. The filenames aren't qualified enough to differentiate the different linux platforms, and I am afraid that the Ubuntu 22/24 builds are randomly swapped or something.

Right, I meant to look at this too but somehow it slipped my mind. Will look into that as well.

@Byron
Copy link
Collaborator

Byron commented Dec 1, 2025

Thanks! Let me create a new nightly build then. This would give you another 20 minutes to download the bundles, I hope it's enough.

@slarse
Copy link
Contributor Author

slarse commented Dec 1, 2025

Darn, I think I win this year's miscommunication award. I'm working full time right now (not on GitButler :( ), and then I have family responsibilities, so work on GB is generally confined to late evenings and weekends. So I didn't have time to pull down those build artifacts :(. Just quickly popped my head in when my work computer had the EGL_BAD_PARAMETER issue.

Sorry I wasted your time today, I was just really eager to get this in after having spent so much time fiddling with builds on different systems. A bit too eager, should have done more testing!

On the bright side, I've diagnosed the EGL_BAD_PARAMETER issue on my system. It's, wait for it... Also a linker issue! Why that doesn't happen with the currently pinned versions is anyone's guess. I put down a pretty little writeup over here: #5282 (comment)

The gist of it is that the system's mesa driver links incorrectly with the bundled libwayland-client.so.0 lib. If I instead use the system's lib by preloading it, it seemingly works fine:

$ ./GitButler.AppImage <snip> Could not create default EGL display: EGL_BAD_PARAMETER. Aborting... $ LD_PRELOAD=/usr/lib/libwayland-client.so ./GitButler.AppImage <snip> TADAAAAA (it does not actually say that, but you get it)

This is a workaround at best. I will give it more of a go tomorrow, but even if it works for me there's no telling if it'll work on a Fedora box, for example.

Linker issues seem to be rather abundant in AppImage Tauri-land, and they are working on something to resolve it: tauri-apps/tauri#12491. I think that will be the be-all-end-all solution to the non-portability of the AppImage.

Not entirely sure what we do in the mean time. I think adding the Ubuntu 22.04 build would be worthwhile regardless, but I'm a bit unsure about bumping the core dependencies given the amount of trouble it causes. Then again, not bumping the deps also causes a whole bunch of random issues. Perhaps the preload workaround is fine for now, and we could always just roll back if that causes too many issues for people.

What do you think?

@Byron
Copy link
Collaborator

Byron commented Dec 2, 2025

Darn, I think I win this year's miscommunication award. I'm working full time right now (not on GitButler :( ), and then I have family responsibilities, so work on GB is generally confined to late evenings and weekends. So I didn't have time to pull down those build artifacts :(. Just quickly popped my head in when my work computer had the EGL_BAD_PARAMETER issue.

No worries, at all! It's good that the binaries can still be downloaded, they have just be superseded by a more recent version.
So I don't think any time was wasted (and even if some was, that's life :D).

Not entirely sure what we do in the mean time. I think adding the Ubuntu 22.04 build would be worthwhile regardless, but I'm a bit unsure about bumping the core dependencies given the amount of trouble it causes. Then again, not bumping the deps also causes a whole bunch of random issues. Perhaps the preload workaround is fine for now, and we could always just roll back if that causes too many issues for people.

What do you think?

Actually I can't really judge if the trade-off is worth it. To me right now it looks like the old dependencies allow the app to launch 'more often', but it may freeze later, and the new dependency version prevent a launch more often unless the LD_PRELOAD workaround is used, but it fixes a bunch of random issues as well.

To me neither seems great and I'd love to downgrade linux support to 'experimental' or something. When that's done, we could move to the latest greatest, have a documentation page with workarounds and make sure people know about it, while waiting for tauri-apps/tauri#12491 to land in the hopes for the one good bundle we can provide.

@slarse
Copy link
Contributor Author

slarse commented Dec 2, 2025

Actually I can't really judge if the trade-off is worth it. To me right now it looks like the old dependencies allow the app to launch 'more often', but it may freeze later, and the new dependency version prevent a launch more often unless the LD_PRELOAD workaround is used, but it fixes a bunch of random issues as well.

It's hard. But to add to the argument for not staying on the old dependencies: they don't get security updates. And WebKitGTK is effectively a web browser. There's a lot to be said about not sticking around on yesterday's web browser version.

I did try the preload workaround a bit more today, and found that it works well. Except that, for some reason, the XWayland session shoots away in CPU usage (10x increase in CPU usage, from ~5% to ~50%). That could just be a performance regression in the system's Wayland client (it happens quite regularly), but it could also be that the system libs don't quite want to play with the AppImage libs. I'm going to try the deb distribution as well and see if that works.

To me neither seems great and I'd love to downgrade linux support to 'experimental' or something.

This seems like the only reasonable approach. The Linux story is just a bit poor at the moment. It doesn't work on old distros because of the hard glibc dependency, and it doesn't work on new distros because of random WebKitGTK issues.

I'll bring this up in the Discord so we can have a slightly more interactive discussion around it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants