Skip to content

build: Optimize build time. Replace custom cross-compilers with musl-cross-make toolchains#5730

Open
rucoder wants to merge 12 commits intolf-edge:masterfrom
rucoder:rucoder/musl-cc-cross-compilers
Open

build: Optimize build time. Replace custom cross-compilers with musl-cross-make toolchains#5730
rucoder wants to merge 12 commits intolf-edge:masterfrom
rucoder:rucoder/musl-cc-cross-compilers

Conversation

@rucoder
Copy link
Copy Markdown
Contributor

@rucoder rucoder commented Apr 1, 2026

Description

Replace the custom cross-compiler build (Alpine aports bootstrap) with toolchains built from musl-cross-make, the upstream tool maintained by the musl developers.

The previous approach bootstrapped GCC+binutils+musl from Alpine 3.16 aports source with custom patches — slow, fragile, and hard to maintain. musl-cross-make is the standard way to build musl cross-compilers, with pinned versions and proper hash verification.

The cross-compilers image is built once manually (via CI workflow or locally) and pushed to DockerHub with a version tag (e.g. gcc-14.2.0-alpine-3.22-<hash>). It is no longer part of the regular make pkgs build — consumers reference it by tag.

Also eliminates the QEMU-based sysroot extraction in consumers (pillar, optee-os, bsp-imx) by adding eve-cross-sysroot.sh to pkg/alpine. This script extracts target-arch packages from the eve-alpine mirror on the build host using apk index --rewrite-arch, without needing a target-arch container.

Key changes

Cross-compiler infrastructure

  • pkg/alpine: add eve-cross-sysroot.sh for cross-arch package extraction
  • Build toolchains from musl-cross-make with pinned versions (GCC 14.2.0, binutils 2.44) for both amd64 and arm64 hosts
  • Update consumers to use new toolchain prefix ({arch}-linux-musl-) and eve-cross-sysroot.sh for sysroot
  • Remove old aports patches

vector: fix cross-compilation for all platforms

  • RUSTFLAGS env var was overriding target-specific rustflags from eve-rust's /usr/local/cargo/config.toml, silently dropping the mold linker (--ld-path=/usr/bin/mold) and the cross-compilation --target flag passed to clang. Add these flags to RUSTFLAGS explicitly.
  • Replace strip with llvm-strip since the native GNU strip cannot recognize cross-compiled binaries (e.g. aarch64 binary on x86_64 host). llvm-strip is architecture-agnostic and already available in the eve-rust image.
  • Bump vector from v0.47.0 to v0.54.0 to be compilable with Rust 1.93.

vtpm: fix cross-compilation by splitting Go and C builds

  • Split the single build stage into build-c and build-go.
  • build-c: C/C++ builds (libtpms, swtpm, tpm2-tss, tpm2-tools, ptpm) remain under QEMU emulation due to autoconf/automake dependencies.
  • build-go: runs natively on the build host using --platform=$BUILDPLATFORM with GOARCH=$TARGETARCH. This avoids QEMU which is known to deadlock on go vet / go build. Both stages run in parallel under BuildKit.

recovertpm, measure-config, newlog: fix cross-compilation by building Go natively

  • Add --platform=$BUILDPLATFORM so Go builds run natively on the host instead of under QEMU emulation.
  • Cross-compile with GOARCH=$TARGETARCH. All three are pure Go with CGO_ENABLED=0, so no C toolchain is needed.
  • For newlog, runtime packages (coreutils) are installed in a separate stage without --platform so they resolve to the correct target architecture.

monitor: bump eve-rust to 1.93.1

  • Bump lfedge/eve-rust from 1.85.1-2 to 1.93.1.

How to test and validate this PR

  • Build cross-compilers image:
    REPO=myrepo/eve-cross-compilers ./build-tools/cross-compilers/build.sh --push --single
  • Build pkg/pillar for arm64 target on amd64 host — verify cross-compilation
  • Build pkg/optee-os for arm64 — verify ARM32+ARM64 cross-compilation
  • Build pkg/bsp-imx for arm64 — verify U-Boot/ATF cross-compilation
  • Build pkg/vector for arm64 — verify Rust cross-compilation with mold and llvm-strip
  • Build pkg/vtpm for arm64 — verify Go builds natively while C builds complete under QEMU
  • Build pkg/recovertpm, pkg/measure-config, pkg/newlog for arm64 — verify Go cross-compilation
  • Build pkg/monitor for arm64 — verify Rust 1.93.1 cross-compilation
  • Build each consumer natively — verify native build path unaffected

Changelog notes

Improve the efficiency of the CI/CD cross compilation

PR Backports

  • 16.0-stable: No, build infrastructure change
  • 14.5-stable: No, build infrastructure change
  • 13.4-stable: No, build infrastructure change

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

@rucoder rucoder force-pushed the rucoder/musl-cc-cross-compilers branch 13 times, most recently from 78c2b9b to 8daba46 Compare April 1, 2026 16:11
@rucoder rucoder force-pushed the rucoder/musl-cc-cross-compilers branch 3 times, most recently from f191fe0 to 39eeac5 Compare April 2, 2026 06:57
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 2, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 28.34%. Comparing base (2281599) to head (6861a61).
⚠️ Report is 451 commits behind head on master.

Additional details and impacted files
@@ Coverage Diff @@ ## master #5730 +/- ## ========================================== + Coverage 19.52% 28.34% +8.81%  ========================================== Files 19 18 -1 Lines 3021 2417 -604 ========================================== + Hits 590 685 +95  + Misses 2310 1588 -722  - Partials 121 144 +23 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@rucoder rucoder force-pushed the rucoder/musl-cc-cross-compilers branch 3 times, most recently from 86703f9 to 4db5ddb Compare April 6, 2026 11:21
Copy link
Copy Markdown
Contributor

@eriknordmark eriknordmark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any impact on reproducible builds e.g., for LTS releases?
What happens if we update and publish a new version of cross-complers - is there a risk that the LTS releases pick up that new version?
(I didn't see where the version is selected when building EVE.)

@rucoder
Copy link
Copy Markdown
Contributor Author

rucoder commented Apr 6, 2026

Any impact on reproducible builds e.g., for LTS releases? What happens if we update and publish a new version of cross-complers - is there a risk that the LTS releases pick up that new version? (I didn't see where the version is selected when building EVE.)

@eriknordmark only 3 packages use cross compiler atm: pillar, bsp-imx, optee-os. They uses corss compiler as FROM mikemzed/eve-cross-compilers:gcc-14.2.0-alpine-3.22-c4b0b3600. I did it to test a workflow first on Rene's repo. Technically cross-compiler pushed as mikemzed/eve-cross-compilers:gcc-14.2.0-alpine-3.22-c4b0b3600 is the same as if we build it from lfedge/eve repo.

I want to split this PR into at least 2:

  1. Workflow to build and publish cross-compiler + ./build-tools/cross-compilers/ folder. then we build cross compiler with correct tags pointing to lfedge/eve-cross-compielrs
  2. follow up PR will replace cross compiler with a new one and deprecate pkg/cross-compilers. in this case we can safly test how this PR works

but first I want all builds in the PR to finish then run eden and test on HW before I move on

@rucoder
Copy link
Copy Markdown
Contributor Author

rucoder commented Apr 6, 2026

@eriknordmark PR description is mess-leading: I still build cross compilers because not all tool-chains are published. I stil think it is better to take the one maintained by musl lib maintainers than try to figure out what patches we need (look at pkg/cross-compilers/Dockerfile). Besides we build it only once anyways

UPDATE: I updated PR description

@rucoder rucoder changed the title build: replace custom cross-compilers with musl.cc toolchains build: replace custom cross-compilers with musl-cross-make toolchains Apr 6, 2026
@rucoder rucoder force-pushed the rucoder/musl-cc-cross-compilers branch from 4db5ddb to 0f996c4 Compare April 6, 2026 19:16
@github-actions github-actions bot requested a review from eriknordmark April 6, 2026 19:16
@rucoder rucoder force-pushed the rucoder/musl-cc-cross-compilers branch 2 times, most recently from 831bf2c to 7feaa12 Compare April 7, 2026 09:17
@rene
Copy link
Copy Markdown
Contributor

rene commented Apr 8, 2026

@rucoder in case of this PR both errors (Docker hashes and Lint Commit) are valid, you must fix them.

@rucoder
Copy link
Copy Markdown
Contributor Author

rucoder commented Apr 8, 2026

@rucoder in case of this PR both errors (Docker hashes and Lint Commit) are valid, you must fix them.

Erik has a fix for rust in installer #5744

ERR=$(gofmt -e -l -s $(find . -name \*.go | grep -v /vendor/)) && \
if [ -n "$ERR" ] ; then echo "go fmt Failed - ERR: $ERR"; exit 1 ; fi
RUN CGO_ENABLED=0 go build -ldflags "-s -w" -o recovertpm .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} go build -ldflags "-s -w" -o recovertpm .
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this necessary? Isn't it always running on linux?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@christoph-zededa it is not really necessary , just to be more explicit. GOOS and GOARCH usually go in pair. I had a PR that clears this uncertainty in other places #5714

RUN tar -C .. -xzvf ../1.5.tar.gz
RUN ./autogen.sh && ./configure && make DESTDIR=/out install
# hadolint ignore=SC2046
RUN ./autogen.sh && ./configure && make -j$(nproc) && make DESTDIR=/out install
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eriknordmark
Copy link
Copy Markdown
Contributor

Replace custom-built cross-compilers with musl-cross-make toolchains. Cross-compiler image is now built on demand and referenced by version tag. Eliminates QEMU dependency for sysroot extraction during cross-compilation. Fix cross-compilation for vector (RUSTFLAGS override, llvm-strip), vtpm, recovertpm, measure-config, and newlog (Go builds natively, no more QEMU deadlocks). Bump eve-rust to 1.93.1 for monitor and vector.

I would put this as a single item in the release note (e.g., "Improve the efficiency of the CI/CD cross compilation") with the above being additional details. That should make it esier for a user to understand the (non-)impact of the change to their use of EVE.

rucoder and others added 12 commits April 9, 2026 08:04
…tion Add a new script that extracts target-architecture Alpine packages from a foreign-arch eve-alpine mirror without requiring QEMU emulation. It uses the existing `apk index --rewrite-arch` trick (already used in build-cache.sh) to make apk treat foreign-arch packages as native. This enables cross-compilation sysroot setup to run entirely on the build host by copying the target-arch eve-alpine mirror and extracting packages locally, eliminating the need for QEMU-based target-arch containers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Migrate pillar, optee-os, and bsp-imx to use cross-compiler toolchains built from musl-cross-make (added in a previous PR). This replaces the old Alpine aports bootstrap approach. Key changes: - Remove old pkg/cross-compilers (aports bootstrap) - Remove cross-compilers from Makefile PKGS lists - Update consumers to use new toolchain from build-tools/cross-compilers with prefix {arch}-linux-musl- (e.g. aarch64-linux-musl-gcc) - Eliminate QEMU-based sysroot extraction using eve-cross-sysroot.sh which extracts target-arch packages from the eve-alpine mirror on the build host without needing a target-arch container Depends on: eve-cross-sysroot.sh in pkg/alpine (commit 1 of this PR) and build-tools/cross-compilers (separate PR lf-edge#5752) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Alpine 3.22 protobuf links dynamically (libprotobuf.so.29) whereas Alpine 3.16 protobuf was statically linked into ptpm. The Dockerfile was explicitly deleting libprotobuf from /out/, which worked before but now causes runtime symbol resolution failures. Keep libprotobuf.so in the runtime image, only remove libprotoc.so (the compiler, not needed at runtime). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
RUSTFLAGS env var was overriding target-specific rustflags from eve-rust's /usr/local/cargo/config.toml, silently dropping the mold linker (--ld-path=/usr/bin/mold) and the cross-compilation --target flag passed to clang. Add these flags to RUSTFLAGS explicitly. Replace strip with llvm-strip since the native GNU strip cannot recognize cross-compiled binaries (e.g. aarch64 binary on x86_64 host). llvm-strip is architecture-agnostic and already available in the eve-rust image. Bump vector from v0.47.0 to v0.54.0 to be compilable with Rust 1.93. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Split the single build stage into build-c and build-go: - build-c: C/C++ builds (libtpms, swtpm, tpm2-tss, tpm2-tools, ptpm) remain under QEMU emulation due to autoconf/automake dependencies. - build-go: runs natively on the build host using --platform=$BUILDPLATFORM with GOARCH=$TARGETARCH. Go has first-class cross-compilation support and CGO_ENABLED=0, so no C toolchain is needed. This avoids QEMU which is known to deadlock on go vet. Both stages run in parallel under BuildKit. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Add --platform=$BUILDPLATFORM so the Go build runs natively on the host instead of under QEMU emulation. Cross-compile with GOARCH=$TARGETARCH. The build is pure Go with CGO_ENABLED=0, so no C toolchain is needed. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Add --platform=$BUILDPLATFORM so the Go build runs natively on the host instead of under QEMU emulation. Cross-compile with GOARCH=$TARGETARCH. The build is pure Go with CGO_ENABLED=0, so no C toolchain is needed. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Add --platform=$BUILDPLATFORM so the Go build runs natively on the host instead of under QEMU emulation. Cross-compile with GOARCH=$TARGETARCH. The build is pure Go with CGO_ENABLED=0, so no C toolchain is needed. Runtime packages (coreutils) are installed in a separate stage without --platform so they resolve to the correct target architecture. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
- bump rust across all rust applications Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
lshw's Makefile has a race condition with parallel builds: the "static" target depends on both the "core" phony target and "core/liblshw.a" file. With -j, make starts building liblshw.a via the "core" target while simultaneously looking for the liblshw.a file which doesn't exist yet. Work around this by building the "core" target first in a separate make invocation, then building "static" once the library is available. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Add --platform=$BUILDPLATFORM so the Go build runs natively on the host instead of under QEMU emulation. Cross-compile with GOARCH=$TARGETARCH. The build is pure Go with CGO_ENABLED=0, so no C toolchain is needed. Runtime packages (iproute2, iptables, etc.) are installed in a separate stage without --platform so they resolve to the correct target architecture. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
- we changed eve-alpine so update all hashes Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

4 participants