build: Optimize build time. Replace custom cross-compilers with musl-cross-make toolchains#5730
build: Optimize build time. Replace custom cross-compilers with musl-cross-make toolchains#5730rucoder wants to merge 12 commits intolf-edge:masterfrom
Conversation
78c2b9b to 8daba46 Compare f191fe0 to 39eeac5 Compare Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@ ## master #5730 +/- ## ========================================== + Coverage 19.52% 28.34% +8.81% ========================================== Files 19 18 -1 Lines 3021 2417 -604 ========================================== + Hits 590 685 +95 + Misses 2310 1588 -722 - Partials 121 144 +23 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
86703f9 to 4db5ddb Compare
eriknordmark left a comment
There was a problem hiding this comment.
Any impact on reproducible builds e.g., for LTS releases?
What happens if we update and publish a new version of cross-complers - is there a risk that the LTS releases pick up that new version?
(I didn't see where the version is selected when building EVE.)
@eriknordmark only 3 packages use cross compiler atm: pillar, bsp-imx, optee-os. They uses corss compiler as I want to split this PR into at least 2:
but first I want all builds in the PR to finish then run eden and test on HW before I move on |
| @eriknordmark PR description is mess-leading: I still build cross compilers because not all tool-chains are published. I stil think it is better to take the one maintained by musl lib maintainers than try to figure out what patches we need (look at pkg/cross-compilers/Dockerfile). Besides we build it only once anyways UPDATE: I updated PR description |
4db5ddb to 0f996c4 Compare 831bf2c to 7feaa12 Compare | @rucoder in case of this PR both errors (Docker hashes and Lint Commit) are valid, you must fix them. |
| ERR=$(gofmt -e -l -s $(find . -name \*.go | grep -v /vendor/)) && \ | ||
| if [ -n "$ERR" ] ; then echo "go fmt Failed - ERR: $ERR"; exit 1 ; fi | ||
| RUN CGO_ENABLED=0 go build -ldflags "-s -w" -o recovertpm . | ||
| RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} go build -ldflags "-s -w" -o recovertpm . |
There was a problem hiding this comment.
Why is this necessary? Isn't it always running on linux?
There was a problem hiding this comment.
@christoph-zededa it is not really necessary , just to be more explicit. GOOS and GOARCH usually go in pair. I had a PR that clears this uncertainty in other places #5714
| RUN tar -C .. -xzvf ../1.5.tar.gz | ||
| RUN ./autogen.sh && ./configure && make DESTDIR=/out install | ||
| # hadolint ignore=SC2046 | ||
| RUN ./autogen.sh && ./configure && make -j$(nproc) && make DESTDIR=/out install |
I would put this as a single item in the release note (e.g., "Improve the efficiency of the CI/CD cross compilation") with the above being additional details. That should make it esier for a user to understand the (non-)impact of the change to their use of EVE. |
…tion Add a new script that extracts target-architecture Alpine packages from a foreign-arch eve-alpine mirror without requiring QEMU emulation. It uses the existing `apk index --rewrite-arch` trick (already used in build-cache.sh) to make apk treat foreign-arch packages as native. This enables cross-compilation sysroot setup to run entirely on the build host by copying the target-arch eve-alpine mirror and extracting packages locally, eliminating the need for QEMU-based target-arch containers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Migrate pillar, optee-os, and bsp-imx to use cross-compiler toolchains built from musl-cross-make (added in a previous PR). This replaces the old Alpine aports bootstrap approach. Key changes: - Remove old pkg/cross-compilers (aports bootstrap) - Remove cross-compilers from Makefile PKGS lists - Update consumers to use new toolchain from build-tools/cross-compilers with prefix {arch}-linux-musl- (e.g. aarch64-linux-musl-gcc) - Eliminate QEMU-based sysroot extraction using eve-cross-sysroot.sh which extracts target-arch packages from the eve-alpine mirror on the build host without needing a target-arch container Depends on: eve-cross-sysroot.sh in pkg/alpine (commit 1 of this PR) and build-tools/cross-compilers (separate PR lf-edge#5752) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> Alpine 3.22 protobuf links dynamically (libprotobuf.so.29) whereas Alpine 3.16 protobuf was statically linked into ptpm. The Dockerfile was explicitly deleting libprotobuf from /out/, which worked before but now causes runtime symbol resolution failures. Keep libprotobuf.so in the runtime image, only remove libprotoc.so (the compiler, not needed at runtime). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
RUSTFLAGS env var was overriding target-specific rustflags from eve-rust's /usr/local/cargo/config.toml, silently dropping the mold linker (--ld-path=/usr/bin/mold) and the cross-compilation --target flag passed to clang. Add these flags to RUSTFLAGS explicitly. Replace strip with llvm-strip since the native GNU strip cannot recognize cross-compiled binaries (e.g. aarch64 binary on x86_64 host). llvm-strip is architecture-agnostic and already available in the eve-rust image. Bump vector from v0.47.0 to v0.54.0 to be compilable with Rust 1.93. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Split the single build stage into build-c and build-go: - build-c: C/C++ builds (libtpms, swtpm, tpm2-tss, tpm2-tools, ptpm) remain under QEMU emulation due to autoconf/automake dependencies. - build-go: runs natively on the build host using --platform=$BUILDPLATFORM with GOARCH=$TARGETARCH. Go has first-class cross-compilation support and CGO_ENABLED=0, so no C toolchain is needed. This avoids QEMU which is known to deadlock on go vet. Both stages run in parallel under BuildKit. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Add --platform=$BUILDPLATFORM so the Go build runs natively on the host instead of under QEMU emulation. Cross-compile with GOARCH=$TARGETARCH. The build is pure Go with CGO_ENABLED=0, so no C toolchain is needed. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Add --platform=$BUILDPLATFORM so the Go build runs natively on the host instead of under QEMU emulation. Cross-compile with GOARCH=$TARGETARCH. The build is pure Go with CGO_ENABLED=0, so no C toolchain is needed. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Add --platform=$BUILDPLATFORM so the Go build runs natively on the host instead of under QEMU emulation. Cross-compile with GOARCH=$TARGETARCH. The build is pure Go with CGO_ENABLED=0, so no C toolchain is needed. Runtime packages (coreutils) are installed in a separate stage without --platform so they resolve to the correct target architecture. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
- bump rust across all rust applications Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
lshw's Makefile has a race condition with parallel builds: the "static" target depends on both the "core" phony target and "core/liblshw.a" file. With -j, make starts building liblshw.a via the "core" target while simultaneously looking for the liblshw.a file which doesn't exist yet. Work around this by building the "core" target first in a separate make invocation, then building "static" once the library is available. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Add --platform=$BUILDPLATFORM so the Go build runs natively on the host instead of under QEMU emulation. Cross-compile with GOARCH=$TARGETARCH. The build is pure Go with CGO_ENABLED=0, so no C toolchain is needed. Runtime packages (iproute2, iptables, etc.) are installed in a separate stage without --platform so they resolve to the correct target architecture. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
- we changed eve-alpine so update all hashes Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Description
Replace the custom cross-compiler build (Alpine aports bootstrap) with toolchains built from musl-cross-make, the upstream tool maintained by the musl developers.
The previous approach bootstrapped GCC+binutils+musl from Alpine 3.16 aports source with custom patches — slow, fragile, and hard to maintain. musl-cross-make is the standard way to build musl cross-compilers, with pinned versions and proper hash verification.
The cross-compilers image is built once manually (via CI workflow or locally) and pushed to DockerHub with a version tag (e.g.
gcc-14.2.0-alpine-3.22-<hash>). It is no longer part of the regularmake pkgsbuild — consumers reference it by tag.Also eliminates the QEMU-based sysroot extraction in consumers (pillar, optee-os, bsp-imx) by adding
eve-cross-sysroot.shtopkg/alpine. This script extracts target-arch packages from the eve-alpine mirror on the build host usingapk index --rewrite-arch, without needing a target-arch container.Key changes
Cross-compiler infrastructure
pkg/alpine: addeve-cross-sysroot.shfor cross-arch package extraction{arch}-linux-musl-) andeve-cross-sysroot.shfor sysrootvector: fix cross-compilation for all platforms
RUSTFLAGSenv var was overriding target-specific rustflags from eve-rust's/usr/local/cargo/config.toml, silently dropping the mold linker (--ld-path=/usr/bin/mold) and the cross-compilation--targetflag passed to clang. Add these flags toRUSTFLAGSexplicitly.stripwithllvm-stripsince the native GNU strip cannot recognize cross-compiled binaries (e.g. aarch64 binary on x86_64 host).llvm-stripis architecture-agnostic and already available in the eve-rust image.vtpm: fix cross-compilation by splitting Go and C builds
buildstage intobuild-candbuild-go.build-c: C/C++ builds (libtpms, swtpm, tpm2-tss, tpm2-tools, ptpm) remain under QEMU emulation due to autoconf/automake dependencies.build-go: runs natively on the build host using--platform=$BUILDPLATFORMwithGOARCH=$TARGETARCH. This avoids QEMU which is known to deadlock ongo vet/go build. Both stages run in parallel under BuildKit.recovertpm, measure-config, newlog: fix cross-compilation by building Go natively
--platform=$BUILDPLATFORMso Go builds run natively on the host instead of under QEMU emulation.GOARCH=$TARGETARCH. All three are pure Go withCGO_ENABLED=0, so no C toolchain is needed.coreutils) are installed in a separate stage without--platformso they resolve to the correct target architecture.monitor: bump eve-rust to 1.93.1
lfedge/eve-rustfrom1.85.1-2to1.93.1.How to test and validate this PR
REPO=myrepo/eve-cross-compilers ./build-tools/cross-compilers/build.sh --push --singlepkg/pillarfor arm64 target on amd64 host — verify cross-compilationpkg/optee-osfor arm64 — verify ARM32+ARM64 cross-compilationpkg/bsp-imxfor arm64 — verify U-Boot/ATF cross-compilationpkg/vectorfor arm64 — verify Rust cross-compilation with mold and llvm-strippkg/vtpmfor arm64 — verify Go builds natively while C builds complete under QEMUpkg/recovertpm,pkg/measure-config,pkg/newlogfor arm64 — verify Go cross-compilationpkg/monitorfor arm64 — verify Rust 1.93.1 cross-compilationChangelog notes
Improve the efficiency of the CI/CD cross compilation
PR Backports
Checklist
I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't
check them.