This directory contains performance benchmarks for Foundry commands across multiple repositories and Foundry versions.

Before running the benchmarks, ensure you have the following installed:

1. Rust and Cargo - Required for building the benchmark binary

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

2. Foundryup - The Foundry toolchain installer

   curl -L https://foundry.paradigm.xyz | bash foundryup

3. Git - For cloning benchmark repositories

4. Hyperfine - The benchmarking tool used by foundry-bench

5. Node.js and npm - Some repositories require npm dependencies

Build and install the benchmark runner:

cargo build --release --bin foundry-bench

To install foundry-bench to your PATH:

cd benches && cargo install --path . --bin foundry-bench

# Run all benchmarks on default repos with stable and nightly versions foundry-bench --versions stable,nightly

# Bench specific versions foundry-bench --versions stable,nightly,v1.0.0 # Run on specific repositories. Default rev for the repo is "main" foundry-bench --repos ithacaxyz/account,Vectorized/solady # Test specific repository with custom revision foundry-bench --repos ithacaxyz/account:main,Vectorized/solady:v0.0.123 # Run only specific benchmarks foundry-bench --benchmarks forge_build_with_cache,forge_test # Run only fuzz tests foundry-bench --benchmarks forge_fuzz_test # Run coverage benchmark foundry-bench --benchmarks forge_coverage # Combine options foundry-bench \ --versions stable,nightly \ --repos ithacaxyz/account \ --benchmarks forge_build_with_cache # Force install Foundry versions foundry-bench --force-install # Verbose output to see hyperfine logs foundry-bench --verbose # Output to specific directory foundry-bench --output-dir ./results --output-file LATEST_RESULTS.md

• --versions <VERSIONS> - Comma-separated list of Foundry versions (default: stable,nightly)
• --repos <REPOS> - Comma-separated list of repos in org/repo[:rev] format (default: ithacaxyz/account:v0.3.2,Vectorized/solady:v0.1.22)
• --benchmarks <BENCHMARKS> - Comma-separated list of benchmarks to run
• --force-install - Force installation of Foundry versions
• --verbose - Show detailed benchmark output
• --output-dir <DIR> - Directory for output files (default: benches)
• --output-file <FILE_NAME.md> - Name of the output file (default: LATEST.md)

• forge_test - Benchmarks forge test command across repos
• forge_build_no_cache - Benchmarks forge build with clean cache
• forge_build_with_cache - Benchmarks forge build with existing cache
• forge_fuzz_test - Benchmarks forge test with only fuzz tests (tests with parameters)
• forge_coverage - Benchmarks forge coverage --ir-minimum command across repos

The benchmark binary uses command-line arguments to configure which repositories and versions to test. The default repositories are:

• ithacaxyz/account:v0.3.2
• Vectorized/solady:v0.1.22

You can override these using the --repos flag with the format org/repo[:rev].

Benchmark results are saved to benches/LATEST.md (or custom output file specified with --output-file). The report includes:

• Summary of versions and repositories tested
• Performance comparison tables for each benchmark type showing:
  • Mean execution time
  • Min/Max times
  • Standard deviation
  • Relative performance comparison between versions
• System information (OS, CPU cores)
• Detailed hyperfine benchmark results in JSON format

1. Foundry version not found: Use --force-install flag or manually install with foundryup --install <version>
2. Repository clone fails: Check network connectivity and repository access
3. Build failures: Some repositories may have specific dependencies - check their README files
4. Hyperfine not found: Install hyperfine using the instructions in Prerequisites
5. npm/Node.js errors: Ensure Node.js and npm are installed for repositories that require them