Performance benchmarking tools for measuring and comparing cold start times across different code changes.

# Run benchmark on current branch uv run pytest tests/test_performance/test_cold_start.py # Compare two branches ./scripts/benchmark_cold_start.sh main my-feature-branch # Compare two existing result files uv run python scripts/compare_benchmarks.py benchmark_results/cold_start_baseline.json benchmark_results/cold_start_latest.json

• Import times: import runpod, import runpod.serverless, import runpod.endpoint
• Module counts: Total modules loaded and runpod-specific modules
• Lazy loading status: Whether paramiko and SSH CLI are eagerly or lazy-loaded
• Statistics: Min, max, mean, median across 10 iterations per measurement

Core benchmark test that measures import performance in fresh Python subprocesses.

# Run as pytest test uv run pytest tests/test_performance/test_cold_start.py -v # Run as standalone script uv run python tests/test_performance/test_cold_start.py # Results saved to: # - benchmark_results/cold_start_<timestamp>.json # - benchmark_results/cold_start_latest.json (always latest)

Output Example:

Running cold start benchmarks... ------------------------------------------------------------ Measuring 'import runpod'... Mean: 273.29ms Measuring 'import runpod.serverless'... Mean: 332.18ms Counting loaded modules... Total modules: 582 Runpod modules: 46 Checking if paramiko is eagerly loaded... Paramiko loaded: False 

Automated benchmark runner that handles git branch switching, dependency installation, and result collection.

# Run on current branch (no git operations) ./scripts/benchmark_cold_start.sh # Run on specific branch ./scripts/benchmark_cold_start.sh main # Compare two branches (runs both, then compares) ./scripts/benchmark_cold_start.sh main feature/lazy-loading

Features:

• Automatic stash/unstash of uncommitted changes
• Dependency installation per branch
• Safe branch switching with restoration
• Timestamped result files
• Automatic comparison when comparing branches

Safety:

• Stashes uncommitted changes before switching branches
• Restores original branch after completion
• Handles errors gracefully

Analyzes and visualizes differences between two benchmark runs with colored terminal output.

uv run python scripts/compare_benchmarks.py <baseline.json> <optimized.json>

Output Example:

====================================================================== COLD START BENCHMARK COMPARISON ====================================================================== IMPORT TIME COMPARISON ---------------------------------------------------------------------- Metric Baseline Optimized Δ ms Δ % ---------------------------------------------------------------------- runpod_total 285.64ms 273.29ms ↓ 12.35ms 4.32% runpod_serverless 376.33ms 395.14ms ↑ -18.81ms -5.00% runpod_endpoint 378.61ms 399.36ms ↑ -20.75ms -5.48% MODULE LOAD COMPARISON ---------------------------------------------------------------------- Total modules loaded: Baseline: 698 Optimized: 582 Δ: 116 Runpod modules loaded: Baseline: 48 Optimized: 46 Δ: 2 LAZY LOADING STATUS ---------------------------------------------------------------------- Paramiko Baseline: LOADED Optimized: NOT LOADED ✓ NOW LAZY SSH CLI Baseline: LOADED Optimized: NOT LOADED ✓ NOW LAZY ====================================================================== SUMMARY ====================================================================== ✓ Cold start improved by 12.35ms ✓ That's a 4.3% improvement over baseline ✓ Baseline: 285.64ms → Optimized: 273.29ms ====================================================================== 

Color coding:

• Green: Improvements (faster times, lazy loading achieved)
• Red: Regressions (slower times, eager loading introduced)
• Yellow: No change

All benchmark results are saved to benchmark_results/ (gitignored).

File naming:

• cold_start_<timestamp>.json - Timestamped result
• cold_start_latest.json - Always contains most recent result
• cold_start_baseline.json - Manually saved baseline for comparison

JSON structure:

{ "timestamp": 1763179522.0437188, "python_version": "3.8.20 (default, Oct 2 2024, 16:12:59) [Clang 18.1.8 ]", "measurements": { "runpod_total": { "min": 375.97, "max": 527.9, "mean": 393.91, "median": 380.4, "iterations": 10 } }, "module_counts": { "total": 698, "filtered": 48 }, "paramiko_eagerly_loaded": true, "ssh_cli_loaded": true }

# 1. Save baseline on main branch git checkout main ./scripts/benchmark_cold_start.sh cp benchmark_results/cold_start_latest.json benchmark_results/cold_start_baseline.json # 2. Switch to feature branch git checkout feature/my-optimization # 3. Run benchmark and compare ./scripts/benchmark_cold_start.sh uv run python scripts/compare_benchmarks.py \ benchmark_results/cold_start_baseline.json \ benchmark_results/cold_start_latest.json

# Compare three different optimization branches ./scripts/benchmark_cold_start.sh main > results_main.txt ./scripts/benchmark_cold_start.sh feature/approach-1 > results_1.txt ./scripts/benchmark_cold_start.sh feature/approach-2 > results_2.txt # Then compare each against baseline uv run python scripts/compare_benchmarks.py \ benchmark_results/cold_start_main_*.json \ benchmark_results/cold_start_approach-1_*.json

Add to your GitHub Actions workflow:

- name: Run cold start benchmark run: |  uv run pytest tests/test_performance/test_cold_start.py --timeout=120  - name: Upload benchmark results uses: actions/upload-artifact@v3 with: name: benchmark-results path: benchmark_results/cold_start_latest.json

Based on testing with Python 3.8:

• Cold start (import runpod): < 300ms (mean)
• Serverless import: < 400ms (mean)
• Module count: < 600 total modules
• Test assertion: Fails if import > 1000ms

Subprocess-based measurements have inherent variance:

• First run in sequence: Often 20-50ms slower (Python startup overhead)
• Subsequent runs: More stable
• Use median or mean for comparison, not single runs

• Fewer modules = faster cold start: Each module has import overhead
• Runpod-specific modules: Should be minimal (40-50)
• Total modules: Includes stdlib and dependencies
• Target reduction: Removing 100+ modules typically saves 10-30ms

• paramiko_eagerly_loaded: false - Good for serverless workers
• ssh_cli_loaded: false - Good for SDK users
• These should only be true when CLI commands are invoked

If you see >100ms variance between runs:

• System is under load
• Disk I/O contention
• Python bytecode cache issues

Solution: Run multiple times and use median values.

# Check git status git status # Manually restore if script failed mid-execution git checkout <original-branch> git stash pop

Ensure dependencies are installed:

uv sync --group test

• Iterations: 10 per measurement (configurable in test)
• Process isolation: Each measurement uses fresh subprocess
• Python cache: Cleared by subprocess creation
• System state: Cannot control OS-level caching

For production performance testing, consider:

• Running on CI with consistent environment
• Multiple runs at different times
• Comparing trends over multiple commits