Welcome to CARP-S! This repository contains a benchmarking framework for optimizers. It allows flexibly combining optimizers and benchmarks via a simple interface, and logging experiment results and trajectories to a database. carps can launch experiment runs in parallel by using hydra, which offers launchers for slurm/submitit, Ray, RQ, and joblib.

The main topics of this README are:

• Installation
• Minimal Example
• Commands
• Adding a new Optimizer or Benchmark

For more details on CARP-S, please have a look at the documentation or our blog post.

To install CARP-S, you can simply use pip:

1. Create virtual env with conda or uv

# Conda conda create -n carps python=3.12 conda activate carps # -OR - # uv pip install uv export PIP="uv pip" # Env var needed for Makefile commands uv venv --python=3.12 carpsenv source carpsenv/bin/activate

2. Install carps.

pip install carps

If you want to install from source, you can clone the repository and install CARP-S via:

git clone https://github.com/AutoML/CARP-S.git cd CARP-S export PIP="pip" conda create -n carps python=3.12 conda activate carps # Install for usage $PIP install .

git clone https://github.com/AutoML/CARP-S.git cd CARP-S pip install uv export PIP="uv pip" uv venv --python=3.12 carpsenv source carpsenv/bin/activate # Install for usage $PIP install . # Install as editable $PIP install -e .

If you want to install CARP-S for development, you can use the following command (from the root of the repo):

$PIP install -e . python -m carps.build.make install-dev

⚠ This is still experimental. You can also use a container as an env, see this guide.

For python3.12, numpy should be numpy>=2.0.0. For python3.10, numpy must be numpy==1.26.4, you can simply pip install numpy==1.26.4 after running the proposed install commands.

Additionally, you need to install the requirements for the benchmark and optimizer that you want to use.

⚠ You can specify the directory of the task data by export CARPS_TASK_DATA_DIR=.... Please use absolute dirnames. The default location is <carps package location>/task_data. If you specify a custom dir, always export the env var. (The carps package location is the root of the package, not of the repo.)

For example, if you want to use the SMAC3 optimizer and the BBOB benchmark, you need to install the requirements for both of them via:

# Install options for optimizers and benchmarks (these are Makefile commands, check the Makefile at carps/build for # more commands) # The commands should be separated by a whitespace python -m carps.build.make benchmark_bbob optimizer_smac

The benchmarks and optimizers can all be installed in one environment (tested with python3.12).

All possible install options for benchmarks are:

benchmark_bbob benchmark_hpobench benchmark_hpob benchmark_mfpbench benchmark_pymoo benchmark_yahpo 

⚠ Some benchmarks require to download surrogate models and/or containers and thus might take disk space and time to download.

All possible install options for optimizers are:

optimizer_smac optimizer_dehb optimizer_nevergrad optimizer_optuna optimizer_ax optimizer_skopt optimizer_synetune 

All of the above except optimizer_hebo work with python3.12.

You can also install all benchmarks in one go with benchmarks and all optimizers with optimizers. Check the carps/build/Makefile in carps for more details.

Once the requirements for both an optimizer and a benchmark, e.g. SMAC2.0 and BBOB, are installed, you can run one of the following minimal examples to benchmark SMAC2.0 on BBOB directly with Hydra:

# Run SMAC BlackBoxFacade on certain BBOB task python -m carps.run +optimizer/smac20=blackbox +task/BBOB=cfg_4_1_0 seed=1 task.optimization_resources.n_trials=25 # Run SMAC BlackBoxFacade on all available BBOB tasks for 10 seeds python -m carps.run +optimizer/smac20=blackbox '+task/BBOB=glob(*)' 'seed=range(1,11)' -m

For the second command, the Hydra -m (or --multirun) option indicates that multiple runs will be performed over a range of parameter values. In this case, it's indicating that the benchmarking should be run for all available BBOB tasks (+task/BBOB=glob(*)) and for 10 different seed values (seed=range(1,11)).

For a complete list see the docs.

You can run a certain task and optimizer combination directly with Hydra via:

python -m carps.run +task=... +optimizer=... seed=... -m

To check whether any runs are missing, you can use the following command. It will create a file runcommands_missing.sh containing the missing runs:

python -m carps.utils.check_missing <rundir>

To collect all run data generated by the file logger into parquet files, use the following command:

python -m carps.analysis.gather_data <rundir>

The parquet files are then located in <rundir>. logs.parquet contain the trial info and values and logs_cfg.parquet contain the experiment configuration. The experiments can be matched via the column experiment_id.

Per default, carps logs to files. This has its caveats: Checking experiment status is a bit more cumbersome (but possible with python -m carps.utils.check_missing <rundir> to check for missing/failed experiments) and reading from the filesystem takes a long time. For this reason, we can also control and log experiments to a MySQL database with PyExperimenter. See the guide in the docs for information about how to set it up.

For instructions on how to add a new optimizer or benchmark, please refer to the contributing guidelines for benchmarks and optimizers.

In the case when you are developing your optimizer or benchmark in a standalone package, you can use carps without directly working in the carps repo. For a custom benchmark we have an example repo. It shows how to use your own benchmark with carps optimizers. For a custom optimizer check this example repo. Information is also available here.

For each task_type (blackbox, multi-fidelity, multi-objective and multi-fidelity-multi-objective) and set (dev and test), we run selected optimizers and provide the data. Here we provide the link to the meta data that contains the detailed optimization setting for each run
and the running results that records the running results of each optimization-benchmark combination.

CARP-S is developed by the AutoML Groups of the Universities of Hannover and Freiburg. It is a featured optimizer on AutoML Space.

If you have found a bug, please report to issues. Moreover, we are appreciating any kind of help. Find our guidelines for contributing to this package here.

If you use CARP-S in one of your research projects, please cite our paper:

@article{benjamins2025carps, title={carps: A framework for comparing n hyperparameter optimizers on m benchmarks}, author={Benjamins, Carolin and Graf, Helena and Segel, Sarah and Deng, Difan and Ruhkopf, Tim and Hennig, Leona and Basu, Soham and Mallik, Neeratyoy and Bergman, Edward and Chen, Deyao and others}, journal={arXiv:2506.06143}, year={2025} } 

Copyright (c) 2025, Leibniz University Hannover - Institute of AI