Skip to content

Latest commit

 

History

History
163 lines (115 loc) · 6.44 KB

README.md

File metadata and controls

163 lines (115 loc) · 6.44 KB

image image image Ruff

hposuite

A lightweight framework for benchmarking HPO algorithms

Minimal Example to run hposuite

from hposuite import create_study

study = create_study(
    name="hposuite_demo",
    output_dir="./hposuite-output",
    optimizers=[...],   #Eg: "RandomSearch"
    benchmarks=[...],   #Eg: "ackley"
    num_seeds=5,
    budget=100,         # Number of iterations
)

study.optimize()

Tip

Installation

Create a Virtual Environment using Venv

python -m venv hposuite_env
source hposuite_env/bin/activate

Installing from PyPI

pip install hposuite

Tip

  • pip install hposuite["notebook"] - For usage in a notebook
  • pip install hposuite["all"] - To install hposuite with all available optimizers and benchmarks
  • pip install hposuite["optimizers"] - To install hposuite with all available optimizers only
  • pip install hposuite["benchmarks"] - To install hposuite with all available benchmarks only

Note

  • We recommend doing doing pip install hposuite["all"] to install all available benchmarks and optimizers, along with ipykernel for running the notebook examples

Installation from source

git clone https://github.com/automl/hposuite.git
cd hposuite

pip install -e . # -e for editable install

Simple example to run multiple Optimizers on multiple benchmarks

from hposuite.benchmarks import BENCHMARKS
from hposuite.optimizers import OPTIMIZERS

from hposuite import create_study

study = create_study(
    name="smachb_dehb_mfh3good_pd1",
    output_dir="./hposuite-output",
    optimizers=[
        OPTIMIZERS["SMAC_Hyperband"],
        OPTIMIZERS["DEHB_Optimizer"]
    ],
    benchmarks=[
        BENCHMARKS["mfh3_good"],
        BENCHMARKS["pd1-imagenet-resnet-512"]
    ],
    num_seeds=5,
    budget=100,
)

study.optimize()

View all available Optimizers and Benchmarks

from hposuite.optimizers import OPTIMIZERS
from hposuite.benchmarks import BENCHMARKS
print(OPTIMIZERS.keys())
print(BENCHMARKS.keys())

Results

hposuite saves the Studies by default to ./hposuite-output/ (relative to the current working directory). Results are saved in the Run subdirectories within the main Study directory as parquet files.
The Study directory and the individual Run directory paths are logged when running Study.optimize()

Plotting

python -m hposuite.plotting.utils \
--study_dir <study directory name>
--output_dir <abspath of dir where study dir is stored>
--save_dir <path relative to study_dir to store the plots> \ 

--save_dir is set by default to study_dir/plots --output_dir by default is ../hposuite-output

Overview of available Optimizers

For a more detailed overview, check here

Overview of Available Optimizers

Optimizer Package Blackbox Multi-Fidelity (MF) Multi-Objective (MO) MO-MF Priors
RandomSearch
RandomSearch with priors
SMAC
SMAC
DEHB
HEBO
Nevergrad
Optuna
Optuna
Scikit-Optimize

Overview of available Benchmarks

For a more detailed overview, check here

Benchmark Package Type Multi-Fidelity Multi-Objective
Ackley Synthetic
Branin Synthetic
mf-prior-bench Synthetic, Surrogate
LCBench-Tabular Tabular
Pymoo Synthetic
IOH (BBOB) Synthetic