OpenBox: Generalized and Efficient Blackbox Optimization System#

OpenBox is an efficient open-source system designed for solving generalized black-box optimization (BBO) problems, such as automatic hyper-parameter tuning, automatic A/B testing, experimental design, database knob tuning, processor architecture and circuit design, resource allocation, automatic chemical design, etc.

The design of OpenBox follows the philosophy of providing “BBO as a service” - we opt to implement OpenBox as a distributed, fault-tolerant, scalable, and efficient service, with a wide range of application scope, stable performance across problems and advantages such as ease of use, portability, and zero maintenance.

There are two ways to use OpenBox: Standalone python package and Online BBO service.

OpenBox GitHub: https://github.com/PKU-DAIR/open-box


News#

  • OpenBox based solutions achieved the First Place of ACM CIKM 2021 AnalyticCup (Track - Automated Hyperparameter Optimization of Recommendation System).

  • OpenBox team won the Top Prize (special prize) in the open-source innovation competition at 2021 CCF ChinaSoft conference.

  • Pasca, which adopts Openbox to support neural architecture search functionality, won the Best Student Paper Award at WWW’22.


Who should consider using OpenBox#

  • Those who want to tune hyper-parameters for their ML tasks automatically.

  • Those who want to find the optimal configuration for their configuration search tasks (e.g., database knob tuning).

  • Data platform owners who want to provide BBO service in their platform.

  • Researchers and data scientists who want to solve generalized BBO problems easily.


OpenBox capabilities#

OpenBox has a wide range of functionality scope, which includes:

  1. BBO with multiple objectives and constraints.

  2. BBO with transfer learning.

  3. BBO with distributed parallelization.

  4. BBO with multi-fidelity acceleration.

  5. BBO with early stops.

In the following, we provide a taxonomy of existing BBO systems:

System/Package

FIOC

Multi-obj.

Constraint

History

Distributed

Hyperopt

×

×

×

Spearmint

×

×

×

×

SMAC3

×

×

×

BoTorch

×

×

×

Ax

×

Optuna

×

GPflowOPT

×

×

×

Vizier

×

HyperMapper

×

×

HpBandSter

×

×

×

OpenBox

  • FIOC: Support different input variable types, including Float, Integer, Ordinal and Categorical.

  • Multi-obj.: Support optimizing multiple objectives.

  • Constraint: Support inequality constraints.

  • History: Support injecting prior knowledge from previous tasks into the current search (i.e. transfer learning).

  • Distributed: Support parallel evaluations in a distributed environment.

  • △ means the system cannot support it for general cases or requires additional dependencies.