Tune Search Algorithms (tune.search)
Contents
Tune Search Algorithms (tune.search)#
Tune’s Search Algorithms are wrappers around open-source optimization libraries for efficient hyperparameter selection.
Each library has a specific way of defining the search space - please refer to their documentation for more details.
Tune will automatically convert search spaces passed to Tuner to the library format in most cases.
You can utilize these search algorithms as follows:
from ray import train, tune
from ray.tune.search.optuna import OptunaSearch
def train_fn(config):
# This objective function is just for demonstration purposes
train.report({"loss": config["param"]})
tuner = tune.Tuner(
train_fn,
tune_config=tune.TuneConfig(
search_alg=OptunaSearch(),
num_samples=100,
metric="loss",
mode="min",
),
param_space={"param": tune.uniform(0, 1)},
)
results = tuner.fit()
Saving and Restoring Tune Search Algorithms#
Certain search algorithms have save/restore implemented,
allowing reuse of searchers that are fitted on the results of multiple tuning runs.
search_alg = HyperOptSearch()
tuner_1 = tune.Tuner(
train_fn,
tune_config=tune.TuneConfig(search_alg=search_alg)
)
results_1 = tuner_1.fit()
search_alg.save("./my-checkpoint.pkl")
# Restore the saved state onto another search algorithm,
# in a new tuning script
search_alg2 = HyperOptSearch()
search_alg2.restore("./my-checkpoint.pkl")
tuner_2 = tune.Tuner(
train_fn,
tune_config=tune.TuneConfig(search_alg=search_alg2)
)
results_2 = tuner_2.fit()
Tune automatically saves searcher state inside the current experiment folder during tuning.
See Result logdir: ... in the output logs for this location.
Note that if you have two Tune runs with the same experiment folder,
the previous state checkpoint will be overwritten. You can
avoid this by making sure air.RunConfig(name=...) is set to a unique
identifier:
search_alg = HyperOptSearch()
tuner_1 = tune.Tuner(
train_fn,
tune_config=tune.TuneConfig(
num_samples=5,
search_alg=search_alg,
),
run_config=air.RunConfig(
name="my-experiment-1",
storage_path="~/my_results",
)
)
results = tuner_1.fit()
search_alg2 = HyperOptSearch()
search_alg2.restore_from_dir(
os.path.join("~/my_results", "my-experiment-1")
)
Random search and grid search (tune.search.basic_variant.BasicVariantGenerator)#
The default and most basic way to do hyperparameter search is via random and grid search.
Ray Tune does this through the BasicVariantGenerator
class that generates trial variants given a search space definition.
The BasicVariantGenerator is used per
default if no search algorithm is passed to
Tuner.
Uses Tune's variant generation for resolving variables. |
Ax (tune.search.ax.AxSearch)#
|
Uses Ax to optimize hyperparameters. |
Bayesian Optimization (tune.search.bayesopt.BayesOptSearch)#
|
Uses fmfn/BayesianOptimization to optimize hyperparameters. |
BOHB (tune.search.bohb.TuneBOHB)#
BOHB (Bayesian Optimization HyperBand) is an algorithm that both terminates bad trials and also uses Bayesian Optimization to improve the hyperparameter search. It is available from the HpBandSter library.
Importantly, BOHB is intended to be paired with a specific scheduler class: HyperBandForBOHB.
In order to use this search algorithm, you will need to install HpBandSter and ConfigSpace:
$ pip install hpbandster ConfigSpace
See the BOHB paper for more details.
|
BOHB suggestion component. |
BlendSearch (tune.search.flaml.BlendSearch)#
BlendSearch is an economical hyperparameter optimization algorithm that combines combines local search with global search. It is backed by the FLAML library. It allows the users to specify a low-cost initial point as input if such point exists.
In order to use this search algorithm, you will need to install flaml:
$ pip install 'flaml[blendsearch]'
See the BlendSearch paper and documentation in FLAML BlendSearch documentation for more details.
alias of |
CFO (tune.search.flaml.CFO)#
CFO (Cost-Frugal hyperparameter Optimization) is a hyperparameter search algorithm based on randomized local search. It is backed by the FLAML library. It allows the users to specify a low-cost initial point as input if such point exists.
In order to use this search algorithm, you will need to install flaml:
$ pip install flaml
See the CFO paper and documentation in FLAML CFO documentation for more details.
alias of |
Dragonfly (tune.search.dragonfly.DragonflySearch)#
|
Uses Dragonfly to optimize hyperparameters. |
HEBO (tune.search.hebo.HEBOSearch)#
|
Uses HEBO (Heteroscedastic Evolutionary Bayesian Optimization) to optimize hyperparameters. |
HyperOpt (tune.search.hyperopt.HyperOptSearch)#
|
A wrapper around HyperOpt to provide trial suggestions. |
Nevergrad (tune.search.nevergrad.NevergradSearch)#
|
Uses Nevergrad to optimize hyperparameters. |
Optuna (tune.search.optuna.OptunaSearch)#
|
A wrapper around Optuna to provide trial suggestions. |
SigOpt (tune.search.sigopt.SigOptSearch)#
You will need to use the SigOpt experiment and space specification to specify your search space.
|
A wrapper around SigOpt to provide trial suggestions. |
Scikit-Optimize (tune.search.skopt.SkOptSearch)#
|
Uses Scikit Optimize (skopt) to optimize hyperparameters. |
ZOOpt (tune.search.zoopt.ZOOptSearch)#
|
A wrapper around ZOOpt to provide trial suggestions. |
Repeated Evaluations (tune.search.Repeater)#
Use ray.tune.search.Repeater to average over multiple evaluations of the same
hyperparameter configurations. This is useful in cases where the evaluated
training procedure has high variance (i.e., in reinforcement learning).
By default, Repeater will take in a repeat parameter and a search_alg.
The search_alg will suggest new configurations to try, and the Repeater
will run repeat trials of the configuration. It will then average the
search_alg.metric from the final results of each repeated trial.
Warning
It is recommended to not use Repeater with a TrialScheduler.
Early termination can negatively affect the average reported metric.
|
A wrapper algorithm for repeating trials of same parameters. |
ConcurrencyLimiter (tune.search.ConcurrencyLimiter)#
Use ray.tune.search.ConcurrencyLimiter to limit the amount of concurrency when using a search algorithm.
This is useful when a given optimization algorithm does not parallelize very well (like a naive Bayesian Optimization).
|
A wrapper algorithm for limiting the number of concurrent trials. |
Custom Search Algorithms (tune.search.Searcher)#
If you are interested in implementing or contributing a new Search Algorithm, provide the following interface:
|
Abstract class for wrapping suggesting algorithms. |
|
Queries the algorithm to retrieve the next set of parameters. |
|
Save state to path for this search algorithm. |
|
Restore state for this search algorithm |
|
Optional notification for result during training. |
|
Notification for the completion of trial. |
If contributing, make sure to add test cases and an entry in the function described below.
Shim Instantiation (tune.create_searcher)#
There is also a shim function that constructs the search algorithm based on the provided string. This can be useful if the search algorithm you want to use changes often (e.g., specifying the search algorithm via a CLI option or config file).
|
Instantiate a search algorithm based on the given string. |