This wraps a torch::torch_optimizer_generator and annotates it with metadata, most importantly a ParamSet.
The optimizer is created for the given parameter values by calling the $generate() method.
This class is usually used to configure the optimizer of a torch learner, e.g.
when constructing a learner or in a ModelDescriptor.
For a list of available optimizers, see mlr3torch_optimizers.
Items from this dictionary can be retrieved using t_opt().
Parameters
Defined by the constructor argument param_set.
If no parameter set is provided during construction, the parameter set is constructed by creating a parameter
for each argument of the wrapped optimizer, where the parameters are then of type ParamUty.
In addition, every TorchOptimizer has the parameter:
param_groups::function(params) -> list()
A function that receives the namedlist()of network parameters and returns alist()of parameter groups, in the format expected by the wrappedtorchoptimizer. This makes it possible to optimize different parts of the network differently, e.g. to give the first layer a different learning rate than the rest:param_groups = function(params) list( list(params = params[startsWith(names(params), "0.")], lr = 0.9), list(params = params[!startsWith(names(params), "0.")], lr = 0.001) )When a
LearnerTorchis configured, this is reachable asopt.param_groups. Default isNULL, i.e. all parameters form a single group.
See also
Other Torch Descriptor:
TorchCallback,
TorchDescriptor,
TorchLoss,
as_torch_callbacks(),
as_torch_loss(),
as_torch_optimizer(),
mlr3torch_losses,
mlr3torch_optimizers,
t_clbk(),
t_loss(),
t_opt()
Super class
TorchDescriptor -> TorchOptimizer
Methods
Inherited methods
TorchOptimizer$new()
Creates a new instance of this R6 class.
Usage
TorchOptimizer$new(
torch_optimizer,
param_set = NULL,
id = NULL,
label = NULL,
packages = NULL,
man = NULL
)Arguments
torch_optimizer(
torch_optimizer_generator)
The torch optimizer.param_set(
ParamSetorNULL)
The parameter set. IfNULL(default) it is inferred fromtorch_optimizer.id(
character(1))
The id for of the new object.label(
character(1))
Label for the new instance.packages(
character())
The R packages this object depends on.man(
character(1))
String in the format[pkg]::[topic]pointing to a manual page for this object. The referenced help package can be opened via method$help().
TorchOptimizer$generate()
Instantiates the optimizer.
Arguments
params(named
list()oftorch_tensors)
The parameters of the network.
Examples
# Create a new torch optimizer
torch_opt = TorchOptimizer$new(optim_ignite_adam, label = "adam")
torch_opt
#> <TorchOptimizer:optim_ignite_adam> adam
#> * Generator: optim_ignite_adam
#> * Parameters: list()
#> * Packages: torch,mlr3torch
# If the param set is not specified, parameters are inferred but are of class ParamUty
torch_opt$param_set
#> <ParamSet(6)>
#> id class lower upper nlevels default value
#> <char> <char> <num> <num> <num> <list> <list>
#> 1: lr ParamUty NA NA Inf <NoDefault[0]> [NULL]
#> 2: betas ParamUty NA NA Inf <NoDefault[0]> [NULL]
#> 3: eps ParamUty NA NA Inf <NoDefault[0]> [NULL]
#> 4: weight_decay ParamUty NA NA Inf <NoDefault[0]> [NULL]
#> 5: amsgrad ParamUty NA NA Inf <NoDefault[0]> [NULL]
#> 6: param_groups ParamUty NA NA Inf <NoDefault[0]> [NULL]
# open the help page of the wrapped optimizer
# torch_opt$help()
# Retrieve an optimizer from the dictionary
torch_opt = t_opt("sgd", lr = 0.1)
torch_opt
#> <TorchOptimizer:sgd> Stochastic Gradient Descent
#> * Generator: optim_ignite_sgd
#> * Parameters: lr=0.1
#> * Packages: torch,mlr3torch
torch_opt$param_set
#> <ParamSet(6)>
#> id class lower upper nlevels default value
#> <char> <char> <num> <num> <num> <list> <list>
#> 1: lr ParamDbl 0 Inf Inf <NoDefault[0]> 0.1
#> 2: momentum ParamDbl 0 1 Inf 0 [NULL]
#> 3: dampening ParamDbl 0 1 Inf 0 [NULL]
#> 4: weight_decay ParamDbl 0 Inf Inf 0 [NULL]
#> 5: nesterov ParamLgl NA NA 2 FALSE [NULL]
#> 6: param_groups ParamUty NA NA Inf <NoDefault[0]> [NULL]
torch_opt$label
#> [1] "Stochastic Gradient Descent"
torch_opt$id
#> [1] "sgd"
# Create the optimizer for a network
net = nn_linear(10, 1)
opt = torch_opt$generate(net$parameters)
# is the same as
optim_sgd(net$parameters, lr = 0.1)
#> <optim_sgd>
#> Inherits from: <torch_optimizer>
#> Public:
#> add_param_group: function (param_group)
#> clone: function (deep = FALSE)
#> defaults: list
#> initialize: function (params, lr = optim_required(), momentum = 0, dampening = 0,
#> load_state_dict: function (state_dict, ..., .refer_to_state_dict = FALSE)
#> param_groups: list
#> state: State, R6
#> state_dict: function ()
#> step: function (closure = NULL)
#> zero_grad: function (set_to_none = FALSE)
#> Private:
#> deep_clone: function (name, value)
#> step_helper: function (closure, loop_fun)
# Use in a learner
learner = lrn("regr.mlp", optimizer = t_opt("sgd"))
# The parameters of the optimizer are added to the learner's parameter set
learner$param_set
#> <ParamSetCollection(39)>
#> id class lower upper nlevels default
#> <char> <char> <num> <num> <num> <list>
#> 1: epochs ParamInt 0 Inf Inf <NoDefault[0]>
#> 2: device ParamFct NA NA 12 <NoDefault[0]>
#> 3: num_threads ParamInt 1 Inf Inf <NoDefault[0]>
#> 4: num_interop_threads ParamInt 1 Inf Inf <NoDefault[0]>
#> 5: seed ParamInt -Inf Inf Inf <NoDefault[0]>
#> 6: eval_freq ParamInt 1 Inf Inf <NoDefault[0]>
#> 7: measures_train ParamUty NA NA Inf <NoDefault[0]>
#> 8: measures_valid ParamUty NA NA Inf <NoDefault[0]>
#> 9: patience ParamInt 0 Inf Inf <NoDefault[0]>
#> 10: min_delta ParamDbl 0 Inf Inf <NoDefault[0]>
#> 11: restore_best_weights ParamLgl NA NA 2 <NoDefault[0]>
#> 12: batch_size ParamInt 1 Inf Inf <NoDefault[0]>
#> 13: batch_size_predict ParamInt 1 Inf Inf <NoDefault[0]>
#> 14: shuffle ParamLgl NA NA 2 FALSE
#> 15: sampler ParamUty NA NA Inf <NoDefault[0]>
#> 16: batch_sampler ParamUty NA NA Inf <NoDefault[0]>
#> 17: num_workers ParamInt 0 Inf Inf 0
#> 18: collate_fn ParamUty NA NA Inf [NULL]
#> 19: pin_memory ParamLgl NA NA 2 FALSE
#> 20: drop_last ParamLgl NA NA 2 FALSE
#> 21: timeout ParamDbl -Inf Inf Inf -1
#> 22: worker_init_fn ParamUty NA NA Inf <NoDefault[0]>
#> 23: worker_globals ParamUty NA NA Inf <NoDefault[0]>
#> 24: worker_packages ParamUty NA NA Inf <NoDefault[0]>
#> 25: tensor_dataset ParamFct NA NA 1 <NoDefault[0]>
#> 26: jit_trace ParamLgl NA NA 2 <NoDefault[0]>
#> 27: neurons ParamUty NA NA Inf <NoDefault[0]>
#> 28: p ParamDbl 0 1 Inf <NoDefault[0]>
#> 29: n_layers ParamInt 1 Inf Inf <NoDefault[0]>
#> 30: activation ParamUty NA NA Inf <NoDefault[0]>
#> 31: activation_args ParamUty NA NA Inf <NoDefault[0]>
#> 32: shape ParamUty NA NA Inf <NoDefault[0]>
#> 33: opt.lr ParamDbl 0 Inf Inf <NoDefault[0]>
#> 34: opt.momentum ParamDbl 0 1 Inf 0
#> 35: opt.dampening ParamDbl 0 1 Inf 0
#> 36: opt.weight_decay ParamDbl 0 Inf Inf 0
#> 37: opt.nesterov ParamLgl NA NA 2 FALSE
#> 38: opt.param_groups ParamUty NA NA Inf <NoDefault[0]>
#> 39: loss.reduction ParamFct NA NA 2 mean
#> id class lower upper nlevels default
#> <char> <char> <num> <num> <num> <list>
#> value
#> <list>
#> 1: [NULL]
#> 2: auto
#> 3: 1
#> 4: [NULL]
#> 5: random
#> 6: 1
#> 7: <list[0]>
#> 8: <list[0]>
#> 9: 0
#> 10: 0
#> 11: FALSE
#> 12: [NULL]
#> 13: [NULL]
#> 14: TRUE
#> 15: [NULL]
#> 16: [NULL]
#> 17: [NULL]
#> 18: [NULL]
#> 19: [NULL]
#> 20: [NULL]
#> 21: [NULL]
#> 22: [NULL]
#> 23: [NULL]
#> 24: [NULL]
#> 25: FALSE
#> 26: FALSE
#> 27:
#> 28: 0.1
#> 29: [NULL]
#> 30: <nn_relu[1]>
#> 31: <list[0]>
#> 32: [NULL]
#> 33: [NULL]
#> 34: [NULL]
#> 35: [NULL]
#> 36: [NULL]
#> 37: [NULL]
#> 38: [NULL]
#> 39: [NULL]
#> value
#> <list>