Skip to contents

Layer Normalization

Source: R/PipeOpTorchLayerNorm.R
mlr_pipeops_nn_layer_norm.Rd

Applies Layer Normalization for last certain number of dimensions.

nn_module

Calls torch::nn_layer_norm() when trained. The parameter normalized_shape is inferred as the dimensions of the last dims dimensions of the input shape.

Parameters

  • dims :: integer(1)
    The number of dimensions over which will be normalized (starting from the last dimension).

  • elementwise_affine :: logical(1)
    Whether to learn affine-linear parameters initialized to 1 for weights and to 0 for biases. The default is TRUE.

  • eps :: numeric(1)
    A value added to the denominator for numerical stability.

State

The state is the value calculated by the public method $shapes_out().

Input and Output Channels

One input channel called "input" and one output channel called "output". For an explanation see PipeOpTorch.

See also

Other PipeOps: mlr_pipeops_nn_adaptive_avg_pool1d, mlr_pipeops_nn_adaptive_avg_pool2d, mlr_pipeops_nn_adaptive_avg_pool3d, mlr_pipeops_nn_avg_pool1d, mlr_pipeops_nn_avg_pool2d, mlr_pipeops_nn_avg_pool3d, mlr_pipeops_nn_batch_norm1d, mlr_pipeops_nn_batch_norm2d, mlr_pipeops_nn_batch_norm3d, mlr_pipeops_nn_block, mlr_pipeops_nn_celu, mlr_pipeops_nn_conv1d, mlr_pipeops_nn_conv2d, mlr_pipeops_nn_conv3d, mlr_pipeops_nn_conv_transpose1d, mlr_pipeops_nn_conv_transpose2d, mlr_pipeops_nn_conv_transpose3d, mlr_pipeops_nn_dropout, mlr_pipeops_nn_elu, mlr_pipeops_nn_flatten, mlr_pipeops_nn_ft_cls, mlr_pipeops_nn_ft_transformer_block, mlr_pipeops_nn_geglu, mlr_pipeops_nn_gelu, mlr_pipeops_nn_glu, mlr_pipeops_nn_hardshrink, mlr_pipeops_nn_hardsigmoid, mlr_pipeops_nn_hardtanh, mlr_pipeops_nn_head, mlr_pipeops_nn_identity, mlr_pipeops_nn_leaky_relu, mlr_pipeops_nn_linear, mlr_pipeops_nn_log_sigmoid, mlr_pipeops_nn_max_pool1d, mlr_pipeops_nn_max_pool2d, mlr_pipeops_nn_max_pool3d, mlr_pipeops_nn_merge, mlr_pipeops_nn_merge_cat, mlr_pipeops_nn_merge_prod, mlr_pipeops_nn_merge_sum, mlr_pipeops_nn_prelu, mlr_pipeops_nn_reglu, mlr_pipeops_nn_relu, mlr_pipeops_nn_relu6, mlr_pipeops_nn_reshape, mlr_pipeops_nn_rrelu, mlr_pipeops_nn_selu, mlr_pipeops_nn_sigmoid, mlr_pipeops_nn_softmax, mlr_pipeops_nn_softplus, mlr_pipeops_nn_softshrink, mlr_pipeops_nn_softsign, mlr_pipeops_nn_squeeze, mlr_pipeops_nn_tanh, mlr_pipeops_nn_tanhshrink, mlr_pipeops_nn_threshold, mlr_pipeops_nn_tokenizer_categ, mlr_pipeops_nn_tokenizer_num, mlr_pipeops_nn_unsqueeze, mlr_pipeops_torch_ingress, mlr_pipeops_torch_ingress_categ, mlr_pipeops_torch_ingress_ltnsr, mlr_pipeops_torch_ingress_num, mlr_pipeops_torch_loss, mlr_pipeops_torch_model, mlr_pipeops_torch_model_classif, mlr_pipeops_torch_model_regr

Super classes

mlr3pipelines::PipeOp -> PipeOpTorch -> PipeOpTorchLayerNorm

Methods

Public methods

Inherited methods

PipeOpTorchLayerNorm$new()

Creates a new instance of this R6 class.

Usage

PipeOpTorchLayerNorm$new(id = "nn_layer_norm", param_vals = list())

Arguments

id

(character(1))
Identifier of the resulting object.

param_vals

(list())
List of hyperparameter settings, overwriting the hyperparameter settings that would otherwise be set during construction.

PipeOpTorchLayerNorm$clone()

The objects of this class are cloneable with this method.

Usage

PipeOpTorchLayerNorm$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

# Construct the PipeOp
pipeop = po("nn_layer_norm", dims = 1)
pipeop
#> 
#> ── PipeOp <nn_layer_norm>: not trained ─────────────────────────────────────────
#> Values: dims=1
#> 
#> ── Input channels: 
#>    name           train predict
#>  <char>          <char>  <char>
#>   input ModelDescriptor    Task
#> 
#> ── Output channels: 
#>    name           train predict
#>  <char>          <char>  <char>
#>  output ModelDescriptor    Task
# The available parameters
pipeop$param_set
#> <ParamSet(3)>
#>                    id    class lower upper nlevels        default  value
#>                <char>   <char> <num> <num>   <num>         <list> <list>
#> 1:               dims ParamInt     1   Inf     Inf <NoDefault[0]>      1
#> 2: elementwise_affine ParamLgl    NA    NA       2           TRUE [NULL]
#> 3:                eps ParamDbl     0   Inf     Inf          1e-05 [NULL]