Single Transformer Block for the FT-Transformer

A transformer block consisting of a multi-head self-attention mechanism followed by a feed-forward network.

This is used in LearnerTorchFTTransformer.

nn_module

Calls nn_ft_transformer_block() when trained.

State

The state is the value calculated by the public method $shapes_out().

Super classes

mlr3pipelines::PipeOp -> PipeOpTorch -> PipeOpTorchFTTransformerBlock

Methods

Public methods

• PipeOpTorchFTTransformerBlock$new()

• PipeOpTorchFTTransformerBlock$clone()

Inherited methods

• mlr3pipelines::PipeOp$help()
• mlr3pipelines::PipeOp$predict()
• mlr3pipelines::PipeOp$print()
• mlr3pipelines::PipeOp$train()
• PipeOpTorch$shapes_out()

---

PipeOpTorchFTTransformerBlock$new()

Create a new instance of this R6 class.

Usage

PipeOpTorchFTTransformerBlock$new(
  id = "nn_ft_transformer_block",
  param_vals = list()
)

Arguments

id

(character(1))
Identifier of the resulting object.

param_vals

(list())
List of hyperparameter settings, overwriting the hyperparameter settings that would otherwise be set during construction.

---

PipeOpTorchFTTransformerBlock$clone()

The objects of this class are cloneable with this method.

Usage

PipeOpTorchFTTransformerBlock$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

# Construct the PipeOp
pipeop = nn("ft_transformer_block")
pipeop
#> 
#> ── PipeOp <ft_transformer_block>: not trained ──────────────────────────────────
#> Values: attention_n_heads=8, attention_dropout=0.2,
#> attention_initialization=kaiming, attention_normalization=<nn_layer_norm>,
#> ffn_dropout=0.1, ffn_activation=<nn_reglu>, ffn_normalization=<nn_layer_norm>,
#> residual_dropout=0, prenormalization=TRUE, is_first_layer=FALSE,
#> query_idx=<NULL>, attention_bias=TRUE, ffn_bias_first=TRUE,
#> ffn_bias_second=TRUE
#> 
#> ── Input channels: 
#>    name           train predict
#>  <char>          <char>  <char>
#>   input ModelDescriptor    Task
#> 
#> ── Output channels: 
#>    name           train predict
#>  <char>          <char>  <char>
#>  output ModelDescriptor    Task
# The available parameters
pipeop$param_set
#> <ParamSet(16)>
#>                           id    class lower upper nlevels        default
#>                       <char>   <char> <num> <num>   <num>         <list>
#>  1:        attention_n_heads ParamInt     1   Inf     Inf <NoDefault[0]>
#>  2:        attention_dropout ParamDbl     0     1     Inf <NoDefault[0]>
#>  3: attention_initialization ParamFct    NA    NA       2 <NoDefault[0]>
#>  4:  attention_normalization ParamUty    NA    NA     Inf <NoDefault[0]>
#>  5:             ffn_d_hidden ParamInt     1   Inf     Inf <NoDefault[0]>
#>  6:  ffn_d_hidden_multiplier ParamDbl     0   Inf     Inf <NoDefault[0]>
#>  7:              ffn_dropout ParamDbl     0     1     Inf <NoDefault[0]>
#>  8:           ffn_activation ParamUty    NA    NA     Inf <NoDefault[0]>
#>  9:        ffn_normalization ParamUty    NA    NA     Inf <NoDefault[0]>
#> 10:         residual_dropout ParamDbl     0     1     Inf <NoDefault[0]>
#> 11:         prenormalization ParamLgl    NA    NA       2 <NoDefault[0]>
#> 12:           is_first_layer ParamLgl    NA    NA       2 <NoDefault[0]>
#> 13:                query_idx ParamUty    NA    NA     Inf <NoDefault[0]>
#> 14:           attention_bias ParamLgl    NA    NA       2 <NoDefault[0]>
#> 15:           ffn_bias_first ParamLgl    NA    NA       2 <NoDefault[0]>
#> 16:          ffn_bias_second ParamLgl    NA    NA       2 <NoDefault[0]>
#>                  value
#>                 <list>
#>  1:                  8
#>  2:                0.2
#>  3:            kaiming
#>  4: <nn_layer_norm[1]>
#>  5:             [NULL]
#>  6:             [NULL]
#>  7:                0.1
#>  8:      <nn_reglu[1]>
#>  9: <nn_layer_norm[1]>
#> 10:                  0
#> 11:               TRUE
#> 12:              FALSE
#> 13:             [NULL]
#> 14:               TRUE
#> 15:               TRUE
#> 16:               TRUE