SwigluFeedForward

class rectools.models.nn.transformers.net_blocks.SwigluFeedForward(n_factors: int, n_factors_ff: int, dropout_rate: float, bias: bool = True)[source]

Bases: Module

Feed-Forward network to introduce nonlinearity into the transformer model. This implementation is based on FuXi and LLama SwigLU https://arxiv.org/pdf/2502.03036, LiGR https://arxiv.org/pdf/2502.03417

Parameters

n_factors (int) – Latent embeddings size.
n_factors_ff (int) – How many hidden units to use in the network.
dropout_rate (float) – Probability of a hidden unit to be zeroed.
bias (bool, default True) – If True, add bias to linear layers.

Methods

forward(seqs)

Forward pass.

Attributes

forward(seqs: Tensor) → Tensor[source]

Forward pass.

Parameters: seqs (torch.Tensor) – User sequences of item embeddings.
Returns: User sequence that passed through all layers.
Return type: torch.Tensor