Keras 3 API documentation
Models API Layers API The base Layer class Layer activations Layer weight initializers Layer weight regularizers Layer weight constraints Core layers Convolution layers Pooling layers Recurrent layers Preprocessing layers Normalization layers Regularization layers Attention layers Reshaping layers Merging layers Activation layers Backend-specific layers Callbacks API Ops API Optimizers Metrics Losses Data loading Built-in small datasets Keras Applications Mixed precision Multi-device distribution RNG API Quantizers Scope Rematerialization Utilities
Keras 2 API documentation
► Keras 3 API documentation / Layers API / Normalization layers / RMSNormalization layer

RMSNormalization layer

[source]

RMSNormalization class

keras.layers.RMSNormalization(axis=-1, epsilon=1e-06, **kwargs)

Root Mean Square (RMS) Normalization layer.

This layer normalizes the input tensor based on its RMS value.

The Keras layer performs the operation as described in Root Mean Square Layer Normalization by Biao Zhang et al.

If scale is enabled, the layer will scale the normalized outputs via a learnable scaling factor.

So, with scaling enabled, the normalization equations are as follows:

Let the intermediate activations for a mini-batch to be the inputs.

rms_normalization(x) = x * rsqrt(mean(square(x))) * scale

For example:

>>> layer = keras.layers.RMSNormalization() >>> layer.build([5, 20, 30, 10]) >>> print(layer.scale.shape) (10,) >>> layer(np.random.rand(1, 10)).numpy() array([[0.35098287, 1.0495652 , 1.4645109 , 1.2944688 , 0.31124955, 1.2768592 , 1.184331 , 0.17474432, 0.49955517, 1.2428929 ]], dtype=float32)

Arguments

  • axis: int. The axis on which to perform the normalization.
  • epsilon: float. A small number to add to avoid division by zero.
RMSNormalization layer
RMSNormalization class