candle-layer-norm
| Crates.io | candle-layer-norm |
| lib.rs | candle-layer-norm |
| version | 0.0.1 |
| created_at | 2025-04-15 10:25:33.627577+00 |
| updated_at | 2025-04-15 10:25:33.627577+00 |
| description | Layer Norm layer for the candle ML framework. |
| homepage | https://github.com/huggingface/candle-extensions/candle-layer-norm/ |
| repository | |
| max_upload_size | |
| id | 1634257 |
| size | 137,550 |
Nicolas Patry (Narsil)
documentation
https://docs.rs/candle-layer-norm
README
Candle Cuda Layer Norm
Layer Norm fused operation for the Candle ML framework.
This Layer was adapted from https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm.
It implements fused dropout + residual + LayerNorm, building on Apex's FastLayerNorm.
Major changes:
- Add residual.
- Make it work for both pre-norm and post-norm architecture.
- Support more hidden dimensions (all dimensions divisible by 8, up to 8192).
- Implement RMSNorm as an option.