Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
-
Updated
May 16, 2025 - Python
#
efficient-attention
Here are 24 public repositories matching this topic...
Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch
-
Updated
Aug 14, 2026 - Python
Official implementation of 'Transformer-VQ: Linear-Time Transformers via Vector Quantization' (ICLR 2024)
-
Updated
Dec 4, 2023 - Python
Efficient Infinite Context Transformers with Infini-attention Pytorch Implementation + QwenMoE Implementation + Training Script + 1M context keypass retrieval
-
Updated
May 9, 2024 - Python
The official PyTorch implementation for CascadedGaze: Efficiency in Global Context Extraction for Image Restoration, TMLR'24.
-
Updated
Feb 13, 2025 - Python
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing (Technical Report)
associative-memory delta-net attention-mechanisms linear-attention efficient-attention kimi-linear gated-deltanet
-
Updated
Jul 10, 2026 - Python
Official repository for "SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space"
-
Updated
May 7, 2026 - Python
Pytorch implementation of "Compact Global Descriptor for Neural Networks" (CGD).
efficient pytorch convolutional-neural-networks attention-mechanism attention-model efficient-attention
-
Updated
Jan 9, 2025 - Python
Implementation of: Hydra Attention: Efficient Attention with Many Heads (https://arxiv.org/abs/2209.07484)
-
Updated
Jan 8, 2023 - Python
Official Implementation of SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024)
-
Updated
Jun 20, 2025 - Python
Two small-scale research threads with pre-registered falsifiable bars + adversarial referee audits: Prizma-Seq (a parameter-free quadratic delta-state sequence mixer, an efficient-attention-replacement candidate) and Prizma (backprop-free, fully-local continual learning).
machine-learning research deep-learning pytorch continual-learning efficient-attention sequence-model
-
Updated
Sep 16, 2026 - Python
DWARF v2 is a research prototype that expands on the original DWARF.
nlp machine-learning language-modeling pytorch language-model attention-mechanism inference-efficiency kv-cache linear-attention efficient-attention sparse-attention
-
Updated
Sep 15, 2026 - Python
O(N) attention with a bounded inference KV cache. D4 Daubechies wavelet field + content-gated Q·K gather at dyadic offsets.
nlp rust machine-learning deep-learning language-modeling pytorch wavelet language-model attention-mechanism wavelet-transform ablation inference-efficiency ablation-study kv-cache linear-attention efficient-attention sparse-attention wavelet-attention
-
Updated
Jun 9, 2026 - Python
HiCI: Hierarchical Construction-Integration for Long-Context Attention
memory transformers attention-mechanism hierarchical-attention efficient-attention llm long-context-modeling hici
-
Updated
May 9, 2026 - Python
Minimal implementation of Samba by Microsoft in PyTorch
language-model ssm pytorch-implementation efficient-attention llm long-context-modeling mamba-state-space-models
-
Updated
Nov 24, 2024 - Python
Forward-evaluation results preview for experimental token mixing in pretrained language models. Research in progress; implementation private.
-
Updated
Oct 13, 2026 - Python
S-grade efficient / sparse attention benchmark (6 variants, G1-G4 gates)
-
Updated
Oct 6, 2026 - Python
Unofficial PyTorch reproduction for Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality.
pytorch reproduction state-space-model sequence-modeling efficient-attention mamba2 unofficial-implementation
-
Updated
Jul 2, 2026 - Python
Log-linear sparse attention (coarse-to-fine hierarchical top-k with KV enrichment) plus a small pixel-space diffusion transformer, in MLX.
-
Updated
Oct 7, 2026 - Python
Code, machine-readable results, and manuscript for a two-axis design study of compressed-and-indexed attention in long-horizon time-series forecasting.
-
Updated
Oct 22, 2026 - Python
Add this topic to your repo
To associate your repository with the efficient-attention topic, visit your repo's landing page and select "manage topics."