What comes after attention? This startup says it already knows.
Notes de l’épisode
Subquadratic is beginning to back up its ambitious claims with benchmarks and third-party validation for its SubQ 1.1 Small model, which uses its proprietary Sparse Attention (SSA) architecture to dramatically improve long-context performance. Rather than comparing every token to every other token, SSA selectively processes relationships, enabling near-linear scaling while maintaining high accuracy across context windows of up to 12 million tokens. The company reports near-perfect retrieval performance, competitive coding and reasoning benchmarks, and compute savings of up to 1,000x at maximum context lengths.
Rather than targeting frontier models immediately, Subquadratic is focusing on enterprise customers that need efficient analysis of massive datasets. The current model was built by replacing the dense attention mechanism in an existing open-weight model and then continuing long-context pretraining. Looking ahead, the startup plans to release a larger mid-tier model while continuing research into "zero attention" architectures that could eliminate attention mechanisms altogether, with the long-term goal of surpassing today's transformer-based AI models in both efficiency and capability.
Learn more from The New Stack around cloud spending:
The context window has been shattered: Subquadratic debuts a 12-million-token window
What comes after attention? This startup says it already knows.
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.