SubQ, developed by the Miami-based startup Subquadratic, introduces a novel LLM architecture built around Sub-quadratic Sparse Attention (SSA). Rather than computing attention across every pair of tokens, SSA selectively focuses on the most relevant connections, dramatically cutting computational requirements.

The architecture achieves sub-quadratic scaling, meaning its computational cost grows much more slowly with input length compared to standard transformers. This makes it particularly well-suited for tasks involving long documents, codebases, or conversation histories.

SubQ represents a growing trend in the AI community: moving away from brute-force scaling toward smarter architectural choices that deliver more performance per compute dollar. If SSA delivers on its promises, it could influence how the next generation of LLMs is designed.

Source