# The Wiola Architecture for Efficient Small Language Models

Source: [The Wiola Architecture for Efficient Small Language Models](<https://www.sota2.com/research/paper/6a4741e2143ebdcacacc59c6>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-04T05:01:04.041Z

Authors: Aryuemaan Kumar Chowdhury, Afreen Shaik, Yaparla Bhargavi, Brahma Kumar

Published: 2026-07-01T18:52:56.000Z

## Abstract

We present Wiola, a fully original Small Language Model (SLM) architecture built from first principles, sharing no structural lineage with any existing model family including GPT, LLaMA, Mistral, or Falcon. Wiola introduces five independently novel components: (i) Spiral Rotary Positional Encoding (SRPE), which embeds token positions on a three-dimensional helical manifold combining absolute, relative, and hierarchical positional signals; (ii) Gated Cross-Layer Attention (GCLA), providing each decoder layer with soft cross-attention access to compressed summaries of two preceding layers for inter-layer coherence; (iii) Adaptive Token Merging (ATM), which dynamically merges se mantically redundant adjacent tokens in middle network layers to reduce attention complexity without information loss; (iv) Dual Stream Feed-Forward (DSFF), replacing the conventional MLP with two parallel streams fused by a learned per-dimension gate; and (v) WiolaRMSNorm, a modified normalisation introducing a per-dimension learned offset vector that prevents representation collapse. We provide complete mathematical derivations, architectural block diagrams, complexity analyses, and systematic comparisons against GPT-2, LLaMA-2, and Mistral. Wiola is released in four sizes (120M, 360M, 700M, and 1.5B parameters) and is fully compatible with the HuggingFace Transformers ecosystem, with all 22 architectural unit tests passing.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [KV-Cache Footprint Analysis on Sequence length T=2048 BF16 (inference)](<https://www.sota2.com/research/sota/kv-cache-footprint-analysis-on-sequence-length-t-2048-bf16-inference.md>): KV-Cache Footprint Analysis; Sequence length T=2048 BF16 (inference); KV Cache Size (MB): 67.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2607.01394>)

[PDF](<https://arxiv.org/pdf/2607.01394.pdf>)
