AI Aulis

AI NEWS IN ENGLISH · 16 November 2026

The most important AI news every morning. AI writes and checks the stories against the original sources – check important facts in the sources.

24 stories · next issue tomorrow morning · How the paper is made →

← Issue 16 November 2026Paper

Raschka: RoPE in all layers is still the most common positional embedding, but its share has fallen

AI-generated contentWritten by AI

magazine.sebastianraschka.com, x.com · 2 sources · Sources published: 15 November 2026

In an article, Sebastian Raschka explains how RoPE tells a language model how far apart tokens are. In Raschka’s gallery, RoPE in all layers is still the most common approach, but partial RoPE and leaving RoPE out of some layers have become more common. The groups are still small.

Machine learning researcher Sebastian Raschka published a tutorial on 15 November in his Ahead of AI newsletter. It explains how large language models get information about the order of tokens. Raschka also compares the approaches used by models released this year.

Without positional information, the same word gets the same vector wherever it appears. Raschka wrote that the causal mask provides some order information, because it lets a token attend only to itself and earlier tokens. The mask cannot, however, tell a distance of one token from a distance of ten.

Rotation encodes distance

GPT-2 adds a learned position vector to each token’s vector, taken from a table with 1,024 positions. A longer context would need a larger table. In Raschka’s calculation, the table makes up 0.30 per cent of GPT-2 Medium’s model weights, but more than 22 per cent at 100,000 positions.

RoPE (Rotary Position Embedding) works inside the attention mechanism. Attention computes a query, key and value vector for each token. The dot product of a query and a key decides how much weight one token gives another. RoPE rotates pairs of numbers in the query and key by an angle equal to the token’s position multiplied by the pair’s frequency. It does not rotate the value vectors.

The dot product depends on the angle between the vectors. The difference between RoPE’s rotations depends only on the distance between the tokens. If 100 tokens are added in front of a sentence, both vectors rotate by the same extra amount, so their dot product does not change. Raschka notes that the dot product stays the same only if the unrotated vectors stay the same; in a real model, the added tokens change those too.

New models cut back on RoPE

Partial RoPE rotates only some of the dimensions. MiniMax-M3, for example, rotates 64 of the 128 dimensions in each attention head. This saves memory for the precomputed sine and cosine tables, Raschka said, but the key-value cache stays full-size. In the Fractional Rotation, Full Potential? paper, rotating around 10 per cent often gave models with 1 billion to 8 billion parameters similar training loss and benchmark accuracy to full RoPE. Full RoPE did better on some evaluations. Raschka thinks partial RoPE is hardly worth the effort.

A NoPE layer has no separate positional embedding; the model relies on the information the causal mask provides. SmolLM3 leaves RoPE out of every fourth layer, and its authors said this improved long-context performance. Thinky’s Inkling instead adds a learned, distance-dependent value to the attention scores.

In an image in his post on X, Raschka has sorted the models in his LLM Architecture Gallery by positional embedding. RoPE in all layers is still the most common, but its share has fallen.

Period Models RoPE in all layers RoPE + NoPE NoPE only Other
2025 32 27 2 2 1
2026, earlier 60 47 5 7 1
2026, later 21 12 5 4 0

Source: Raschka’s gallery as of 9 November 2026. Each gallery card is counted once, so versions of the same architecture may be overrepresented.

Raschka said the groups are still so small that it is hard to judge whether the trends will continue. In both 2026 groups, most of the models that use RoPE use partial RoPE. Raschka expects RoPE to remain dominant in 2027 as well.

Report an error in this story