Thinking Machines releases Inkling 2 Small weights under Apache 2.0 licence
Written by AI
The weights of Thinking Machines’ 594-billion-parameter Inkling 2 Small model are available to download from Hugging Face, and vLLM has supported the model since release day. In measurements by the company and Inferact, a single user reached about 630 tokens per second on four GB200 GPUs with speculative decoding.
Thinking Machines said on X on 12 November that the model weights for Inkling 2 Small are available to download from Hugging Face under an Apache 2.0 licence. Apache 2.0 is a permissive licence that also allows modification and commercial use.
Mira Murati of Thinking Machines wrote the same day that the weights were being released, as promised. Murati thanked the company’s team and the partners involved in safety testing, who were not named.
The model’s technical details and performance figures come from the vLLM project blog. vLLM is open-source software for serving large language models. The blog’s authors, Thinking Machines and Inferact, made the measurements themselves.
The model reads text and images
Inkling 2 Small is a mixture-of-experts (MoE) model. Each of its 48 layers has 384 experts, and eight of them process each token. The model has 594 billion parameters in total but uses 17 billion to process each token. Its context window is up to 1 million tokens. The model takes text and images as input and produces text. Its predecessor, Inkling, also accepted audio.
The expert weights are quantised: the first 40 layers in 4-bit NVFP4 format and the last eight in 8-bit MXFP8. This means the model can run on a single server node, the blog said.
The model ships with a DSpark draft model for speculative decoding. In speculative decoding, the draft model proposes several tokens at once, in this case seven, and the main model checks them in a single pass. Acceptance length is the average number of tokens accepted per step.
Measurements by Thinking Machines and Inferact
On four GB200 GPUs, a single user got about 630 tokens per second with an acceptance length of 3.5. The companies measured an agentic workload on eight GPUs across two nodes. In each turn, about 3,000 new tokens were added to the session’s long context, and the model generated 1,024 tokens with reasoning enabled. The table shows tokens per second per GPU when each user received 100 tokens per second.
| Session context | Tokens/s per GPU | Concurrent sessions | Cache hit rate | Acceptance length |
|---|---|---|---|---|
| 32K | c. 1,070 | c. 84 | 91% | 3.3 |
| 64K | c. 915 | c. 72 | 95% | 3.4 |
| 128K | c. 700 | c. 55 | 98% | 3.5 |
| 256K | c. 450 | c. 37 | 99% | 3.5 |
Source: vLLM blog, measurements by Thinking Machines and Inferact, with DSpark and the cache enabled.
The figures apply to a workload in which most of the context was already in the cache from earlier turns. The blog authors tested accuracy on the NIAH, MMMU-Pro and HLE benchmarks with DSpark enabled. They said the results matched runs without speculative decoding. The blog gives no scores, so it does not compare the model’s performance with other models.
The vLLM blog includes a launch command that splits the model across eight GPUs and enables DSpark. vLLM also supports LoRA fine-tuning for the model, as well as reinforcement learning through the SkyRL software.
Read more: Models