New llama.cpp release runs Inkling 2 with text and images
llama.cpp pre-release b12406 can convert Thinking Machines Lab’s Inkling 2 model to GGUF format and run it with text and images. The conversion has not yet been tested on a real model file.
The llama.cpp project released version b12406 on GitHub on 15 November. llama.cpp is a program for running language models on your own computer, and GGUF is its file format for model weights. The changelog says the new version converts Inkling 2’s weights to GGUF, runs the model with text and reads still images. The Japanese site Signal Yomi and the news digest Briefed Today reported that GitHub marks the release as a pre-release.
Signal Yomi reported that Thinking Machines Lab’s model card describes Inkling 2 as taking text and images as input and responding with text. The company announced on 12 November that the Inkling 2 Small weights were available for download, the site said.
Support is limited to single images. The developers removed video frame averaging and the audio support they had initially added, and left alpha channel handling for later. Other changes include flash attention computation on CUDA GPUs.
Version b12413, released the same day, fixes the placement of the image start and end tokens in the input. A maintainer said that before the fix an empty text block was inserted before the image, Signal Yomi reported. The order now matches the model’s reference template. The site recommends b12413 or later for experimenting with images and notes that no measurements of image recognition accuracy or speed have been published.
The converter decodes quantised weights in small chunks
Inkling 2 is a mixture-of-experts (MoE) model: it consists of many expert networks, and each token uses only some of them. In the original file, the expert weights are quantised in the NVFP4 and MXFP8 formats, in which each number takes only a few bits. The converter decodes them into 32-bit floating-point numbers, 128 rows per expert at a time.
The changelog says a w13 weight chunk takes 2.25 MiB, whereas decoding a whole layer at once would take 30.375 GiB. The developers estimate the memory requirement at about 16 MiB per weight matrix, excluding runtime memory, other tensors and the operating system’s cache.
GGUF weights converted from quantised synthetic weights matched weights decoded separately with vLLM bit for bit. In logits computed on the CPU – the scores the model gives to candidate next tokens – the absolute error was at most 3.1e-6, and the most likely next token was the same in every case. Converting a real model file has not been tested, the changelog says.
Ready-made files and memory figures
Signal Yomi reported that Unsloth distributes pre-quantised GGUF files on Hugging Face. The model file alone is enough for text. Images also require a visual projector file, which converts images into a form the model can process and is passed at launch with the mmproj option. Unsloth’s table lists the model file for the smallest quantisation, UD-IQ1_S, at 171.7 GB.
Thinking Machines Lab’s model card says running the official NVFP4 version requires at least 500 GB of GPU memory in total, the site said. Signal Yomi notes that the figures measure different things. Neither shows how much memory running the GGUF model with llama.cpp requires.
Pre-built binaries for macOS, Linux, Windows and Android can be downloaded from the llama.cpp GitHub releases page.
Read more: Models