Marin releases all 57 checkpoints of its 535-billion-parameter Mochi model
The Marin project has released 57 pretraining checkpoints of the Mochi 535B-A23B model on Hugging Face, about 61 terabytes of weights in total. According to the model card, the base model has not been properly tested or evaluated and is not a finished chat model.
On 17 November, Marin released all of Mochi’s intermediate checkpoints on Hugging Face. These are the copies of the model weights saved during pretraining. The model card says they cover training steps 6,000 to 393,491. Each checkpoint sits in its own branch, and the main branch holds the final one.
Mochi is a mixture-of-experts (MoE) model, so only some of its parameters process each token. It has about 535 billion stored parameters. According to the model card, about 22.76 billion non-embedding parameters are active per token, and the A23B in the name is this figure rounded to the nearest billion. The small active count does not reduce storage needs, however: a single checkpoint takes up about 1.07 terabytes in BF16. @dlwh, who announced the release on X, wrote that the checkpoints total about 61 terabytes and that the model was trained on 18.2 trillion tokens.
The checkpoints also show the stages of training. The context window is 4K tokens in the early stages and 16K later, the model card says. Steps 378,000, 384,000 and 390,251 use a 65K window, and steps 390,500–393,491 a 262K window. @dlwh wrote that 10 checkpoints from the long-context extension have been released.
Sebastian Raschka wrote on X on 19 November that the model’s technical report justifies its architectural components in terms of training stability and GPU efficiency. Raschka said latent MoE began to work better once RMSNorm normalisation was added to the latent representation. Raschka also highlighted a 3:1 ratio of local to global attention, short convolutions and exclusive self-attention (XSA). According to an image in Raschka’s post, the model has 48 layers and 384 experts, eight of which are selected for each token.
The model card describes Mochi as a research artefact that has not been properly tested or evaluated and may not be safe. As a base model, it is used for continuing text, not for chat. The bundled tokeniser is the version used by the Hero training run’s own text-generation settings. The training run recorded the tokeniser repository but not its exact version.
Running the model requires Marin’s vLLM branch and its split-expert loader. The final checkpoints also need support for global-layer query scaling, and older loaders do not accept them. Full-model numerical parity, text generation, standard Transformers loading and the run instructions have not been validated. Loader support alone does not prove numerical parity, the model card says. The weights are on Hugging Face as open-athena/Marin-Mochi-535B-A23B under the OpenMDW 1.1 licence.
Read more: Models