Pandaily, which covers China’s technology sector, reported on 8 November that Huawei’s Ascend computing team has adapted Moonshot AI’s Kimi K3.1 Pro for LoRA fine-tuning. Pandaily draws on a Guandian news story and a Chinese-language technical post published on 6 and 7 November. The team says it has validated training on a 128-card Atlas 950 SuperPoD.

LoRA is a fine-tuning method that keeps a model’s original weights unchanged and trains a small set of new parameters alongside them. Training is a separate stage from inference, which means running a finished model. Ascend announced inference support on the day Moonshot released the model weights. Pandaily reported that the weights were released on 4 November; Asia Daily reported that the release announcement came early in the morning of 5 November, Beijing time.

Kimi K3.1 Pro is an open-weight mixture-of-experts (MoE) model that handles images as well as text. It has about 2.8 trillion parameters. Pandaily reported 896 routed experts, 16 of which are activated per token. The context window is about 1 million tokens.

Ascend spread the memory load across accelerators

The BF16 weights alone occupy roughly 5.6TB, Pandaily reported. Training also consumes memory for gradients, optimiser states and intermediate results from long inputs. Ascend used FSDP2 to shard parameters, gradients and optimiser states across the accelerators, and distributed the experts across different accelerators. Validation used an FSDP2 plus EP64 configuration.

The model’s attention mechanism, the way it weights different parts of the input, is a hybrid. About three quarters of the attention blocks use KDA linear attention, alongside gated sparse attention. The team says it built large fused kernels for KDA attention and used fused AscendC operators for sparse attention. It claims these are more than twice as fast as the earlier implementation built from small operators.

A selective cache called Swap Core stores the results of selected matrix multiplications in tiered HBM and DDR memory. Ascend said the cache sped up training by an average of 15 per cent in the current test configuration. The team did not publish absolute throughput, and the figure has not been independently verified.

Loss fell in both runs

The team published loss curves from the 128-card runs. In training from random initialisation, loss fell from about 13.5 to about 1.7 over more than 400 steps. In LoRA fine-tuning from the released weights, loss fell from about 1.85 over more than 500 steps and settled at about 0.27. The curves show that training progressed, but the sources contain no test results on the quality of the fine-tuned model.

Ascend said it got the model working within a few days because AI agents guided the development work.

Moonshot distributes the weights on Hugging Face under its own licence governing commercial use, Asia Daily reported. The company has not published its training code or original training data. The sources do not say whether Ascend’s training implementation is available to others.