To do this, we are discovering and assembling the knowledge required to
build artificial intelligence and
returning it to the world.
We've completed pre-training of Mochi, a
535 billion parameter mixture-of-experts model
with 23 billion active parameters, on 18.2 trillion tokens of data. Explore the training run
here.
Selected writing:
All Marin writing
8B dense retro
32B dense retro
Data pipeline (25T tokens)
Delphi scaling suite
Cluster scheduling with Iris
LLM pretraining efficiency
MoE quantile balancing
Posttraining data commons
Snowball post-training
Mochi pre-training overview
Mochi architecture decisions
◇