Marin Mochi 535B-A23B

Research artifact. This model has not been properly tested or evaluated and is not necessarily secure. Use at your own risk in production settings.

BF16 weights-only exports of retained Hero pretraining checkpoints. The model has 535,477,106,688 stored parameters and approximately 22.76B active non-embedding parameters per token; A23B rounds that active count to the nearest billion.

Checkpoint branches

Each checkpoint uses a branch named step-N. The retained inventory contains 57 checkpoints from step 6000 through step 393491. All 57 checkpoint uploads are complete and independently verified. A branch is complete only when hf-upload-manifest.json exists and pins a verified weights commit. The main branch contains the final checkpoint, step 393491, matching step-393491.

Read the checkpoint branch's model card and config.json for its training phase, context settings and source revision. Earlier phases use 4K context, later phases use 16K. Steps 378000, 384000 and 390251 use 65K. Steps 390500 through 393491 use 262K and GrugMoe artifact schema 3 with global-layer query log scaling. Each checkpoint contains 193 safetensors shards with a BF16 tensor payload of 1,070,954,213,376 bytes. Optimizer and data-loader state are excluded.

Tokenizer and inference

The bundled tokenizer is marin-community/marin-tokenizer at revision a5ca45f2feb6c959bd87b81689aa7279b5bdcaa2, the pin used by Hero's native sampling configuration. The training run records the tokenizer repository but does not record an exact tokenizer revision.

Serving requires the Marin vLLM fork with its split-expert loader. This export uses GrugMoe artifact schema 3 and preserves global-layer query log scaling: (1 + 0.1 * ln(max(n / 65536, 1))) ** 2, where n counts visible keys in the query’s document. It requires a schema-3-compatible loader with this scaling; older loaders reject schema 3. Full-model inference parity and generation remain unvalidated. Loader support alone does not establish numerical parity. Generic Transformers loading and a serving launch recipe have not been validated for this release. This is a base model; the bundled chat template does not indicate instruction tuning. Use text completions.

License

OpenMDW 1.1. See LICENSE for the complete text.

We release our tools to the public with the intent that they are used safely, ethically, and responsibly. They are meant to contribute to the public's understanding of science, and aid in its advancement. By using this Marin model, you agree to comply with all applicable law.

Downloads last month
121
Safetensors
Model size
535B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Space using open-athena/Marin-Mochi-535B-A23B 1