Marin Mochi 535B-A23B
Research artifact. This model has not been properly tested or evaluated and is not necessarily secure. Use at your own risk in production settings.
BF16 weights-only exports of retained Hero pretraining checkpoints. The model
has 535,477,106,688 stored parameters and approximately 22.76B active
non-embedding parameters per token; A23B rounds that active count to the
nearest billion.
Checkpoint branches
Each checkpoint uses a branch named step-N. The retained inventory contains
57 checkpoints from step 6000 through step 393491. All 57 checkpoint uploads are complete and independently verified.
A branch is complete only when hf-upload-manifest.json exists and pins a
verified weights commit. The main branch contains the final checkpoint, step 393491, matching step-393491.
Read the checkpoint branch's model card and config.json for its training
phase, context settings and source revision. Earlier phases use 4K context,
later phases use 16K. Steps 378000, 384000 and 390251 use 65K. Steps
390500 through 393491 use 262K and GrugMoe artifact schema 3 with global-layer
query log scaling. Each checkpoint contains
193 safetensors shards with a BF16 tensor payload of 1,070,954,213,376 bytes.
Optimizer and data-loader state are excluded.
Tokenizer and inference
The bundled tokenizer is marin-community/marin-tokenizer at revision
a5ca45f2feb6c959bd87b81689aa7279b5bdcaa2, the pin used by Hero's native sampling configuration.
The training run records the tokenizer repository but does not record an exact
tokenizer revision.
Serving requires the Marin vLLM fork
with its split-expert loader.
This export uses GrugMoe artifact schema 3 and preserves global-layer query log scaling: (1 + 0.1 * ln(max(n / 65536, 1))) ** 2, where n counts visible keys in the query’s document. It requires a schema-3-compatible loader with this scaling; older loaders reject schema 3. Full-model inference parity and generation remain unvalidated. Loader support alone does not establish numerical parity. Generic Transformers
loading and a serving launch recipe have not been validated for this release.
This is a base model; the bundled chat template does not indicate instruction
tuning. Use text completions.
License
OpenMDW 1.1. See LICENSE for the complete text.
We release our tools to the public with the intent that they are used safely, ethically, and responsibly. They are meant to contribute to the public's understanding of science, and aid in its advancement. By using this Marin model, you agree to comply with all applicable law.
- Downloads last month
- 121