Dataset Viewer
Duplicate
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code:   StreamingRowsError
Exception:    TypeError
Message:      Couldn't cast array of type
struct<target_mass_at_prediction: double, uniform_accuracy_baseline: double, fixed_answer_accuracy_baseline: double, baseline_adjusted_skill: double, baseline_adjusted_score: double>
to
{'binary_brier': Value('float64'), 'accuracy': Value('float64'), 'positive_recall': Value('float64'), 'specificity': Value('float64'), 'precision': Value('float64'), 'balanced_accuracy': Value('float64'), 'f1': Value('float64'), 'true_positive': Value('float64'), 'false_positive': Value('float64'), 'false_negative': Value('float64'), 'true_negative': Value('float64'), 'positive_prevalence': Value('float64'), 'majority_accuracy_baseline': Value('float64'), 'constant_brier_baseline': Value('float64'), 'brier_skill': Value('float64'), 'baseline_adjusted_skill': Value('float64'), 'baseline_adjusted_score': Value('float64')}
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/utils.py", line 147, in get_rows_or_raise
                  return get_rows(
                      dataset=dataset,
                  ...<4 lines>...
                      column_names=column_names,
                  )
                File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
                  return func(*args, **kwargs)
                File "/src/services/worker/src/worker/utils.py", line 127, in get_rows
                  rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
                File "/src/services/worker/src/worker/utils.py", line 483, in safe_iter
                  yield from ds.decode(False) if ds.features else ds
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2951, in __iter__
                  for key, example in ex_iterable:
                                      ^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2461, in __iter__
                  for key, pa_table in self._iter_arrow():
                                       ~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2486, in _iter_arrow
                  for key, pa_table in self.ex_iterable._iter_arrow():
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 547, in _iter_arrow
                  for key, pa_table in iterator:
                                       ^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 430, in _iter_arrow
                  for key, pa_table in self.generate_tables_fn(**gen_kwags):
                                       ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 343, in _generate_tables
                  self._cast_table(pa_table, json_field_paths=json_field_paths),
                  ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
                  pa_table = table_cast(pa_table, self.info.features.arrow_schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2378, in table_cast
                  return cast_table_to_schema(table, schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2312, in cast_table_to_schema
                  cast_array_to_feature(
                  ~~~~~~~~~~~~~~~~~~~~~^
                      table[name] if name in table_column_names else pa.array([None] * len(table), type=schema.field(name).type),
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      feature,
                      ^^^^^^^^
                  )
                  ^
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 1861, in wrapper
                  return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
                                           ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2158, in cast_array_to_feature
                  raise TypeError(f"Couldn't cast array of type\n{_short_str(array.type)}\nto\n{_short_str(feature)}")
              TypeError: Couldn't cast array of type
              struct<target_mass_at_prediction: double, uniform_accuracy_baseline: double, fixed_answer_accuracy_baseline: double, baseline_adjusted_skill: double, baseline_adjusted_score: double>
              to
              {'binary_brier': Value('float64'), 'accuracy': Value('float64'), 'positive_recall': Value('float64'), 'specificity': Value('float64'), 'precision': Value('float64'), 'balanced_accuracy': Value('float64'), 'f1': Value('float64'), 'true_positive': Value('float64'), 'false_positive': Value('float64'), 'false_negative': Value('float64'), 'true_negative': Value('float64'), 'positive_prevalence': Value('float64'), 'majority_accuracy_baseline': Value('float64'), 'constant_brier_baseline': Value('float64'), 'brier_skill': Value('float64'), 'baseline_adjusted_skill': Value('float64'), 'baseline_adjusted_score': Value('float64')}

Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.

S1MB evaluation results

View the S1MB Leaderboard

Public English and Japanese model evaluation results for System One Mosaic Benchmark (S1MB), combining Choice, Noul and Score tasks. Both english-v1 and japanese-v1 measurements belong in this Dataset. The same leaderboard selects English or Japanese through the category query parameter. Evaluate and rank each scope independently. Japanese evaluation inputs and targets remain private; publishing measurements does not redistribute the evaluation Dataset.

Repository layout and submissions

Each <organization-or-user>__<model-id>/ folder represents one canonical model and may contain benchmarks from either language or both. It contains metadata.json plus per-benchmark <benchmark-id>.json.xz files. Measurements retain original run IDs, exact model and evaluation Dataset revisions, evaluator provenance and effective settings. Different recorded Dataset revisions may coexist; validate each result against its own recorded revision. Historical Dataset repo IDs and SHAs remain unchanged after repository renames.

Follow the submission guide to evaluate, validate, export and open a Dataset PR. Include evaluation commands, coverage, failures, model/source pins and material runtime settings. Explain replacements and configuration changes; do not assemble an undisclosed best-of-many row. Fix settings independently of test labels and scores. Missing and partial measurements remain incomplete and cannot claim complete aggregate scores.

Adapter/code changes belong in a separate source PR. Upload only the intended model metadata and compressed measurements; exclude credentials, raw input text, checkpoints, caches, reports and local lock files. Authored training disclosures are provenance, never inference instructions or scoring adjustments; missing metadata means unknown. S1MB scores do not establish unseen-task generalization or training-data non-overlap.

Leaderboard display updates

The offline publisher prepares English and Japanese scopes from one exact results commit. After merging a results Dataset PR, run from evaluator/:

uv run python ../viewer/scripts/publish-display.py
# Review the report, then publish the summary:
uv run python ../viewer/scripts/publish-display.py --publish

The reserved root file viewer-summary.json contains prepared display data, not a measurement submission. Maintainers generate it offline from validated public results, compare with the published summary, review reductions, and publish with a parent-SHA guard. Rebuild/redeploy the Space to load it; measurement publication alone does not update the running leaderboard. The viewer reads bundled JSON without runtime Hub access. Local viewers also require explicit preparation or fetching and a restart. See the deployment runbook.

License and usage terms

This collection has source-dependent rights and usage terms, represented by license: other; it has no blanket MIT grant. The S1MB project's MIT license covers source code only. Submitted measurements retain their contributor's applicable terms, and source-derived material retains the terms and attribution requirements of each recorded evaluation Dataset revision. Consult the English source notices and Japanese source notices for the applicable recorded release; access to Japanese inputs requires authorization. Model checkpoint and inference-service terms remain separate. Where permission is unspecified, obtain it from the relevant rights holder before redistribution; an unknown license is not a reuse grant.

See the S1MB project for code and documentation.

Japanese results

Publish Japanese measurements evaluated against hotchpotch/S1MB-japanese-v1. Historical measurements referencing the private hotchpotch/JDMB-v1 Dataset have been removed from this shared results Dataset; those models require fresh S1MB evaluations. JDMB-v1 and the original result archives remain private.

The S1MB evaluations for PelaAI/KnowLine-4B-Gen4 and convaiinnovations/laya-multilingual are retained: 54 measurements across two model folders, with 27 benchmarks / 8,067 judgments per model. Recorded Dataset IDs, exact SHAs, model revisions and evaluator provenance remain unchanged. Existing English measurements and canonical model metadata are preserved.

Submit new English and Japanese measurements here. Do not upload private raw inputs, targets, local lock files or reports. Prepare and publish the display summary, then rebuild/redeploy the Space to show updated results.

Downloads last month
2,666

Articles mentioning hotchpotch/s1mb-result