Only the decoded state, question definitions, and criteria enter inference. Labels, annotation metadata, group/source identifiers, evaluation-family flags, and provenance must remain outside model prompts. A neutral anonymous question or option identifier may replace a source-derived identifier, provided predictions are mapped back without changing the supplied option semantics or order.

Preserve authored Noul meanings: true does not always mean “good” or “acceptable.” Preserve numeric Score values rather than interpreting option indices as scores. Probability vectors align by option ID, not incidental dictionary order. Do not harden soft targets, round continuous gold values to classes, or silently truncate long inputs. The physical release uses Parquet; JSON fields preserve structured values without requiring every source to share the same state schema.

The actual query/qrels/hybrid-candidate acquisition source is NanoJMTEB-v2, revision 322a48e91659b48c78583bb37367c62917d89776. Where recorded, its intermediate distribution is SB Intuitions / JMTEB-lite. That wrapper is distinguished from the original authors and corpus rights holders.

Wikipedia-derived components still have incomplete article-version/license-chain reconstruction in the recorded source notices. The webpage and FAQ components likewise retain unresolved source-specific scope questions. These qualifications are part of the dataset documentation, not permission to disregard underlying rights. Consult the individual records in the retrieval source directory before redistributing those components.

Copyright © Yuichi Tateno. All generalization-* subsets are licensed under Creative Commons Attribution-NonCommercial 4.0 International, to the extent the maintainer holds the relevant rights. Attribute the maintainer and this release, identify modifications, and comply with the license's noncommercial condition. Third-party rights remain intact.

The maintainer declares that these tasks were generated/authored with OpenAI GPT-6.1-Sol. The choice of noncommercial distribution reflects the maintainer's concern about OpenAI's restrictions on using generated output to develop competing models. Applicable OpenAI contractual obligations are separate from the CC license: CC BY-NC itself is not a blanket prohibition on model training, and noncommercial status does not by itself establish permission to develop a competing model.

See OpenAI Terms of Use and OpenAI Services Agreement. The Services Agreement includes a competing-model restriction with permitted exceptions; applicability depends on the service and agreement. The license notice records a terms check dated 2026-10-06. JDMB does not claim that every downstream user is party to the same agreement or that a CC license can waive contractual restrictions.

Acquire each dataset at the linked revision before interpreting its schema. For directly sourced tasks, use the original author's labeled test partition, except for the explicitly identified JCoLA validation, unpartitioned JSeM suite, GPQA-JA main evaluation pool, unpartitioned JMMLU benchmark, and SOBACO original training pool. These exceptions do not turn the original partitions into author-defined test splits. Retain the complete extracted test pool before filtering or sampling, together with original record IDs, source split names, revision, and file/record identities. S1MB-japanese-v1 distributes the selected rows, not those complete initial pools.

Explicit retrieval exception: JDMB adopts the evaluation queries supplied by NanoJMTEB-v2. Its wrapper evaluation split is not asserted to be an original-author test split for every underlying corpus. In particular, JAQKET's quiz lineage includes original dev2. Record wrapper and original partition provenance separately; the JDMB test name describes the benchmark's evaluation role, not a retrospective renaming of the original author's partition.

Choice. Retain the source question/context and its candidate texts in their original order, then assign unit reference mass to the source gold candidate. Preserve each task's meaning: JaNLI has entailment/non-entailment; JNLI has entailment/contradiction/neutral. Entailment is directional. Non-entailment is not treated as synonymous with contradiction. KUCI asks for a plausible causal continuation, not logical necessity. MMLU-ProX retains nonempty source options and their corresponding original answer identity; nominal labels are never converted into numeric Score values.

Noul. Use original Boolean labels when available. Commonsense morality maps source label 1 to true meaning a moral violation, and label 0 to false. Mintaka includes only original Boolean-answer questions with an existing Japanese translation; it does not generate new Japanese translations or turn open-ended answers into unsupported Boolean labels. JCulture expands a question into candidate-membership judgments: for candidate position j, the reference is true if and only if j belongs to the author's complete answer set. Multiple correct candidates remain multiple true judgments. A negative question stem does not invert this membership definition. Rationales and answer references remain outside inference inputs.

Continuous STS Score. JSICK uses original numeric anchors 1, 2, 3, 4, 5; JSTS uses 0, 1, 2, 3, 4, 5. Given original gold value y between adjacent anchors a and b, assign masses

q(a) = (b-y)/(b-a) and q(b) = (y-a)/(b-a),

with zero mass elsewhere. An exact anchor receives unit mass. This preserves sum_k value(k) q(k) = y without rounding or claiming an annotator-vote distribution. Middle-anchor descriptions explain the scale; they do not replace the source scalar. Images, translations, NLI labels, raw ratings, and gold scores are not placed in the STS state. Source scales remain distinct.

Authored Score. Synthetic tasks provide explicit numeric criteria as part of the task specification. Interpret those declared values as given, including any stated scoring convention. Do not infer a common physical unit across tasks, manufacture interval scores from unrelated category labels, or treat all Score outputs as probabilities of an external event.

Initial length screening uses the tokenizer and chat template of Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. Render the full JSON request—decoded state, questions, instructions, and every candidate—in a user message with a generation prompt and thinking disabled. Count the complete rendered input; retain at most 7,936 input tokens, reserving 256 tokens within an 8,192-token budget. This is a reference-renderer eligibility criterion, not a guarantee for every model tokenizer.

For the directly converted Choice/Noul source pools, before final capping, downsample imbalanced pools by stratum so no stratum exceeds 1.5 times the least-populated stratum. Strata are semantic labels for NLI, true/false for Noul, and correct-option position for QA. STS selection instead uses five equal-width bands over the original numeric range at the capped-selection stage. Within each stratum, order groups and cases deterministically by SHA-256 of a canonical JSON seed/identifier tuple and select cases by group round-robin. This reduces domination by a repeated source group but may leave partial original groups.

For the original source-derived subsets, the final cap is 300 judgments per ordinary subset. JCoLA, Jamp, JSeM and GPQA-JA use the constrained selection procedure below; JMMLU and SOBACO use their source-specific procedure in section 7. For an eligible pool already below the cap, retain all rows without padding. Otherwise, allocate near-equal quotas across available strata, then greedily add complete eligible-pool groups that fit both the remaining count budget and every stratum quota. Prioritize the greatest normalized remaining-quota benefit per judgment; resolve ties by smaller group size and a seeded SHA-256 order. Use seed namespace jdmb-cap300-whole-group-v1:<configuration> and return retained rows in source-pool order. This is a deterministic greedy approximation, not an optimal group-selection solution. Whole-group and quota constraints account for the 285 JaNLI judgments and 283 JCulture judgments.

For STS, band boundaries are proportional to the original scale: JSICK [1,1.8), [1.8,2.6), [2.6,3.4), [3.4,4.2), [4.2,5]; JSTS uses five unit-width bands over 0–5, including 5 in the last band. Gold values remain continuous. Each released STS subset has 60 judgments per band.

Balance of answer positions is not balance of topic, difficulty, subject, or candidate count. “Complete groups” refers to the final eligible pool; earlier filtering/balancing may already have removed other original-group members. Model-specific evaluation must measure its own full input lengths and reject overflow rather than silently clipping state or candidate text.

Use NanoJMTEB-v2's evaluation queries, corpus texts, qrels, and reranking_hybrid candidate rankings. For each query, collect all reference-positive document IDs. Choose one available, nonempty qrels-positive document deterministically with a query-specific seeded random generator. Build the distractor pool from hybrid-ranked documents, excluding every qrels-positive ID, empty text, and normalized duplicate positive/candidate texts. Other known positives must never become distractors.

Sample three to five distinct distractors, yielding four to six total candidates. If a sampled set exceeds the full-input reference budget, retry up to 16 samples at that candidate count, then reduce the number of distractors to a minimum of three. Use cached document-length estimates only to prune candidate attempts; measure every accepted full request exactly. A failure to find a fitting set is an exclusion under this bounded sampling procedure, not proof that no possible set fits.

For each candidate-count cohort, distribute the correct-answer position evenly by a seeded query order and rotating position offset. Select 50 queries from each of six components. Allocate candidate-count quotas approximately in proportion to the eligible component pool using largest remainders, then balance answer positions within each candidate count where available. The six components contribute equally to the 300-row retrieval subset, rather than being six separately weighted JDMB benchmarks.

The final retrieval input ceilings are 4,096 reference tokens, except 6,144 for JAQKET, including full state, instructions, candidates, and chat-template overhead. Documents are retained in full. For JAQKET, retain each original query and positive document but resample distractors from the same pinned hybrid source, preferring a shorter candidate pool to meet that ceiling; update candidate alignment and the reference position. This introduces document-length and candidate-selection bias.

Only paper title → abstract is included from NLPJournal; the reverse and other paper views are excluded. A retrieval gold candidate means “the selected source-qrels reference in this closed candidate set.” Unjudged distractors are not independently established false or irrelevant, and these labels must not be interpreted as universal document relevance grades.

The ordinary synthetic inventory has six knowledge/formal subsets (300 judgments each) and two diverse subsets (300 each). The diverse subsets cover 30 daily-life, business, and technology categories with ten separately authored scenes per type per category. Noul has 150 true and 150 false references; Choice has three options and 100 answers at each position. States include text, structured objects, and dialogue-like lists. Labels and rationales are authored with the items, rather than inferred from their positions.

Knowledge/formal items specify their own decision criteria, domain conventions, and numeric scales. Recorded review and mathematical-check metadata, where present, document specific checks rather than establishing independent human annotation for the entire collection. The terms hard and generalization are subset names/design intentions, not measured difficulty thresholds or proof of unseen-task generalization.

Synthetic construction and review do not establish independence from model training data. Exact synthetic wording cannot be reconstructed from a random seed alone; the distributed states, criteria, and reference targets define the frozen authored evaluation items.

These subsets use selection that minimizes the sum of squared source-group counts subject to stratum quotas. Their original partition identities are preserved separately from JDMB's common test role.

JCoLA — Japanese acceptability. Combine 865 in-domain and 685 out-of-domain validation sentences. Use the author's normalized sentence, not the original string with acceptability diacritics. Target 75 examples in each domain × label stratum while balancing bibliographic source groups. Exclude cases overlapping the LLM-jp-eval training partition; replace them with unused evaluation cases from the same source publication and label where available, and omit a case if no matching replacement exists. The resulting subset has 299 cases: 150 true and 149 false. Map label 1 to true and 0 to false. State is the sentence alone, and the task defines grammatical/lexical acceptability rather than factual truth or morality. Context-dependent ellipsis, honorifics, and discourse effects remain source limitations. The original split stays recorded as validation.

Jamp — temporal NLI. Start with the 348 rows of test_2.tsv. Exclude 21 premises with a “since” starting date later than the stated present date, comparing parsed year/month/day/hour/minute tuples. This leaves 327 eligible examples. Select 100 entailment, 100 contradiction, and 100 neutral cases while balancing template groups; the release spans 79 templates with two to seven examples each. Preserve premise and hypothesis strings and the original gold_label. Do not repair dates or rewrite text to match gold. Instruction asks for the relation assuming the premise true; no universal same-event assumption is injected, since that would change neutral cases.

JSeM — semantic inference. Read the XML categories in this order: Adjectives, Attitudes, Comparatives, Ellipsis, GeneralizedQuantifier, NominalAnaphora, Plurals, TemporalReference, Verbs, preserving XML problem order. Retain semantic answers yes, no, unknown, and undef; omit acceptability labels and extension tasks with incompatible no semantics. Of 789 candidate cases, remove duplicate category/ID pairs Comparatives:588, Plurals:409, and Plurals:410, retaining their matching earlier counterparts. Exclude definition/annotation disagreements at GeneralizedQuantifier:110–114, Plurals:400, Adjectives:508–509, GeneralizedQuantifier:304–306, and GeneralizedQuantifier:362–364. The resulting pool has 772 cases.

Select every available no (66) and undef (11), plus 112 yes and 111 unknown. Category balancing yields 35 Adjectives, 34 GeneralizedQuantifier, and 33 in each remaining category. yes means the hypothesis follows; no means its negation follows; unknown means neither follows under a fixed interpretation; undef means the source's interpretation/denotation conditions do not support an ordinary determinate truth-value judgment. State retains all premises and the hypothesis. For six source-specified ambiguous-reading cases, include only the necessary interpretation condition, never gold or its explanation. The suite has no original author-defined split; do not describe these examples as independently held-out original test data.

GPQA-JA main — scientific QA. Use the Japanese main evaluation pool of 448 questions, without merging its overlapping diamond or extended variants. Preserve original Record IDs. Exclude three records whose stripped answer texts do not form four distinct options and three further records with unresolved scientific assumptions, an unversioned external-sequence dependency, or a conflicting answer/explanation. Do not repair gold from source explanations. This leaves 442 eligible items.

Allocate 300 places across biology, chemistry, and physics by repeatedly assigning one place to the least-filled domain with remaining capacity, breaking domain ties by the original English name. The release has 77 biology, 112 chemistry, and 111 physics items. Minimize squared counts of domain/subdomain groups within these quotas. In fixed selected-row order, rotate the correct position A/B/C/D by row index modulo four, giving 75 at each position. Shuffle the three original distractors using a query-specific seeded generator and insert the unchanged correct-answer text at the designated position. State contains only the original question; answer texts are supplied as candidates, while explanations remain outside inference.

The GPQA-JA source requires question examples and explanations to remain offline. Its CC BY 4.0 declaration does not remove that explicit handling condition. This component remains private, and inclusion here is not authorization to publish its question text online.

JMMLU Japan. Retain the original four Japan-specific subject pools: history 150, idiom 150, civics 150 and geography 139. Select 75 rows per subject with seed 42, stratifying by original correct-option position. Per-subject position quotas are 18 or 19 and rotate across subjects, yielding 75 references at each of A/B/C/D overall. Preserve original question text, options, option order and gold. Original split remains unspecified (null); JMMLU's evaluation purpose is not represented as author-held-out test membership.

SOBACO culture. Use only the culture portion: age 2,016, gender 1,872 and hierarchy 2,088 original rows. Group templates by their complete sets of name-role-normalized context and additional-context pairs. Each matching pair contains two complementary questions, yielding 10 age, 12 gender and 11 hierarchy families (33 total; 66 templates). Names are normalized only for grouping; original names and full text remain in inference.

Select 50 concrete context pairs per category and retain both complementary questions, yielding 100 rows per category and 300 total. Allocate pairs as evenly as possible across families, then prioritize normalized wording and name-pair coverage with seed 42. Select one existing authored option ordering per question while balancing gold positions 100/100/100 globally. Preserve UNKNOWN answers where authored; do not fabricate or relabel them. No identical context/question is repeated under a different ordering in this 300-row subset.

The state joins original context, additional context and question with newlines. The original three candidates remain in their selected authored order. The original partition is train, retained in provenance. JDMB's test-only role does not make these rows an original-author test set or establish training-data independence. 300 rows span 33 dependent template families, not 300 independent cultural topics. Use family group IDs for any bootstrap or splitting. The source's age/gender counts differ from paper Table 2; category labels follow the actual pinned files.

Compute a per-benchmark adjusted score and clip it to [0,1]:

• Choice: (accuracy - b)/(1-b), where b is the strongest of uniform, fixed-option-ID, and fixed-position reference accuracy on the entire benchmark.
• Noul: 2 × balanced_accuracy - 1, using threshold 0.5 and fractional confusion counts for soft targets. This is not Brier skill; retain raw Brier alongside it.
• Score: 1 - MAE/b, where b is the full-benchmark MAE of a constant prediction equal to the median normalized reference expectation.

AVG is the arithmetic mean of all 27 benchmark adjusted scores, displayed on a 0–100 scale. Each benchmark contributes 1/27. This is equal-benchmark weighting; do not substitute row-weighted averaging or average raw Brier and MAE values. The reference baselines are computed from evaluation labels for descriptive comparison; they are not independently trained baseline models.

A complete AVG requires successful measured coverage and a defined score for every active benchmark. Missing, failed, partial, or undefined benchmarks cannot be removed from its denominator. Decision-type metrics remain useful diagnostics.

The aggregation is a declared comparison, not an empirically validated universal capability scale.

JSICK's published relatedness_score_Ja is retained as authoritative source gold. Source inspection found cases where it differs from the mean of all available ratings and matches the first two ratings; the aggregation rationale is unresolved. Some source sentence pairs also recur across the author's train/test files, and duplicate pairs can have different ratings. JDMB does not silently relabel them or claim contamination-free evaluation.

SOBACO questions share 33 template families; JCulture judgments can share an original question; NLI/STS examples can share image or sentence groups. Preserve grouping when inspecting errors or estimating uncertainty. Any bootstrap/confidence interval should respect the relevant source groups, rather than treating every judgment as independent. This release does not supply a validated significance analysis for leaderboard differences.

Synthetic subset names do not establish real-world probability calibration, robustness, or broad generalization. The retrieval task measures choice among sampled candidates, with qrels incompleteness and length-selection effects, rather than unrestricted corpus retrieval.

Use JDMB primarily as a frozen evaluation set. Training, prompt tuning, or selecting checkpoints on its labels compromises its value as held-out evidence and must be disclosed. No independent human-gold certification or absence of training overlap is claimed.