[KennethEnevoldsen]

for siglip, I used the fix proposed in:
embeddings-benchmark/mteb#4205

checked that all scores are there using "only-cache":

import mteb
from pathlib import Path

vidore = mteb.get_benchmark("ViDoRe(v3)")
tasks = mteb.get_tasks([t.metadata.superseded_by for t in vidore.tasks])

model_names = [
    "google/siglip-so400m-patch14-384",
    "BAAI/bge-large-en",
    "google/embeddinggemma-300m",
    "nomic-ai/nomic-embed-multimodal-3b",
    "nomic-ai/nomic-embed-multimodal-7b",
    # close
    # "nvidia/llama-nemotron-embed-vl-1b-v2"
    # missing:
    # "llamaindex/vdr-2b-multi-v1"
    # "Qwen/Qwen3-VL-Embedding-2B"
]
cache = mteb.ResultCache(Path(__file__).parent.parent)

for name in model_names:
    print(name)
    model = mteb.get_model_meta(name)
    res = mteb.evaluate(
        model,
        tasks,
        num_proc=16,
        encode_kwargs={"batch_size": 32},
        cache=cache,
        overwrite_strategy="only-cache" # just load the result from cache if present otherwise throws an error
    )

Also added results from #433 and #432

[Samoed]

bge, gemma, siglib, nomic results are inside results/results instead of results

[KennethEnevoldsen]

fixed!

[github-actions]

Model Results Comparison

Reference models: intfloat/multilingual-e5-large, google/gemini-embedding-001
New models evaluated: BAAI/bge-large-en, Qwen/Qwen3-VL-Embedding-2B, google/embeddinggemma-300m, google/siglip-so400m-patch14-384, nomic-ai/nomic-embed-multimodal-3b, nomic-ai/nomic-embed-multimodal-7b, nvidia/llama-nemotron-embed-vl-1b-v2
Tasks: Vidore3ComputerScienceRetrieval.v2, Vidore3EnergyRetrieval.v2, Vidore3FinanceEnRetrieval.v2, Vidore3FinanceFrRetrieval.v2, Vidore3HrRetrieval.v2, Vidore3IndustrialRetrieval.v2, Vidore3NuclearRetrieval.v2, Vidore3PharmaceuticalsRetrieval.v2, Vidore3PhysicsRetrieval.v2, Vidore3TelecomRetrieval.v2

Results for BAAI/bge-large-en

task_name	BAAI/bge-large-en	Max result	Model with max result	In Training Data
Vidore3ComputerScienceRetrieval.v2	0.4239			False
Vidore3EnergyRetrieval.v2	0.2726			False
Vidore3FinanceEnRetrieval.v2	0.2442			False
Vidore3FinanceFrRetrieval.v2	0.1100			False
Vidore3HrRetrieval.v2	0.2278			False
Vidore3IndustrialRetrieval.v2	0.1675			False
Vidore3NuclearRetrieval.v2	0.1905			False
Vidore3PharmaceuticalsRetrieval.v2	0.3463			False
Vidore3PhysicsRetrieval.v2	0.2791			False
Vidore3TelecomRetrieval.v2	0.3402			False
Average	0.2602		nan	-

Training datasets: AmazonReviewsClassification, AmazonReviewsVNClassification, DuRetrieval, MLQARetrieval, NQ, NQ-Fa, NQ-FaHardNegatives, NQ-NL, NQ-PL, NQ-PLHardNegatives, NQ-VN, NQHardNegatives, NanoNQ-VN, NanoNQRetrieval

---

Results for Qwen/Qwen3-VL-Embedding-2B

task_name	Qwen/Qwen3-VL-Embedding-2B	Max result	Model with max result	In Training Data
Vidore3ComputerScienceRetrieval.v2	0.7262			False
Vidore3EnergyRetrieval.v2	0.5701			False
Vidore3FinanceEnRetrieval.v2	0.5650			False
Vidore3FinanceFrRetrieval.v2	0.3636			False
Vidore3HrRetrieval.v2	0.5193			False
Vidore3IndustrialRetrieval.v2	0.4039			False
Vidore3NuclearRetrieval.v2	0.4077			False
Vidore3PharmaceuticalsRetrieval.v2	0.6169			False
Vidore3PhysicsRetrieval.v2	0.4676			False
Vidore3TelecomRetrieval.v2	0.6241			False
Average	0.5264		nan	-

---

Results for google/embeddinggemma-300m

task_name	google/embeddinggemma-300m	Max result	Model with max result	In Training Data
Vidore3ComputerScienceRetrieval.v2	0.6455			False
Vidore3EnergyRetrieval.v2	0.5586			False
Vidore3FinanceEnRetrieval.v2	0.4620			False
Vidore3FinanceFrRetrieval.v2	0.3249			False
Vidore3HrRetrieval.v2	0.4580			False
Vidore3IndustrialRetrieval.v2	0.3751			False
Vidore3NuclearRetrieval.v2	0.3426			False
Vidore3PharmaceuticalsRetrieval.v2	0.5776			False
Vidore3PhysicsRetrieval.v2	0.4248			False
Vidore3TelecomRetrieval.v2	0.5837			False
Average	0.4753		nan	-

Training datasets: FEVER, FEVER-FaHardNegatives, FEVER-NL, FEVER-VN, FEVERHardNegatives, FEVERHardNegatives.v2, HotpotQA, HotpotQA-Fa, HotpotQA-FaHardNegatives, HotpotQA-NL, HotpotQA-PL, HotpotQA-PLHardNegatives, HotpotQA-VN, HotpotQAHardNegatives, HotpotQAHardNegatives.v2, MIRACLJaRetrievalLite, MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLRetrievalHardNegatives.v2, NQ, NQ-Fa, NQ-FaHardNegatives, NQ-NL, NQ-PL, NQ-PLHardNegatives, NQ-VN, NQHardNegatives, NanoFEVER-VN, NanoFEVERRetrieval, NanoHotpotQA-VN, NanoHotpotQARetrieval, NanoNQ-VN, NanoNQRetrieval

---

Results for google/siglip-so400m-patch14-384

task_name	google/siglip-so400m-patch14-384	Max result	Model with max result	In Training Data
Vidore3ComputerScienceRetrieval.v2	0.1078			False
Vidore3EnergyRetrieval.v2	0.0745			False
Vidore3FinanceEnRetrieval.v2	0.0507			False
Vidore3FinanceFrRetrieval.v2	0.0233			False
Vidore3HrRetrieval.v2	0.0565			False
Vidore3IndustrialRetrieval.v2	0.0365			False
Vidore3NuclearRetrieval.v2	0.0450			False
Vidore3PharmaceuticalsRetrieval.v2	0.1200			False
Vidore3PhysicsRetrieval.v2	0.0877			False
Vidore3TelecomRetrieval.v2	0.0406			False
Average	0.0643		nan	-

---

Results for nomic-ai/nomic-embed-multimodal-3b

task_name	nomic-ai/nomic-embed-multimodal-3b	Max result	Model with max result	In Training Data
Vidore3ComputerScienceRetrieval.v2	0.6740			False
Vidore3EnergyRetrieval.v2	0.5580			False
Vidore3FinanceEnRetrieval.v2	0.4886			False
Vidore3FinanceFrRetrieval.v2	0.3356			False
Vidore3HrRetrieval.v2	0.4792			False
Vidore3IndustrialRetrieval.v2	0.3764			False
Vidore3NuclearRetrieval.v2	0.3622			False
Vidore3PharmaceuticalsRetrieval.v2	0.5682			False
Vidore3PhysicsRetrieval.v2	0.4482			False
Vidore3TelecomRetrieval.v2	0.5886			False
Average	0.4879		nan	-

Training datasets: JinaVDRArxivQARetrieval, JinaVDRDocQAAI, JinaVDRDocQAEnergyRetrieval, JinaVDRDocQAGovReportRetrieval, JinaVDRDocQAHealthcareIndustryRetrieval, JinaVDRDocVQARetrieval, JinaVDRInfovqaRetrieval, JinaVDRTatQARetrieval, VDRMultilingualRetrieval, VidoreArxivQARetrieval, VidoreDocVQARetrieval, VidoreInfoVQARetrieval, VidoreSyntheticDocQAAIRetrieval, VidoreSyntheticDocQAEnergyRetrieval, VidoreSyntheticDocQAGovernmentReportsRetrieval, VidoreSyntheticDocQAHealthcareIndustryRetrieval, VidoreTatdqaRetrieval

---

Results for nomic-ai/nomic-embed-multimodal-7b

task_name	nomic-ai/nomic-embed-multimodal-7b	Max result	Model with max result	In Training Data
Vidore3ComputerScienceRetrieval.v2	0.6913			False
Vidore3EnergyRetrieval.v2	0.5779			False
Vidore3FinanceEnRetrieval.v2	0.5074			False
Vidore3FinanceFrRetrieval.v2	0.3689			False
Vidore3HrRetrieval.v2	0.4916			False
Vidore3IndustrialRetrieval.v2	0.3932			False
Vidore3NuclearRetrieval.v2	0.3942			False
Vidore3PharmaceuticalsRetrieval.v2	0.5912			False
Vidore3PhysicsRetrieval.v2	0.4562			False
Vidore3TelecomRetrieval.v2	0.6068			False
Average	0.5079		nan	-

Training datasets: JinaVDRArxivQARetrieval, JinaVDRDocQAAI, JinaVDRDocQAEnergyRetrieval, JinaVDRDocQAGovReportRetrieval, JinaVDRDocQAHealthcareIndustryRetrieval, JinaVDRDocVQARetrieval, JinaVDRInfovqaRetrieval, JinaVDRTatQARetrieval, VDRMultilingualRetrieval, VidoreArxivQARetrieval, VidoreDocVQARetrieval, VidoreInfoVQARetrieval, VidoreSyntheticDocQAAIRetrieval, VidoreSyntheticDocQAEnergyRetrieval, VidoreSyntheticDocQAGovernmentReportsRetrieval, VidoreSyntheticDocQAHealthcareIndustryRetrieval, VidoreTatdqaRetrieval

---

Results for nvidia/llama-nemotron-embed-vl-1b-v2

task_name	nvidia/llama-nemotron-embed-vl-1b-v2	Max result	Model with max result	In Training Data
Vidore3ComputerScienceRetrieval.v2	0.7057			False
Vidore3EnergyRetrieval.v2	0.5819			False
Vidore3FinanceEnRetrieval.v2	0.5589			False
Vidore3FinanceFrRetrieval.v2	0.3595			False
Vidore3HrRetrieval.v2	0.5301			False
Vidore3IndustrialRetrieval.v2	0.3816			False
Vidore3NuclearRetrieval.v2	0.3927			False
Vidore3PharmaceuticalsRetrieval.v2	0.5995			False
Vidore3PhysicsRetrieval.v2	0.4449			False
Vidore3TelecomRetrieval.v2	0.6073			False
Average	0.5162		nan	-

Training datasets: Cauldron (AI2D, OCRVQA, Websight), HotpotQA, HotpotQA-Fa, HotpotQA-FaHardNegatives, HotpotQA-NL, HotpotQA-PL, HotpotQA-PLHardNegatives, HotpotQA-VN, HotpotQAHardNegatives, HotpotQAHardNegatives.v2, JinaVDRArxivQARetrieval, JinaVDRDocQAAI, JinaVDRDocQAEnergyRetrieval, JinaVDRDocQAGovReportRetrieval, JinaVDRDocQAHealthcareIndustryRetrieval, JinaVDRDocVQARetrieval, JinaVDRInfovqaRetrieval, JinaVDRTatQARetrieval, MIRACLJaRetrievalLite, MIRACLReranking, MIRACLRetrieval, MIRACLRetrievalHardNegatives, MIRACLRetrievalHardNegatives.v2, MLQARetrieval, MultiLongDocReranking, MultiLongDocRetrieval, NQ, NQ-Fa, NQ-FaHardNegatives, NQ-NL, NQ-PL, NQ-PLHardNegatives, NQ-VN, NQHardNegatives, NanoHotpotQA-VN, NanoHotpotQARetrieval, NanoNQ-VN, NanoNQRetrieval, SQuAD, StackExchangeClustering, StackExchangeClustering-VN, StackExchangeClustering.v2, Tiger Math/Stack, VDRMultilingualRetrieval, VidoreArxivQARetrieval, VidoreDocVQARetrieval, VidoreInfoVQARetrieval, VidoreSyntheticDocQAAIRetrieval, VidoreSyntheticDocQAEnergyRetrieval, VidoreSyntheticDocQAGovernmentReportsRetrieval, VidoreSyntheticDocQAHealthcareIndustryRetrieval, VidoreTatdqaRetrieval, docmatix-ir, wiki-ss-nq

---

---

Note: Content truncated due to GitHub API limits. See the full report in the workflow artifacts.

[Samoed]

Looks good. I think we need to wait for qwen and nemotron model will merge to main

[Samoed]

Can we remove nvidia/llama-nemotron-embed-vl-1b-v2 and merge then?

[Samoed]

Currently, revision for siglip 9fdffc58afc957d1a03a25b10dba0329ab15c2a3-but-my-prev

[KennethEnevoldsen]

Yep noticed - was just waiting to add some more results in that we are currently missing

[KennethEnevoldsen]

I believe that is everything!