Agent Recovery on Kaggle: A Submission Attempt with Separate Research Results

This is a submission attempt for the Kaggle Benchmarking Challenge. I am submitting it for review while explicitly disclosing the missing Kaggle-platform results. The research results below are external-provider observations, not a Kaggle leaderboard or a claim of contest eligibility.

What I Benchmarked

A business system accepts an agent's operation, but the acknowledgement is lost. Should the agent retrieve the original result, reconnect to running work, preserve uncertainty, or submit a new write?

I designed 24 synthetic cases covering running operations, lost acknowledgements, unknown outcomes, legitimate new intents, authorization changes, and target-binding changes. The domains are tickets, inspections, appointments, and inventory deductions. Five tools operate on an in-memory simulator; no real customer system or production transaction is involved.

The evaluation distinguishes tool attempts, accepted actions, actual effects, and final reporting. Seventeen cases require business progress; seven require an unknown, blocked, or pending outcome. Refusing everything therefore cannot succeed. New writes are deliberately non-idempotent, so the simulator does not hide an unsafe redispatch through automatic deduplication.

Models Tested

The frozen research batch, 20261009-full96-v1, compared deepseek-v4.1-flash and deepseek-v4-pro through Tencent Cloud TokenHub using the Kaggle Benchmarks SDK. Each model ran all 24 cases twice: 48 episodes each, 96 overall, with fresh conversations and simulator states and caching disabled.

These were local external-provider runs. The provider-facing identifiers do not pin immutable model revisions. Two repeats are limited repeatability evidence, not independent random samples or a significance test.

Separately, I attempted native Kaggle access using my original account. Earlier calls to google/gemini-3.7-flash and qwen/qwen3-next-80b-a3b-instruct returned a CN-location restriction. On October 11, a new, uncached private access-probe task also failed during native creation with deepseek-ai/deepseek-r1-0528, openai/gpt-oss-120b, and zai/glm-5: HTTP403/PermissionDenied with no successful model response. Those versions were Errored; they did not reach a completed task that could subsequently be scheduled with tasks run.

That tests five named interfaces across separate access attempts, not every model in the visible directory. The missing native observations are not zero benchmark scores. No cached external outputs are presented here as native Kaggle results, and no external model is relabelled as a Kaggle model.

Findings

The external research observed all 96 planned episodes, retaining failures:

The strict contract includes final JSON, evidence-supported identities and claims, the effect ledger, and permitted tool behavior. Business progress is a separate diagnostic. Neither a parser failure nor a strict zero establishes that the original write failed; automatically resubmitting on that basis could introduce an effect the experiment itself did not observe.

Successful execution also did not erase prohibited tool attempts or payload-binding mistakes. The runtime could reject an unauthorized request while the trajectory still revealed an incorrect model decision. Conversely, identical payloads could represent distinct authorized intents that should not be collapsed into one operation.

I retain the original strict scores. A recorded final_not_json label alone does not establish that every later semantic check would pass after extracting JSON: failed parsing skips those checks. The bundle's retained final text, visible messages, scorer, and effect ledgers allow readers to inspect these distinctions rather than trusting only a summary label.

These are findings under an explicit, small synthetic contract, two models in one family, and one provider. They do not establish production safety or general model equivalence. I would next vary recovery-tool wording, evidence noise, and explanation checks without replacing the frozen observations.

My Benchmark

There is currently no publicly accessible Kaggle benchmark URL or successful native Kaggle result for this submission. Private task creation, a notebook save, and an empty results table are not substitutes. The organizer has confirmed that external API and cached results do not satisfy this challenge's platform-run requirement; I disclose that qualification gap rather than presenting this attempt as accepted.

The independently published research remains inspectable:

• Pinned reproduction source
• Public full96-v1 release, including cases, scorer, manifests, and all 96 episode records
• Earlier research article, published separately without this challenge's tags

Those links document the research; they are not Kaggle benchmark links. The experiment uses the Kaggle Benchmarks SDK, with recovery and binding questions informed by ACC and BailingHub. This attempted entry records both what was measured and what the platform path has not yet produced.