Skip to content

[Bug] Optuna sweeper raises uncaught exception when run ends with nan value #2237

Closed

Description

@jbaczek

🐛 Bug

Description

I perform hp search for a deep learning model and there are situations when model diverges and starts producing nans in it's output. My code exits gracefully returning tuple of nans as metrics. This is an expected behavior. But optuna sweeper doesn't think so.

  1. The problem starts here: https://github.com/facebookresearch/hydra/blob/main/plugins/hydra_optuna_sweeper/hydra_plugins/hydra_optuna_sweeper/_impl.py#L377 . Here values is a tuple of NaNs and state is COMPLETED (because the experiment code exited gracefully)
  2. study.tell raises an error here: https://github.com/optuna/optuna/blob/release-v2.10.0/optuna/study/study.py#L652
    because of the check preformed here https://github.com/optuna/optuna/blob/release-v2.10.0/optuna/study/_optimize.py#L319
  3. Error is finally reraised here: https://github.com/facebookresearch/hydra/blob/main/plugins/hydra_optuna_sweeper/hydra_plugins/hydra_optuna_sweeper/_impl.py#L391 ,which ultimately crashes hydra.

Checklist

  • I checked on the latest version of Hydra
  • I created a minimal repro (See this for tips).

To reproduce

** Minimal Code/Config snippet to reproduce **

** Stack trace/error message **

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/hydra_plugins/hydra_optuna_sweeper/_impl.py", line 237, in sweep
    study.tell(trial=trial, state=state, values=values)
  File "/opt/conda/lib/python3.8/site-packages/optuna/study/study.py", line 652, in tell
    raise ValueError(values_conversion_failure_message)
ValueError: Trial 0 failed, because the objective function returned nan.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 386, in <lambda>
    lambda: hydra.multirun(
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 140, in multirun
    ret = sweeper.sweep(arguments=task_overrides)
  File "/opt/conda/lib/python3.8/site-packages/hydra_plugins/hydra_optuna_sweeper/optuna_sweeper.py", line 42, in sweep
    return self.sweeper.sweep(arguments)
  File "/opt/conda/lib/python3.8/site-packages/hydra_plugins/hydra_optuna_sweeper/_impl.py", line 240, in sweep
    study.tell(trial=trial, state=state, values=values)
  File "/opt/conda/lib/python3.8/site-packages/optuna/study/study.py", line 592, in tell
    raise ValueError(
ValueError: Values were told. Values cannot be specified when state is TrialState.PRUNED or TrialState.FAIL.

Expected Behavior

If a code returns NaN, then mark trial as failed and proceed without crashing.

System information

  • Hydra Version : 1.2
  • Python version : 3.8.12
  • Virtual environment type and version : NGC Docker container
  • Operating system : GNU/Linux

Activity

  1. Jasha10 commented on May 26, 2022

    @Jasha10
    Contributor

    Thanks for the report!

  2. self-assigned this
    on May 26, 2022
  3. linked a pull request that will close this issueUpdate Optuna Sweeper #2225on May 27, 2022
  4. added this to the Hydra 1.4.0 milestone on Jul 15, 2026
  5. omry commented on Jul 15, 2026

    @omry
    Collaborator

    Thanks for the report, and sorry for the long delay. I confirmed that this bug is still present on current main after the Optuna 4.9 upgrade. Hydra first calls study.tell() with state=COMPLETE and the returned NaN values. Optuna rejects those values, and Hydra then retries with state=FAIL while incorrectly passing the same values. Optuna requires values=None for failed trials, so the retry raises another exception.

    #2225 identified the correct one-line change, but it was closed as stale rather than because the fix was invalid. The current code also needs a regression test covering NaN results.

    I am tracking this for Hydra 1.4.0.

  6. saivedant169 commented on Sep 16, 2026

    @saivedant169
    Contributor

    Hi @omry, I'd like to pick this up and finish it per your review on #3333.

    Reproduced on main (plugin at 89e45f3, optuna 5.0.0): a NaN objective fails the COMPLETE tell, then the FAIL retry at _impl.py:375 raises again because it still passes the values. With values=None the trial is marked FAIL and the rest of the sweep runs.

    One question first: with the default max_failure_rate=0.0, the check at L380 only logs an error and the sweep carries on, since the job itself returned normally. Should a NaN stop the sweep there, or is log-and-continue fine? I'll write the test to match.

    jieru-hu is still assigned from 2022, happy to step back if that's still active.

  7. omry commented on Sep 16, 2026

    @omry
    Collaborator

    Feel free to submit a PR.
    Thanks.

  8. added a commit that references this issue on Sep 17, 2026
    b2f1949
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingoptunapluginPlugins realted issuessweep

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions