Repository navigation
Closed
Closed
Copy link
Labels
bugSomething isn't workingSomething isn't workingoptunapluginPlugins realted issuesPlugins realted issuessweep
Milestone
Description
🐛 Bug
Description
I perform hp search for a deep learning model and there are situations when model diverges and starts producing nans in it's output. My code exits gracefully returning tuple of nans as metrics. This is an expected behavior. But optuna sweeper doesn't think so.
- The problem starts here: https://github.com/facebookresearch/hydra/blob/main/plugins/hydra_optuna_sweeper/hydra_plugins/hydra_optuna_sweeper/_impl.py#L377 . Here values is a tuple of NaNs and state is COMPLETED (because the experiment code exited gracefully)
study.tellraises an error here: https://github.com/optuna/optuna/blob/release-v2.10.0/optuna/study/study.py#L652
because of the check preformed here https://github.com/optuna/optuna/blob/release-v2.10.0/optuna/study/_optimize.py#L319- Error is finally reraised here: https://github.com/facebookresearch/hydra/blob/main/plugins/hydra_optuna_sweeper/hydra_plugins/hydra_optuna_sweeper/_impl.py#L391 ,which ultimately crashes hydra.
Checklist
- I checked on the latest version of Hydra
- I created a minimal repro (See this for tips).
To reproduce
** Minimal Code/Config snippet to reproduce **
** Stack trace/error message **
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/hydra_plugins/hydra_optuna_sweeper/_impl.py", line 237, in sweep
study.tell(trial=trial, state=state, values=values)
File "/opt/conda/lib/python3.8/site-packages/optuna/study/study.py", line 652, in tell
raise ValueError(values_conversion_failure_message)
ValueError: Trial 0 failed, because the objective function returned nan.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
return func()
File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 386, in <lambda>
lambda: hydra.multirun(
File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 140, in multirun
ret = sweeper.sweep(arguments=task_overrides)
File "/opt/conda/lib/python3.8/site-packages/hydra_plugins/hydra_optuna_sweeper/optuna_sweeper.py", line 42, in sweep
return self.sweeper.sweep(arguments)
File "/opt/conda/lib/python3.8/site-packages/hydra_plugins/hydra_optuna_sweeper/_impl.py", line 240, in sweep
study.tell(trial=trial, state=state, values=values)
File "/opt/conda/lib/python3.8/site-packages/optuna/study/study.py", line 592, in tell
raise ValueError(
ValueError: Values were told. Values cannot be specified when state is TrialState.PRUNED or TrialState.FAIL.
Expected Behavior
If a code returns NaN, then mark trial as failed and proceed without crashing.
System information
- Hydra Version : 1.2
- Python version : 3.8.12
- Virtual environment type and version : NGC Docker container
- Operating system : GNU/Linux
Reacted by Tobias and Muthu Kamalan
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingoptunapluginPlugins realted issuesPlugins realted issuessweep
Activity
Jasha10 commentedon May 26, 2022
Thanks for the report!
omry commentedon Jul 15, 2026
Thanks for the report, and sorry for the long delay. I confirmed that this bug is still present on current
mainafter the Optuna 4.9 upgrade. Hydra first callsstudy.tell()withstate=COMPLETEand the returned NaN values. Optuna rejects those values, and Hydra then retries withstate=FAILwhile incorrectly passing the same values. Optuna requiresvalues=Nonefor failed trials, so the retry raises another exception.#2225 identified the correct one-line change, but it was closed as stale rather than because the fix was invalid. The current code also needs a regression test covering NaN results.
I am tracking this for Hydra 1.4.0.
saivedant169 commentedon Sep 16, 2026
Hi @omry, I'd like to pick this up and finish it per your review on #3333.
Reproduced on main (plugin at 89e45f3, optuna 5.0.0): a NaN objective fails the COMPLETE tell, then the FAIL retry at
_impl.py:375raises again because it still passes the values. Withvalues=Nonethe trial is marked FAIL and the rest of the sweep runs.One question first: with the default
max_failure_rate=0.0, the check at L380 only logs an error and the sweep carries on, since the job itself returned normally. Should a NaN stop the sweep there, or is log-and-continue fine? I'll write the test to match.jieru-hu is still assigned from 2022, happy to step back if that's still active.
omry commentedon Sep 16, 2026
Feel free to submit a PR.
Thanks.
Fix Optuna sweeper crash on NaN or invalid trial results