Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't working
Milestone
Description
🐛 Bug
Hi, should bugs related to hydra dev version be reported? If so I have a couple of interesting edge cases. Here comes the first one.
When instantiating optimizer with Hydra using PyTorch Lightning, using "betas" parameter crashes a run after end of first epoch.
def configure_optimizers(self):
optim_conf = {
"_target_": "torch.optim.Adam",
"lr": 0.001,
"eps": 1e-08,
"weight_decay": 0,
"betas": [ 0.9, 0.999 ] # without this param everything works correctly
}
# this doesnt work :(
optim = hydra.utils.instantiate(optim_conf, params=self.parameters())
# this works
# optim = torch.optim.Adam(self.parameters())
return optimEverything works correctly when using hydra 1.0.6
Checklist
- I checked on the latest version of Hydra
- I created a minimal repro (See this for tips).
To reproduce
Repo:
https://github.com/hobogalaxy/lit-bug1
Stack trace/error message
Traceback (most recent call last):
File "main.py", line 86, in <module>
trainer.fit(model, train, val)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 498, in fit
self.dispatch()
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 545, in dispatch
self.accelerator.start_training(self)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 73, in start_training
self.training_type_plugin.start_training(trainer)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 114, in start_training
self._results = trainer.run_train()
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 668, in run_train
self.train_loop.on_train_end()
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 134, in on_train_end
self.check_checkpoint_callback(should_update=True, is_last=True)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 164, in check_checkpoint_callback
cb.on_validation_end(self.trainer, model)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 212, in on_validation_end
self.save_checkpoint(trainer, pl_module)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 262, in save_checkpoint
self._save_last_checkpoint(trainer, pl_module, monitor_candidates)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 546, in _save_last_checkpoint
self._save_model(last_filepath, trainer, pl_module)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 335, in _save_model
self.save_function(filepath, self.save_weights_only)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/properties.py", line 327, in save_checkpoint
self.checkpoint_connector.save_checkpoint(filepath, weights_only)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 404, in save_checkpoint
atomic_save(checkpoint, filepath)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/utilities/cloud_io.py", line 63, in atomic_save
torch.save(checkpoint, bytesbuffer)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/torch/serialization.py", line 372, in save
_save(obj, opened_zipfile, pickle_module, pickle_protocol)
File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/torch/serialization.py", line 476, in _save
pickler.dump(obj)
TypeError: cannot pickle 'generator' object
Should I post it on PyTorch Lightning repo instead?
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
tmke8 commentedon Mar 21, 2021
I think you might need
_convert_="partial". See here: https://hydra.cc/docs/next/advanced/instantiate_objects/overview#parameter-conversion-strategiesashleve commentedon Mar 21, 2021
@thomkeh thank you, this works. Closing.
omry commentedon Mar 22, 2021
What version is this with?
can you provide a complete runnable repro (as minimal as possible?)
ashleve commentedon Mar 22, 2021
@omry the repro was in my post: https://github.com/hobogalaxy/lit-bug1
hydra-core==1.1.0.dev4
torch==1.8.0
pytorch-lightning==1.2.4
omry commentedon Mar 23, 2021
Great, thanks.