Skip to content

[Bug] PyTorch Lightning crashes run when optimizer is instatiated with Hydra (dev version) #1483

Closed

Description

@ashleve

🐛 Bug

Hi, should bugs related to hydra dev version be reported? If so I have a couple of interesting edge cases. Here comes the first one.

When instantiating optimizer with Hydra using PyTorch Lightning, using "betas" parameter crashes a run after end of first epoch.

def configure_optimizers(self):
        optim_conf = {
            "_target_": "torch.optim.Adam",
            "lr": 0.001,
            "eps": 1e-08,
            "weight_decay": 0,
            "betas": [ 0.9, 0.999 ] # without this param everything works correctly
        }
        
        # this doesnt work :(
        optim = hydra.utils.instantiate(optim_conf, params=self.parameters())
        
        # this works
        # optim = torch.optim.Adam(self.parameters())

        return optim

Everything works correctly when using hydra 1.0.6

Checklist

  • I checked on the latest version of Hydra
  • I created a minimal repro (See this for tips).

To reproduce

Repo:
https://github.com/hobogalaxy/lit-bug1

Stack trace/error message

Traceback (most recent call last):
  File "main.py", line 86, in <module>
    trainer.fit(model, train, val)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 498, in fit
    self.dispatch()
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 545, in dispatch
    self.accelerator.start_training(self)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 73, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 114, in start_training
    self._results = trainer.run_train()
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 668, in run_train
    self.train_loop.on_train_end()
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 134, in on_train_end
    self.check_checkpoint_callback(should_update=True, is_last=True)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 164, in check_checkpoint_callback
    cb.on_validation_end(self.trainer, model)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 212, in on_validation_end
    self.save_checkpoint(trainer, pl_module)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 262, in save_checkpoint
    self._save_last_checkpoint(trainer, pl_module, monitor_candidates)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 546, in _save_last_checkpoint
    self._save_model(last_filepath, trainer, pl_module)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 335, in _save_model
    self.save_function(filepath, self.save_weights_only)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/properties.py", line 327, in save_checkpoint
    self.checkpoint_connector.save_checkpoint(filepath, weights_only)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 404, in save_checkpoint
    atomic_save(checkpoint, filepath)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/utilities/cloud_io.py", line 63, in atomic_save
    torch.save(checkpoint, bytesbuffer)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/torch/serialization.py", line 372, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/torch/serialization.py", line 476, in _save
    pickler.dump(obj)
TypeError: cannot pickle 'generator' object

Should I post it on PyTorch Lightning repo instead?

Activity

  1. tmke8 commented on Mar 21, 2021

    @tmke8
    Contributor
  2. ashleve commented on Mar 21, 2021

    @ashleve
    Author

    @thomkeh thank you, this works. Closing.

  3. omry commented on Mar 22, 2021

    @omry
    Collaborator

    What version is this with?
    can you provide a complete runnable repro (as minimal as possible?)

  4. reopened this on Mar 22, 2021
  5. ashleve commented on Mar 22, 2021

    @ashleve
    Author

    @omry the repro was in my post: https://github.com/hobogalaxy/lit-bug1
    hydra-core==1.1.0.dev4
    torch==1.8.0
    pytorch-lightning==1.2.4

  6. omry commented on Mar 23, 2021

    @omry
    Collaborator

    Great, thanks.

  7. added this to the Hydra 1.1.0 milestone on Mar 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions