Skip to content

[Bug] 1.4.0rc1: instantiated objects keep a parent link to the source config, breaking pickling #3537

Closed

Description

@rsokl

🐛 Bug

Description

In 1.4.0rc1, OmegaConf containers that instantiate() passes to a target keep a parent link back to the source config. In 1.3.x, _call_target detached them ("the parent link can cause issues when serializing objects in some scenarios"). That detach step is gone in 1.4, which I think is a side effect of the copy-free instantiate change (#3216).

As a result, pickling an instantiated object also pickles the entire source config tree:

  • If any structured config in the tree can't be pickled (for example, a dataclass created programmatically or defined in a local scope), pickling the instantiated object now fails, even though the object itself is fine.
  • Otherwise the pickle silently includes the whole config. In the size example below, pickling a 3-element list grows from 595 bytes to 6.4 MB.

This affects common workflows like torch.save, checkpointing, and multiprocessing/DDP spawn of instantiated objects.

Checklist

  • I checked on the latest version of Hydra
  • I created a minimal repro (See this for tips).

To reproduce

** Minimal Code/Config snippet to reproduce **

import pickle
from dataclasses import dataclass, field

import hydra
from hydra.utils import instantiate


class Model:
    def __init__(self, sizes):
        self.sizes = sizes


def make_config():
    # e.g. a dataclass generated programmatically (not importable by pickle)
    @dataclass
    class ModelConf:
        _target_: str = "__main__.Model"
        sizes: list = field(default_factory=lambda: [1, 2, 3])

    return ModelConf


model = instantiate(make_config())
print("hydra", hydra.__version__)
print("model.sizes parent:", model.sizes._get_parent())
pickle.dumps(model)

** Stack trace/error message **

# hydra-core 1.3.7
hydra 1.3.7
model.sizes parent: None
(pickles fine)

# hydra-core 1.4.0rc1
hydra 1.4.0rc1
model.sizes parent: [1, 2, 3]
AttributeError: Can't get local object 'make_config.<locals>.ModelConf'

Size, with importable dataclasses only:

@dataclass
class ModelConf:
    _target_: str = "__main__.Model"
    sizes: list = field(default_factory=lambda: [1, 2, 3])

@dataclass
class Experiment:
    model: ModelConf = field(default_factory=ModelConf)
    data: list = field(default_factory=lambda: list(range(100_000)))  # unrelated

cfg = instantiate(Experiment)
len(pickle.dumps(cfg.model.sizes))
# 1.3.7:        595 bytes
# 1.4.0rc1: 6,439,714 bytes

Expected Behavior

Containers passed to a target are self-contained, as in 1.3.x. Pickling an instantiated object shouldn't depend on, or include, unrelated parts of the source config.

I understand the parent link now matters because interpolations are resolved lazily, so simply detaching could break interpolations that point outside the container. One option is to detach after resolving: for containers handed to a target (when they're not passed through by identity with _convert_="none" and _recursive_=False), resolve their interpolations and then detach.

System information

  • Hydra Version : 1.4.0rc1 (compared against 1.3.7)
  • OmegaConf Version : 2.4.0
  • Python version : 3.12.12
  • Virtual environment type and version : uv venv
  • Operating system : Windows 11 (not OS-specific)

Additional context

Found while testing hydra-zen against 1.4.0rc1. hydra-zen's builds() generates its config dataclasses dynamically, so on 1.4 every object instantiated from a hydra-zen config with list/dict arguments becomes unpicklable.

Activity

  1. omry commented on Oct 12, 2026

    @omry
    Collaborator

    The change in Hydra 1.4 is intentional: containers passed through to called code
    should behave consistently with ordinary OmegaConf configs. Your examples expose
    a serialization consequence of that choice that deserves clearer migration
    guidance.

    Lazy interpolation is OmegaConf's default behavior, and users are accustomed to
    values resolving when they access them. The old instantiation path broke that
    expectation by eagerly resolving the input configuration before constructing
    targets. Even when a constructor received an OmegaConf container, its
    interpolations had already been resolved.

    That behavior came from an earlier serialization problem,
    #1483. An Adam optimizer
    retained a ListConfig for betas, whose parent links led back to the config
    containing the runtime params generator. Saving a PyTorch Lightning checkpoint
    followed those links and failed because the generator could not be pickled. The
    fix eagerly resolved the configuration and detached OmegaConf arguments before
    calling targets. Resolving first allowed detachment without breaking ancestor
    interpolations, but it also removed their normal lazy behavior. Deep copying
    already existed and added a separate cost.

    Hydra 1.4 stops forcing whole-tree resolution before target calls. Values can
    resolve during Hydra's traversal, inside the called function or constructor, or
    later when application code accesses a retained container. This restores normal
    OmegaConf behavior for containers passed through to called code, including the
    ability to establish runtime state before accessing dependent values. Their
    ancestry, resolver caches, and inherited flags remain available. With
    _recursive_=False, _convert_="none", and no call-site overrides, configured
    containers also retain their original identity. Calls on OmegaConf inputs without
    call-site overrides avoid the additional full-tree copy too.

    The cost is that receiving an OmegaConf container no longer implies receiving an
    independent snapshot. Its parent links can retain the source configuration,
    including unrelated data and Structured Config type metadata. That explains both
    the larger pickle and the failure involving a dynamically generated or locally
    defined schema class.

    There are several ways to choose the boundary:

    • If the constructor needs ordinary data, _convert_="partial" or "all" can
      provide native containers. "partial" preserves Structured Config containers,
      so it is not a universal serialization solution.
    • If the constructor needs an independent OmegaConf config, explicitly resolve
      and reconstruct the relevant values before passing them. This moves the copying
      and resolution cost to the point where isolation is needed.
    • If retaining the config graph is intentional, its contents and schema classes
      must be supported by the serializer. A serializer supporting generated classes
      can address the class failure, but still includes the retained graph.

    An opt-in mode restoring the earlier copy/eager-resolution/detachment behavior is
    possible in principle. It could let integrations request isolation automatically
    through configuration, while callers retain the choice. Such a mode would
    deliberately restore those costs and give up lazy dependencies and source
    identity; its scope could be limited to isolation without restoring every Hydra
    1.3 behavior.

    For hydra-zen, the existing
    hydra_convert
    control is an immediate option for configs whose targets can accept native
    containers, with the Structured Config caveat above. Its
    zen_dataclass
    options also support pickle-compatible generated classes: specify an importable
    module and class name, and bind the class under that name in the module. That
    addresses the class-lookup failure, but does not reduce the retained graph. If a
    config-selectable isolation mode were added, hydra-zen could expose it through its
    config-generation API and record the choice in generated configs, so ordinary
    Hydra instantiation would honor it without requiring a special hydra-zen entry
    point. This could provide an explicit compatibility option for consumers needing
    independent snapshots while leaving normal OmegaConf semantics available.

    The current migration guidance should explain these serialization and isolation
    consequences more explicitly, including for recursively rebuilt containers.

  2. rsokl commented on Oct 13, 2026

    @rsokl
    ContributorAuthor

    Thanks for the detailed explanation. The motivation makes sense, and restoring lazy interpolation is a reasonable default for Hydra.

    I want to flag how this lands for hydra-zen users, though, because it's more severe than a larger pickle. Every hydra-zen config is a dynamically generated dataclass. In 1.4, any instantiated object that retains an OmegaConf container argument now carries a link to that generated class, and pickling fails outright. This is the most basic hydra-zen usage:

    import pickle
    from hydra_zen import builds, instantiate
    
    class Optimizer:  # stand-in for e.g. torch.optim.Adam
        def __init__(self, lr: float, betas: list[float]):
            self.lr, self.betas = lr, betas
    
    opt = instantiate(builds(Optimizer, lr=0.1, betas=[0.9, 0.999]))
    pickle.dumps(opt)
    # hydra-core 1.3.7:   ok
    # hydra-core 1.4.0rc1: PicklingError: Can't pickle <class 'types.Builds_Optimizer'>:
    #                      attribute lookup Builds_Optimizer on types failed

    The same happens for nested configs (make_config(trainer=builds(Trainer, model=builds(Model, layers=[1, 2]))) → cfg.trainer.model is unpicklable). Many hydra-zen users are ML practitioners, and pickling instantiated objects is routine for them: Lightning checkpoints (exactly the #1483 scenario), torch.save, DataLoader workers under spawn (the default on macOS and Windows), and multiprocessing generally. For them, upgrading to 1.4 turns working code into hard failures.

    The workarounds don't scale well for us:

    • zen_dataclass={"module": ..., "cls_name": ...} requires every builds/make_config call, including nested ones, to be bound to a unique importable module-level name. That defeats the point of generating configs dynamically, and as you note, it still retains the whole graph.
    • _convert_="all" works, but it changes what every target receives, and users would need to know to apply it everywhere.

    So I'd strongly support the opt-in isolation mode you describe, as a config-level setting (something like an _isolate_ flag that instantiate honors per node). hydra-zen could then set it in the configs it generates, so ordinary hydra.utils.instantiate and @hydra.main apps get the 1.3 behavior for hydra-zen configs without a special entry point. Users who want lazy dependencies could opt out. Restoring just the isolation part (resolved, detached containers), as you suggest, is all we need.

  3. rsokl commented on Oct 13, 2026

    @rsokl
    ContributorAuthor

    (but also the larger pickles is really really bad for users too!)

  4. omry commented on Oct 14, 2026

    @omry
    Collaborator

    Please evaluate from main. I will aim for a second rc in the coming days/week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions