Repository navigation
Description
🐛 Bug
Description
In 1.4.0rc1, OmegaConf containers that instantiate() passes to a target keep a parent link back to the source config. In 1.3.x, _call_target detached them ("the parent link can cause issues when serializing objects in some scenarios"). That detach step is gone in 1.4, which I think is a side effect of the copy-free instantiate change (#3216).
As a result, pickling an instantiated object also pickles the entire source config tree:
- If any structured config in the tree can't be pickled (for example, a dataclass created programmatically or defined in a local scope), pickling the instantiated object now fails, even though the object itself is fine.
- Otherwise the pickle silently includes the whole config. In the size example below, pickling a 3-element list grows from 595 bytes to 6.4 MB.
This affects common workflows like torch.save, checkpointing, and multiprocessing/DDP spawn of instantiated objects.
Checklist
- I checked on the latest version of Hydra
- I created a minimal repro (See this for tips).
To reproduce
** Minimal Code/Config snippet to reproduce **
import pickle
from dataclasses import dataclass, field
import hydra
from hydra.utils import instantiate
class Model:
def __init__(self, sizes):
self.sizes = sizes
def make_config():
# e.g. a dataclass generated programmatically (not importable by pickle)
@dataclass
class ModelConf:
_target_: str = "__main__.Model"
sizes: list = field(default_factory=lambda: [1, 2, 3])
return ModelConf
model = instantiate(make_config())
print("hydra", hydra.__version__)
print("model.sizes parent:", model.sizes._get_parent())
pickle.dumps(model)** Stack trace/error message **
# hydra-core 1.3.7
hydra 1.3.7
model.sizes parent: None
(pickles fine)
# hydra-core 1.4.0rc1
hydra 1.4.0rc1
model.sizes parent: [1, 2, 3]
AttributeError: Can't get local object 'make_config.<locals>.ModelConf'
Size, with importable dataclasses only:
@dataclass
class ModelConf:
_target_: str = "__main__.Model"
sizes: list = field(default_factory=lambda: [1, 2, 3])
@dataclass
class Experiment:
model: ModelConf = field(default_factory=ModelConf)
data: list = field(default_factory=lambda: list(range(100_000))) # unrelated
cfg = instantiate(Experiment)
len(pickle.dumps(cfg.model.sizes))
# 1.3.7: 595 bytes
# 1.4.0rc1: 6,439,714 bytesExpected Behavior
Containers passed to a target are self-contained, as in 1.3.x. Pickling an instantiated object shouldn't depend on, or include, unrelated parts of the source config.
I understand the parent link now matters because interpolations are resolved lazily, so simply detaching could break interpolations that point outside the container. One option is to detach after resolving: for containers handed to a target (when they're not passed through by identity with _convert_="none" and _recursive_=False), resolve their interpolations and then detach.
System information
- Hydra Version : 1.4.0rc1 (compared against 1.3.7)
- OmegaConf Version : 2.4.0
- Python version : 3.12.12
- Virtual environment type and version : uv venv
- Operating system : Windows 11 (not OS-specific)
Additional context
Found while testing hydra-zen against 1.4.0rc1. hydra-zen's builds() generates its config dataclasses dynamically, so on 1.4 every object instantiated from a hydra-zen config with list/dict arguments becomes unpicklable.
Activity
omry commentedon Oct 12, 2026
The change in Hydra 1.4 is intentional: containers passed through to called code
should behave consistently with ordinary OmegaConf configs. Your examples expose
a serialization consequence of that choice that deserves clearer migration
guidance.
Lazy interpolation is OmegaConf's default behavior, and users are accustomed to
values resolving when they access them. The old instantiation path broke that
expectation by eagerly resolving the input configuration before constructing
targets. Even when a constructor received an OmegaConf container, its
interpolations had already been resolved.
That behavior came from an earlier serialization problem,
#1483. An Adam optimizer
retained a
ListConfigforbetas, whose parent links led back to the configcontaining the runtime
paramsgenerator. Saving a PyTorch Lightning checkpointfollowed those links and failed because the generator could not be pickled. The
fix eagerly resolved the configuration and detached OmegaConf arguments before
calling targets. Resolving first allowed detachment without breaking ancestor
interpolations, but it also removed their normal lazy behavior. Deep copying
already existed and added a separate cost.
Hydra 1.4 stops forcing whole-tree resolution before target calls. Values can
resolve during Hydra's traversal, inside the called function or constructor, or
later when application code accesses a retained container. This restores normal
OmegaConf behavior for containers passed through to called code, including the
ability to establish runtime state before accessing dependent values. Their
ancestry, resolver caches, and inherited flags remain available. With
_recursive_=False,_convert_="none", and no call-site overrides, configuredcontainers also retain their original identity. Calls on OmegaConf inputs without
call-site overrides avoid the additional full-tree copy too.
The cost is that receiving an OmegaConf container no longer implies receiving an
independent snapshot. Its parent links can retain the source configuration,
including unrelated data and Structured Config type metadata. That explains both
the larger pickle and the failure involving a dynamically generated or locally
defined schema class.
There are several ways to choose the boundary:
_convert_="partial"or"all"canprovide native containers.
"partial"preserves Structured Config containers,so it is not a universal serialization solution.
and reconstruct the relevant values before passing them. This moves the copying
and resolution cost to the point where isolation is needed.
must be supported by the serializer. A serializer supporting generated classes
can address the class failure, but still includes the retained graph.
An opt-in mode restoring the earlier copy/eager-resolution/detachment behavior is
possible in principle. It could let integrations request isolation automatically
through configuration, while callers retain the choice. Such a mode would
deliberately restore those costs and give up lazy dependencies and source
identity; its scope could be limited to isolation without restoring every Hydra
1.3 behavior.
For hydra-zen, the existing
hydra_convertcontrol is an immediate option for configs whose targets can accept native
containers, with the Structured Config caveat above. Its
zen_dataclassoptions also support pickle-compatible generated classes: specify an importable
module and class name, and bind the class under that name in the module. That
addresses the class-lookup failure, but does not reduce the retained graph. If a
config-selectable isolation mode were added, hydra-zen could expose it through its
config-generation API and record the choice in generated configs, so ordinary
Hydra instantiation would honor it without requiring a special hydra-zen entry
point. This could provide an explicit compatibility option for consumers needing
independent snapshots while leaving normal OmegaConf semantics available.
The current migration guidance should explain these serialization and isolation
consequences more explicitly, including for recursively rebuilt containers.
Update tests for Hydra 1.4 instantiate semantics
rsokl commentedon Oct 13, 2026
Thanks for the detailed explanation. The motivation makes sense, and restoring lazy interpolation is a reasonable default for Hydra.
I want to flag how this lands for hydra-zen users, though, because it's more severe than a larger pickle. Every hydra-zen config is a dynamically generated dataclass. In 1.4, any instantiated object that retains an OmegaConf container argument now carries a link to that generated class, and pickling fails outright. This is the most basic hydra-zen usage:
The same happens for nested configs (
make_config(trainer=builds(Trainer, model=builds(Model, layers=[1, 2])))→cfg.trainer.modelis unpicklable). Many hydra-zen users are ML practitioners, and pickling instantiated objects is routine for them: Lightning checkpoints (exactly the #1483 scenario),torch.save, DataLoader workers underspawn(the default on macOS and Windows), and multiprocessing generally. For them, upgrading to 1.4 turns working code into hard failures.The workarounds don't scale well for us:
zen_dataclass={"module": ..., "cls_name": ...}requires everybuilds/make_configcall, including nested ones, to be bound to a unique importable module-level name. That defeats the point of generating configs dynamically, and as you note, it still retains the whole graph._convert_="all"works, but it changes what every target receives, and users would need to know to apply it everywhere.So I'd strongly support the opt-in isolation mode you describe, as a config-level setting (something like an
_isolate_flag thatinstantiatehonors per node). hydra-zen could then set it in the configs it generates, so ordinaryhydra.utils.instantiateand@hydra.mainapps get the 1.3 behavior for hydra-zen configs without a special entry point. Users who want lazy dependencies could opt out. Restoring just the isolation part (resolved, detached containers), as you suggest, is all we need.rsokl commentedon Oct 13, 2026
(but also the larger pickles is really really bad for users too!)
Add opt-in instantiation isolation
Add opt-in instantiation isolation
Add opt-in instantiation isolation (#3547)
omry commentedon Oct 14, 2026
Please evaluate from main. I will aim for a second rc in the coming days/week.