You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Reproducing a Hydra run from previous Hydra job configs #1805
When I want to reproduce the experiment, I can simply load the config.yaml: python run.py --config-path /logs/runs/.../.hydra --config-name config.yaml
The problem is, config.yaml doesn't contain the hydra logs output paths configuration, which means that after running the line above, the logging path will revert to default.
Is there a workaround for that?
Is it possible to additionally load hydra specific config from some other path with a simple oneliner?
Hi, how can I load the config with properly formatted placeholders like ${abc.xyz} in scripts? Can I do this without wrapping another main function? I'm not trying to re-run, but to use the saved output. To instantiate some classes related to the output, I'd like to read the params used saved in .hydra/config.yaml but some of them are placeholders.
@Jasha10 Hi, I tried to load .hydra/config.yaml of a run by OmegaConf.load, but got the following error when I access the config.work_dir
UnsupportedInterpolationType: Unsupported interpolation type hydra
full_key: work_dir
object_type=dict
And I don't seem to see any information related to the interpolated values of work_dir and when in .hydra/config.yaml, .hydra/hydra.yaml and .hydra/overrides.yaml of that run .
@Yevgnen thank you for trying it out and the repro code. Could you provide a bit more details on what you tried with OmegaConf.load so we can repro? Many thanks
I've had success reproducing the config by using the saved outputs/.../.hydra/overrides.yaml file. Specifically, suppose I run my_app.py as follows:
$ python my_app.py a=b +c/d=e
This produces a file outputs/.../.hydra/overrides.yaml with the following contents:
- a=b
- +c/d=e
This list of overrides can be used to reproduce the config. The following recipe has worked for me:
withhydra.initialize(config_path=config_path): # same config_path as used by @hydra.mainrecomposed_config=hydra.compose(
config_name="main", # same config_name as used by @hydra.mainoverrides=OmegaConf.load("outputs/.../.hydra/overrides.yaml"),
)
There are a few caveats here:
Currently the ${hydra:...} resolver cannot be used outside the context of @hydra.main. Using compose does not play well with the ${hydra:...} resolver.
If your config uses non-deterministic resolvers, e.g. to generate a random number or get the current time of day, those resolvers will be run anew (and will probably give different values from the original run). If you need to re-use the values from the initial run, you could try something like this: config = OmegaConf.merge(recomposed_config, OmegaConf.load("outputs/.../.hydra/config.yaml")) EDIT: I think that config = OmegaConf.merge(recomposed_config, OmegaConf.load("outputs/.../.hydra/config.yaml")) will not work because .hydra/config.yaml may contain unresolved interpolations/OmegaConf resolvers. A yaml file containing resolved values could be useful here, as @jieru-hu pointed out below.
how about we jus save the final composed config of everything related to a job in a separate YAML file (everything including Hydra and the job config, all resolved, no OmegConf syntax in the YAML) we can also provide an experimental CLI option for rerun an application from that finalized composed config. we will only support single run for now and see how it goes.
Saving the finalized YAML will also help with debugging, I've heard from users that the YAML files saved has OC resolvers in them and sometimes is hard to figure out what's going on.
Reacted by Robert Jan Schlimbach, Yihui "Ray" Ren, Bram O. and Addison Klinke
Saving a resolved yaml config (e.g. outputs/.../.hydra/resolved.yaml) could certainly be useful! I've just edited the end of my previous comment since this made me realize that loading .hydra/config.yaml is not good enough to reconstruct the resolved values.
That being said, creating resolved.yaml could be expensive, e.g. if the user has defined custom resolvers to do some complicated calculation. For this reason, I think it might make sense to have automatic saving of resolved.yaml be an opt-in feature (or at least have the option to disable saving resolved.yaml).
Another potential complication is that some application use-cases could rely on interpolations or OmegaConf resolvers being unresolved, e.g. if the user app wants to call these resolvers at runtime.
For re-running the application, more information is needed than just what is in resolved.yaml. The resolved yaml config will not contain typing information. Info about structured config types or enum types cannot be reconstructed from the resolved.yaml file. This is why the compose+overrides.yaml approach is useful: it produces a typed config.
Reacted by Jieru Hu, Boxi Xia, Yevgnen and Addison Klinke
thanks @Jasha10 those are great points! I agree that eagerly resolve all configs is not the best way to go - especially when user applications also plays a role here.
arguably, we should be able to repro a run if the task_cfg and hydra_cfg passed into the task_function stay the same, the rest will be left up to the user application. I think one lightweight way of enabling this could be pickling JobReturn both before and after the run of user application.
Right now JobReturn( which can be very helpful) is not saved anywhere - I think we can enable the opt in of this by adding an experimental CallBack which dumps the JobReturn for each run.
The second part of this would providing an experiment CLI option that takes in a JobReturn pickle and single run the application.
I think that using a Callback to pickle the JobReturn (and later using a CLI option to load the pickle) sounds like a great idea.
I think one lightweight way of enabling this could be pickling JobReturn both before and after the run of user application.
To pickle before the run, we might need to change the signature of the on_job_start hook, since on_job_start currently does not accept a JobReturn argument.
I think that using a Callback to pickle the JobReturn (and later using a CLI option to load the pickle) sounds like a great idea.
I think one lightweight way of enabling this could be pickling JobReturn both before and after the run of user application.
To pickle before the run, we might need to change the signature of the on_job_start hook, since on_job_start currently does not accept a JobReturn argument.
I think for now we can just pickle the config object instead of a uncomplete JobReturn - that seems like it should be enough (both task and hydra config are included in the config object)
Hi @ashleve@Yevgnen
i just created #2098 to address this. It is not a perfect solution but could be useful for what you are trying to achieve.
If you are interested, here is the documentation on this how to use this particular experimental CLI option.
I have some questions to better understand the differences between these approaches
Do approaches 1 and 2 end up generating the same config object?
If the above answer is yes, should approach 1 be preferred whenever @hydra.main is applicable? I say that based on this docs recommendation
Please avoid using the Compose API in cases where @hydra.main() can be used. Doing so forfeits many of the benefits of Hydra (e.g., Tab completion, Multirun, Working directory management, Logging management and more)
Is approach 3 only necessary if you want to preserve the original output directory and append to the old log file? Are there other advantages over approaches 1 or 2 that I'm missing?
Do approaches 1 and 2 end up generating the same config object?
Approaches 1 involves a round-trip through yaml, which does not always capture the original config in full fidelity. Meanwhile, the approach2 config should be basically identical to the original config.
To see some of the differences, you can experiment with round-tripping a config through yaml as follows:
approach1=OmegaConf.create(OmegaConf.to_yaml(original_config)) # round-trip through yaml
Approach 1 will produce an untyped config (i.e. a config that is not structured). This means you'll miss out on the runtime type-safety that comes with structured configs. Also, you'll see a difference when calling routines such as OmegaConf.to_object (since OmegaConf.to_object treats structured configs differently from unstructured configs). Another pitfall with the round-trip through yaml comes up if you use Enum values in your config:
If the above answer is yes, should approach 1 be preferred whenever @hydra.main is applicable?
If you use structured configs or enum values, I'd recommend against approach 1.
Another difference between approaches 1 and 2 is that, if you refactor your python dataclasses or change your input yaml files, the result of approach 2 will change accordingly. Approach 2 reconstructs the config using Hydra's compose process; the output of that process can change if your dataclasses or your input yaml files change. Meanwhile, the result of approach 1 is always the same (as it depend only on the static config.yaml file that was the output from your previous experiment).
The docs do recommend using @hydra.main over compose if possible, since @hydra.main can do several things that compose cannot (including sweeping, launching, etc).
Actually, approach 2 can be adapted to work with @hydra.main if you don't mind monkeying with sys.argv:
importsys
...
# approach2 adapted to @hydra.main:overrides: list[str] =list(OmegaConf.load("outputs/.../.hydra/overrides.yaml"))
sys.argv+=overrides@hydra.main(config_path=..., config_name="main") # use same config_path and config_name as you used in the original rundefmy_app(cfg):
...
I wonder if it would make sense for @hydra.main to expose an overrides parameter so that we could avoid this sys.argv hack...
Is approach 3 only necessary if you want to preserve the original output directory and append to the old log file? Are there other advantages over approaches 1 or 2 that I'm missing?
Yes, using the --experimental-rerun flag is only necessary if you want to preserve the original output directory and append to the old log file.
There is also a fourth approach that combines a few of these ideas: you can use PickleJobInfoCallback to produce a config.pickle file, and then directly call pickle.load(open("outputs/.../.hydra/config.pickle", "rb")) to load that pickled config later. This fourth approach circumvents the need to call @hydra.main or compose.
In summary, here are the four discussed approaches for re-animating a config from a previous experiment:
Approach 1: Run Hydra with config_name="outputs/../.hydra/config.yaml"
Approach 1 involves a round-trip through yaml, which does not always capture the original config in full fidelity
Thanks for clarifying this subtle difference - that makes a lot of sense
Approach 1 always produces the same config even if you refactor your application / input yaml files. This may or may not be desirable.
I'll have to think more about whether this is desirable for my ML experiments. Typically, I would plan to checkout the git commit of the original run before reproducing, in which case there couldn't be any changes to the application code
I wonder if it would make sense for @hydra.main to expose an overrides parameter so that we could avoid this sys.argv hack...
I agree it could definitely be convenient to officially support that pattern. Worth opening a new issue?
Activity
omry commentedon Aug 31, 2021
This is not an officially supported workflow.
Try overriding the run path as well from the command line.
Yevgnen commentedon Feb 10, 2022
@omry
Hi, how can I load the config with properly formatted placeholders like
${abc.xyz}in scripts? Can I do this without wrapping another main function? I'm not trying to re-run, but to use the saved output. To instantiate some classes related to the output, I'd like to read the params used saved in.hydra/config.yamlbut some of them are placeholders.Jasha10 commentedon Feb 10, 2022
@Yevgnen see OmegaConf.load.
Yevgnen commentedon Feb 18, 2022
config/config.yaml
main.py
@Jasha10 Hi, I tried to load
.hydra/config.yamlof a run byOmegaConf.load, but got the following error when I access theconfig.work_dirAnd I don't seem to see any information related to the interpolated values of
work_dirandwhenin.hydra/config.yaml,.hydra/hydra.yamland.hydra/overrides.yamlof that run .jieru-hu commentedon Feb 25, 2022
@Yevgnen thank you for trying it out and the repro code. Could you provide a bit more details on what you tried with OmegaConf.load so we can repro? Many thanks
[-]Reproducing the run[/-][+]Reproducing a Hydra run from previous Hydra job configs[/+]jieru-hu commentedon Feb 25, 2022
reopening, we should look into this and at least provide some documentation on the approach.
Jasha10 commentedon Mar 2, 2022
I've had success reproducing the config by using the saved
outputs/.../.hydra/overrides.yamlfile. Specifically, suppose I runmy_app.pyas follows:This produces a file
outputs/.../.hydra/overrides.yamlwith the following contents:This list of overrides can be used to reproduce the config. The following recipe has worked for me:
There are a few caveats here:
${hydra:...}resolver cannot be used outside the context of@hydra.main. Usingcomposedoes not play well with the${hydra:...}resolver.If your config uses non-deterministic resolvers, e.g. to generate a random number or get the current time of day, those resolvers will be run anew (and will probably give different values from the original run). If you need to re-use the values from the initial run, you could try something like this:
config = OmegaConf.merge(recomposed_config, OmegaConf.load("outputs/.../.hydra/config.yaml"))EDIT: I think that
config = OmegaConf.merge(recomposed_config, OmegaConf.load("outputs/.../.hydra/config.yaml"))will not work because.hydra/config.yamlmay contain unresolved interpolations/OmegaConf resolvers. A yaml file containing resolved values could be useful here, as @jieru-hu pointed out below.jieru-hu commentedon Mar 4, 2022
how about we jus save the final composed config of everything related to a job in a separate YAML file (everything including Hydra and the job config, all resolved, no OmegConf syntax in the YAML) we can also provide an experimental CLI option for rerun an application from that finalized composed config. we will only support single run for now and see how it goes.
Saving the finalized YAML will also help with debugging, I've heard from users that the YAML files saved has OC resolvers in them and sometimes is hard to figure out what's going on.
Jasha10 commentedon Mar 5, 2022
Saving a resolved yaml config (e.g.
outputs/.../.hydra/resolved.yaml) could certainly be useful! I've just edited the end of my previous comment since this made me realize that loading.hydra/config.yamlis not good enough to reconstruct the resolved values.That being said, creating
resolved.yamlcould be expensive, e.g. if the user has defined custom resolvers to do some complicated calculation. For this reason, I think it might make sense to have automatic saving ofresolved.yamlbe an opt-in feature (or at least have the option to disable savingresolved.yaml).Another potential complication is that some application use-cases could rely on interpolations or OmegaConf resolvers being unresolved, e.g. if the user app wants to call these resolvers at runtime.
For re-running the application, more information is needed than just what is in
resolved.yaml. The resolved yaml config will not contain typing information. Info about structured config types or enum types cannot be reconstructed from theresolved.yamlfile. This is why thecompose+overrides.yamlapproach is useful: it produces a typed config.jieru-hu commentedon Mar 14, 2022
thanks @Jasha10 those are great points! I agree that eagerly resolve all configs is not the best way to go - especially when user applications also plays a role here.
arguably, we should be able to repro a run if the
task_cfgandhydra_cfgpassed into thetask_functionstay the same, the rest will be left up to the user application. I think one lightweight way of enabling this could be picklingJobReturnboth before and after the run of user application.Right now
JobReturn( which can be very helpful) is not saved anywhere - I think we can enable the opt in of this by adding an experimentalCallBackwhich dumps the JobReturn for each run.The second part of this would providing an experiment CLI option that takes in a JobReturn pickle and single run the application.
Jasha10 commentedon Mar 14, 2022
I think that using a
Callbackto pickle theJobReturn(and later using a CLI option to load the pickle) sounds like a great idea.To pickle before the run, we might need to change the signature of the
on_job_starthook, sinceon_job_startcurrently does not accept aJobReturnargument.jieru-hu commentedon Mar 14, 2022
I think for now we can just pickle the config object instead of a uncomplete JobReturn - that seems like it should be enough (both task and hydra config are included in the
configobject)jieru-hu commentedon Mar 17, 2022
Hi @ashleve @Yevgnen
i just created #2098 to address this. It is not a perfect solution but could be useful for what you are trying to achieve.
If you are interested, here is the documentation on this how to use this particular experimental CLI option.
addisonklinke commentedon Nov 7, 2022
I'm see three potential approaches suggested in this thread
--config-pathand--config-nameflags to point the script to an old config [comment]--experimental-rerunflag [comment]I have some questions to better understand the differences between these approaches
@hydra.mainis applicable? I say that based on this docs recommendationThanks everyone for the clarification!
Jasha10 commentedon Nov 8, 2022
Approaches 1 involves a round-trip through
yaml, which does not always capture the original config in full fidelity. Meanwhile, the approach2 config should be basically identical to the original config.To see some of the differences, you can experiment with round-tripping a config through yaml as follows:
Approach 1 will produce an untyped config (i.e. a config that is not structured). This means you'll miss out on the runtime type-safety that comes with structured configs. Also, you'll see a difference when calling routines such as
OmegaConf.to_object(sinceOmegaConf.to_objecttreats structured configs differently from unstructured configs). Another pitfall with the round-trip through yaml comes up if you useEnumvalues in your config:If you use structured configs or enum values, I'd recommend against approach 1.
Another difference between approaches 1 and 2 is that, if you refactor your python dataclasses or change your input yaml files, the result of approach 2 will change accordingly. Approach 2 reconstructs the config using Hydra's compose process; the output of that process can change if your dataclasses or your input yaml files change. Meanwhile, the result of approach 1 is always the same (as it depend only on the static
config.yamlfile that was the output from your previous experiment).The docs do recommend using
@hydra.mainovercomposeif possible, since@hydra.maincan do several things thatcomposecannot (including sweeping, launching, etc).Actually, approach 2 can be adapted to work with
@hydra.mainif you don't mind monkeying withsys.argv:I wonder if it would make sense for
@hydra.mainto expose anoverridesparameter so that we could avoid thissys.argvhack...Yes, using the
--experimental-rerunflag is only necessary if you want to preserve the original output directory and append to the old log file.There is also a fourth approach that combines a few of these ideas: you can use PickleJobInfoCallback to produce a
config.picklefile, and then directly callpickle.load(open("outputs/.../.hydra/config.pickle", "rb"))to load that pickled config later. This fourth approach circumvents the need to call@hydra.mainorcompose.In summary, here are the four discussed approaches for re-animating a config from a previous experiment:
Approach 1: Run Hydra with
config_name="outputs/../.hydra/config.yaml"This approach comes from the OP.
outputs/.../.hydra/config.yamlfile.config.yamlto re-run experiments using the--config-nameand--config-pathflags:Pros:
Cons:
Approach 1 always produces the same config even if you refactor your application / input yaml files. This may or may not be desirable.
Approach 2: Recompose the config with
overrides="outputs/../.hydra/overrides.yaml"This approach comes from my comment above.
outputs/.../.hydra/overrides.yamlfile.overrides.yamland using those overrides as input tocomposeor@hydra.main.Pros:
Cons:
@hydra.mainrequires monkeying withsys.argv.The config produced by approach 2 will change if you refactor your dataclasses or your input yaml files. This may or may not be desirable.
Approach 3: PickleJobInfoCallback +
--experimental-rerunThis approach is only necessary if you want to re-use the same output directory and append to the log file from your previous experiment.
Approach 4: PickleJobInfoCallback +
pickle.load(open("config.pickle", "rb")).This approach avoids using
@hydra.mainandcompose.addisonklinke commentedon Nov 8, 2022
Thanks for clarifying this subtle difference - that makes a lot of sense
I'll have to think more about whether this is desirable for my ML experiments. Typically, I would plan to checkout the git commit of the original run before reproducing, in which case there couldn't be any changes to the application code
I agree it could definitely be convenient to officially support that pattern. Worth opening a new issue?
Jasha10 commentedon Nov 8, 2022
Yes, that would be great, thanks!
npuichigo commentedon Sep 6, 2023
@Jasha10 For Approach 2, how can we pass the overrides path from command line since we may want the overrides dynamic
asimukaye commentedon Oct 13, 2023
@npuichigo I haven't tried it but this: #1022 seems relevant to your question and might help.
bkmi commentedon Jan 2, 2024
the
compose+overrideslooks ok for my purposes, but is there any way to include the${hydra:...}resolves in it as well at the current time?