Add PyTorch Lightning experiment adapter (TorchExperiment) #201
Conversation
src/hyperactive/experiment/integrations/torch_lightning_experiment.py
Outdated
Show resolved
Hide resolved
| """ | ||
|
|
||
| model = self.lightning_module(**params) | ||
| self.trainer.fit(model, self.datamodule) |
There was a problem hiding this comment.
Is the trainer designed to be fitted multiple/many times like this?
There was a problem hiding this comment.
Ohh, I'll modify this to create a new trainer instance for every run!
| return params | ||
|
|
||
| @classmethod | ||
| def _get_score_params(self): |
| else: | ||
| val_results = self.trainer.callback_metrics.get(self.objective_metric, float('inf')) | ||
| metadata = {} | ||
| import numpy as np |
There was a problem hiding this comment.
put this to the top of the file
There was a problem hiding this comment.
Okay, I'll address all the issues!
| self.trainer.fit(model, self.datamodule) | ||
| # get validation results | ||
| if self.objective_metric not in self.trainer.callback_metrics: | ||
| raise ValueError(f"objective metric {self.objective_metric} not found in trainer callback metrics") |
There was a problem hiding this comment.
This is redundant. The get method already handles of objective_metric is not in objective_metrics.
| if self.objective_metric not in self.trainer.callback_metrics: | ||
| raise ValueError(f"objective metric {self.objective_metric} not found in trainer callback metrics") | ||
| else: | ||
| val_results = self.trainer.callback_metrics.get(self.objective_metric, float('inf')) |
There was a problem hiding this comment.
val_results -> val_result
It is one result, right?
There was a problem hiding this comment.
Deep-Learning frameworks often have very verbose logging behaviour per default. Since we run multiple trainings (in each iteration), this can get overwhelming in the command-line. Maybe we should address this.
Also answer the questions in the code and make necessary changes.
| >>> | ||
| >>> val_result, metadata = experiment._evaluate(params) | ||
| """ | ||
|
|
There was a problem hiding this comment.
Since the objective_metric is a loss the property:higher_or_lower_is_better in the _tags must be changed.
There was a problem hiding this comment.
Yeah, I was actually a bit confused about the usage of tags and the author credit. Can you help me with it.
There was a problem hiding this comment.
re author credit, that is ["amitsubhashchejara"], i.e., your GitHub account name
|
|
||
| Examples | ||
| -------- | ||
| >>> from hyperactive.experiment.integrations import TorchExperiment |
There was a problem hiding this comment.
This assumes, that "TorchExperiment" is in the init of the integrations module, which it isn't.
There was a problem hiding this comment.
Okay, I'll add TorchExperiment to the __init__ of integrations module.
| val_results = self.trainer.callback_metrics.get(self.objective_metric, float('inf')) | ||
| metadata = {} | ||
| import numpy as np | ||
| return np.float64(val_results.detach().cpu().item()), metadata |
There was a problem hiding this comment.
Is val_results always a tensor? If not this would fail. If there is a possibility, that val_results another type then we should account for that.
|
@amitsubhashchejara Thanks for opening this PR. There is still some work to do, but I am sure you can do this. |
|
Sure, I'll make the changes! |
There was a problem hiding this comment.
The tests fail since lightning is not installed, and the dependency is not explicit
- add
lightningto thepyprojectdepsetall_extras - add
"lightning"to the tagpython_dependenciesof the new experiment - make sure no
lightningortorchimports happen outside the class
|
I've pushed some changes per your suggestions. |
| "objective_metric": "val_loss", | ||
| } | ||
|
|
||
| return params |
There was a problem hiding this comment.
I think the issue might be with this - try returning a list of dict
3 participants
This pull request is to merge a new experiment
torch-lightning-experimentfor the issue195. The new classTorchExperimentis an experiment adapter for PyTorch Lightning experiments and is used to perform experiments using PyTorch Lightning modules. It allows for hyperparameter tuning and evaluation of the model's performance using specified metrics.The
TorchExperimentclass accepts aLightningModule,DataModule,Trainer, and anobjective_matricwith default valueval_loss. The_evaluatefunction internally performs a training run and returns a score of theobjective_matricas anumpy.float64.Fixes #195