@@ -8,22 +8,22 @@ status](https://www.r-pkg.org/badges/version/mlr3batchmark)](https://CRAN.R-proj
8
8
[ ![ Mattermost] ( https://img.shields.io/badge/chat-mattermost-orange.svg )] ( https://lmmisld-lmu-stats-slds.srv.mwn.de/mlr_invite/ )
9
9
10
10
A connector between [ mlr3] ( https://github.com/mlr-org/mlr3 ) and
11
- [ batchtools] ( https ://mllg.github.io/batchtools /) . This allows to run
11
+ [ batchtools] ( http ://batchtools.mlr-org.com /) . This allows to run
12
12
large-scale benchmark experiments on scheduled high-performance
13
13
computing clusters.
14
14
15
15
The package comes with two core functions for switching between ` mlr3 `
16
16
and ` batchtools ` to perform a benchmark:
17
17
18
- - After creating a ` design ` object (as required for ` mlr3 ` ’s
19
- ` benchmark() ` function), instead of ` benchmark() ` call ` batchmark() `
20
- which populates an ` ExperimentRegistry ` for the computational jobs
21
- of the benchmark. You are now in the world of ` batchtools ` where you
22
- can selectively submit jobs with different resources, monitor the
23
- progress or resubmit as needed.
24
- - After the computations are finished, collect the results with
25
- ` reduceResultsBatchmark() ` to return to ` mlr3 ` . The resulting object
26
- is a regular ` BenchmarkResult ` .
18
+ - After creating a ` design ` object (as required for ` mlr3 ` ’s
19
+ ` benchmark() ` function), instead of ` benchmark() ` call ` batchmark() `
20
+ which populates an ` ExperimentRegistry ` for the computational jobs of
21
+ the benchmark. You are now in the world of ` batchtools ` where you can
22
+ selectively submit jobs with different resources, monitor the progress
23
+ or resubmit as needed.
24
+ - After the computations are finished, collect the results with
25
+ ` reduceResultsBatchmark() ` to return to ` mlr3 ` . The resulting object
26
+ is a regular ` BenchmarkResult ` .
27
27
28
28
## Example
29
29
@@ -46,61 +46,86 @@ reg = makeExperimentRegistry(NA)
46
46
47
47
## No readable configuration file found
48
48
49
- ## Created registry in '/tmp/Rtmp8DlMZQ/registry704553adf7a88 ' using cluster functions 'Interactive'
49
+ ## Created registry in '/tmp/RtmpbcuMc4/registry27b8961304f5da ' using cluster functions 'Interactive'
50
50
51
51
``` r
52
52
ids = batchmark(design , reg = reg )
53
53
```
54
54
55
55
## Adding algorithm 'run_learner'
56
56
57
- ## Adding problem 'b39ef23a66b1f1ee '
57
+ ## Adding problem 'abc694dd29a7a8ce '
58
58
59
- ## Exporting new objects: '5ec484de3f93431b ' ...
59
+ ## Exporting new objects: '2da7eeb80b94fc3b ' ...
60
60
61
- ## Exporting new objects: '7c35d835f3dfae37 ' ...
61
+ ## Exporting new objects: 'c905990877a775af ' ...
62
62
63
- ## Exporting new objects: '70dd22724e5c724d ' ...
63
+ ## Exporting new objects: '3acc41a799a260d8 ' ...
64
64
65
- ## Adding 6 experiments ('b39ef23a66b1f1ee'[1] x 'run_learner'[2] x repls[3]) ...
65
+ ## Exporting new objects: 'ecf8ee265ec56766' ...
66
66
67
- ## Adding problem '76c4fc7a533d41b7 '
67
+ ## Overwriting previously exported object: 'ecf8ee265ec56766 '
68
68
69
- ## Exporting new objects: 'b209de197d6cbe75' ...
69
+ ## Adding 6 experiments ('abc694dd29a7a8ce'[1] x 'run_learner'[2] x repls[3]) ...
70
70
71
- ## Adding 6 experiments ('76c4fc7a533d41b7'[1] x 'run_learner'[2] x repls[3]) ...
71
+ ## Adding problem 'f9791e97f9813150'
72
+
73
+ ## Exporting new objects: '62ac3bb85aabfbaf' ...
74
+
75
+ ## Adding 6 experiments ('f9791e97f9813150'[1] x 'run_learner'[2] x repls[3]) ...
72
76
73
77
``` r
74
78
submitJobs()
75
79
```
76
80
77
81
## Submitting 12 jobs in 12 chunks using cluster functions 'Interactive' ...
78
82
83
+ ## Error in workhorse(iteration = job$repl, task = data, learner = learner, :
84
+ ## unused argument (lgr_threshold = lgr::get_logger("mlr3")$threshold)
85
+ ## Error in workhorse(iteration = job$repl, task = data, learner = learner, :
86
+ ## unused argument (lgr_threshold = lgr::get_logger("mlr3")$threshold)
87
+ ## Error in workhorse(iteration = job$repl, task = data, learner = learner, :
88
+ ## unused argument (lgr_threshold = lgr::get_logger("mlr3")$threshold)
89
+ ## Error in workhorse(iteration = job$repl, task = data, learner = learner, :
90
+ ## unused argument (lgr_threshold = lgr::get_logger("mlr3")$threshold)
91
+ ## Error in workhorse(iteration = job$repl, task = data, learner = learner, :
92
+ ## unused argument (lgr_threshold = lgr::get_logger("mlr3")$threshold)
93
+ ## Error in workhorse(iteration = job$repl, task = data, learner = learner, :
94
+ ## unused argument (lgr_threshold = lgr::get_logger("mlr3")$threshold)
95
+ ## Error in workhorse(iteration = job$repl, task = data, learner = learner, :
96
+ ## unused argument (lgr_threshold = lgr::get_logger("mlr3")$threshold)
97
+ ## Error in workhorse(iteration = job$repl, task = data, learner = learner, :
98
+ ## unused argument (lgr_threshold = lgr::get_logger("mlr3")$threshold)
99
+ ## Error in workhorse(iteration = job$repl, task = data, learner = learner, :
100
+ ## unused argument (lgr_threshold = lgr::get_logger("mlr3")$threshold)
101
+ ## Error in workhorse(iteration = job$repl, task = data, learner = learner, :
102
+ ## unused argument (lgr_threshold = lgr::get_logger("mlr3")$threshold)
103
+ ## Error in workhorse(iteration = job$repl, task = data, learner = learner, :
104
+ ## unused argument (lgr_threshold = lgr::get_logger("mlr3")$threshold)
105
+ ## Error in workhorse(iteration = job$repl, task = data, learner = learner, :
106
+ ## unused argument (lgr_threshold = lgr::get_logger("mlr3")$threshold)
107
+
79
108
``` r
80
109
getStatus()
81
110
```
82
111
83
- ## Status for 12 jobs at 2023-11-13 19:32:20 :
112
+ ## Status for 12 jobs at 2025-05-26 09:23:22 :
84
113
## Submitted : 12 (100.0%)
85
114
## -- Queued : 0 ( 0.0%)
86
115
## -- Started : 12 (100.0%)
87
116
## ---- Running : 0 ( 0.0%)
88
- ## ---- Done : 12 (100 .0%)
89
- ## ---- Error : 0 ( 0 .0%)
117
+ ## ---- Done : 0 ( 0 .0%)
118
+ ## ---- Error : 12 (100 .0%)
90
119
## ---- Expired : 0 ( 0.0%)
91
120
92
121
``` r
93
122
reduceResultsBatchmark()
94
123
```
95
124
96
- ## <BenchmarkResult> of 12 rows with 4 resampling runs
97
- ## nr task_id learner_id resampling_id iters warnings errors
98
- ## 1 iris classif.featureless cv 3 0 0
99
- ## 2 iris classif.rpart cv 3 0 0
100
- ## 3 sonar classif.featureless cv 3 0 0
101
- ## 4 sonar classif.rpart cv 3 0 0
125
+ ##
126
+ ## ── <BenchmarkResult> of 0 rows with 0 resampling run ───────────────────────────
102
127
103
128
## Resources
104
129
105
- - The * Large-Scale Benchmarking* chapter of the [ mlr3
106
- book] ( https://mlr3book.mlr-org.com/ )
130
+ - The * Large-Scale Benchmarking* chapter of the [ mlr3
131
+ book] ( https://mlr3book.mlr-org.com/ )
0 commit comments