I noticed that the multinode case for AWS Batch (which I believe is what is called for @metaflow_ray) seems to rely on UBF_CONTROL , which makes me think that we can't use call a multinode batch step in a foreach
task (because then presumably we would be using UBF_TASK
). Is this correct?
I'm wondering because my use case has multiple job groups that I would like to submit jobs for, where each job group is a parallelizable set of subjobs. I wanted distinction between the individual job groups themselves so I thought combining the foreach
and the metaflow_ray
decorator would be useful