Recursion bug - Excessive number of processes spawned #5305
Answered
by
bentsherman
JoshuaEPSamuel
asked this question in
Q&A
-
|
I am facing an odd issue where Nextflow is running the first process up to hundreds/thousands of times simultaneously. This has only started occurring when I began experimenting with recursion. This bug occurs whether I am running recursion using .times() - e.g. .times(4) spawns PROC1 four times This bug also occurs when I am running recursion using .until() - where my condition expects a minimum number of files in a shared folder, it will spawn hundreds/thousands of PROC1 jobs simultaneously until the GPU crashes the workflow script. Here is roughly what my workflow looks like |
Beta Was this translation helpful? Give feedback.
Answered by
bentsherman
Sep 17, 2024
Replies: 3 comments 2 replies
-
|
Can you share the definition for |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Sure, here is the PROC1 looks like: I can also confirm that the inside the PROC1 When I look at the nextflow.log file, before the script crashes due to the GPU, it seems to be running the |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I think the problem is that the recursive worklfow is only emitting I would just find a way to include some output from PROC5 into the workflow output, even if that means the first invocation has some dummy value |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
I see you're including the results directory as one of the recursion params. Maybe the last process could forward that directory as an output, then emit that directory. Then Nextflow will see that the workflow emit depends on PROC5. Also, you can use that reference in the until closure instead of grabbing the directory anew with .until { params ->
// Get folder path
def folder = params[0] // whichever index it is
// ...
} |
Beta Was this translation helpful? Give feedback.
-
|
Solution here works thanks! Looks like it only required to update the workflow process to explicitly recycle a process output to solve the issue. I've added below an example of how you can re-write it: Updating the .until{} function may have tidied up the code, to reuse the shared_folder parameter instead of manually defining it, but alone did not stop the issue of spawning hundreds of PROC1. N.B please excuse myself referring to |
Beta Was this translation helpful? Give feedback.
Answer selected by
JoshuaEPSamuel
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I think the problem is that the recursive worklfow is only emitting
paramsand none of the process outputs. As a result, the feedback channel doesn't have to wait on the processes and can simply recurse at maximum speed. Looks like you found a dangerous pitfall of workflow recursion 😅I would just find a way to include some output from PROC5 into the workflow output, even if that means the first invocation has some dummy value