Is possible to restrict number of threads?

[1991jhf]

Is possible to restrict number of threads for groupby->transform/combine?
I have some function i want to use is not thread safe however I do not want to run entire julia session on single thread.

[bkamins]

Currently not. However, we probably should add this feature.

CC @nalimilan

[nalimilan]

We had discussed this when we added multithreading support. There are two options:

• add a keyword arguments to all functions that support multithreading
• have a global variable setting the number of threads for all DataFrames operations

We could also have both.

Something which might be useful is to allow not only to choose between single-threading and multi-threading, but also to set the number of threads, as sometimes using all threads isn't faster. This is sometimes easy to implement, but sometimes more difficult. It doesn't have to be supported immediately but it's important to keep in mind when choosing the name of the keyword argument (so that it can take either a Boolean or an integer).

[bkamins]

@nalimilan - I understand that the plan would be to:

• ntasks argument in select, combine, etc. (name selected for consistency with CSV.jl); by default the argument would take the Threads.nthreads() value, but it could be also changed to ntasks=1 to avoid using multi-threading; these two values for now would be the only supported values;
• We then would implement Support multithreading in groupreduce #2491 also.

[nalimilan]

Maybe we should use the same strategy as Arrow.jl:

• ntasks::Int: number of concurrent threaded tasks to allow while writing input partitions out as arrow record batches; default is no limit; to disable multithreaded writing, pass ntasks=1

https://github.com/apache/arrow-julia/blob/52bfe1f72fa72b0b2d150efeeeb9edacd853f96f/src/write.jl#L49

The default is Inf (could also be typemax(Int)). This better reflects what we do as we spawn as many tasks as we can, independent of the number of threads.

[bkamins]

OK - but we should add a comment that currently only 1 or "unlimited" is supported.

[bkamins]

Following #3019 the key decision to make is if we:

• want to be able to disable multi-threading by a global flag
• or, we want to be able to disable it only in split-apply-combine functions, but we are fine with it in joins and indexing

[haberdashPI]

For my use cases I primarily need it in the latter case (split-apply-combine) where a user function can be called that might require some debugging (e.g. via Infiltrator). I think a global flag is also very nice, but less important to me.

[bkamins]

After several discussions I am now leaning towards a global switch rather than argument passed to functions. In particular it will be then easier to handle with DTable.

What @jpsamaroo suggested is that we could consider using https://github.com/tkf/ContextVariablesX.jl instead of a global switch.

@tkf the question is if you consider ContextVariablesX.jl stable/mature enough to be used (and what would be a recommended pattern here). What we need is a switch that will enable/disable using multi threading in DataFrames.jl operations.

CC @krynju

[tkf]

if you consider ContextVariablesX.jl stable/mature enough

ContextVariablesX.jl is a hack and "X" here means "eXperimental" (not, e.g., "eXtended"). It uses logger as a "payload." It means that with_logger can reset the context variable.

julia> @contextvar CTX = 0
Main.CTX :: ContextVar [3eaa8739-b516-401d-aa2a-f5dd85d0f89c] => 0

julia> with_context(CTX => 1) do
           @show CTX[]
           with_logger(NullLogger()) do
               @show CTX[]
           end
           @show CTX[]
       end;
CTX[] = 1
CTX[] = 0
CTX[] = 1

That said, I don't think people call with_logger frequently and so this mechanism may be OK for your use case. But we need something like JuliaLang/julia#35833 for "true" dynamically scoped configuration.

Maybe we should use the same strategy as Arrow.jl:

• ntasks::Int: number of concurrent threaded tasks to allow while writing input partitions out as arrow record batches; default is no limit; to disable multithreaded writing, pass ntasks=1

FWIW, I don't think this pattern is great since it does not generalize over the amount of data to be processed. I think a much more composable pattern is to specify the size of per-task problem size (what is called basesize in JuliaFolds). Note that it automatically disables parallel execution for a sufficiently small problems.

[nalimilan]

After several discussions I am now leaning towards a global switch rather than argument passed to functions. In particular it will be then easier to handle with DTable.

@bkamins What is needed exactly with DTable?

A global variable doesn't seem like the best API if it's intended to be used when passing a function that isn't thread safe (you'd have to set it before and unset it after an operation). Though we could easily have both a global variable, and use its value as the default for the argument that relevant functions would take.

FWIW, I don't think this pattern is great since it does not generalize over the amount of data to be processed. I think a much more composable pattern is to specify the size of per-task problem size (what is called basesize in JuliaFolds). Note that it automatically disables parallel execution for a sufficiently small problems.

@tkf Actually we already have this kind of mechanism for some operations internally. But the logic to choose the number of tasks is relatively complex so I'm not sure we can easily expose this to users. For example, for grouped operations, we spawn one task per operation (this is cheap so it's probably always worth it), and then one task per thread (as having one task per group would have a large overhead). We could allow specifying the number of groups per task, but the total number of rows can also be a relevant criterion depending on operations.

For now, the idea is mainly to have a way to disable multithreading as that's the most common need.