unexpected overload resolution with numpy arrays -- bug or working as expected?

[nurpax]

I recently hit a surprise in how function overload is selected. I crafted a small repro that can be dropped into the nanobind_example project:

#include <nanobind/nanobind.h>
#include <nanobind/ndarray.h>

namespace nb = nanobind;

using namespace nb::literals;

typedef nb::ndarray<const double, nb::ndim<1>, nb::device::cpu, nb::c_contig> ndarray_1d;

NB_MODULE(nanobind_example_ext, m) {
    m.doc() = "This is a \"hello world\" example with nanobind";
    m.def("add", [](int a, int b) { return a + b; }, "a"_a, "b"_a);

    m.def("plot_scatter", [](const char* label_id, ndarray_1d& values, double xscale) {
        printf("plot_scatter(label_id, values, xscale) called\n");
    }, "label_id"_a, "values"_a, "xscale"_a = 1.0);
    m.def("plot_scatter", [](const char* label_id, ndarray_1d& xs, ndarray_1d& ys) {
        printf("plot_scatter(label_id, xs, ys) called\n");
    }, "label_id"_a, "xs"_a, "ys"_a);
}

In the below example, the second overload is chosen. This is what I'd expect to happen as both inputs are ndarrays.

>>> import numpy as np
>>> import nanobind_example
>>>
>>> a = np.array([1.0])
>>> b = np.array([1.0])
>>> nanobind_example.plot_scatter("foo", a, b)
plot_scatter(label_id, xs, ys) called

However, if I any other element type than the default np.float64, say, np.float32 or np.uint8, looks like the second argument is implicitly converted to a scalar float, and the first overload is chosen instead:

>>> c = np.array([1.0], dtype=np.float32)
>>> nanobind_example.plot_scatter("foo", a, c)
<stdin>:1: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
plot_scatter(label_id, values, xscale) called

It works again if I re-order the overloads in the C++ binding code to put the label_id, xs, ys variant at the top.

Numpy also issues a deprecation warning about, as can be seen above.

Is this a bug? Or can I turn off 1-dim arrays from being automatically converted to scalars? Or is the right fix to reorder the overloads in the binding implementation? I don't know the overload resolution algorithm well enough to know what workaround to apply.

[hpkfft]

I think this is expected.
Reording the overloads may be the right thing for you; it depends on exactly what combination of inputs you want to accept for each function.
You can turn off conversions for a function argument with noconvert. See ndarray overload resolution. You can also use it on the xscale parameter. If you do, be aware that the integer 3, for example, will then not convert to the double 3.0.

[nurpax]

I don't think I will want noconvert though as I think that means even includes like arrays of ints or np.float32s won't be automatically converted to np.float64 which is what my bindings code wants.

In this API I'd definitely prefer arrays like np.array([1.0], dtype=np.float32) to not get implicitly cast to scalars by NB. Not only will this choose the wrong overload, but it will also issue the deprecation warning from numpy like is shown in the above Python example.

Anyway, thanks for the links! This is good information. I had not read all the separate passages. I think reordering the overloads will work for my code.

[hpkfft]

Nanobind is not doing the conversion; NumPy is.
I agree it should not. The NumPy developers must agree as well since they deprecated it.
Maybe they've started throwing TypeError in a newer release?
If not, you can suggest that it might now be better to start doing so.

If NumPy did not do the conversion, nanobind would move on to the next overload.

[nurpax]

Thanks for the clarification! I think I get it now. I'll summarize what I think happens:

1. First overload check:

   • "foo" → const char* (perfect match)
   • a (float64 array) → ndarray_1d& (perfect match)
   • c (float32 array) → double (fails, needs conversion)

2. Second overload check:

   • "foo" → const char* (perfect match)
   • a (float64 array) → ndarray_1d& (perfect match)
   • c (float32 array) → ndarray_1d& (fails, float32 can't convert to double ndarray)

3. Back to first overload with conversions enabled:

   • "foo" → const char* (perfect match)
   • a (float64 array) → ndarray_1d& (perfect match)
   • c (float32 array) → double (np.array converted to scalar)

In step 3, something like below happens, except coded in C. Since despite the deprecation warning the conversion succeeds, the overload is selected.

>>> import numpy as np
>>> arr = np.array([1.0], dtype=np.float32)
>>> float(arr)
<stdin>:1: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
1.0

[hpkfft]

Yes.
BTW, the C code is https://docs.python.org/3/c-api/float.html#c.PyFloat_AsDouble
Nanobind does not need to understand NumPy's types, and it does not.
It just calls this C function, which asks the NumPy object to convert itself into a double if it is willing to do so.

[hpkfft]

See also: implicit conversions and overload resolution order

[hpkfft]

I suggested NumPy to raise TypeError: numpy/numpy#29835