Feature Request: Remove PyArrow requirement for Polars import/export

[henryharbeck]

Now that support for the Arrow PyCapsule Interface has been implemented in duckdb/duckdb#10716, it's possible to exchange Arrow data with Polars without PyArrow installed. However, this doesn't work in practice because DuckDB checks if the input is a Polars object before checking for existence of the Arrow PyCapsule Interface. So reading Polars data works without pyarrow only if you hide the fact that it's a polars.DataFrame.

Feature request lifted from discussion here: duckdb/duckdb#13827

[paultiq]

Yah, polars_io.py relies on fetch_arrow_stream in all paths, but, it looks like the method in 10716 still works which is neat:

import duckdb
import polars as pl

class ArrowStream:
    def __init__(self, obj):
        self.obj = obj

    def __arrow_c_stream__(self, requested_schema=None):
        return self.obj.__arrow_c_stream__(requested_schema=requested_schema)


df = pl.DataFrame({"a": [1, 2, 3, 4]})
stream = ArrowStream(df)
con = duckdb.connect()
# Reading from the wrapper
sql = "SELECT * FROM stream"
con.query(sql)

Perhaps that could be used as a fallback for when pyarrow isn't installed?

[henryharbeck]

It is good it works, but the above workaround is not really end-user facing or very ergonomic

Perhaps that could be used as a fallback for when pyarrow isn't installed?

There is no need to go via PyArrow, even when it is installed.
Not requiring PyArrow is one the the goals of the PyCapsule interface ("...instead of hardcoding support for specific Arrow producers." - i.e., PyArrow)

[paultiq]

For sure, I meant this as a workaround.

That said, I'm curious what happens if you pass a relation to a relation in this manner.

[paultiq]

I tested reverting the change in #13798, which was to fix duckdb/duckdb#13793:

On the plus side, using the pycapsule stream was 30% faster and slightly less (9%) memory.

On the negative side, the same error* occurs from 13793 when the stream is consumed twice: _duckdb.InternalException: INTERNAL Error: ArrowArrayStream was released by another thread/library

This error is the same as #105.

My test case was:

    df = duckdb.sql("SELECT range AS id, 'value_' || range::VARCHAR AS name FROM range(10000000)").pl()
    result = duckdb.sql("SELECT COUNT(*) as cnt FROM df").fetchone()[0]

---- Edit
See #70 . Fixing this would fix the original issue (13793), which would allow 13798 to be reverted.

[paultiq]

Oh, and for the reverse, pyrelation.cpp would need a small change to construct the Polars dataframe directly, instead of going through auto arrow = ToArrowTableInternal(batch_size, true);

PolarsDataFrame DuckDBPyRelation::ToPolars(idx_t batch_size, bool lazy) {
	if (!lazy) {
		auto polars_module = pybind11::module_::import("polars");
		return py::cast<PolarsDataFrame>(polars_module.attr("DataFrame")(*this));
	}

This would allow the following without pyarrow installed:

import duckdb
duckdb.execute("select * from range(1)").pl();

Alternatively

Just create the dataframe in Polars from the relation, no pyarrow needed.

import duckdb
import polars as pl
rel = duckdb.sql("select * from range(10)")
df = pl.DataFrame(rel)

* And a similar change to the lazy path, and probably polars_io.py

** happy to open a PR

[paultiq]

I wonder if this would be a better solution to duckdb/duckdb#19356.... Oh wow, it is!

With this change, peak memory in the 19356 test case is cut from 3.5x to 2.5x. Which makes sense, since we're going direct from relation -> polars, instead of relation -> pyarrow ->rechunking -> polars.

edit: Altho, it takes twice as long... tanstaafl.- nm

See #137