Support pyarrow MonthDayNano and Arrow Interval(DayTime) intervals for DuckDB interoperability #15969
Description
Description
I have run into some problems trying to convert data with interval columns from DuckDB to Polars.
Direct conversion
This fails:
import duckdb
duckdb.sql('CREATE TABLE test (duration INTERVAL);')
duckdb.sql('INSERT INTO test VALUES (INTERVAL 1 hour);')
duckdb.sql('SELECT * FROM test;').pl()with the slightly cryptic polars.exceptions.ComputeError: The datatype "tin" is still not supported in Rust implementation.
I think the problem is that DuckDB uses pyarrow.month_day_nano_interval for representing intervals.
This fails with the same error message:
import polars as pl
import pyarrow as pa
pl.DataFrame(pa.table({'duration': [pa.scalar((1, 15, -30), type=pa.month_day_nano_interval())]}))While using pyarrow directly yields:
>>> duckdb.sql('SELECT * FROM test;').arrow()
pyarrow.Table
duration: month_day_nano_interval
----
duration: [[0M0d3600000000000ns]]Parquet compatibility
Additionally, trying to read a parquet file written from DuckDB also fails:
>>> duckdb.sql('COPY test TO test.parquet;')
>>> pl.read_parquet('test.parquet')
thread '<unnamed>' panicked at crates\polars-core\src\datatypes\field.rs:189:19:
...
pyo3_runtime.PanicException: Arrow datatype Interval(DayTime) not supported by Polars. You probably need to activate that data-type feature.Current workaround
Detouring via pandas works:
>>> pl.DataFrame(duckdb.sql('SELECT * FROM test;').df())
shape: (1, 1)
┌──────────────┐
│ duration │
│ --- │
│ duration[ns] │
╞══════════════╡
│ 1h │
└──────────────┘Possible solution
Support month_day_nano_interval and DayTimeInterval types?