feat: datafusion table provider next

[roeap]

Description

• stacked-on: feat: allow for lazy loading files in operations #3872

This PR adds a new table provider, in the hopes of applying all the learnings we had and leveraging modern datafusion APIs. There are several aspects we need to consider.

Implementations

Thus far we implement TableProvider for DeltaTable and a dedicated DeltaTableProvider. Specifically the implementation for DeltaTable is problematic, since we do not (or at least may not) know important information (i.e. schema) about the table.

For log replay we implement ScanFileStream which consumes the kernel ScanMetadata stream and processes it to collect file skipping stats and extract datafusion Statistics to include in parquet execution planning.

Statistics & file skipping

Both delta-kernel and datafusion's parquet handling allow optimising queries via predicates. We pass the predicate into the kernel scan to leverage kernels file skipping. We also add statistics to the PartitionedFiles the get passed into the parquet plan to allow datrafusion to do its thing.

However we no longer expose statistics on the TableProvider since this would always require a full log replay prior to constructing the TableProvider, which we do want to move away from. ListingTable in datafusion - which is likely most similar to our provider - takes a similar approach.

Execution metrics

Thus far we collect operation statistics in several ways, including the custom MetricsObserver node. While we likely need to retain this functionality, there are several stats we can collect more efficiently. Specifically we track files skipped and scanned when we do the log replay to plan the scan.

Future work

push deletion vectors into parquet read

Currently we process deletion vectors after loading the data from the parquet file. This is due to uncertainties in handling row ids and other features that might be affected by skipping individual rows.

[codecov]

Codecov Report

❌ Patch coverage is 80.75314% with 322 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.26%. Comparing base (a22a97e) to head (7130d3c).

Files with missing lines	Patch %	Lines
...c/delta_datafusion/engine/expressions/to_kernel.rs	68.85%	92 Missing and 17 partials ⚠️
...re/src/delta_datafusion/table_provider/next/mod.rs	72.03%	51 Missing and 22 partials ⚠️
...e/src/delta_datafusion/table_provider/next/scan.rs	65.31%	53 Missing and 7 partials ⚠️
...src/delta_datafusion/table_provider/next/replay.rs	80.20%	32 Missing and 7 partials ⚠️
...e/src/delta_datafusion/engine/expressions/to_df.rs	94.39%	22 Missing and 14 partials ⚠️
crates/core/src/operations/load.rs	76.47%	0 Missing and 4 partials ⚠️
...tes/core/src/kernel/snapshot/iterators/scan_row.rs	91.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3849      +/-   ##
==========================================
+ Coverage   73.76%   74.26%   +0.50%     
==========================================
  Files         151      156       +5     
  Lines       39396    41033    +1637     
  Branches    39396    41033    +1637     
==========================================
+ Hits        29061    30474    +1413     
- Misses       9023     9175     +152     
- Partials     1312     1384      +72     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.