Skip to content

feat: update to DataFusion 50, pyo3 24, pyo3-arrow 0.11 #3749

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 19, 2025
Merged

feat: update to DataFusion 50, pyo3 24, pyo3-arrow 0.11 #3749

merged 1 commit into from
Sep 19, 2025

Conversation

alamb
Copy link
Contributor

@alamb alamb commented Sep 10, 2025

Description

The description of the main changes of your pull request

Related Issue(s)

Documentation

As part of preparing for the DataFusion 50 release I am making a PR to upgrade delta.rs

@alamb alamb requested a review from ion-elgreco as a code owner September 10, 2025 21:11
@alamb alamb marked this pull request as draft September 10, 2025 21:11
@github-actions github-actions bot added the binding/python Issues for the Python package label Sep 10, 2025
@github-actions GitHub Actions
Copy link

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

@alamb alamb changed the title feat: Update to DataFusion 50, pyo3 24, pyo3-arrow 0.11 feat: update to DataFusion 50, pyo3 24, pyo3-arrow 0.11 Sep 10, 2025
@alamb alamb force-pushed the alamb/update_datafusion_50 branch from b1da222 to dab5841 Compare September 10, 2025 21:28
@github-actions github-actions bot added the binding/rust Issues for the Rust crate label Sep 10, 2025
@alamb alamb mentioned this pull request Sep 10, 2025
Closed
45 tasks
@codecov Codecov
Copy link

codecov bot commented Sep 10, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 33.33333% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.69%. Comparing base (0ca1a3b) to head (8de2a6a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
python/src/datafusion.rs 0.00% 3 Missing ⚠️
python/src/writer.rs 0.00% 3 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #3749      +/-   ##
==========================================
- Coverage   75.72%   75.69%   -0.03%     
==========================================
  Files         145      145              
  Lines       44713    44719       +6     
  Branches    44713    44719       +6     
==========================================
- Hits        33857    33852       -5     
- Misses       9183     9195      +12     
+ Partials     1673     1672       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@alamb alamb force-pushed the alamb/update_datafusion_50 branch from dab5841 to 2ec3e1c Compare September 10, 2025 21:35
alamb
alamb commented Sep 10, 2025
View reviewed changes
.github/workflows/build.yml
with:
profile: default
toolchain: "1.85"
toolchain: "1.86"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to https://datafusion.apache.org/library-user-guide/upgrading.html#msrv-updated-to-1-86-0

crates/core/src/delta_datafusion/expr.rs
/// This struct is like Datafusion's MakeArray but ensures that `element` is used rather than `item
/// as the field name within the list.
#[derive(Debug)]
#[derive(Debug, Hash, PartialEq, Eq)]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to https://datafusion.apache.org/library-user-guide/upgrading.html#scalarudfimpl-aggregateudfimpl-and-windowudfimpl-traits-now-require-partialeq-eq-and-hash-traits

python/src/datafusion.rs
}

impl LazyBatchGenerator for TestBatchGenerator {
fn as_any(&self) -> &dyn Any {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

due to https://datafusion.apache.org/library-user-guide/upgrading.html#add-as-any-method-to-lazybatchgenerator

@rtyler rtyler self-assigned this Sep 11, 2025
@abhiaagarwal
Copy link
Contributor

@alamb are you planning on rebasing this branch with DF50 out? I've been testing this internally and saw a moderate ram improvement in my workloads, I'd love to be able to point to this ref without needing to define multiple patches :)

@alamb
Copy link
Contributor Author

alamb commented Sep 19, 2025

@alamb are you planning on rebasing this branch with DF50 out? I've been testing this internally and saw a moderate ram improvement in my workloads, I'd love to be able to point to this ref without needing to define multiple patches :)

Done!

@alamb alamb force-pushed the alamb/update_datafusion_50 branch from 2ec3e1c to 8de2a6a Compare September 19, 2025 16:21
@alamb alamb marked this pull request as ready for review September 19, 2025 16:21
ion-elgreco
ion-elgreco approved these changes Sep 19, 2025
View reviewed changes
Copy link
Collaborator

@ion-elgreco ion-elgreco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @alamb :)

@ion-elgreco ion-elgreco merged commit 0864b21 into delta-io:main Sep 19, 2025
28 of 29 checks passed
@vegarsti vegarsti mentioned this pull request Sep 20, 2025
Closed
@alamb
Copy link
Contributor Author

alamb commented Sep 20, 2025

This was one of the easiest upgrades in a while. 🥳

@alamb alamb deleted the alamb/update_datafusion_50 branch September 20, 2025 10:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ion-elgreco ion-elgreco ion-elgreco approved these changes

@rtyler rtyler Awaiting requested review from rtyler rtyler is a code owner

@roeap roeap Awaiting requested review from roeap roeap is a code owner

@hntd187 hntd187 Awaiting requested review from hntd187 hntd187 is a code owner

Assignees

@rtyler rtyler

Labels
binding/python Issues for the Python package binding/rust Issues for the Rust crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@alamb @abhiaagarwal @ion-elgreco @rtyler