Skip to content

Conversation

nicklan
Copy link
Collaborator

@nicklan nicklan commented Sep 11, 2025

What changes are proposed in this pull request?

Currently there's no way for a connector using ffi to know if a &DvInfo actually has a vector in it. We have has_vector() on the rust side, but this isn't exposed via ffi. So this just wraps the &DvInfo in another struct which includes a boolean that says if there is a dv to consider or not.

This allows engines to ignore dv info if there isn't any without needing to make another ffi call at all.

This PR affects the following public APIs

The CScanCallback now takes a &CDvInfo and not a &DvInfo. This will be a compilation breaker so no risk that engines will silently fail to update to support this change.

How was this change tested?

Added a DV test to make sure this works end to end with read_table

Copy link

codecov bot commented Sep 11, 2025

Codecov Report

❌ Patch coverage is 0% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.97%. Comparing base (a7889af) to head (07f1d4b).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
ffi/src/scan.rs 0.00% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1286      +/-   ##
==========================================
- Coverage   83.98%   83.97%   -0.02%     
==========================================
  Files         111      111              
  Lines       26246    26250       +4     
  Branches    26246    26250       +4     
==========================================
  Hits        22044    22044              
- Misses       3105     3109       +4     
  Partials     1097     1097              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Member

@zachschuermann zachschuermann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

possibly naive question: is another alternative to just make DvInfo repr(C) itself? and then could we just keep the bool in there directly? though I suppose that would be some FFI-specific code bleeding in to core kernel so perhaps this is actually better..

Copy link
Member

@zachschuermann zachschuermann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM one quick nit (and I think I answered myself in last comment already)

size: i64,
stats: Option<&Stats>,
dv_info: &DvInfo,
dv_info: &CDvInfo,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're okay with using stats: Option<&Stats>, why not just do dv_info: Option<&DvInfo>?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great question! Because that would go from a pointer that could not be null to one that could be null, and I felt like that was a more dangerous change that wasn't worth it.

I'd actually consider also not having the stats be allowed to be null in the future too :)

@nicklan nicklan added the breaking-change Change that require a major version bump label Sep 17, 2025
@nicklan
Copy link
Collaborator Author

nicklan commented Sep 17, 2025

possibly naive question: is another alternative to just make DvInfo repr(C) itself? and then could we just keep the bool in there directly? though I suppose that would be some FFI-specific code bleeding in to core kernel so perhaps this is actually better..

To answer this, yeah, I'd prefer to keep all the C stuff in the ffi crate

@nicklan nicklan merged commit a5705c5 into delta-io:main Sep 17, 2025
19 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change Change that require a major version bump
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants