feat: get the delta table row count based on the table history

[ohadmata]

Description

Currently, delta-rs does not have an option to get the number of rows for a specific table.
I want to get the number of rows from the delta log without reading the whole table to the RAM of my application.
The idea is to handle a row counter and iterate over the log in order to get the number.

Related Issue(s)

- closes #3731

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[ion-elgreco]

Thanks for your PR @ohadmata, unfortunately this wouldn't be a stable interface to get the row count since operations and metrics are optional and not agreed upon in delta spec.

You can however sum all the numRecords of each file add action.

[rtyler]

@ion-elgreco 🤷 this seems like a kind of useful thing to have right? If I put an interface for counting file stats down in the 🦀 layer, would that be cool to expose upwards?

[ion-elgreco]

@ion-elgreco 🤷 this seems like a kind of useful thing to have right? If I put an interface for counting file stats down in the 🦀 layer, would that be cool to expose upwards?

Definitely! As long as its done based on the file stats

[ohadmata]

@ion-elgreco @rtyler I have change the function to get the count from the parquet files metadata. WDYT?

[ion-elgreco]

@ion-elgreco @rtyler I have change the function to get the count from the parquet files metadata. WDYT?

This won't work on a remote object storage. The most efficient and correct way is to do it base on the delta log stats field "numRecords"

[ohadmata]

@ion-elgreco I updated it again to retrieve the number of records from the add action in the Delta log. This approach is much more efficient and avoids opening all the Parquet files.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.08%. Comparing base (9b35849) to head (4ba2865).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3732      +/-   ##
==========================================
- Coverage   76.09%   76.08%   -0.01%     
==========================================
  Files         145      145              
  Lines       45117    45117              
  Branches    45117    45117              
==========================================
- Hits        34330    34328       -2     
- Misses       9098     9100       +2     
  Partials     1689     1689              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ion-elgreco]

@ohadmata can you add a small test? Then we are good to go!

[rtyler]

@ion-elgreco I have added a regression test and some improved docstrings, I think this is ready to go!

[ohadmata]

Hi, I just got back from vacation and saw that you added a test and merged it. Thanks!