Change storage precision and data size for parameters to 32bit #11830
Conversation
xjules
added
the
release-notes:skip
CodSpeed Performance ReportMerging #11830 will not alter performanceComparing Summary
|
There was a problem hiding this comment.
Seem sensible.
But, do we have a general "philosophy" on float precision? For example the parameters stuff here:
ert/src/ert/config/gen_kw_config.py
Line 320 in 26203ad
And tests are failing :-/
When it is stored as 32bit the schema should say 32 bit. The tests are failing due to the snapshots. |
xjules
force-pushed
the
data_prec_32bit
branch
from
db311ce to
b1274c5
Compare
xjules
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #11830 +/- ##
==========================================
- Coverage 90.74% 90.61% -0.13%
==========================================
Files 432 432
Lines 28985 28985
==========================================
- Hits 26302 26265 -37
- Misses 2683 2720 +37
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
xjules
force-pushed
the
data_prec_32bit
branch
from
020961a to
7d3381c
Compare
xjules
force-pushed
the
data_prec_32bit
branch
from
5d7bf12 to
f09754c
Compare
- Give a precision interval in test - update snapshots
|
|
||
| def cluster_responses( | ||
| responses: npt.NDArray[np.float64], | ||
| responses: npt.NDArray[np.float32], |
There was a problem hiding this comment.
should this be np.floating instead? it seems responses can be np.float64 coming from main?
There was a problem hiding this comment.
yes, probably it should be np.floating I have changed it in some places 😄
xjules
force-pushed
the
data_prec_32bit
branch
from
afa2661 to
91335c1
Compare
|
|
||
| snapshot.assert_match( | ||
| result.to_csv(float_format="%.12g"), | ||
| result.to_csv(float_format="%.7g"), |
There was a problem hiding this comment.
I need to revisit the precision.
4 participants
Update step requires 32bit so there is no need to store 64bit dataframes
Approach
(Screenshot of new behavior in GUI if applicable)
change the snapshots
PR title captures the intent of the changes, and is fitting for release notes.
Added appropriate release note label
Commit history is consistent and clean, in line with the contribution guidelines.
Make sure unit tests pass locally after every commit (
git rebase -i main --exec 'just rapid-tests')When applicable