Simplified obs.setup with metadata #10795
Description
Background:
The current ERT observation format has some limitations:
- it is not compatible with adding context/metadata on observations
- it is not user friendly to set up for standard observation types like RFT (which requires consistent configuration in 4-5 different files today)
- for "summary" observations the current format is verbose and not user friendly to set up
New observation format (draft):
In early 2024, team members from Atlas, Sudo, Scout, Webviz and (what today is) Targa, designed a draft for new format. At a high level these were the suggested design principles for the new format:
- The top level observation input file is
.yaml. - To avoid this
.yamlbecoming very large/verbose, for relevant observation types it refers to.csv/.xlsxcontaining observation points.- This can also be extended to seismic observation type, where it refers to e.g. binary files (since the seismic observation data itself could be massive), compared to today's requirement where ERT requires all observation types coming as text files. This might also improve FMU run performance, since the forward models can dump to binary files instead of text files.
YAML schem:
- label: Gas-oil-ratio observations
type: timeseries
error: 10% # optional (% indicates relative, otherwise absolute)
min_error: 20 # optional, always absolute
max_error: 30 # optional, always absolute
# Columns: vector, date, value, error, min_error, max_error
# (last three columns optional for each row, overrides global error for the rows where they are present)
observation: "./input/observations/summary_gor.csv"
# Optional. Columns: vector (wildcard supported), xpos, ypos, range1, range2, angle, zone (wildcard supported)
localization: "./input/observations/localization_config_timeseries.csv"
# The user can include the same observation type multiple times,
# with the benefit of easily giving different error to e.g. gas-oil-ratio observations
# compared with e.g. pressure observations, and also have different names useful in post-run mismatch
# analysis to group time series observations in different relevant "buckets" .
- name: Bottom hole pressure observations
type: timeseries
error: 5%
observation: "./input/observations/summary_bhp.csv"
# Example of another observation type, rft:
- label: RFT pressure observations
type: rft # we have a plugin that "knows about rft"
plugin_arguments:
zonemap: "./rms/output/zone/layer_zone_table.txt"
trajectories: "./rms/output/trajectories.csv"
error: 5
# Columns: well, date, measured_depth, zone, value, error, min_error, max_error (last three columns optional)
observation: "./input/observations/rft_pressure.csv"
metadata:
# should the metadata below have default values? TBD with `fmu-dataio`
columns:
measured_depth:
unit: m
reference: msl
value:
unit: bar
# Example of another observation type, rft, in this case saturation:
- label: RFT water saturation observations
type: rft # we have a plugin that "knows about rft"
plugin_arguments:
zonemap: "<RUNPATH>/rms/output/zone/layer_zone_table.txt"
trajectories: "./rms/output/trajectories.csv"
error: 5
# TBD: Should we support saturation values for multiple phases in the csv file below?
# Columns: well, date, measured_depth, zone, value, error, min_error, max_error (last three columns optional)
observation: "./input/observations/rft_saturation.csv"
metadata:
columns:
measured_depth:
unit: m
reference: msl
value:
unit: fraction
# Other observation types this can be extended to support:
- label: 4D gravity
type: gravity
...
- label: 4D seismic
type: seismic
...
- label: Breakthrough time
type: breakthrough-time
...For each observation type, ERT knows how to internally translate it down to:
- An array of values and errors (since this is what ERT needs to do AHM).
- A corresponding "simulated response extraction job".
- Today ERT automatically extracts simulated
.UNSMRYwhen a realization is finished, and implicitly know this should be compared with the list of (value, error) pairs from theSUMMARY_OBSERVATION. The proposal in the improved format is that this is extended to natively support also other observations.- In other words, the type
timeserieswould extract simulated responses similar to what ERT does today from.UNSMRY. - For e.g. the observation type
rft, ERT would extract simulated responses similar to what theFORWARD_MODEL GENDATA_RFTdoes today.
- In other words, the type
- Today ERT automatically extracts simulated
This could either be maintained centrally in ERT, or maybe as a plugin system (similar to FORWARD_MODELS today).