[eval][1122] Plot scores on a map

[iluise]

Description

To better investigate spatial patterns in the scores and the ensemble distributions, this PR adds the possibility to plot the scores (all of them, not only RMSE and spread) on a map.
The code recomputes the scores averaging over the sample (initialization time) dimension and produces a plot for each forecast step, ensemble member and metric .

Examples:

Issue Number

Closes #1122

Is this PR a draft? Mark it as draft.

Checklist before asking for review

• I have performed a self-review of my code
• My changes comply with basic sanity checks:
  • I have fixed formatting issues with ./scripts/actions.sh lint
  • I have run unit tests with ./scripts/actions.sh unit-test
  • I have documented my code and I have updated the docstrings.
  • I have added unit tests, if relevant
• I have tried my changes with data and code:
  • I have run the integration tests with ./scripts/actions.sh integration-test
  • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
  • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
• I have informed and aligned with people impacted by my change:
  • for config changes: the MatterMost channels and/or a design doc
  • for changes of dependencies: the MatterMost software development channel

[SavvasMel]

How can I replicate this? What should I add to the config?

[SavvasMel]

I am also a bit confused, since you average over all samples how you plot for each ensemble member? Do I understand wrongly what an ensemble is?

[iluise]

How can I replicate this? What should I add to the config?

evaluation:
  metrics  : ["rmse", "mae"]
  regions: ["global", "nhem"]
  summary_plots : true
  summary_dir: "./plots/"
  plot_ensemble: "members" #supported: false, "std", "minmax", "members"
  plot_score_maps: true <----
  print_summary: false #print out score values on screen. it can be verbose
  log_scale: false
  add_grid: false

[iluise]

I am also a bit confused, since you average over all samples how you plot for each ensemble member? Do I understand wrongly what an ensemble is?

There's a loop over ensemble members, so each ensemble member has its own plot:
https://github.com/ecmwf/WeatherGenerator/pull/1176/files#diff-fb043e50e73406a916b50c2ebd2247b619bcdeb851d3e23dc0bf71a522498c89R243

[SavvasMel]

It is not working for me if I have something like:

global_plotting_options:
  # image_format : "png" #options: "png", "pdf", "svg", "eps", "jpg" ..
  # dpi_val : 300
  ERA5:
    marker_size: 2
    scale_marker_size: 1
    marker: "o"
    2t: 
      vmin: 
      vmax: 

I get:

  File "/p/project1/hclimrep/melidonis1/WG_Fork/WeatherGenerator/.venv/bin/evaluate", line 10, in <module>
    sys.exit(evaluate())
             ^^^^^^^^^^
  File "/p/project1/hclimrep/melidonis1/WG_Fork/WeatherGenerator/packages/evaluate/src/weathergen/evaluate/run_evaluation.py", line 36, in evaluate
    evaluate_from_args(sys.argv[1:])
  File "/p/project1/hclimrep/melidonis1/WG_Fork/WeatherGenerator/packages/evaluate/src/weathergen/evaluate/run_evaluation.py", line 56, in evaluate_from_args
    evaluate_from_config(OmegaConf.load(config))
  File "/p/project1/hclimrep/melidonis1/WG_Fork/WeatherGenerator/packages/evaluate/src/weathergen/evaluate/run_evaluation.py", line 101, in evaluate_from_config
    _ = plot_data(reader, stream, global_plotting_opts)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/p/project1/hclimrep/melidonis1/WG_Fork/WeatherGenerator/packages/evaluate/src/weathergen/evaluate/utils.py", line 338, in plot_data
    maps_config = common_ranges(
                  ^^^^^^^^^^^^^^
  File "/p/project1/hclimrep/melidonis1/WG_Fork/WeatherGenerator/packages/evaluate/src/weathergen/evaluate/utils.py", line 571, in common_ranges
    maps_config[var].update({"vmin": float(min(list_min))})
                                           ^^^^^^^^^^^^^
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()