Add full cut optimization as introduced in pyirf 0.13 #2789
Conversation
This comment has been minimized.
This comment has been minimized.
|
How fine did you make the scanning of the cuts? In principle, the full cut opt should always be at least as good as the one that is restricted,.if it is allowed to find the same cuts. |
This comment has been minimized.
This comment has been minimized.
I kept everything at the default values, as we have them in here right now. So, if I'm not mistaken, the only difference would be that the EventDisplay-like optimization has a theta cut with 68% efficiency, while the full optimization only tries 60% and 70%. And, I think, there is no check for a minimum number of events per bin in the full cut optimization, while the 68% theta cut for the EventDisplay-like optimization has a minimum of 10 events per bin. However, this only seems to play a role for the two lowest energy bins, if at all. |
|
There was still an error with the application of the cuts in the irf tool. It should be correct now and this improved the sensitivity situation a lot, but there are still some bins where the EventDisplay-like optimization outperforms the full optimization, even though the first uses a theta cut with 70% efficiency for this plot, which also gets tested in the full optimization. |
Analysis Details1 Issue
Coverage and Duplications
Project ID: cta-observatory_ctapipe_AY52EYhuvuGcMFidNyUs |
|
If I remember correctly, these percentiles cannot be compared directly, as the eventdisplay-like optimization computes the percentile on the events surviving the initial cut, whereas the full optimization computes it on all events. |
Ah, good point, I forgot about that. I'll run the full optimization again with a finer gridding. I guess, these underperformance for some bins will disappear then. |
|
Running both optimizations with finer grids: PointSourceSensitivityOptimizer:
gh_cut_efficiency_step=0.02
theta_cut_efficiency_step=0.02
PointSourceSensitivityGhOptimizer:
gh_cut_efficiency_step=0.02gets the performance of the full optimization closer, but some bins are still worse then with EventDisplay-like optimization. |
There was a problem hiding this comment.
gh_cut_efficiency_step=0.02
theta_cut_efficiency_step=0.02
If a smaller step is important to get a good sensitivity, please also update the ctapipe-quickstart configurations to provide the best values for users, and also the default values in the tool itself.
I just noticed this (similar for the full optimization): The config is exactly the same (default values) besides the stepsize. This doesn't make sense. I'm re-running everything now and, if this behavior is still there, I'll convert this PR to draft until I figure out whats going on here. |
Is this an error introduced here? Or does it affect main? If it affects also main, could you open a PR just with the bugfix please? |
| class PointSourceSensitivityOptimizer(PointSourceSensitivityOptimizerBase): | ||
| """ | ||
| Finds the combination of G/H cut, theta cut, and cut on the event-multiplicity | ||
| for each energy bin with the maximum point source sensitivity in a grid search. |
There was a problem hiding this comment.
| for each energy bin with the maximum point source sensitivity in a grid search. | |
| for each energy bin with the best point source sensitivity in a grid search. |
sensitivity is lower = better
| theta_min_angle = AstroQuantity( | ||
| default_value=u.Quantity(-1, u.deg), | ||
| physical_type=u.physical.angle, | ||
| help="Smallest angular cut value allowed (-1 means no cut)", |
| ).tag(config=True) | ||
|
|
||
| theta_max_angle = AstroQuantity( | ||
| default_value=u.Quantity(0.32, u.deg), |
There was a problem hiding this comment.
0.32 degrees is a special case for 0.4 degree wobble observations. I'm not sure this is a good default to use here, as we 0.4 degree is not a wobble offset we intend to use for CTAO.
It is used by LST to enable common observations with MAGIC, but a larger value would be much better for the large FoVs of the CTAO telescopes and the relatively large PSF of a single LST at low energies.
This does not affect main, it was related to the application of the multiplicity cut introduced here. |
This changes the
PointSourceSensitivityOptimizerto use the full cut optimization introduced in pyirf 0.13. The previous EventDisplay-like optimization can now be used via thePointSourceSensitivityGhOptimizer.I did a quick comparison of the three optimizers using prod6 files (multiplicity >= 2 for gh opt and percentile cuts, as the HillasReconstructer is used):
I am not sure, why the EventDisplay-like optimization results in a better sensitivity at high energies.
Fixes #2771