ESQL: Implement percentiles agg for exponential_histogram

[JonasKunz]

Implements PERCENTILE agg support for exponential_histogram (part of #137549):

• Adds a new MERGE aggregation to merge exponential histograms. This aggregation is intentionally not registered and can only be used in surrogates
• Adds a HISTOGRAM_PERCENTILE(MERGE(field), percentile) surrogate to implement PERCENTILES.

This time I remembered to also run the tests locally in non-snapshot mode, so something like #137693 shouldn't happen again.

Looking at the logical plans for the CSV tests also shows the performance benefit of splitting the percentile extraction and the merge aggregation:

Project[[instance{f}#29, p0{r}#6, p50{r}#18, p99{r}#21, p100{r}#15]]
\_TopN[[Order[instance{f}#29,ASC,LAST]],1000[INTEGER],false]
  \_Eval[[HISTOGRAMPERCENTILE($$MERGE$p0$0{r$}#32,0[INTEGER]) AS p0#6,
          HISTOGRAMPERCENTILE($$MERGE$p0$0{r$}#32,50[INTEGER]) AS p50#9,
          HISTOGRAMPERCENTILE($$MERGE$p0$0{r$}#32,99[INTEGER]) AS p99#12,
          HISTOGRAMPERCENTILE($$MERGE$p0$0{r$}#32,100[INTEGER]) AS p100#15,
          ROUND(p50{r}#9,7[INTEGER]) AS p50#18,
          ROUND(p99{r}#12,7[INTEGER]) AS p99#21]]
    \_Aggregate[[instance{f}#29],[MERGE(responseTime{f}#31,true[BOOLEAN],PT0S[TIME_DURATION]) AS $$MERGE$p0$0#32, instance{f}#29]]
      \_Filter[NOT(STARTSWITH(instance{f}#29,dummy[KEYWORD]))]
        \_EsRelation[exp_histo_sample][@timestamp{f}#30, instance{f}#29, responseTime{f}#3..]

Even though we query four percentiles, we only do the merge aggregation once. This would not be the case if merge + percentile extraction were fused in a single aggregation.

@kkrik-es @not-napoleon : I intended to add you as optional reviewers. So feel free to ignore the request.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[kkrik-es]

nit: round p100 too?

[JonasKunz]

That's not necessary, because p0 is the min and p100 is the max. Those are exact across machines and merging

[kkrik-es]

Fix also AGGREGATE_METRIC_DOUBLE? We have a type for it, no?

[JonasKunz]

We do, but we don't have any non-surrogate aggregations for AGGREGATE_METRIC_DOUBLE yet. So this would be effectively dead and untested code, which I would want to avoid adding right now. I think we should only add it when we actually add a non-surrogate aggregation for aggregate metric double.

[kkrik-es]

LGTM, leaving it to Nhat to stamp.

[JonasKunz]

I had an interesting test failure where the percentiles where a tiny, tiny bit off the exact values on CI.
This also wouldn't reproduce locally with the same seed.
My guess here is that this is due to the different implementation of Math.log / Math.exp on my machine (Mac M1) vs the CI runners. See also this SO thread

For that reason I've added the rounding to make the tests more robust.

[JonasKunz]

I have removed this assert, because this can now actually occur: We merge histograms, which means the zero-threshold of the result can be defined by a (scale, index) tuple instead of a double value. So when storing that merged histogram into the block, we have to "round" that to the nearest double, introducing a tiny bit of error.

This won't matter in practice because the error is tiny and we are only estimating percentiles anyway, but the errors could add up. I say this is okay for a tech preview, but the TODO above still stands that we should fix this latest before moving out of tech preview.

Ideally we can add some benchmarks first, so that we can also see if the encoding intended for disk-storage performed below is too costly (=it appears in the flame graphs) and we should do something special or not.

[JonasKunz]

That's not necessary, because p0 is the min and p100 is the max. Those are exact across machines and merging

[JonasKunz]

We do, but we don't have any non-surrogate aggregations for AGGREGATE_METRIC_DOUBLE yet. So this would be effectively dead and untested code, which I would want to avoid adding right now. I think we should only add it when we actually add a non-surrogate aggregation for aggregate metric double.