Fix encoding selection logic in Lucene104PostingsWriter

[RamakrishnaChilaka]

The previous code compared numBitsNextBitsPerValue ≤ docRange to decide whether to store the doc-delta block with PackedInts or as a bit-set. docRange is the number of documents, not the number of bits the bit-set would occupy, so the test did not reflect the actual space that would be written.

Change

Replace the comparison with numBitsNextBitsPerValue ≤ numBitSetLongs * Long.SIZE
so the decision is based on the exact bit footprint of each alternative. The bit-set path is now taken only when it is strictly smaller on disk.

We can skip change-log as it is a marginal change, I will add it, if you prefer

[github-actions]

This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.

[jpountz]

This is fine with me.

I'm trying to figure out if this could result in different choices of encoding, but I believe not since this essentially rounds up to the next multiple of 64 while the number of bits required when encoding with FOR are always multiples of 256?

[RamakrishnaChilaka]

Thanks for the review @jpountz.

I think in some cases this will change encoding choice.

For example, Please consider

• If maxBitsPerValue is 4, largest delta can be 15 i.e., cost would be 1280 ((4 + 1) * 256).
• The smallest possible docRange is then 269 (15 + 254 × 1) and the largest is 3825 (15 × 255).

Therefore, blocks that were previously encoded with FOR will continue to be encoded with FOR. However, some blocks that were previously encoded with BitSet may now switch to FOR, because we now account for the true size of the BitSet.

[jpountz]

Ahh, you are right, thanks for explaining. I suggest we wait a couple days so that the performance impact on nightly benchmarks can be easily distinguished from the impact of bumping the block size to 256.

[jpountz]

I think we've waited long enough, feel free to merge!