Better leaderboards could be cool

[benshindel]

Current behavior

Currently, baseline accuracy scales almost linearly with number of questions forecasted, to the point that you could easily win it by just forecasting it on every single question on the platform and constantly mimicking the community forecast. This seems not optimal. Also, peer accuracy rewards you for only participating in very LARGE tournaments where ppl don't update their forecast. You could imagine that if I forecasted on the ACX tournament, and then also forecasted on 50 coin flip questions, I would have a far lower peer accuracy than someone who only forecasted on the ACX tournament, despite possibly being a better forecaster (in relation to my peers).

Expected/desired behavior

1. The peer accuracy prior should be like 50 or 75 questions or something, to actually incentivize doing well across multiple tournaments. Alternatively, the peer accuracy scores could be normalized by volatility (within tournaments or between them), so that participating in tournaments with higher- or lower-than-average peer accuracy scores doesn't guarantee you or lock you out of the leaderboard. I'm currently 1st on peer accuracy so I think I'm unbiased in saying it's gameable, lol.

2. It might be nice to have a leaderboard that captures both of these in some way. A graph plotting peer accuracy against baseline accuracy for the top x users or whatever would be really cool?!

3. Y'all love h-indexes (as should everyone). What about an h-index for peer accuracy? Something like (# of questions with a certain peer accuracy score)? I feel like that could capture some unique new information. That might also work with baseline accuracy. All 4 scores could be h-indexes, that would be neat lol.

[SylvainChevalier]

Thank you for your ideas @benshindel!

First, I agree that Peer leaderboards currently have a problem. See recent discussion here.

Regarding your suggestions:

peer accuracy prior should be like 50 or 75 questions or something
One of the objectives of Peer leaderboards is to let time-constrained forecasters fairly compete without having to spend 10 hours a week forecasting. The 30 question handicap was chosen at a time when we didn't have as many novice-heavy tournaments, and of course before people started gooharting it. We'll look into bumping the handicap, but we want to be cautious here to keep the barrier of entry sufficiently low.

peer accuracy scores could be normalized
This would very likely not be proper.

h-index for peer accuracy
That's... very intriguing. I'll think about it more.