[Meta] NPoS + Babe Roadmap

[liamaharon]

As suggested by @shamil-gadelshin, I'm creating a roadmap describing the pathway to a world with decentralised NPoS + Babe block authoring and consensus.

The roadmap is split into "phases", with the idea of merging progress incrementally as it is ready, rather than accumulating all changes into a huge PR which we try to merge all at once.

Phase 1: Hybrid Consensus Node

Since we plan to switch from Aura consensus to Babe consensus mid-chain, we require a node which is capable of importing and producing both Babe and Aura blocks. This, I am calling a "Hybrid Node". I recorded a video explaining why we need a Hybrid Node, and demonstrating an implementation of it here.

The Hybrid Node in phase 1 supports block production and validation of both Aura and Babe based runtimes. It does NOT yet support seemless switching of Aura -> Babe in the event of a runtime upgrade, which is achieved in phase 2.

I'd like to get this merged without the work planned for phase 2, just because it is a non-breaking change, and will save me lots of time resolving merge conflicts in the future :)

• Hybrid Node PR: Hybrid Consensus Node #1876

Phase 2: Seemless Aura -> Babe Upgrade

There are a few additional things required for the Hybrid Node from phase 1 to support seemless Aura -> Babe runtime upgrades, which requires a slightly adjusted Polkadot SDK:

• The vulnerability raised by @shamil-gadelshin here requires the SDK to allow constructing a BabeVerifier outside of the crate.
• We must replace hardcoded assumptions that a Babe "Genesis" can only occur at block 0 with something more flexible.
• If pallet_babe::CurrentSlot != timestamp_slot, it should log a warning rather than panic in the runtime. This is because I found initializing pallet_babe::CurrentSlot in the runtime upgrade brittle, and want to remove any potential panic vectors however small they are. Note that after the runtime upgrade, the brittleness disappears and pallet_babe::CurrentSlot always equals timestamp_slot.
• Fix a small annoyance when trying to initialise the node logger multiple times.

TODOs

• Switch to Polkadot SDK fork (branch here), which I am using in my Phase 3 draft PR.
• Fix the vulnerability raised by @shamil-gadelshin.
• Node Support for Aura -> Babe Runtime Upgrades #1927

Phase 3: Permissioned NPoS Authorities

Now that we have a Hybrid Node that supports seamless Aura to Babe upgrades and block imports, it is time to enact the Aura PoA to Babe NPoS runtime upgrade.

In the initial stage of NPoS, we will simply migrate the Aura PoA authorities to Babe NPoS authorities, and disallow changes to the authority set.

At this stage, NPoS authorities will receive no incentives for authoring.

TODOs

• Sync with Medulla about running Staking Miners
  • Although these are not technically required for this phase due to the permissioned nature of the authority set, it will be nice to set them up and see them running smoothly in advance of future stages
• Permissioned Babe NPoS Runtime #1708

Phase 4: Expand Authority Set Beyond OTF

Now that we have a Babe + NPoS runtime, we can start diversifying the authority set, swapping out OTF authorities with community authorities. Keep it permissioned while we monitor network health.

TODOs

• Write documentation for participating in NPoS
• Find teams beyond the OTF eager and suitable to be the first to participate in NPoS consensus
• Consider NPoS incentives
• One authority at a time, add community validators until >50% of nodes are community operated and there are no issues with the network.

Phase 5: Open Authority Set

Now that the network has demonstrated the ability to be run among a distributed set of authorities, we are ready to open NPoS registrations to everyone and decommission the OTF nodes.

TODOs

• Finalize NPoS incentives
• Open NPoS staking to everyone
• Monitor staking participant health, tweak incentives as required to ensure secure diverse set
• Decommission OTF authorities