Overhaul block attachment and request dispatch

[pfmooney]

This addresses considerable lock contention and thundering herd issues present in the block IO abstractions. Devices with multiple queues (read: NVMe), serviced by multiple workers, should be in much better shape when it comes to efficiently farming out work.

[pfmooney]

This should not be merged before the R17 release branch is cut. We'll want soak time, consider the scope of work. I wanted to get it up and available for folks to review.

[iximeow]

i've mostly not looked at propolis/src/block yet but figured this first round of thoughts would be useful. on the whole this seems pretty reasonable! but I see that transmute in attachment.rs which I look forward to in.. round 2

[iximeow]

somewhat fewer comments on the back half (plus what we've been talking about re. NoneProcessing)

[pfmooney]

Thanks for looking through so far. I think I've at least touched on all the points you've raised.

[pfmooney]

I'm not sold on returning the QueueId at the moment, but we definitely need some stronger checks here. I've included some additional logic around the admin queues, and fixed the part where we were not doing proper association during state import.

[leftwo]

I have no further comments, thanks.

[iximeow]

just getting back to earlier conversations and resolving where appropriate. if only there was some kind of collaborative forge where the review interface was amenable to conversations...