Add re-ordering buffer to the actor side (meta-pytorch#1244)

Summary:

In Rust V1, we want to move the re-ordering buffer from [comm actor](https://www.internalfb.com/code/fbsource/[aea72188b23ef1539c1a04996566a8f188e42644]/fbcode/monarch/hyperactor_mesh/src/comm.rs?lines=383-384) to the actor side. The basic idea is:

* client assigns the seq no for messages it sends to actors;
* the `(client_name, seq_no)` tuple is added to the message's header.
    * for direct send, client will add that header;
    * for multicast, a actor->seq map is passed to comm actors, and comm actor will add that header before delivering the message to actors.
* Each actor instance maintains its own re-ordering buffer, and use the `seq_no` to buffer and re-order the message, before passing the message to the actor loop.

This diff implements the last part, the re-ordering buffer in actor instance.  Specifically, this diff wrap the re-ordering buffer under the `OrderChannel` abstraction, so it can fit to the current implementation seamlessly.

The change is currently disabled behind the `ENABLE_CLIENT_SEQ_ASSIGNMENT` gate.

Reviewed By: mariusae, shayne-fletcher

Differential Revision: D82573932

Author: pzhan9

hyperactor/src/config.rs

@@ -66,6 +66,9 @@ declare_attrs! {
     /// Sampling rate for logging message latency
     /// Set to 0.01 for 1% sampling, 0.1 for 10% sampling, 0.90 for 90% sampling, etc.
     pub attr MESSAGE_LATENCY_SAMPLING_RATE: f32 = 0.01;
+
+    /// Whether to enable client sequence assignment.
+    pub attr ENABLE_CLIENT_SEQ_ASSIGNMENT: bool = false;
 }
 
 /// Load configuration from environment variables