adapter: gracefully handle newer versions in migration shard

Prior to this change, if the leader environment restarted either during
an in-progress 0dt upgrade or after an aborted one, it would fail to
come up if that 0dt upgrade caused builtin item migrations to occur. The
new version would poison the migration shard, and the old version would
halt itself upon reading it.

This commit applies the minimal fix: When performing builtin item
migrations in leader mode, we now ignore the existence of newer versions
in the migration shard, instead of crashing upon observing them.
Read-only processes still crash upon observing newer versions.

This change is meant to derisk subsequent releases before the rewrite of
the builtin item migrations lands. In contrast to that rewrite, this
change is small enough to be backported into previous versions, so we
can deploy it prior to the next release that requires migrations to
occur.

Author: teskje

src/adapter/src/catalog/open/builtin_item_migration.rs

@@ -40,7 +40,7 @@ use mz_storage_client::controller::StorageTxn;
 use mz_storage_types::StorageDiff;
 use mz_storage_types::sources::SourceData;
 use timely::progress::{Antichain, Timestamp as TimelyTimestamp};
-use tracing::{debug, error, info};
+use tracing::{debug, error, info, warn};
 
 use crate::catalog::open::builtin_item_migration::persist_schema::{TableKey, TableKeySchema};
 use crate::catalog::state::LocalExpressionCache;
@@ -415,11 +415,23 @@ async fn migrate_builtin_collections_incompatible(
     let storage_collection_metadata = txn.get_collection_metadata();
     for (table_key, shard_id) in global_id_shards.clone() {
         if table_key.build_version > build_version {
-            halt!(
-                "saw build version {}, which is greater than current build version {}",
-                table_key.build_version,
-                build_version
-            );
+            if read_only {
+                halt!(
+                    "saw build version {}, which is greater than current build version {}",
+                    table_key.build_version,
+                    build_version
+                );
+            } else {
+                // If we are in leader mode, and a newer (read-only) version has started a
+                // migration, we must not allow ourselves to get fenced out! Continuing here might
+                // confuse any read-only process running the migrations concurrently, but it's
+                // better for the read-only env to crash than the leader.
+                // TODO(#9755): handle this in a more principled way
+                warn!(
+                    %table_key.build_version, %build_version,
+                    "saw build version which is greater than current build version",
+                );
+            }
         }
 
         if !migrated_storage_collections.contains(&table_key.global_id)