System DB retries #93
Description
Make the system database resilient to connection issues. Right now if a connection is interrupted, a step / DBOS operation will not be retried.
For now we're targeting these common connection errors:
"08006" // connection_failure - Connection lost during operation
"08003" // connection_does_not_exist - Stale/terminated connection
"57P01" // admin_shutdown - Server is shutting down
"57P03" // cannot_connect_now - Server is starting up
"08001" // sqlclient_unable_to_establish_sqlconnection - Can't connect
"08004" // sqlserver_rejected_establishment_of_sqlconnection - Rejected (too many connections)
Interesting library for implementation: https://github.com/avast/retry-go
Interesting for testing: https://github.com/Shopify/toxiproxy