Retry upstreams that fail transiently before any content

CHANGELOG.md

@@ -11,6 +11,74 @@ target.
 
 ## [Unreleased]
 
+### Read before upgrading
+
+- **A request whose upstreams fail for the moment can take up to two
+  minutes to fail.** When every route of a model failed with an overload
+  or a server error, or could not be reached, before any of the answer
+  reached the client, the gateway now waits and sends the request again
+  instead of returning the error at once, for up to
+  `gateway.retry_wait_secs` (120 seconds by default) after the request
+  arrived. Clients and load balancers with a shorter timeout than that may
+  give up first. Set `gateway.retry_wait_secs` to 0 to keep the earlier
+  behaviour.
+- **A whole answer that runs into the request timeout is now an error
+  with a log row.** A non-streaming request's upstream attempt is cut
+  2 seconds before the gateway's 120-second request timeout (counted from
+  when the request reached the gateway port), and the client gets the
+  last upstream error, or a 504 `ProviderTimeout` when there was none,
+  with its `gateway_logs` row, instead of an empty 408. A retry starts
+  only when its wait and another attempt as long as the last one fit
+  before then.
+- **A stream's opening events arrive with its first content.** While a
+  stream can still be sent again, what the upstream sends before any
+  content (Anthropic `message_start` and `ping`, Responses
+  `response.created` and `response.in_progress`, a Chat chunk with only
+  the role) is held back and goes out together with the first content.
+  While it is held, and while the gateway waits to send again, the client
+  receives an SSE comment (`: keep-alive`) after every 15 seconds without
+  data — except Gemini clients, which never receive one. The status is
+  200 as before.
+
+### Added
+
+- **Transient upstream failures are retried.** When every candidate route
+  has been tried and the last one failed for the moment — 500, 502, 503,
+  504 or 529, a connection that could not be made or broke before the
+  answer's head, a stream that broke off or reported an overload or a
+  server error (`overloaded_error`, `api_error`, `server_error`,
+  `server_is_overloaded`, `slow_down`, Gemini `UNAVAILABLE`) before any
+  content — and none of the answer reached the client, the request waits
+  2, 4, 8, then 16 seconds (each give or take 20%), or as long as the
+  upstream said in `retry-after-ms` or `Retry-After` (a zero wait is
+  ignored, and a shorter one than 250 ms is taken as 250 ms), and goes out again
+  to the routes that failed that way. A model with one route, and a
+  stream, goes back to the same route. Each attempt is picked as before:
+  the strategy, the circuit breaker, the route caps, session affinity
+  and the provider holding a conversation's encrypted history all apply,
+  and each failed attempt counts against its route's breaker as it did.
+  Retries stop when the next wait would end past the deadline; an attempt
+  already under way is not cut short. Rate limits and spent quota (429),
+  refused credentials (401, 403), any other 4xx, timeouts, the gateway's
+  own limits, budgets and policies, and a client that left are never
+  retried, and neither is a stream after part of its answer went out.
+- **`gateway.retry_wait_secs`** (Settings → Gateway → Routing strategy):
+  how long after a request arrives it may still be sent again, 0 to 600
+  seconds, default 120; 0 turns retries off.
+- **`gateway_provider_retry_total{provider}`**: attempts sent again after
+  a transient failure, by the provider they went to, next to
+  `gateway_provider_fallback_total`.
+- The `gateway_logs` row of a request that was sent again records
+  `"retries": n` in its detail, whether it then succeeded or failed.
+
+### Changed
+
+- A provider whose base URL cannot be made into a request fails as
+  `ProviderError` instead of `NetworkError`, and is not retried.
+- An error a stream reports as OpenAI's `server_error` is logged as the
+  upstream's 500 (`ProviderHttpError`), as the same error answered with
+  HTTP 500 is, instead of 502 (`ProviderError`).
+
 ## [3.9.1] — 2026-10-23
 
 A conversation whose history a provider encrypted, such as a Codex

crates/common/src/dynamic_config.rs

@@ -216,6 +216,12 @@ macro_rules! dc_getters_u64_from_i64 {
     };
 }
 
+/// `gateway.retry_wait_secs` when the setting is missing.
+pub const RETRY_WAIT_DEFAULT_SECS: i64 = 120;
+/// Upper bound on `gateway.retry_wait_secs`, enforced both at write time
+/// (the settings handler) and at read time.
+pub const RETRY_WAIT_MAX_SECS: i64 = 600;
+
 /// Upper bound on `security.signature_drift_secs`, enforced both at
 /// write time (admin settings validation) and at read time (the
 /// getter below). Anything beyond five minutes weakens replay
@@ -440,6 +446,18 @@ impl DynamicConfig {
         raw.clamp(0, 86400)
     }
 
+    /// How long after it arrived a request may still be sent again to
+    /// upstreams that failed for the moment (overloaded, a server error,
+    /// a broken connection) before any of the answer reached the client.
+    /// Clamped to [0, 600]; 0 never sends a request again.
+    pub async fn retry_wait_secs(&self) -> u64 {
+        let raw = self
+            .get_i64("gateway.retry_wait_secs")
+            .await
+            .unwrap_or(RETRY_WAIT_DEFAULT_SECS);
+        raw.clamp(0, RETRY_WAIT_MAX_SECS) as u64
+    }
+
     /// Exponent for the `latency` strategy: higher = more aggressive
     /// (heavier penalty on slow routes). Default 2.0 (aggressive).
     /// Clamped to [0.5, 8.0] so a typo can't make weights collapse

crates/gateway/Cargo.toml

@@ -40,3 +40,4 @@ rust_decimal = { workspace = true }
 
 [dev-dependencies]
 http_1x = { package = "http", version = "1" }
+tokio = { workspace = true, features = ["test-util"] }

crates/gateway/src/error.rs

@@ -12,9 +12,17 @@ pub enum GatewayError {
     /// Upstream returned a non-2xx, non-429, non-401 status. The
     /// status is kept structured so error-classifier metrics stay
     /// readable and the gateway can classify retry-eligible 5xx
-    /// versus poison 4xx without parsing the message.
+    /// versus poison 4xx without parsing the message. `retry_after` is
+    /// when the upstream said to try again (`retry-after-ms` or
+    /// `Retry-After`), which an overloaded upstream can send with a 503:
+    /// the gateway waits that long before sending to it again (see
+    /// `proxy::retry`).
     #[error("Provider HTTP {status}: {message}")]
-    ProviderHttpError { status: u16, message: String },
+    ProviderHttpError {
+        status: u16,
+        message: String,
+        retry_after: Option<std::time::Duration>,
+    },
     /// Upstream took longer than the configured timeout. Distinct
     /// from a network drop because the request reached the upstream
     /// — only the response was missing in time.