997 Advanced

997 Retry Backoff

Functional Programming

Tutorial

The Problem

Implement a retry combinator with exponential backoff: given a fallible operation FnMut() -> Result<T, E>, try up to max_attempts times, sleeping base_delay * 2^attempt milliseconds between failures. Also implement a jitter variant to avoid thundering-herd problems. The combinator is purely functional — it wraps any fallible operation.

🎯 Learning Outcomes

• Implement retry<T, E, F: FnMut() -> Result<T,E>>(max, base_ms, f) -> Result<T,E>

• Use 1 << attempt for exponential backoff: attempt 0 = base, attempt 1 = 2×base, attempt 2 = 4×base

• Skip the final sleep: if attempt + 1 < max_attempts { sleep(...) }

• Add jitter: add base / 3 * (attempt % 3) to spread retry times across time

• Recognize the combinator pattern: retry is higher-order, wrapping any FnMut() -> Result

Code Example

#![allow(clippy::all)]
// 997: Retry with Exponential Backoff
// Pure functional combinator — wraps any FnMut() -> Result<T,E>

use std::thread;
use std::time::Duration;

// --- Core retry with exponential backoff ---
fn retry<T, E, F>(max_attempts: usize, base_delay_ms: u64, mut f: F) -> Result<T, E>
where
    F: FnMut() -> Result<T, E>,
{
    let mut last_err = None;
    for attempt in 0..max_attempts {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => {
                last_err = Some(e);
                if attempt + 1 < max_attempts {
                    let delay = base_delay_ms * (1 << attempt);
                    thread::sleep(Duration::from_millis(delay));
                }
            }
        }
    }
    Err(last_err.unwrap())
}

// --- Retry with jitter to avoid thundering herd ---
fn retry_with_jitter<T, E, F>(max_attempts: usize, base_delay_ms: u64, mut f: F) -> Result<T, E>
where
    F: FnMut() -> Result<T, E>,
{
    let mut last_err = None;
    for attempt in 0..max_attempts {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => {
                last_err = Some(e);
                if attempt + 1 < max_attempts {
                    let base = base_delay_ms * (1 << attempt);
                    // Simple deterministic "jitter" using attempt number
                    let jitter = base / 3 * (attempt as u64 % 3);
                    thread::sleep(Duration::from_millis(base + jitter));
                }
            }
        }
    }
    Err(last_err.unwrap())
}

// --- Retry only for retryable errors ---
#[derive(Debug, PartialEq)]
enum MyError {
    Transient(String),
    Permanent(String),
}

fn retry_if<T, F, P>(
    max_attempts: usize,
    base_delay_ms: u64,
    is_retryable: P,
    mut f: F,
) -> Result<T, MyError>
where
    F: FnMut() -> Result<T, MyError>,
    P: Fn(&MyError) -> bool,
{
    let mut last_err = None;
    for attempt in 0..max_attempts {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) if !is_retryable(&e) => return Err(e),
            Err(e) => {
                last_err = Some(e);
                if attempt + 1 < max_attempts {
                    let delay = base_delay_ms * (1 << attempt);
                    thread::sleep(Duration::from_millis(delay));
                }
            }
        }
    }
    Err(last_err.unwrap())
}

// --- Builder pattern for ergonomic retry configuration ---
struct RetryConfig {
    max_attempts: usize,
    base_delay_ms: u64,
}

impl RetryConfig {
    fn new() -> Self {
        RetryConfig {
            max_attempts: 3,
            base_delay_ms: 100,
        }
    }
    fn attempts(mut self, n: usize) -> Self {
        self.max_attempts = n;
        self
    }
    fn base_delay(mut self, ms: u64) -> Self {
        self.base_delay_ms = ms;
        self
    }

    fn run<T, E, F: FnMut() -> Result<T, E>>(&self, f: F) -> Result<T, E> {
        retry(self.max_attempts, self.base_delay_ms, f)
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_retry_succeeds_on_third_attempt() {
        let mut count = 0;
        let result = retry(5, 1, || {
            count += 1;
            if count < 3 {
                Err("fail")
            } else {
                Ok(count)
            }
        });
        assert_eq!(result, Ok(3));
        assert_eq!(count, 3);
    }

    #[test]
    fn test_retry_exhausts_attempts() {
        let mut count = 0;
        let result = retry::<i32, &str, _>(3, 1, || {
            count += 1;
            Err("always fails")
        });
        assert!(result.is_err());
        assert_eq!(count, 3);
    }

    #[test]
    fn test_retry_succeeds_first_try() {
        let result = retry::<i32, &str, _>(3, 1, || Ok(42));
        assert_eq!(result, Ok(42));
    }

    #[test]
    fn test_retry_if_stops_on_permanent() {
        let mut count = 0;
        let result: Result<i32, MyError> = retry_if(
            5,
            1,
            |e| matches!(e, MyError::Transient(_)),
            || {
                count += 1;
                Err(MyError::Permanent("unrecoverable".to_string()))
            },
        );
        assert!(result.is_err());
        assert_eq!(count, 1); // stopped immediately on Permanent error
    }

    #[test]
    fn test_retry_if_retries_transient() {
        let mut count = 0;
        let result = retry_if(
            5,
            1,
            |e| matches!(e, MyError::Transient(_)),
            || {
                count += 1;
                if count < 3 {
                    Err(MyError::Transient("temp".to_string()))
                } else {
                    Ok(42)
                }
            },
        );
        assert_eq!(result, Ok(42));
        assert_eq!(count, 3);
    }

    #[test]
    fn test_builder_config() {
        let mut n = 0;
        let r = RetryConfig::new().attempts(4).base_delay(1).run(|| {
            n += 1;
            if n < 2 {
                Err("x")
            } else {
                Ok(n)
            }
        });
        assert_eq!(r, Ok(2));
    }
}

(* 997: Retry with Exponential Backoff *)
(* Pure functional combinator — no concurrency needed *)

(* --- Core retry combinator --- *)

let retry ?(max_attempts=3) ?(base_delay_ms=100) f =
  let rec loop attempt =
    match f () with
    | Ok v -> Ok v
    | Error e when attempt >= max_attempts -> Error e
    | Error _ ->
      let delay = base_delay_ms * (1 lsl (attempt - 1)) in
      Unix.sleepf (float_of_int delay /. 1000.0);
      loop (attempt + 1)
  in
  loop 1

(* --- Approach 1: Retry a flaky operation --- *)

let () =
  let attempt_count = ref 0 in
  let result = retry ~max_attempts:5 ~base_delay_ms:1 (fun () ->
    incr attempt_count;
    if !attempt_count < 3 then Error "not ready yet"
    else Ok !attempt_count
  ) in
  assert (result = Ok 3);
  assert (!attempt_count = 3);
  Printf.printf "Approach 1 (retry): succeeded on attempt %d\n" !attempt_count

(* --- Approach 2: Retry combinator with jitter --- *)

let retry_with_jitter ?(max_attempts=5) ?(base_ms=50) f =
  let rec loop attempt =
    match f () with
    | Ok v -> Ok v
    | Error e when attempt >= max_attempts -> Error e
    | Error _ ->
      let delay = base_ms * (1 lsl (attempt - 1)) in
      let jitter = Random.int (delay / 2 + 1) in
      Unix.sleepf (float_of_int (delay + jitter) /. 1000.0);
      loop (attempt + 1)
  in
  loop 1

let () =
  let calls = ref 0 in
  let result = retry_with_jitter ~max_attempts:4 ~base_ms:1 (fun () ->
    incr calls;
    if !calls < 2 then Error "busy"
    else Ok 42
  ) in
  assert (result = Ok 42);
  Printf.printf "Approach 2 (with jitter): ok after %d calls\n" !calls

(* --- Approach 3: Retry with predicate (not all errors are retryable) --- *)

type error = Transient of string | Permanent of string

let retry_if ~is_retryable ?(max_attempts=3) ?(base_ms=10) f =
  let rec loop attempt =
    match f () with
    | Ok v -> Ok v
    | Error (Permanent _ as e) -> Error e
    | Error e when attempt >= max_attempts -> Error e
    | Error e when not (is_retryable e) -> Error e
    | Error _ ->
      Unix.sleepf (float_of_int (base_ms * (1 lsl (attempt - 1))) /. 1000.0);
      loop (attempt + 1)
  in
  loop 1

let () =
  let calls = ref 0 in
  let result = retry_if ~is_retryable:(function Transient _ -> true | _ -> false)
    ~max_attempts:5 ~base_ms:1 (fun () ->
      incr calls;
      if !calls < 3 then Error (Transient "network error")
      else Ok "connected"
    )
  in
  assert (result = Ok "connected");
  Printf.printf "Approach 3 (selective retry): ok after %d calls\n" !calls

let () = Printf.printf "✓ All tests passed\n"

Key Differences

Aspect	Rust	OCaml
Loop style	Imperative `for` loop	Recursive `go` function
Exponential delay	`1 << attempt`	`1 lsl attempt`
Skip final sleep	`if attempt + 1 < max`	`when attempt + 1 >= max -> Error e`
Max delay cap	Manual `min(max_delay, delay)`	Same

Jitter prevents "thundering herd": when many clients retry simultaneously, jitter spreads them out in time. Adding rand::thread_rng().gen_range(0..base/2) provides true randomized jitter in production.

OCaml Approach

let retry max_attempts base_delay_ms f =
  let rec go attempt =
    match f () with
    | Ok v -> Ok v
    | Error e when attempt + 1 >= max_attempts -> Error e
    | Error _ ->
      let delay = base_delay_ms * (1 lsl attempt) in
      Thread.delay (float_of_int delay /. 1000.0);
      go (attempt + 1)
  in
  go 0

let retry_with_backoff ~max ~base_ms ?(max_delay_ms = 30_000) f =
  let rec loop attempt =
    match f () with
    | Ok v -> Ok v
    | Error e when attempt >= max -> Error e
    | Error _ ->
      let delay = min max_delay_ms (base_ms * (1 lsl attempt)) in
      Unix.sleepf (float_of_int delay /. 1000.0);
      loop (attempt + 1)
  in
  loop 0

OCaml's tail-recursive go avoids accumulating stack frames across retries. The when guard in the Error arm cleanly separates "final attempt" from "retry" without a separate branch.

Full Source

#![allow(clippy::all)]
// 997: Retry with Exponential Backoff
// Pure functional combinator — wraps any FnMut() -> Result<T,E>

use std::thread;
use std::time::Duration;

// --- Core retry with exponential backoff ---
fn retry<T, E, F>(max_attempts: usize, base_delay_ms: u64, mut f: F) -> Result<T, E>
where
    F: FnMut() -> Result<T, E>,
{
    let mut last_err = None;
    for attempt in 0..max_attempts {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => {
                last_err = Some(e);
                if attempt + 1 < max_attempts {
                    let delay = base_delay_ms * (1 << attempt);
                    thread::sleep(Duration::from_millis(delay));
                }
            }
        }
    }
    Err(last_err.unwrap())
}

// --- Retry with jitter to avoid thundering herd ---
fn retry_with_jitter<T, E, F>(max_attempts: usize, base_delay_ms: u64, mut f: F) -> Result<T, E>
where
    F: FnMut() -> Result<T, E>,
{
    let mut last_err = None;
    for attempt in 0..max_attempts {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) => {
                last_err = Some(e);
                if attempt + 1 < max_attempts {
                    let base = base_delay_ms * (1 << attempt);
                    // Simple deterministic "jitter" using attempt number
                    let jitter = base / 3 * (attempt as u64 % 3);
                    thread::sleep(Duration::from_millis(base + jitter));
                }
            }
        }
    }
    Err(last_err.unwrap())
}

// --- Retry only for retryable errors ---
#[derive(Debug, PartialEq)]
enum MyError {
    Transient(String),
    Permanent(String),
}

fn retry_if<T, F, P>(
    max_attempts: usize,
    base_delay_ms: u64,
    is_retryable: P,
    mut f: F,
) -> Result<T, MyError>
where
    F: FnMut() -> Result<T, MyError>,
    P: Fn(&MyError) -> bool,
{
    let mut last_err = None;
    for attempt in 0..max_attempts {
        match f() {
            Ok(v) => return Ok(v),
            Err(e) if !is_retryable(&e) => return Err(e),
            Err(e) => {
                last_err = Some(e);
                if attempt + 1 < max_attempts {
                    let delay = base_delay_ms * (1 << attempt);
                    thread::sleep(Duration::from_millis(delay));
                }
            }
        }
    }
    Err(last_err.unwrap())
}

// --- Builder pattern for ergonomic retry configuration ---
struct RetryConfig {
    max_attempts: usize,
    base_delay_ms: u64,
}

impl RetryConfig {
    fn new() -> Self {
        RetryConfig {
            max_attempts: 3,
            base_delay_ms: 100,
        }
    }
    fn attempts(mut self, n: usize) -> Self {
        self.max_attempts = n;
        self
    }
    fn base_delay(mut self, ms: u64) -> Self {
        self.base_delay_ms = ms;
        self
    }

    fn run<T, E, F: FnMut() -> Result<T, E>>(&self, f: F) -> Result<T, E> {
        retry(self.max_attempts, self.base_delay_ms, f)
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_retry_succeeds_on_third_attempt() {
        let mut count = 0;
        let result = retry(5, 1, || {
            count += 1;
            if count < 3 {
                Err("fail")
            } else {
                Ok(count)
            }
        });
        assert_eq!(result, Ok(3));
        assert_eq!(count, 3);
    }

    #[test]
    fn test_retry_exhausts_attempts() {
        let mut count = 0;
        let result = retry::<i32, &str, _>(3, 1, || {
            count += 1;
            Err("always fails")
        });
        assert!(result.is_err());
        assert_eq!(count, 3);
    }

    #[test]
    fn test_retry_succeeds_first_try() {
        let result = retry::<i32, &str, _>(3, 1, || Ok(42));
        assert_eq!(result, Ok(42));
    }

    #[test]
    fn test_retry_if_stops_on_permanent() {
        let mut count = 0;
        let result: Result<i32, MyError> = retry_if(
            5,
            1,
            |e| matches!(e, MyError::Transient(_)),
            || {
                count += 1;
                Err(MyError::Permanent("unrecoverable".to_string()))
            },
        );
        assert!(result.is_err());
        assert_eq!(count, 1); // stopped immediately on Permanent error
    }

    #[test]
    fn test_retry_if_retries_transient() {
        let mut count = 0;
        let result = retry_if(
            5,
            1,
            |e| matches!(e, MyError::Transient(_)),
            || {
                count += 1;
                if count < 3 {
                    Err(MyError::Transient("temp".to_string()))
                } else {
                    Ok(42)
                }
            },
        );
        assert_eq!(result, Ok(42));
        assert_eq!(count, 3);
    }

    #[test]
    fn test_builder_config() {
        let mut n = 0;
        let r = RetryConfig::new().attempts(4).base_delay(1).run(|| {
            n += 1;
            if n < 2 {
                Err("x")
            } else {
                Ok(n)
            }
        });
        assert_eq!(r, Ok(2));
    }
}

(* 997: Retry with Exponential Backoff *)
(* Pure functional combinator — no concurrency needed *)

(* --- Core retry combinator --- *)

let retry ?(max_attempts=3) ?(base_delay_ms=100) f =
  let rec loop attempt =
    match f () with
    | Ok v -> Ok v
    | Error e when attempt >= max_attempts -> Error e
    | Error _ ->
      let delay = base_delay_ms * (1 lsl (attempt - 1)) in
      Unix.sleepf (float_of_int delay /. 1000.0);
      loop (attempt + 1)
  in
  loop 1

(* --- Approach 1: Retry a flaky operation --- *)

let () =
  let attempt_count = ref 0 in
  let result = retry ~max_attempts:5 ~base_delay_ms:1 (fun () ->
    incr attempt_count;
    if !attempt_count < 3 then Error "not ready yet"
    else Ok !attempt_count
  ) in
  assert (result = Ok 3);
  assert (!attempt_count = 3);
  Printf.printf "Approach 1 (retry): succeeded on attempt %d\n" !attempt_count

(* --- Approach 2: Retry combinator with jitter --- *)

let retry_with_jitter ?(max_attempts=5) ?(base_ms=50) f =
  let rec loop attempt =
    match f () with
    | Ok v -> Ok v
    | Error e when attempt >= max_attempts -> Error e
    | Error _ ->
      let delay = base_ms * (1 lsl (attempt - 1)) in
      let jitter = Random.int (delay / 2 + 1) in
      Unix.sleepf (float_of_int (delay + jitter) /. 1000.0);
      loop (attempt + 1)
  in
  loop 1

let () =
  let calls = ref 0 in
  let result = retry_with_jitter ~max_attempts:4 ~base_ms:1 (fun () ->
    incr calls;
    if !calls < 2 then Error "busy"
    else Ok 42
  ) in
  assert (result = Ok 42);
  Printf.printf "Approach 2 (with jitter): ok after %d calls\n" !calls

(* --- Approach 3: Retry with predicate (not all errors are retryable) --- *)

type error = Transient of string | Permanent of string

let retry_if ~is_retryable ?(max_attempts=3) ?(base_ms=10) f =
  let rec loop attempt =
    match f () with
    | Ok v -> Ok v
    | Error (Permanent _ as e) -> Error e
    | Error e when attempt >= max_attempts -> Error e
    | Error e when not (is_retryable e) -> Error e
    | Error _ ->
      Unix.sleepf (float_of_int (base_ms * (1 lsl (attempt - 1))) /. 1000.0);
      loop (attempt + 1)
  in
  loop 1

let () =
  let calls = ref 0 in
  let result = retry_if ~is_retryable:(function Transient _ -> true | _ -> false)
    ~max_attempts:5 ~base_ms:1 (fun () ->
      incr calls;
      if !calls < 3 then Error (Transient "network error")
      else Ok "connected"
    )
  in
  assert (result = Ok "connected");
  Printf.printf "Approach 3 (selective retry): ok after %d calls\n" !calls

let () = Printf.printf "✓ All tests passed\n"

✓ Tests Rust test suite

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_retry_succeeds_on_third_attempt() {
        let mut count = 0;
        let result = retry(5, 1, || {
            count += 1;
            if count < 3 {
                Err("fail")
            } else {
                Ok(count)
            }
        });
        assert_eq!(result, Ok(3));
        assert_eq!(count, 3);
    }

    #[test]
    fn test_retry_exhausts_attempts() {
        let mut count = 0;
        let result = retry::<i32, &str, _>(3, 1, || {
            count += 1;
            Err("always fails")
        });
        assert!(result.is_err());
        assert_eq!(count, 3);
    }

    #[test]
    fn test_retry_succeeds_first_try() {
        let result = retry::<i32, &str, _>(3, 1, || Ok(42));
        assert_eq!(result, Ok(42));
    }

    #[test]
    fn test_retry_if_stops_on_permanent() {
        let mut count = 0;
        let result: Result<i32, MyError> = retry_if(
            5,
            1,
            |e| matches!(e, MyError::Transient(_)),
            || {
                count += 1;
                Err(MyError::Permanent("unrecoverable".to_string()))
            },
        );
        assert!(result.is_err());
        assert_eq!(count, 1); // stopped immediately on Permanent error
    }

    #[test]
    fn test_retry_if_retries_transient() {
        let mut count = 0;
        let result = retry_if(
            5,
            1,
            |e| matches!(e, MyError::Transient(_)),
            || {
                count += 1;
                if count < 3 {
                    Err(MyError::Transient("temp".to_string()))
                } else {
                    Ok(42)
                }
            },
        );
        assert_eq!(result, Ok(42));
        assert_eq!(count, 3);
    }

    #[test]
    fn test_builder_config() {
        let mut n = 0;
        let r = RetryConfig::new().attempts(4).base_delay(1).run(|| {
            n += 1;
            if n < 2 {
                Err("x")
            } else {
                Ok(n)
            }
        });
        assert_eq!(r, Ok(2));
    }
}

Deep Comparison

Retry with Exponential Backoff — Comparison

Core Insight

Retry is a higher-order function that wraps a fallible computation and adds resilience. It's pure functional: takes f: unit -> Result<T,E>, returns Result<T,E>. The combinator doesn't care what f does — separation of concerns.

OCaml Approach

• retry ~max_attempts ~base_delay_ms f takes a unit -> ('a, 'e) result thunk

• Named arguments (?max_attempts) give ergonomic defaults

• Unix.sleepf for the delay (seconds as float)

• Pattern match on Error e when attempt >= max_attempts for clean exit

• Recursive loop with accumulator for attempt count

Rust Approach

• FnMut() -> Result<T, E> — FnMut because we call it multiple times

• delay = base_ms * (1 << attempt) — bit shift for powers of 2

• if !is_retryable(&e) { return Err(e) } — short-circuit for permanent errors

• Builder pattern (RetryConfig) for ergonomic configuration

• thread::sleep(Duration::from_millis(delay)) for backoff

Comparison Table

Concept	OCaml	Rust
Function signature	`(unit -> ('a,'e) result) -> result`	`FnMut() -> Result<T,E>`
Default args	`?max_attempts=3`	Struct/builder pattern
Delay calculation	`base * 2^(attempt-1)`	`base_ms * (1 << attempt)`
Sleep primitive	`Unix.sleepf secs`	`thread::sleep(Duration::from_millis)`
Selective retry	`when is_retryable e`	`if !is_retryable(&e) { return Err }`
Jitter	`Random.int (delay/2)`	Deterministic or `rand` crate
Configuration	Named arguments	Builder struct or direct params

std vs tokio

Aspect	std version	tokio version
Runtime	OS threads via `std::thread`	Async tasks on tokio runtime
Synchronization	`std::sync::Mutex`, `Condvar`	`tokio::sync::Mutex`, channels
Channels	`std::sync::mpsc` (unbounded)	`tokio::sync::mpsc` (bounded, async)
Blocking	Thread blocks on lock/recv	Task yields, runtime switches tasks
Overhead	One OS thread per task	Many tasks per thread (M:N)
Best for	CPU-bound, simple concurrency	I/O-bound, high-concurrency servers

Exercises

Add a maximum delay cap: delay = delay.min(max_delay_ms) before sleeping.

Implement retry_until<F, P: Fn(&T) -> bool>(f, pred, max) -> Option<T> — retry until the result satisfies pred.

Add an on_retry: impl Fn(usize, &E) callback that is called before each retry with attempt number and error.

Implement async retry: async fn retry_async<T, E, F: AsyncFn>(...) -> Result<T,E> using tokio::time::sleep.

Implement full binary exponential backoff with jitter: delay = random(0, base * 2^attempt) bounded by a cap.

Open Source Repos

functional-rust

View the source for this example on GitHub — OCaml and Rust side by side in the repo.

Rust