348 Advanced

348: Async Generator Pattern

Functional Programming

Tutorial

The Problem

Some computations produce values lazily — database result sets, network streams, event queues — where computing all values upfront would be wasteful or impossible. Generators (Python's yield, JavaScript's function*, C#'s yield return) allow a function to produce values one at a time, suspending between yields. Rust doesn't have stable generators yet (RFC 2996 is in progress), but the pattern is emulated via the Iterator trait for synchronous generators and async streams (Stream trait from futures) for async generators. Understanding this pattern prepares you for when Rust's native generator syntax lands.

🎯 Learning Outcomes

• Implement a generator-like struct that holds mutable state and produces values via next()

• Derive the Iterator trait to make the generator composable with iterator adapters

• Use reset() to replay a generator from the beginning

• Understand that generators are stateful iterators — each call to next() advances the state

• Recognize the difference between a generator (lazy sequence) and a Vec (eager sequence)

• See how from_fn and successors are Rust's built-in generator constructors

Code Example

#![allow(clippy::all)]
//! # Async Generator Pattern
//! Yield values one at a time from async computations.

pub struct Generator<T> {
    items: Vec<T>,
    index: usize,
}

impl<T: Clone> Generator<T> {
    pub fn new(items: Vec<T>) -> Self {
        Self { items, index: 0 }
    }
    pub fn next(&mut self) -> Option<T> {
        if self.index < self.items.len() {
            let item = self.items[self.index].clone();
            self.index += 1;
            Some(item)
        } else {
            None
        }
    }
    pub fn reset(&mut self) {
        self.index = 0;
    }
}

impl<T: Clone> Iterator for Generator<T> {
    type Item = T;
    fn next(&mut self) -> Option<Self::Item> {
        Generator::next(self)
    }
}

pub fn range_generator(start: i32, end: i32) -> Generator<i32> {
    Generator::new((start..end).collect())
}

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn generator_yields() {
        let mut g = Generator::new(vec![1, 2, 3]);
        assert_eq!(g.next(), Some(1));
        assert_eq!(g.next(), Some(2));
        assert_eq!(g.next(), Some(3));
        assert_eq!(g.next(), None);
    }
    #[test]
    fn generator_reset() {
        let mut g = Generator::new(vec![1, 2]);
        g.next();
        g.next();
        g.reset();
        assert_eq!(g.next(), Some(1));
    }
    #[test]
    fn as_iterator() {
        let g = range_generator(0, 5);
        let v: Vec<_> = g.collect();
        assert_eq!(v, vec![0, 1, 2, 3, 4]);
    }
}

(* OCaml: generator via Seq (lazy sequence) *)

let fibonacci () : int Seq.t =
  let rec go a b () =
    Seq.Cons (a, go b (a + b))
  in go 0 1

let range_gen start stop step () =
  let rec go i () =
    if i >= stop then Seq.Nil
    else Seq.Cons (i, go (i + step))
  in go start ()

let () =
  (* Take first 10 Fibonacci numbers *)
  let fibs = fibonacci () |> Seq.take 10 |> List.of_seq in
  List.iter (Printf.printf "%d ") fibs;
  print_newline ();

  let r = range_gen 0 20 3 () |> List.of_seq in
  List.iter (Printf.printf "%d ") r;
  print_newline ()

Key Differences

Aspect	Rust `Iterator`	OCaml `Seq.t`
Laziness	On-demand `next()`	Thunk-per-node
Mutability	Mutable state in closure/struct	Immutable (new thunk each time)
Native yield	Not yet (generator RFC pending)	Effect-based in OCaml 5
Composability	Full iterator adapter chain	`Seq.map`, `Seq.filter`, `Seq.flat_map`
Async equivalent	`futures::Stream`	`Lwt_seq` / streaming Lwt promises

OCaml Approach

OCaml 4.14+ has Seq for lazy sequences:

let range start stop =
  let rec go i () =
    if i >= stop then Seq.Nil
    else Seq.Cons (i, go (i + 1))
  in
  go start

(* Use as: Seq.take 5 (range 0 100) |> List.of_seq *)

Seq.t is a lazy list — each node is a thunk (unit -> 'a Seq.node) computed on demand. In OCaml 5, Effect-based generators allow yield-like semantics within a delimited continuation, closer to Python generators.

Full Source

#![allow(clippy::all)]
//! # Async Generator Pattern
//! Yield values one at a time from async computations.

pub struct Generator<T> {
    items: Vec<T>,
    index: usize,
}

impl<T: Clone> Generator<T> {
    pub fn new(items: Vec<T>) -> Self {
        Self { items, index: 0 }
    }
    pub fn next(&mut self) -> Option<T> {
        if self.index < self.items.len() {
            let item = self.items[self.index].clone();
            self.index += 1;
            Some(item)
        } else {
            None
        }
    }
    pub fn reset(&mut self) {
        self.index = 0;
    }
}

impl<T: Clone> Iterator for Generator<T> {
    type Item = T;
    fn next(&mut self) -> Option<Self::Item> {
        Generator::next(self)
    }
}

pub fn range_generator(start: i32, end: i32) -> Generator<i32> {
    Generator::new((start..end).collect())
}

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn generator_yields() {
        let mut g = Generator::new(vec![1, 2, 3]);
        assert_eq!(g.next(), Some(1));
        assert_eq!(g.next(), Some(2));
        assert_eq!(g.next(), Some(3));
        assert_eq!(g.next(), None);
    }
    #[test]
    fn generator_reset() {
        let mut g = Generator::new(vec![1, 2]);
        g.next();
        g.next();
        g.reset();
        assert_eq!(g.next(), Some(1));
    }
    #[test]
    fn as_iterator() {
        let g = range_generator(0, 5);
        let v: Vec<_> = g.collect();
        assert_eq!(v, vec![0, 1, 2, 3, 4]);
    }
}

(* OCaml: generator via Seq (lazy sequence) *)

let fibonacci () : int Seq.t =
  let rec go a b () =
    Seq.Cons (a, go b (a + b))
  in go 0 1

let range_gen start stop step () =
  let rec go i () =
    if i >= stop then Seq.Nil
    else Seq.Cons (i, go (i + step))
  in go start ()

let () =
  (* Take first 10 Fibonacci numbers *)
  let fibs = fibonacci () |> Seq.take 10 |> List.of_seq in
  List.iter (Printf.printf "%d ") fibs;
  print_newline ();

  let r = range_gen 0 20 3 () |> List.of_seq in
  List.iter (Printf.printf "%d ") r;
  print_newline ()

✓ Tests Rust test suite

#[cfg(test)]
mod tests {
    use super::*;
    #[test]
    fn generator_yields() {
        let mut g = Generator::new(vec![1, 2, 3]);
        assert_eq!(g.next(), Some(1));
        assert_eq!(g.next(), Some(2));
        assert_eq!(g.next(), Some(3));
        assert_eq!(g.next(), None);
    }
    #[test]
    fn generator_reset() {
        let mut g = Generator::new(vec![1, 2]);
        g.next();
        g.next();
        g.reset();
        assert_eq!(g.next(), Some(1));
    }
    #[test]
    fn as_iterator() {
        let g = range_generator(0, 5);
        let v: Vec<_> = g.collect();
        assert_eq!(v, vec![0, 1, 2, 3, 4]);
    }
}

Deep Comparison

OCaml vs Rust: Async Generator Pattern

Overview

See the example.rs and example.ml files for detailed implementations.

Key Differences

Aspect	OCaml	Rust
Type system	Hindley-Milner	Ownership + traits
Memory	GC	Zero-cost abstractions
Mutability	Explicit ref	mut keyword
Error handling	Option/Result	Result<T, E>

See README.md for detailed comparison.

Exercises

Fibonacci generator: Implement a Fibonacci struct that implements Iterator<Item = u64>, producing the infinite Fibonacci sequence; take the first 20 values with .take(20).collect::<Vec<_>>().

Chunked generator: Add a chunks(size: usize) method to Generator<T> that returns groups of size items at a time as Vec<T>; handle the last partial chunk correctly.

Async stream: Using tokio_stream or futures::stream::unfold, implement an async generator that yields numbers 1..N with a 10ms delay between each; collect the first 5.

Open Source Repos

functional-rust

View the source for this example on GitHub — OCaml and Rust side by side in the repo.

Rust