907 Intermediate

907-iterator-chunks — Iterator Chunks

Functional Programming

Tutorial

The Problem

Processing data in fixed-size batches is fundamental to I/O buffering, pagination, parallel work distribution, and batch database operations. Reading 4096-byte I/O blocks, processing 100 database rows at a time, distributing work across 8 threads — all require splitting a sequence into non-overlapping fixed-size groups. Rust provides .chunks(n) for variable-size last chunk and .chunks_exact(n) for uniform-size-only processing. These are zero-copy slice operations returning references into the original data. OCaml requires recursive functions or Array.sub for equivalent functionality.

🎯 Learning Outcomes

• Use .chunks(n) to split a slice into groups of at most n elements

• Use .chunks_exact(n) when only full chunks are valid and remainders need separate handling

• Access the remainder via .chunks_exact(n).remainder()

• Implement the recursive OCaml-style chunking for comparison

• Understand the difference between overlapping windows and non-overlapping chunks

Code Example

pub fn chunk_sums(data: &[i32], n: usize) -> Vec<i32> {
    data.chunks(n).map(|c| c.iter().sum()).collect()
}

pub fn full_chunks<T: Clone>(data: &[T], n: usize) -> Vec<Vec<T>> {
    data.chunks_exact(n).map(<[T]>::to_vec).collect()
}

let chunks n lst =
  let rec aux acc current count = function
    | [] ->
      if current = [] then List.rev acc
      else List.rev (List.rev current :: acc)
    | x :: xs ->
      if count = n then aux (List.rev current :: acc) [x] 1 xs
      else aux acc (x :: current) (count + 1) xs
  in
  aux [] [] 0 lst

Key Differences

Zero-copy: Rust .chunks(n) yields references into the original slice; OCaml Array.sub allocates new arrays.

Exact chunks: chunks_exact and .remainder() provide clean separation of full and partial chunks; OCaml requires explicit length checking.

Standard library: Rust has first-class .chunks() and .chunks_exact() on all slices; OCaml requires the Base library or manual recursion.

Mutable chunks: Rust also has .chunks_mut(n) for in-place processing of each chunk; OCaml Array.blit for equivalent mutation.

OCaml Approach

OCaml's Array.init with Array.sub can chunk arrays: Array.init (n / k) (fun i -> Array.sub arr (i*k) k). For lists: recursive chunking via let rec take n = function ... in let rec chunks n xs = match take n xs with | .... The standard library lacks built-in chunk functions for lists. The Base library provides List.chunks_of: 'a list -> length:int -> 'a list list as a first-class function.

Full Source

#![allow(clippy::all)]
//! 263. Fixed-size chunks iteration
//!
//! `chunks(n)` splits a slice into non-overlapping sub-slices of at most n elements.
//! `chunks_exact(n)` yields only full-size chunks; the remainder is accessible separately.

/// Sum each chunk of size `n` in a slice. Returns a Vec of chunk sums.
///
/// Uses `chunks(n)` — the last chunk may be shorter if `len % n != 0`.
pub fn chunk_sums(data: &[i32], n: usize) -> Vec<i32> {
    data.chunks(n).map(|c| c.iter().sum()).collect()
}

/// Split a slice into owned Vec-of-Vecs with at most `n` elements each.
pub fn chunks_owned<T: Clone>(data: &[T], n: usize) -> Vec<Vec<T>> {
    data.chunks(n).map(<[T]>::to_vec).collect()
}

/// Return only the full chunks of size `n`, discarding any remainder.
pub fn full_chunks<T: Clone>(data: &[T], n: usize) -> Vec<Vec<T>> {
    data.chunks_exact(n).map(<[T]>::to_vec).collect()
}

/// Return the remainder after taking all full chunks of size `n`.
pub fn chunks_remainder<T>(data: &[T], n: usize) -> &[T] {
    data.chunks_exact(n).remainder()
}

/// Functional / recursive OCaml-style chunking (no std chunk helpers).
pub fn chunks_recursive<T: Clone>(data: &[T], n: usize) -> Vec<Vec<T>> {
    if data.is_empty() || n == 0 {
        return vec![];
    }
    let (head, tail) = data.split_at(n.min(data.len()));
    let mut result = vec![head.to_vec()];
    result.extend(chunks_recursive(tail, n));
    result
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_chunk_sums_even_division() {
        let data = [1, 2, 3, 4, 5, 6];
        assert_eq!(chunk_sums(&data, 3), vec![6, 15]);
    }

    #[test]
    fn test_chunk_sums_with_remainder() {
        let data = [1, 2, 3, 4, 5, 6, 7];
        // chunks: [1,2,3]=6, [4,5,6]=15, [7]=7
        assert_eq!(chunk_sums(&data, 3), vec![6, 15, 7]);
    }

    #[test]
    fn test_chunk_sums_empty() {
        assert_eq!(chunk_sums(&[], 3), Vec::<i32>::new());
    }

    #[test]
    fn test_chunks_owned_shape() {
        let data = [1, 2, 3, 4, 5];
        let result = chunks_owned(&data, 2);
        assert_eq!(result, vec![vec![1, 2], vec![3, 4], vec![5]]);
    }

    #[test]
    fn test_full_chunks_drops_remainder() {
        let data = [1, 2, 3, 4, 5, 6, 7];
        let result = full_chunks(&data, 3);
        assert_eq!(result, vec![vec![1, 2, 3], vec![4, 5, 6]]);
    }

    #[test]
    fn test_chunks_remainder() {
        let data = [1, 2, 3, 4, 5, 6, 7];
        assert_eq!(chunks_remainder(&data, 3), &[7]);
    }

    #[test]
    fn test_chunks_remainder_empty_when_evenly_divisible() {
        let data = [1, 2, 3, 4, 5, 6];
        assert_eq!(chunks_remainder(&data, 3), &[] as &[i32]);
    }

    #[test]
    fn test_chunks_recursive_matches_std() {
        let data: Vec<i32> = (1..=7).collect();
        let recursive = chunks_recursive(&data, 3);
        let std_chunks: Vec<Vec<i32>> = data.chunks(3).map(|c| c.to_vec()).collect();
        assert_eq!(recursive, std_chunks);
    }

    #[test]
    fn test_chunks_recursive_empty() {
        let empty: Vec<i32> = vec![];
        assert_eq!(chunks_recursive(&empty, 3), Vec::<Vec<i32>>::new());
    }
}

(* 263. Fixed-size chunks iteration - OCaml *)

let chunks n lst =
  let rec aux acc current count = function
    | [] ->
      if current = [] then List.rev acc
      else List.rev (List.rev current :: acc)
    | x :: xs ->
      if count = n then aux (List.rev current :: acc) [x] 1 xs
      else aux acc (x :: current) (count + 1) xs
  in
  aux [] [] 0 lst

let () =
  let nums = [1; 2; 3; 4; 5; 6; 7] in
  let cs = chunks 3 nums in
  Printf.printf "Chunks of 3:\n";
  List.iter (fun chunk ->
    Printf.printf "[%s]\n" (String.concat "; " (List.map string_of_int chunk))
  ) cs;

  let sums = List.map (List.fold_left (+) 0) cs in
  Printf.printf "Chunk sums: %s\n"
    (String.concat ", " (List.map string_of_int sums));

  let batches = chunks 4 (List.init 10 (fun i -> i + 1)) in
  List.iteri (fun i batch ->
    Printf.printf "Batch %d: [%s]\n" i
      (String.concat ", " (List.map string_of_int batch))
  ) batches

✓ Tests Rust test suite

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_chunk_sums_even_division() {
        let data = [1, 2, 3, 4, 5, 6];
        assert_eq!(chunk_sums(&data, 3), vec![6, 15]);
    }

    #[test]
    fn test_chunk_sums_with_remainder() {
        let data = [1, 2, 3, 4, 5, 6, 7];
        // chunks: [1,2,3]=6, [4,5,6]=15, [7]=7
        assert_eq!(chunk_sums(&data, 3), vec![6, 15, 7]);
    }

    #[test]
    fn test_chunk_sums_empty() {
        assert_eq!(chunk_sums(&[], 3), Vec::<i32>::new());
    }

    #[test]
    fn test_chunks_owned_shape() {
        let data = [1, 2, 3, 4, 5];
        let result = chunks_owned(&data, 2);
        assert_eq!(result, vec![vec![1, 2], vec![3, 4], vec![5]]);
    }

    #[test]
    fn test_full_chunks_drops_remainder() {
        let data = [1, 2, 3, 4, 5, 6, 7];
        let result = full_chunks(&data, 3);
        assert_eq!(result, vec![vec![1, 2, 3], vec![4, 5, 6]]);
    }

    #[test]
    fn test_chunks_remainder() {
        let data = [1, 2, 3, 4, 5, 6, 7];
        assert_eq!(chunks_remainder(&data, 3), &[7]);
    }

    #[test]
    fn test_chunks_remainder_empty_when_evenly_divisible() {
        let data = [1, 2, 3, 4, 5, 6];
        assert_eq!(chunks_remainder(&data, 3), &[] as &[i32]);
    }

    #[test]
    fn test_chunks_recursive_matches_std() {
        let data: Vec<i32> = (1..=7).collect();
        let recursive = chunks_recursive(&data, 3);
        let std_chunks: Vec<Vec<i32>> = data.chunks(3).map(|c| c.to_vec()).collect();
        assert_eq!(recursive, std_chunks);
    }

    #[test]
    fn test_chunks_recursive_empty() {
        let empty: Vec<i32> = vec![];
        assert_eq!(chunks_recursive(&empty, 3), Vec::<Vec<i32>>::new());
    }
}

Deep Comparison

OCaml vs Rust: Fixed-Size Chunks Iteration

Side-by-Side Code

OCaml

let chunks n lst =
  let rec aux acc current count = function
    | [] ->
      if current = [] then List.rev acc
      else List.rev (List.rev current :: acc)
    | x :: xs ->
      if count = n then aux (List.rev current :: acc) [x] 1 xs
      else aux acc (x :: current) (count + 1) xs
  in
  aux [] [] 0 lst

Rust (idiomatic)

pub fn chunk_sums(data: &[i32], n: usize) -> Vec<i32> {
    data.chunks(n).map(|c| c.iter().sum()).collect()
}

pub fn full_chunks<T: Clone>(data: &[T], n: usize) -> Vec<Vec<T>> {
    data.chunks_exact(n).map(<[T]>::to_vec).collect()
}

Rust (functional/recursive — OCaml-style)

pub fn chunks_recursive<T: Clone>(data: &[T], n: usize) -> Vec<Vec<T>> {
    if data.is_empty() || n == 0 {
        return vec![];
    }
    let (head, tail) = data.split_at(n.min(data.len()));
    let mut result = vec![head.to_vec()];
    result.extend(chunks_recursive(tail, n));
    result
}

Type Signatures

Concept	OCaml	Rust
Chunk function	`val chunks : int -> 'a list -> 'a list list`	`fn chunks(n: usize) -> ChunksIter<T>` (slice method)
Element collection	`'a list`	`&[T]` (slice)
Result	`'a list list`	`impl Iterator<Item = &[T]>`
Partial last chunk	handled via `[]` base case	automatic — last chunk may be shorter
Exact chunks only	filter manually	`chunks_exact(n)` + `.remainder()`

Key Insights

Zero allocation vs. recursive building: Rust's chunks() is a zero-copy iterator that yields sub-slices (&[T]) directly from the original data. OCaml must build new lists via accumulator recursion, allocating on every step.

**chunks vs chunks_exact:** Rust provides two variants — chunks(n) includes the final short chunk if len % n != 0, while chunks_exact(n) skips it and exposes the leftover via .remainder(). OCaml's manual implementation must handle the short tail in the base case.

Ownership and borrowing: Because chunks() returns sub-slices (&[T]), no data is copied. Turning them into owned Vec<T> requires an explicit .to_vec() call — making the allocation visible and optional.

Iterator composability: The returned iterator composes freely with .map(), .filter(), .enumerate() etc., enabling batch-processing pipelines without intermediate allocations. OCaml would need List.map over the constructed list.

Stack safety: The recursive Rust version mirrors OCaml's accumulator pattern but is not tail-recursive in safe Rust — for large inputs the idiomatic iterator form is always preferred.

When to Use Each Style

**Use idiomatic Rust (chunks / chunks_exact) when:** Processing real data in batches — database writes, image row rendering, pagination — where zero-copy sub-slice access and iterator composability matter.

Use recursive Rust when: Teaching the OCaml-to-Rust translation, or when you need to understand the underlying algorithm without relying on stdlib methods.

Exercises

Implement process_in_batches<T: Clone, U, F: Fn(&[T]) -> U>(data: &[T], batch_size: usize, f: F) -> Vec<U> that applies f to each chunk.

Write pad_to_multiple<T: Clone>(data: &[T], n: usize, pad: T) -> Vec<T> that extends the last chunk to full size.

Implement chunk_by_weight<T>(data: &[T], weight: impl Fn(&T) -> usize, max_weight: usize) -> Vec<Vec<&T>> that creates chunks where total weight stays below the limit.

Open Source Repos

functional-rust

View the source for this example on GitHub — OCaml and Rust side by side in the repo.

Rust