263 Intermediate

263: Fixed-Size Chunks Iteration

Functional Programming

Tutorial

The Problem

Batch processing is fundamental to systems programming: sending data in fixed-size network packets, processing database rows in pages, dividing work among threads, or encoding binary data in base64 groups of 3 bytes. The chunks(n) method partitions a slice into non-overlapping sub-slices of at most n elements, handling the remainder (a possibly smaller final chunk) automatically.

🎯 Learning Outcomes

• Understand how chunks(n) divides a slice into non-overlapping groups of at most n elements

• Distinguish chunks() from chunks_exact(): the latter excludes the remainder

• Access the remainder from chunks_exact() separately for clean batch+remainder logic

• Process batches in parallel by distributing chunks across threads

Code Example

#![allow(clippy::all)]
//! 263. Fixed-size chunks iteration
//!
//! `chunks(n)` splits a slice into non-overlapping sub-slices of at most n elements.

#[cfg(test)]
mod tests {
    #[test]
    fn test_chunks_basic() {
        let data = [1i32, 2, 3, 4, 5];
        let chunks: Vec<&[i32]> = data.chunks(2).collect();
        assert_eq!(chunks.len(), 3);
        assert_eq!(chunks[0], &[1, 2]);
        assert_eq!(chunks[2], &[5]);
    }

    #[test]
    fn test_chunks_exact_remainder() {
        let data = [1i32, 2, 3, 4, 5];
        let exact = data.chunks_exact(2);
        assert_eq!(exact.remainder(), &[5]);
    }

    #[test]
    fn test_chunks_divisible() {
        let data = [1i32, 2, 3, 4];
        let chunks: Vec<_> = data.chunks(2).collect();
        assert_eq!(chunks.len(), 2);
        assert!(chunks.iter().all(|c| c.len() == 2));
    }
}

(* 263. Fixed-size chunks iteration - OCaml *)

let chunks n lst =
  let rec aux acc current count = function
    | [] ->
      if current = [] then List.rev acc
      else List.rev (List.rev current :: acc)
    | x :: xs ->
      if count = n then aux (List.rev current :: acc) [x] 1 xs
      else aux acc (x :: current) (count + 1) xs
  in
  aux [] [] 0 lst

let () =
  let nums = [1; 2; 3; 4; 5; 6; 7] in
  let cs = chunks 3 nums in
  Printf.printf "Chunks of 3:\n";
  List.iter (fun chunk ->
    Printf.printf "[%s]\n" (String.concat "; " (List.map string_of_int chunk))
  ) cs;

  let sums = List.map (List.fold_left (+) 0) cs in
  Printf.printf "Chunk sums: %s\n"
    (String.concat ", " (List.map string_of_int sums));

  let batches = chunks 4 (List.init 10 (fun i -> i + 1)) in
  List.iteri (fun i batch ->
    Printf.printf "Batch %d: [%s]\n" i
      (String.concat ", " (List.map string_of_int batch))
  ) batches

Key Differences

Remainder handling: Rust provides chunks_exact() + .remainder() to separate full chunks from the tail; OCaml requires manual logic.

Zero-copy: Rust yields borrowed &[T] slices; OCaml creates new sub-lists or arrays.

Standard library: chunks() is built into Rust's slice API; OCaml requires third-party libraries or custom code.

Industrial use: Batch processing in production systems: page-based DB reads, TLS record splitting, parallel work distribution.

OCaml Approach

OCaml lacks a standard chunks function on lists. The standard approach uses List.filteri with modular arithmetic or a recursive accumulator:

let rec chunks n lst = match lst with
  | [] -> []
  | _ ->
    let (h, t) = (List.filteri (fun i _ -> i < n) lst,
                  List.filteri (fun i _ -> i >= n) lst) in
    h :: chunks n t

For arrays, Array.sub arr (i * n) n achieves the same but allocates new arrays.

Full Source

#![allow(clippy::all)]
//! 263. Fixed-size chunks iteration
//!
//! `chunks(n)` splits a slice into non-overlapping sub-slices of at most n elements.

#[cfg(test)]
mod tests {
    #[test]
    fn test_chunks_basic() {
        let data = [1i32, 2, 3, 4, 5];
        let chunks: Vec<&[i32]> = data.chunks(2).collect();
        assert_eq!(chunks.len(), 3);
        assert_eq!(chunks[0], &[1, 2]);
        assert_eq!(chunks[2], &[5]);
    }

    #[test]
    fn test_chunks_exact_remainder() {
        let data = [1i32, 2, 3, 4, 5];
        let exact = data.chunks_exact(2);
        assert_eq!(exact.remainder(), &[5]);
    }

    #[test]
    fn test_chunks_divisible() {
        let data = [1i32, 2, 3, 4];
        let chunks: Vec<_> = data.chunks(2).collect();
        assert_eq!(chunks.len(), 2);
        assert!(chunks.iter().all(|c| c.len() == 2));
    }
}

(* 263. Fixed-size chunks iteration - OCaml *)

let chunks n lst =
  let rec aux acc current count = function
    | [] ->
      if current = [] then List.rev acc
      else List.rev (List.rev current :: acc)
    | x :: xs ->
      if count = n then aux (List.rev current :: acc) [x] 1 xs
      else aux acc (x :: current) (count + 1) xs
  in
  aux [] [] 0 lst

let () =
  let nums = [1; 2; 3; 4; 5; 6; 7] in
  let cs = chunks 3 nums in
  Printf.printf "Chunks of 3:\n";
  List.iter (fun chunk ->
    Printf.printf "[%s]\n" (String.concat "; " (List.map string_of_int chunk))
  ) cs;

  let sums = List.map (List.fold_left (+) 0) cs in
  Printf.printf "Chunk sums: %s\n"
    (String.concat ", " (List.map string_of_int sums));

  let batches = chunks 4 (List.init 10 (fun i -> i + 1)) in
  List.iteri (fun i batch ->
    Printf.printf "Batch %d: [%s]\n" i
      (String.concat ", " (List.map string_of_int batch))
  ) batches

✓ Tests Rust test suite

#[cfg(test)]
mod tests {
    #[test]
    fn test_chunks_basic() {
        let data = [1i32, 2, 3, 4, 5];
        let chunks: Vec<&[i32]> = data.chunks(2).collect();
        assert_eq!(chunks.len(), 3);
        assert_eq!(chunks[0], &[1, 2]);
        assert_eq!(chunks[2], &[5]);
    }

    #[test]
    fn test_chunks_exact_remainder() {
        let data = [1i32, 2, 3, 4, 5];
        let exact = data.chunks_exact(2);
        assert_eq!(exact.remainder(), &[5]);
    }

    #[test]
    fn test_chunks_divisible() {
        let data = [1i32, 2, 3, 4];
        let chunks: Vec<_> = data.chunks(2).collect();
        assert_eq!(chunks.len(), 2);
        assert!(chunks.iter().all(|c| c.len() == 2));
    }
}

Exercises

Use chunks(3) to implement a simple base64 encoder that processes three bytes at a time and handles padding for the remainder.

Divide a Vec<i32> into N approximately equal chunks and compute the sum of each chunk in parallel using threads.

Process a binary file's bytes in 512-byte chunks, computing a checksum for each chunk independently.

Open Source Repos

functional-rust

View the source for this example on GitHub — OCaml and Rust side by side in the repo.

Rust