759 Fundamental

759-manual-serialize-trait — Manual Serialize Trait

Functional Programming

Tutorial Video

Text description (accessibility)

This video demonstrates the "759-manual-serialize-trait — Manual Serialize Trait" functional Rust example. Difficulty level: Fundamental. Key concepts covered: Functional Programming. Before `serde` existed, Rust programmers wrote serialization by hand. Key difference from OCaml: 1. **Trait vs function**: Rust uses a `Serialize` trait with method dispatch; OCaml's `Bin_prot` uses module values (`bin_writer_t`) without a unified trait.

Tutorial

The Problem

Before serde existed, Rust programmers wrote serialization by hand. Understanding manual serialization reveals what #[derive(Serialize)] generates and why serde's design is what it is. Manual serialization is still needed when: the format has custom requirements, you target no_std environments, you need maximum performance without abstraction overhead, or you are implementing a wire protocol like MessagePack or Protocol Buffers.

🎯 Learning Outcomes

• Design a Serialize trait with serialize(&self, out: &mut Output) method

• Implement an Output buffer with typed write methods (write_u32, write_string, write_byte)

• Implement Serialize for primitives, strings, and nested structs manually

• Create a corresponding Deserialize trait and Input buffer

• Understand the difference between text serialization (JSON) and binary serialization (this example)

Code Example

pub trait Serialize {
    fn serialize(&self, out: &mut Output);
    
    fn to_bytes(&self) -> Vec<u8> {
        let mut out = Output::new();
        self.serialize(&mut out);
        out.into_bytes()
    }
}

module type SERIALIZE = sig
  type t
  val serialize : t -> Buffer.t -> unit
  val to_bytes : t -> bytes
end

Key Differences

Trait vs function: Rust uses a Serialize trait with method dispatch; OCaml's Bin_prot uses module values (bin_writer_t) without a unified trait.

derive: Both languages have derive macros for automatic implementation; Rust's #[derive(Serialize)] and OCaml's [@@deriving bin_io] generate similar code.

Generic containers: Rust's impl<T: Serialize> Serialize for Vec<T> is idiomatic; OCaml uses parametrized bin_write_list functions.

no_std: Rust's manual serialization works in no_std environments; OCaml's runtime always requires a full GC.

OCaml Approach

OCaml's Marshal module provides automatic binary serialization of any value, including closures and mutable references. For typed serialization, Bin_prot (Jane Street) provides a manual trait-like approach: bin_write_t, bin_read_t, and bin_size_t functions per type. sexplib (also Jane Street) does the same for S-expression serialization. These are the OCaml equivalents of Rust's serde.

Full Source

#![allow(clippy::all)]
//! # Manual Serialize Trait
//!
//! Building a serialization trait from scratch.

/// A simple output buffer for serialization
pub struct Output {
    buffer: Vec<u8>,
}

impl Output {
    pub fn new() -> Self {
        Output { buffer: Vec::new() }
    }

    pub fn write_byte(&mut self, b: u8) {
        self.buffer.push(b);
    }

    pub fn write_bytes(&mut self, bytes: &[u8]) {
        self.buffer.extend_from_slice(bytes);
    }

    pub fn write_u32(&mut self, n: u32) {
        self.buffer.extend_from_slice(&n.to_le_bytes());
    }

    pub fn write_string(&mut self, s: &str) {
        self.write_u32(s.len() as u32);
        self.write_bytes(s.as_bytes());
    }

    pub fn into_bytes(self) -> Vec<u8> {
        self.buffer
    }
}

impl Default for Output {
    fn default() -> Self {
        Self::new()
    }
}

/// The serialize trait
pub trait Serialize {
    fn serialize(&self, out: &mut Output);

    fn to_bytes(&self) -> Vec<u8> {
        let mut out = Output::new();
        self.serialize(&mut out);
        out.into_bytes()
    }
}

// Implement for primitive types

impl Serialize for u8 {
    fn serialize(&self, out: &mut Output) {
        out.write_byte(*self);
    }
}

impl Serialize for u32 {
    fn serialize(&self, out: &mut Output) {
        out.write_u32(*self);
    }
}

impl Serialize for i32 {
    fn serialize(&self, out: &mut Output) {
        out.write_u32(*self as u32);
    }
}

impl Serialize for bool {
    fn serialize(&self, out: &mut Output) {
        out.write_byte(if *self { 1 } else { 0 });
    }
}

impl Serialize for String {
    fn serialize(&self, out: &mut Output) {
        out.write_string(self);
    }
}

impl Serialize for &str {
    fn serialize(&self, out: &mut Output) {
        out.write_string(self);
    }
}

impl<T: Serialize> Serialize for Vec<T> {
    fn serialize(&self, out: &mut Output) {
        out.write_u32(self.len() as u32);
        for item in self {
            item.serialize(out);
        }
    }
}

impl<T: Serialize> Serialize for Option<T> {
    fn serialize(&self, out: &mut Output) {
        match self {
            Some(v) => {
                out.write_byte(1);
                v.serialize(out);
            }
            None => {
                out.write_byte(0);
            }
        }
    }
}

/// Example: A user-defined struct
#[derive(Debug, PartialEq)]
pub struct Person {
    pub name: String,
    pub age: u32,
    pub active: bool,
}

impl Serialize for Person {
    fn serialize(&self, out: &mut Output) {
        self.name.serialize(out);
        self.age.serialize(out);
        self.active.serialize(out);
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_serialize_u8() {
        let bytes = 42u8.to_bytes();
        assert_eq!(bytes, vec![42]);
    }

    #[test]
    fn test_serialize_u32() {
        let bytes = 0x12345678u32.to_bytes();
        assert_eq!(bytes, vec![0x78, 0x56, 0x34, 0x12]); // little-endian
    }

    #[test]
    fn test_serialize_bool() {
        assert_eq!(true.to_bytes(), vec![1]);
        assert_eq!(false.to_bytes(), vec![0]);
    }

    #[test]
    fn test_serialize_string() {
        let bytes = "hi".to_bytes();
        // length (4 bytes) + content
        assert_eq!(bytes, vec![2, 0, 0, 0, b'h', b'i']);
    }

    #[test]
    fn test_serialize_vec() {
        let v: Vec<u8> = vec![1, 2, 3];
        let bytes = v.to_bytes();
        assert_eq!(bytes, vec![3, 0, 0, 0, 1, 2, 3]);
    }

    #[test]
    fn test_serialize_option_some() {
        let opt: Option<u8> = Some(42);
        let bytes = opt.to_bytes();
        assert_eq!(bytes, vec![1, 42]);
    }

    #[test]
    fn test_serialize_option_none() {
        let opt: Option<u8> = None;
        let bytes = opt.to_bytes();
        assert_eq!(bytes, vec![0]);
    }

    #[test]
    fn test_serialize_person() {
        let person = Person {
            name: "Al".to_string(),
            age: 30,
            active: true,
        };
        let bytes = person.to_bytes();
        // name: 4 (len) + 2 (chars) = 6 bytes
        // age: 4 bytes
        // active: 1 byte
        assert_eq!(bytes.len(), 6 + 4 + 1);
    }
}

(* Manual Serialize / Deserialize in OCaml
   We define a module type that mirrors Rust's Serialize trait,
   then implement it for a concrete record type. *)

(* ---------- Serializer "trait" as a module type ---------- *)
module type SERIALIZER = sig
  type t
  val serialize : t -> string
  val deserialize : string -> t option
end

(* ---------- A simple key=value wire format ---------- *)
(* Helper: escape '|' and '\' in strings *)
let escape s =
  let buf = Buffer.create (String.length s + 4) in
  String.iter (fun c ->
    if c = '|' || c = '\\' then Buffer.add_char buf '\\';
    Buffer.add_char buf c
  ) s;
  Buffer.contents buf

let unescape s =
  let buf = Buffer.create (String.length s) in
  let len = String.length s in
  let i = ref 0 in
  while !i < len do
    if s.[!i] = '\\' && !i + 1 < len then begin
      Buffer.add_char buf s.[!i + 1];
      i := !i + 2
    end else begin
      Buffer.add_char buf s.[!i];
      i := !i + 1
    end
  done;
  Buffer.contents buf

(* ---------- Domain type ---------- *)
type person = { name: string; age: int; active: bool }

(* ---------- Implementation ---------- *)
module PersonSerializer : SERIALIZER with type t = person = struct
  type t = person

  let serialize p =
    Printf.sprintf "name=%s|age=%d|active=%b"
      (escape p.name) p.age p.active

  let deserialize s =
    let fields = String.split_on_char '|' s in
    let tbl = Hashtbl.create 4 in
    List.iter (fun field ->
      match String.split_on_char '=' field with
      | k :: rest -> Hashtbl.replace tbl k (unescape (String.concat "=" rest))
      | [] -> ()
    ) fields;
    try
      let name   = Hashtbl.find tbl "name" in
      let age    = int_of_string (Hashtbl.find tbl "age") in
      let active = bool_of_string (Hashtbl.find tbl "active") in
      Some { name; age; active }
    with Not_found | Failure _ -> None
end

(* ---------- Demo ---------- *)
let () =
  let alice = { name = "Alice|Wonder"; age = 30; active = true } in
  let encoded = PersonSerializer.serialize alice in
  Printf.printf "Encoded : %s\n" encoded;
  match PersonSerializer.deserialize encoded with
  | Some p ->
    Printf.printf "Decoded : name=%s age=%d active=%b\n" p.name p.age p.active
  | None ->
    Printf.printf "Decode failed!\n"

✓ Tests Rust test suite

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_serialize_u8() {
        let bytes = 42u8.to_bytes();
        assert_eq!(bytes, vec![42]);
    }

    #[test]
    fn test_serialize_u32() {
        let bytes = 0x12345678u32.to_bytes();
        assert_eq!(bytes, vec![0x78, 0x56, 0x34, 0x12]); // little-endian
    }

    #[test]
    fn test_serialize_bool() {
        assert_eq!(true.to_bytes(), vec![1]);
        assert_eq!(false.to_bytes(), vec![0]);
    }

    #[test]
    fn test_serialize_string() {
        let bytes = "hi".to_bytes();
        // length (4 bytes) + content
        assert_eq!(bytes, vec![2, 0, 0, 0, b'h', b'i']);
    }

    #[test]
    fn test_serialize_vec() {
        let v: Vec<u8> = vec![1, 2, 3];
        let bytes = v.to_bytes();
        assert_eq!(bytes, vec![3, 0, 0, 0, 1, 2, 3]);
    }

    #[test]
    fn test_serialize_option_some() {
        let opt: Option<u8> = Some(42);
        let bytes = opt.to_bytes();
        assert_eq!(bytes, vec![1, 42]);
    }

    #[test]
    fn test_serialize_option_none() {
        let opt: Option<u8> = None;
        let bytes = opt.to_bytes();
        assert_eq!(bytes, vec![0]);
    }

    #[test]
    fn test_serialize_person() {
        let person = Person {
            name: "Al".to_string(),
            age: 30,
            active: true,
        };
        let bytes = person.to_bytes();
        // name: 4 (len) + 2 (chars) = 6 bytes
        // age: 4 bytes
        // active: 1 byte
        assert_eq!(bytes.len(), 6 + 4 + 1);
    }
}

Deep Comparison

OCaml vs Rust: Manual Serialize Trait

Trait Definition

Rust

pub trait Serialize {
    fn serialize(&self, out: &mut Output);
    
    fn to_bytes(&self) -> Vec<u8> {
        let mut out = Output::new();
        self.serialize(&mut out);
        out.into_bytes()
    }
}

OCaml

module type SERIALIZE = sig
  type t
  val serialize : t -> Buffer.t -> unit
  val to_bytes : t -> bytes
end

Implementing for Types

Rust

impl Serialize for u32 {
    fn serialize(&self, out: &mut Output) {
        out.write_u32(*self);
    }
}

impl<T: Serialize> Serialize for Vec<T> {
    fn serialize(&self, out: &mut Output) {
        out.write_u32(self.len() as u32);
        for item in self {
            item.serialize(out);
        }
    }
}

OCaml

let serialize_int32 n buf =
  Buffer.add_int32_le buf n

let serialize_list serialize_elem lst buf =
  serialize_int32 (Int32.of_int (List.length lst)) buf;
  List.iter (fun x -> serialize_elem x buf) lst

Key Differences

Aspect	OCaml	Rust
Polymorphism	Functions	Traits
Generic impl	Manual	`impl<T: Trait>`
Buffer	`Buffer.t`	Custom `Output`
Endianness	`add_int32_le`	`to_le_bytes()`

Exercises

Implement Serialize for Option<T: Serialize> using a one-byte discriminant (0 for None, 1 for Some) followed by the serialized value.

Add Serialize for Vec<T: Serialize> using a 4-byte length prefix, and write a roundtrip test for a struct containing a Vec<String>.

Implement a length-delimited framing layer: write [4 bytes: total_len][serialized_data] and read it back, enabling streaming protocols over TCP.

Open Source Repos

functional-rust

View the source for this example on GitHub — OCaml and Rust side by side in the repo.

Rust