tx2-pack

Binary world snapshot format for ECS persistence, checkpointing, and time-travel replay.

tx2-pack is the storage layer of the TX-2 ecosystem, providing efficient serialization of Entity-Component-System worlds to disk with compression, encryption, and time-travel capabilities. It enables save/load, replay, and session persistence without requiring a traditional database.

Features

Efficient Storage Format

• Struct-of-arrays layout - Cache-friendly memory organization
• Multiple serialization formats - Bincode (fast) or MessagePack (compact)
• Component archetype grouping - Entities organized by component types
• Versioned format - Magic number + version for compatibility checks

Compression

• Zstd compression - Best compression ratio (configurable levels 1-19)
• LZ4 compression - Fast compression/decompression
• No compression - Raw storage for maximum speed
• Transparent operation - Compression handled automatically

Encryption

• AES-256-GCM - Authenticated encryption with galois counter mode
• Key management - Generate or provide encryption keys
• Optional per-snapshot - Enable encryption as needed
• Secure storage - Protect sensitive world data

Checkpointing

• Checkpoint manager - Save/load/delete checkpoints by ID
• Parent tracking - Checkpoint chains for history
• Metadata support - Tags, descriptions, custom fields
• Pruning - Keep only N most recent checkpoints
• Chain traversal - Navigate checkpoint history

Replay & Time-Travel

• Replay engine - Step forward/backward through checkpoints
• Loop support - Wrap around at start/end
• Seek operations - Jump to specific checkpoint
• Time-travel - Store snapshots at specific timestamps
• Time-based queries - Find snapshot closest to target time
• Forking - Clone world state at any point in time
• Pruning - Remove snapshots before/after timestamp

Data Integrity

• SHA-256 checksums - Verify data integrity on load
• Header validation - Detect corrupted or incompatible files
• Version checking - Ensure format compatibility

Quick Start

Saving and Loading Snapshots

use tx2_pack::{PackedSnapshot, SnapshotWriter, SnapshotReader};

// Create a snapshot from your world
let snapshot = PackedSnapshot::from_world_snapshot(world.create_snapshot());

// Write to file with compression
let writer = SnapshotWriter::new()
    .with_compression(CompressionCodec::zstd_default());

writer.write_to_file(&snapshot, "world.tx2pack")?;

// Read back
let reader = SnapshotReader::new();
let loaded = reader.read_from_file("world.tx2pack")?;

Checkpoints

use tx2_pack::{CheckpointManager, SnapshotMetadata};

// Create checkpoint manager
let mut manager = CheckpointManager::new("./checkpoints")?;

// Save checkpoint
let snapshot = PackedSnapshot::from_world_snapshot(world.create_snapshot());
let metadata = SnapshotMetadata::new("level-1-complete".to_string())
    .with_name("Level 1 Complete")
    .with_description("Player beat first boss")
    .with_tag("milestone");

manager.create_checkpoint("cp1".to_string(), snapshot)?;

// Load checkpoint
let checkpoint = manager.load_checkpoint("cp1")?;
world.restore_from_snapshot(&checkpoint.snapshot)?;

// List all checkpoints
let checkpoints = manager.list_checkpoints()?;

// Delete old checkpoints
manager.prune_old_checkpoints(5)?; // Keep only 5 most recent

Replay Engine

use tx2_pack::{ReplayEngine, CheckpointManager};

// Create replay engine
let mut replay = ReplayEngine::new();

// Load checkpoints from manager
replay.load_from_manager(&mut manager)?;

// Navigate through checkpoints
replay.next(); // Move forward
replay.previous(); // Move backward
replay.seek(5)?; // Jump to index 5
replay.seek_to_start(); // Jump to beginning
replay.seek_to_end(); // Jump to end

// Get current checkpoint
if let Some(checkpoint) = replay.current() {
    world.restore_from_snapshot(&checkpoint.snapshot)?;
}

// Enable looping
let mut replay = ReplayEngine::new().with_loop(true);

Time-Travel

use tx2_pack::TimeTravel;

// Create time-travel system
let mut tt = TimeTravel::new();

// Record snapshots at specific times
for t in 0..100 {
    let snapshot = world.create_snapshot();
    tt.record(t as f64, snapshot);
}

// Seek to specific time
if let Some(snapshot) = tt.seek_to_time(45.0) {
    world.restore_from_snapshot(snapshot)?;
}

// Fork from a specific time
if let Some(forked) = tt.fork_at_time(30.0) {
    // Create alternate timeline from this point
}

// Prune old snapshots
tt.prune_before(20.0); // Remove snapshots before t=20
tt.prune_after(80.0);  // Remove snapshots after t=80

Encryption

use tx2_pack::{SnapshotWriter, SnapshotReader, EncryptionKey};

// Generate encryption key
let key = EncryptionKey::generate();

// Write encrypted snapshot
let writer = SnapshotWriter::new()
    .with_compression(CompressionCodec::zstd_default())
    .with_encryption(key.clone());

writer.write_to_file(&snapshot, "world.tx2pack")?;

// Read encrypted snapshot
let reader = SnapshotReader::new()
    .with_encryption(key);

let loaded = reader.read_from_file("world.tx2pack")?;

Architecture

File Format

[Header][Data]

Header (bincode-serialized):

pub struct SnapshotHeader {
    pub magic: [u8; 8],           // "TX2PACK\0"
    pub version: u32,             // Format version
    pub format: PackFormat,       // Bincode or MessagePack
    pub compression: CompressionType,
    pub encrypted: bool,
    pub checksum: [u8; 32],       // SHA-256 of data
    pub timestamp: i64,
    pub entity_count: u64,
    pub component_count: u64,
    pub archetype_count: u64,
    pub data_offset: u64,         // Offset to data section
    pub data_size: u64,           // Size of data section
}

Data (compressed, optionally encrypted):

pub struct PackedSnapshot {
    pub header: SnapshotHeader,
    pub archetypes: Vec<ComponentArchetype>,
    pub entity_metadata: HashMap<EntityId, EntityMetadata>,
}

Component Archetype

Components are stored in struct-of-arrays layout:

pub struct ComponentArchetype {
    pub component_id: ComponentId,
    pub entity_ids: Vec<EntityId>,
    pub data: ComponentData,
}

pub enum ComponentData {
    StructOfArrays(StructOfArraysData),
    Blob(Vec<u8>),
}

pub struct StructOfArraysData {
    pub field_names: Vec<String>,
    pub field_types: Vec<FieldType>,
    pub field_data: Vec<FieldArray>,
}

Example SoA layout for Position component:

component_id: "Position"
entity_ids: [1, 2, 3, 4, 5]
field_names: ["x", "y", "z"]
field_types: [F32, F32, F32]
field_data: [
    [10.0, 20.0, 30.0, 40.0, 50.0], // x values
    [15.0, 25.0, 35.0, 45.0, 55.0], // y values
    [5.0,  10.0, 15.0, 20.0, 25.0], // z values
]

Benefits:

• Cache-friendly iteration
• SIMD-friendly operations
• Efficient compression (similar values together)

Compression Performance

Tested with 10,000 entities containing Position, Velocity, and Health components:

Storage Operations

SnapshotStore

Manages multiple snapshots in a directory:

use tx2_pack::{SnapshotStore, SnapshotWriter, SnapshotReader};

// Create store
let store = SnapshotStore::new("./snapshots")?;

// Save snapshot with metadata
let writer = SnapshotWriter::new();
let metadata = SnapshotMetadata::new("save-001".to_string());
store.save(&snapshot, &metadata, &writer)?;

// Files created:
// - ./snapshots/save-001.tx2pack
// - ./snapshots/save-001.meta.json

// Load snapshot
let reader = SnapshotReader::new();
let (snapshot, metadata) = store.load("save-001", &reader)?;

// List all snapshots
let ids = store.list()?;

// Delete snapshot
store.delete("save-001")?;

Metadata

pub struct SnapshotMetadata {
    pub id: String,
    pub name: Option<String>,
    pub description: Option<String>,
    pub created_at: i64,              // Unix timestamp
    pub world_time: f64,              // In-game time
    pub schema_version: u32,
    pub custom_fields: HashMap<String, String>,
    pub tags: Vec<String>,
}

Usage:

let metadata = SnapshotMetadata::new("save-001".to_string())
    .with_name("Before Boss Fight")
    .with_description("Full health, all items")
    .with_tag("boss")
    .with_tag("milestone")
    .with_custom_field("level", "5")
    .with_custom_field("difficulty", "hard");

Use Cases

Game Save/Load

// Save game
let snapshot = game_world.create_snapshot();
let metadata = SnapshotMetadata::new(format!("save-{}", slot))
    .with_name(&player_name)
    .with_tag("player-save");

manager.create_checkpoint(format!("slot-{}", slot), snapshot)?;

// Load game
let checkpoint = manager.load_checkpoint(&format!("slot-{}", slot))?;
game_world.restore_from_snapshot(&checkpoint.snapshot)?;

Replay System

// Record gameplay every 5 seconds
loop {
    // ... game logic ...

    if elapsed >= 5.0 {
        let snapshot = world.create_snapshot();
        tt.record(world_time, snapshot);
        elapsed = 0.0;
    }
}

// Replay from any time
let start_time = 120.0;
if let Some(snapshot) = tt.seek_to_time(start_time) {
    world.restore_from_snapshot(snapshot)?;
    replay_mode = true;
}

Session Persistence (Agent IDEs)

// Save session on exit
let snapshot = agent_world.create_snapshot();
let metadata = SnapshotMetadata::new("last-session".to_string())
    .with_tag("auto-save")
    .with_custom_field("tab_count", &open_tabs.len().to_string());

store.save(&snapshot, &metadata, &writer)?;

// Restore session on launch
if let Ok((snapshot, _)) = store.load("last-session", &reader) {
    agent_world.restore_from_snapshot(&snapshot)?;
}

Time-Travel Debugging

// Record checkpoints during simulation
for tick in 0..1000 {
    world.update(delta_time);

    if tick % 10 == 0 {
        let snapshot = world.create_snapshot();
        tt.record(tick as f64, snapshot);
    }
}

// Jump to problematic tick
if let Some(snapshot) = tt.seek_to_time(450.0) {
    world.restore_from_snapshot(snapshot)?;
    // Inspect world state at tick 450
}

Integration with TX-2 Ecosystem

tx2-pack works with the broader TX-2 stack:

• tx2-core (Rust): Uses tx2-pack for save/load operations
• tx2-link: Shares WorldSnapshot format for network sync
• tx2-ecs (TypeScript): Can load tx2-pack snapshots via WASM

Relationship to tx2-link

tx2-pack and tx2-link both work with world snapshots but serve different purposes:

• tx2-link: Network synchronization with delta compression
• tx2-pack: Disk persistence with compression and encryption

Both use the same WorldSnapshot structure from tx2-link.

Running Tests

cargo test

All 14 tests should pass, covering:

• Write/read snapshots with checksum validation
• Compression (Zstd, LZ4, None)
• Encryption (AES-256-GCM)
• Checkpoint management
• Replay engine navigation
• Time-travel queries

Running Benchmarks

cargo build --benches  # Build without running

Benchmarks measure:

• Write performance (compression formats, entity counts)
• Read performance (decompression overhead)
• Roundtrip time (write + read)
• Compression ratios
• Encryption overhead

Error Handling

pub enum PackError {
    Io(std::io::Error),
    Serialization(String),
    Deserialization(String),
    Compression(String),
    Decompression(String),
    Encryption(String),
    Decryption(String),
    InvalidFormat(String),
    VersionMismatch { expected: String, actual: String },
    ChecksumMismatch,
    SnapshotNotFound(String),
    InvalidCheckpoint(String),
}

All operations return Result<T, PackError> for proper error handling.

Development Status

• Binary snapshot format with SoA layout
• Versioned serialization (Bincode, MessagePack)
• Compression (Zstd, LZ4, None)
• AES-256-GCM encryption
• SHA-256 checksums
• Checkpoint management
• Replay engine
• Time-travel system
• Metadata with tags
• SnapshotStore for multi-file management
• Comprehensive tests
• Benchmarks
• Incremental snapshots (only changed archetypes)
• Snapshot diffs for version control
• Streaming read/write for large worlds

Dependencies

• tx2-link - Shared snapshot format
• serde - Serialization framework
• bincode - Fast binary serialization
• rmp-serde - MessagePack format
• zstd - Zstd compression
• lz4 - LZ4 compression
• sha2 - SHA-256 checksums
• aes-gcm - AES-256-GCM encryption
• chrono - Timestamp handling
• ahash - Fast hashing

License

MIT

Contributing

Contributions are welcome! This is part of the broader TX-2 project for building isomorphic applications with a unified world model.

Learn More

• TX-2 Framework Outline
• tx2-core Native Engine
• tx2-link Protocol
• tx2-ecs TypeScript Runtime