nautilus-serialization

Data serialization and format conversion for NautilusTrader.

The nautilus-serialization crate provides comprehensive data serialization capabilities for converting trading data between different formats including Apache Arrow, Parquet, and Cap'n Proto. This enables efficient data storage, retrieval, and interoperability across different systems:

• Apache Arrow integration: Schema definitions and encoding/decoding for market data types.
• Parquet file operations: High-performance columnar storage for historical data analysis.
• Record batch processing: Efficient batch operations for time-series data.
• Schema management: Type-safe schema definitions with metadata preservation.
• Cross-format conversion: Seamless data interchange between Arrow, Parquet, and native types.
• Cap'n Proto serialization: Zero-copy, schema-based serialization for efficient data interchange.

Platform

NautilusTrader is an open-source, high-performance, production-grade algorithmic trading platform, providing quantitative traders with the ability to backtest portfolios of automated trading strategies on historical data with an event-driven engine, and also deploy those same strategies live, with no code changes.

NautilusTrader's design, architecture, and implementation philosophy prioritizes software correctness and safety at the highest level, with the aim of supporting mission-critical, trading system backtesting and live deployment workloads.

Feature flags

This crate provides feature flags to control source code inclusion during compilation, depending on the intended use case, i.e. whether to provide Python bindings for the nautilus_trader Python package, or as part of a Rust only build.

• python: Enables Python bindings from PyO3.
• extension-module: Builds as a Python extension module (used with python).
• high-precision: Enables high-precision mode to use 128-bit value types.
• capnp: Enables Cap'n Proto serialization support.

Building with Cap'n Proto support

To build with Cap'n Proto serialization enabled:

cargo build -p nautilus-serialization --features capnp

The Cap'n Proto compiler is required. See the Environment Setup guide for installation instructions. The required version is specified in the capnp-version file in the repository root.

Cap'n Proto schemas

When the capnp feature is enabled, this crate provides zero-copy serialization using Cap'n Proto schemas.

Schema location

Cap'n Proto schemas are bundled with the crate in schemas/capnp/:

• common/identifiers.capnp - Identifier types (TraderId, InstrumentId, etc.)
• common/types.capnp - Value types (Price, Quantity, Money, etc.)
• common/enums.capnp - Trading enumerations
• commands/trading.capnp - Trading commands
• commands/data.capnp - Data subscription/request commands
• events/order.capnp - Order events
• events/position.capnp - Position events
• events/account.capnp - Account events
• data/market.capnp - Market data types (quotes, trades, bars, order books)

Generated modules

During build, schemas are compiled to Rust code and made available as:

• nautilus_serialization::identifiers_capnp
• nautilus_serialization::types_capnp
• nautilus_serialization::enums_capnp
• nautilus_serialization::trading_capnp
• nautilus_serialization::data_capnp
• nautilus_serialization::order_capnp
• nautilus_serialization::position_capnp
• nautilus_serialization::account_capnp
• nautilus_serialization::market_capnp

Usage example

use nautilus_model::types::Price;
use nautilus_serialization::capnp::{ToCapnp, FromCapnp};

// Serialize a Price
let price = Price::from("123.45");
let bytes = nautilus_serialization::capnp::conversions::serialize_price(&price).unwrap();

// Deserialize back
let decoded = nautilus_serialization::capnp::conversions::deserialize_price(&bytes).unwrap();
assert_eq!(price, decoded);

See the conversions module for trait-based serialization patterns:

use nautilus_model::identifiers::InstrumentId;
use nautilus_serialization::capnp::{ToCapnp, FromCapnp, identifiers_capnp};

let instrument_id = InstrumentId::from("AAPL.NASDAQ");

// Using traits
let mut message = capnp::message::Builder::new_default();
let builder = message.init_root::<identifiers_capnp::instrument_id::Builder>();
instrument_id.to_capnp(builder);

// Serialize to bytes
let mut bytes = Vec::new();
capnp::serialize::write_message(&mut bytes, &message).unwrap();

// Deserialize
let reader = capnp::serialize::read_message(
    &mut &bytes[..],
    capnp::message::ReaderOptions::new()
).unwrap();
let root = reader.get_root::<identifiers_capnp::instrument_id::Reader>().unwrap();
let decoded = InstrumentId::from_capnp(root).unwrap();

Contributing schemas

When adding or modifying schemas:

1. Edit schema files in the appropriate subdirectory under schemas/capnp/.
2. Use lowerCamelCase for field names to match Cap'n Proto conventions.
3. Generate a unique schema ID using: capnp id.
4. Implement ToCapnp and FromCapnp traits in src/capnp/conversions.rs.
5. Add integration tests in tests/ to verify roundtrip serialization.

The build script (build.rs) automatically discovers and compiles all .capnp files during build.

Serialization format comparison

This crate supports three serialization formats for market data types. Choose the format based on your use case:

Performance numbers shown for QuoteTick serialization (measured on AMD Ryzen 9 7950X). Cap'n Proto provides the fastest serialization and deserialization, while MsgPack offers the smallest size. Arrow is optimized for batch processing rather than individual messages.

Note: Cap'n Proto performance can be further optimized through zero-copy techniques and direct buffer manipulation for specialized use cases.

Usage examples

JSON serialization

use nautilus_core::serialization::Serializable;
use nautilus_model::data::QuoteTick;

let quote = QuoteTick { /* ... */ };

// Serialize to JSON
let json_bytes = quote.to_json_bytes()?;

// Deserialize from JSON
let decoded = QuoteTick::from_json_bytes(&json_bytes)?;

MsgPack serialization

use nautilus_core::serialization::{ToMsgPack, FromMsgPack};
use nautilus_model::data::QuoteTick;

let quote = QuoteTick { /* ... */ };

// Serialize to MsgPack
let msgpack_bytes = quote.to_msgpack_bytes()?;

// Deserialize from MsgPack
let decoded = QuoteTick::from_msgpack_bytes(&msgpack_bytes)?;

Cap'n Proto serialization

use nautilus_model::data::QuoteTick;
use nautilus_serialization::capnp::{ToCapnp, FromCapnp, market_capnp};

let quote = QuoteTick { /* ... */ };

// Serialize to Cap'n Proto
let mut message = capnp::message::Builder::new_default();
let builder = message.init_root::<market_capnp::quote_tick::Builder>();
quote.to_capnp(builder);

let mut bytes = Vec::new();
capnp::serialize::write_message(&mut bytes, &message)?;

// Deserialize from Cap'n Proto
let reader = capnp::serialize::read_message(
    &mut &bytes[..],
    capnp::message::ReaderOptions::new()
)?;
let root = reader.get_root::<market_capnp::quote_tick::Reader>()?;
let decoded = QuoteTick::from_capnp(root)?;

Benchmarking

Run benchmarks to compare serialization performance across formats:

# Compare all formats for QuoteTick
cargo bench -p nautilus-serialization --features capnp --bench serialization_comparison -- QuoteTick

# Compare all formats for TradeTick
cargo bench -p nautilus-serialization --features capnp --bench serialization_comparison -- TradeTick

# Compare all formats for Bar
cargo bench -p nautilus-serialization --features capnp --bench serialization_comparison -- Bar

# Run all Cap'n Proto benchmarks (including OrderBookDeltas with varying sizes)
cargo bench -p nautilus-serialization --features capnp --bench capnp_serialization

# Run all comparison benchmarks
cargo bench -p nautilus-serialization --features capnp --bench serialization_comparison

Benchmark results include serialization and deserialization times for each format.

Documentation

See the docs for more detailed usage.

License

The source code for NautilusTrader is available on GitHub under the GNU Lesser General Public License v3.0. Contributions to the project are welcome and require the completion of a standard Contributor License Agreement (CLA).

NautilusTrader™ is developed and maintained by Nautech Systems, a technology company specializing in the development of high-performance trading systems. For more information, visit https://nautilustrader.io.

© 2015-2026 Nautech Systems Pty Ltd. All rights reserved.