openzl_sddl_derive

openzl_sddl_derive provides a procedural derive macro for generating OpenZL Simple Data Description Language (SDDL) definitions from Rust structs at compile time.

This crate is a component for integrating Rust applications with the OpenZL compression framework, enabling high-performance, structure-aware compression by describing data layouts declaratively.

Overview

OpenZL achieves superior compression ratios by parsing data into homogeneous streams before compression. SDDL is a scripting language used by OpenZL's engine to describe the structure of this data. Instead of writing these SDDL scripts manually, this crate allows you to generate them automatically from your existing Rust struct definitions.

The #[derive(OpenZlSddl)] macro inspects a struct and generates a const SDDL_DEFINITION: &'static str containing the corresponding SDDL "Record" definition. This constant can then be passed to the OpenZL compression library (via FFI) to configure a structure-aware compression graph.

Key Features

• Automatic SDDL Generation: Converts Rust struct definitions into valid SDDL Record schemas.
• Support for Primitive Types: Correctly maps Rust numeric types (u8, i16, f64, etc.) to their SDDL equivalents (e.g., UInt8, Int16LE, Float64LE).
• Endianness Control: Defaults to little-endian (LE) and allows explicit configuration with #[sddl(endian = "be")].
• Variable-Length Data: Natively handles Vec<T> and String by generating the required two-part SDDL declaration for a length-prefixed array.
• Nested Structs: Supports composition by allowing fields that are themselves structs deriving OpenZlSddl.
• Compile-Time Safety: Validates struct layouts and attributes at compile time, providing clear error messages for unsupported configurations.

Usage

Add this crate to your Cargo.toml:

[dependencies]
openzl_sddl_derive = "0.1.0" # Replace with the latest version

Basic Example

Simply add #[derive(OpenZlSddl)] to your struct.

use openzl_sddl_derive::OpenZlSddl;

#[derive(OpenZlSddl)]
struct PacketHeader {
    version: u8,
    sequence: u64,
    flags: u32,
}

fn main() {
    // The macro generates a constant SDDL_DEFINITION
    println!("{}", PacketHeader::SDDL_DEFINITION);
}

This will print the following SDDL script, which describes the PacketHeader struct's memory layout to OpenZL:

PacketHeader = {
    version : UInt8;
    sequence : UInt64LE;
    flags : UInt32LE;
};

Advanced Example with Attributes

The macro supports attributes to control serialization details.

use openzl_sddl_derive::OpenZlSddl;

#[derive(OpenZlSddl)]
struct NetworkMessage {
    // Use big-endian for this network-ordered field
    #[sddl(endian = "be")]
    message_id: u32,

    // A variable-length payload.
    // The length will be represented by a u32 instead of the default u64.
    #[sddl(len_type = "u32")]
    payload: Vec<u8>,

    // This field is for internal application logic and should not be part
    // of the compressed data structure.
    #[sddl(skip)]
    internal_state: bool,

    // A nested struct that also derives OpenZlSddl
    footer: MessageFooter,
}

#[derive(OpenZlSddl)]
struct MessageFooter {
    checksum: u16,
}

This generates the following comprehensive SDDL script:

NetworkMessage = {
    message_id : UInt32BE;
    __payload_len : UInt32LE;
    payload : UInt8[__payload_len];
    footer : MessageFooter;
};

Note: The SDDL for MessageFooter would also need to be registered with the OpenZL engine.

Supported Attributes

• #[sddl(skip)]: Excludes the field from the generated SDDL definition.
• #[sddl(endian = "le" | "be")]: Specifies the endianness for multi-byte numeric types. Defaults to "le" (little-endian).
• #[sddl(len_type = "<type>")]: For Vec<T> or String fields, specifies the integer type used to represent the length prefix. Supported types are u8, i8, u16, i16, u32, i32, u64, i64. Defaults to u64.

How It Fits into the OpenZL Ecosystem

This crate provides the first crucial step for structured compression in Rust: describing the data. The complete workflow requires two additional steps:

1. Serialization: The Rust struct instance must be serialized into a flat byte vector that precisely matches the SDDL layout. This crate can be extended with a companion macro to auto-generate a serialize() method.
2. FFI Compression: The generated SDDL string and the serialized bytes must be passed to the OpenZL C/C++ library via Foreign Function Interface (FFI) to perform the actual compression.

License

This project is licensed under the terms of the MIT license.