📦 lencode

Compact, fast binary encoding with varints, optional deduplication, and opportunistic zstd compression for bytes and strings. no_std by default, with an opt‑in std feature.

Highlights

• Fast varints: efficient for small and large integers
• Optional deduplication: replace repeats with compact IDs for supported types
• Bytes/strings compression: flagged header + zstd when smaller
• no_std + alloc: works without std (uses zstd-safe)
• Derive macros: #[derive(Encode, Decode)] for your types
• Solana support: feature solana adds v2/v3 SDK types
• Big-endian ready: CI runs tests on s390x

Install

[dependencies]
lencode = "0.1"

# With standard library types (e.g., Cow)
lencode = { version = "0.1", features = ["std"] }

# With Solana type support (implies std)
lencode = { version = "0.1", features = ["solana"] }

Quick start

Derive and round‑trip

use lencode::prelude::*;

#[derive(Encode, Decode, PartialEq, Debug)]
struct Point { x: u64, y: u64 }

let p = Point { x: 3, y: 5 };
let mut buf = Vec::new();
encode(&p, &mut buf)?;
let q: Point = decode(&mut Cursor::new(&buf))?;
assert_eq!(p, q);

Collections and primitives

use lencode::prelude::*;

let values: Vec<u128> = (0..10).collect();
let mut buf = Vec::new();
encode(&values, &mut buf)?;
let rt: Vec<u128> = decode(&mut Cursor::new(&buf))?;
assert_eq!(values, rt);

Deduplication (optional)

To benefit from deduplication for your own types, implement Pack and the marker traits and pass the encoder/decoder via encode_ext/decode_ext.

use lencode::prelude::*;

#[derive(Clone, Copy, PartialEq, Eq, Debug, Hash)]
struct MyId(u32);

impl Pack for MyId {
    fn pack(&self, w: &mut impl Write) -> Result<usize> { self.0.pack(w) }
    fn unpack(r: &mut impl Read) -> Result<Self> { Ok(Self(u32::unpack(r)?)) }
}
impl DedupeEncodeable for MyId {}
impl DedupeDecodeable for MyId {}

let vals = vec![MyId(42), MyId(7), MyId(42), MyId(7), MyId(42)];

// Encode with deduplication enabled
let mut enc = DedupeEncoder::new();
let mut buf = Vec::new();
encode_ext(&vals, &mut buf, Some(&mut enc))?;

// Decode with deduplication enabled
let mut dec = DedupeDecoder::new();
let roundtrip: Vec<MyId> = decode_ext(&mut Cursor::new(&buf), Some(&mut dec))?;
assert_eq!(roundtrip, vals);

Compact bytes and strings

&[u8], Vec<u8], VecDeque<u8], &str, and String use a compact flagged header: varint((payload_len << 1) | flag) + payload.

• flag = 0 → raw bytes/UTF‑8
• flag = 1 → zstd frame (original size stored inside the frame)

The encoder picks whichever is smaller per value.

Supported types

• Primitives: all ints, bool, f32, f64
• Arrays: [T; N]
• Option: Option<T>
• Bytes/strings: &[u8], Vec<u8], VecDeque<u8], &str, String
• Collections (alloc): Vec<T>, BTreeMap<K,V>, BTreeSet<V>, VecDeque<T>, LinkedList<T>, BinaryHeap<T>
• Tuples: (T1,) … up to 11 elements
• std feature: adds support for std::borrow::Cow<'_, T>
• solana feature: Pubkey, Signature, Hash, messages (legacy/v0), and related v2/v3 types

Note: HashMap/HashSet are not implemented.

Cargo features

• default: core + no_std (uses alloc)
• std: enables std adapters and Cow
• solana: Solana SDK v2 + Agave v3 types (implies std)

Big‑endian and portability

• Varints are decoded efficiently on little‑endian and portably on big‑endian
• CI runs tests on s390x-unknown-linux-gnu using cross
• Pack always uses a stable little‑endian layout

Benchmarks

# Full suite
cargo bench --all-features

# Compare against borsh/bincode
cargo bench --bench roundup --features std

# Solana‑specific
cargo bench --bench solana_bench --features solana

Errors

Errors use lencode::io::Error and map to std::io::Error under std.

use lencode::prelude::*;
use lencode::io::Error;

let mut buf = Vec::new();
match encode(&123u64, &mut buf) {
    Ok(n) => eprintln!("wrote {n} bytes"),
    Err(Error::WriterOutOfSpace) => eprintln!("buffer too small"),
    Err(Error::ReaderOutOfData) => eprintln!("unexpected EOF"),
    Err(Error::InvalidData) => eprintln!("corrupted data"),
    Err(e) => eprintln!("other error: {e}"),
}

Examples

• examples/size_comparison.rs: space savings on repeated Solana pubkeys
• examples/versioned_tx_compression.rs: end‑to‑end on Solana versioned transactions

Run with --features solana.

License

MIT

Changelog

0.1.0

• Lencode varints for signed/unsigned ints
• Optional deduplication via DedupeEncoder/DedupeDecoder
• Bytes/strings with flagged header + zstd
• no_std by default; std and solana features
• Derive macros for Encode/Decode
• Big‑endian test coverage