wral

Crates.iowral
lib.rswral
version0.2.0
sourcesrc
created_at2020-12-05 08:21:01.154319
updated_at2021-03-12 11:13:58.56799
descriptionWrite ahead logging for durability
homepage
repositoryhttps://github.com/bnclabs/wral
max_upload_size
id319819
size73,367
(prataprc)

documentation

https://docs.rs/wral/

README

Documentation

Write ahead logging for rust applications. Write ahead logging is a crucial component for applications requiring data durability. Many times it is inefficient to flush and sync new data (or modifications to existing data) to on-disk data-structures, like an index. Write-ahead-logging facilitates by ingesting write operations by appending and syncing it to disk and allows applications to pre-process a batch of write-operations and write them to on-disk structures in the most efficient manner.

Goals

  • Serialize write operations to an append only journal file.
  • Generate monotonically increasing sequence-number and return the same to the application.
  • Configurable limit for journal file, beyond which log files are rotate.
  • Configurable fsync, for persistence guarantee.
  • Iterate over all entries persisted in the log file in monotonically increasing seqno order.
  • Range over a subset of entries, specified with start-seqno and end-seqno.
  • Wal type is parameterized over a state type S. This is helpful for using Wal type to be used with consensus protocol like Raft.
  • Concurrent readers and writers into single log instance.

Concurrency

A single log-instance can be cloned and shared among multiple threads for concurrent writing and reading. All write operations are serialized. While read operations and write-operation are mutually exclusive, concurrent reads are allowed.

Performance

Single threaded write performance with different payload size and fsync enabled.

payload total-entries elapsed-time throughput
100 10000 31s 300/s
1000 10000 31s 300/s
10000 10000 31s 300/s
10000 1000 3.1s 300/s

Multi-threaded write performance with constant payload size of 100-bytes per operation and fsync enabled.

threads total-entries elapsed-time throughput
1 10000 31s 300/s
2 20000 60s 300/s
4 40000 59s 650/s
8 80000 54s 1300/s
16 160000 50s 3200/s

Multi-threaded read performance with constant payload size of 100-bytes per operation and fsync enabled.

threads total-entries elapsed-time throughput
1 10000 .15s 66000/s
2 20000 .28s 71000/s
4 40000 .38s 105000/s
8 80000 .62s 130000/s
16 160000 1.10s 150000/s

Contribution

  • Simple workflow. Fork - Modify - Pull request.
  • Before creating a PR,
    • Run make build to confirm all versions of build is passing with 0 warnings and 0 errors.
    • Run check.sh with 0 warnings, 0 errors and all testcases passing.
    • Run perf.sh with 0 warnings, 0 errors and all testcases passing.
    • Install and run cargo spellcheck to remove common spelling mistakes.
  • Developer certificate of origin is preferred.
Commit count: 47

cargo fmt