wral

Crates.io	wral
lib.rs	wral
version	0.2.0
source	src
created_at	2020-12-05 08:21:01.154319
updated_at	2021-03-12 11:13:58.56799
description	Write ahead logging for durability
homepage
repository	https://github.com/bnclabs/wral
max_upload_size
id	319819
size	73,367

(prataprc)

documentation

https://docs.rs/wral/

README

Write ahead logging for rust applications. Write ahead logging is a crucial component for applications requiring data durability. Many times it is inefficient to flush and sync new data (or modifications to existing data) to on-disk data-structures, like an index. Write-ahead-logging facilitates by ingesting write operations by appending and syncing it to disk and allows applications to pre-process a batch of write-operations and write them to on-disk structures in the most efficient manner.

Goals

Serialize write operations to an append only journal file.
Generate monotonically increasing sequence-number and return the same to the application.
Configurable limit for journal file, beyond which log files are rotate.
Configurable fsync, for persistence guarantee.
Iterate over all entries persisted in the log file in monotonically increasing seqno order.
Range over a subset of entries, specified with start-seqno and end-seqno.
Wal type is parameterized over a state type S. This is helpful for using Wal type to be used with consensus protocol like Raft.
Concurrent readers and writers into single log instance.

Concurrency

A single log-instance can be cloned and shared among multiple threads for concurrent writing and reading. All write operations are serialized. While read operations and write-operation are mutually exclusive, concurrent reads are allowed.

Performance

Single threaded write performance with different payload size and fsync enabled.

payload	total-entries	elapsed-time	throughput
100	10000	31s	300/s
1000	10000	31s	300/s
10000	10000	31s	300/s
10000	1000	3.1s	300/s

Multi-threaded write performance with constant payload size of 100-bytes per operation and fsync enabled.

threads	total-entries	elapsed-time	throughput
1	10000	31s	300/s
2	20000	60s	300/s
4	40000	59s	650/s
8	80000	54s	1300/s
16	160000	50s	3200/s

Multi-threaded read performance with constant payload size of 100-bytes per operation and fsync enabled.

threads	total-entries	elapsed-time	throughput
1	10000	.15s	66000/s
2	20000	.28s	71000/s
4	40000	.38s	105000/s
8	80000	.62s	130000/s
16	160000	1.10s	150000/s

Contribution

Simple workflow. Fork - Modify - Pull request.
Before creating a PR,
- Run make build to confirm all versions of build is passing with 0 warnings and 0 errors.
- Run check.sh with 0 warnings, 0 errors and all testcases passing.
- Run perf.sh with 0 warnings, 0 errors and all testcases passing.
- Install and run cargo spellcheck to remove common spelling mistakes.
Developer certificate of origin is preferred.