rhuffle

rhuffle is a random shuffler for large file with many lines which can exceed available RAM.

rhuffle supports:

• shuffling huge files which does not fit in memory
• skipping head lines which should not include for shuffling (e.g. csv/tsv)
• multiple file input and flexible input formats
• rhuffle works very fast (see benchmark results.)

Installation

See lib.rs.

Usage

USAGE:
    rhuffle [OPTIONS]

FLAGS:
        --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -b, --buf <NUMBER>
            Sets buffer size which is smaller than available RAM with bytes (default: 4294967296).

        --dst <PATH>
            Sets destination file path. If not set, destination sets to stdout. (default: None)

        --feed <LF|LF_CRLF>                        Sets acceptable line feed as EOL (default: LF_CRLF).
    -h, --head <NUMBER>
            Sets first `n` lines without shuffling (default: 0). For multiple input sources, take README a look.

        --log <off|error|warn|info|debug|trace>    Sets log level. (default: off)
        --src <[PATH]>
            Sets source file paths (space separated). If not set, source sets to stdin. (default: None)

--head n Option

• For multiple input sources, first n lines in the first input source forwards to output source without shuffling.
• For second input source and later, first n lines in the first input source are skipped.
• Here is an example below:

in1.txt

head1-1
head2-1
line1-1
line2-1

in2.txt

head1-2
head2-2
line1-2
line2-2

$ rhuffle --src in1.txt in2.txt --dst out.txt --head 2

out.txt

head1-1 // L1-L2: fixed
head2-1 
line2-1 // L3-L6: shuffled globally
line1-2
line2-2
line1-1

--feed Option

• LF_CRLF(default): accepts LF or CRLF as newline
• LF: accepts only LF as newline
• No option for CR

Benchmarks

The results shown below are focused on execution time in a limited memory space. Two datasets are used for testing.

• Kaggle competition dataset (New York City Taxi Fare Prediction)
• (self-owned) custom dataset

Three softwares are used for performance comparison.

• GNU shuf
  • command: shuf {src} -o {dst}
• terashuf
  • command: terashuf < {src} > {dst}
• rhuffle
  • command: rhuffle --src {src} --dst {dst}

Benchmarks are executed on MacBook Pro 2017, Core i7 3.1GHz, RAM 16GB. Execution time is measured by time.

Kaggle competition dataset

5.3GB size, 55423856 lines

Custom dataset

9.0GB size, 21550072 lines

GNU shuf was impossible to measure (very slow).