redditrip

A versatile tool for downloading the linked contents of entire subreddits fast and efficiently.

Note that this is not the tool for the job if you want to retrieve information about the individual reddit posts.

Features

• bypass reddit API limitation of 1000 posts

• update local copy

• banned subreddits work

• fast (because of an asynchronous job queue, powered by the fantastic tokio)

• custom title formatting

• (for the Rust programmers) #![forbid(unsafe_code)]

Installation

Prebuilt binaries can be found in the Releases tab.

If you want a locally compiled version, you can use cargo:

1. Install Rust according to https://rustup.rs. The default configuration is sufficient.

2. $ cargo install redditrip

Usage

The usage can also be acquired by running redditrip --help.

The base command is redditrip [SUBREDDITS]....
The following arguments can optionally be used:

• --update/-u: Stop at the first already existing file.

• --force/-f: Force downloads from unsupported domains by simpling writing whatever is on the page to disk.

• --after <date>: Only download posts after this date.

• --before <date>: Only download posts before this date.

• --selfposts/-s: Download self posts as text files.

• --exclude <domain>/-e <domain>: Prevents downloading from a domain.

• --title <formatter>: Use a custom title format.

There are a couple of more advanced options described in the --help output.

Downloading large amounts of data

It is recommended to use -q/--quiet to see only the individual errors. You should also use a high --batch-size, and subsequently a high ulimit -n (open files) because every download job takes >= 1 open file descriptor. Finally, if you expect to run into a lot of unsupported sites, which can directly be saved, use --force.

Title formatting

redditrip supports custom titles. To use this feature, a formatting string must be provided with --title <formatter>. Placeholders are the field names enclosed in curly brackets. For example: --title "{id}-{author}_{title}". The file extension is always appended to the title.

The available fields can be queried by running the program with --formatting-fields. They correspond to data from the Pushshift API. An rough overview can be seen on https://api.pushshift.io/reddit/search/submission?size=1, though not all fields can be present.

Characters which are not allowed in file names are replaced with _.

Because reddit titles alone can be longer than the maximum file name length on many systems, one should know about --max-file-name-length <length>, which is used to truncate file names.

Not using the {id} placeholder can lead to file name collisions, thus it is advised to always include it somewhere in the formatter, preferably at the front so it is not at the risk of being truncated.

The most useful placeholders are:

Compiling

It's quite easy to compile the master branch yourself. Make sure that Rust and cargo are installed like described above. Then:

$ git clone https://github.com/Draphar/redditrip
$ cd redditrip

Then this is sufficient:

$ cargo build --release
  # The binary is `target/release/redditrip`

If you, for whatever reason, want a highly optimized build:

$ cargo rustc --release -- -C lto -C codegen-units=1 -C target_cpu=native
$ strip target/release/redditrip

Todo

• implement a map of already downloaded links, and symlink instead of redownloading

• filter posts

License

Copyright 2020 Joshua Prieth

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.