redditrip

Crates.ioredditrip
lib.rsredditrip
version0.2.1
sourcesrc
created_at2020-04-21 11:38:08.162779
updated_at2020-05-20 16:43:36.507128
descriptionA versatile tool for downloading the linked contents of entire subreddits fast and efficiently
homepage
repositoryhttps://github.com/Draphar/redditrip/
max_upload_size
id232529
size126,818
(Draphar)

documentation

README

redditrip

crates.io-badge

A versatile tool for downloading the linked contents of entire subreddits fast and efficiently.

Note that this is not the tool for the job if you want to retrieve information about the individual reddit posts.

Features

  • bypass reddit API limitation of 1000 posts

  • update local copy

  • banned subreddits work

  • fast (because of an asynchronous job queue, powered by the fantastic tokio)

  • custom title formatting

  • (for the Rust programmers) #![forbid(unsafe_code)]

Installation

Prebuilt binaries can be found in the Releases tab.

If you want a locally compiled version, you can use cargo:

  1. Install Rust according to https://rustup.rs. The default configuration is sufficient.

  2. $ cargo install redditrip

Usage

The usage can also be acquired by running redditrip --help.

The base command is redditrip [SUBREDDITS]....
The following arguments can optionally be used:

  • --update/-u: Stop at the first already existing file.

  • --force/-f: Force downloads from unsupported domains by simpling writing whatever is on the page to disk.

  • --after <date>: Only download posts after this date.

  • --before <date>: Only download posts before this date.

  • --selfposts/-s: Download self posts as text files.

  • --exclude <domain>/-e <domain>: Prevents downloading from a domain.

  • --title <formatter>: Use a custom title format.

There are a couple of more advanced options described in the --help output.

Downloading large amounts of data

It is recommended to use -q/--quiet to see only the individual errors. You should also use a high --batch-size, and subsequently a high ulimit -n (open files) because every download job takes >= 1 open file descriptor. Finally, if you expect to run into a lot of unsupported sites, which can directly be saved, use --force.

Title formatting

redditrip supports custom titles. To use this feature, a formatting string must be provided with --title <formatter>. Placeholders are the field names enclosed in curly brackets. For example: --title "{id}-{author}_{title}". The file extension is always appended to the title.

The available fields can be queried by running the program with --formatting-fields. They correspond to data from the Pushshift API. An rough overview can be seen on https://api.pushshift.io/reddit/search/submission?size=1, though not all fields can be present.

Characters which are not allowed in file names are replaced with _.

Because reddit titles alone can be longer than the maximum file name length on many systems, one should know about --max-file-name-length <length>, which is used to truncate file names.

Not using the {id} placeholder can lead to file name collisions, thus it is advised to always include it somewhere in the formatter, preferably at the front so it is not at the risk of being truncated.

The most useful placeholders are:

Placeholder Type Purpose
{id} string The post ID
{title} string The post title
{author} string The post author name
{created_utc} integer The UNIX timestamp when the post was created
{link_flair_text} string The text of the post flair
{author_flair_text} string The text of the author flair
{domain} string The domain of the link the post points to
{over_18} bool Whether the post is NSFW

Compiling

It's quite easy to compile the master branch yourself. Make sure that Rust and cargo are installed like described above. Then:

$ git clone https://github.com/Draphar/redditrip
$ cd redditrip

Then this is sufficient:

$ cargo build --release
  # The binary is `target/release/redditrip`

If you, for whatever reason, want a highly optimized build:

$ cargo rustc --release -- -C lto -C codegen-units=1 -C target_cpu=native
$ strip target/release/redditrip

Todo

  • implement a map of already downloaded links, and symlink instead of redownloading

  • filter posts

License

Copyright 2020 Joshua Prieth

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Commit count: 23

cargo fmt