ptt-crawler (ptc) 

A crawler for the web version of PTT, the largest online community in Taiwan.

Yet another PTT crawler but written in Rust. Can be used as binary directly or as crate.

Table of Contents

• ptt-crawler (ptc)
• Table of Contents
  • Features
  • Getting started
    • Installation
      • From crates.io
      • From the sources
    • How to use
  • Used as crate
  • Run unit tests
  • Contributing
  • Links
  • Versioning
  • License

Created by gh-md-toc

Features

• Single executable without any dependence
• Cross platforms supported
• Crawl single article or multiple articles in one board
• Anti-Anti-Crawler with random user agent and proxy server

Getting started

Installation

The binary name for ptt-crawler is ptc . Currently, no precompiled binary is available. You need Rust 1.40 or higher and use cargo to build ptt-crawler from the sources.

From crates.io

> cargo install ptt-crawler

From the sources

> git clone https://github.com/cwouyang/ptt-crawler.git
> cd ptt-crawler
> cargo build --release

How to use

• Crawls specific article

> ptc url https://www.ptt.cc/bbs/Gossiping/M.1597463395.A.478.html

Specify flags user agent -u and proxy -p used during crawling

> ptc -u "user/agent/string" -p "https://some.proxy" url https://www.ptt.cc/bbs/Gossiping/M.1597463395.A.478.html

# pass "random" to use randomly generated user agent
> ptc -u "random" https://www.ptt.cc/bbs/Gossiping/M.1597463395.A.478.html

• Crawls articles of board within page range

# From page 100 (https://www.ptt.cc/bbs/Gossiping/index100.html) to 200 (https://www.ptt.cc/bbs/Gossiping/index200.html)
> ptc board Gossiping -r 100 200

# From page 1 to latest page
> ptc board Gossiping

Use -l flag to list supported boards

> ptc board Gossiping --list

Used as crate

Add ptt-crawler as dependence in Cargo.toml file

[dependencies]
ptt-crawler = "0.1"

See document for usages.

Run unit tests

> cargo test --all

Contributing

If you'd like to contribute, please fork the repository and use a feature branch. Pull requests are warmly welcome.

Before submit pull request, make sure

• clippy was applied
• Commit messages with Conventional Commits (See here for detailed format)

Links

• Project homepage: https://github.com/cwouyang/ptt-crawler/
• Issue tracker: https://github.com/cwouyang/ptt-crawler/issues
• Related projects

  • ptt-web-crawler: PTT crawler in Python version

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

License

Copyright (c) 2020 cwouyang.

This project is licensed under the terms of MIT License. See the LICENSE file for details.