spider_worker

Crates.iospider_worker
lib.rsspider_worker
version1.98.1
sourcesrc
created_at2023-03-10 21:26:36.958866
updated_at2024-06-25 11:07:17.135934
descriptionThe fastest web crawler as a worker or proxy.
homepage
repositoryhttps://github.com/spider-rs/spider
max_upload_size
id806750
size93,150
Jeff Mendez (j-mendez)

documentation

https://docs.rs/spider

README

Spider Worker

crate version

A spider worker to decentralize the crawl lifting.

Dependencies

This project depends on the spider crate.

Usage

The worker starts on port 3030 and the scraper for html gathering on 3031 by default.

SPIDER_WORKER_PORT=3030 SPIDER_WORKER_SCRAPER_PORT=3031 cargo run

Feature Flags

  1. scrape - When the html is needed run the instance with the flag. Requires spider feature flag matching on the client to start. This also starts the instance on port 3031 instead.
  2. full_resources - Start the basic worker to gather links and scraper together.
  3. tls - Enable tls support use the env variables SPIDER_WORKER_CERT_PATH for the .pem file and SPIDER_WORKER_KEY_PATH with your .rsa file. Defaults to /cert.pem and /key.rsa.

Ports

By default the instance runs on port 3030 use SPIDER_WORKER_PORT to adjust the port. The scraper runs on port 3031 when enabled use SPIDER_WORKER_SCRAPER_PORT to adjust the port.

Commit count: 683

cargo fmt