maman

Crates.io	maman
lib.rs	maman
version	0.13.1
source	src
created_at	2016-05-03 20:13:45.227109
updated_at	2018-12-01 16:05:45.084095
description	Rust Web Crawler
homepage	https://github.com/spk/maman
repository	https://github.com/spk/maman.git
max_upload_size
id	4964
size	28,132

Laurent Arnoud (spk)

documentation

README

Maman

Maman is a Rust Web Crawler saving pages on Redis.

Pages are send to list <MAMAN_ENV>:queue:maman using Sidekiq job format

{
"class": "Maman",
"jid": "b4a577edbccf1d805744efa9",
"retry": true,
"created_at": 1461789979, "enqueued_at": 1461789979,
"args": {
    "document":"<html><body><a href='#' /><a href='/new' /></html>",
    "urls": ["https://example.net/new"],
    "headers": {"content-type": "text/html"},
    "url": "https://example.net/"
    }
}