Web Scraping Library

Efficient and scalable web scraping for Rust applications.

Introduction

This library allows users to perform recursive web scraping, media downloading, and content extraction from web pages with minimal configuration. The library supports HTML parsing, media extraction, and error logging.

Installation

To add this library to your project, include the following in your Cargo.toml:

[dependencies]
web_scraper = "0.1.0"

Usage

Below is a minimal example for scraping a website:

use web_scraper::{Client, recursive_scrape};
use std::collections::HashSet;

#[tokio::main]
async fn main() {
    let client = Client::new();
    let mut visited = HashSet::new();
    recursive_scrape("https://example.com", &client, &mut visited).await;
}