product-os-crawler
| Crates.io | product-os-crawler |
| lib.rs | product-os-crawler |
| version | 0.0.11 |
| source | src |
| created_at | 2023-08-20 12:29:59.902332 |
| updated_at | 2024-06-12 12:26:11.171114 |
| description | Product OS : Crawler is a browser based cralwer that utilises Product OS : Browser to perform advanced url crawling leveraging headless browsing and automation. |
| homepage | |
| repository | |
| max_upload_size | |
| id | 949357 |
| size | 96,491 |
Jeremy (eremyjay)
documentation
README
Product OS : Crawler
Product OS : Crawler is a browser based cralwer that utilises Product OS : Browser to perform advanced url crawling leveraging headless browsing and automation.
What is Product OS?
Product OS is a collection of packages that provide different tools and features that can work together to build products more easily for the Rust ecosystem.
Installation
Use the Rust crate package manager cargo to install Product OS : Crawler.
cargo add product-os-crawler
or add Product OS : Crawler to your cargo.toml [packages] section.
product-os-crawler = { version = "0.0.11", features = [], default-features = true, optional = false }
Features
Product OS Crawler supports a number of features leveraging existing Rust libraries to crawl and perform instructions including:
- Basic crawling capabilities with back-off rules
- Revisit logic and full configuration for tuning
- Scoring system fully configurable to determine value of content
- Ability to hand-off to indexing and storage services using a processor
// Feature samples TODO
Usage
// Examples TODO
Contributing
Contributions are not currently available but will be available on a public repository soon.