# Product OS : Crawler

Product OS : Crawler is a browser based cralwer that utilises Product OS : Browser to perform advanced url crawling leveraging headless browsing and automation.

### What is Product OS?

Product OS is a collection of packages that provide different tools and features that can work together to build products more easily for the Rust ecosystem.

## Installation

Use the Rust crate package manager [cargo](https://www.rust-lang.org/tools/install) to install Product OS : Crawler.

```bash
cargo add product-os-crawler
```

or add Product OS : Crawler to your cargo.toml ```[packages]``` section.

```rust
product-os-crawler = { version = "0.0.11", features = [], default-features = true, optional = false }
```

## Features

Product OS Crawler supports a number of features leveraging existing Rust libraries to crawl and perform instructions including:

- Basic crawling capabilities with back-off rules
- Revisit logic and full configuration for tuning
- Scoring system fully configurable to determine value of content
- Ability to hand-off to indexing and storage services using a processor

```rust
// Feature samples TODO
```

## Usage

```rust
// Examples TODO
```

## Contributing

Contributions are not currently available but will be available on a public repository soon.

## License

[GNU AGPLv3](https://choosealicense.com/licenses/agpl-3.0/)