# oscar-io
Types and IO (Reader/Writer) for OSCAR Corpus processing and generation.

The crate provides basic abstractions around Corpus items and generic readers/writers useable in OSCAR Corpus files.
At some time, it should replace reader implementations in both [Ungoliant](https://github.com/oscar-corpus/Ungoliant) and [oscar-tools](https://github.com/oscar-corpus/oscar-tools).


## Features

`oscar-io` aims to provide readers/writers for numerous types of OSCAR Corpora.


### OSCAR v2
- Reader 
    - [x] Uncompressed [oscar_doc::Reader::new]
    - [x] GZipped [oscar_doc::Reader::from_gzip]
    - [ ] Parquet
- Writer
    - [x] Uncompressed [oscar_doc::Writer::new]
    - [ ] GZipped [oscar_doc::Writer::new] (using a [GzEncoder] reader, `from_gzip` not yet implemented)
    - [ ] Parquet
- SplitReader (Should be unified with SplitReader with `split_size: Option<u64>`)
    - [x] Uncompressed
    - [ ] GZipped
- SplitWriter (Same)
    - [ ] Uncompressed
    - [ ] GZipped

### OSCAR v1.1
- [ ] Reader
- [ ] Writer
- [ ] SplitReader (Should be unified with SplitReader with `split_size: Option<u64>`)
- [ ] SplitWriter (Same)

### OSCAR v1
- [ ] Reader
- [ ] Writer
- [ ] SplitReader
- [ ] SplitWriter