oscar-io
| Crates.io | oscar-io |
| lib.rs | oscar-io |
| version | 0.4.0 |
| source | src |
| created_at | 2022-05-13 09:08:59.665991 |
| updated_at | 2023-08-08 10:08:21.152194 |
| description | Readers/Writers for OSCAR Corpora. |
| homepage | https://oscar-corpus.com |
| repository | https://github.com/oscar-corpus/oscar-io |
| max_upload_size | |
| id | 585723 |
| size | 3,756,114 |
Julien "uj" Abadji (Uinelj)
documentation
https://docs.rs/oscar-io
README
oscar-io
Types and IO (Reader/Writer) for OSCAR Corpus processing and generation.
The crate provides basic abstractions around Corpus items and generic readers/writers useable in OSCAR Corpus files. At some time, it should replace reader implementations in both Ungoliant and oscar-tools.
Features
oscar-io aims to provide readers/writers for numerous types of OSCAR Corpora.
OSCAR v2
- Reader
- Uncompressed [oscar_doc::Reader::new]
- GZipped [oscar_doc::Reader::from_gzip]
- Parquet
- Writer
- Uncompressed [oscar_doc::Writer::new]
- GZipped [oscar_doc::Writer::new] (using a [GzEncoder] reader,
from_gzipnot yet implemented) - Parquet
- SplitReader (Should be unified with SplitReader with
split_size: Option<u64>)- Uncompressed
- GZipped
- SplitWriter (Same)
- Uncompressed
- GZipped
OSCAR v1.1
- Reader
- Writer
- SplitReader (Should be unified with SplitReader with
split_size: Option<u64>) - SplitWriter (Same)
OSCAR v1
- Reader
- Writer
- SplitReader
- SplitWriter