sqlite-collections

Rust collection types backed by sqlite database files.

This provides some standard-library-like collections, which may be serialized and deserialized with arbitrary serializers. This allows you to use an interface very similar to the std::collections ones, with these characteristics:

• You can persist your collections to disk without serializing and deserializing the whole set. Opening a very large collection and making a small change is very efficient compared to just using serde to load and dump the whole thing.
• You can store very large structures, as big as your hard drive can handle, instead of just your memory. This handles many hundreds of gigabytes in the same way that plain SQLite does.
• You can use transactions and savepoints to roll back changes to any of your collections, or even many of them together.
• You can keep your collections across multiple files, with transactional integrity of them all together.

Portability and stability

ds::set::DSSet and ds::map::DSMap types depend on deterministic serialization. You aren't prevented from storing whatever is serializable in a Set, for instance, but keep in mind:

• If you store something like a HashMap, its order can change between runs.
• Some formats have multiple representations for the same data.
  • Ciborium (the cbor feature) does not support half-precision floating point yet, so it being added would break determinism in the future.
    • In this way, ciborium doesn't implement CBOR's deterministic encoding.
  • serde_json choosing to escape or not escape some unicode values in strings across versions can also break.
  • Accessing the containers from another programming language that doesn't serialize the same way will cause issues
• Some serializers will serialize differently on different architectures. If your serializer doesn't behave consistently on different endianness, it will not be portable across these different architectures.
• Some datatypes are different on different architectures, primarily usize and isize. Some serialization formats will encode these differently.

To be safe, make sure you do not update your serializers without some thorough testing. Direct is always safe. postmark is probably the most reliable serde format you can use, keeping in mind to not depend on types that use the Hash trait to determine ordering unless you can ensure a consistent order separately. cbor and json are useful for inter-language use, but keep in mind the caveats, and make sure that other languages serialize the exact same way.

In the future, a BTreeSet and BTreeMap will be added to support non- deterministic encodings. These will be slower, but will be guaranteed to match just based on Ord and Eq alone. This will not solve platform portability problems, but will solve deterministic serialization concerns.

Concurrency safety

Your collections might be open in a different thread or process through another connection. SAVEPOINT is used internally to prevent inconsistent states.

Performance

This library uses internal SAVEPOINTs to prevent inconsistent states of the database. To get the most performance without sacrificing reliability:

• Use as large a transaction as you reasonably use around all operations.
• Use an IMMEDIATE transaction when you know you will modify the database (upgrading transactions may deadlock and cause errors).
• Switching the database into journal_mode = WAL with synchronized = NORMAL can give some performance gains when there is a lot of writing.

Collections

Currently implemented

• DSSet
  • A regular set, sorted lexicographically by the stored representation.
  • Requires deterministic serialization.
  • Not completed yet. Most necessary functionality is present, but not all functionality.

To be implemented

• DSMap
  • A regular map, sorted lexicographically by the stored representation of the key.
  • Requires deterministic serialization.
• BTreeSet
  • A set allowing non-deterministic serialization, as long as Ord is implemented. Should be quite a lot slower than a normal Set, as every comparison requires a full deserialization.
• BTreeMap
  • A map allowing non-deterministic serialization.
• List
  • A sequence ordered by an integer index.
  • This is not called Vec because it's not really an array and is not accessible as a contiguous slice.
• Deque
  • A sequence ordered by an integer index, with efficient insertion and removal at both ends.

Can't I just use SQLite directly?

Yes. This is mostly to make it easy to efficiently interact with a SQLite- backed collection without having to think too hard about the SQL details, as well as making it not-too-painful to swap out an existing std::collections struct and keep the same functionality, when you need persistence and/or huge data without filling your RAM.

If you really just want a large, persistent, and/or transaction-safe set of collections and don't need any other RDBMS functionality, this library is a good choice.