# Title?

The Nix daemon uses a custom binary protocol -- the *nix protocol* -- to communicate with just about
everything. When you run `nix build` on your machine, the Nix binary opens up a Unix socket to the Nix
daemon and talks to it using the nix protocol. When you administer a nix server remotely using
`nix store --store ssh://example.com [...]`,
the Nix binary opens up an ssh connection to a remote machine and tunnels the nix protocol over ssh.
When you use remote builders to speed up your Nix builds, the various Nix daemons speak the nix protocol
to one another.

Despite its importance in the Nix world, the Nix protocol has no specification or reference documentation.
Besides the original implementation in the Nix project itself, the [hnix-store] project contains
a reimplementation of the client end of the protocol. We do not know of any other implementations. (The
Tvix project has its own [gRPC-based](https://cs.tvl.fyi/depot@842d5ed3ee68f819742cbd3a81dac8f3410161dc/-/blob/tvix/store/docs/api.md)
protocol instead of re-implementing a Nix-compatible one.)

So we wrote another one, in rust. Unlike the `hnix-store` re-implementation, we've implemented both ends
of the protocol. (In fact, we tested the correctness of our implementation by writing a simple
nix protocol proxy and verifying that our proxied stream is byte-for-byte identical to
the original.) And thanks to rust's procedural macros and the [`serde`] crate, our implementation
is declarative, meaning that it also serves as a concise documentation of the protocol.

## Structure of the nix protocol

A nix communication starts with the exchange of a few magic bytes, followed by
some version negotiation. Both the client and server maintain compatibility with
old versions of the protocol, and they always agree to speak the newest version
supported by both.

The main protocol loop is initiated by the client, which sends a "worker op" consisting
of an opcode and some data. The server gets to work on carrying out the requested operation.
While it does so, it enters a "stderr streaming" mode in which it sends a stream of
logging or tracing messages back to the client. (This is how nix's progress messages
make their way to your terminal when you run a `nix build`.) The stream of stderr messages
is terminated by a special `STDERR_LAST` message. After that, the server sends the operation's
result back to the client (if there is one), and waits for the next worker op to come along.

## The nix wire format

Nix's wire format starts out simple. It has two basic types:
- unsigned 64-bit integers, encoded in little-endian order; and
- byte buffers, written as a length (a 64-bit integer) followed by the bytes in the buffer.
  If the length of the buffer is not a multiple of 8, it is zero-padded to a multiple of 8 bytes.
  Strings on the wire are just byte buffers, with no specific encoding.

Compound types are built up in terms of these two pieces:
- Variable-length collections like lists, sets, or maps are represented by their length (as a 64-bit integer) followed by their contents.
  For example, a list of two strings is represented by:
    - the number 2 (the length of the list),
    - the first string (its length, followed by its contents)
    - the second string (its length, followed by its contents)
- Product types (i.e. structs) are represented by listing out their fields one-by-one.
- Sum types (i.e. unions) are serialized with a tag followed by the contents.

This wire format is not self-describing: in order to read it, you need
to know in advance which data-type you're expecting. If you get confused or misaligned somehow,
you'll end up reading complete garbage. In my experience, this usually leads to
reading a "length" field that isn't actually a length, followed by an attempt to allocate
petabytes of memory.

There's one important exception to the main wire format: "framed data."
Some worker ops need to transfer source trees or build artifacts that are too
large to comfortably fit in memory; these large chunks of data need to be
handled differently than the rest of the protocol. Specifically, they're transmitted
as a sequence of length-delimited byte buffers, the idea being that you can read one
buffer at a time, and stream it back out or write it to disk before reading the next
one.
Two
features make this framed data unusual:
the sequence of buffers are terminated by an empty buffer instead of being length-delimited
like most of the protocol, and the individual buffers are not padded out to a multiple of 8 bytes.

## Serde

Serde is the de-facto standard for serialization and deserialization in rust. It
defines an interface between serialization formats (like JSON, or the nix wire
protocol) on the one hand and serializable data types on the other. This divides our
work into two parts: first, we implement the serialization format, by specifying the
correspondence between serde's data model and the nix wire format we described above.
Then we descibe how the nix protocol's messages map to the serde data model.

The best part about using serde for this task is that the second step becomes
straightforward and completely declarative. For example, the `AddToStore` worker op
is implememented like
```rust
#[derive(serde::Deserialize, serde::Serialize)]
pub struct AddToStore {
    pub name: StorePath,
    pub cam_str: StorePath,
    pub refs: StorePathSet,
    pub repair: bool,
    pub data: FramedData,
}
```
These few lines handle both serialization and deserialization (while ensuring that
they remain in-sync).

## Mismatches with the serde data model

While serde gave us some useful tools and shortcuts, it isn't a perfect fit for our case.
For a start, we don't benefit much from one of serde's most important benefits: the decoupling
between serialization formats and serializable data types. We're interested in a specific
serialization format (the nix wire format) *and* a specific collection of data types (the ones
used in the nix protocol); we don't gain much by being able to, say,
serialize the nix protocol to JSON.

The main disadvantage of using serde is that we need to match the nix protocol to serde's data
model. Most things match fairly well; serde has native support for integers, byte buffers,
sequences, and structs. But there were a few mismatches that we had to work around:

- Different kinds of sequences: serde has native support for sequences, and it can support
  sequences that are either length-delimited or not. However, serde does not make it easy to support
  length-delimited and non-length-delimited sequences in the same serialization format. And
  although most sequences in the nix format are length-delimited, the sequence of chunks in a
  *framed source* are not. We hacked around this restriction by treating a framed source not
  as a sequence but as a tuple with `2^64` elements, relying on the fact that serde doesn't care
  if you terminate a tuple early.
- The serde data model is larger than the nix protocol needs; for example, it supports floating
  point numbers, and integers of different sizes and signedness. Our serde de/serializer raises
  an error at runtime if it encounters any of these data types. Our nix protocol implementation
  avoids these forbidden data types, but the serde abstraction between the serializer and the
  data types means that any mistakes will not be caught at compile time.
- Sum types tagged with integers: serde has native support for tagged unions, but it assumes
  that they're tagged with either the variant name (i.e. a string) or the variant's index within
  a list of all possible variants. The nix protocol uses numeric tags, but we can't just
  use the variant's index: we need to specify specific tags for specific variants, to match the
  ones used by nix. We solved this by using our own derive macro for tagged unions. Instead of
  using serde's native unions, we map a union to a serde tuple consisting of a tag followed by
  its payload.

But with these mismatches resolved, our final definition of the nix protocol is fully declarative
and pretty straightforward:
```rust
#[derive(TaggedSerde)]
//       ^^ our custom procedural macro for unions tagged with integers
pub enum WorkerOp {
    #[tagged_serde = 1]
    //              ^^ this op has opcode 1
    IsValidPath(StorePath, Resp<bool>),
    //             ^^            ^^ the op's response type
    //             || the op's payload
    #[tagged_serde = 6]
    QueryReferrers(StorePath, Resp<StorePathSet>),
    #[tagged_serde = 7]
    AddToStore(AddToStore, Resp<ValidPathInfoWithPath>),
    #[tagged_serde = 9]
    BuildPaths(BuildPaths, Resp<u64>),
    #[tagged_serde = 10]
    EnsurePath(StorePath, Resp<u64>),
    #[tagged_serde = 11]
    AddTempRoot(StorePath, Resp<u64>),
    #[tagged_serde = 14]
    FindRoots((), Resp<FindRootsResponse>),
    // ... another dozen or so ops
}
```

## Next steps

Now that we have the protocol implemented, we're going to see if there is anything useful we can
do with it. Our first attempt is an experimental server for proxying nix remote builds to the Bazel
Remote Execution protocol. If you have any other good ideas, let us know!

[hnix-store]: https://github.com/haskell-nix/hnix-store
[`serde`]: https://crates.io/crates/serde