# elfshaker File Format

The format of `.pack` and `.pack.idx` files is stable and must only change with major version changes.

## `.pack` format

The pack file format is a subset of the Zstandard format. Zstandard compressed data is made of one or more frames.

A special kind of frame called a Skippable Frame can be used to inject custom uncompressed data.

We use a Skippable Frame to write some additional metadata about the number of frames and their offsets in the file. Knowing these offsets beforehand allows us to split the file in chucks and decompress the frames individually. This header is the `PackHeader` struct serialized into the msgpack serialization format.

The N compressed frames are regular Zstandard data frames. The decompressed content of each frame is the concatenation of the objects assigned to that frame. Our current heuristic for assigning objects works by sorting the full set of objects by size and then assigning those objects to equally-sized buckets. Each buckets forms a frame in the pack.

🛈 The compression level can be set during packing. See the help output:
```bash
elfshaker pack --help
```

<!-- The following table was generated by https://ozh.github.io/ascii-tables/ -->
```
+--------------+------------------+
| Description  |    Frame Type    |
+--------------+------------------+
| PackHeader   | Skippable Frame  |
| ObjectFrame0 | Compressed Frame |
| ...          | Compressed Frame |
| ObjectFrameN | Compressed Frame |
+--------------+------------------+
```

## `.pack.idx` format
The pack index format is an entirely custom format. It starts with a magic number (`0x454c4653` — aka "ELFS"), followed by a sequence of msgpack-serialized values. See [`struct PackIndex`](../../src/packidx.rs) for more information.

The pack index contains essential metadata needed to extract anything from the pack:
- The list of snapshots
- The list of files stored in each snapshot
- The corresponding objects for these files
- The SHA-1 checksum, size and offset of these objects in the `.pack`