arx: A Fast, Mountable File Archive

Arx is a high-performance file archive format built upon the Jubako container format. It offers a compelling alternative to traditional archive formats like zip and tar, providing significant speed advantages, especially for large archives and random access operations. Arx archives can even be mounted as read-only filesystems.

Key Features

• Fast Creation and Extraction: Arx leverages optimized compression algorithms and a structured data layout for significantly faster archive creation and extraction times compared to traditional methods, particularly for larger datasets.

• Random Access: Access individual files within the archive without needing to decompress the entire archive. This is particularly beneficial for large archives.

• Read-Only Mounting (Linux and MacoOS): Mount Arx archives as read-only filesystems using FUSE, allowing you to directly access and work with files within the archive without decompression.

• Versatile Compression: Supports various compression algorithms, including zstd (default), lz4, and lzma, allowing you to choose the best option for your data and performance needs.

• Comprehensive CLI Tool: A command-line interface simplifies archive creation, extraction, listing, and mounting.

• Python Bindings: A Python wrapper facilitates integration with Python projects.

Installation

Using Cargo

The easiest way to install arx is via Cargo, Rust's package manager:

cargo install arx

Pre-built Binaries

Pre-built binaries for Windows, macOS, and Linux are available for each release on GitHub Releases. Download the appropriate binary for your operating system and add it to your system's PATH environment variable.

Usage Examples

Create an Archive:

Create an archive named my_archive.arx from the directory my_directory:

arx create -o my_archive.arx -r my_directory

The -r flag indicates recursive inclusion of subdirectories. You can omit this for non-recursive creation.

To strip a common prefix from the file paths within the archive, use the --strip-prefix option:

arx create -o my_archive.arx -r --strip-prefix /home/user/documents /home/user/documents/my_directory

Extract an Archive:

Extract the contents of my_archive.arx to the directory my_output_dir:

arx extract my_archive.arx -C my_output_dir

The -C flag specifies the output directory. If omitted, extraction happens in the current directory.

List Archive Contents:

List the files and directories within my_archive.arx:

arx list my_archive.arx

For a more machine-readable output suitable for scripting, use the --stable-output option:

arx list --stable-output my_archive.arx

Dump a Single File:

Dump the contents of a specific file (my_directory/my_file.txt) within the archive to standard output:

arx dump my_archive.arx my_directory/my_file.txt

To redirect the output to a file, use redirection:

arx dump my_archive.arx my_directory/my_file.txt my_file.txt

Mount the Archive (Linux and MacOS):

Mount my_archive.arx to a mount point (requires libfuse-dev on Linux and macfuse on macOS):

mkdir mount_point
arx mount my_archive.arx mount_point

Unmount using the standard umount command. If mount_point is not provided, a temporary mount point will be created. The arx mount command runs in the background by default. Use the --foreground flag to keep it in the foreground.

Convert Zip/Tar Archives:

Convert a zip archive (my_archive.zip) or a tar archive (my_archive.tar.gz) to an Arx archive:

zip2arx -o my_archive.arx my_archive.zip
tar2arx -o my_archive.arx my_archive.tar.gz

You may need to install zip2arx and tar2arx tools, the same you have installed arx tool.

Remote tar archives can also be converted using tar2arx:

tar2arx -o my_archive.arx https://example.com/my_archive.tar.gz

Performance

The following tables compare the performance of Arx to different archive formats. Tests were conducted on various datasets (the entire Linux kernel, its drivers directory, and its documentation directory) stored on an SSD. All tests were run on a tmpfs (archive and extracted files stored in memory). Mount diff time measures the time to diff the mounted archive with the source directory using diff -r. Mounting of tar and zip archives was performed using the archivemount tool. Arx mount is implemented using the fuse API. Squashfs was mounted using the kernel; SquashfsFuse was mounted using the fuse API; Only Mount diff differs between the two.

"Mount diff" times for tar and zip are significantly longer and may not always be fully measured depending on the dataset and system specifications.

The comparaison script is available at script/compare_archive.py

Linux doc (Documentation directory only of Linux source code):

Ratio <Archive> time / Arx time (A ratio > 100% means Arx is better):

Linux Driver (Driver directory only of Linux source code):

Ratio <Archive> time / Arx time (A ratio > 100% means Arx is better):

Linux Source Code (Entire Linux source code):

Ratio <Archive> time / Arx time (A ratio > 100% means Arx is better):

Kernel Compilation Time (Time needed to compile the whole kernel with default configuration -j8):

Arx archives are slightly larger (about 1%) than tar.zst archives but 15% smaller than squashfs. Creation and full extraction times are comparable to other formats, but listing files and accessing individual files from the archive are much faster using arx or squashfs. Access time is almost constant independently of the archive size, unlike tar, where access time increases significantly with archive size. Mounting an arx archive makes the archive usable without extraction.

Contributing

Contributions are welcome! Please open an issue or submit a pull request.

Sponsoring

I (@mgautierfr) am a freelance developer. All jubako projects are created in my free time, which competes with my paid work. If you want me to be able to spend more time on Jubako projects, please consider sponsoring me. You can also donate on liberapay or buy me a coffee.

License

This project is licensed under the MIT License - see the LICENSE-MIT file for details.