Crates.io | finalfusion-utils |
lib.rs | finalfusion-utils |
version | 0.14.1 |
source | src |
created_at | 2019-03-19 18:20:55.471675 |
updated_at | 2021-12-17 12:28:26.072043 |
description | finalfusion utilities |
homepage | https://github.com/finalfusion/finalfusion-rust |
repository | https://github.com/finalfusion/finalfusion-rust |
max_upload_size | |
id | 122501 |
size | 113,622 |
finalfusion-utils
is a Rust crate offering various
functionalities to process and query embeddings.
finalfusion-utils
supports conversion between different
formats, quantization of embedding matrices, similarity and
analogy queries as well as evaluation on analogy datasets.
The following precompiled binaries can be found on the releases page:
x86_64-unknown-linux-gnu-mkl
: glibc Linux build, statically linked
against Intel MKL. This is the recommended build for Intel (non-AMD)
CPUs.
x86_64-unknown-linux-musl
: static Linux build using the MUSL C
library. This binary does not link against a BLAS/LAPACK implementation
and therefore does not support optimized product quantization.
universal-macos
: dynamic macOS build. Supports both the x86_64 and
ARM64 architectures. Linked against the Accelerate framework for
BLAS/LAPACK.
cargo
finalfusion-utils
can be installed using an up-to-date Rust
toolchain, which can be installed with rustup.
With a valid Rust toolchain, the crate is most easily installed through
cargo
:
$ cargo install finalfusion-utils
Typically, you will want to enable support for a BLAS/LAPACK library to speed up matrix multiplication and enable optimized product quantization support. In order to do so, run
$ cargo install finalfusion-utils --features implementation
where implementation
is one of the following:
accelerate
: the macOS Accelerate framework.intel-mkl
: Intel MKL (downloaded and statically linked).intel-mkl-amd
: Intel MKL, preinstalled MKL libaries expected, override
CPU detection for AMD CPUs.netlib
: any compatible system BLAS/LAPACK implementation(s).openblas
: system-installed OpenBLAS. This option is discouraged,
unless the system OpenBLAS library is a single-threaded build with
locking. Otherwise, OpenBLAS' threading interacts badly with application
threads.finalfusion-utils
can also be built from source,
after cloning this repository execute the following
command in the directory to find the exectuable under
target/release/finalfusion
:
$ cargo build --release
finalfusion-utils
is built as a single binary, the
different functionality is invoked through subcommands:
# Convert embeddings in fastText format to finalfusion
$ finalfusion convert -f fasttext -t finalfusion \
embeddings.bin embeddings.fifu
# Convert embeddings in word2vec format to finalfusion
$ finalfusion convert -f word2vec -t finalfusion \
embeddings.w2v embeddings.fifu
# Print help with all supported combinations:
$ finalfusion convert --help
# Quantize embeddings in finalfusion format with a
# single attempt through product quantization
$ finalfusion quantize -f finalfusion -q pq -a 1 \
embeddings.pq
# Get the 15 nearest neighbours of "Tübingen" for
# embeddings in finalfusion format.
$ finalfusion similar -f finalfusion -k 15 \
embeddings.fifu
# Get the 5 best answers for the analogy query
# "Berlin" is to "Deutschland" as "Amsterdam" to:
$ finalfusion analogy -f finalfusion -k 5 \
Berlin Deutschland Amsterdam embeddings.fifu
# Evaluate embeddings on some analogy dataset
$ finalfusion compute-accuracy embeddings.fifu \
analogies.txt
# Dump optionally stored metadata and store in
# metadata.txt, only supported for finalfusion
# format
$ finalfusion metadata embeddings.fifu \
> metadata.txt
# Converts a hash-bucket based subword vocab to
# one with explicitly stored n-grams.
$ finalfusion bucket-to-explicit buckets.fifu \
explicit.fifu
# Print completion script for zsh
$ finalfusion completions zsh