# AllMyToes Provides thumbnails by using the [freedesktop-specified](https://specifications.freedesktop.org/thumbnail-spec/thumbnail-spec-latest.html) thumbnail data-base (aka XDG standard). Thumbnails are shared with other programs via a common cache. The (expensive) creation of thumbnails is only done if a thumbnails has not been created yet or if the original file has changed. AllMyToes can create thumbnails for many image formats. (See “Decoding” support of the [image crate](https://crates.io/crates/image).) Additionally, AllMyToes can create thumbnails for other kind of files using other programs to create the base image. (See section [Provider](#provider).) AllMyToes is both, a non-interactive program for the shell and scripts, and a Rust library. It's abbreviated “AMT” (or “amt”) in code or structured data. [[_TOC_]] # Usage Call `allmytoes` with a file as parameter. The program will print the path to a thumbnail to `stdout`. ``` > allmytoes some_image.jpg /home/me/.cache/thumbnails/large/b7931d1d6e0439c1a6e2e6b02c5b21a6.png ``` If the thumbnail already exists, AllMyToes will just return the path to the existing thumbnail file. If not, AllMyToes will create the image’s thumbnail first. ## Thumbnail Size The latest freedesktop.org specification (0.9.0) defines four different thumbnail sizes: normal, large, x-large, and xx-large with maximum edge lengths of 128 px, 256 px, 512 px, and 1024 px respectively. By default, AllMyToes returns the path to the _large_ (256 px) thumbnail. The size to be returned can be chosen with the `-s` (`--size`) option by giving a value out of {`n`, `l`, `x`, `xx`}. For example, to get a xx-large thumbnail, the call would look like this: ``` > allmytoes -sxx some_image.jpg /home/me/.cache/thumbnails/xx-large/ad0779df58de36f038bdc4040a322bfe.png ``` ### Small Input Images If the input image is smaller than the requested thumbnail size, the thumbnail will have the same dimensions as the input image. A thumbnail will never be bigger than the original image. If a thumb for the requested size does not already exist, AllMyToes will check if the input image is smaller than the requested thumb size. If so, AllMyToes will determine the smallest thumb size that still covers the input image dimensions (is bigger or equal than the input image). If that “feasible” thumb size is different from the requested thumb size, AllMyToes will start the job all over with that smaller, “feasible” thumb size. AllMyToes does that to avoid unnecessary, duplicated thumbnails for small images. That implies that for small input images, the returned thumbnail path might not correspond to the requested thumb size, because creating and keeping thumbnails for the (bigger) requested size would be a waste of computation time and disk space. If – for whatever reason – one must get a result path to the thumbnail of the requested size, one can use the `-F` (`--force-size`) switch. However, one should do that only if there is a really good reason. ## Show Extensive Information Calling AllMyToes with the `--extensive` (or `-e`) option makes it print out more information than just the path to the thumb. The provided information is * The path to the thumb * The URI-hash of the input file that is used for the XDG-thumb name * All meta data (key-value pairs) stored in the thumb (as “tEXt” entries) It does not matter if the thumb did already exist or if it has just been created by AllMyToes. All data is printed as a list of key-value pairs, each tuple in one line. The key is separated from the value by “`: `”. Key-value pairs which show meta data from the thumbnail file are preceded with a colon and are printed in alphabetical order. Example: ```bash $ allmytoes --extensive some_image.jpg Thumb path: /home/dude/.cache/thumbnails/large/c000276b8378a33e16e3fac005eebc02.png Thumb hash: c000276b8378a33e16e3fac005eebc02 :Software: allmytoes :Software::Version: 0.1.0 :Thumb::Image::Dimensions: 4608x3456 :Thumb::Image::Height: 3456 :Thumb::Image::Reorientation: Rotated 180 degrees, not flipped :Thumb::Image::Width: 4608 :Thumb::MTime: 1693737198 :Thumb::Mimetype: image/jpeg :Thumb::Size: 7777807 :Thumb::URI: file:///path/to/some_image.jpg ``` What meta-data is available in the thumb of course depends on the software which created the thumb. So, the lines starting with a colon (`:`) may vary. ## Logging By default, AllMyToes will print warnings and errors to `stderr`. A different log-level can be set via the environment variable `RUST_LOG`. For example, do a ```sh export RUST_LOG=trace ``` to get all log entries, down to the most detailed “trace”-level. # Provider AllMyToes can create thumbnails for non-image files if a “provider” is defined. A provider is a shell-command or a script that provides an image for a certain set of MIME types. AllMyToes has inbuilt support for some file formats: | Format | Dependencies | |------------|------------------------------------------------------------------| | PDF | (`evince-thumbnailer` or `magick`) and (`exiftool` or `pdfinfo` or `gs`) | | Postscript | (`evince-thumbnailer` or `magick`) and (`exiftool` or `gs`) | | SVG | `inkscape` or `magick` | | various video formats | `ffmpeg` (and optionally `ffprobe` & `magick` & `bc` for more complex thumbs) and (`ffprobe` or (`mediainfo` and `bc`) or (`exiftool` and `bc`)) | The exact MIME-types and the provider definitions can be found in [`/conf/provider.yaml`](https://gitlab.com/allmytoes/allmytoes/-/blob/main/conf/provider.yaml). ## Defining Provider Users can use their own provider configuration in a YAML file to add support for more MIME types or to implement more fancy thumbnails. To do so, copy [`/conf/provider.yaml`](https://gitlab.com/allmytoes/allmytoes/-/blob/main/conf/provider.yaml) to `~/.config/allmytoes/provider.yaml` (or `$XDG_CONFIG_HOME/allmytoes/provider.yaml` if `$XDG_CONFIG_HOME` is defined) and adapt it. You can also use the `-p` (`--provider-config-file`) option to explicitly choose another provider configuration. ❗ Be aware that the provider-feature is pretty new and the format of the provider configuration will likely change here and there before it stabilizes. The provider configuration YAML file has a list of dictionaries on the top level. Each list entry defines a provider for one or more MIME types. For each entry, there are two mandatory and two optional keys. | Key | Mandatory | Short description | |-------------|-----------|---------------------------------------------------| | `mimes` | Yes | _List_ of mime types, handled by the provider definition | | `commands` | Yes | _List_ of commands providing the thumbnail, processed top down until one suceeds | | `meta` | No | _Dictionary_ mapping thumb-meta-keys to a list of commands to provide them | | `revision` | No | _integer_ to track the version of the provider definition | ### Key `mimes` The mime type list defines for which mime types the provider is executed. MIME types are specifies as `/`, like `video/x-matroska`. If AllMyToes is called for a file for which no provider can be found, the MIME type is printed in an error message. MIME types for which a provider is found are printed in a debug message and also annotated as meta-data in the thumbnail itself. (Which you can see if you use the `--extensive` (`-e`) option.) ### Key `commmands` Each provider-entry must have a list of commands. Commands are shell commands (or scripts) that take the input file and return an image back to AllMyToes. AllMyToes takes that image and creates the XDG-conforming thumbnail from it. Commands provide that image by copying it to a defined location. When one of the MIME types from the `mimes` list matches the input file, AllMyToes executes the first command from the `commands` list. If the command succeeds (exit code `0`), AllMyToes stops and uses the thumbnail which has been provided by that command. If the command does not succeed (exit code `≠ 0`), AllMyToes tries the next command, and so forth. This allows to have commands which use different programs to provide the thumb. If a user does not have the programs used in the first command, another option to calculate the thumb may work. If a command returns `0` but still does not provide a valid thumb, AllMyToes returns with an error. A command provides the thumb by copying it to a certain path in a temporary directory, which is deleted by AllMyToes afterwards. Commands need certain information to do their job, for example the input file and the path to copy the thumbnail to. Therefore, commands can use certain variables. | Variable | Content | |---------------|--------------------------------------------------------| | `%(file)s` | The full path to the input file | | `%(size)s` | The size in pixels that the longer edge should ideally have. | | `%(outfile)s` | The path to the temporary file where the thumbnail shall be copied to | | `%(tmpdir)s` | The temporary directory that contains `%(outfile)s`, and that can be used other temporary files that might be needed during the thumb-creation | If the created image (at `%(outfile)s`) is bigger than `%(size)s`, AllMyToes will scale the image down accordingly. AllMyToes also does other necessary conversions like turning the image into a 8-bit per channel RGBA PNG image and add all necessary meta chunks. So, the commands can provide their thumbnails in any supported image format. AllMyToes will take care to turn the image into a freedesktop.org-conform thumbnail afterwards. ### Key `meta` freedesktop.org-thumbnails have “meta data chunks”, meta-data attributes in key-value pairs. (See [explanation of meta-data chunks](#thumbnail-meta-data-chunks).) Providers can also provide such meta-data entries. Some are even [recommended by the standard](https://specifications.freedesktop.org/thumbnail-spec/thumbnail-spec-latest.html#CREATION) (`Thumb::Document::Pages` for paper-oriented documents and `Thumb::Movie::Length` for the length of video files in seconds). The `meta` key in the provider specification contains a dictionary, mapping meta-data keys to lists of commands. For each key, the list of commands are executed like for the thumb image top down until one of them exits successfully with `0`. The value for the meta-chunk has to be returned via `stdout`. With the current implementation, the thumb-generation fails if none of the commands for a meta-chunk is successful. The commands for a meta-chunk can use the same variables as the commands for the image creation. The specification of meta-chunks is optional. ### Key `revision` The revision key is intended as a “version” for the provider. It's optional and as of now, it does not have any effect other than being added as a meta-chunk to the thumbnail. The idea is that this revision number can be used in the future to re-create thumbs if a provider is available with a higher revision. ### Notes on commands One should consider a few things for both, commands for the images in the `commands` key, and for commands under a `meta` key. First, commands can be **multi-line shell scripts**, simply by using multi-line strings in the YAML file. This example for an SVG-provider shows how multi-line commands may look like: ```yaml - mimes: - image/svg+xml commands: - | which inkscape || exit 1 w=$(inkscape -W "%(file)s" | sed 's/\..*//') h=$(inkscape -H "%(file)s" | sed 's/\..*//') test $w -gt $h && size_arg="-w %(size)s" || size_arg="-h %(size)s" inkscape --export-area-page --export-type=png $size_arg -o "%(tmpdir)s/outfile.png" "%(file)s" && \ mv "%(tmpdir)s/outfile.png" "%(outfile)s" - | convert -background none -resize %(size)sx%(size)s "%(file)s" "%(tmpdir)s/outfile.png" && \ mv "%(tmpdir)s/outfile.png" "%(outfile)s" ``` Secondly, commands should be **POSIX shell compatible**, not - for example - depend on BASH. This assures that the provider commands will work on most machines and accounts. Third, commands should **fail with a return code `> 0`** if some environment preconditions are not fulfilled. Keep in mind that a pipe of commands exits with the return code of the last command, no matter if an earlier command did fail. So, it's a good idea to chain programs with `&&` or to extend program calls with `|| exit 1` to make a provider command fail properly. The availability of required programs that are used in pipes can for example be tested with `which || exit 1`. Also remember to **quote the input file** (like `"%(file)s"`) as the path may contain spaces. The `%(outfile)s` (and the `%(tmpdir)s`) should not contain any spaces, but it doesn't hurt to also quote them. ### Example Let's look at an example and specify a provider that provides thumbnails for PDF _and_ postscript files. Both can be turned into a thumbnail image with the same command, which is why it's possible to handle both in one provider. Be aware that the real default configuration for AllMyToes uses a different provider definition for these formats to implement more specific meta-chunk commands. This definition serves only as an example. PDF files have a MIME type `application/pdf`, postscript files have `application/postscript`. ```yaml - mimes: [application/pdf, application/postscript] commands: - evince-thumbnailer -s %(size)s "%(file)s" %(outfile)s - convert -thumbnail %(size)sx%(size)s -background white %(file)s[0] PNG:%(outfile)s meta: 'Thumb::Document::Pages': - | which exiftool || exit 1 exiftool "%(file)s" | awk '/^Page Count/ {print $4}' | sed 's/[^0-9]*//g' - | which gs || exit 1 gs -o /dev/null -sDEVICE=bbox "%(file)s" 2>&1 | grep HiResBoundingBox | wc -l revision: 1 ``` First, the MIME types for which this provider shall be used are defined. The `commands` section first tries to create the thumbnail (in `%(outfile)s`) with `evince-thumbnailer`. If `evince-thumbnailer` exists, the task of providing the image is done and the second command is not executed. If `evince-thumbnailer` is not available, the first command fails and AllMyToes will execute the second command which uses `convert` from ImageMagick. If also `convert` is not available, the thumb creation has failed and AllMyToes returns with an error. The `evince-thumbnailer` is tried first because it's faster and we prefer that one. If one of the two commands was successful, AllMyToes evaluates the `meta` section. In this example, we have only one meta-chunk with the meta-key `Thumb::Document::Pages`. This meta-chunk has also two commands and again, AllMyToes will first try the first and execute the second one only if the first one fails. The first command depends on `exiftool`, the second depends on Ghostscript (`gs`). Because both of them are used in a pipe, we first check if they exist explicitly and exit with `1` otherwise. The example shows how a multi-line script can be used as a command. Both of the meta-data commands return the value for the `Thumb::Document::Pages` meta data chunk as output on `stdout`. Check the [default provider configuration](https://gitlab.com/allmytoes/allmytoes/-/blob/main/conf/provider.yaml) for more examples. # How is this useful? My personal motivation was the use of thumbnails for image previews in terminal-based file managers like [joshuto](https://github.com/kamiyaa/joshuto) and [ranger](https://github.com/ranger/ranger). But AllMyToes can be useful in any situation where one wants to show a size-limited image in scripted environments, maybe as icons in [desktop notifications](https://specifications.freedesktop.org/notification-spec/notification-spec-latest.html), other kind of image-overlays in terminal-based applications, or desktop widgets; wherever loading full sized-images would consume unnecessary time and CPU power. # Installation #### Linux X86-64 Binary (Experimental) Download the latest release as binary from [the release page](https://gitlab.com/allmytoes/allmytoes/-/releases). Make the downloaded file executable (`chmod +x allmytoes`) and place it in a directory which is part of your `PATH` (for example `mv allmytoes ~/.local/bin`). #### Latest release from source via `cargo` ``` cargo install allmytoes ``` #### Bleeding edge from source via `cargo` ``` cargo install --git https://gitlab.com/allmytoes/allmytoes.git ``` #### From repo clone ``` git clone https://gitlab.com/allmytoes/allmytoes.git cd allmytoes cargo build --release ``` The built binary will be at `target/release/allmytoes`. # Some Specs * Freedesktop.org Thumbnail Specification: AllMyToes strives for compliance to the [thumbnail freedesktop.org specification](https://specifications.freedesktop.org/thumbnail-spec/thumbnail-spec-latest.html). All relevant parts of the standard should be implemented by now. * AllMyToes will not create the cache directory (`$XDG_CACHE_HOME` or `~/.cache`). In case the cache directory does not exist, AllMyToes will terminate with an error. (Might be changed later.) * AllMyToes will create the thumbnail directory beneath the cache directory and the size-specific sub-directories if they do not exist. * AllMyToes will only accept regular files and symlinks to regular files as input. Directories may get supported in the future. There are no plans for other [file types](https://en.wikipedia.org/wiki/Unix_file_types). ## Thumbnail Meta-Data Chunks Thumbnails - all in PNG format - have certain meta-data entries, each of them a PNG “tEXt” [chunk](https://en.wikipedia.org/wiki/PNG#%22Chunks%22_within_the_file) with a key and a value. The freedesktop.org thumbnail standard describes some mandatory and some optional keys. (See the “[Thumbnail Creation section](https://specifications.freedesktop.org/thumbnail-spec/thumbnail-spec-latest.html#CREATION)” in the standard.) AllMyToes implements all of them. The following table lists all meta-data entries created by AllMyToes. | Key | XDG standard | Description | | --------------- | ------------ | -------------------------------------------------------------| | `Thumb::URI` | mandatory | The URI of the thumb's source file. | | `Thumb::MTime` | mandatory | The mtime of the thumb's source file. (Functionally required.) | | `Software` | optional | The name of the software that created the thumb. Also described by the PNG standard.
This is always set to `allmytoes`. | | `Software::Version` | - | The version of AllMyToes that created the thumb. | | `AMT::ProviderRevision` | - | For thumbs from providers (non image sources), the revision of the provider as given in the config. | | `Thumb::Size` | optional | The size of the thumb's source file in bytes. | | `Thumb::Mimetype` | optional | The mimetype of the thumb's source file. | | `Thumb::Image::Width` | optional | The width of the source image in pixel. † | | `Thumb::Image::Height` | optional | The height of the source image in pixel. † | | `Thumb::Image::Dimensions` | - | A string `x`, describing the dimensions of the thumb's source image. †| | `Thumb::Image::Reorientation` | - | A textual description of the rotation and flipping done my AllMyToes on creation of the thumb based on the EXIF data of the thumb's source image.
Only if the source of a thumb is an image format with EXIF data that contains an `Orientation` flag. | | `Thumb::Document::Pages` | optional | The number of pages (only for PDF and postscript files). | | `Thumb::Movie::Length` | optional | The length of a video in seconds (only for video files). | > † Only for image input files # Roadmap Major milestones planned: * [x] Comply to mandatory requirements from the freedesktop.org specification and have basic robustness (see %1). AllMyToes will go to version 0.1.0 then. * [x] Support for non-image file formats (videos, fonts, documents,...) by configuring other programs providing previews. (#13) * [ ] Support for [shared repositories](https://specifications.freedesktop.org/thumbnail-spec/thumbnail-spec-latest.html#SHARED). * [ ] Support for other thumbnail repository locations depending on configured path patterns. (Not part of the freedesktop.org standard.) See also [my development board](https://gitlab.com/allmytoes/allmytoes/-/boards). # Testing Few things are tested by standard Rust unit test. Those tests can be run with `cargo test`. The majority of AllMyToes’ functionality is tested with end-to-end tests by a BDD test framework, [Python Behave](https://behave.readthedocs.io/en/stable/). Those tests use real images on the file system and an `allmytoes` debug build, started as a sub-process. The BDD tests live in the `test` sub-directory. The test-definitions can be found as `feature`-files in `test/features`. ## Running the BDD tests Have a Python >= 3.10 environment with the required dependencies. E.g., use a virtualenv and then `pip install -r test/requirements.txt`. Have a _debug_ build of AllMyToes. `target/debug/allmytoes` will be the binary under test. Then, to run the tests, `cd` into the `test` directory and run `behave`. ## Notes on missing Tests / untested things * evaluation of `XDG_CACHE_HOME` * MIME type probing for wrong _and_ missing file extensions (if this fails on CI, check if the MIME-db is installed on the used docker img) * `--force-creation` * Return error when input file is not readable, no matter if valid thumb exists. * Updating existing but outdated thumbs * Error on non-existing, non-readable, not-decodable input-file * Deny to process files which are neither a regular nor a symlink to a regular file * All tEXt meta-data (mtime and URI are tested implicitly, of course) * #61 # You may prefer to use this instead of AllMyToes * [Tumbler](https://gitlab.xfce.org/xfce/tumbler/) is a much more mature and feature-rich tool for the same purpose, but needs to run as a service and uses DBUS for communication. I did not use Tumbler as I wanted to have something more simple for my scripting. # AllMyToes as a Rust Library AllMyToes can be used as a Rust library to obtain a thumbnail for a given image. You can find the crate [on crates.io](https://crates.io/crates/allmytoes). AllMyToes has a very small interface. There's one struct for the configuration (`AMT`) that also provides the one function (`get`) to get a thumb. Then, there's one enumeration to specify the thumbnail size (`ThumbSize`), one struct for the result (`Thumb`), and one enumeration for the possible errors (`ToeErrorType`). ### Example ```rust use std::path::Path; use allmytoes::{AMTConfiguration, AMT, ThumbSize}; fn main() { // The configuration for allmytoes // Usually, the defaults are fine. let configuration = AMTConfiguration::default(); // An instance of allmytoes that can be used to provide thumbnails let amt = AMT::new(&configuration); // The file for which we want a thumbnail as a `std::path::Path`. let input_file = Path::new("/tmp/image.jpg"); // The size of the thumbnail we want let thumb_size = ThumbSize::Large; // Get a thumbnail match amt.get( input_file, thumb_size, ) { Ok(thumb) => { println!("The thumb is here: {}", thumb.path) } Err(error) => println!( "Error '{:?}' occurred when trying to provide the thumb. ({})", error, error.msg(), ), } } ``` # License: GPL 3 AllMyToes is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. AllMyToes is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with AllMyToes. If not, see .