caesura 𝄓
A versatile command line tool for automated verifying and transcoding of all your torrents. ## Features Most gazelle based indexers/trackers are supported - RED - **[[new](https://github.com/RogueOneEcho/caesura/issues/7)]** OPS. Tested on Linux, theoretically works on Windows. Fully configurable, if there's something hard coded that you think should be configurable then [open a discussion on GitHub](https://github.com/RogueOneEcho/caesura/discussions). ### Source Verification Each source is verified to ensure: - A lossless FLAC - Not a scene, lossy, unconfirmed, or trumpable release - Files match the torrent hash - Audio tags for artist, album, title and track number are set - **[[fixed](https://github.com/RogueOneEcho/caesura/issues/47)]** Classical sources have a composer tag. - **[[fixed](https://github.com/RogueOneEcho/caesura/issues/18)]** Vinyl track numbering is converted to numeric - Sample rate and channels are suitable ### Spectrogram Generation - Full and zoomed spectrograms generated for review ### Transcoding - **[fixed]** Multi-threaded transcoding with optional CPU limit - FLAC and FLAC 24 bit sources are supported - FLAC, MP3 320 (CBR) and MP3 V0 (VBR) target formats - Existing formats are skipped - **[[fixed](https://github.com/RogueOneEcho/caesura/issues/21)]** Nested sub directories are fully supported (i.e. CD1, and CD2 etc) - **[[fixed](https://github.com/RogueOneEcho/caesura/issues/22)]** Automatic naming following established conventions, with decoding of HTML entities. - **[[fixed](https://github.com/RogueOneEcho/caesura/issues/24)]** Shorter file names. - Automatic torrent file creation - **[new]** Images in the root and first nested directory are included and all other files ignored. - **[new]** Images larger than 750 KB are reduced to less than 1280 px, converted to JPG and compressed. ### Upload - Copy transcodes to content directory - Copy torrent file to client auto-add directory ### Batch / Queue - **[new]** Verify, transcode and upload with one command for every torrent file in a directory. - **[new]** Source torrents are added to a queue to track their progress reducing duplicate work and speeding up subsequent runs. *The application will crunch through your torrent directory and automatically determine which are FLAC sources suitable for transcoding.* ## Getting started Docker is the recommended way to run the application across all platforms. - All dependencies are built into the image - Runs in an isolated environment reducing risks to your system > [!TIP] > **[Configuration options and the commands they apply to are documented in COMMANDS.md](COMMANDS.md)** ### 0. Install Docker [Install Docker Engine](https://docs.docker.com/engine/install/) for your OS. ### 1. Run the `help` command Run the `help` command to see the available commands and options. ```bash docker run ghcr.io/rogueoneecho/caesura --help ``` > [!TIP] > You can append `--help` to any command to see the available options. > > ```bash > docker run ghcr.io/rogueoneecho/caesura verify --help > ``` ### 2. Create a configuration file Create a `config.yml` file with the following content: - `announce_url` Your personal announce URL. Find it on upload page. - `api_key` Create an API key with `Torrents` permission `Settings > Access Settings > Create an API Key` Refer to [COMMANDS.md](COMMANDS.md) for full documentation of options. ```yaml announce_url: https://flacsfor.me/YOUR_ANNOUNCE_KEY/announce api_key: "YOUR_API_KEY" ``` You can then run the `config` command to see how the full configuration including default values the application will use: ```bash docker run ghcr.io/rogueoneecho/caesura config ``` > [!TIP] > The following fields are optional, if not set they're set based on the `announce_url`: > - `indexer` the id of the indexer: `red`, `pth`, `ops`. > - `indexer_url` the URL of the indexer: `https://redacted.ch`, `https://orpheus.network`. ### 3. Create storage directories Create a directory for the application to output files to: ```bash mkdir ./output ``` Create a directory for the application to cache files to: ```bash mkdir ./output ``` > [!TIP] > Refer to the [directory structure](#directory-structure) section for documentation on the purpose and structure of these directories. ### 4. Verify a source Run the `verify` command with the source as an argument. > [!NOTE] > Because the application is running in a Docker container, you need to mount the config file, content directory, output directory and cache directory. > [!TIP] > For the source you can use a permalink, the numeric torrent id or a path to a torrent file: > > Each step of this guide will use a different source to demonstrate, but feel free to use whichever suits you best. ```bash docker run \ -v ./config.yml:/config.yml \ -v /path/to/your/content:/content \ -v ./output:/output \ -v ./cache:/cache \ ghcr.io/rogueoneecho/caesura \ verify https://redacted.ch/torrents.php?id=80518&torrentid=142659#torrent142659 ``` If it looks good you can proceed to the next step, otherwise try another source. ### 5. Use Docker Compose Docker is great but specifying the volumes everytime is tedious and prone to error. Using Docker Compose simplifies this by storing the configuration in a `docker-compose.yml` file. Create a `docker-compose.yml` file with the following content: ```yaml services: caesura: container_name: caesura image: ghcr.io/rogueoneecho/caesura volumes: - ./config.yml:/config.yml:ro - /path/to/your/content:/content:ro - ./output:/output - ./cache:/cache ``` > [!NOTE] > The `:ro` suffix makes the volume read-only which is a good security practice. > > If you intend to use the `--copy-transcode-to-content-dir` option then you must remove the `:ro` suffix from the content volume. > > If you intend to use the `--hard-link` option then the `content` and `output` paths must be inside the same volume and you will need to update the `config.yml` accordingly. Now run the verify command again but this time using Docker Compose: ```bash docker compose run --rm caesura verify 142659 ``` ### 6. Generate spectrograms of a source Run the `spectrogram` command with the source as an argument. ```bash docker compose run --rm caesura spectrogram 142659 ``` Inspect the spectrograms in the output directory. ### 7. Transcode a source Run the `transcode` command with the source as an argument. ```bash docker compose run --rm caesura transcode "Khotin - Hello World [2014].torrent" ``` Inspect the transcodes in the output directory. > [!TIP] > Things to check: > - Folder structure > - File names > - Tags > - Audio quality > - Image size and compression quality ### 8. Upload transcodes > [!WARNING] > You are responsible for everything you upload. > > Misuse of this application can result in the loss of your upload privileges. Run the `upload` command with the source as an argument. > [!TIP] > Ideally you've already checked everything and nothing will go wrong but just in case there is a grace period after uploading in which you can remove the upload from your indexer. > [!TIP] > If you're unsure about this then you can append `--dry-run` to the command and instead of uploading it will print the data that would be submitted. ```bash docker compose run --rm caesura upload https://redacted.ch/torrents.php?id=80518&torrentid=142659#torrent142659 ``` Go to your indexer and check your uploads to make sure everything has gone to plan. ### 9. Batch processing > [!WARNING] > You are responsible for everything you upload. > > Misuse of this application, especially the `batch` command, can result in the loss of your upload privileges or a ban. Now that you have the hang of the application we can speed things up with the `queue` and `batch` commands. The `batch` command handles `verify`, `spectrogram`, `transcode` and `upload` in a single command. Run the `queue add` command to search through a directory of torrents and queue them for batch processing: > [!NOTE] > The `batch` and `queue` commands use the cache directory to store progress helping speed up subsequent runs. > > Make sure the cache directory is in a mounted volume so it's not deleted between runs. ```bash docker compose run --rm caesura queue add /path/to/your/torrents ``` Run the `queue list` command to see what is next in the queue for the current `indexer`: ```bash docker compose run --rm caesura queue list ``` By default the `batch` command will limit to processing just `3` transcodes and it won't create spectrograms or upload unless explicitly instructed. These safeguards are in place to prevent mistakenly uploading a bunch of sources that you haven't checked. Run the command to batch verify and transcode the three sources in the queue: ```bash docker compose run --rm caesura batch --transcode ``` > [!TIP] > Add the `--spectrogram` flag to generate spectrograms. If everything goes to plan three sources should have transcoded to your output directory. Use the `queue summary` command to see the progress: ```bash docker compose run --rm caesura queue summary ``` > [!TIP] > Refer to the [analyzing the queue](#analyzing-the-queue) section to inspect the queue in greater detail. Nothing was uploaded in the previous run of the `batch` command giving you a chance to check the transcodes and spectrograms. Once you're satisfied run again but with the `--upload` flag. ```bash docker compose run --rm caesura batch --transcode --upload ``` Check the uploads on your indexer to make sure everything has gone to plan. Now, we can set the batch command loose with the `--no-limit` option to transcode (but not upload) every source in the directory: ```bash docker compose run --rm caesura batch --transcode --no-limit ``` Once you've checked the transcodes you can start to upload them in batches. The `--wait-before-upload 30s` option will add a 30 second wait interval between uploads to give you time to check everything looks good, and spread out the load on your indexer: ```bash docker compose run --rm caesura batch --upload --limit 10 --wait-before-upload 30s ``` > [!WARNING] > In theory you can execute with both `--upload --no-limit` but that is probably a bad idea and a very fast way to lose your upload privileges. > > If you are going to do so then you should definitely use a long wait interval: > `--upload --no-limit --wait-before-upload 2m` ### 10. Next steps Check out the [full documentation of configuration options in COMMANDS.md](COMMANDS.md), in particular you may want to use `--copy-transcode-to-content-dir` and `--copy-torrent-to` to suit your preferred setup. ## Directory Structure The application requires two writable directories. ### Cache Directory The `verify` command will download `.torrent` files for each source to `{CACHE}/torrents/{ID}.{INDEXER}.torrent` > [!TIP] > You can delete the cached `.torrent` files at any time. The application will just download them again if required. The `queue` and `batch` commands will read and write the source statues to `{CACHE}/queue/{FIRST_BYTE_OF_HASH}.yml` > [!WARNING] > In theory you can delete the `cache/queue` files as they can be re-created using `queue add` however: > - subsequent `batch` will be slow as it will need to re-process everything from scratch making an unnecessary number of I/O and API calls > - `queue summary` will no longer include your uploads. Instead `verify` will just see them as all formats being transcoded already. > It's therefore recommended to leave these files alone. > [!TIP] > The `cache/queue` can be checked into version control. It uses a flat file format so changes can easily be tracked, backed up, and even reverted using `git`. ### Output Directory The `spectrogram` command will generate spectrograms inside to `{OUTPUT}/{ARTIST} - {ALBUM} [{YEAR}] [{MEDIA} SPECTROGRAMS]/` > [!TIP] > Once you've reviewed the spectrograms you can freely delete each sectrograms directory (it can always be re-generated). The `transcode` command will transcode to `{OUTPUT}/{ARTIST} - {ALBUM} [{YEAR}] [{MEDIA} {FORMAT}]/` > [!TIP] > You can delete each transcode directory if you: > - Store the transcode elsewhere for seeding > - Don't intend to produce transcodes or cross seed to another indexer. Then `transcode` will create two `.torrent` files: - `{OUTPUT}/{ARTIST} - {ALBUM} [{YEAR}] [{MEDIA} {FORMAT}].{INDEXER}.torrent` - `{OUTPUT}/{ARTIST} - {ALBUM} [{YEAR}] [{MEDIA} {FORMAT}].torrent` > [!TIP] > You can delete the `.torrent` files if you: > - Have already uploaded to the indexer > - Don't intend to produce transcodes or cross seed to another indexer. ## Commands and Configuration > [!TIP] > **[Configuration options and the commands they apply to are documented in COMMANDS.md](COMMANDS.md)** Configuration options are sourced first from the command line arguments, then from a configuration file. By default the application loads `config.yml` from the current working directory, but this can be overridden with the `--config