[](https://git.sr.ht/~alip/jja)
Swiss army knife for chess file formats
=======================================
[](https://en.wikipedia.org/wiki/Woman,_Life,_Freedom)
[](https://crates.io/crates/jja)
[](https://repology.org/project/jja/versions)
[](https://blog.rust-lang.org/2022/09/22/Rust-1.70.0.html)
[](https://docs.rs/jja)
[](https://builds.sr.ht/~alip/jja?)
[](https://crates.io/crates/jja)
[](https://crowdin.com/project/jja)
[](https://github.com/mkenney/software-guides/blob/master/STABILITY-BADGES.md#beta)
[](https://git.sr.ht/~alip/jja/tree/main/item/COPYING)
[](https://deps.rs/repo/sourcehut/~alip/jja)
[](https://git.sr.ht/~alip/jja)
This is ___jja___, a command line utility to interact with various chess file
formats. It is still in its early stages of development. The initial intention
of the author was to convert their opening books which were saved with ChessBase's
proprietary [CTG](https://www.chessprogramming.org/CTG) format to the free and open
[PolyGlot](http://hgm.nubati.net/book_format.html) format. Overtime they intend to
add support for other chess file formats (
[cbh](https://www.chessprogramming.org/ChessBase_%28Database%29%23Formats),
[si4](https://www.chessprogramming.org/SCID#Database_Format),
[si5](https://www.chessprogramming.org/SCID#Database_Format)
and so on).
# Demo
# Jin, Jîyan, Azadî
I've started hacking this on International Women's Day 2023, a day to honor the
achievements of women and advocate for their rights worldwide. As a person of
Kurdish heritage, I am particularly moved by the slogan "Woman, Life, Freedom",
which has become a symbol of resistance against oppression and a call for
equality. In the spirit of free software and free speech, I strive to
contribute to the creation of a more just and inclusive society, where every
human being is granted the freedom to express themselves and pursue their
dreams. I also honor the memory of Mahsa Amini, whose tragic death reminds us
of the urgent need to fight for women's freedom and safety.
More on Wikipedia,
[WikiPedia::Woman, Life, Freedom](https://en.wikipedia.org/wiki/Woman,_Life,_Freedom)
# Formats
As of version 0.7.0, ___jja___ supports reading/querying:
- [PolyGlot](http://hgm.nubati.net/book_format.html), aka `bin`
- [Arena](https://www.chessprogramming.org/ABK), aka `abk`
- [ChessBase](https://www.chessprogramming.org/CTG), aka `ctg`
- [ChessMaster](https://www.chessprogramming.org/Chessmaster), aka `obk`
(version 1 and 2, w\o,with text notes)
- [BrainLearn](https://github.com/amchess/BrainLearn), aka `exp`
opening book files, whereas it supports writing/converting to:
- [PolyGlot](http://hgm.nubati.net/book_format.html), aka `bin`
- [Arena](https://www.chessprogramming.org/ABK), aka `abk`
- [BrainLearn](https://github.com/amchess/BrainLearn), aka `exp`
opening book files.
As of version 0.5.0, ___jja___ supports exporting all the supported opening book
formats to [PGN](http://www.saremba.de/chessgml/standards/pgn/pgn-complete.htm). To
use this functionality, specify an output file with `pgn` extension as an argument
to `jja edit`.
As of version 0.8.0, ___jja___ supports exporting all positions in a
[PGN](http://www.saremba.de/chessgml/standards/pgn/pgn-complete.htm) file into an
output [EPD](https://www.chessprogramming.org/Extended_Position_Description) file.
To use this functionality, specify an input file with `pgn` extension (or a
compressed pgn: `pgn.bz2`, `pgn.gz`, `pgn.lz4`, `pgn.xz` or `pgn.zst`) and an output
file with `epd` extension as arguments to `jja edit`.
During opening book conversion, ___jja___ uses the information provided in various
input opening book formats to come up with a **move weight** which accompanies the
move in the [PolyGlot](http://hgm.nubati.net/book_format.html) opening file.
___jja___ also writes some custom numbers in the **learn** field, such as
[NAGs](https://en.wikipedia.org/wiki/Numeric_Annotation_Glyphs) during `ctg`
conversion or **priority** during `abk` conversion. You may disable this custom
usage using `--no-learn` as it may confuse other software making use of this field.
Note, [Arena](https://www.chessprogramming.org/ABK), aka `abk`, opening book file
writing support is only supported from
[ChessBase](https://www.chessprogramming.org/CTG), aka `ctg` books. Use the command
line flags `--author`, `--comment`, `--probability-priority`, `--probability-games`,
`--probability-win-percent` to configure [ABK](https://www.chessprogramming.org/ABK)
header metadata. Game statistics (minimum number of games/wins, win percentages for
both sides) are managed automatically by ___jja___.
In-place editing for [Arena](https://www.chessprogramming.org/ABK) opening books is
also possible using `-i, --in-place=SUFFIX` command line option. Conversion from
[PolyGlot](http://hgm.nubati.net/book_format.html), aka `bin`, and
[ChessMaster](https://www.chessprogramming.org/Chessmaster), aka `obk` opening books
to [Arena](https://www.chessprogramming.org/ABK), aka `abk` opening book files is
planned for a future release.
# Usage
- Use `jja info` to get brief information about the chess file.
- Use `jja find` to search for a position in a chess file.
- Use `jja edit` to edit opening book files and convert to
[PolyGlot](http://hgm.nubati.net/book_format.html) files or
[Arena](https://www.chessprogramming.org/ABK) files.
- Use `jja make` to compile
[PGN](https://en.wikipedia.org/wiki/Portable_Game_Notation) files into
[PolyGlot](http://hgm.nubati.net/book_format.html) opening books.
- Use `jja dump` to dump a [PolyGlot](http://hgm.nubati.net/book_format.html)
or [BrainLearn](https://github.com/amchess/BrainLearn) file as a stream of
[JSON](https://en.wikipedia.org/wiki/JSON) arrays.
- Use `jja restore` to restore a [PolyGlot](http://hgm.nubati.net/book_format.html)
or [BrainLearn](https://github.com/amchess/BrainLearn) file from a stream of
[JSON](https://en.wikipedia.org/wiki/JSON) arrays.
- Use `jja merge` to merge two [PolyGlot](http://hgm.nubati.net/book_format.html)
opening books.
- Use `jja match` to arrange book matches using random playouts.
- Use `jja play` to make random playouts, optionally using books.
- Use `jja digest` to calculate Zobrist hash of a given chess position.
- Use `jja perft` to calculate node count from a given position up to a given depth.
- Use `jja open` to browse [ECO](https://www.chessprogramming.org/ECO)
classification.
- Use `jja quote` to print a chess quote.
___jja___ determines the type of the file using its file extension. Files with the
extension `.bin` are considered [PolyGlot](http://hgm.nubati.net/book_format.html)
books. Files with the extension `.ctg` are considered
[ChessBase](https://www.chessprogramming.org/CTG) books. Files with the extension
`.abk` are considered [Arena](https://www.chessprogramming.org/ABK) books. Files
with extension `.obk` are considered
[ChessMaster](https://www.chessprogramming.org/Chessmaster) books. Files with
extension `.exp` are considered [BrainLearn](https://github.com/amchess/BrainLearn)
experience files.
By default if the standard output is a
[TTY](https://en.wikipedia.org/wiki/Computer_terminal), ___jja___ will display
information using fancy tables. Use `--porcelain` command line option to get the
output in [CSV](https://en.wikipedia.org/wiki/Comma-separated_values)
(comma-separated values) format instead.
# Install
To compile from source, use `cargo install jja`. This requires the [Rust
Toolchain](https://rustup.rs/) to be installed. In addition you are going to need
[OpenSSL](https://www.openssl.org/) libraries on
[UNIX](https://en.wikipedia.org/wiki/Unix) systems. Moreover you need
[liburing](https://github.com/axboe/liburing) on [Linux](https://kernel.org). If
you're on a [Linux](https://kernel.org) system older than
[5.1](https://en.wikipedia.org/wiki/Io_uring#History) or you are unable to install
[liburing](https://github.com/axboe/liburing) for another reason, you may disable
the feature by building ___jja___ with `cargo install jja --no-default-features`.
As an alternative, release builds of ___jja___ are hosted on
[chesswob.org](https://chesswob.org/jja) for
[64-bit](https://en.wikipedia.org/wiki/64-bit_computing)
[Linux](https://en.wikipedia.org/wiki/Linux) and
[Windows](https://en.wikipedia.org/wiki/Microsoft_Windows). These versions are
signed by [GnuPG](https://gnupg.org/), using key
[D076A377FB27DE70](https://keybase.io/alip/pgp_keys.asc). To install, acquire the
latest version from [chesswob.org](https://chesswob.org/jja), verify the
[checksum](https://en.wikipedia.org/wiki/Checksum) and the
[GnuPG](https://gnupg.org/)
[signature](https://en.wikipedia.org/wiki/Digital_signature):
```
$> export JJA_VERSION=0.7.1
$> export JJA_FLAVOUR=glibc
$> curl https://keybase.io/alip/pgp_keys.asc | gpg --import
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13292 100 13292 0 0 13535 0 --:--:-- --:--:-- --:--:-- 26584
gpg: key D076A377FB27DE70: public key "Ali Polatel (Caissa AI) " imported
gpg: Total number processed: 1
gpg: imported: 1
$> for f in jja-${JJA_VERSION}-${JJA_FLAVOUR}.bin{,.sha512sum,.sha512sum.asc}; do wget -q https://chesswob.org/jja/${f}; done
$> gpg --verify jja-${JJA_VERSION}.bin.sha512sum.asc jja-${JJA_VERSION}.bin.sha512sum
gpg: Signature made Sun Mar 19 20:52:41 2023 CET
gpg: using RSA key 5DF763560390A149AC6C14C7D076A377FB27DE70
gpg: Good signature from "Ali Polatel (Caissa AI) ...
$> sha512sum -c jja-${JJA_VERSION}.bin.sha512sum
jja: OK
$> sudo install -m755 jja-${JJA_VERSION}-${JJA_FLAVOUR}.bin /usr/local/bin
```
Finally you may download the builds of the latest git version via the
[SourceHut build server](https://builds.sr.ht/~alip/jja?). There're three flavours,
`windows`, `linux-glibc`, and `linux-musl`. Simply browse to the latest build and
download the artifact listed on the left. Note: these artifacts are kept for 90
days.
# Packaging Status
[](https://repology.org/project/jja/versions)
# License
__jja__ is free software: you can redistribute it and/or modify it under the
terms of the __GNU General Public License__ as published by the Free Software
Foundation, either __version 3__ of the License, or (at your option) any later
version.
This program is distributed in the hope that it will be useful, but __WITHOUT
ANY WARRANTY__; without even the implied warranty of __MERCHANTABILITY__ or
__FITNESS FOR A PARTICULAR PURPOSE__. See the GNU General Public License for
more details.
You should have received a copy of the GNU General Public License along with
this program. If not, see .
# Bugs
```
Hey you, out there beyond the wall,
Breaking bottles in the hall,
Can you help me?
```
Report bugs to ___jja___'s bug tracker at :
1. Always **be polite**, respectful, and kind:
2. Keep your final change as **small and neat** as possible:
3. Attaching poems with the bug report encourages consideration tremendously.
# PGN Book Making
Since version 0.4.0, ___jja___ can make
[PolyGlot](http://hgm.nubati.net/book_format.html) books out of
[PGN](http://www.saremba.de/chessgml/standards/pgn/pgn-complete.htm)
files. This feature is similar to `polyglot make-book` with the following
differences:
1. ___jja___ may directly read compressed
[PGN](http://www.saremba.de/chessgml/standards/pgn/pgn-complete.htm) files
.pgn.{bz2,gz,lz4,xz,zst}
2. ___jja___ can process
[PGN](http://www.saremba.de/chessgml/standards/pgn/pgn-complete.htm) files bigger
than your system's available memory, by persisting statistics in a temporary
[RocksDB](https://rocksdb.org/) database.
3. ___jja___ scales move weights by default to prevent potential overflows with huge
[PGN](http://www.saremba.de/chessgml/standards/pgn/pgn-complete.htm) files bigger
files, use `--no-scale` to disable.
4. ___jja___ can filter moves using *Filter Expressions*, allowing the user to
filter-out unwanted games and create specialised opening books.
## Filter Expressions
The filter expression string should contain filter conditions, which consist of a
tag name, a comparison operator, and a value. The following operators are supported:
- `>` (greater than)
- `>=` (greater than or equal to)
- `<` (less than)
- `<=` (less than or equal to)
- `=` (equal to)
- `!=` (not equal to)
- `=~` (regex match, case insensitive)
- `!~` (negated regex match, case insensitive)
Filter conditions can be combined using the following logical operators:
- `AND` (logical AND)
- `OR` (logical OR)
Example:
`--filter="Event =~ World AND White =~ Carlsen AND ( Result = 1-0 OR ECO = B33 )"`
Supported tags are Event, Site, Date, UTCDate, Round, Black, White, Result,
BlackElo, WhiteElo, BlackRatingDiff, WhiteRatingDiff, BlackTitle, WhiteTitle, ECO,
Opening, TimeControl, Termination, TotalPlyCount, and ScidFlags.
In addition to these are four special variables, namely, Player, Elo, Title, and
RatingDiff. These variables may be used to match the relevant header from either
one of the sides. E.g the filter:
`--filter="Player =~ Carlsen"`
is functionally equivalent to
`--filter="( White =~ Carlsen OR Black =~ Carlsen )"`
Note: The filtering is designed to be simple and fast. The tokens, including
parantheses are split by whitespace. Quoting values is not allowed. For more
sophisticated filtering needs, use pgn-extract.
### Scid Flags
Scid uses one character flags, `DWBMENPTKQ!?U123456`, for each field where:
- `D` - Deleted
- `W` - White opening
- `B` - Black opening
- `M` - Middlegame
- `E` - Endgame
- `N` - Novelty
- `P` - Pawn structure
- `T` - Tactics
- `K` - Kingside play
- `Q` - Queenside play
- `!` - Brilliancy
- `?` - Blunder
- `U` - User-defined
- `1..6` - Custom flags
It is ill-advised to rely on the order of the characters flags.
Use a regex match if/when you can.
## Tips and Tricks about PGN Book Making
0. The defaults run best on my laptop and in my personal benchmarks on the
[SourceHut build
server](https://builds.sr.ht/~alip/jja/commits/main/benchmark.yml),
they're not universal truth.
1. ___jja___ processes input PGN files in parallel. You can use this to your
advantage by giving many split PGNs as input to increase parallelism and
performance.
2. Increasing batch size is good as long as you have constant memory usage. When
threads can't keep up, you'll get increased memory usage so that's the point you
really know your limit.
3. Try increasing max open files to the point you don't get too many open files
error from your operating system. You may specify `--max-open-files=-1` to keep
files always open.
4. Try different compression algorithms for the temporary
[RocksDB](https://rocksdb.org/) database, or try disabling compression completely
if you have enough space. ___jja___ writes the temporary
[RocksDB](https://rocksdb.org/) database in the same directory as the first
pgn file argument. The default compression algorithm,
[Lz4](https://lz4.github.io/lz4/), and the default compression level, `4`, are
aimed at speedy conversion with relatively moderate space usage. If you run out
of space during conversion, try to use an algorithm like
[Zstd](https://github.com/facebook/zstd) with an "ultra" level, ie. a level
greater or equal to `20`.
5. Use filters which are processed during PGN traversal when possible. Due to the
fact that these filters are matched before writing the game data to the temporary
database, when use wisely, they may have a vast on impact space and memory costs
and therefore improve overall performance. These filters are `--filter=`,
`--max-ply=`, `--min-pieces=`, `--only-white`, and
`--only-black`.
# Acknowledgements
Thanks to ___Steinar H. Gunderson___, for publishing the [CTG
Specification](https://web.archive.org/web/20210129162445/https://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?tid=2319),
and authoring the [remoteglot](http://git.sesse.net/?p=remoteglot;a=summary) tool:
The [CTG](https://www.chessprogramming.org/CTG) probing code in ___jja___ is very
directly ported from their
[C](https://en.wikipedia.org%2Fwiki%2FC_%28programming_language%29%) probing code,
and the specification has been an enormous help in clearing up various rough edges.
Thanks to ___Fabien Letouzey___, the author of the original
[PolyGlot](http://hgm.nubati.net/book_format.html) software: The
[PolyGlot](http://hgm.nubati.net/book_format.html) probing, book making and merging
code in ___jja___ is mostly ported from their respective
[C](https://en.wikipedia.org%2Fwiki%2FC_%28programming_language%29%) code. Thanks to
___Michel Van den Bergh___, the author of pg\_utils, a collection of tools to
interact with [PolyGlot](http://hgm.nubati.net/book_format.html) opening books: The
[PolyGlot](http://hgm.nubati.net/book_format.html) book editing code of ___jja___
uses many ideas and code excerpts from pg\_utils. Thanks to ___Peter Österlund___,
the author of [DroidFish](https://www.chessprogramming.org/DroidFish): The
[ABK](https://www.chessprogramming.org/ABK) opening book interface code in ___jja___
makes use of ideas and code excerpts from
[DroidFish](https://www.chessprogramming.org/DroidFish). Thanks to ___Jens
Nissen___, the author of [ChessX](https://chessx.sourceforge.io/): The
[CTG](https://www.chessprogramming.org/CTG) and
[ABK](https://www.chessprogramming.org/ABK) probing codes in ___jja___ use ideas and
code excerpts from [ChessX](https://chessx.sourceforge.io/). Thanks to
[LiChess](https://lichess.org), the best chess website on the planet. The `quote`
command of ___jja___ has a selection of quotes imported from the
[LiChess](https://lichess.org) codebase. Thanks to ___Shane Hudson___, the author of
[Scid vs. PC](https://scidvspc.sourceforge.net/): The `jja eco` command uses the
[ECO](https://www.chessprogramming.org/ECO) classification which has been done by
the [Scid](https://scidvspc.sourceforge.net/) project. In addition, the
[PolyGlot](http://hgm.nubati.net/book_format.html) editing code of ___jja___ uses
ideas and code from [Scid](https://scidvspc.sourceforge.net/). Thanks to ___Marcus
Bufett___, the author of
[chess-tactics-cli](https://github.com/marcusbuffett/chess-tactics-cli): The
chessboard displaying code in [PolyGlot](http://hgm.nubati.net/book_format.html) and
[ABK](https://www.chessprogramming.org/ABK) edit screens is borrowed from
[chess-tactics-cli](https://github.com/marcusbuffett/chess-tactics-cli).
# ChangeLog
## 0.9.1
- upgrade `nix` crate from `0.27` to `0.28`.
- upgrade `tempfile` crate from `3.8` to `3.10`.
- upgrade `rust-embed` crate from `8.1` to `8.4`.
- upgrade `rusqlite` crate from `0.30` to `0.31`.
- upgrade `rayon` crate from `1.8` to `1.10`.
- upgrade `hostname` crate from `0.3` to `0.4`.
- Add support for restoring BrainLearn experience files from Lichess
evaluations export JSON.
- `TotalPlyCount` PGN tag is now supported in make filters.
- fix `--win,draw,loss-factor` calculation for make
- Add 50-moves detection and 3-position repetition detection to the play
subcommand.
- upgrade `once_cell` crate from `1.18` to `1.19`.
- upgrade `rayon` crate from `1.7` to `1.8`.
- upgrade `zstd` crate from `0.12` to `0.13`.
- upgrade `rusqlite` crate from `0.29` to `0.30`.
- upgrade `dialoguer` crate from `0.10` to `0.11`.
- upgrade `i18n-embed` crate from `0.13` to `0.14`.
- upgrade `rust-embed` crate from `6.8` to `8.0`.
- upgrade `built` crate from `0.6` to `0.7`.
- fix quote matching at word boundaries when no wildcard characters are given.
- merge learned `-O, --outlier-threshold=` to specify an outlier threshold
where moves with weights higher than the given threshold are filtered out from the
merged output book. This stands from the fact that if the weight differences of
the same move in two different books have a high difference, calculating any
average may not yield meaningful results. The default is 0, which disables this
feature.
- merge learned `--strategy dynmid` which stands for
`jja::merge::MergeStrategy::DynamicMidpoint`.
- merge learned `--strategy wdist` which stands for
`jja::merge::MergeStrategy::WeightedDistance`.
- upgrade `nix` crate from `0.26` to `0.27` and add the `signal` feature.
- merge learned `--strategy wmedian` which stands for
`jja::merge::MergeStrategy::WeightedMedian`.
- merge learned `--strategy entropy` which stands for
`jja::merge::MergeStrategy::Entropy`.
- merge learned `--strategy quadratic` which stands for
`jja::merge::MergeStrategy::QuadraticMean`. This strategy merges using the Quadratic
Mean (Root Mean Square) approach. The Quadratic Mean (also known as the Root Mean
Square) is a statistical measure of the magnitude of a set of numbers. It offers a
more balanced view, especially when dealing with numbers of varying magnitudes.
- merge learned `--strategy lavg` which stands for
`jja::merge::MergeStrategy::LogarithmicAverage`. This strategy merges using a
logarithmic averaging approach. Given that logarithmic functions compress large
values and expand small values, we can use them to get a merge strategy that's
sensitive to differences in smaller weights while being more resistant to
disparities in larger weights.
## 0.9.0
- merge cutoff specified using `-c, --cutoff` is now applied after the merge
strategy assigns new weights. This way, cutoff may also be used to filter out
unwanted entries after the merge.
- the default value of `edit --color-weight-blue` has been changed from `32768` to
`65280` for consistency with `--nag-weight-good`.
- merge learned new merge strategy `jja::merge::MergeStrategy::Sort` to calculate
weight based on the relative position of the move entry in sorted move entries.
- merge learned new merge strategy `jja::merge::MergeStrategy::GeometricScaling` to
calculate the weight using geometric scaling: The geometric scale focuses on
multiplying numbers together rather than adding, which can help in equalizing
disparities.
- merge learned new merge strategy `jja::merge::MergeStrategy::HarmonicMean` to
calculate the harmonic mean weight of the identical move entries in both books.
This approach tends to favor more balanced weights and is less influenced by
extreme values.
- merge learned new merge strategy `jja::merge::MergeStrategy::Sigmoid` to remap the
weights in a non-linear fashion using the sigmoid function. The idea here is to
diminish the influence of extreme values, which might be causing the
dissatisfaction in previous strategies.
- edit now properly prioritizes CTG move entries with higher combined color, NAG,
performance score such that the ordering is more close to the original.
- edit learned new commandline flags `-C, --no-colors`, and `-N, --no-nags` to avoid
using CTG move colours and NAGs to assign PolyGlot move weights or Arena move
priorities.
- merge and make commands now outputs `info` on the output book after successful run.
- merge strategy defaults to the new `pavg` strategy rather than `sum` now.
- merge learned `--strategy pavg` which stands for
`jja::merge::MergeStrategy::PercentageAverage`. This strategy merges by computing
the weighted average of percentage weights, taking into account the total number
of entries in each book. This approach gives higher importance to moves from
larger books versus smaller ones, ensuring that the resultant weights reflect the
relative contributions of each book based on its size.
- merge learned `--rescale` to rescale weights of merged book entries globally to
fit into 16 bits. This is similar to `edit --rescale` but works on the final
merged book rather than the input books.
- make now uses `estimate-num-keys` property of the temporary RocksDB database to
determine the _approximate_ unique position count rather than bulk scanning the
database twice which improves performance.
- make now uses read ahead during the bulk scan of the temporary RocksDB database.
This increases performance significantly especially on spinning disks.
The read ahead size may be configured using the `--read-ahead=` commandline
option. This defaults to `4 MB` and may be disabled by passing `0` as size.
- make now uses asynchronous I/O during the bulk scan of the temporary RocksDB
database. This increases performance significantly especially with the `io_uring`
feature enabled. This may be disabled using the new `--sync` commandline argument
or using the environment variable `JJA_SYNC`.
- drop `smallvec` crate usage, and remove dependency.
- replace `humansize` with `bytefmt` crate.
- `jja::abk::traverse_tree` function now returns a `BTreeMap` rather than a `BTreeMap` which is a
**breaking change**. Moreover this function now sorts the value vectors using
binary search which makes `abk->bin` both more memory efficient and faster.
- `jja::ctg::ByteBoard::from_position` has been replaced with the function
`from_board` which is a **breaking change**.
- `jja::ctgbook::CtgBook::process_move` function now utilizes binary search for move
table lookup improving efficiency. `jja::ctg::MOVETABLE`'s `encoding` element has
changed its type from `char` to `u8` which is a **breaking change**.
- edit converts CTG books to BIN books with 20% less memory usage using the new compact
`jja::polyglot::CompactBookEntry` data structure.
- `jja::abk::PackedSBookMoveEntry` has been renamed to `CompactSBookMoveEntry` which
is a **breaking change**.
- drop the unused functions `jja::ctgbook::CtgBook::extract_all{,2}` and the unused
type `jja::ctgbook::CtgTree` both of which are **breaking changes**.
- rename `jja::ctgbook::CtgBook::extract_map{,2}` to `extract_abk{,2}` for
consistency which is a **breaking change**.
- use `extract_bin` in `CTG->BIN` to improve efficiency
- implement new functions `jja::ctgbook::CtgBook::extract_bin{,2}`
- use `human-panic` crate for user friendly panic messages.
## 0.8.1
- upgrade `tempfile` crate from `3.7` to `3.8`.
- upgrade `clap` crate from `4.3` to `4.4`.
- info learned to print the sha256 checksum of the chess file.
- fix a memory leak in builds with the `i18n` feature disabled.
- edit now converts CTG books to ABK books around 5% times faster using 30% less
memory thanks to the new public functions `jja::ctgbook::CtgBook::extract_map{,2}`
which directly returns a `BTreeMap>` rather than processing a
`CtgTree` into a `SBookMoveEntryHashMap` as an additional step.
- info now prints information about the size of a single entry.
- info now prints information about ABK number of entries.
- New public function `jja::abkbook::AbkBook::total_entries()` to calculate the
number of entries in an Arena opening book file.
- Various memory efficiency improvements for `CTG->ABK` conversion.
- edit now reads from standard input rather than spawning the default editor if the
standard input does not refer to a terminal, ie is not a TTY. This is useful for
programmatic editing of opening books. find output may be piped to edit which
makes it practical to write entries from an opening book to another opening book
of the same type. Finally this also gives users who are unable to use the editor
functionality a chance to edit their opening books by submitting a custom CSV
file via standard input.
- replace the final `lazy_static!` usages with `once_cell` and drop the dependency
on `lazy_static` crate. Our usage of `once_cell` is currently in Rust nightly and
is hopefully soon going to land in stable Rust.
## 0.8.0
- new `perft` subcommand to count legal move paths of a given length
- upgrade `pgn-reader` crate from `0.24` to `0.25`
- upgrade `shakmaty` crate from `0.25` to `0.26`
- upgrade `tempfile` create from `3.6` to `3.7`
- **important fix** in PGN generation to avoid skip some lines due to the incorrect
usage of the repetition tracker. This effects PGN generation from all supported
opening book formats (`abk`, `bin`, `ctg`, `exp`, `obk`) so users are highly
recommended to re-export their previously exported PGNs using `jja edit
book.{abk,bin,ctg,exp,obk} -o book.pgn`.
- bump MSRV (minimal supported Rust version) from `1.64` to `1.70`.
- quote now matches at word boundaries unless the given quote pattern includes the
wildcard characters `.`, `?`, or `*`.
- jja now ignores invalid castling rights and en passant squares in PGN games' FEN
header rather than skipping the game.
- edit learned to dump all positions in a PGN file to an output EPD file. Compressed
PGN files are supported, hence the usage is
`jja edit source.pgn{,.bzip2,gz,lz4,xz,zst} -o output.epd`.
- dump learned `-e`, `--elements` to specify a list of elements to dump which
defaults to `id, position`. This option works only for PGN dumps.
- new subcommand `probe` which can be used to probe Syzygy tablebases up to 7
pieces. This is almost functionally identical to the awesome Fathom tool, but also
offers some alternative modes such as `--test --fast` to skip walking the DTZ line
at the cost of misevaluating `MaybeWin` and `MaybeLoss` positions.
- edit learned two new command line options `--max-ply=`, and
`--look-ahead=`. The former limits PGN generation to a certain number of
plies, and defaults to `1024`. The latter is used to specify the number of plies
to look ahead on PolyGlot book lookup misses during PGN generation which is useful
to generate PGNs of book generated by `--only-black|white`. Currently
`--look-ahead` only supports `0`, and `1` as argument, of which `0` being the
default value, other values will generate an `unimplemented` panic which directs
the user to report a bug. This change comes along with a change in the public
function signature of `jja::polyglotbook::PolyGlotBook::write_pgn` which is a
**breaking change**.
- The function `jja::polyglot::from_move` now takes argument a *reference* to a
`shakmarty::Move` rather than a `shakmaty::Move` which is a **breaking change**.
- The default values of some weight conversion arguments to `jja edit` have been
changed to be more intuitive, in that default value of `--color-weight-green` has
been changed from `10000` to `65520`, `--color-weight-blue` from `1000` to
`32768`, `--nag-weight-good` from `9000` to `65280`, `--nag-weight-hard` from
`10000` to `65520`, `--nag-weight-interesting` from `7500` to `61440`,
`--nag-weight--forced` from `10000` to `65520`, and `--nag-weight-only` from
`10000` to `65520`. The new defaults are empirical, and advanced users are
recommended to use `--color-weight-*`, and `--nag-weight-*` arguments with `jja
edit` for fine-tuning during `CTG->BIN` or `CTG->ABK` conversions.
- **Important fix** in `jja::ctgbook::CtgBook::read_page` function to prevent panics
due to out-of-bounds access. This fixes search & conversion with some huge CTG
books.
- The function `jja::abkbook::AbkBook::traverse_book_and_merge` function now returns
nothing rather than `Result<(), Box>` which is a **breaking
change**.
- The function `jja::chess::lines_from_tree` now returns a `Table` rather than a
`Result>` which is a **breaking change**.
- Use `anyhow` crate for error handling. This is mainly used in the main code, and
not the library code but there are points where the library code is changed, and
there are **breaking changes**: `jja::pgnbook::create_opening_book` now returns an
`anyhow::Result`, rather than
`Result>`. Similarly the function
`jja::pgn::pgn_dump` now returns `anyhow::Result<()>` rather than
`Result<(), Box>`.
- The functions `jja::system::get_progress_{bar,spinner}` now return a `ProgressBar`
rather than a `Result>`. These functions
now panic when there is an error in the template, which shouldn't happen normally.
This change is a **breaking change**.
- The function `jja::system::edit_tempfile` now returns `EditTempfileError` on error
rather than `Box`. Moreover the `EditTempfileError` enum
has a new member `EditTempfileError::InputOutputError(std::io::Error)`. These
changes are in the public API and hence are **breaking changes**.
- restore learned experimental EPD output support. This may be used in a pipeline
with dump, e.g: `jja dump -fcsv file.pgn | jja restore file.epd` to create an
Extended Position Description file of all the positions in the given Portable Game
Notation file. The EPD entries include the Zobrist hash of the positions in the `id`
field.
- fix deserializing of Chess960 castling rights in `jja::chess::deserialize_chess`.
- `jja::chess::deserialize_chess` function now panics on invalid piece indexes
rather than silently continuing.
- `hash` subcommand has been renamed to `digest`. The former will work as an alias
until the next major version bump.
## 0.7.1
- fix build with `i18n` feature disabled.
- enable the lint `#![deny(clippy::cast_lossless)]` for jja library and fix
offending code.
## 0.7.0
- enable the lint `#![deny(clippy::cast_precision_loss)]` for jja library and fix
offending code.
- edit learned to use buffered I/O, rather than direct I/O during PGN conversion to
improve efficiency, especially with huge opening books.
- edit learned to display a progress bar during PGN conversion.
- CTG interface has seen many **breaking change**s to make the interface leaner,
with regards to the recent mmap changes. As a result, the function
`jja::ctg::find_piece` returns an `Option` rather than a `Result`, the functions `jja::ctgbook::CtgBook::extract_all{,2}` return a
`CtgTree` rather than a `Result`, and the function
`jja::ctgbook::CtgBook::lookup_moves` returns an `Option>` rather
than a `Result>, std:io::Error>`.
- edit now uses a much more efficient PGN converter implementation which does not
load the whole tree of variations in memory. This makes it possible to convert
huge opening book files into PGN without running out of system memory. Moreover,
the PGN converter now respects the `-f, --fen`, and `-p, --pgn` arguments of `jja
edit` so it is possible to convert a subtree of the opening book into PGN. See the
respective issues [#14](https://todo.sr.ht/~alip/jja/14), and
[#16](https://todo.sr.ht/~alip/jja/16) for more details. This change deprecates
the function `jja::chess::pgn_from_tree`, it is recommended to use the new
`write_pgn` function of the respective opening book file.
- `jja::polyglotbook::PolyGlotBook` implementation has seen many improvements to be
a leaner interface. `PolyGlotBook::get{,_key}` functions no longer panic on
invalid indexes, rather return an `Option` rather than a `BookEntry`.
This implementation avoids an ugly hack to map a dummy anonymous memory region for
PolyGlot books with zero-size to allow creating PolyGlot books from scratch (e.g:
`touch new-file.bin && jja edit -i new-file.bin`). As these functions are public,
this is a **breaking change**.
- merge now supports merging BrainLearn experience files together. The option
`--weight-cutoff` has been renamed to `--cutoff` and now supports filtering out by
min depth in BrainLearn experience files.
- edit now supports editing BrainLearn experience files. In-place editing of such
files is also supported.
- reduce minimum supported Rust version (MSRV) from `1.70` to `1.64` for portability.
- bring back the dependency on `is_terminal` crate rather than depending on >=rust-1.70.
- downgrade `pgn-reader` crate from `0.25` to `0.24`.
- downgrade `shakmaty` crate from `0.26` to `0.25`.
- upgrade `regex` crate from `1.8` to `1.9`.
- upgrade `smallvec` crate from `1.10` to `1.11`.
- The `progress_bar` member of `jja::obkbook::ObkBook` has been removed, in return
the public functions `jja::obkbook::ObkBook::{tree,traverse_tree}` require an
optional reference to a progress bar now. This avoids a needless clone of the
progress bar and it is a **breaking change**. Moreover, the function
`jja::obkbook::ObkBook::read_moves` has been renamed to `load` which is again a
**breaking change**.
- find now memory maps the OBK opening book files rather than reading the whole file
into memory at once for efficiency. This caused a change in public function
signatures of `jja::obkbook::ObkBook::{read_moves,traverse_tree,tree}` which is a
**breaking change**.
- dump learned `binary` format to dump PGNs in PostgreSQL binary output format. This
brings in a dependency on crate `pgcopy`.
- find now memory maps the PolyGlot opening book files rather than maintaining a
`BufReader` handle to them. This removes the `book` public member of
`jja::polyglotbook::PolyGlotBook`, and changes signatures of public functions
`jja::polyglotbook::PolyGlotBook::{lookup_moves,tree}` which is a **breaking
change**. This also changes names of the public functions
`jja::polyglotbook::PolyGlotBook::{find_book_key,read_book_entry,read_book_key}`
to `jja::polyglotbook::PolyGlotBook::{find,get,get_key}` respectively which is
again a **breaking change**. Moreover `jja::polyglotbook::PolyGlotBook`'s default
iterator implementation has been changed to iterate over single book entries. New
function introduced `jja::polyglotbook::PolyGlotBook::into_iter_grouped()` may be
used to iterate over entries grouped by key.
- check for whether standard error is a TTY, rather than standard output when
displaying progress bars. This allows commands such as dump to display progress
bars during execution.
- optimize the stockfish hash function implementation, making it almost double as
fast.
- hash learned `-B`, `--benchmark`, and `-I`, `--benchmark-iterations` to benchmark
Stockfish and Zobrist hash functions. This brings in a dependency on
`benchmarking` crate.
- Avoid translating CSV headers of ABK entries in `jja find` output. Note, this
change is for CSV output only which is printed when the output is not a TTY or the
option `--porcelain=csv` is given.
- The `progress_bar` member of `jja::ctgbook::CtgBook` has been removed, in return
the public functions `jja::ctgbook::CtgBook::extract_all{,2}` require an optional
reference to a progress bar now. This avoids a needless clone of the progress bar
and it is a **breaking change**.
- avoid needless conversion to and from EPD in `jja::ctgbook::CtgBook` functions
improving efficiency. The function `jja::ctgbook::CtgBook::lookup_moves` now
accepts a `&dyn shakmaty::Position` rather than an EPD string which is a
**breaking change**.
- drop unused public functions `jja::ctg::find_piece`, `jja::ctg::decode_fen_board`,
`jja::ctg::invert_board`, `jja::ctg::needs_flipping`, `jja::ctg::flip_board`
which is a **breaking change**.
- quote now also accepts a search term as a case-insensitive regular expression as
well as a quote index.
- The `progress_bar` member of `jja::abkbook::AbkBook` has been removed, in return
many public functions of `jja::abkbook::AbkBook` require an optional reference to
a progress bar now. This avoids a needless clone of the progress bar and it is a
**breaking change**.
- find now memory maps the ABK opening book files rather than reading the whole file
into memory at once for efficiency. `jja::abkbook::AbkBook` no longer implements
`Clone` which is a **breaking change**.
- edit now uses buffered writing when converting books to the ABK opening book
format which improves efficiency. The function `jja::AbkBook::write_file` has been
changed to take an argument a `BufWriter` rather than a `File`
which is a **breaking change**.
- the positions table, `p`, in jja-0 databases now have an index on id, `p_idx` so
as to be able to query for Zobrist hash collisions more efficiently.
- info prints file type in uppercase rather than lowercase now.
- edit learned to print Polyglot book information after a successful CTG conversion,
like we already do for CTG to ABK conversions.
- CTG gained support for the full range of NAGs, `$0 - $255`, thus edit no longer
panics when stumbling upon a previously unsupported NAG. Note, only the move
assessments, `$1 - $9`, are used in Polyglot weight and ABK priority calculation
during edit. Other NAGs are merely used for display for the find subcommand. Note,
this changes the `jja::ctg::Nag` public type, and hence is a **breaking change**.
- dump learned `-f=`, `--format=` argument to choose the dump format
of PGN dumps. This option has no effect on non-PGN dumps. The function
`jja::pgn::pgn2csv` has been renamed to `jja::pgn::pgn_dump` which is a
**breaking change**.
- prioritize CTG coloured move recommendations over Numeric Annotation Glpyhs (NAGs)
in ABK priority calculation. edit learned three command-line parameters which are
`--color-priority-green`, `--color-priority-blue`, and `--color-priority-red`.
Their default values are `9`, `5`, and `1` respectively.
- make learned `--min-wins` which can be used to filter moves by their win count.
The default value is `0` which has no effect.
- prioritize CTG coloured recommendations over Numeric Annotation Glpyhs (NAGs) in
Polyglot weight calculation. edit learned three command-line parameters which are
`--color-weight-green`, `--color-weight-blue`, and `--color-weight-red`. Their
default values are `10000`, `1000`, and `1` respectively.
- CTG numerical annotation glyph `7` which is a forced move is now supported.
Previously, we mistakenly used `8` which was an only move, not a forced move.
Although the distinction is not really clear, we've implemented `ctg::Nag::Only`
in addition to `ctg::Nag::Forced`, and the corresponding command-line flags for
jja edit are `--nag-weight-only=`, and `--nag-priority-only=`. Both
defaults are identical to the default values of `--nag-weight-forced`, and
`--nag-priority-forced`.
- **important fix** `jja::CtgBook::search_position` function from mistakenly missing
some positions causing some huge CTG books, larger than ~2,5-3G, to be seen as
having 0 positions during edit, or causing position lookups to fail during find. A
multiplication overflow in `jja::CtgBook::read_page` function is also fixed.
- make informs user about the `--{win,draw,loss}-factor` values during filtering as
they're also important in determining weight, and preserval of entries.
- fix a bug in make where negative values in `--draw-factor`, and `--loss-factor`
was not counted as deficits.
- dump and restore learned an experimental PGN to an array of chess position Zobrist
keys and setups in streaming JSON. restore can save these dump into a sqlite3
database with `.jja-0` extension. The format of this file is **experimental**, and
is subject to change. Once this format is stable, the extension `.jja-1` is going
to be used. We're using this format currently only to detect Zobrist hash
collisions.
- use the [`XorShift`](https://www.jstatsoft.org/v08/i14/paper) random number
generator to randomly pick moves during random playouts using the play command. This
algorithm is cryptographically insecure but is very fast. See the benchmark
[here](https://git.sr.ht/~alip/jja/commit/b2659d05c883f33d03d3e5c717759580b528919b)
- **breaking change**: `jja::quote::print_quote` now requires a second argument which
is a boolean which specifies whether the output should be formatted with ANSI
colour codes or not. By default, when the standard output is not a TTY,
`print_quote` will now print quote, and author information without styling.
- fix PGN dumps to produce proper JSON arrays, previously the array markers `[]`
were erroneously not printed out.
## 0.6.1
- fix deserializing of promotions of BrainLearn experience file entries.
- edit learned to convert BrainLearn experience files to PGN. Note a lot of memory
may be required in converting these files. This is planned to be improved in the
future.
- fix deserializing of castling moves of BrainLearn experience file entries.
- fix `jja restore` which incorrectly wrote BrainLearn entries in big-endian rather
than little-endian.
- fix sorting of BrainLearn entries in `jja find` tree output.
- upgrade `rust-embed` crate from `6.7` to `6.8`.
- upgrade `num_cpus` crate from `1.15` to `1.16`.
- find can now correctly query BrainLearn experience files using Stockfish
compatible Zobrist hashes.
- hash learned `-S, --stockfish` to generate Stockfish compatible Zobrist hashes.
- New public module `jja::stockfish`, and new public function
`jja::stockfish::stockfish_hash` to generate Stockfish compatible Zobrist hashes.
Before calling this function `jja::stokfish::zobrist::init` function must be
called once to compute hashtables used in Zobrist hashing.
- New public functions `jja::chess::{de,}serialize_chess` to serialize/deserialize a
`shakmaty::Chess` instance to/from an array of 5 unsigned 64-bit numbers. `jja
dump` uses this functionality in PGN dumps.
## 0.6.0
- `jja::pgnfilt::Operator` and `jja::pgnfilt::LogicalOperator` implements `Eq` as
well as `PartialEq` now.
- new `restore` command to accompany the `dump` command which restores JSON
serialized PolyGlot or BrainLearn file entries into the given output file.
- fix `SIGPIPE` handling on UNIX systems so that `jja` does not panic when the
output is piped to another program such as a pager.
- find learned `-z , --hash=` to query PolyGlot opening books and
BrainLearn experience files by Zobrist hash.
- readonly support for BrainLearn experience file format. The subcommands `info`,
`dump`, and `find` are able to handle files in BrainLearn experience file format
with the extension `.exp`.
- improve polyglot key lookup by reading only the key rather than the whole entry
from file. The new public function `polyglotbook::PolyGlotBook::read_book_key` is
used for that.
- new `dump` command to dump the full contents of a PolyGlot opening book or a PGN
file. The dump format of the PolyGlot opening book is JSON, whereas for PGN files
this is CSV.
- **breaking change**: `polyglotbook::PolyglotBook::lookup_moves` has been changed
to take a zobrist hash of a chess position as an argument rather than the
`shakmaty::Chess` position itself.
- hash learned `--signed` to print Zobrist hashes as signed decimal numbers.
- new module `jja::file` which exports utilities for binary file i/o.
- **breaking change**: `jja::polyglot::entry_{from,to}_file` have been renamed to
`jja::polyglot::bin_entry_{from,to}_file`.
- use the [`XorShift`](https://www.jstatsoft.org/v08/i14/paper) random number
generator to randomly pick moves during book matches using the match command. This
algorithm is cryptographically insecure but is very fast.
- fix match command from panicing on certain cases when there is a book lookup miss.
- Print more detailed build information on long version, `--version` output.
- Memory map CTG files to speed up random access in return for increased memory
usage. This brings a new dependency upon the crate `memmap`.
- use buffered read/write in interacting with PolyGlot books which reduces the
read/write system calls by a huge margin and thereby improves performance. The
`book` member of `PolyGlotBook` is now a `BufReader` rather than a `File`
which is a **breaking change**.
- **important fix** for calculating PolyGlot compatible Zobrist hashes wrt.
en-passant legality. In PolyGlot format, en-passant moves are only pseudo-legal
whereas previously the `jja::hash::zobrist_hash` function mistakenly checked
for full legality.
- **breaking change**: new type `ctgbook::CtgTree` which holds the new return value
of the functions `CtgBook::extract_all`, and `CtgBook::extract_all2`. The `tree`
element of `CtgEntry` which were used by these functions has also been dropped and
the functions have been implemented in a much more performant way using
considerably less memory. As a result, most ctg to abk/polyglot conversions are
almost double as fast.
- **breaking change**: `ctgbook::CtgEntry` member uci's type has been changed from
`String` to `shakmaty::uci::Uci`, and the `nags` member has been renamed to `nag`
and its type has been changed from `Option` to `Option`.
- ctg move comment entries were parsed and silently discarded, they're no longer
parsed. Moreover, `ctgbook::CtgEntry` no longer has a `comment` member which is a
**breaking change**.
- **breaking change**: `ctg::colored_uci` function now accepts a
`shakmaty::uci::Uci` rather than a UCI string. The order of the function arguments
is also changed.
- ctg: new type `Ctg::Nag` to abstract CTG NAG (Numeric Annotation Glyph) entries
- obk entries with zero weight, are now assigned weight `1` during PolyGlot
conversion to prevent skipping these entries. We plan to make this
user-configurable in the future.
- use the standard `IsTerminal` trait, and drop the dependency on `is_terminal`
crate
- use the standard `to_le_bytes()`, `to_be_bytes()`, and drop the dependency on
`byteorder` crate
- hash learned `-e=`, `--enpassant-mode=` to select en-passant mode when
to include the en-passant square in Zobrist hash calculation.
- **important fix** for encoding castling moves during book conversions to the
PolyGlot format. See details in the [respective issue](https://todo.sr.ht/~alip/jja/13)
- **breaking change**: `polyglot.from_uci` now expects a `bool` argument
to correctly encode castling positions by determining whether the king to move is
on their starting square.
- bump minimum supported Rust version (MSRV) from `1.64` to `1.70` due to `shakmaty` bump
- drop dependency on the unmaintained and insecure `chrono` crate
- `PolyGlotBook` has two new public functions: `find_book_key`, and
`read_book_entry`
- hash learned `-x`, `--hex` to print hash as hexadecimal rather than decimal
- upgrade `pgn-reader` crate from `0.24` to `0.25`
- upgrade `shakmaty` crate from `0.25` to `0.26`
- optimize various ctg functions
- info learned to print the total number of positions for `ctg` opening books
- **breaking change**: `CtgBook::num_entries` has been renamed to `total_positions`.
- **breaking change**: `CtgBook::total_pages` function accepts a reference to
`self`, rather than consuming `self`. The return type is `usize` now.
- **breaking change**: Drop `close` functions of `CtgBook` and `PolyGlotBook`,
improve `CtgBook` to close the cto file immediately after open, and keep a `File`,
rather than an `Option` in `CtgBook`. The now unused `path` element of
`CtgBook` is also dropped.
- find: simplify & optimize polyglot entry lookup
- display license, version and author information in `--help` output
- set minimum supported Rust version (MSRV) to `1.64` as determined by `cargo-msrv`
- fix a bug which caused key and epd to be displayed incorrectly in editor screen
- edit: implement `--rescale` for PolyGlot books. When specified, edit will rescale
weights of all entries in the book, rather than a single entry. This is useful to
quickly correct/optimize PolyGlot books which were generated without weight scaling.
- fix build with `i18n` feature disabled
- upgrade `tempfile` crate from `3.5` to `3.6`
- upgrade `rust-embed` crate from `6.6` to `6.7`
- upgrade `once_cell` crate from `1.17` to `1.18`
- upgrade `ctrlc` crate from `3.3` to `3.4`
- **important fix** CTG to Polyglot weight conversion which caused all entries in
CTG books with missing performance information to be skipped from the output
PolyGlot book. Read more about it in the [respective
issue](https://todo.sr.ht/~alip/jja/12).
- make learned `-H`, `--hashcode` argument to skip duplicate games based on the
**HashCode** PGN tag. PGN files may be tagged using `pgn-extract --addhashcode`
## 0.5.0
- **important fix, for encoding of castling in polyglot books**. Previously, we
encoded castling as e1g1, e1c1, e8g8, and e8c8, whereas the correct encoding is
e1h1, e1a1, e8h8, and e8a8.
- when printing version with `--version`, prefer `git` version over package version
for git builds
- upgrade `clap` crate from `4.2` to `4.3`
- upgrade `ctrlc` crate from `3.2` to `3.3`
- upgrade `rocksdb` crate to `0.21.0` which bundles RocksDB-8.1.1
- document all the public code and enable the lint `#[deny(missing_docs)]`
- edit learned to export opening books in PGN format, use an output file with `pgn`
extension to export an opening book to a PGN
- add back the build dependency upon the [built](https://crates.io/crates/built) crate
- find learned `-l `, `--line=` to display lines from the opening
book as a table of opening variations reverse-sorted by cumulative weight
- make `--min-score` now accepts floating point values as argument rather than an
unsigned 64-bit integer
- make learned `--win-factor`, `--draw-factor`, and `--loss-factor` to specify
respective factors during score calculation, the defaults, `2`, `1`, and `0`
respectively, resembles the original `polyglot` tool
- find `--tree=` no longer panics on broken pipe, so it's more convenient
to use with a pager
- make learned `-p, --min-pieces` to specify the minimum number of pieces on the
board for a position to be included in the book, defaults to `8`
- merge learned `-c, --weight-cutoff` to specify the minimum weight of entries to be
included in the book
- merge learned about merge strategies `avg` and `wavg` to merge using average
weight or weighted average weight respectively; the weighted averages should be
specified for `wavg` using `-w, --weight1`, and `-W, --weight2`
- in-place editing of polyglot files allows editing empty books which makes it
practical to create polyglot book from scratch
- merge learned about merge strategies `max`, `min`, `ours`, and `sum`, default is
`sum` which adds together move weights, `max` picks the one with the maximum
weight, `min` picks up the one with the minimum weight, and `ours` always picks
the entries from the first book
- translate the README to German language
- turn i18n support into a feature which defaults to on. For static linking this
must be disabled as embed makes use of proc macros.
- drop depedency on unused [libc](https://crates.io/crates/libc) crate
## 0.4.1
- fix [docs.rs](https://docs.rs/jja) build
- update [shakmaty](https://crates.io/crates/shakmaty), and
[pgn-reader](https://crates.io/crates/pgn-reader) creates
- add initial translation to Turkish language
- add initial translation to German language
- use gettext for i18n
## 0.4.0
- replace [atty](https://crates.io/crates/atty) with the better maintained
[is-terminal](https://crates.io/crates/is-terminal) crate.
- replace [default-editor](https://crates.io/crates/default-editor) with the more
advanced [dialoguer](https://crates.io/crates/dialoguer) crate.
- replace [colored](https://crates.io/crates/colored) with the more portable
[console](https://crates.io/crates/console) crate.
- match learned `-i`, `--irreversible` to prefer irreversible moves during random
playouts. Pawn moves, captures, moves that destroy castling rights, and moves that
cede en-passant are irreversible.
- replace the [progress_bar](https://crates-io/crates/progress_bar)
crate with the more portable and advanced
[indicatif](https://crates.io/creates/indicatif) crate.
- make learned `-B `, `--batch-size=` to determine write batch size in
number of games
- Linux builds require [liburing](https://github.com/axboe/liburing) to be installed
by default. This may be disabled using `--no-default-features` on installation.
- make learned about `ScidFlags`, a set of character flags used by the
[Scid](https://scidvspc.sourceforge.net/) software, which may be used in filter
expressions
- make learned `--debug` flag to print information on matching filter expressions
- quote learned many more chess quotes which were imported from goodreads.com
- make learned `--max-open-files=` to specify the maximum number of open files
per thread opened by the temporary [RocksDB](https://rocksdb.org) database.
- match command is now multithreaded and uses as many threads as there're cpus on
the machine by default. Use `-T=`, `--threads=` or the
`JJA_NPROC` environment variable to override
- fix match to alternate colour between books during random playouts
- quote accepts an optional index argument to print the quote at the specified
index, rather than a random quote
- quote learned many more chess quotes which were imported from archive.org,
goodreads.com and nitter.net
- make filter learned special variables `Player`, `Elo`, `Title`, and `RatingDiff`
to match the relevant field from either colour
- make learned `--filter=` to filter PGN games by headers. The filtering
is designed to be simple and fast. The tokens, including parantheses are split by
whitespace. Quoting variables is not allowed. See `jja make --help` for more
information on Filter Expressions
- find learned `-t `, `--tree=` to display lines from the opening
book as a tree using the nice [termtree](https://crates.io/crates/termtree) crate
- drop the build dependency upon the [built](https://crates.io/crates/built) crate
- drop the unused dependency on the
[ħyphenation](https://crates.io/crates/hyphenation) crate
- replace [csv](https://crates.io/crates/csv) crate with the more
lightweight [quick-csv](https://crates.io/crates/quick-csv) crate
- replace [cli-table](https://crates.io/crates/cli-table) crate with the more
lightweight [prettytable-rs](https://crates.io/crates/prettytable-rs) crate
- use [pgn-reader](https://crates.io/crates/pgn-reader) crate instead of
[pgnparse](https://crates.io/crates/pgnparse) for parsing the `--pgn`
commandline option
- strip off the unneeded
[cli-table-derive](https://crates.io/crates/cli-table-derive) dependency
- fix hash subcommand option parsing causing panic
- ctg find prioritises wins & draws over performance
- make learned to preserve moves with null moves in the book with `-0`, `--null`
- make learned to avoid scaling weights using `--no-scale`
- make learned to configure compression for the rocksdb database using
`--compression={none,bzip2,lz4,lz4hc,snappy,zlib,zstd}`,`--compression-level=`,
defaults to `lz4`, level 4
- use a temporary [RocksDB](https://crates.io/crates/rocksdb) database during pgn book
make so as to better make use of memory, this reduces the performance a little,
however in return makes importing huge PGN files possible.
- new make command to make polyglot books out of pgn files, runs multithreaded
with as many threads as the cpu number of the system by default, use `-T`, `--threads`
to override, works transparently with compressed PGN files (zstd, bzip2, gunzip, lz4)
## 0.3.2
- edit writes the name of the user and jja's version as comment to Arena opening
book metadata on ChessBase to Arena opening book conversion, override with
--author, --comment
- edit displays a unicode chess board in edit tempfile
- fix book traversal on Arena opening book to PolyGlot conversion
- enable ansi colors when running on windows terminal
- drop unixisms, cross-compiles for windows
- fix yet another bug with castle decoding on polyglot read/query
- edit no longer tries to spawn the default editor if standard output is not a TTY
- fix find and edit for Arena opening book reading, move selection is on par with
the Arena GUI
- new hash command to calculate the Zobrist hash of the given position
- fix infinite loop while converting some big Arena opening book files
- improve hashing performance by avoiding double hashing using a hasher builder for
Zobrist hashes
- improve hashing performance using
[shakmaty](https://docs.rs/shakmaty/latest/shakmaty/zobrist/trait.ZobristHash.html#tymethod.zobrist_hash)
crate's [Zobrist Hashing](https://www.chessprogramming.org/Zobrist_Hashing)
implementation rather than the internal one.
## 0.3.1
- edit learned to calculate & write ABK header game statistics fields
- edit learned to convert CTG book files to ABK book files
- edit learned --author and --comment to specify metadata for Arena opening books
- edit can edit Arena opening book (abk) files in-place with -i, --in-place=SUFFIX
- support for writing Arena (abk) opening books
- open learned to wrap long ECO opening lines into multiple lines
- find no longer panics on some abk books with entries having invalid uci
- edit takes move priority into account for weight on abk to bin conversion
- match learned --move-selection={best\_move,uniform\_random,weighted\_random} to
pick move selection algoritm for book moves
- fix castle decoding on polyglot read/query
- fix error return when no positions found in abk, obk and ctg find
- fix promotion handling in ctg edit
- many improvements to ctg find (move coloring & sorting, average statistics)
- new merge command to merge two PolyGlot opening books
- new match command to arrange book matches with random playouts
## 0.3.0
- refactor code to unify various opening book reader interfaces
- support obk version 1 as well as 2 (ChessMaster books with and without notes)
- support for reading obk (ChessMaster) books and converting them to polyglot books
- do not display progress bar if standard output is not a TTY
- support for reading abk (Arena) books and converting them to polyglot books
- new play command can be used to make random playouts using opening books
## 0.2.1
- weight conversion in ctg to polyglot edit can be tuned with
--nag-weight-{good,mistake,hard,blunder,interesting,dubious,forced}=
- edit learned --no-scale to avoid scaling weights globally to fit into 16 bits
- the code is now relatively well documented
- edit --in-place now properly deletes the output temp file on interrupt
- edit filters out moves with zero weights, use -0, --null to preserve them
## 0.2.0
- edit window lists position info (key, epd, legal moves) as comment
- edit no longer silently discards illegal moves
- edit can edit PolyGlot files in-place with -i, --in-place=SUFFIX
- edit can convert CTG opening books into PolyGlot opening books
- default to start position when no --fen or --pgn is given for edit and find
- info prints number of total pages in CTG books
## 0.1.1
- new positions can be added to polyglot files
- many bugs fixed with polyglot edit
- quote command added to print a random chess quote
- open command added to query ECO classification
## 0.1.0
- edit polyglot files, only editing present positions work
- read polyglot files
- read ctg files