# jaq
[](https://crates.io/crates/jaq-core)
[](https://docs.rs/jaq-core)
[](https://www.rust-lang.org)
jaq (pronounced /ʒaːk/, like *Jacques*[^jacques]) is a clone of the JSON data processing tool [jq].
jaq aims to support a large subset of jq's syntax and operations.
You can try jaq online on the [jaq playground](https://gedenkt.at/jaq/).
Instructions for the playground can be found [here](jaq-play/).
jaq focuses on three goals:
* **Correctness**:
jaq aims to provide a more correct and predictable implementation of jq,
while preserving compatibility with jq in most cases.
* **Performance**:
I created jaq originally because I was bothered by
[the long start-up time of jq 1.6](https://github.com/jqlang/jq/issues/1411),
which amounts to about 50ms on my machine.
This can be particularly seen when processing a large number of small files.
Although the startup time has been vastly improved in jq 1.7,
jaq is still faster than jq on many other [benchmarks](#performance).
* **Simplicity**:
jaq aims to have a simple and small implementation, in order to
reduce the potential for bugs and to
facilitate contributions.
I drew inspiration from another Rust program, namely [jql].
However, unlike jql, jaq aims to closely imitate jq's syntax and semantics.
This should allow users proficient in jq to easily use jaq.
[jq]: https://jqlang.github.io/jq/
[jql]: https://github.com/yamafaktory/jql
[^jacques]: I wanted to create a tool that should be discreet and obliging, like a good waiter.
And when I think of a typical name for a (French) waiter, to my mind comes "Jacques".
Later, I found out about the old French word *jacquet*, meaning "squirrel",
which makes for a nice *ex post* inspiration for the name.
# Installation
## Binaries
You can download binaries for Linux, Mac, and Windows on the [releases page](https://github.com/01mf02/jaq/releases).
You may also install jaq using [homebrew](https://formulae.brew.sh/formula/jaq) on macOS or Linux:
$ brew install jaq
$ brew install --HEAD jaq # latest development version
Or using [scoop](https://scoop.sh/#/apps?q=jaq&id=59dbaf2bb778402cd8ec50d0ad4cdae8a6814fc3) on Windows:
$ scoop install main/jaq
## From Source
To compile jaq, you need a Rust toolchain.
See for instructions.
(Note that Rust compilers shipped with Linux distributions
may be too outdated to compile jaq.)
Any of the following commands install jaq:
$ cargo install --locked jaq
$ cargo install --locked --git https://github.com/01mf02/jaq # latest development version
On my system, both commands place the executable at `~/.cargo/bin/jaq`.
If you have cloned this repository, you can also build jaq by executing one of the commands in the cloned repository:
$ cargo build --release # places binary into target/release/jaq
$ cargo install --locked --path jaq # installs binary
jaq should work on any system supported by Rust.
If it does not, please file an issue.
# Examples
The following examples should give an impression of what jaq can currently do.
You should obtain the same outputs by replacing jaq with jq.
If not, your filing an issue would be appreciated. :)
The syntax is documented in the [jq manual].
[jq manual]: https://jqlang.github.io/jq/manual/v1.6/
Access a field:
$ echo '{"a": 1, "b": 2}' | jaq '.a'
1
Add values:
$ echo '{"a": 1, "b": 2}' | jaq 'add'
3
Construct an array from an object in two ways and show that they are equal:
$ echo '{"a": 1, "b": 2}' | jaq '[.a, .b] == [.[]]'
true
Apply a filter to all elements of an array and filter the results:
$ echo '[0, 1, 2, 3]' | jaq 'map(.*2) | [.[] | select(. < 5)]'
[0, 2, 4]
Read (slurp) input values into an array and get the average of its elements:
$ echo '1 2 3 4' | jaq -s 'add / length'
2.5
Repeatedly apply a filter to itself and output the intermediate results:
$ echo '0' | jaq '[recurse(.+1; . < 3)]'
[0, 1, 2]
Lazily fold over inputs and output intermediate results:
$ seq 1000 | jaq -n 'foreach inputs as $x (0; . + $x)'
1 3 6 10 15 [...]
# Performance
The following evaluation consists of several benchmarks that
allow comparing the performance of jaq, jq, and [gojq].
The `empty` benchmark runs `n` times the filter `empty` with null input,
serving to measure the startup time.
The `bf-fib` benchmark runs a Brainfuck interpreter written in jq,
interpreting a Brainfuck script that produces `n` Fibonacci numbers.
The other benchmarks evaluate various filters with `n` as input;
see [`bench.sh`](bench.sh) for details.
I generated the benchmark data with
`bench.sh target/release/jaq jq-1.7.1 gojq-0.12.16 | tee bench.json`
on a Linux system with an AMD Ryzen 5 5500U.[^binaries]
I then processed the results with a "one-liner" (stretching the term and the line a bit):
jq -rs '.[] | "|`\(.name)`|\(.n)|" + ([.time[] | min | (.*1000|round)? // "N/A"] | min as $total_min | map(if . == $total_min then "**\(.)**" else "\(.)" end) | join("|"))' bench.json
(Of course, you can also use jaq here instead of jq.)
Finally, I concatenated the table header with the output and piped it through `pandoc -t gfm`.
[^binaries]: The binaries for jq-1.7.1 and gojq-0.12.16 were retrieved from their GitHub release pages.
Table: Evaluation results in milliseconds ("N/A" if error or more than 10 seconds).
| Benchmark | n | jaq-2.0 | jq-1.7.1 | gojq-0.12.16 |
|-----------------|--------:|--------:|---------:|-------------:|
| `empty` | 512 | 300 | 500 | **230** |
| `bf-fib` | 13 | **440** | 1230 | 570 |
| `defs` | 100000 | **60** | N/A | 1020 |
| `upto` | 8192 | **0** | 470 | 460 |
| `reduce-update` | 16384 | **10** | 550 | 1340 |
| `reverse` | 1048576 | **40** | 690 | 280 |
| `sort` | 1048576 | **110** | 530 | 630 |
| `group-by` | 1048576 | **500** | 1920 | 1500 |
| `min-max` | 1048576 | **210** | 320 | 260 |
| `add` | 1048576 | **460** | 630 | 1300 |
| `kv` | 131072 | **110** | 150 | 230 |
| `kv-update` | 131072 | **130** | 540 | 470 |
| `kv-entries` | 131072 | **570** | 1150 | 730 |
| `ex-implode` | 1048576 | **520** | 1110 | 580 |
| `reduce` | 1048576 | **770** | 890 | N/A |
| `try-catch` | 1048576 | **290** | 320 | 370 |
| `repeat` | 1048576 | **140** | 840 | 530 |
| `from` | 1048576 | **320** | 1010 | 590 |
| `last` | 1048576 | **40** | 240 | 110 |
| `pyramid` | 524288 | **340** | 350 | 480 |
| `tree-contains` | 23 | **70** | 610 | 210 |
| `tree-flatten` | 17 | 780 | 360 | **10** |
| `tree-update` | 17 | **700** | 970 | 1340 |
| `tree-paths` | 17 | 440 | **280** | 870 |
| `to-fromjson` | 65536 | **40** | 360 | 110 |
| `ack` | 7 | **520** | 710 | 1220 |
| `range-prop` | 128 | 360 | 320 | **230** |
| `cumsum` | 1048576 | **280** | 380 | 450 |
| `cumsum-xy` | 1048576 | **430** | 470 | 710 |
The results show that
jaq-2.0 is fastest on 25 benchmarks, whereas
jq-1.7.1 is fastest on 1 benchmark and
gojq-0.12.16 is fastest on 3 benchmarks.
gojq is much faster on `tree-flatten` because it implements the filter `flatten` natively instead of by definition.
[gojq]: https://github.com/itchyny/gojq
# Features
Here is an overview that summarises:
* [x] features already implemented, and
* [ ] features not yet implemented.
[Contributions to extend jaq are highly welcome.](#contributing)
## Basics
- [x] Identity (`.`)
- [x] Recursion (`..`)
- [x] Basic data types (null, boolean, number, string, array, object)
- [x] if-then-else (`if .a < .b then .a else .b end`)
- [x] Folding (`reduce .[] as $x (0; . + $x)`, `foreach .[] as $x (0; . + $x; . + .)`)
- [x] Error handling (`try ... catch ...`)
- [x] Breaking (`label $x | f | ., break $x`)
- [x] String interpolation (`"The successor of \(.) is \(.+1)."`)
- [x] Format strings (`@json`, `@text`, `@csv`, `@tsv`, `@html`, `@sh`, `@base64`, `@base64d`)
## Paths
- [x] Indexing of arrays/objects (`.[0]`, `.a`, `.["a"]`)
- [x] Iterating over arrays/objects (`.[]`)
- [x] Optional indexing/iteration (`.a?`, `.[]?`)
- [x] Array slices (`.[3:7]`, `.[0:-1]`)
- [x] String slices
## Operators
- [x] Composition (`|`)
- [x] Variable binding (`. as $x | $x`)
- [x] Pattern binding (`. as {a: [$x, {("b", "c"): $y, $z}]} | $x, $y, $z`)
- [x] Concatenation (`,`)
- [x] Plain assignment (`=`)
- [x] Update assignment (`|=`)
- [x] Arithmetic update assignment (`+=`, `-=`, ...)
- [x] Alternation (`//`)
- [x] Logic (`or`, `and`)
- [x] Equality and comparison (`.a == .b`, `.a < .b`)
- [x] Arithmetic (`+`, `-`, `*`, `/`, `%`)
- [x] Negation (`-`)
- [x] Error suppression (`?`)
## Definitions
- [x] Basic definitions (`def map(f): [.[] | f];`)
- [x] Recursive definitions (`def r: r; r`)
## Core filters
- [x] Empty (`empty`)
- [x] Errors (`error`)
- [x] Input (`inputs`)
- [x] Length (`length`, `utf8bytelength`)
- [x] Rounding (`floor`, `round`, `ceil`)
- [x] String <-> JSON (`fromjson`, `tojson`)
- [x] String <-> integers (`explode`, `implode`)
- [x] String normalisation (`ascii_downcase`, `ascii_upcase`)
- [x] String prefix/postfix (`startswith`, `endswith`, `ltrimstr`, `rtrimstr`)
- [x] String whitespace trimming (`trim`, `ltrim`, `rtrim`)
- [x] String splitting (`split("foo")`)
- [x] Array filters (`reverse`, `sort`, `sort_by(-.)`, `group_by`, `min_by`, `max_by`)
- [x] Stream consumers (`first`, `last`, `range`, `fold`)
- [x] Stream generators (`range`, `recurse`)
- [x] Time (`now`, `fromdateiso8601`, `todateiso8601`)
- [x] More numeric filters (`sqrt`, `sin`, `log`, `pow`, ...) ([list of numeric filters](#numeric-filters))
- [ ] More time filters (`strptime`, `strftime`, `strflocaltime`, `mktime`, `gmtime`, and `localtime`)
## Standard filters
These filters are defined via more basic filters.
Their definitions are at [`std.jq`](jaq-std/src/std.jq).
- [x] Undefined (`null`)
- [x] Booleans (`true`, `false`, `not`)
- [x] Special numbers (`nan`, `infinite`, `isnan`, `isinfinite`, `isfinite`, `isnormal`)
- [x] Type (`type`)
- [x] Filtering (`select(. >= 0)`)
- [x] Selection (`values`, `nulls`, `booleans`, `numbers`, `strings`, `arrays`, `objects`, `iterables`, `scalars`)
- [x] Conversion (`tostring`, `tonumber`)
- [x] Iterable filters (`map(.+1)`, `map_values(.+1)`, `add`, `join("a")`)
- [x] Array filters (`transpose`, `first`, `last`, `nth(10)`, `flatten`, `min`, `max`)
- [x] Object-array conversion (`to_entries`, `from_entries`, `with_entries`)
- [x] Universal/existential (`all`, `any`)
- [x] Recursion (`walk`)
- [x] I/O (`input`)
- [x] Regular expressions (`test`, `scan`, `match`, `capture`, `splits`, `sub`, `gsub`)
- [x] Time (`fromdate`, `todate`)
## Numeric filters
jaq imports many filters from [libm](https://crates.io/crates/libm)
and follows their type signature.
Full list of numeric filters defined in jaq
Zero-argument filters:
- [x] `acos`
- [x] `acosh`
- [x] `asin`
- [x] `asinh`
- [x] `atan`
- [x] `atanh`
- [x] `cbrt`
- [x] `cos`
- [x] `cosh`
- [x] `erf`
- [x] `erfc`
- [x] `exp`
- [x] `exp10`
- [x] `exp2`
- [x] `expm1`
- [x] `fabs`
- [x] `frexp`, which returns pairs of (float, integer).
- [x] `gamma`
- [x] `ilogb`, which returns integers.
- [x] `j0`
- [x] `j1`
- [x] `lgamma`
- [x] `log`
- [x] `log10`
- [x] `log1p`
- [x] `log2`
- [x] `logb`
- [x] `modf`, which returns pairs of (float, float).
- [x] `nearbyint`
- [x] `pow10`
- [x] `rint`
- [x] `significand`
- [x] `sin`
- [x] `sinh`
- [x] `sqrt`
- [x] `tan`
- [x] `tanh`
- [x] `tgamma`
- [x] `trunc`
- [x] `y0`
- [x] `y1`
Two-argument filters that ignore `.`:
- [x] `atan2`
- [x] `copysign`
- [x] `drem`
- [x] `fdim`
- [x] `fmax`
- [x] `fmin`
- [x] `fmod`
- [x] `hypot`
- [x] `jn`, which takes an integer as first argument.
- [x] `ldexp`, which takes an integer as second argument.
- [x] `nextafter`
- [x] `nexttoward`
- [x] `pow`
- [x] `remainder`
- [x] `scalb`
- [x] `scalbln`, which takes as integer as second argument.
- [x] `yn`, which takes an integer as first argument.
Three-argument filters that ignore `.`:
- [x] `fma`
Example
`foreach (5, 10) as $x (1; .+$x, -.)` yields
`6, -1, 9, 1` in jq, whereas it yields
`6, 16, -6, -1, 9, 1` in jaq.
We can see that both jq and jaq yield the values `6` and `-1`
resulting from the first iteration (where `$x` is 5), namely
`1 | 5 as $x | (.+$x, -.)`.
However, jq performs the second iteration (where `$x` is 10)
*only on the last value* returned from the first iteration, namely `-1`,
yielding the values `9` and `1` resulting from
`-1 | 10 as $x | (.+$x, -.)`.
jaq yields these values too, but it also performs the second iteration
on all other values returned from the first iteration, namely `6`,
yielding the values `16` and `-6` that result from
` 6 | 10 as $x | (.+$x, -.)`.