Fluvio Jolt JSON library

Overview

JSON to JSON transformation where the "specification" for the transform is itself a JSON document.

Port of Java Jolt library written in Rust.

Usage Example

Add fluvio-jolt crate to your Cargo.toml file:

[dependencies]
fluvio-jolt = { version = "0.3"}

Then, for example, if you want to repack your JSON record, you can do the following:

use serde_json::{json, Value};
use fluvio_jolt::{transform, TransformSpec};

let input: Value = serde_json::from_str(r#"
    {
        "id": 1,
        "name": "John Smith",
        "account": {
            "id": 1000,
            "type": "Checking"
        }
    }
"#).unwrap();

let spec: TransformSpec =
serde_json::from_str(r#"[
    {
      "operation": "shift",
      "spec": {
        "name": "data.name",
        "account": "data.account"
      }
    }
  ]"#).unwrap();

let output = transform(input, &spec);

assert_eq!(output, json!({
    "data" : {
      "name": "John Smith",
      "account": {
        "id": 1000,
        "type": "Checking"
      }
    }
}));

Supported Operations

1. shift: copy data from the input tree and put it the output tree
2. default: apply default values to the tree
3. remove: remove data from the tree

See SPEC.md for more info on specifics of execution order and DSL grammar.

Specification

Composes a list of operation specifications. Each operation has its own DSL (Domain Specific Language) in order to facilitate its narrow job.

use fluvio_jolt::TransformSpec;

let spec: TransformSpec =
serde_json::from_str(r#"[
    {
      "operation": "shift",
      "spec": {
        "name": "data.name",
        "account": "data.account"
      }
    }
  ]"#).unwrap();

Shift operation

Specifies where the data from the input JSON should be placed in the output JSON, or in other words, how the input JSON/data should be shifted around to make the output JSON/data.

At a base level, a single shift operation is a mapping from an input path to an output path, similar to the mv command in Unix, mv /var/data /var/backup/data.

The input path is a JSON tree structure, and the output path is flattened "dot notation" path notation.

For example, given this simple input JSON:

{
    "id": 1,
    "name": "John Smith",
    "account": {
        "id": 1000,
        "type": "Checking"
    }
}

A simple spec could be constructed by copying that input, and modifying it to supply an output path for each piece of data:

{
    "id": "data.id",
    "name": "data.name",
    "account": "data.account"
}

would produce the following output JSON:

{
    "data" : {
        "id": 1,
        "name": "John Smith",
        "account": {
            "id": 1000,
            "type": "Checking"
        }
    }
}

Wildcards

The shift specification on the keys level supports wildcards and conditions:
1. * - match everything
2. name1|name2|nameN - match any of the specified names

& Wildcard

& lookup allows referencing the values captured by the * or |.

&(x,y) means go up the path x levels and get the yth match from that level.

0th match is always the entire input they and the rest are the specific things the *s matched.

& == &(0) == &(0,0) and &(x) == &(x,0)

It allows for specs to be more compact. For example, for this input:

{
    "id": 1,
    "name": "John Smith",
    "account": {
        "id": 1000,
        "type": "Checking"
    }
}

to get the output:

{
    "data" : {
        "id": 1,
        "name": "John Smith",
        "account": {
            "id": 1000,
            "type": "Checking"
        }
    }
}

the spec with wildcards would be:

{
    "*": "data.&0"
}

If you want only id and name in the output, the spec is:

{
    "id|name": "data.&(0)"
}

& wildcard also allows to dereference any level of the path of given node:

{
    "foo": {
        "bar" : {
            "baz": "new_location.&(0).&(1).&(2)" // &(0) = baz, &(1) = bar, &(2) = foo
            }
        }
    }
}

for the input:

{
    "foo": {
      "bar": {
        "baz": "value"
      }
    }
  }

will produce:

{
    "new_location": {
      "baz": {
        "bar": {
          "foo": "value"
        }
      }
    }
}

$ Wildcard

$ wildcard allows accessing matched keys from the path and use them on the right hand side.

See tests in tests/java/resources/shift for usage examples.

See java library docs here.

@ Wildcard

@ wildcard allows accessing values of matched keys from the path and use them on the right hand side.

See tests in tests/java/resources/shift for usage examples.

See java library docs here.

Default operation

Applies default values if the value is not present in the input JSON.

For example, given this simple input JSON:

{
    "phones": {
        "mobile": 01234567,
        "country": "US"
    }
}

with the following specification for default operation:

{
    "phones": {
        "mobile": 0000000,
        "code": "+1"
    }
}

the output JSON will be:

{
    "phones": {
        "mobile": 01234567,
        "country": "US",
        "code": "+1"
    }
}

As you can see, the field mobile remains not affected while the code has a default '+1' value.

Remove operation

Removes content from the input JSON. The spec structure matches the input JSON structure. The value of fields is ignored.

For example, given this simple input JSON:

{
    "phones": {
        "mobile": 01234567,
        "country": "US"
    }
}

you can remove the country by the following specification for remove operation:

{
    "phones": {
        "country": ""
    }
}

the output JSON will be:

{
    "phones": {
        "mobile": 01234567
    }
}

Contributing

If you'd like to contribute to the project, please read our Contributing guide.

License

This project is licensed under the Apache license.