# <img src="media/icon.png" alt="fiddler" width="80"/> Fiddler [![Crates.io][crates-badge]][crates-url] [![Apache 2.0 licensed][apache-badge]][apache-url] [![Documentation][doc-badge]][doc-url]

[crates-badge]: https://shields.io/crates/v/fiddler.svg
[crates-url]: https://crates.io/crates/fiddler
[apache-badge]: https://img.shields.io/badge/license-Apache2.0-blue.svg
[apache-url]: https://github.com/rc1405/fiddler/blob/main/LICENSE
[doc-badge]: https://img.shields.io/badge/docs-API%20Docs-green.svg
[doc-url]: https://docs.rs/fiddler/latest/fiddler

<br>
Fiddler is a stream processor built in rust that uses a yaml based configuration file syntax to map together inputs, processors, and outputs. 

<br>
<br>
Such as:
<br>
<br>

```
input:
    stdin: {}
pipeline:
  max_in_flight: 10
  processors:
    - python: 
        string: true
        code: |
            import json
            msg = json.loads(root)
            msg['Python'] = 'rocks'
            root = json.dumps(msg)
output:
  switch:
    - check:
        condition: '\"Hello World\" > `5`'
        output:
          validate: 
            expected: []
    - stdout: {}
```

The main inline message manipulation is provided by Python, and requires the Python shared library.

To install the Python shared library on Ubuntu:
`sudo apt install python3-dev`
To install the Python shared library on RPM based distributions (e.g. Fedora, Red Hat, SuSE), install the `python3-devel` package.

Any third party packages utilized in Python will have to be installed on the OS running fiddler.  

The message format in and out is simple, the source of the event is stored in the local variable named `root`; and the output taken from executing the code is expected to be named `root`.  By default `root` is bytes of the message, unless the argument `string: true` is proved, in which case `root` is converted to a string.
<br>
<br>
Conditional execution of steps for processing, or selection of outputs can be obtained through the use of `check` and `switch` plugins.  For processing, it looks like:
<br>
```
- switch:
    - check: 
        condition: '\"Hello World\" <= `5`'
        processors:
            - python: 
                string: true
                code: |
                    import json
                    new_string = f\"python: {root}\"
                    root = new_string
            - echo: {}
    - label: my_cool_mapping
        echo: {}
```
<br>
The condition format utilized [jmespath](https://jmespath.org/specification.html) syntax for evaluation.  As such, this can only be utilized with JSON documents.

# Install
1. Grab a release for your OS [here](https://github.com/rc1405/fiddler/releases)
1. Install with Cargo `cargo install fiddler`

# Run
1. `fiddler run -c <path_to_config_file> [ -c ... ]`

# Lint
1. `fiddler lint -c <path_to_config_file> [ -c ... ]`

# Test
Tests are expected in the format of `<filename>_test.yaml`.  I.e. if you have a configuration `input.yaml`.  The expected filename is `input_test.yaml`.  The test file syntax is as follows:
```
- name: name_of_test
  inputs:
   - list of expected input strings
  expected_outputs:
   - list of expected output strings
```

Tests can be run as `fiddler-cli test -c <path_to_configuration>.yaml`

# Build
Build with [Cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html) `cargo build --release --features all`

# Plugins
## Input
### File
```
input:
  file: 
    filename: tests/data/input.txt
    codec: ToEnd
```
* `filename`: `string`: path to file to be read
* `codec`:  `string`: Possible Values: `ToEnd`: Read entire file in one message.  `Lines`: Read file line by line

### Stdin
```
input:
  stdin: {}
```
## Processors
### Lines
```
processors:
  - lines: {}
```
### NoOp
```
processors:
  - noop: {}
```
### Python
```
processors:
  - python: 
      string: true
      code: |
          import json
          msg = json.loads(root)
          msg['Python'] = 'rocks'
          root = json.dumps(msg)
```
* `string`: `bool`: whether or not the message contents are passed as a string or bytes (default)
* `code`: `string`: python code to execute
### Check
```
processors:
  - check: 
      condition: '\"Hello World\" <= `5`'
      processors:
        - python: 
            string: true
            code: |
              import json
              new_string = f\"python: {root}\"
              root = new_string
```
* `condition`: `string`: jmespath expression to evaluate if processors are run
* `processors`: `list`: list of processors to run if condition is met
### Switch
```
processors:
  - switch:
      - check: 
          condition: '\"Hello World\" <= `5`'
          processors:
            - python: 
                string: true
                code: |
                  import json
                  new_string = f\"python: {root}\"
                  root = new_string
            - echo: {}
      - echo: {}
```
* `array`: array of processors to run, typically used with the `check` processor.  once a processor is successful, no other processors are ran
## Output
### StdOut
```
output:
  stdout: {}
```
### Check
```
output:
  check:
    condition: '\"Hello World\" > `5`'
    output:
      stdout: {}
```
* `condition`: `string`: jmespath expression to evaluate if processors are run
* `output`: `object`: output to use if check is true
### Switch
```
output:
  switch:
    - check:
        condition: '\"Hello World\" > `5`'
        output:
          validate: 
            expected: []
```
* `array`: array of outputs, typically used with the `check` output.  

### Elasticsearch
```
output:
  elasticsearch:
    url: http://example.com:9200
    username: user
    password: password
    cloud_id: some_cloud_id
    index: example
```
* `url`: if not using elasticsearch cloud, specify the url of the es cluster. i.e. http://127.0.0.1:9200
* `username`: username for authentication
* `password`: password for authentication
* `cloud_id`: if using elasticsearch cloud, specify the cloud_id
* `index`: index to insert events into

**Note:**  Variables can be collected from the environment, so when providing password, you may set it to `password: "{{ ElasticSearchPassword }}"` and ElasticSearchPassword will be pulled from the environment variables and provided to the configuration.

# Contributing

Contributions are welcome and encouraged! Contributions
come in many forms. You could:

  1. Submit a feature request or bug report as an [issue].
  2. Ask for improved documentation as an [issue].
  3. Comment on [issues that require feedback].
  4. Contribute code via [pull requests].

[issue]: https://github.com/rc1405/fiddler/issues
[issues that require feedback]: https://github.com/rc1405/fiddler/issues?q=is%3Aopen+is%3Aissue+label%3A%22feedback+wanted%22
[pull requests]: https://github.com/rc1405/fiddler/pulls

Note that unless you explicitly state otherwise, any contribution intentionally 
submitted for inclusion in fiddler by you shall be licensed under the
Apache License, Version 2.0, without any additional terms or conditions.

# Known Issues
* Cargo shows warnings on switch plugins when compiled on Windows that they are skipped when in fact they are not.
* build.rs doesn't validate Python is present when compiled on Windows
* Integration tests failing on windows GHA