# Resource Control Demo Hash Daemon

`rd-hashd` is a workload simulator for `resctl-demo` and `resctl-bench`. Its
primary goal is simulating a latency-senstive and throttleable primary
workload which can saturate the machine in all local resources.

Imagine a latency-sensitive user-request-servicing application which is load
balanced and configured to use all available resources of the machine under
full load. Under nominal load, it'd consume smaller amounts of resources and
show tighter latency profile. As load gets close to full, it'll consume most
of the machine and the latencies would increase but, hopefully, stay within
the target envelope. If the application gets stalled for whatever reasons
including resource conflicts, it'd experience latency spikes and the load
balancer would allocate it less requests until it can catch up.

`rd-hashd` simulates such workload in a self-contained manner. It sets up
test files and memory heap with random contents and keeps calculating SHA1s
using multiple worker threads. The concurrency level is modulated so that
the RPS converges on the target while not exceeding the latency target. The
RPS and latency targets can be dynamically modified. The memory access
patterns follow normal distributions, small random sleeps are injected to
emulate network interactions, and it also generates log writes.

Many aspects of `rd-hashd`'s resource consumption behaviors are
configureable and are tuned, by default, to behave similarly to a popular
Facebook web workload, especially under memory and IO contentions.

`rd-hashd` is used as the primary workload for both `resctl-demo` and
`resctl-bench`. While a single workload cannot possibly represent the many
different ways systems are used, it can successfully capture many aspects of
typical human-interactive workloads, reproduce a wide variety of resource
conflict issues and verify their remedies, and be used as an approximate
measure in evaluating hardware and operating system behaviors.

While `rd-hashd` is usually used a part of `resctl-demo` and `resctl-bench`,
it can be useful as a stand-alone pseudo workload too. For more information
on the containing projects, visit:

  https://github.com/facebookexperimental/resctl-demo


# Configuration, Report and Log Files

There are two configuration channels - command line arguments and runtime
parameters. The former can be specified as command line options or with the
`--args` file. The latter can be specified using the `--params` file and
dynamically updated while `rd-hashd` is running - just edit and save, the
changes will be applied immediately.

If the specified `--args` and/or `--params` files don't exist, they will be
created with the default values. Any configurations in the `--args` file can
be overridden on the command line and the file will be updated accordingly.
Note that only the arguments which are listed above `--args` in the help
message are saved. If `--params` is not specified, the defaults are used and
the parameters can't be updated while `rd-hashd` is running.

`rd-hashd` reports the current status in the optional `--report` file and
the hash results are saved in the optional log files in the `--log-dir`
directory.

The following will create the `--args` and `--params` files and exit.

```
$ rd-hashd --testfiles ~/rd-hashd/testfiles --args ~/rd-hashd/args.json \
  --params ~/rd-hashd/params.json --report ~/rd-hashd/report.json \
  --log-dir ~/rd-hashd/logs --interval 1 --prepare-config
```

Afterwards, `rd-hashd` can be run with the same configurations with:

```
  $ rd-hashd --args ~/rd-hashd/args.json
```


# Benchmarking

It is challenging to find the right parameters to maximize resource
utilization. To help determining the parameters, `--bench` runs a series of
tests and records the determined parameters in the specified `--args` and
`--params` files.

With the resulting parameters, `rd-hashd` should saturate CPU and memory and
use some amount of IO at the full load. RPS will be sensitive to memory and
thus IO availability and resource conflicts will lead to raised request
processing latencies and lowered RPS.

`--bench` may take over ten minutes and the system should be idle otherwise.
While it tries its best, due to long tail memory accesses and fluctuating
hardware IO performance characteristics, there is a low chance that the
resulting configuration might not be sustainable in extended runs. If
`rd-hashd` fails to keep CPU saturated and deviates significantly from the
target RPS, try lowering the runtime parameter `mem_frac`. If not enough IO
is being generated, try raising.

`--bench` preserves the existing parameters in the configuration files as
much as possible. If the benchmark behaves in an unexpected way, try
removing the configuration files to start from a clean slate.


# Usage Example

The following is an example workflow. It clears the existing configurations,
performs a benchmark to determine the parameters and then starts a normal
run.

```
  $ mkdir -p ~/rd-hashd
  $ rm -f ~/rd-hashd/*.json
  $ rd-hashd --args ~/rd-hashd/args.json --testfiles ~/rd-hashd/testfiles \
             --params ~/rd-hashd/params.json --report ~/rd-hashd/report.json \
             --log-dir ~/rd-hashd/logs --interval 1 --bench
  $ rd-hashd --args ~/rd-hashd/args.json
```