Resource Control Benchmarks

Resource control aims to control compute resource distribution to improve reliability and utilization of a system. resctl-bench is a collection of whole-system benchmarks to evaluate resource control and hardware behaviors using realistic simulated workloads.

Comprehensive resource control involves the whole system - kernel subsystems such as cgroup2, memory management, file system and block layer, userspace system components and even the SSD. Furthermore, testing resource control end-to-end requires scenarios involving realistic workloads and monitoring their interactions. The combination makes benchmarking resource control challenging and error-prone. It's easy to slip up on a configuration and testing with real workloads can be tedious and unreliable.

resctl-bench encapsulates the whole process so that resource control benchmarks can be performed easily and reliably. It verifies and updates system configurations, reproduces resource contention scenarios with a realistic latency-sensitive workload simulator and other secondary workloads, analyzes the resulting system and workload behaviors, and generates easily understandable reports.

resctl-bench is a part of resctl-demo suite, which gives a guided tour of various resource control strategies using live scenarios built on the same components. The benchmarks implemented in resctl-bench involve concepts and components which are documented in resctl-demo in depth. For more information on resctl-demo, visit:

https://github.com/facebookexperimental/resctl-demo

Premade System Images

Comprehensive resource control has many requirements, some of which can be difficult to configure on an existing system. resctl-demo provides premade images to help getting started. Visit the following page for details:

https://facebookmicrosites.github.io/resctl-demo-website

For other installation options, visit:

https://github.com/facebookexperimental/resctl-demo

An Example Session

Let's say we want to see how well iocost can protect rd-hashd and designed a bench sequence like the following:

1. Run hashd-params to determine hashd parameters.
2. Run protection with iocost disabled to establish the baseline.
3. Run iocost-params to determine the iocost parameters.
4. Run protection with iocost enabled and compare the results.

Assuming the root device is nvme0n1 with the device number 259:0, this maps to:

   $ echo '259:0 enable=0' > /sys/fs/cgroup/io.cost.qos
   $ echo 0 > /sys/block/nvme0n1/queue/wbt_lat_usec
   $ resctl-bench -r result.json run \
     hashd-params:passive=io \
     protection:id=iocost-off,passive=io \
     iocost-params \
     protection:id=iocost-on

• We want to run the first two benchmarks with iocost off. Turn it off manually and tell the first two to not touch IO related configurations. wbt is turned off too to stay consistent with iocost enabled configuration.
• To reduce confusion, we're marking the two protection runs with different IDs.

Here are the example outputs:

• Summary: https://github.com/facebookexperimental/resctl-demo/blob/main/resctl-bench/examples/prot-iocost-off-on-summary.txt
• Format: https://github.com/facebookexperimental/resctl-demo/blob/main/resctl-bench/examples/prot-iocost-off-on-format.txt

Let's look at the result of the first benchmark - hashd-params.

   [hashd-params result] 2021-06-22 17:23:03 - 17:43:47

   System info: kernel="5.6.13-0_fbk16_5756_gdcbe47195163"
                nr_cpus=36 memory=63.9G swap=32.0G swappiness=60 zswap
                mem_profile=16 (avail=57.4G share=12.0G target=11.0G)
                passive=io

   IO info: dev=nvme0n1(259:0) model="WDC CL SN720 SDAQNTW-512G-1020" firmware="10105120" size=477G
            iosched=none wbt=off iocost=off other=off

   Params: log_bps=1.0M

   Result: hash_size=1.2M rps_max=1029 mem_actual=16.1G chunk_pages=25

After the header, the following three blocks are showing the system and bench configurations followed by the result.

• passive=io, so IO configurations were left as-are. We can see that iocost and other IO controllers were off.
• zswap is reported on. I forgot to turn it off. The subsequent benchmarks will automatically turn off zswap as they are storage focused benchmarks. It'd have been better if zswap were off here too but it shouldn't make much difference given that all data are incompressible and the primary goal of this bench is establishing the common measuring standard.
• The determined memory footprint is 16.1G, which is pretty good given that the amount of memory available to the benchmark - mem_target - was only 11G.

Let's now take a look at the first next result. Partial header:

   [protection result] "iocost-off" 2021-06-22 19:13:37 - 19:30:25
   ...
   IO info: dev=nvme0n1(259:0) model="WDC CL SN720 SDAQNTW-512G-1020" firmware="10105120" size=477G
            iosched=none wbt=off iocost=off other=off

shows that this is protection result with ID iocost-off. Skipping over to the result:

   Memory Hog Summary
   ==================

   IO Latency: R p50=885u:3.7m/49.5m p90=4.7m:12.7m/150m p99=13.1m:25.1m/350m max=30.4m:65.4m/750m
               W p50=5.0m:16.3m/99.5m p90=17.6m:28.3m/250m p99=29.0m:38.8m/450m max=48.9m:87.0m/850m

   Isolation and Request Latency Impact Distributions:

                 min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
   isol%           0  0.49  1.65  2.24 13.12 50.90 72.52 82.12 88.56 100.0 100.0 45.50 30.72
   lat-imp%        0     0     0     0  4.69 17.00 40.54 75.06 121.9 380.3 882.5 39.42 81.53

   Result: isol=45.50:30.72% lat_imp=39.42%:81.53 work_csv=100.0% missing=0.26%

For brevity, let's just focus on the isol=45.50:30.72% on the last line, which is indicating that the isolation factor - how well the RPS of the rd-hashd could be protected against interferences from memory hogs - averaged 45.5% with the standard deviation of 30.72%. Roughly speaking, our main workload's RPS halved while the system was experiencing memory shortage. For more information on the output format:

• $ resctl-bench doc protection
• https://github.com/facebookexperimental/resctl-demo/blob/main/resctl-bench/doc/protection.md

So, we now know that without iocost, the protection isn't great. The next iocost-params benchmark determines the parameters so that we can enable it. The result:

   iocost model: rbps=1348822120 rseqiops=235687 rrandiops=218614
                 wbps=601694170 wseqiops=133453 wrandiops=69308
   iocost QoS: rpct=95.00 rlat=19562 wpct=95.00 wlat=65667 min=60.00 max=100.00

iocost-params automatically applies the determined parameters for the subsequent benchmarks. The QoS parameters determined here are very naive but should do for our purpose. For determining more accurate QoS parameters and evaluating storage devices comprehensively, see the iocost-tune benchmark.

Let's see whether the protection result is any better with iocost on:

   [protection result] "iocost-on" 2021-06-22 19:38:53 - 20:02:27
   ...
   IO info: dev=nvme0n1(259:0) model="WDC CL SN720 SDAQNTW-512G-1020" firmware="10105120" size=477G
            iosched=none wbt=off iocost=on other=off
            iocost model: rbps=1348822120 rseqiops=235687 rrandiops=218614
                          wbps=601694170 wseqiops=133453 wrandiops=69308
            iocost QoS: rpct=95.00 rlat=19562 wpct=95.00 wlat=65667 min=60.00 max=100.00

The header confirms that we are testing the correct configuration. The result:

   Memory Hog Summary
   ==================

   IO Latency: R p50=164u:42.2u/415u p90=915u:827u/17.5m p99=3.4m:4.5m/97.5m max=8.8m:10.3m/250m
               W p50=158u:1.7m/41.5m p90=2.3m:9.1m/95.5m p99=5.1m:14.3m/97.5m max=8.8m:21.7m/350m

   Isolation and Request Latency Impact Distributions:

                 min   p01   p05   p10   p25   p50   p75   p90   p95   p99   max  mean stdev
   isol%           0     0 88.34 90.57 93.78 97.30 100.0 100.0 100.0 100.0 100.0 95.18 11.06
   lat-imp%        0     0  0.96  2.20  3.79  6.49 10.22 15.63 18.32 29.55 263.0  8.14  9.99

   Result: isol=95.18:11.06% lat_imp=8.14%:9.99 work_csv=42.89% missing=0.21%

The isolation factor average is now 95.18% with the standard deviation of 11.06%, a significant improvement over 45.5% without iocost.

This example shows that testing resource control behaviors using scenarios that exercise every layer of the tall stack realistically is easy and reliable with resctl-bench. For more information, explore the doc pages:

• $ resctl-bench doc --help
• https://github.com/facebookexperimental/resctl-demo/tree/main/resctl-bench/doc