========================================================================================== [hashd-params result] 2021-06-22 17:23:03 - 17:43:47 System info: kernel="5.6.13-0_fbk16_5756_gdcbe47195163" nr_cpus=36 memory=63.9G swap=32.0G swappiness=60 zswap mem_profile=16 (avail=57.4G share=12.0G target=11.0G) passive=io IO info: dev=nvme0n1(259:0) model="WDC CL SN720 SDAQNTW-512G-1020" firmware="10105120" size=477G iosched=none wbt=off iocost=off other=off Params: log_bps=1.0M Result: hash_size=1.2M rps_max=1029 mem_actual=16.1G chunk_pages=25 ========================================================================================== [protection result] "iocost-off" 2021-06-22 19:13:37 - 19:30:25 System info: kernel="5.6.13-0_fbk16_5756_gdcbe47195163" nr_cpus=36 memory=63.9G swap=32.0G swappiness=60 mem_profile=16 (avail=58.3G share=12.0G target=11.0G) passive=io IO info: dev=nvme0n1(259:0) model="WDC CL SN720 SDAQNTW-512G-1020" firmware="10105120" size=477G iosched=none wbt=off iocost=off other=off hashd params: hash_size=1.2M rps_max=1029 mem_actual=16.1G chunk_pages=25 Scenario 1/2 - Memory Hog ========================= Params: loops=2 load=1 speed=2x Info: baseline_rps=1029.00 baseline_lat=69.6m:1.7m vrate=0.00:0.00 io_usage=0.0 io_unused=0.0 hog_io_usage=0.0 hog_io_loss=0.0 hog_bytes=53.1G hog_lost_bytes=333G IO Latency Distribution: READ min p25 p50 p75 p90 p95 p99 p99.9 max cum mean stdev min 5.0u 5.0u 5.0u 35.0u 55.0u 65.0u 105u 865u 865u 35.0u 25.6u 59.1u p01 5.0u 35.0u 65.0u 75.0u 95.0u 115u 335u 1.5m 1.5m 65.0u 70.7u 131u p05 5.0u 75.0u 85.0u 105u 115u 195u 1.5m 3.5m 3.5m 95.0u 125u 299u p10 5.0u 85.0u 95.0u 115u 165u 285u 2.5m 6.5m 6.5m 105u 188u 589u p25 75.0u 115u 135u 185u 395u 1.5m 7.5m 19.5m 19.5m 165u 451u 1.8m p50 95.0u 165u 225u 505u 1.5m 4.5m 24.5m 49.5m 49.5m 495u 1.2m 4.9m p75 115u 195u 595u 1.5m 4.5m 12.5m 59.5m 99.5m 99.5m 1.5m 3.1m 10.5m p90 145u 375u 1.5m 3.5m 10.5m 30.5m 96.5m 150m 150m 5.5m 6.1m 16.1m p95 175u 555u 2.5m 5.5m 22.5m 52.5m 97.5m 250m 250m 10.5m 9.1m 22.0m p99 315u 895u 5.5m 12.5m 45.5m 96.5m 98.5m 350m 350m 50.5m 16.1m 31.2m p99.9 375u 945u 7.5m 26.5m 87.5m 97.5m 99.5m 450m 450m 150m 22.6m 40.2m p99.99 395u 965u 8.5m 27.5m 88.5m 98.5m 150m 550m 550m 350m 25.3m 48.1m p99.999 435u 995u 9.5m 28.5m 94.5m 99.5m 250m 650m 650m 450m 29.4m 60.7m max 445u 1.5m 10.5m 31.5m 98.5m 150m 350m 750m 750m 750m 37.7m 80.3m WRITE min p25 p50 p75 p90 p95 p99 p99.9 max cum mean stdev min 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 28.5m 73.5m 73.5m 5.0u 657u 6.2m p01 5.0u 5.0u 5.0u 5.0u 5.0u 15.0u 50.5m 91.5m 91.5m 5.0u 1.3m 8.9m p05 5.0u 5.0u 5.0u 5.0u 15.0u 75.0u 62.5m 94.5m 94.5m 5.0u 1.8m 10.2m p10 5.0u 5.0u 5.0u 15.0u 25.0u 2.5m 69.5m 95.5m 95.5m 5.0u 2.2m 11.4m p25 5.0u 5.0u 25.0u 35.0u 1.5m 25.5m 91.5m 96.5m 96.5m 25.0u 3.6m 14.5m p50 5.0u 25.0u 55.0u 135u 20.5m 60.5m 95.5m 99.5m 99.5m 55.0u 7.3m 20.1m p75 5.0u 35.0u 185u 14.5m 47.5m 93.5m 96.5m 150m 150m 12.5m 14.1m 26.8m p90 5.0u 45.0u 2.5m 33.5m 78.5m 95.5m 97.5m 250m 250m 45.5m 21.7m 32.5m p95 5.0u 65.0u 6.5m 45.5m 92.5m 96.5m 98.5m 350m 350m 78.5m 26.7m 37.5m p99 5.0u 105u 11.5m 59.5m 96.5m 97.5m 99.5m 450m 450m 150m 33.0m 44.9m p99.9 5.0u 125u 13.5m 66.5m 97.5m 98.5m 150m 550m 550m 350m 36.4m 52.3m p99.99 5.0u 175u 14.5m 68.5m 98.5m 99.5m 250m 650m 650m 550m 40.2m 63.5m p99.999 5.0u 205u 15.5m 69.5m 99.5m 150m 350m 750m 750m 750m 46.7m 80.3m max 15.0u 295u 17.5m 70.5m 150m 250m 450m 850m 850m 850m 60.2m 106m IO Latency: R p50=1.2m:4.9m/49.5m p90=6.1m:16.1m/150m p99=16.1m:31.2m/350m max=37.7m:80.3m/750m W p50=7.3m:20.1m/99.5m p90=21.7m:32.5m/250m p99=33.0m:44.9m/450m max=60.2m:106m/850m Slice resource stat: [ROOT] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 9.47 13.16 15.57 18.31 31.93 49.31 70.78 82.19 86.68 91.62 92.50 50.29 22.90 sys% 2.41 6.79 10.44 12.19 15.17 19.71 25.79 32.60 36.39 50.38 56.26 21.25 8.58 mem 61.9G 62.0G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 19.5M io% - - - - - - - - - - - - - rbps 279M 314M 399M 432M 536M 653M 787M 908M 965M 1.2G 7.5G 697M 501M wbps 55.5M 59.0M 183M 205M 239M 275M 317M 343M 367M 414M 1.1G 278M 81.7M cpu-some% 0.28 0.41 0.54 0.63 0.88 1.55 4.21 8.77 13.50 26.92 62.13 3.68 5.91 mem-some% 4.53 5.09 14.80 18.25 24.31 31.64 46.06 64.37 80.16 89.52 192.1 37.76 21.01 mem-full% 3.29 3.41 13.50 17.34 22.67 29.45 40.23 61.57 75.86 84.09 181.7 34.63 19.96 io-some% 1.90 2.03 6.21 7.79 11.22 16.17 27.52 47.90 57.52 80.75 149.9 22.59 18.48 io-full% 1.09 1.20 4.81 5.59 8.08 12.48 20.31 42.77 48.65 65.95 142.5 18.13 16.44 [WORKLOAD] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 0.43 0.46 0.86 1.25 2.87 27.78 51.67 58.38 65.84 81.70 82.81 29.91 23.28 sys% 0.09 0.11 0.22 0.27 0.67 2.80 4.85 6.74 7.39 9.79 10.16 3.19 2.47 mem 9.1G 9.1G 9.1G 9.1G 9.3G 9.5G 9.7G 10.5G 10.6G 11.4G 11.5G 9.6G 537M io% - - - - - - - - - - - - - rbps 1.3M 1.5M 3.6M 5.1M 9.6M 51.5M 71.1M 141M 164M 182M 204M 58.0M 49.1M wbps 680K 740K 1.3M 1.8M 3.7M 12.0M 20.7M 34.2M 42.7M 62.4M 67.6M 15.0M 13.5M cpu-some% 0.02 0.02 0.03 0.04 0.07 0.16 1.09 4.40 6.82 11.66 19.46 1.28 2.55 mem-some% 0.57 1.05 2.88 3.66 5.22 8.28 14.35 21.12 28.68 41.71 53.33 11.04 8.82 mem-full% 0.34 0.49 2.50 3.46 4.94 7.64 13.09 19.08 28.36 41.67 51.48 10.30 8.44 io-some% 0.44 0.97 1.38 1.91 2.85 4.78 10.25 16.93 24.36 41.93 48.91 8.03 8.40 io-full% 0.30 0.60 1.38 1.85 2.58 4.25 9.58 16.35 23.96 41.93 48.91 7.44 8.11 [SYSTEM] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 4.12 4.43 6.50 8.51 10.65 14.15 19.89 24.96 31.29 44.48 48.55 16.07 7.56 sys% 2.27 2.28 5.15 6.38 8.78 12.46 17.24 24.10 28.95 44.04 46.55 14.16 7.58 mem 1.5G 1.6G 2.6G 2.7G 3.5G 3.7G 4.0G 4.1G 4.1G 4.2G 4.2G 3.6G 554M io% - - - - - - - - - - - - - rbps 125M 249M 342M 385M 480M 598M 714M 830M 912M 1.1G 1.1G 606M 176M wbps 33.7M 40.5M 174M 194M 222M 257M 295M 328M 345M 356M 435M 257M 56.6M cpu-some% 0.08 0.15 0.20 0.24 0.29 0.55 1.85 4.55 6.79 15.97 21.03 1.76 2.90 mem-some% 1.33 2.26 11.92 14.22 19.82 26.19 38.61 61.14 71.19 85.38 89.20 32.34 18.79 mem-full% 0.78 1.58 11.50 13.30 18.71 25.21 36.22 59.76 69.47 83.12 86.71 30.83 18.19 io-some% 0.81 0.89 5.09 6.98 10.14 14.68 24.43 47.49 53.64 75.58 83.45 20.65 16.54 io-full% 0.76 0.84 4.69 6.40 9.08 13.30 21.48 43.21 49.84 66.62 79.36 18.41 14.66 Isolation and Request Latency Impact Distributions: min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev isol% 0 0 1.07 1.65 2.82 41.40 63.36 71.33 76.68 95.24 97.28 38.52 28.14 lat-imp% 0 0 0 0 0 11.27 45.26 87.49 306.3 557.1 882.5 46.35 106.4 Result: isol=38.52:28.14% lat_imp=46.35%:106.4 work_csv=100.0% missing=0.25% Scenario 2/2 - Memory Hog ========================= Params: loops=2 load=0.8 speed=2x Info: baseline_rps=823.20 baseline_lat=61.7m:565u vrate=0.00:0.00 io_usage=0.0 io_unused=0.0 hog_io_usage=0.0 hog_io_loss=0.0 hog_bytes=52.6G hog_lost_bytes=367G IO Latency Distribution: READ min p25 p50 p75 p90 p95 p99 p99.9 max cum mean stdev min 5.0u 5.0u 5.0u 35.0u 35.0u 45.0u 65.0u 275u 275u 35.0u 18.2u 22.0u p01 5.0u 35.0u 45.0u 65.0u 85.0u 95.0u 115u 675u 675u 65.0u 51.9u 44.6u p05 5.0u 75.0u 85.0u 95.0u 105u 115u 255u 1.5m 1.5m 95.0u 80.1u 89.7u p10 5.0u 85.0u 95.0u 105u 115u 155u 425u 2.5m 2.5m 105u 101u 147u p25 85.0u 105u 115u 155u 235u 385u 1.5m 6.5m 6.5m 155u 198u 433u p50 105u 115u 245u 425u 625u 1.5m 5.5m 18.5m 18.5m 415u 481u 1.2m p75 115u 165u 625u 1.5m 2.5m 3.5m 12.5m 47.5m 47.5m 1.5m 1.3m 3.2m p90 145u 285u 1.5m 3.5m 5.5m 7.5m 27.5m 87.5m 87.5m 4.5m 3.1m 6.7m p95 175u 545u 2.5m 5.5m 8.5m 12.5m 40.5m 98.5m 98.5m 8.5m 4.8m 9.3m p99 205u 955u 6.5m 9.5m 21.5m 38.5m 80.5m 99.5m 99.5m 35.5m 9.7m 14.8m p99.9 255u 975u 8.5m 13.5m 42.5m 74.5m 97.5m 150m 150m 150m 14.9m 22.6m p99.99 305u 985u 9.5m 15.5m 44.5m 81.5m 98.5m 250m 250m 350m 16.7m 26.0m p99.999 315u 1.5m 10.5m 16.5m 45.5m 84.5m 99.5m 350m 350m 450m 18.0m 29.6m max 365u 2.5m 11.5m 18.5m 48.5m 89.5m 150m 450m 450m 750m 22.0m 40.7m WRITE min p25 p50 p75 p90 p95 p99 p99.9 max cum mean stdev min 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 15.0u 33.5m 33.5m 5.0u 161u 2.0m p01 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 35.0u 34.5m 34.5m 5.0u 192u 2.2m p05 5.0u 5.0u 5.0u 5.0u 5.0u 15.0u 6.5m 36.5m 36.5m 5.0u 277u 2.6m p10 5.0u 5.0u 15.0u 15.0u 15.0u 25.0u 15.5m 41.5m 41.5m 15.0u 422u 3.3m p25 5.0u 15.0u 25.0u 25.0u 45.0u 75.0u 30.5m 54.5m 54.5m 25.0u 889u 5.4m p50 5.0u 25.0u 35.0u 65.0u 2.5m 13.5m 50.5m 96.5m 96.5m 55.0u 2.4m 9.8m p75 5.0u 55.0u 85.0u 4.5m 22.5m 36.5m 65.5m 97.5m 97.5m 4.5m 6.4m 15.0m p90 5.0u 95.0u 2.5m 15.5m 38.5m 64.5m 94.5m 98.5m 98.5m 27.5m 13.1m 21.7m p95 5.0u 155u 6.5m 23.5m 52.5m 80.5m 96.5m 99.5m 99.5m 50.5m 17.7m 25.5m p99 5.0u 345u 13.5m 35.5m 75.5m 96.5m 97.5m 150m 150m 150m 24.4m 29.9m p99.9 5.0u 375u 15.5m 38.5m 84.5m 97.5m 98.5m 250m 250m 250m 27.0m 32.9m p99.99 5.0u 435u 16.5m 39.5m 86.5m 98.5m 99.5m 350m 350m 450m 28.0m 35.4m p99.999 5.0u 485u 17.5m 40.5m 88.5m 99.5m 150m 450m 450m 750m 29.8m 40.3m max 15.0u 615u 18.5m 42.5m 90.5m 150m 250m 550m 550m 850m 36.1m 55.4m IO Latency: R p50=481u:1.2m/18.5m p90=3.1m:6.7m/87.5m p99=9.7m:14.8m/99.5m max=22.0m:40.7m/450m W p50=2.4m:9.8m/96.5m p90=13.1m:21.7m/98.5m p99=24.4m:29.9m/150m max=36.1m:55.4m/550m Slice resource stat: [ROOT] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 16.85 17.80 20.55 24.12 31.76 60.78 68.34 74.09 76.27 82.42 88.19 53.10 19.38 sys% 2.37 3.68 11.21 12.58 16.77 21.47 25.90 34.91 44.30 62.61 67.02 23.01 10.08 mem 62.0G 62.0G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 16.4M io% - - - - - - - - - - - - - rbps 29.6M 31.1M 372M 416M 491M 604M 715M 833M 901M 1.0G 4.3G 621M 295M wbps 8.4M 14.3M 163M 189M 215M 246M 279M 313M 327M 371M 470M 246M 56.1M cpu-some% 0.13 0.14 0.50 0.64 0.90 1.39 3.12 6.61 8.50 13.82 17.73 2.60 2.92 mem-some% 1.22 1.36 15.32 18.50 22.66 29.76 38.80 52.82 71.40 84.30 116.7 33.05 15.99 mem-full% 1.20 1.34 14.76 17.33 22.13 28.02 35.02 47.13 63.42 77.88 112.1 30.64 14.44 io-some% 0.34 0.40 5.03 6.40 8.33 12.26 18.63 28.78 39.24 61.55 92.65 16.01 12.41 io-full% 0.33 0.37 4.47 5.53 7.07 9.53 14.36 22.38 29.96 48.10 88.28 12.31 9.40 [WORKLOAD] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 3.94 3.97 5.22 5.38 7.99 38.46 49.50 53.39 55.60 57.50 58.40 31.33 19.88 sys% 0.50 0.65 0.81 0.97 1.52 3.69 5.42 6.51 6.99 9.82 12.79 3.69 2.27 mem 9.2G 9.2G 9.2G 9.2G 9.3G 9.5G 9.8G 10.1G 10.3G 10.9G 10.9G 9.6G 377M io% - - - - - - - - - - - - - rbps 6.7M 8.8M 11.8M 13.1M 16.8M 45.4M 61.6M 108M 117M 143M 185M 50.0M 35.5M wbps 716K 1.4M 2.7M 3.8M 4.8M 7.1M 10.4M 22.7M 29.5M 38.4M 42.4M 10.0M 8.1M cpu-some% 0.03 0.04 0.05 0.05 0.07 0.24 0.91 1.74 2.62 6.09 8.16 0.70 1.13 mem-some% 0.24 0.26 3.07 3.66 5.07 7.09 9.71 13.96 17.17 31.43 47.60 8.24 5.53 mem-full% 0.21 0.25 3.04 3.59 4.93 6.71 9.08 12.49 15.81 27.56 43.95 7.74 5.00 io-some% 0.27 0.31 1.34 1.66 2.31 3.47 5.75 8.54 11.34 25.76 43.52 4.78 4.75 io-full% 0.27 0.30 1.33 1.65 2.25 3.25 5.17 7.57 10.02 22.14 39.10 4.38 4.27 [SYSTEM] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 3.16 3.27 7.07 8.46 11.70 15.83 20.42 29.88 35.21 55.82 58.77 17.43 9.07 sys% 1.34 1.35 5.72 6.81 9.83 13.32 18.16 26.64 32.70 54.80 57.01 15.29 8.88 mem 2.2G 2.2G 2.9G 3.1G 3.4G 3.7G 3.9G 4.0G 4.0G 4.0G 4.0G 3.6G 389M io% - - - - - - - - - - - - - rbps 7.2M 15.3M 314M 367M 442M 550M 673M 766M 848M 920M 973M 555M 167M wbps 6.7M 11.0M 155M 183M 205M 232M 267M 297M 317M 361M 446M 234M 54.6M cpu-some% 0.01 0.01 0.18 0.22 0.31 0.55 1.43 4.15 5.57 8.98 11.72 1.36 1.93 mem-some% 0.02 0.05 11.27 13.36 18.45 24.77 32.86 46.41 58.97 72.99 81.06 27.60 14.65 mem-full% 0.01 0.05 10.81 12.94 17.95 23.98 31.11 43.53 57.20 69.76 76.47 26.41 13.70 io-some% 0.01 0.04 3.82 5.13 7.07 11.24 18.28 28.30 35.64 58.00 69.60 14.41 11.15 io-full% 0.01 0.04 3.73 4.82 6.68 10.05 15.14 22.35 31.32 46.32 56.32 12.47 8.79 Isolation and Request Latency Impact Distributions: min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev isol% 0 6.92 9.48 10.45 14.46 65.72 76.77 85.03 92.44 100.0 100.0 51.53 31.46 lat-imp% 0 0 0 3.30 11.61 20.30 37.31 63.29 92.84 317.9 380.3 32.99 47.43 Result: isol=51.53:31.46% lat_imp=32.99%:47.43 work_csv=100.0% missing=0.28% Memory Hog Summary ================== Info: baseline_rps=926.10 baseline_lat=65.7m:1.3m vrate=0.00:0.00 io_usage=0.0 io_unused=0.0 hog_io_usage=0.0 hog_io_loss=0.0 hog_bytes=106G hog_lost_bytes=701G IO Latency Distribution: READ min p25 p50 p75 p90 p95 p99 p99.9 max cum mean stdev min 5.0u 5.0u 5.0u 35.0u 45.0u 65.0u 95.0u 515u 865u 35.0u 22.2u 45.9u p01 5.0u 35.0u 55.0u 75.0u 95.0u 95.0u 265u 1.5m 1.5m 65.0u 61.9u 101u p05 5.0u 75.0u 85.0u 95.0u 115u 135u 795u 2.5m 3.5m 95.0u 104u 228u p10 5.0u 85.0u 95.0u 115u 135u 205u 1.5m 5.5m 6.5m 105u 147u 444u p25 75.0u 105u 125u 175u 275u 565u 4.5m 17.5m 19.5m 155u 333u 1.3m p50 95.0u 145u 235u 445u 1.5m 2.5m 11.5m 49.5m 49.5m 415u 885u 3.7m p75 115u 185u 615u 1.5m 3.5m 7.5m 33.5m 97.5m 99.5m 1.5m 2.3m 8.1m p90 145u 325u 1.5m 3.5m 7.5m 19.5m 71.5m 98.5m 150m 4.5m 4.7m 12.7m p95 175u 555u 2.5m 5.5m 11.5m 31.5m 97.5m 99.5m 250m 8.5m 7.1m 17.4m p99 205u 925u 5.5m 10.5m 34.5m 67.5m 98.5m 150m 350m 35.5m 13.1m 25.1m p99.9 255u 965u 7.5m 17.5m 60.5m 96.5m 99.5m 250m 450m 150m 19.0m 33.4m p99.99 305u 985u 8.5m 19.5m 67.5m 98.5m 150m 350m 550m 350m 21.3m 39.6m p99.999 315u 995u 9.5m 20.5m 68.5m 99.5m 250m 450m 650m 450m 24.1m 49.1m max 365u 1.5m 11.5m 22.5m 73.5m 150m 350m 550m 750m 750m 30.4m 65.4m WRITE min p25 p50 p75 p90 p95 p99 p99.9 max cum mean stdev min 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 105u 73.5m 73.5m 5.0u 425u 4.7m p01 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 30.5m 91.5m 91.5m 5.0u 805u 6.7m p05 5.0u 5.0u 5.0u 5.0u 15.0u 25.0u 36.5m 94.5m 94.5m 5.0u 1.1m 7.7m p10 5.0u 5.0u 5.0u 15.0u 25.0u 45.0u 42.5m 95.5m 95.5m 15.0u 1.4m 8.7m p25 5.0u 15.0u 25.0u 25.0u 55.0u 8.5m 56.5m 96.5m 96.5m 25.0u 2.3m 11.3m p50 5.0u 25.0u 45.0u 75.0u 10.5m 36.5m 90.5m 97.5m 99.5m 55.0u 5.0m 16.3m p75 5.0u 45.0u 115u 9.5m 33.5m 63.5m 96.5m 98.5m 150m 4.5m 10.5m 22.4m p90 5.0u 75.0u 2.5m 23.5m 61.5m 94.5m 97.5m 99.5m 250m 27.5m 17.6m 28.3m p95 5.0u 115u 6.5m 35.5m 78.5m 95.5m 98.5m 150m 350m 50.5m 22.5m 32.8m p99 5.0u 195u 12.5m 47.5m 95.5m 96.5m 99.5m 250m 450m 150m 29.0m 38.8m p99.9 5.0u 225u 15.5m 53.5m 97.5m 97.5m 150m 350m 550m 250m 32.0m 44.5m p99.99 5.0u 275u 16.5m 54.5m 98.5m 98.5m 250m 450m 650m 450m 34.5m 52.6m p99.999 5.0u 325u 17.5m 56.5m 99.5m 99.5m 350m 550m 750m 750m 38.8m 65.3m max 15.0u 345u 18.5m 57.5m 150m 150m 450m 650m 850m 850m 48.9m 87.0m IO Latency: R p50=885u:3.7m/49.5m p90=4.7m:12.7m/150m p99=13.1m:25.1m/350m max=30.4m:65.4m/750m W p50=5.0m:16.3m/99.5m p90=17.6m:28.3m/250m p99=29.0m:38.8m/450m max=48.9m:87.0m/850m Slice resource stat: [ROOT] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 9.47 13.91 18.12 20.93 31.93 54.97 69.47 76.93 82.69 89.97 92.50 51.76 21.15 sys% 2.37 3.71 10.94 12.32 15.96 20.38 25.87 33.93 37.58 52.14 67.02 22.13 9.45 mem 61.9G 62.0G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 62.1G 17.9M io% - - - - - - - - - - - - - rbps 29.6M 279M 378M 427M 508M 627M 748M 874M 940M 1.1G 7.5G 665M 437M wbps 8.4M 55.5M 170M 195M 224M 258M 299M 331M 350M 392M 1.1G 262M 71.5M cpu-some% 0.13 0.28 0.52 0.64 0.90 1.45 3.45 7.85 10.69 19.25 62.13 3.21 5.04 mem-some% 1.22 4.53 15.24 18.50 23.69 30.38 41.71 60.83 74.55 86.85 192.1 35.51 19.19 mem-full% 1.20 3.29 13.91 17.34 22.31 28.41 37.55 56.44 68.76 83.57 181.7 32.71 17.69 io-some% 0.34 1.90 5.85 6.84 9.41 14.31 22.83 38.36 51.13 76.06 149.9 19.30 16.18 io-full% 0.33 1.09 4.74 5.56 7.32 10.86 16.95 30.14 43.51 61.59 142.5 15.19 13.65 [WORKLOAD] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 0.43 0.68 1.27 1.43 7.45 34.71 50.12 55.53 58.38 78.23 82.81 30.72 21.53 sys% 0.09 0.18 0.27 0.42 1.34 3.13 5.08 6.64 7.39 9.79 12.79 3.44 2.38 mem 9.1G 9.1G 9.1G 9.1G 9.3G 9.5G 9.7G 10.2G 10.5G 11.1G 11.5G 9.6G 461M io% - - - - - - - - - - - - - rbps 1.3M 2.2M 5.1M 7.5M 16.2M 47.5M 70.1M 120M 147M 185M 2.7G 65.7M 162M wbps 680K 956K 1.7M 2.5M 4.5M 8.6M 16.3M 28.6M 35.6M 54.6M 150M 12.8M 13.2M cpu-some% 0.02 0.03 0.04 0.05 0.07 0.22 0.95 2.73 5.26 10.26 42.95 1.11 2.88 mem-some% 0.24 0.57 3.07 3.66 5.13 7.31 11.28 18.70 24.29 42.33 62.02 9.83 8.05 mem-full% 0.21 0.34 2.87 3.55 4.94 7.04 10.50 16.85 22.99 41.67 58.37 9.20 7.55 io-some% 0.27 0.44 1.38 1.80 2.58 4.08 7.03 13.22 21.22 41.93 65.40 6.59 7.68 io-full% 0.27 0.37 1.38 1.79 2.43 3.75 6.29 12.17 19.16 39.10 63.91 6.09 7.35 [SYSTEM] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 1.96 3.27 6.67 8.46 10.89 15.17 20.20 26.94 32.01 45.63 58.77 16.69 8.45 sys% 1.04 1.52 5.15 6.58 9.28 12.81 17.81 24.61 30.40 44.85 57.01 14.66 8.33 mem 1.5G 1.8G 2.6G 3.0G 3.5G 3.7G 3.9G 4.1G 4.1G 4.1G 4.2G 3.6G 475M io% - - - - - - - - - - - - - rbps 7.2M 125M 335M 373M 450M 573M 689M 811M 898M 1.1G 4.8G 598M 296M wbps 6.7M 36.7M 159M 185M 212M 245M 280M 317M 335M 361M 900M 246M 64.0M cpu-some% 0.01 0.08 0.19 0.24 0.30 0.55 1.69 4.50 6.42 15.26 32.18 1.67 3.00 mem-some% 0.02 1.33 11.80 13.72 18.90 25.73 36.43 56.56 69.03 85.38 187.2 30.64 19.49 mem-full% 0.01 0.78 11.32 13.18 18.26 24.70 34.00 55.21 66.27 83.12 180.5 29.24 18.54 io-some% 0.01 0.81 4.56 5.83 8.32 13.23 20.34 36.62 51.67 75.58 167.8 18.11 17.03 io-full% 0.01 0.76 4.01 5.44 7.47 11.84 17.99 31.75 46.20 65.03 165.4 15.99 15.26 Isolation and Request Latency Impact Distributions: min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev isol% 0 0.49 1.65 2.24 13.12 50.90 72.52 82.12 88.56 100.0 100.0 45.50 30.72 lat-imp% 0 0 0 0 4.69 17.00 40.54 75.06 121.9 380.3 882.5 39.42 81.53 Result: isol=45.50:30.72% lat_imp=39.42%:81.53 work_csv=100.0% missing=0.26% ========================================================================================== [iocost-params result] 2021-06-22 19:30:28 - 19:38:52 System info: kernel="5.6.13-0_fbk16_5756_gdcbe47195163" nr_cpus=36 memory=63.9G swap=32.0G swappiness=60 mem_profile=16 (avail=58.3G share=12.0G target=11.0G) IO info: dev=nvme0n1(259:0) model="WDC CL SN720 SDAQNTW-512G-1020" firmware="10105120" size=477G iosched=none wbt=off iocost=off other=off iocost model: rbps=1348822120 rseqiops=235687 rrandiops=218614 wbps=601694170 wseqiops=133453 wrandiops=69308 iocost QoS: rpct=95.00 rlat=19562 wpct=95.00 wlat=65667 min=60.00 max=100.00 ========================================================================================== [protection result] "iocost-on" 2021-06-22 19:38:53 - 20:02:27 System info: kernel="5.6.13-0_fbk16_5756_gdcbe47195163" nr_cpus=36 memory=63.9G swap=32.0G swappiness=60 mem_profile=16 (avail=58.3G share=12.0G target=11.0G) IO info: dev=nvme0n1(259:0) model="WDC CL SN720 SDAQNTW-512G-1020" firmware="10105120" size=477G iosched=none wbt=off iocost=on other=off iocost model: rbps=1348822120 rseqiops=235687 rrandiops=218614 wbps=601694170 wseqiops=133453 wrandiops=69308 iocost QoS: rpct=95.00 rlat=19562 wpct=95.00 wlat=65667 min=60.00 max=100.00 hashd params: hash_size=1.2M rps_max=1029 mem_actual=16.1G chunk_pages=25 Scenario 1/2 - Memory Hog ========================= Params: loops=2 load=1 speed=2x Info: baseline_rps=1029.00 baseline_lat=67.8m:1.1m vrate=99.79:0.56 io_usage=276.8 io_unused=323.0 hog_io_usage=39.6 hog_io_loss=1121.8 hog_bytes=22.1G hog_lost_bytes=627G IO Latency Distribution: READ min p25 p50 p75 p90 p95 p99 p99.9 max cum mean stdev min 5.0u 5.0u 5.0u 35.0u 45.0u 65.0u 95.0u 95.0u 95.0u 25.0u 23.0u 22.1u p01 5.0u 55.0u 75.0u 75.0u 75.0u 85.0u 105u 105u 125u 75.0u 59.9u 26.1u p05 5.0u 75.0u 85.0u 95.0u 95.0u 105u 125u 135u 145u 85.0u 83.1u 19.3u p10 5.0u 95.0u 95.0u 105u 105u 115u 145u 155u 155u 95.0u 96.9u 15.7u p25 5.0u 105u 115u 115u 135u 165u 165u 165u 165u 115u 117u 16.0u p50 75.0u 145u 165u 185u 205u 235u 365u 415u 415u 165u 170u 41.2u p75 85.0u 245u 325u 445u 555u 635u 1.5m 1.5m 1.5m 385u 366u 192u p90 105u 525u 725u 995u 1.5m 2.5m 3.5m 6.5m 17.5m 935u 909u 882u p95 115u 735u 1.5m 1.5m 2.5m 3.5m 6.5m 13.5m 38.5m 1.5m 1.6m 1.8m p99 225u 1.5m 2.5m 4.5m 5.5m 6.5m 15.5m 90.5m 97.5m 4.5m 3.4m 5.5m p99.9 265u 2.5m 4.5m 6.5m 8.5m 9.5m 20.5m 94.5m 98.5m 10.5m 5.1m 6.9m p99.99 325u 3.5m 5.5m 7.5m 9.5m 11.5m 26.5m 96.5m 99.5m 42.5m 6.2m 7.7m p99.999 335u 4.5m 6.5m 8.5m 10.5m 12.5m 27.5m 98.5m 150m 150m 7.2m 9.0m max 445u 5.5m 7.5m 9.5m 12.5m 14.5m 29.5m 150m 250m 250m 8.7m 13.0m WRITE min p25 p50 p75 p90 p95 p99 p99.9 max cum mean stdev min 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 0 p01 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 15.0u 15.0u 15.0u 5.0u 5.1u 1.1u p05 5.0u 5.0u 5.0u 5.0u 5.0u 15.0u 25.0u 35.0u 35.0u 5.0u 6.0u 3.6u p10 5.0u 5.0u 15.0u 15.0u 15.0u 25.0u 35.0u 55.0u 215u 15.0u 13.6u 9.8u p25 5.0u 15.0u 25.0u 25.0u 35.0u 35.0u 145u 345u 3.5m 25.0u 31.1u 132u p50 5.0u 35.0u 35.0u 55.0u 85.0u 175u 475u 15.5m 31.5m 35.0u 169u 1.5m p75 5.0u 65.0u 105u 235u 335u 345u 15.5m 72.5m 94.5m 85.0u 841u 6.0m p90 5.0u 185u 335u 345u 975u 10.5m 58.5m 76.5m 95.5m 345u 2.1m 8.6m p95 5.0u 315u 345u 365u 4.5m 15.5m 63.5m 92.5m 96.5m 2.5m 2.9m 10.8m p99 5.0u 345u 385u 725u 12.5m 22.5m 78.5m 94.5m 97.5m 36.5m 4.3m 13.2m p99.9 5.0u 365u 445u 1.5m 13.5m 24.5m 87.5m 96.5m 98.5m 99.5m 4.7m 13.8m p99.99 5.0u 375u 485u 2.5m 14.5m 25.5m 93.5m 97.5m 150m 150m 5.2m 14.8m p99.999 5.0u 395u 645u 3.5m 15.5m 26.5m 95.5m 99.5m 250m 250m 6.0m 16.8m max 15.0u 465u 2.5m 5.5m 18.5m 28.5m 150m 150m 350m 350m 7.8m 22.7m IO Latency: R p50=170u:41.2u/415u p90=909u:882u/17.5m p99=3.4m:5.5m/97.5m max=8.7m:13.0m/250m W p50=169u:1.5m/31.5m p90=2.1m:8.6m/95.5m p99=4.3m:13.2m/97.5m max=7.8m:22.7m/350m Slice resource stat: [ROOT] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 63.05 73.19 80.89 83.62 85.90 87.64 89.37 90.70 91.59 92.95 93.30 87.20 3.49 sys% 3.16 3.95 4.84 5.39 6.19 7.72 9.65 12.03 13.53 16.34 20.61 8.23 2.71 mem 61.8G 61.9G 61.9G 62.0G 62.0G 62.0G 62.1G 62.1G 62.1G 62.1G 62.1G 62.0G 55.6M io% 12.61 17.39 25.08 27.90 34.95 44.29 54.37 66.79 75.68 99.59 719.6 47.27 31.70 rbps 131M 161M 211M 238M 295M 383M 490M 604M 718M 986M 6.9G 420M 316M wbps 19.3M 22.3M 31.0M 35.9M 46.2M 62.0M 83.3M 103M 121M 165M 817M 68.0M 41.7M cpu-some% 0.42 1.00 3.03 3.88 5.65 7.72 10.31 13.28 15.63 19.11 164.8 8.48 7.39 mem-some% 4.34 5.34 7.52 8.81 11.53 18.78 45.90 74.92 85.55 97.62 264.6 31.58 27.82 mem-full% 2.35 2.92 4.09 4.75 6.27 8.48 12.73 16.11 18.11 23.11 111.7 9.82 6.19 io-some% 2.19 3.56 4.43 5.59 7.66 13.40 43.68 74.29 85.50 97.43 195.4 27.99 28.02 io-full% 1.16 1.68 2.13 2.45 3.00 4.31 7.99 11.62 13.46 16.82 57.45 5.91 4.37 [WORKLOAD] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 56.74 66.31 74.19 76.32 78.58 80.35 82.02 83.63 84.37 85.54 87.59 79.94 3.40 sys% 1.58 1.68 1.82 1.88 2.01 2.31 2.92 3.69 4.21 5.35 7.61 2.59 0.83 mem 10.8G 10.8G 11.0G 11.1G 11.2G 11.3G 11.3G 11.4G 11.4G 11.5G 11.7G 11.3G 141M io% 2.20 5.25 6.73 7.40 8.41 10.16 12.21 15.33 18.17 24.31 27.40 10.86 3.59 rbps 22.8M 48.3M 64.3M 68.3M 76.8M 88.9M 106M 142M 191M 256M 297M 99.5M 39.4M wbps 576K 2.6M 4.1M 5.3M 7.5M 11.7M 18.1M 25.6M 29.0M 42.4M 53.1M 13.9M 8.5M cpu-some% 0.30 0.61 2.00 2.79 4.16 5.54 7.24 8.84 9.88 12.00 14.05 5.72 2.37 mem-some% 0.20 0.73 1.21 1.49 2.04 2.80 4.37 6.20 7.68 12.23 20.68 3.53 2.41 mem-full% 0.12 0.44 0.67 0.93 1.25 1.80 2.83 4.10 5.32 8.53 18.60 2.31 1.81 io-some% 0.18 0.51 0.88 1.13 1.45 1.90 2.75 4.04 5.04 8.83 19.83 2.38 1.81 io-full% 0.14 0.42 0.64 0.87 1.11 1.43 1.95 2.83 3.63 6.40 17.94 1.75 1.40 [SYSTEM] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 0.72 0.82 1.28 1.52 2.33 3.51 5.28 7.18 9.17 11.28 13.49 4.06 2.35 sys% 0.40 0.53 0.84 1.02 1.45 2.15 3.31 4.75 5.76 7.05 9.34 2.57 1.54 mem 1.4G 1.5G 1.7G 1.7G 1.8G 1.8G 1.9G 2.1G 2.2G 2.4G 2.4G 1.9G 158M io% 9.62 10.46 16.59 19.23 25.35 33.71 42.63 51.26 59.52 75.41 83.50 34.96 13.19 rbps 72.2M 94.6M 134M 158M 221M 303M 396M 486M 567M 767M 892M 321M 139M wbps 12.8M 16.0M 21.2M 25.3M 33.5M 45.8M 63.6M 82.3M 97.7M 142M 179M 51.4M 25.0M cpu-some% 0.12 0.33 0.73 1.12 1.68 2.64 4.02 6.12 7.88 10.71 13.68 3.23 2.22 mem-some% 2.61 4.49 5.93 7.06 11.17 20.92 61.91 88.90 94.33 98.51 146.9 36.12 31.07 mem-full% 2.43 3.80 5.09 6.17 9.34 18.66 55.76 84.29 89.49 95.80 132.8 33.18 29.77 io-some% 1.82 3.63 5.13 6.37 10.37 19.52 62.59 89.47 95.45 98.89 145.0 35.84 31.73 io-full% 1.79 3.35 4.83 5.98 9.28 18.22 57.35 84.61 89.94 95.68 130.7 33.35 30.07 Isolation and Request Latency Impact Distributions: min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev isol% 45.77 84.06 87.66 89.30 91.84 94.46 96.50 98.35 99.22 100.0 100.0 93.89 4.61 lat-imp% 0 0 2.22 3.77 6.35 9.14 13.15 17.37 20.01 29.63 263.0 10.74 12.70 Result: isol=93.89:4.61% lat_imp=10.74%:12.70 work_csv=46.15% missing=0.27% Scenario 2/2 - Memory Hog ========================= Params: loops=2 load=0.8 speed=2x Info: baseline_rps=823.20 baseline_lat=61.6m:712u vrate=100.00:0.00 io_usage=237.8 io_unused=362.2 hog_io_usage=45.7 hog_io_loss=1167.5 hog_bytes=24.3G hog_lost_bytes=621G IO Latency Distribution: READ min p25 p50 p75 p90 p95 p99 p99.9 max cum mean stdev min 5.0u 5.0u 5.0u 35.0u 35.0u 45.0u 65.0u 65.0u 75.0u 25.0u 19.4u 16.8u p01 5.0u 45.0u 55.0u 75.0u 75.0u 75.0u 85.0u 95.0u 95.0u 65.0u 50.5u 23.7u p05 5.0u 85.0u 85.0u 85.0u 85.0u 85.0u 95.0u 105u 105u 85.0u 80.5u 16.1u p10 5.0u 95.0u 95.0u 95.0u 95.0u 95.0u 105u 115u 115u 95.0u 92.5u 13.3u p25 95.0u 105u 105u 115u 115u 125u 135u 155u 165u 115u 109u 8.8u p50 105u 125u 145u 175u 205u 235u 315u 385u 405u 165u 157u 42.3u p75 115u 225u 315u 455u 585u 655u 1.5m 2.5m 3.5m 385u 372u 245u p90 135u 525u 725u 995u 1.5m 1.5m 3.5m 6.5m 11.5m 935u 921u 763u p95 165u 725u 1.5m 1.5m 2.5m 3.5m 5.5m 14.5m 20.5m 1.5m 1.6m 1.4m p99 265u 1.5m 2.5m 4.5m 6.5m 7.5m 10.5m 32.5m 35.5m 4.5m 3.4m 2.9m p99.9 615u 2.5m 4.5m 6.5m 8.5m 11.5m 20.5m 71.5m 74.5m 10.5m 5.5m 5.4m p99.99 645u 4.5m 5.5m 7.5m 10.5m 12.5m 23.5m 72.5m 82.5m 36.5m 6.5m 5.8m p99.999 665u 5.5m 6.5m 9.5m 11.5m 13.5m 24.5m 73.5m 83.5m 150m 7.6m 5.9m max 765u 6.5m 7.5m 10.5m 12.5m 15.5m 25.5m 74.5m 84.5m 250m 8.9m 6.1m WRITE min p25 p50 p75 p90 p95 p99 p99.9 max cum mean stdev min 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 0 p01 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 15.0u 5.0u 5.0u 385n p05 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 15.0u 25.0u 25.0u 5.0u 5.5u 2.3u p10 5.0u 5.0u 15.0u 15.0u 15.0u 15.0u 25.0u 65.0u 85.0u 15.0u 11.7u 6.7u p25 5.0u 15.0u 25.0u 25.0u 25.0u 35.0u 65.0u 145u 13.5m 25.0u 42.6u 520u p50 5.0u 25.0u 35.0u 45.0u 65.0u 85.0u 315u 25.5m 41.5m 35.0u 146u 1.9m p75 5.0u 45.0u 55.0u 105u 315u 345u 14.5m 42.5m 47.5m 65.0u 568u 3.5m p90 5.0u 95.0u 155u 335u 4.5m 14.5m 53.5m 86.5m 93.5m 315u 2.5m 9.5m p95 5.0u 155u 255u 365u 12.5m 26.5m 68.5m 94.5m 94.5m 1.5m 4.1m 12.3m p99 5.0u 335u 375u 1.5m 20.5m 39.5m 79.5m 96.5m 96.5m 29.5m 6.1m 15.3m p99.9 5.0u 375u 505u 2.5m 23.5m 41.5m 81.5m 97.5m 97.5m 99.5m 6.8m 15.9m p99.99 5.0u 415u 665u 4.5m 24.5m 42.5m 84.5m 98.5m 98.5m 150m 7.3m 16.2m p99.999 5.0u 455u 1.5m 5.5m 25.5m 43.5m 85.5m 99.5m 150m 250m 8.1m 17.0m max 15.0u 585u 3.5m 7.5m 27.5m 45.5m 86.5m 150m 250m 350m 9.8m 20.6m IO Latency: R p50=157u:42.3u/405u p90=921u:763u/11.5m p99=3.4m:2.9m/35.5m max=8.9m:6.1m/84.5m W p50=146u:1.9m/41.5m p90=2.5m:9.5m/93.5m p99=6.1m:15.3m/96.5m max=9.8m:20.6m/250m Slice resource stat: [ROOT] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 56.24 57.61 58.70 59.46 60.63 62.65 65.31 69.19 72.69 78.19 79.52 63.67 4.32 sys% 2.77 3.10 3.79 4.31 5.12 6.34 8.59 11.62 13.53 16.70 22.05 7.22 3.09 mem 61.8G 61.9G 61.9G 62.0G 62.0G 62.0G 62.1G 62.1G 62.1G 62.1G 62.1G 62.0G 56.0M io% 11.03 14.25 19.89 22.51 28.41 37.22 48.37 60.92 68.69 82.91 366.9 40.18 20.19 rbps 84.0M 112M 150M 174M 228M 307M 401M 516M 609M 757M 3.3G 335M 185M wbps 17.3M 19.7M 28.7M 34.1M 46.2M 58.3M 77.9M 103M 125M 167M 489M 65.6M 33.9M cpu-some% 0.14 0.17 0.20 0.22 0.28 0.44 0.96 2.33 3.70 6.17 9.87 0.94 1.27 mem-some% 4.10 5.16 6.49 7.39 9.70 13.81 28.98 56.45 76.58 95.37 139.7 23.79 22.37 mem-full% 3.45 4.45 5.26 6.05 7.79 10.38 17.34 27.76 35.42 41.78 93.12 14.00 9.63 io-some% 1.47 2.91 3.93 4.79 6.47 10.14 25.17 56.34 74.59 95.27 112.5 20.76 22.76 io-full% 1.30 2.23 3.22 3.47 4.49 6.24 12.62 23.05 28.85 38.50 65.47 10.11 8.74 [WORKLOAD] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 44.82 50.31 54.36 54.86 55.75 56.44 57.56 58.50 59.13 61.35 62.45 56.58 1.72 sys% 1.15 1.18 1.23 1.26 1.34 1.51 1.92 2.62 3.15 4.09 5.65 1.76 0.65 mem 10.4G 10.5G 10.5G 10.6G 10.6G 10.7G 10.8G 10.9G 10.9G 11.0G 11.6G 10.7G 133M io% 2.90 3.70 4.22 4.60 5.24 6.28 7.72 10.01 12.18 15.74 16.95 6.91 2.45 rbps 16.5M 33.0M 39.3M 41.3M 46.5M 54.1M 67.6M 90.1M 121M 155M 173M 62.0M 25.1M wbps 292K 1.0M 2.5M 3.3M 5.5M 8.0M 11.2M 14.2M 17.6M 26.0M 37.8M 8.7M 4.9M cpu-some% 0.07 0.08 0.09 0.10 0.11 0.16 0.34 0.83 1.31 2.16 3.79 0.34 0.45 mem-some% 0.42 0.68 0.85 0.98 1.21 1.75 2.63 4.02 5.37 11.07 25.43 2.31 2.07 mem-full% 0.39 0.64 0.80 0.91 1.13 1.59 2.39 3.64 4.71 9.18 22.15 2.09 1.79 io-some% 0.40 0.64 0.80 0.92 1.10 1.45 2.08 3.10 3.67 9.51 22.26 1.86 1.65 io-full% 0.38 0.63 0.77 0.87 1.05 1.35 1.91 2.77 3.25 8.21 19.20 1.70 1.41 [SYSTEM] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 0.50 0.60 0.93 1.19 1.74 2.97 4.92 7.92 10.95 15.07 17.02 3.91 3.09 sys% 0.32 0.35 0.58 0.73 1.06 1.72 3.20 5.21 6.69 9.44 12.25 2.46 2.04 mem 1.5G 2.0G 2.1G 2.2G 2.3G 2.4G 2.5G 2.6G 2.6G 2.7G 2.7G 2.4G 152M io% 7.08 9.77 14.59 17.12 22.62 30.37 40.33 50.61 56.79 68.55 82.79 32.44 13.33 rbps 47.1M 66.4M 102M 131M 178M 261M 351M 445M 531M 611M 831M 278M 128M wbps 8.8M 11.6M 21.2M 26.1M 37.1M 50.0M 66.1M 90.4M 107M 145M 175M 55.0M 26.8M cpu-some% 0.04 0.04 0.06 0.07 0.11 0.20 0.52 1.48 2.50 4.43 7.00 0.55 0.91 mem-some% 2.17 4.18 7.20 8.37 11.01 17.72 36.42 72.26 89.68 97.06 118.9 28.43 25.28 mem-full% 2.13 4.08 6.89 7.97 10.78 16.97 33.24 69.28 85.34 94.77 116.6 27.37 24.42 io-some% 0.50 2.93 7.04 8.32 11.06 17.30 38.89 74.14 90.70 97.05 118.8 28.75 25.78 io-full% 0.49 2.88 6.82 8.10 10.77 16.76 35.36 70.56 84.43 94.69 115.2 27.44 24.59 Isolation and Request Latency Impact Distributions: min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev isol% 0 0 92.57 95.48 98.15 100.0 100.0 100.0 100.0 100.0 100.0 96.44 14.85 lat-imp% 0 0 0.80 1.72 2.93 4.55 6.66 10.02 12.86 22.62 53.89 5.55 4.99 Result: isol=96.44:14.85% lat_imp=5.55%:4.99 work_csv=39.64% missing=0.15% Memory Hog Summary ================== Info: baseline_rps=926.10 baseline_lat=64.7m:899u vrate=99.90:0.39 io_usage=514.6 io_unused=685.2 hog_io_usage=85.3 hog_io_loss=2289.2 hog_bytes=46.4G hog_lost_bytes=1.2P IO Latency Distribution: READ min p25 p50 p75 p90 p95 p99 p99.9 max cum mean stdev min 5.0u 5.0u 5.0u 35.0u 45.0u 55.0u 75.0u 95.0u 95.0u 25.0u 21.3u 19.8u p01 5.0u 45.0u 65.0u 75.0u 75.0u 75.0u 105u 105u 125u 65.0u 55.4u 25.4u p05 5.0u 85.0u 85.0u 85.0u 95.0u 95.0u 115u 135u 145u 85.0u 81.8u 17.9u p10 5.0u 95.0u 95.0u 95.0u 105u 105u 135u 155u 155u 95.0u 94.8u 14.7u p25 5.0u 105u 115u 115u 125u 135u 165u 165u 165u 115u 113u 13.7u p50 75.0u 135u 155u 185u 205u 235u 335u 415u 415u 165u 164u 42.2u p75 85.0u 235u 315u 445u 575u 635u 1.5m 1.5m 3.5m 385u 369u 219u p90 105u 525u 725u 995u 1.5m 2.5m 3.5m 11.5m 17.5m 935u 915u 827u p95 115u 725u 1.5m 1.5m 2.5m 3.5m 6.5m 20.5m 38.5m 1.5m 1.6m 1.6m p99 225u 1.5m 2.5m 4.5m 6.5m 7.5m 13.5m 90.5m 97.5m 4.5m 3.4m 4.5m p99.9 265u 2.5m 4.5m 6.5m 8.5m 10.5m 20.5m 94.5m 98.5m 10.5m 5.3m 6.2m p99.99 325u 3.5m 5.5m 7.5m 9.5m 11.5m 25.5m 96.5m 99.5m 36.5m 6.4m 6.9m p99.999 335u 4.5m 6.5m 8.5m 11.5m 13.5m 27.5m 98.5m 150m 150m 7.4m 7.7m max 445u 5.5m 7.5m 10.5m 12.5m 14.5m 29.5m 150m 250m 250m 8.8m 10.3m WRITE min p25 p50 p75 p90 p95 p99 p99.9 max cum mean stdev min 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 0 p01 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 5.0u 15.0u 15.0u 5.0u 5.1u 843n p05 5.0u 5.0u 5.0u 5.0u 5.0u 15.0u 15.0u 35.0u 35.0u 5.0u 5.8u 3.0u p10 5.0u 5.0u 15.0u 15.0u 15.0u 25.0u 25.0u 65.0u 215u 15.0u 12.7u 8.5u p25 5.0u 15.0u 25.0u 25.0u 35.0u 35.0u 125u 3.5m 13.5m 25.0u 36.7u 373u p50 5.0u 25.0u 35.0u 45.0u 65.0u 135u 335u 31.5m 41.5m 35.0u 158u 1.7m p75 5.0u 55.0u 75.0u 175u 325u 345u 15.5m 72.5m 94.5m 65.0u 710u 4.9m p90 5.0u 115u 265u 345u 2.5m 12.5m 57.5m 93.5m 95.5m 315u 2.3m 9.1m p95 5.0u 185u 345u 365u 7.5m 22.5m 68.5m 94.5m 96.5m 1.5m 3.5m 11.6m p99 5.0u 345u 385u 965u 15.5m 32.5m 90.5m 96.5m 97.5m 29.5m 5.1m 14.3m p99.9 5.0u 375u 465u 2.5m 17.5m 33.5m 91.5m 97.5m 98.5m 99.5m 5.7m 14.9m p99.99 5.0u 385u 545u 3.5m 18.5m 34.5m 94.5m 98.5m 150m 150m 6.2m 15.5m p99.999 5.0u 425u 995u 4.5m 19.5m 38.5m 96.5m 150m 250m 250m 7.0m 16.9m max 15.0u 495u 2.5m 6.5m 22.5m 42.5m 99.5m 250m 350m 350m 8.8m 21.7m IO Latency: R p50=164u:42.2u/415u p90=915u:827u/17.5m p99=3.4m:4.5m/97.5m max=8.8m:10.3m/250m W p50=158u:1.7m/41.5m p90=2.3m:9.1m/95.5m p99=5.1m:14.3m/97.5m max=8.8m:21.7m/350m Slice resource stat: [ROOT] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 56.24 57.90 59.46 60.21 62.65 76.84 87.62 89.68 90.70 92.49 93.30 75.42 12.41 sys% 2.77 3.29 4.13 4.69 5.57 7.02 9.20 11.79 13.53 16.69 22.05 7.72 2.95 mem 61.8G 61.9G 61.9G 62.0G 62.0G 62.0G 62.1G 62.1G 62.1G 62.1G 62.1G 62.0G 55.8M io% 11.03 15.99 21.06 24.74 31.51 40.64 52.19 64.11 73.99 92.16 813.0 44.37 34.78 rbps 84.0M 124M 170M 200M 262M 345M 442M 572M 670M 930M 7.6G 384M 339M wbps 17.3M 21.3M 29.8M 35.1M 46.2M 59.8M 80.5M 104M 123M 167M 878M 67.5M 44.6M cpu-some% 0.14 0.18 0.22 0.26 0.44 3.35 7.76 10.85 13.35 17.49 164.8 4.75 6.62 mem-some% 4.10 5.28 6.75 8.09 10.40 15.96 38.26 69.82 82.89 97.90 264.6 27.86 26.22 mem-full% 2.35 3.37 4.58 5.31 6.84 9.22 14.38 21.93 28.40 40.74 147.8 12.03 9.22 io-some% 1.47 3.21 4.22 5.09 6.97 11.74 35.05 68.07 82.10 97.81 195.4 24.50 26.14 io-full% 1.16 1.77 2.38 2.73 3.63 5.33 9.48 16.67 23.63 36.42 91.78 8.08 7.61 [WORKLOAD] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 44.82 53.19 54.86 55.50 56.44 61.65 80.35 82.44 83.63 85.07 87.59 68.26 11.99 sys% 1.15 1.20 1.26 1.31 1.51 1.99 2.55 3.33 3.86 4.88 7.61 2.17 0.85 mem 10.4G 10.5G 10.6G 10.6G 10.7G 10.9G 11.3G 11.4G 11.4G 11.5G 11.7G 11.0G 310M io% 2.20 3.78 4.59 4.97 6.15 8.19 10.75 13.59 16.01 22.22 220.4 9.25 8.71 rbps 16.5M 35.5M 41.2M 44.9M 53.6M 74.2M 94.3M 125M 155M 252M 1.8G 84.2M 82.9M wbps 292K 1.5M 3.1M 4.0M 6.2M 9.4M 14.2M 21.8M 26.9M 40.0M 319M 11.7M 12.0M cpu-some% 0.07 0.09 0.10 0.11 0.16 1.54 5.54 7.70 8.88 11.27 127.2 3.16 4.85 mem-some% 0.20 0.71 0.94 1.09 1.48 2.28 3.55 5.40 7.06 13.07 57.47 3.02 3.15 mem-full% 0.12 0.52 0.75 0.92 1.18 1.68 2.65 3.89 5.29 9.49 47.47 2.28 2.44 io-some% 0.18 0.61 0.81 0.95 1.25 1.68 2.42 3.56 4.69 11.20 43.43 2.19 2.33 io-full% 0.14 0.50 0.73 0.87 1.08 1.38 1.93 2.81 3.56 9.63 37.51 1.79 1.92 [SYSTEM] min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev cpu% 0.50 0.74 1.07 1.34 2.01 3.23 5.11 7.57 9.52 13.90 17.02 3.98 2.74 sys% 0.32 0.43 0.67 0.85 1.22 1.95 3.28 4.86 6.18 8.96 12.25 2.51 1.81 mem 1.4G 1.6G 1.7G 1.7G 1.8G 2.1G 2.4G 2.5G 2.6G 2.6G 2.7G 2.1G 312M io% 7.08 10.29 15.25 18.05 23.97 31.99 41.64 50.95 58.81 75.17 580.9 34.78 26.28 rbps 47.1M 73.8M 118M 145M 202M 282M 376M 478M 557M 763M 5.9G 310M 265M wbps 8.8M 15.6M 21.2M 25.6M 35.2M 48.1M 64.5M 86.1M 106M 150M 657M 54.4M 35.3M cpu-some% 0.04 0.05 0.07 0.10 0.20 1.24 2.83 4.78 6.20 9.88 49.45 1.95 2.62 mem-some% 2.17 4.49 6.36 7.72 11.09 18.52 49.63 84.45 92.73 99.13 517.8 33.30 35.27 mem-full% 2.13 3.92 5.50 6.91 10.33 17.48 45.36 80.26 89.07 95.97 478.9 31.25 33.69 io-some% 0.50 3.45 5.92 7.32 10.85 18.01 49.81 85.43 92.84 99.09 499.8 33.29 35.39 io-full% 0.49 3.32 5.36 7.02 10.33 17.11 46.23 81.15 89.12 96.96 471.7 31.36 33.72 Isolation and Request Latency Impact Distributions: min p01 p05 p10 p25 p50 p75 p90 p95 p99 max mean stdev isol% 0 0 88.34 90.57 93.78 97.30 100.0 100.0 100.0 100.0 100.0 95.18 11.06 lat-imp% 0 0 0.96 2.20 3.79 6.49 10.22 15.63 18.32 29.55 263.0 8.14 9.99 Result: isol=95.18:11.06% lat_imp=8.14%:9.99 work_csv=42.89% missing=0.21%