Add streaming algorithms and SLA to check for outliers
In the common.streaming_algorithms module, we add the base for the streaming algorithms that take values from the input stream and compute certain quantities needed for the benchmark results processing. These algorithms should use constant memory and be able to scale. Two streaming algorithms for computing mean and standard deviation are implemented. There is also a new "max_outliers" SLA that checks for the maximum number of outliers based on the mean and standard deviation of the durations, computed using these new streaming algorithms. By default, the outliers are detected only when the total number of iterations reaches 10 (can be configured). Example: 3.1 4.2 3.6 4.5 2.8 3.3 4.1 3.8 4.3 2.9 10.2 11.2 3.4 has 2 outliers (10.2 and 11.2), so: {"outliers": {"max": 2}} -> PASS {"outliers": {"max": 1}} -> FAIL Bonus: * Add gate testing of different SLAs with the Dummy scenario * Add samples for all SLAs * Fix detailed message for max_avg_duration Change-Id: I7c4f77c418c7b61f71b43216110fa4c7aaccc2f5
This commit is contained in:
parent
92cef8ebb8
commit
6f350c38f1
@ -307,6 +307,12 @@
|
|||||||
sla:
|
sla:
|
||||||
failure_rate:
|
failure_rate:
|
||||||
max: 0
|
max: 0
|
||||||
|
max_seconds_per_iteration: 1.0
|
||||||
|
max_avg_duration: 0.5
|
||||||
|
outliers:
|
||||||
|
max: 1
|
||||||
|
min_iterations: 10
|
||||||
|
sigmas: 10
|
||||||
|
|
||||||
-
|
-
|
||||||
args:
|
args:
|
||||||
|
@ -10,9 +10,13 @@
|
|||||||
"concurrency": 10
|
"concurrency": 10
|
||||||
},
|
},
|
||||||
"sla": {
|
"sla": {
|
||||||
"max_seconds_per_iteration": 4,
|
"max_seconds_per_iteration": 4.0,
|
||||||
"failure_rate": {
|
"failure_rate": {"max": 1},
|
||||||
"max": 1
|
"max_avg_duration": 3.0,
|
||||||
|
"outliers": {
|
||||||
|
"max": 1,
|
||||||
|
"min_iterations": 10,
|
||||||
|
"sigmas": 10
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
@ -8,6 +8,11 @@
|
|||||||
times: 100
|
times: 100
|
||||||
concurrency: 10
|
concurrency: 10
|
||||||
sla:
|
sla:
|
||||||
max_seconds_per_iteration: 4
|
max_seconds_per_iteration: 4.0
|
||||||
failure_rate:
|
failure_rate:
|
||||||
max: 1
|
max: 1
|
||||||
|
max_avg_duration: 3.0
|
||||||
|
outliers:
|
||||||
|
max: 1
|
||||||
|
min_iterations: 10
|
||||||
|
sigmas: 10
|
||||||
|
Loading…
Reference in New Issue
Block a user