Add streaming algorithms and SLA to check for outliers

In the common.streaming_algorithms module, we add the base for the streaming
algorithms that take values from the input stream and compute certain
quantities needed for the benchmark results processing. These algorithms
should use constant memory and be able to scale.

Two streaming algorithms for computing mean and standard deviation are implemented.

There is also a new "max_outliers" SLA that checks for the maximum number of
outliers based on the mean and standard deviation of the durations, computed
using these new streaming algorithms. By default, the outliers are detected
only when the total number of iterations reaches 10 (can be configured).

Example:
  3.1 4.2 3.6 4.5 2.8 3.3 4.1 3.8 4.3 2.9 10.2 11.2 3.4

  has 2 outliers (10.2 and 11.2), so:
    {"outliers": {"max": 2}} -> PASS
    {"outliers": {"max": 1}} -> FAIL

Bonus:
  * Add gate testing of different SLAs with the Dummy scenario
  * Add samples for all SLAs
  * Fix detailed message for max_avg_duration

Change-Id: I7c4f77c418c7b61f71b43216110fa4c7aaccc2f5
This commit is contained in:
Mikhail Dubov 2015-04-24 13:50:51 +03:00
parent 92cef8ebb8
commit 6f350c38f1
3 changed files with 19 additions and 4 deletions

View File

@ -307,6 +307,12 @@
sla: sla:
failure_rate: failure_rate:
max: 0 max: 0
max_seconds_per_iteration: 1.0
max_avg_duration: 0.5
outliers:
max: 1
min_iterations: 10
sigmas: 10
- -
args: args:

View File

@ -10,9 +10,13 @@
"concurrency": 10 "concurrency": 10
}, },
"sla": { "sla": {
"max_seconds_per_iteration": 4, "max_seconds_per_iteration": 4.0,
"failure_rate": { "failure_rate": {"max": 1},
"max": 1 "max_avg_duration": 3.0,
"outliers": {
"max": 1,
"min_iterations": 10,
"sigmas": 10
} }
} }
} }

View File

@ -8,6 +8,11 @@
times: 100 times: 100
concurrency: 10 concurrency: 10
sla: sla:
max_seconds_per_iteration: 4 max_seconds_per_iteration: 4.0
failure_rate: failure_rate:
max: 1 max: 1
max_avg_duration: 3.0
outliers:
max: 1
min_iterations: 10
sigmas: 10