soledad.git/testing/tests/benchmarks, branch 0.10.2

[benchmarks] add docstrings for benchmarks

2017-07-18T16:51:09+00:00

[benchmarks] store test function docstring

2017-07-18T16:51:08+00:00

[doc] add info on how to update benchmarks website

2017-07-13T16:02:10+00:00

[benchmarks] change 20_500k to 10_1000k

2017-07-13T14:27:57+00:00

[doc] mark which sync benchmark scenarios are actually run

2017-07-12T18:11:35+00:00

[benchmarks] limit number of runs of sqlcipher benchmark tests

2017-07-12T15:29:45+00:00

As sqlcipher benchmark tests take longer, we want to limit the number of
repetitions. Previous to this change, these tests were being calibrated
automatically and would run 5 times becuase it is the default minimum
number of times for pytest-benchmark. By changing the runner to pedantic
mode, now they will be run 4 times, the same number of times as
benchmark sync tests.

[benchmarks] allow passing args and kwargs to txbenchmark_with_setup

2017-07-12T15:29:21+00:00

[benchmarks] separate memory sampling from cpu measurement

2017-07-09T14:14:29+00:00

[benchmarks] run benchmarks twice, for time and resources

2017-07-08T10:48:47+00:00

We noticed that instrumentation added for watching resources has an
impact in time statistics (i.e. it increases average and stddev). This
commit makes the benchmark tests run twice: once for measuring time and
a second time for measuring resources.

[benchmarks] add --watch-resources option

2017-07-08T10:48:47+00:00

This commit adds the --watch-resources command line option for
benchmarks tests, and allows to running the benchmark test suite with
and without resource monitoring instrumentation code.

This is needed because resource consumption monitoring impacts the mean
time and standard deviation of time taken to run benchmarked tests.