Benchmark all commits ===================== The script in this directory is used to run the benchmarking tests for all commits in the repository. It is actually not possible to run all benchmarking tests for all commits because: - at some points in the repo history, the tested code didn't exist. - at some poitns in the repo history, the teste code might have been refactored and the api would not match the one used in tests. So what this script does is the following: - check out .gitlab-ci.yml from current master. - use gitlab-runner to exec a build locally. - set an environment variable (BENCHMARK_ALL_COMMITS=1)so the build script knows that it has to setup the environment. - store timestamp and exit status of each run. - checkout previous commit and iterate. How to use the script --------------------- Once you have gitlab-runner setup in the machine and the elasticsearch credentials, use the following commands to run the script: git clone https://0xacab.org/leap/soledad cd soledad/ export PYTEST_OPTS="--benchmark-autosave --benchmark-storage=\"elasticsearch+https://USER:PASSWORD@moose.leap.se:9200\"" scripts/benchmark/benchmark-all-commits.sh