Age | Commit message (Collapse) | Author |
|
|
|
|
|
|
|
We noticed that instrumentation added for watching resources has an
impact in time statistics (i.e. it increases average and stddev). This
commit makes the benchmark tests run twice: once for measuring time and
a second time for measuring resources.
|
|
pysqlcipher just has to be reinstalled for benchmark tests if current
installed version fails the HAVE_USLEEP flag test. This commit moves the
code for checking and reinstalling to a script and bypasses the
reinstallation when it is unneeded.
|
|
|
|
This needs OpenSSL >= 1.1, otherwise it will keep using the scrypt
dependency.
We should think about deprecating scrypt as a dependency when we can be
sure that the adoption of libssl 1.1 is wide enough. I think that at
some point (soledad 0.11 or so) we can drop the scrypt dependency, which
was being somehow problematic at times (the _scrypt.so was not appearing
when installing with pip, needed workarounds). From that moment on, we
can raise an error if an old libssl is found and no scrypt can be
imported - leaving that to the user/packager.
In debian stretch and afterwards, you can get that version by installing
libssl-dev
- Related: #8472
|
|
|
|
|
|
|
|
|
|
|
|
|
|
SQLCipher synchronous benchmark tests were introduced when we started
developing benchmark tests to compare synchronous and asynchronous code.
Synchronous access to sqlcipher database is not used in soledad, and
those tests are much slower than asynchronous tests (more than 10 times
using ssd drive), so we want to avoid running them on ci. This commit
introduces a "synchronous" marker and avoid running tests markes as such
in ci environment.
|
|
There were some changes needed in pytest-benchmark so we could
successfully use it for soledad benchmarks graphing:
https://github.com/ionelmc/pytest-benchmark/pull/73
https://github.com/ionelmc/pytest-benchmark/pull/74
https://github.com/ionelmc/pytest-benchmark/pull/75
The contributions were accepted but not released yet, so this commit
uses the code from upstream git repository's master branch.
|
|
The debian package for libsqlcipher does not enable the use of usleep(),
and that might cause timeouts when doing concurrent access to sqlcipher
databases.
See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859685
In benchmark tests we use twisted's asynchronous dbapi for creating
thousands of documents, and if usleep() is not enabled this operation is
very likelly to fail with a "db is locked" error.
This commit adds a workaround for benchmark tests by reinstalling
pysqlcipher with the "--bundled" option, what causes libsqlcipher to be
built and bundled in the python package, with usleep() enabled.
Resolves: #8835
|
|
|
|
Caching .tox breaks when installing new dependencies. Caching pip
instead and asking tox to always recreate will ensure newer dependencies
are installed while getting existing ones from cache.
|
|
In order to do this from CI, please create a secret variable
(at i.e. https://0xacab.org/varac/soledad/settings/ci_cd)
called PYTEST_OPTS and use this example:
--benchmark-autosave --benchmark-storage="elasticsearch+https://USER:PASSWORD@moose.leap.se:9200"
|
|
At this point tests are not yet collectable
in the new environment. But no regressions observed.
|
|
When we use marks the new pytests from benchmarks folder are collected
and ignored, but this causes trial to fail sometimes. Using --ignore
avoids it from being loaded while --benchmark-only will properly select
the benchmarks for tox, as intended.
|
|
|
|
|
|
Fixes setup.cfg, adding current exclude rules, simplified tox.ini to use
setup.cfg and fixed all.
|
|
|
|
code-check is running with py3 randomly on CI, this commit should pin
it.
|
|
|
|
In order to configure performance tests to run in a specific machine we
need to add a tagged job to .gitlab-ci.yml file. That job will only
execute the perf tests, and then we can have runners that will only run
those jobs.
|
|
|
|
add coverage reports too.
(hereby we swear not to write stupid tests just because it feels good to
have an increased coverage metric).
- Resolves: #8416
|
|
Adapted pytest-benchmark to Twisted as it's synchronous and added
fixtures for benchmarking.
|
|
tox was configured to change to the testing/tests directory before
executing pytest, by using tox's "changedir" configuration option. The
reason why this was the case is that we wanted to discover tests inside
the testing/tests directory only.
The problem with that approach is that if we wanted to point to a
specific test file, for example "tests/perf/test_sync.py", we would have
to omit the "tests" part and write "tox perf/test_sync.py" because the
argument would be understood as relative to the changed dir. That is not
practical as doesn't allow to use the shell autocomplete, and is also
not the only way to achieve what we want.
Actually, pytest has a configuration option called "testpaths" where you
can indicate where it should discover tests. This commit changes one
approach by the other and allows to user shell autocomplete for easyness
of testing during development.
|
|
|
|
|
|
"tox -e pep8" runs it standalone and "tox" includes the pep8 env.
|
|
Recent versions of pip will ignore that option and use a cache anyway.
|
|
|
|
|
|
|
|
|
|
- move tests to root directory
- split tests in different subdirectories
- setup a small package with common test dependencies in /testing/test_soledad
- add tox.ini that will:
- install the test_soledad package and other test dependencies
- install soledad common, client, server from the repository
- run tests contianed in /testing/tests directory using pytest
This commit also removes all oauth code from tests, as we have removed the
u1db dependency (by importing it into the repo and naming it l2db) and don't
neet oauth at all right now.
|