summaryrefslogtreecommitdiff
path: root/testing/tox.ini
AgeCommit message (Collapse)Author
2017-08-03[test] add sqlcipher check also to parallel test suiteVictor Shyba
2017-07-18[test] avoid reinstalling pysqlcipher if unneededdrebs
2017-07-09[benchmarks] separate memory sampling from cpu measurementdrebs
2017-07-08[benchmarks] run benchmarks twice, for time and resourcesdrebs
We noticed that instrumentation added for watching resources has an impact in time statistics (i.e. it increases average and stddev). This commit makes the benchmark tests run twice: once for measuring time and a second time for measuring resources.
2017-07-08[benchmarks] avoid reinstalling pysqlcipher if unneededdrebs
pysqlcipher just has to be reinstalled for benchmark tests if current installed version fails the HAVE_USLEEP flag test. This commit moves the code for checking and reinstalling to a script and bypasses the reinstallation when it is unneeded.
2017-07-08[benchmarks] skip some tests by defaultdrebs
2017-07-08[feat] use OpenSSL backend for scrypt if availableKali Kaneko
This needs OpenSSL >= 1.1, otherwise it will keep using the scrypt dependency. We should think about deprecating scrypt as a dependency when we can be sure that the adoption of libssl 1.1 is wide enough. I think that at some point (soledad 0.11 or so) we can drop the scrypt dependency, which was being somehow problematic at times (the _scrypt.so was not appearing when installing with pip, needed workarounds). From that moment on, we can raise an error if an old libssl is found and no scrypt can be imported - leaving that to the user/packager. In debian stretch and afterwards, you can get that version by installing libssl-dev - Related: #8472
2017-06-28[test] use pip -e for installing soledad for testsdrebs
2017-06-24[bug] fix testsVictor Shyba
2017-06-24[tests] install unified package with extras in toxKali Kaneko
2017-06-24[test] adapt tests to new python packagedrebs
2017-05-31[tests] use pysqlcipher workaround here tooKali Kaneko
2017-04-27[test] measure cpu percentage during benchmarkdrebs
2017-04-19[test] avoid running sqlcipher synchonous tests when benchmarkingdrebs
SQLCipher synchronous benchmark tests were introduced when we started developing benchmark tests to compare synchronous and asynchronous code. Synchronous access to sqlcipher database is not used in soledad, and those tests are much slower than asynchronous tests (more than 10 times using ssd drive), so we want to avoid running them on ci. This commit introduces a "synchronous" marker and avoid running tests markes as such in ci environment.
2017-04-19[test] use latest pysqlcipher-benchmark for benchmarksdrebs
There were some changes needed in pytest-benchmark so we could successfully use it for soledad benchmarks graphing: https://github.com/ionelmc/pytest-benchmark/pull/73 https://github.com/ionelmc/pytest-benchmark/pull/74 https://github.com/ionelmc/pytest-benchmark/pull/75 The contributions were accepted but not released yet, so this commit uses the code from upstream git repository's master branch.
2017-04-15[test] enforce bundled pysqlcipher in benchmark testsdrebs
The debian package for libsqlcipher does not enable the use of usleep(), and that might cause timeouts when doing concurrent access to sqlcipher databases. See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859685 In benchmark tests we use twisted's asynchronous dbapi for creating thousands of documents, and if usleep() is not enabled this operation is very likelly to fail with a "db is locked" error. This commit adds a workaround for benchmark tests by reinstalling pysqlcipher with the "--bundled" option, what causes libsqlcipher to be built and bundled in the python package, with usleep() enabled. Resolves: #8835
2017-04-04[feat] add the host hostname to benchmark machine infodrebs
2017-04-04[bug] cache pip instead of toxVictor Shyba
Caching .tox breaks when installing new dependencies. Caching pip instead and asking tox to always recreate will ensure newer dependencies are installed while getting existing ones from cache.
2017-04-03Post benchmark results to elastticsearchvarac
In order to do this from CI, please create a secret variable (at i.e. https://0xacab.org/varac/soledad/settings/ci_cd) called PYTEST_OPTS and use this example: --benchmark-autosave --benchmark-storage="elasticsearch+https://USER:PASSWORD@moose.leap.se:9200"
2017-03-17[test] Prepare new tox environmentefkin
At this point tests are not yet collectable in the new environment. But no regressions observed.
2016-12-12[tests] use options instead of marksVictor Shyba
When we use marks the new pytests from benchmarks folder are collected and ignored, but this causes trial to fail sometimes. Using --ignore avoids it from being loaded while --benchmark-only will properly select the benchmarks for tox, as intended.
2016-12-12[test] rename benchmark tests directory and tagdrebs
2016-12-12[test] use tags for selecting benchmark testsdrebs
2016-12-12[style] fix pep8 and confsVictor Shyba
Fixes setup.cfg, adding current exclude rules, simplified tox.ini to use setup.cfg and fixed all.
2016-12-12[tests] adapt testsKali Kaneko
2016-10-21[tests] specify basepython on root envVictor Shyba
code-check is running with py3 randomly on CI, this commit should pin it.
2016-09-30[test] add flake8 code check and generalize name of tox envdrebs
2016-09-09[test] add tagged perf job for gitlab-cidrebs
In order to configure performance tests to run in a specific machine we need to add a tagged job to .gitlab-ci.yml file. That job will only execute the perf tests, and then we can have runners that will only run those jobs.
2016-09-06[test] add tox env for perf testsdrebs
2016-09-02[tests] add pep8 in main tox run from within the gitlab scriptKali Kaneko
add coverage reports too. (hereby we swear not to write stupid tests just because it feels good to have an increased coverage metric). - Resolves: #8416
2016-08-22[test] Adds pytest-benchmark adapted to TwistedVictor Shyba
Adapted pytest-benchmark to Twisted as it's synchronous and added fixtures for benchmarking.
2016-08-15[test] allow for shell autocomplete for test namesdrebs
tox was configured to change to the testing/tests directory before executing pytest, by using tox's "changedir" configuration option. The reason why this was the case is that we wanted to discover tests inside the testing/tests directory only. The problem with that approach is that if we wanted to point to a specific test file, for example "tests/perf/test_sync.py", we would have to omit the "tests" part and write "tox perf/test_sync.py" because the argument would be understood as relative to the changed dir. That is not practical as doesn't allow to use the shell autocomplete, and is also not the only way to achieve what we want. Actually, pytest has a configuration option called "testpaths" where you can indicate where it should discover tests. This commit changes one approach by the other and allows to user shell autocomplete for easyness of testing during development.
2016-08-01[test] do not run pep8 tox env by defaultdrebs
2016-08-01[test] adds optional parallel env for local devVictor Shyba
2016-08-01[test] adds pep8 as a tox envVictor Shyba
"tox -e pep8" runs it standalone and "tox" includes the pep8 env.
2016-08-01[test] remove pip download cachedrebs
Recent versions of pip will ignore that option and use a cache anyway.
2016-08-01[test] use pip download cache for tests and docker imagedrebs
2016-08-01[test] remove pep8 from tox configdrebs
2016-08-01[test] add pytest initial setup for performance testsdrebs
2016-07-12add pep8/flake8 to toxKali Kaneko
2016-07-12[test] toxify testsdrebs
- move tests to root directory - split tests in different subdirectories - setup a small package with common test dependencies in /testing/test_soledad - add tox.ini that will: - install the test_soledad package and other test dependencies - install soledad common, client, server from the repository - run tests contianed in /testing/tests directory using pytest This commit also removes all oauth code from tests, as we have removed the u1db dependency (by importing it into the repo and naming it l2db) and don't neet oauth at all right now.