soledad.git - [soledad]

Age	Commit message (Collapse)	Author
2017-09-11	[test] add --subdir option and blacklist	drebs
	Use subdir path to select/deselect benchmark tests. Because of the way pytest and pytest benchamrk select/deselect tests, it makes more sense for this test suite to use a --subdir option to select only a subset of tests and a blacklist of subdirectories that should not be run unless explicitelly asked to. This commit adds a --subdir option, that will deselect all tests no in ./tests/<subdir> when the option is passed to pytest. Also, a blacklist is added so, unless explicitelly added as a --subdir, tests in blacklisted subdirectories will not be run. The reasons for this modification are: - pytest-benchmarks selects/deselects tests based on the type of their fixtures. This means that fixtures have to be instantiated previous to test selection/deselection, something that may cause side effects that we want to avoid. - the usual test suite is run for all benchmarks, while some tests should only be run in specific situations (as is the case of benchmarks and responsiveness tests). It is saner to implement subdir selection than marking all tests with pytest marks and using them to select/deselect.
2017-09-11	[benchmarks] add responsiveness test with watchdog	drebs

2017-08-23	[tests] use latest version of pytest-benchmark	drebs

2017-08-03	[test] add sqlcipher check also to parallel test suite	Victor Shyba

2017-07-18	[test] avoid reinstalling pysqlcipher if unneeded	drebs

2017-07-09	[benchmarks] separate memory sampling from cpu measurement	drebs

2017-07-08	[benchmarks] run benchmarks twice, for time and resources	drebs
	We noticed that instrumentation added for watching resources has an impact in time statistics (i.e. it increases average and stddev). This commit makes the benchmark tests run twice: once for measuring time and a second time for measuring resources.
2017-07-08	[benchmarks] avoid reinstalling pysqlcipher if unneeded	drebs
	pysqlcipher just has to be reinstalled for benchmark tests if current installed version fails the HAVE_USLEEP flag test. This commit moves the code for checking and reinstalling to a script and bypasses the reinstallation when it is unneeded.
2017-07-08	[benchmarks] skip some tests by default	drebs

2017-07-08	[feat] use OpenSSL backend for scrypt if available	Kali Kaneko
	This needs OpenSSL >= 1.1, otherwise it will keep using the scrypt dependency. We should think about deprecating scrypt as a dependency when we can be sure that the adoption of libssl 1.1 is wide enough. I think that at some point (soledad 0.11 or so) we can drop the scrypt dependency, which was being somehow problematic at times (the _scrypt.so was not appearing when installing with pip, needed workarounds). From that moment on, we can raise an error if an old libssl is found and no scrypt can be imported - leaving that to the user/packager. In debian stretch and afterwards, you can get that version by installing libssl-dev - Related: #8472
2017-06-28	[test] use pip -e for installing soledad for tests	drebs

2017-06-24	[bug] fix tests	Victor Shyba

2017-06-24	[tests] install unified package with extras in tox	Kali Kaneko

2017-06-24	[test] adapt tests to new python package	drebs

2017-05-31	[tests] use pysqlcipher workaround here too	Kali Kaneko

2017-04-27	[test] measure cpu percentage during benchmark	drebs

2017-04-19	[test] avoid running sqlcipher synchonous tests when benchmarking	drebs
	SQLCipher synchronous benchmark tests were introduced when we started developing benchmark tests to compare synchronous and asynchronous code. Synchronous access to sqlcipher database is not used in soledad, and those tests are much slower than asynchronous tests (more than 10 times using ssd drive), so we want to avoid running them on ci. This commit introduces a "synchronous" marker and avoid running tests markes as such in ci environment.
2017-04-19	[test] use latest pysqlcipher-benchmark for benchmarks	drebs
	There were some changes needed in pytest-benchmark so we could successfully use it for soledad benchmarks graphing: https://github.com/ionelmc/pytest-benchmark/pull/73 https://github.com/ionelmc/pytest-benchmark/pull/74 https://github.com/ionelmc/pytest-benchmark/pull/75 The contributions were accepted but not released yet, so this commit uses the code from upstream git repository's master branch.
2017-04-15	[test] enforce bundled pysqlcipher in benchmark tests	drebs
	The debian package for libsqlcipher does not enable the use of usleep(), and that might cause timeouts when doing concurrent access to sqlcipher databases. See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859685 In benchmark tests we use twisted's asynchronous dbapi for creating thousands of documents, and if usleep() is not enabled this operation is very likelly to fail with a "db is locked" error. This commit adds a workaround for benchmark tests by reinstalling pysqlcipher with the "--bundled" option, what causes libsqlcipher to be built and bundled in the python package, with usleep() enabled. Resolves: #8835
2017-04-04	[feat] add the host hostname to benchmark machine info	drebs

2017-04-04	[bug] cache pip instead of tox	Victor Shyba
	Caching .tox breaks when installing new dependencies. Caching pip instead and asking tox to always recreate will ensure newer dependencies are installed while getting existing ones from cache.
2017-04-03	Post benchmark results to elastticsearch	varac
	In order to do this from CI, please create a secret variable (at i.e. https://0xacab.org/varac/soledad/settings/ci_cd) called PYTEST_OPTS and use this example: --benchmark-autosave --benchmark-storage="elasticsearch+https://USER:PASSWORD@moose.leap.se:9200"
2017-03-17	[test] Prepare new tox environment	efkin
	At this point tests are not yet collectable in the new environment. But no regressions observed.
2016-12-12	[tests] use options instead of marks	Victor Shyba
	When we use marks the new pytests from benchmarks folder are collected and ignored, but this causes trial to fail sometimes. Using --ignore avoids it from being loaded while --benchmark-only will properly select the benchmarks for tox, as intended.
2016-12-12	[test] rename benchmark tests directory and tag	drebs

2016-12-12	[test] use tags for selecting benchmark tests	drebs

2016-12-12	[style] fix pep8 and confs	Victor Shyba
	Fixes setup.cfg, adding current exclude rules, simplified tox.ini to use setup.cfg and fixed all.
2016-12-12	[tests] adapt tests	Kali Kaneko

2016-10-21	[tests] specify basepython on root env	Victor Shyba
	code-check is running with py3 randomly on CI, this commit should pin it.
2016-09-30	[test] add flake8 code check and generalize name of tox env	drebs

2016-09-09	[test] add tagged perf job for gitlab-ci	drebs
	In order to configure performance tests to run in a specific machine we need to add a tagged job to .gitlab-ci.yml file. That job will only execute the perf tests, and then we can have runners that will only run those jobs.
2016-09-06	[test] add tox env for perf tests	drebs

2016-09-02	[tests] add pep8 in main tox run from within the gitlab script	Kali Kaneko
	add coverage reports too. (hereby we swear not to write stupid tests just because it feels good to have an increased coverage metric). - Resolves: #8416
2016-08-22	[test] Adds pytest-benchmark adapted to Twisted	Victor Shyba
	Adapted pytest-benchmark to Twisted as it's synchronous and added fixtures for benchmarking.
2016-08-15	[test] allow for shell autocomplete for test names	drebs
	tox was configured to change to the testing/tests directory before executing pytest, by using tox's "changedir" configuration option. The reason why this was the case is that we wanted to discover tests inside the testing/tests directory only. The problem with that approach is that if we wanted to point to a specific test file, for example "tests/perf/test_sync.py", we would have to omit the "tests" part and write "tox perf/test_sync.py" because the argument would be understood as relative to the changed dir. That is not practical as doesn't allow to use the shell autocomplete, and is also not the only way to achieve what we want. Actually, pytest has a configuration option called "testpaths" where you can indicate where it should discover tests. This commit changes one approach by the other and allows to user shell autocomplete for easyness of testing during development.
2016-08-01	[test] do not run pep8 tox env by default	drebs

2016-08-01	[test] adds optional parallel env for local dev	Victor Shyba

2016-08-01	[test] adds pep8 as a tox env	Victor Shyba
	"tox -e pep8" runs it standalone and "tox" includes the pep8 env.
2016-08-01	[test] remove pip download cache	drebs
	Recent versions of pip will ignore that option and use a cache anyway.
2016-08-01	[test] use pip download cache for tests and docker image	drebs

2016-08-01	[test] remove pep8 from tox config	drebs

2016-08-01	[test] add pytest initial setup for performance tests	drebs

2016-07-12	add pep8/flake8 to tox	Kali Kaneko

2016-07-12	[test] toxify tests	drebs
	- move tests to root directory - split tests in different subdirectories - setup a small package with common test dependencies in /testing/test_soledad - add tox.ini that will: - install the test_soledad package and other test dependencies - install soledad common, client, server from the repository - run tests contianed in /testing/tests directory using pytest This commit also removes all oauth code from tests, as we have removed the u1db dependency (by importing it into the repo and naming it l2db) and don't neet oauth at all right now.