summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2016-08-23[pkg] add leftovers deletion to couch scehma migration scriptdrebs
Previous versions of the couchdb schema used documents "u1db_sync_log" and "u1db_sync_state" to store sync metadata. At some point this was changed, but the documents might have stayed as leftovers. This commit adds the deletion of such documents to the migration script.
2016-08-23[pkg] add --pdb option to migration scriptdrebs
2016-08-23[test] test_instance -> test_initializationVictor Shyba
This isnt a test, but a benchmark. Initialization sounds more like an operation while instance is just something.
2016-08-23[doc] improve SQLITE_MAX_VARIABLE_NUMBER commentsVictor Shyba
2016-08-23[test] point issue #7370 as reason for low valuesVictor Shyba
We are using lower values on test_encdecpool due high memory usage, described in #7370. Added a comment to explain it.
2016-08-22[test] refactor load_up for readabilityVictor Shyba
defer parameter wasnt clear
2016-08-22[tests] move doc creation to setupVictor Shyba
Otherwise it will add unrelated overhead to results.
2016-08-22[tests] remove unused new=False logicVictor Shyba
2016-08-22[test] make txbench ignore kwargs for readabilityVictor Shyba
They arent used so far and using empty dicts to make them work is ugly. Removing it leaves the return function on setup code clean and readable.
2016-08-22[test] calibrate encdecpool bench for memoryVictor Shyba
1000 docs at 100k~500k are exploding memory (4Gb+4Gb swap). Changed for 100 docs in order to be able to get measures on higher loads. Now its 10k, 100k and 500k
2016-08-22[test] adds tests for raw encryptionVictor Shyba
Hypothesis: raw vs doc Added the same sizes set (10k, 100k, 500k, 1M, 10M, 50M) as the document crypto test, so we can compare how close to raw the higher level operation is.
2016-08-22[test] adds test for SoledadCryptoVictor Shyba
10k, 100k, 500k, 1m, 10m and 50m for encryption and decryption of a whole document.
2016-08-22[bug] limit pool comnsumption to 900 docsVictor Shyba
This was discovered during load tests: Trying to process more than 999 docs triggers an error on SQLite due a select query not supporting 999 values to query.
2016-08-22[test] adds encdecpool testsVictor Shyba
Most of them are commented as memory usage is going out of control for now.
2016-08-22[test] test soledad instatiation timeVictor Shyba
It has a heavy scrypt hashing processing with room for improvement.
2016-08-22[test] sync without changesVictor Shyba
Syncing without any changes was reported as slow. This benchmark will help measure it.
2016-08-22[test] remove fixed default uuidVictor Shyba
Use a new one to avoid reusing the same database.
2016-08-22[test] make all nested fixture function-scopedVictor Shyba
function is the default scope, so there is no need to pass this parameter. Previously, one of the scopes was 'module', but it is a nested function that fires on demand, so it should clean up itself from test to test in order to avoid conflict while putting.
2016-08-22[test] adds sqlcipher synchronous testsVictor Shyba
2016-08-22[test] adds sqlcipher create testsVictor Shyba
Creating 20/500k, 100/100k and 1000/10k.
2016-08-22[test] use a nested func to simplify scenariosVictor Shyba
If we have many scenarios (like 20/500k, 100/100k, 1000,10k) then making a nested function to generate tests based on scenario parameters simplifies the code a lot.
2016-08-22[test] Adds pytest-benchmark adapted to TwistedVictor Shyba
Adapted pytest-benchmark to Twisted as it's synchronous and added fixtures for benchmarking.
2016-08-17[tests] Adapt tests for ensure_ddocs deathVictor Shyba
2016-08-17[bug] remove misleading ensure_ddocVictor Shyba
ensure_ddoc doesnt make sense anymore as we dont have any ddoc other than _security, which has its own method for setting. 'ensure_security' is explicit and is set internally when user is creating a database, otherwise it will be False as it's only used during creation. This isn't exposed externally (of couch module) to avoid confusion. This confusion was making create-user-db fail to create a security ddoc as it wasn't passing ensure_ddocs=True. -- Resolves: #8388
2016-08-15[test] allow for shell autocomplete for test namesdrebs
tox was configured to change to the testing/tests directory before executing pytest, by using tox's "changedir" configuration option. The reason why this was the case is that we wanted to discover tests inside the testing/tests directory only. The problem with that approach is that if we wanted to point to a specific test file, for example "tests/perf/test_sync.py", we would have to omit the "tests" part and write "tox perf/test_sync.py" because the argument would be understood as relative to the changed dir. That is not practical as doesn't allow to use the shell autocomplete, and is also not the only way to achieve what we want. Actually, pytest has a configuration option called "testpaths" where you can indicate where it should discover tests. This commit changes one approach by the other and allows to user shell autocomplete for easyness of testing during development.
2016-08-09[test] use "leapcode" in docker image namedrebs
"leapcode" is the LEAP docker hub organisation varac could squat (https://hub.docker.com/r/leap/ was already taken).
2016-08-09[feat] log to syslog (#8286)drebs
2016-08-08[pkg] remove version pinning for couchdb and beakerdrebs
We will not maintain support for older versions of debian as that introduces some unneeded complexity for now. Also, the version pinned for couchdb python library has a bug that makes some requests slow. Because of those, we remove the pinning for now.
2016-08-05[bug] create gen document after saving the actual document in couchdrebs
If we create the gen document before saving the actual document in couch, we may run into problems if more than one client is syncing and trying to save documents with the same id at the same time. By moving the gen document creation to after the actual document save in couch, we rely on couch/u1db resolution of conflicts before actually allocating a new generation, and the problem above doesn't occur.
2016-08-03[pkg] support netrc couch access in migrate scriptdrebs
2016-08-02[test] avoid race condition on test_processing_orderVictor Shyba
test_processing_order aims to check that unordered docs wont be processed, but if we let the pool start and advance Twisted LoopingCall clock right before calling the processing method manually, the process method will run concurrently and cause a race condition issue.
2016-08-01[test] do not run pep8 tox env by defaultdrebs
2016-08-01[test] adds optional parallel env for local devVictor Shyba
2016-08-01[test] adds pep8 as a tox envVictor Shyba
"tox -e pep8" runs it standalone and "tox" includes the pep8 env.
2016-08-01[refactor] remove unused design docs compilation codedrebs
2016-08-01[bug] retry allocation of gen instead of using a lockdrebs
The use of a lock to allocate the next generation of a change in couch backend suffers from at least 2 problems: 1. all modification to the couch database would have to be made through a soledad server entrypoint, otherwise the lock would have no effect. 2. introducing a lock makes code uglier, harder to debug, and prone to undesired blocks. The solution implemented by this commit is not so elegant, but works for what we need right now. Now, concurrent threads updating the couch database will race for the allocation of a new generation, and retry when they fail to do so. There's no high risk of getting blocked for too much time in the while loop because (1) there's always one thread that wins (what makes the expected number of retries to be N/2 if N is the number of concurrent threads), and (2) the number of concurrent attempts to update the user database is limited by the number of devices syncing at the same time.
2016-08-01[feat] use couch _all_docs for get_docs() and get_all_docs()drebs
The previous solution would make use of concurrent get's to couch backend in a pool of threads to implement the get_docs() and get_all_docs() CouchDatabase backend methods. This commit replaces those by a simpler implementation use the `_all_docs` couchdb view api. It passes all needed IDs to the view and r etrieves all documents with content in the same request. A comparison between both implementations shows an improvement of at least 15 times for large number of documents. The table below shows the time for different implementations of get_all_docs() for different number of documents and threads versus _all_docs implementation: +-------+-----------------+------------------+-------------+ | | threads | _all_docs | improvement | +-------+-----------------+------------------+-------------+ | 10 | 0.0728030204773 | 0.00782012939453 | 9.3 | | 100 | 0.609349966049 | 0.0377721786499 | 16.1 | | 1000 | 5.86522197723 | 0.370730876923 | 15.8 | | 10000 | 66.1713931561 | 3.61764383316 | 18.3 | +-------+-----------------+------------------+-------------+
2016-08-01[refactor] simplify couch whats_changed calculationdrebs
2016-08-01[bug] use couch lock to atomize saving of documentdrebs
2016-08-01[test] remove pip download cachedrebs
Recent versions of pip will ignore that option and use a cache anyway.
2016-08-01[pkg] add couch schema migration scriptdrebs
2016-08-01[feat] standardize metadata storage in couch backend.drebs
2016-08-01[test] remove duplicated function declarationVictor Shyba
`pytest_addoption` was declared twice making the second declaration replace the first, thus removing couch url parameter.
2016-08-01[test] add custom couchdb docker imagedrebs
2016-08-01[test] allow passing number of docs on command line on perf testsdrebs
2016-08-01[test] use random name for couchdb container in docker perf testdrebs
2016-08-01[test] add rules to run perf test on docker with separate couchdb server ↵drebs
container
2016-08-01[test] use pip download cache for tests and docker imagedrebs
2016-08-01[test] use docker image with couchdb service to run testsdrebs
2016-08-01[test] use tox and couchdb image to run testsdrebs