summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2016-08-03[pkg] support netrc couch access in migrate scriptdrebs
2016-08-02[test] avoid race condition on test_processing_orderVictor Shyba
test_processing_order aims to check that unordered docs wont be processed, but if we let the pool start and advance Twisted LoopingCall clock right before calling the processing method manually, the process method will run concurrently and cause a race condition issue.
2016-08-01[test] do not run pep8 tox env by defaultdrebs
2016-08-01[test] adds optional parallel env for local devVictor Shyba
2016-08-01[test] adds pep8 as a tox envVictor Shyba
"tox -e pep8" runs it standalone and "tox" includes the pep8 env.
2016-08-01[refactor] remove unused design docs compilation codedrebs
2016-08-01[bug] retry allocation of gen instead of using a lockdrebs
The use of a lock to allocate the next generation of a change in couch backend suffers from at least 2 problems: 1. all modification to the couch database would have to be made through a soledad server entrypoint, otherwise the lock would have no effect. 2. introducing a lock makes code uglier, harder to debug, and prone to undesired blocks. The solution implemented by this commit is not so elegant, but works for what we need right now. Now, concurrent threads updating the couch database will race for the allocation of a new generation, and retry when they fail to do so. There's no high risk of getting blocked for too much time in the while loop because (1) there's always one thread that wins (what makes the expected number of retries to be N/2 if N is the number of concurrent threads), and (2) the number of concurrent attempts to update the user database is limited by the number of devices syncing at the same time.
2016-08-01[feat] use couch _all_docs for get_docs() and get_all_docs()drebs
The previous solution would make use of concurrent get's to couch backend in a pool of threads to implement the get_docs() and get_all_docs() CouchDatabase backend methods. This commit replaces those by a simpler implementation use the `_all_docs` couchdb view api. It passes all needed IDs to the view and r etrieves all documents with content in the same request. A comparison between both implementations shows an improvement of at least 15 times for large number of documents. The table below shows the time for different implementations of get_all_docs() for different number of documents and threads versus _all_docs implementation: +-------+-----------------+------------------+-------------+ | | threads | _all_docs | improvement | +-------+-----------------+------------------+-------------+ | 10 | 0.0728030204773 | 0.00782012939453 | 9.3 | | 100 | 0.609349966049 | 0.0377721786499 | 16.1 | | 1000 | 5.86522197723 | 0.370730876923 | 15.8 | | 10000 | 66.1713931561 | 3.61764383316 | 18.3 | +-------+-----------------+------------------+-------------+
2016-08-01[refactor] simplify couch whats_changed calculationdrebs
2016-08-01[bug] use couch lock to atomize saving of documentdrebs
2016-08-01[test] remove pip download cachedrebs
Recent versions of pip will ignore that option and use a cache anyway.
2016-08-01[pkg] add couch schema migration scriptdrebs
2016-08-01[feat] standardize metadata storage in couch backend.drebs
2016-08-01[test] remove duplicated function declarationVictor Shyba
`pytest_addoption` was declared twice making the second declaration replace the first, thus removing couch url parameter.
2016-08-01[test] add custom couchdb docker imagedrebs
2016-08-01[test] allow passing number of docs on command line on perf testsdrebs
2016-08-01[test] use random name for couchdb container in docker perf testdrebs
2016-08-01[test] add rules to run perf test on docker with separate couchdb server ↵drebs
container
2016-08-01[test] use pip download cache for tests and docker imagedrebs
2016-08-01[test] use docker image with couchdb service to run testsdrebs
2016-08-01[test] use tox and couchdb image to run testsdrebs
2016-08-01[test] use tox to create docker imagedrebs
2016-08-01[test] remove pep8 from tox configdrebs
2016-08-01[test] avoid perf tests to be run on normal tox callsdrebs
Currently the perf tests use pytest-twisted plugin, and this has some implications in the old tests adapted from u1db that now use trial classes. Because of that, we exclude perf tests from usual tox calls, but you can still run them by explicitelly calling `tox perf`.
2016-08-01[test] allow custom couch url for perf testsdrebs
2016-08-01[test] allow custom couch url for couch testsdrebs
2016-08-01[test] add some payload to perf sync testsdrebs
2016-08-01[test] use pytest fixture scopes to provide per module soledad server for ↵drebs
perf tests
2016-08-01[test] add pytest initial setup for performance testsdrebs
2016-08-01[feat] use a lock for updating couch gen datadrebs
2016-08-01[bug] fix order of multipart serialization when writing to couchdrebs
The couch backend makes use of attachments and multipart structure for writing the document to the couch database. For that to work, the order in which attachments are described must match the actual order in which attachments are written to the couch http stream. This was not being properly taken care of, and eventually the json serializer was arbitrarilly ordering the attachments description in a way that it didn't match the actual order of attachments writing. This commit fixes that by using json.dumps() sort_keys parameter and making sure conflicts are always written before content.
2016-08-01[test] remove traces of design docs from couch testsdrebs
2016-08-01[feat] remove usage of design documents in couchdrebs
Design documents are slow and we already have alternatives to all uses we used to make of them, so this commit completelly removes all usage of design documents.
2016-07-25[test] adapt couch tests to use new generation/transaction storage schemedrebs
2016-07-25[feat] do not use couch views for sync metadatadrebs
When compared to plain couch document get, the use of the simplest view functions takes around double the time, while the use of the simplest list function can take more than 8 times: get 100 docs: total: 0.440337 secs mean: 0.004403 query 100 views: total: 0.911425 secs mean: 0.009114 query 100 lists: total: 3.711537 secs mean: 0.037115 Besides that, the current implementation of sync metadata storage over couch is dependent of timestamps of document puts, what can lead to metadata corruption if the clock of the system is changed for any reason. Because of these reasons, we seek to change the implementation of database metadata. This commit implements the storage of transaction log data on couch documents with special ids, in the form "gen-xxxxxxxxxx", where the x's are replaced by the generation index. Each generation document holds a dictionary containing the generation, doc_id and transaction_id for the changed document. For each modified document, a generation document is inserted holding the transaction metadata.
2016-07-25[feat] use _local couch docs for metadata storagedrebs
2016-07-25[test] remove ddocs param from docker setup scriptdrebs
2016-07-25[test] update docker readme and tododrebs
2016-07-25[test] fail gracefully on cert deletedrebs
2016-07-18[pkg] remove pixelated from requirements-latest.pipChristoph Kluenter
modifying original PR [0] by cristoph to account for the recent vendoring of l2db code, which means we no longer depend on u1db/dirspec. I expect the whole mess about the venv setup to be further simplified pretty soon, since we are going to merge most of the leap.* packages into a couple of repos. [0] https://github.com/leapcode/soledad/pull/327
2016-07-14[pkg] bump changelog to 0.8.10.8.1Kali Kaneko
2016-07-13[style] pep8Kali Kaneko
2016-07-12add pep8/flake8 to toxKali Kaneko
2016-07-12[tests] ignore tox folderKali Kaneko
2016-07-12[test] use tox in gitlab-cidrebs
2016-07-12[test] refactor test filesdrebs
2016-07-12[refactor] make u1db connection pool args explicitdrebs
2016-07-12[bug] properly trap db errors and close resourcesdrebs
SQLCipher database access errors can raise Soledad exceptions. Database access and multithreading resources are allocated in different places, so we have to be careful to close all multithreading mechanismis in case of database access errors. If we don't, zombie threads may haunt the reactor. This commit adds SQLCipher exception trapping and Soledad exception raising for database access errors, while properly shutting down multithreading resources.
2016-07-12[test] properly close dbpool on async testdrebs
2016-07-12[pkg] remove unneeded dirspec exceptionsdrebs