summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2015-08-20[bug] update target metadata after first receivedrebs
After we receive one document from the target database, we have to update the target metadata or else we will not be able to succesfully return the new generation and transaction id of the target when receiving exactly one document during a sync.
2015-08-20[bug] return tuple in get_sync_infodrebs
Tests actually expect a tuple instead of a list on the return value of get_sync_info().
2015-08-19[style] pep8 cleanupsKali Kaneko
2015-08-19[bug] wait for db init on sync decrypter pooldrebs
Previous to this modification, the initialization of the sync decrypter pool could happen concurrently with other database operations. That could cause the pool to hang because it could be waiting for something that was mistakenly deleted because of the wrong order of database operations. This commit implements a standard which we already use in leap.keymanager and leap.mail which makes some methods wait for the initialization operation before they are actually called. Closes: #7386
2015-08-19[refactor] remove threadpool from sqlcipher backenddrebs
The old strategy for having Soledad sync running asynchronously with other API calls was to have the sync running in its own threadpool. This is not needed now that all sync code uses deferreds and will not block. This commit removes what's left from the old threadpool.
2015-08-19[bug] hold sync lock while running post-sync hooksdrebs
Previous to this modification, the post-sync hooks were run after the sync lock was released. In that scenario, the next sync process could start before the previous sync's post-sync hooks were run. In general, we want the hooks to run while the current sync lock is still locked, even though for some plugins this might not make a difference.
2015-08-18[refactor] rename sync lock and improve docstringsdrebs
2015-08-18[refactor] clarify sync target closingdrebs
2015-08-18[bug] always release lock on spam scriptdrebs
2015-08-17[style] pep8 cleanupKali Kaneko
2015-08-17[tests] add pep8/flake8 ignoresKali Kaneko
2015-08-17[tests] add mutex sync testsdrebs
2015-08-17[tests] fix test_sqlcipher_syncdrebs
2015-08-17[tests] fix test_sync_deferred testdrebs
* Add possibility of passing sync_db and sync_enc_pool to target creation. * Ensure defer_encryption and defer_decryption are True when setting up the test environment.
2015-08-17[tests] fix tests from test_sync moduledrebs
* Remove test to assert database autocreation. We don't have that feature in soledad and so don't need that test here. * Remove redundant test without token authentication. * Fix server setup. * Skip interruptable sync test case, for now.
2015-08-17[bug] add missing parameter to unauthorized errordrebs
2015-08-12[bug] wrong sqlcipher passphrase now raises correctlyBruno Wagner
When you tried to start a local sqlcipher that was created before, with the wrong passphrase, the code was raising a sqlcipher DatabaseError, there were tests covering this but they were expecting a WrongMacError that was never raised. I added code to wrap the DatabaseError and raise a new exception DatabaseAccessError that is specific to soledad and adapted the tests to expect it
2015-08-12[tests] added inlineCallbacks to test with yieldBruno Wagner
test_db_sync_autocreate was using yield but didn't have the inlineCallbacks decorator, that made it a generator but the trial tests don't support generators. Fixed this
2015-08-12[bug] changes multiprocessing.Queue to Twisted'sVictor Shyba
multiprocessing.Queue is suitable for process communication, but its not the ideal for a reactor model. This commit changes it to DeferredQueue, where consumers and producers doesnt block and Twisted can handle them better.
2015-08-12[tests] Test encdecpool couldn't resolve inheritanceBruno Wagner
It was inheriting from twisted testcase in both u1db_tests.TestCase and in BaseSoledadTest, removed u1db testcase and everything is ok now
2015-08-12[style] Fixed pep8 warnings after PRBruno Wagner
Fixed pep8 warnings after latest PR
2015-08-12[bug] fix several things on testsdrebs
* redefine the BaseLeapTest class so it inherits from twisted.trial.unittest.TestCase. * allow passing the source_replica_uid when creating the sync target. This is needed because the SoledadHTTPSyncTarget needs to be instantiated with the source_replica_uid passed as a parameter. * close the sync target in tests that use it. * raise and U1DBError instead of an Exception on failure_and_retry test, because otherwise the u1db http app server will re-raise the Exception and twisted's WSGI server will raise the exception * set defer_decryption=False in some sync_exchange() calls so the call is consistent with soledad instantiation in tests and the new signature of that method. * yield get_sync_info() in tests that use this method.
2015-08-12[bug] raise u1db errors based on http responsedrebs
When we started to use the twisted http agent, we forgot to intercept http response and raise the appropriate u1db errors based on the response status code and messages. This commit implements that by redefining the http body reader used by the http agent.
2015-08-12[tests] #7370 opened, skipping test for nowVictor Shyba
While #7370 is being researched, it doesnt make sense to have this test running for now. Will be enabled again after investigation finishes.
2015-08-12[bug] Using lambda to avoid mistake on argumentsVictor Shyba
As suggested by drebs, the callback chaining can put None to sync, bringing undesired behavior.
2015-08-12[tests] Fixed double getURL in some testsBruno Wagner
Some calls to soledad_sync_target were being made with a full path, but the the method itself handles that, so I removed the extraneous calls those problems
2015-08-12[tests] Fixed flakiness in test_sync_targetBruno Wagner
The first test to be run in the TestSoledadParseReceivedDocResponse suite was erroring with connection refused, that is because it actually connects to the server, but wasn't setting it up, I changed it to inherit the simpler SoledadWithCouchServerMixin and adapted the setUp and tearDown, now it works everytime with no problems
2015-08-12[bug] Encdecpool won't explode if stopped twiceBruno Wagner
The encryption pool could be stopped twice and would break on the second attempt because it deletes the encryption queue variable. Added a condition to make sure it only deletes the encryption queue if it exists, making it more idempotent
2015-08-12[tests] Test sync adapted without u1db testsBruno Wagner
Test sync now doesn't inherit from u1db tests, it's tests will now be run and were failing, so I fixed them. Sync deferred will not be correctly handled in these tests. I also pulled over a method from the u1db test and there seems to be a pattern, I'll try to deduplicate these methods as soon as it gets clearer
2015-08-12[bug] Sync Locking based on class + db file pathVictor Shyba
Change locking to be class based and each lock generated by db file path.
2015-08-12[tests] Pulled test scenario req into sqlcipher sync testBruno Wagner
Pulled the requirement for the test scenario in sqlcipher sync so we could remove the dependency on the u1db test_sync
2015-08-12[tests] Fixed running sqlcipher_sync testsBruno Wagner
Sqlcipher sync tests were not being run because they were inheriting from a u1db class that was having the tests skipped, I removed that dependency and fixed the tests (they were expecting the synchronous, old http_target api).
2015-08-12[tests] Removed unused import in test_http_targetBruno Wagner
Test_http_target was using u1db test_sync in a lot of places, I removed those references on the last changes and am removing the dependency at once in this one
2015-08-12[tests] Adapted test_sync_target tests after changesBruno Wagner
Removed inherit from u1db test that was causing these tests to be ignored, the tests were broken with the latest changes to the sync_target, so I fixed them, no tests are being skipped on test_sync_target anymore
2015-08-12[tests] Sync very large files won't explode memoryBruno Wagner
Sync very large files will now fail early because it was exploding the memory, it needs some love, but for now I'll just make it fail and open an issue to take a better look at it
2015-08-12[bug] fixes concurrent sync and their testsVictor Shyba
Changes threading.lock to DeferredLock and checks syncing attribute by looking into the lock state. Also, applies more of startTwistedServer on tests that relies on HTTP/1.1. Fixes mock for events
2015-08-12[style] Fixed pep8 warningBruno Wagner
Two spaces before class, not one
2015-08-12[tests] Removed some of the dependencies on u1db_testsBruno Wagner
Some mixins were being called from u1db_tests but not used, what was really needed was pulled from there and the remaining dependencies were identified and used from the base (u1db_tests.__init__.py)
2015-08-12[tests] Removed dependency u1db test in parse testBruno Wagner
Started looking at what we can do to remove the u1db test dependencies directly, the parse was calling another class setup that was in fact the u1db testcase, changed that to testcase directly and it still works
2015-08-12[tests] U1db test file was not being used anywhereBruno Wagner
U1DB test_http_app file was not being required anywhere and it's tests were being skipped, removing it to clean up
2015-08-12[tests] Twisted WSGI is optionalVictor Shyba
Some tests still rely on python WSGI logic while others expect HTTP/1.1 from Twisted WSGI. While we can't use a single implementation for all, both are needed. startServer -> python default wsgi (http 1.0) startTwistedServer -> Twisted WSGI (http 1.1 'production-like')
2015-08-12[bug] close http_target when soledad closesVictor Shyba
Soledad has a close method that wasn't calling http_target close. The reference to sync exchange was being deleted without proper closing of underlying resources.
2015-08-12[test] Change test server to use TwistedVictor Shyba
Soledad was being served with python's default wsgi server, which uses HTTP 1.0. Since Soledad server doesn't send length or chunked headers, Twisted HTTP Client gets partial downloads when served with HTTP1.0. Soledad is meant to be used with Twisted WSGI container on HTTP/1.1 and the tests should reflect that.
2015-08-11[tests] add util function to instantiate dowser mem profilerKali Kaneko
2015-08-11[feat] control sending messages in scriptKali Kaneko
- ability to disable sending (to get raw receive times) - ability to specify size and number of payloads - ability to point to a file to be used as raw source of payloads
2015-08-11[feat] add support for twisted-aware theseus profilerKali Kaneko
2015-08-11[bug] disable events on profilingKali Kaneko
2015-08-11[feat] update profile-sync script, add plop profilingKali Kaneko
Updating the profile-sync script to the latest deferred-based sync. - Added a couple of options to have finer control about what output to get. - Add support for the plop profiler https://pypi.python.org/pypi/plop/ - To get meaningful plop profiles, make sure to disable the system stats collection, like this:: ./profile-sync.py --no-stats --plop -b /tmp/syncdata/ -p sikret user@provider A good practice for this is having a pre-seeded soledad account (currently you have to do that through the wizard, a cli will be very nice to have in the coming future) with a known amount of data (for instance, sending some mails with known payload weight). It is VERY IMPORTANT that you *NEVER* process the data in this account (ie, do not ever log in with a mail client, for instance, since that will alter the original documents). In order to have comparable results, you should always run this sync script in similar conditions. Ideally, on a machine with runlevel 1. Also, make sure of deleting the contents in the folder if you are not using a temporal dir.
2015-08-11[feat] emit a dict instead of a strIvan Alejandro
2015-08-06[feat] WHEELHOUSE can be a url + --use-leap-wheelsParménides GV
--use-leap-wheels sets --trusted-host (remove it when we have a proper cert) and WHEELHOUSE to https://ftp.lizard.leap.se Until we get ftp.lizard cname, use lizard as the wheels server. - Related: #7339