summaryrefslogtreecommitdiff
path: root/client/src/leap
AgeCommit message (Collapse)Author
2015-07-22[refactor] use a deferred to signal when SyncDecriptionPool has finishedRuben Pollan
It makes the code simpler and clearer to use a deferred instead of having to pull on 'has_finished'. - Related: #7234
2015-06-30[feature] add close() method to client http targetdrebs
HTTP client cached connections will hang around in the reactor if they are not properly cleaned up, and might raise a "reactor unclean" message on shutdown. This commit adds a close() method to the client http target that will cleanup those connections.
2015-06-15[style] minor style fixes for correctionKali Kaneko
after suggestions in the review
2015-06-15[refactor] use twisted logging facility to log failureKali Kaneko
2015-06-15[refactor] move collect_plugins to leap.commonKali Kaneko
2015-06-15[feature] filter documents before passing them to the pluginKali Kaneko
2015-06-15[feature] use raw operation, returns no resultKali Kaneko
2015-06-15[feature] add post-sync hooks using twisted pluginsKali Kaneko
implementing a generic plugin interface to allow other modules to react to soledad syncs, receiving a list of document ids that they've subscribed to. - Resolves: #6996 - Releases: 0.7.1
2015-06-15[bug] allow reuse of decr poolKali Kaneko
2015-06-03[feature] increase the async encryption perioddrebs
When async decrypting, we want to finish as fast as possible. When encrypting, though, we don't have such a rush. With an encryption loop period of 2 seconds, we're able to encrypt 30 documents in one minute (the current bitmask client sync period), which is meaningful: should moderatelly use the processor while not syncing and relief from some work when actually syncing.
2015-06-03[bug] do not block when getting doc for async encdrebs
Previous to this change, the actual encryption method used to run on its own thread. When the close method was called from another thread, the queue could be deleted after the encryption method loop had started, but before the queue was checked for new items. By removing that thread and moving the encryption loop to the reactor, that race condition should disappear. Closes: #7088.
2015-06-03[bug] use exception from correct moduledrebs
Queue exceptions are not in multiprocessing.Queue module, but in plain Queue instead.
2015-06-03[bug] use correct sync enc pool queue attributedrebs
2015-05-27[feat] adapt to new events api on commonIvan Alejandro
- Related: #6359
2015-05-27[refactor] move the twisted http code to leap.commonRuben Pollan
2015-05-26[bug] dictionary cant be modified during iterationVictor Shyba
I tested that code and this cant happen. We need to iterate keys and then ask 'del'. The previous method raised: RuntimeError: dictionary changed size during iteration
2015-05-26[bug] Empty comes from QueueVictor Shyba
When handling this exception Python got lost because the import was incorrect. Queue.Empty comes from Queue, not from multiprocessing.Queue
2015-05-25[bug] remove client syncer call to close methoddrebs
2015-05-25[feature] add pool of http/https connectionsdrebs
Instead of opening one TCP connection for each HTTP request, we want to reuse connections. Also, we need to be able to verify SSL certificates. This commit implements both features in the twisted http client sync.
2015-05-22[refactor] remove inline enc/dec from client pooldrebs
The whole idea of the encrypter/decrypter pool is to be able to use multiple cores to allow parallel encryption/decryption. Previous to this commit, the encryptor/decryptor pools could be configured to not use workers and instead do encryption/decryption inline. That was meant for testing purposes and defeated the purpose of the pools. This commit removes the possibility of inline encrypting/decrypting when using the pools. It also refactors the enc/dec pool code so any failures while using the pool are correctly grabbed and raised to the top of the sync deferred chain.
2015-05-21[refactor] remove unneeded proxy for insert_doc_cbdrebs
When we initialized the async decrypter pool in the target's init method we needed a proxy to ensure we could update the insert doc callback with the correct method later on. Now we initialize the decrypter only when we need it, so we don't need this proxy anymore. This commit removes the unneeded proxy.
2015-05-20[bug] ensure async decryption failures are loggeddrebs
We have to make sure any failures in asynchronous decryption code is grabbed and properly transmitted up the deferred chain so it can be logged. This commit adds errbacks in the decryption pool that grab any failure and a check on the http target the failure if that is the case.
2015-05-20[feature] ensure reactor stops on client db scriptdrebs
2015-05-20[bug] wrap unauth errors as invalid token errorsdrebs
2015-05-20[bug] ensure sync failures are not ignoreddrebs
2015-05-20[refactor] cleanup sync, remove unused stuffdrebs
This commit does the following: * Remove the autocreate parameter from the sync() method. * Remove the syncing lock from the sync module because it did the same job as the lock in the sqlcipher module. * Remove the close/stop methods from sync module as they don't make sense after we started to use twisted in client-side sync.
2015-05-20[feature] use twisted.web.client in client syncdrebs
This change uses twisted deferreds for the whole syncing process and paves the way to implementing other transport schemes. It removes a lot of threaded code that used locks and was very difficult to maintain, and lets twisted to the dirty work. Furthermore, all blocking network i/o is now handled asynchronously by the twisted. This commit removes the possibility of interrupting a sync, and we should reimplement it using cancellable deferreds if we need it.
2015-05-20[feature] use twisted adbapi for async encryptiondrebs
The access to the sync db was modified to use twisted.enterprise.adbapi, but only the asynchronous decryption of incoming documents during sync was adapted. This commit modifies the asynchornous encryption of documents to also use the adbapi for accessing the sync db.
2015-05-20[bug] remove unused pending documents exceptiondrebs
When we started implementing the sync db, one of the ideas was to reuse the data in the database in the case of a sync interruption. We don't do that now and thus the pending documents exception is unneeded. This commit removes that exception from the code.
2015-05-20[bug] remove illegal CR from auth headerKali Kaneko
The b64 encoding of the auth token was introducing an illegal character (\n), which was breaking the authentication step since an exception was being raised - when that multi-line header was attempted to be built. this commit fixes that bug. - Resolves: #6959
2015-05-20[bug] fix order of insertion of decrypted docsdrebs
This commit actually does some different things: * When doing asynchronous decryption of incoming documents in soledad client during a sync, there was the possibility that a document corresponding to a newer generation would be decrypted and inserted in the local database before a document corresponding to an older generation. When this happened, the metadata about the target database (i.e. its locally-known generation) would be first updated to the newer generation, and then an attempt to insert a document corresponding to an older generation would cause the infamous InvalidGeneration error. To fix that we use the sync-index information that is contained in the sync stream to correctly find the insertable docs to be inserted in the local database, thus avoiding the problem described above. * Refactor the sync encrypt/decrypt pool to its own file. * Fix the use of twisted adbapi with multiprocessing. Closes: #6757.
2015-05-20[feature] use async adbapi for async decryptiondrebs
Since we started implementing twisted api in soledad, some pieces are missing. Accessing the sqlcipher database directly with the twisted adbapi facilities is one of them. The async encryption/decryption was touching the database directly, and this was causing some difficulties like having different threads accessing the same database. This commit implements the twisted adbapi stuff for the asynchronous encryption/decryption facilities. Next steps would be use async adbapi for async encryption and use async adbapi for all sqlcipher access.
2015-05-20[bug] fix log messages for secrets in storagedrebs
2015-05-20[bug] fix log messages when fetching documentsdrebs
We always got a log message saying "canceling sync threads" in the end of the sync process, even when there was no error during the sync. This commit changes that in a way that we only have that log when the sync was actually cancelled because of an error.
2015-05-20[bug] always initialize sync dbdrebs
Both deferred encryption and decryption rely on a special sync db. Previous to this fix, the sync db was only initialized if a syncer was configured with deferred encryption capabilities. This was a problem when the syncer was not configured like so, but the actual sync method was initiated configured to do deferred decryption. This commit fixes this by always initializing the sync db, so we have the option of doing all combinations of deferred encryption and decryption.
2015-04-28[doc] fix doc on why to re-raise sync exceptionsdrebs
2015-04-28[bug] remove old code for sync deferred db initdrebs
The database initialization on the client sync module is deferred to another thread. As there is only one thread in the thread pool, this should not be a problem for now, as operations will actually be queued in that thread. There was some old code left from when we had to explicitelly wait for the db to be initialize before using it. This commit removes that old code and introduces some documentation so we remember to deal with deferred db init if we ever change the number of threads in the thread pool.
2015-04-28[refactor] rename sync callbacks on client apidrebs
2015-04-28[bug] log traceback on sync failures on clientdrebs
Conversion of Twisted failures to string that rely on __str__ or __repr__ might not return all the information we would like to have, especially on sync failures. This commit asks for a detailed traceback of such failures and logs them both in Twisted and client logs.
2015-04-23[feat] ensure sync fails will raise an exceptiondrebs
This commit makes 2 changes that allow sync failures to raise exceptions that can be caught by the api: 1. Remove try/except statements in sync.py level that would prevent an exception to be caught by the soledad client api. 2. Ensure that if an asynchronous decrypting process fails the exception will be re-raised to eventually reach the api. Related: #6757.
2015-04-23[bug] fail gracefully when sync failsdrebs
With new soledad async api, we need to catch errors using errbacks instead of catching exceptions explicitelly. This commit fixed the api sync() call to intercept sync failures, log them, and do not propagate them down the callback chain.
2015-03-19[fix] add explicit dependency on leap.commondrebs
In the past, we wanted dependency on leap.common to be optional, but now because of the explicit use of the config path prefix and signaling, we want to enforce dependency on leap.common.
2015-02-20cutoff for encoding detectionKali Kaneko
2015-02-11Retry on SQLCipher timeout (#6625).drebs
2015-02-11add some benchmarking skeletonKali Kaneko
2015-02-11add raw sqlcipher query methodKali Kaneko
2015-02-11Do not try to unlock shared db if db is not syncable.drebs
2015-02-11Do not try to close db syncer if db is not syncable.drebs
2015-02-11Add local replica info to sync debug output.drebs
2015-02-11Several fixes in SoledadSyncTarget:drebs
* Fix arg passing to syncing failure method. * Do not try to start sync loop which should be already running. * Adapt to removal of old multiprocessing safe db, now accesses the sqlcipher database directly.