| Age | Commit message (Collapse) | Author |
|---|
|
When async decrypting, we want to finish as fast as possible. When encrypting,
though, we don't have such a rush. With an encryption loop period of 2
seconds, we're able to encrypt 30 documents in one minute (the current bitmask
client sync period), which is meaningful: should moderatelly use the processor
while not syncing and relief from some work when actually syncing.
|
|
Previous to this change, the actual encryption method used to run on its own
thread. When the close method was called from another thread, the queue could
be deleted after the encryption method loop had started, but before the queue
was checked for new items.
By removing that thread and moving the encryption loop to the reactor, that
race condition should disappear.
Closes: #7088.
|
|
Queue exceptions are not in multiprocessing.Queue module, but in plain Queue
instead.
|
|
|
|
|
|
- Related: #6359
|
|
|
|
Now the CouchDB based tests will try 3 times before give up due to
timeout or some temporary error. The stop function will also execute
properly even if the process wasn't created, leaving no lost files or
folders behind.
|
|
I tested that code and this cant happen. We need to iterate keys and
then ask 'del'. The previous method raised: RuntimeError: dictionary
changed size during iteration
|
|
When handling this exception Python got lost because the import was
incorrect. Queue.Empty comes from Queue, not from multiprocessing.Queue
|
|
As described in #4691, sometimes couch just hangs and all testing
freezes. This is due to no output from couch (sometimes meaning that an
error on boot wasn't logged). This adds a timeout and checks for it
while booting for tests. Closes #4691
|
|
CouchServerState does not use two extra parameters anymore. Tests are
now reflecting this change. Relate to #6833
|
|
The .ini used before was very old and did not contain all the config
values that couch needs. Using -n resets the configuration file chain,
making it use what is provided without including the system defaults
needed. To make it work with all versions the util will now copy
default.ini to temp dir and use it as additional config, overwriting
only what is needed with couchdb.ini.template provided.
|
|
this branch includes many changes that improve the asynchronous
retrieval of the sync docs, and the parallel decryption of the encrypted
documents.
|
|
|
|
Instead of opening one TCP connection for each HTTP request, we want to reuse
connections. Also, we need to be able to verify SSL certificates. This commit
implements both features in the twisted http client sync.
|
|
The whole idea of the encrypter/decrypter pool is to be able to use multiple
cores to allow parallel encryption/decryption. Previous to this commit, the
encryptor/decryptor pools could be configured to not use workers and instead
do encryption/decryption inline. That was meant for testing purposes and
defeated the purpose of the pools.
This commit removes the possibility of inline encrypting/decrypting when using
the pools. It also refactors the enc/dec pool code so any failures while using
the pool are correctly grabbed and raised to the top of the sync deferred
chain.
|
|
When we initialized the async decrypter pool in the target's init method we
needed a proxy to ensure we could update the insert doc callback with the
correct method later on. Now we initialize the decrypter only when we need it,
so we don't need this proxy anymore. This commit removes the unneeded proxy.
|
|
We have to make sure any failures in asynchronous decryption code is grabbed
and properly transmitted up the deferred chain so it can be logged. This
commit adds errbacks in the decryption pool that grab any failure and a
check on the http target the failure if that is the case.
|
|
|
|
|
|
|
|
This commit does the following:
* Remove the autocreate parameter from the sync() method.
* Remove the syncing lock from the sync module because it did the same job
as the lock in the sqlcipher module.
* Remove the close/stop methods from sync module as they don't make sense
after we started to use twisted in client-side sync.
|
|
This change uses twisted deferreds for the whole syncing process and paves the
way to implementing other transport schemes. It removes a lot of threaded code
that used locks and was very difficult to maintain, and lets twisted to the
dirty work. Furthermore, all blocking network i/o is now handled
asynchronously by the twisted.
This commit removes the possibility of interrupting a sync, and we should
reimplement it using cancellable deferreds if we need it.
|
|
The access to the sync db was modified to use twisted.enterprise.adbapi, but
only the asynchronous decryption of incoming documents during sync was
adapted. This commit modifies the asynchornous encryption of documents to also
use the adbapi for accessing the sync db.
|
|
|
|
When we started implementing the sync db, one of the ideas was to reuse the
data in the database in the case of a sync interruption. We don't do that now
and thus the pending documents exception is unneeded. This commit removes that
exception from the code.
|
|
The b64 encoding of the auth token was introducing an illegal character
(\n), which was breaking the authentication step since an exception was
being raised - when that multi-line header was attempted to be built.
this commit fixes that bug.
- Resolves: #6959
|
|
This commit actually does some different things:
* When doing asynchronous decryption of incoming documents in soledad client
during a sync, there was the possibility that a document corresponding to
a newer generation would be decrypted and inserted in the local database
before a document corresponding to an older generation. When this
happened, the metadata about the target database (i.e. its locally-known
generation) would be first updated to the newer generation, and then an
attempt to insert a document corresponding to an older generation would
cause the infamous InvalidGeneration error.
To fix that we use the sync-index information that is contained in the
sync stream to correctly find the insertable docs to be inserted in the
local database, thus avoiding the problem described above.
* Refactor the sync encrypt/decrypt pool to its own file.
* Fix the use of twisted adbapi with multiprocessing.
Closes: #6757.
|
|
Since we started implementing twisted api in soledad, some pieces are missing.
Accessing the sqlcipher database directly with the twisted adbapi facilities
is one of them. The async encryption/decryption was touching the database
directly, and this was causing some difficulties like having different threads
accessing the same database. This commit implements the twisted adbapi stuff
for the asynchronous encryption/decryption facilities.
Next steps would be use async adbapi for async encryption and use async adbapi
for all sqlcipher access.
|
|
|
|
We always got a log message saying "canceling sync threads" in the end of the
sync process, even when there was no error during the sync. This commit
changes that in a way that we only have that log when the sync was actually
cancelled because of an error.
|
|
Both deferred encryption and decryption rely on a special sync db. Previous to
this fix, the sync db was only initialized if a syncer was configured with
deferred encryption capabilities. This was a problem when the syncer was not
configured like so, but the actual sync method was initiated configured to do
deferred decryption.
This commit fixes this by always initializing the sync db, so we have the
option of doing all combinations of deferred encryption and decryption.
|
|
soledad-common versions before 0.6.5 do not contain the fix for #6833 and thus
will not work with most recent server. That is why we have to bump this
soledad-server dependency on soledad.common.
|
|
|
|
The database initialization on the client sync module is deferred to another
thread. As there is only one thread in the thread pool, this should not be a
problem for now, as operations will actually be queued in that thread. There
was some old code left from when we had to explicitelly wait for the db to be
initialize before using it. This commit removes that old code and introduces
some documentation so we remember to deal with deferred db init if we ever
change the number of threads in the thread pool.
|
|
|
|
Conversion of Twisted failures to string that rely on __str__ or __repr__
might not return all the information we would like to have, especially on sync
failures. This commit asks for a detailed traceback of such failures and logs
them both in Twisted and client logs.
|
|
This commit makes 2 changes that allow sync failures to raise exceptions that
can be caught by the api:
1. Remove try/except statements in sync.py level that would prevent an
exception to be caught by the soledad client api.
2. Ensure that if an asynchronous decrypting process fails the exception
will be re-raised to eventually reach the api.
Related: #6757.
|
|
With new soledad async api, we need to catch errors using errbacks instead of
catching exceptions explicitelly. This commit fixed the api sync() call to
intercept sync failures, log them, and do not propagate them down the callback
chain.
|
|
|
|
This commit removes some leftover code from a time when Soledad Server used to
check for permissions on certain databases when starting (i.e. shared and
tokens databases). This was later removed as correct permissions enforcement
was relayed to tapicero.
Closes: #6833.
|
|
With changes in soledad client api to make it async, the supporting scripts
also have to be updated. This commit also adds more functionalities as
exporting incoming mail and public or private keys.
|
|
because in OCD we trust.
|
|
In the past, we wanted dependency on leap.common to be optional, but now
because of the explicit use of the config path prefix and signaling, we want
to enforce dependency on leap.common.
|
|
Previous to this modification, leap.soledad.common.tests.u1db_tests was being
installed and its files were being included in the debian package. By
excluding *.tests and *.tests.* from find_packages() in setup.py, we make sure
that no test file will be installed not included in the final debian package.
|
|
Add dependency on twisted for Soledad Client. Also remove minimum twisted
version for Soledad Server because debian stable currently distributes 12.0.0
and pypi currently distributes 15.0.0.
Closes: #6797
|
|
Any solead release that includes this commit will be incompatible with LEAP
Platform < 0.6.1 because only from that version on the platform implements the
ephemeral monthly tokens databases.
Closes: #6785.
|
|
|
|
|
