| Age | Commit message (Collapse) | Author |
|---|
|
The access to the sync db was modified to use twisted.enterprise.adbapi, but
only the asynchronous decryption of incoming documents during sync was
adapted. This commit modifies the asynchornous encryption of documents to also
use the adbapi for accessing the sync db.
|
|
When we started implementing the sync db, one of the ideas was to reuse the
data in the database in the case of a sync interruption. We don't do that now
and thus the pending documents exception is unneeded. This commit removes that
exception from the code.
|
|
The b64 encoding of the auth token was introducing an illegal character
(\n), which was breaking the authentication step since an exception was
being raised - when that multi-line header was attempted to be built.
this commit fixes that bug.
- Resolves: #6959
|
|
This commit actually does some different things:
* When doing asynchronous decryption of incoming documents in soledad client
during a sync, there was the possibility that a document corresponding to
a newer generation would be decrypted and inserted in the local database
before a document corresponding to an older generation. When this
happened, the metadata about the target database (i.e. its locally-known
generation) would be first updated to the newer generation, and then an
attempt to insert a document corresponding to an older generation would
cause the infamous InvalidGeneration error.
To fix that we use the sync-index information that is contained in the
sync stream to correctly find the insertable docs to be inserted in the
local database, thus avoiding the problem described above.
* Refactor the sync encrypt/decrypt pool to its own file.
* Fix the use of twisted adbapi with multiprocessing.
Closes: #6757.
|
|
Since we started implementing twisted api in soledad, some pieces are missing.
Accessing the sqlcipher database directly with the twisted adbapi facilities
is one of them. The async encryption/decryption was touching the database
directly, and this was causing some difficulties like having different threads
accessing the same database. This commit implements the twisted adbapi stuff
for the asynchronous encryption/decryption facilities.
Next steps would be use async adbapi for async encryption and use async adbapi
for all sqlcipher access.
|
|
|
|
We always got a log message saying "canceling sync threads" in the end of the
sync process, even when there was no error during the sync. This commit
changes that in a way that we only have that log when the sync was actually
cancelled because of an error.
|
|
Both deferred encryption and decryption rely on a special sync db. Previous to
this fix, the sync db was only initialized if a syncer was configured with
deferred encryption capabilities. This was a problem when the syncer was not
configured like so, but the actual sync method was initiated configured to do
deferred decryption.
This commit fixes this by always initializing the sync db, so we have the
option of doing all combinations of deferred encryption and decryption.
|
|
|
|
The database initialization on the client sync module is deferred to another
thread. As there is only one thread in the thread pool, this should not be a
problem for now, as operations will actually be queued in that thread. There
was some old code left from when we had to explicitelly wait for the db to be
initialize before using it. This commit removes that old code and introduces
some documentation so we remember to deal with deferred db init if we ever
change the number of threads in the thread pool.
|
|
|
|
Conversion of Twisted failures to string that rely on __str__ or __repr__
might not return all the information we would like to have, especially on sync
failures. This commit asks for a detailed traceback of such failures and logs
them both in Twisted and client logs.
|
|
This commit makes 2 changes that allow sync failures to raise exceptions that
can be caught by the api:
1. Remove try/except statements in sync.py level that would prevent an
exception to be caught by the soledad client api.
2. Ensure that if an asynchronous decrypting process fails the exception
will be re-raised to eventually reach the api.
Related: #6757.
|
|
With new soledad async api, we need to catch errors using errbacks instead of
catching exceptions explicitelly. This commit fixed the api sync() call to
intercept sync failures, log them, and do not propagate them down the callback
chain.
|
|
In the past, we wanted dependency on leap.common to be optional, but now
because of the explicit use of the config path prefix and signaling, we want
to enforce dependency on leap.common.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
* Fix arg passing to syncing failure method.
* Do not try to start sync loop which should be already running.
* Adapt to removal of old multiprocessing safe db, now accesses the
sqlcipher database directly.
|
|
* Add copy of SQLCipherOptions object to avoid modifying the options object
in place when instantiating the sync db.
* Add string representation of SQLCipherOptions for easiness of debugging.
* Make sync db always "ready".
* Fix passing options for sync db initialization.
* Fix typ0 that made SQLCipherU1DBSync._sync_loop be a tuple.
* Do not defer requests for stopping sync to a thread pool.
* Do not make pysqlcipher check if object is using in distinct threads.
* Reset the sync loop when stopping the syncer.
* Fix docstrings.
* Check for _db_handle attribute when closing the database.
|
|
* Remove check for HTTPS address.
* Remove creation of shared database.
* Fix docstrings.
|
|
* Adapt to removal of the old multiprocessing safe database, by directly
querying the sync database.
* Fix docstrings.
|
|
* Get replica uid upon U1DBConnectionPool initialization.
* Fix docstrings.
|
|
* Allow passing shared_db to Soledad constructor.
* Close syncers on Soledad close.
* Fix docstrings.
|
|
|
|
|
|
|
|
after drebs review
|
|
|
|
|
|
* Completed mapping of async dbpool
* Fixed shared db initialization.
Stuff To Be Fixed yet:
[ ] All inserts have to be done from the sync threadpool.
Right now we're reusing the connection from multiple
threads in the syncer. I'm assuming the writes are automatically
locking the file at the sqlite level, so this shouldn't pose a
problem.
[ ] Correctly handle the multiprocessing pool, and the callback
execution.
|
|
|
|
* Separate local storage, syncers and shared_db
* Comment out unused need_sync method
* Use twisted LoopingCall
* Create a threadpool for syncs
* Return deferred from sync method
* Do not pass crypto to SQLCipherDatabase
* Pass replica_uid to u1db_syncer
* Rename / reorganize some initialization methods
|
|
* add examples and benchmarks
* remove autocommit mode, allow wal disabling
* lock initialization
* make api use async calls
|
|
|
|
|
|
|
|
|
|
The changes introduced in aafa79c0f5 having to do with the cert
verification are incorrect, regarding the use of the newest ssl context
api introduced in python 2.7.9. There the use of the server setup was
taken, instead of the correct client options.
I hereby apologize for the insuficient testing on that fix. It happens
that I wrongly tested in an evironment that did the fallback to
pre-2.7.9 interpreter.
|
|
(#6400).
|
|
|
|
since ssl.SSLContext does not exist prior to python 2.7.9
|
|
Although the API can be misleading, PROTOCOL_SSLv23 selects the highest
protocol version that both the client and server support. Despite the
name, this option can select “TLS” protocols as well as “SSL”.
In this way, we can use TLSv1.2 (PROTOCOL_TLSv1 will *only* give us TLS
v1.0)
In the client side, we try to disable SSLv2 and SSLv3 options
explicitely.
The python version in wheezy does not offer PROTOCOL_TLSv1_2 nor
OP_NO_SSLv2 or OP_NO_SSLv3 (It's new in 2.7.9)
|
|
|
|
|
|
sync_target connection for it to be able to sync again
|
