| Age | Commit message (Collapse) | Author |
|---|
|
|
|
We are getting "too many files open" while running tests with 1024 max
files open. This is a leak from close methods. Some of them should be fixed
on this commit, but further investigation may be necessary.
|
|
in this way we use the reactor pattern to dispatch the events, instead
of having the overhead of running a separate client thread.
- Resolves: #7274
|
|
Code is trying to close a closed threadpool. This raises errors on
Twisted 15.4.
|
|
- Resolves: #7412
|
|
The http_target.py refactor started in 8adf2dedb74941352520d8de42326b0c59818728
forgot to remove the old file.
|
|
|
|
Preparing many docs is useful for batching only. As we are sendind one
by one I will leave prepare_one_doc method to do the encrypt as the docs
goes to upload.
Also fixes method name as kaliy suggested.
|
|
isinstance is better, as kaliy pointed out, and the constructor is also
in a safer place on __init__.py to be explicit.
Also re-apply a change from last rebase;
|
|
There were some missing or on incorrect format (sphinx) as drebs and
kaliy pointed out.
|
|
* file headers
* variable names
* missing docstrings
* prune_conflicts
** extra: tests failed on a 1-based index bug
|
|
SoledadHTTPSyncTarget is composed of 4 main groups of responsibility:
* api.py - Public and main methods of a SyncTarget
* fetch.py - Document fetching logic
* send.py - Document sending logic
* support.py - Support functions and patches
Previous single file had ~600 lines with those 4 logic groups mixed,
making it harder to read and understand.
|
|
Just extracted some common logic to create u1db formatted requests
into RequestBody class and created new methods to represent operations
done during send_docs. This also removes send_one_doc, but does not add
batching yet. The single send behavior still the same, represented by the
parameter passed into RequestBody 'remove' method.
|
|
_prepare was being used to concatenate and prepare request body to send
or receive data on the format expected by the server. This behavior
wasnt clear, so I added a new class to abstract this out.
Content type and auth headers was being copied around methods. Now the
request method accepts a content_type parameter to remove this
duplication.
|
|
Creating a message, emitting an event and logging afterwards is a single
operation outside of of those method's responsabilities.
|
|
|
|
|
|
After we receive one document from the target database, we have to update the
target metadata or else we will not be able to succesfully return the new
generation and transaction id of the target when receiving exactly one
document during a sync.
|
|
Tests actually expect a tuple instead of a list on the return value of
get_sync_info().
|
|
|
|
Previous to this modification, the initialization of the sync decrypter pool
could happen concurrently with other database operations. That could cause the
pool to hang because it could be waiting for something that was mistakenly
deleted because of the wrong order of database operations.
This commit implements a standard which we already use in leap.keymanager and
leap.mail which makes some methods wait for the initialization operation
before they are actually called.
Closes: #7386
|
|
The old strategy for having Soledad sync running asynchronously with other
API calls was to have the sync running in its own threadpool. This is not
needed now that all sync code uses deferreds and will not block. This commit
removes what's left from the old threadpool.
|
|
Previous to this modification, the post-sync hooks were run after the sync
lock was released. In that scenario, the next sync process could start before
the previous sync's post-sync hooks were run. In general, we want the hooks to
run while the current sync lock is still locked, even though for some plugins
this might not make a difference.
|
|
|
|
|
|
|
|
When you tried to start a local sqlcipher that was created
before, with the wrong passphrase, the code was raising
a sqlcipher DatabaseError, there were tests covering this
but they were expecting a WrongMacError that was never raised.
I added code to wrap the DatabaseError and raise a new exception
DatabaseAccessError that is specific to soledad and adapted the
tests to expect it
|
|
multiprocessing.Queue is suitable for process communication, but its not
the ideal for a reactor model. This commit changes it to DeferredQueue,
where consumers and producers doesnt block and Twisted can handle them
better.
|
|
When we started to use the twisted http agent, we forgot to intercept http
response and raise the appropriate u1db errors based on the response status
code and messages. This commit implements that by redefining the http body
reader used by the http agent.
|
|
The encryption pool could be stopped twice and would break
on the second attempt because it deletes the encryption queue
variable. Added a condition to make sure it only deletes the
encryption queue if it exists, making it more idempotent
|
|
Change locking to be class based and each lock generated by db file
path.
|
|
Changes threading.lock to DeferredLock and checks syncing attribute by
looking into the lock state.
Also, applies more of startTwistedServer on tests that relies on
HTTP/1.1.
Fixes mock for events
|
|
Soledad has a close method that wasn't calling http_target close. The
reference to sync exchange was being deleted without proper closing of
underlying resources.
|
|
|
|
All the response parse tests are passing now, response
with no entries was broken because it wasn't being treated
and the others were broken because of calls that no longer
existed
|
|
Created a setup for the http target tests
Fixed two tests relying on http target that were
outdated
Fixed a call for an exception that doesn't exist, it
won't break anymore if it gets to that exception
|
|
Line break before binary operator breaks PEP8, fixed
that in the client api.py
|
|
before sqlcipher backend, or the attribute is not found.
this is a leftover of the recent refactor
|
|
|
|
|
|
|
|
SoledadCrypto had Soledad as parameter to be able to use
SoledadSecrets. SoledadSecrets had SoledadCrypto as parameter to use
*crypt_sym. This commit removes this circular dependency passing
directly the secret that SoledadCrypto cares about to the constructor
and removing the *crypt_sym methods from SoledadCrypto.
- Resolves: #7338
|
|
* change close method name to stop
* add start/stop methods to both enc/dec clases
* remove any delayed calls on pool shutdown
|
|
|
|
The bolean operator must come before a line break, not after
according to pep8
|
|
Because of how the incoming document queue is implemented, it could be the
case that a document was sent to async decryption queue more than once. This
commit creates a list of documents to be decrypted, so we avoid sending the
same document to the queue more than once.
|
|
|
|
The incoming documents events are meant to be used by a progress bar for
soledad sync, yet to be implemented. When deferred decryption was used, the
events were sent out of order, depending on the order of arrival of the
documents. This commit changes it so that the content of the emited events are
in order, so it is meaningful for the implementation of a progress bar.
Note that even after documents are received from the server, they will still
be decrypted asynchronously, so another signal could be implemented to signal
for the waiting of the decryption of incoming documents.
|
|
This is how a secret was stored in the secrets json file:
* each secret is symmetrically encrypted amd MACed with keys derived from
the user's passphrase.
* the encrypted secrets dictionary is then MACed with another key derived
* from the user's passphrase.
* each key is derived using scrypt and a unique random salt.
There are disadvantages to this approach:
* repeating scrypt many times is a waste of time.
* an attacker could crack whichever has weaker parameters, if they get out
of sync.
* if an attacker can modify the secret in a way it is good to decrypt the
database, then she can also modify the MAC.
The solution for this is:
* completelly eliminate the MAC from the storage secrets file.
* attempt to decrypt the database with whatever is got from the decryption
of the secret. If that is wrong, report an error.
Closes #6980.
|
|
resulting from the previous pep8 cleanup
|
