| Age | Commit message (Collapse) | Author |
|---|
|
Created two methods on the backend to start and finish a batch. A dict of
callbacks is available to defer actions for the last document, allowing
temporary (changing often) metadata to be recorded only once.
Using those methods we will also be able to put all docs in one go on
the CouchDatabase implementation, but that is another step.
|
|
u1db provides batching by default. Current Soledad HTTPS Sync Target was
stuck at 1 doc per request. This commit adds batching capability,
limiting the size to a predefined value.
Default limit size: 500kB
|
|
|
|
This line was missing an yield and without it we end up inserting a
document that is being retrieved and bad things happen.
This is the core fix from yesterday debugging session. During sequential
syncs the pool was inserting and querying at the same time and sometimes
repeating or failing to delete documents.
|
|
If we reset the vars after firing the finish callback, other thread can pick
up a dirty state on due concurrency.
|
|
by subclassing the MissingDesignDocError, we don't have to import the
soledad.common.couch submodule into the soledad.client.sync
- Resolves: #7626
|
|
errors.py was holding a few specific CouchDB errors, now moved into
couch.errors module. Also, some of CouchDatabase methods were declared
as private, but external classes needs them.
|
|
MissingDesignDocError raised on get_sync_info due to a missing design
document will be handled by the server during sync.
Ensure is now False by default, and thus database creation can deliver
an empty one that will be ensured during sync, following the ensure
parameter.
|
|
this is a workaroud to reduce the chances of failed sync due to
timeouts. this should be properly tackled by:
1. implementing proper cancellable for the sync operation.
2. implementing a retry count at the level of a single request, handled
internally by soledad.
in this way we can remove the retries logic from the soledadbootstrapper
in the bitmask client.
- Related: #7382
|
|
- Related: #7503
|
|
This error raises while getting info on target (or server) replica. On
previous implementation there was nothing to do here, but now that we
have db creation in place this error should be handled just like u1db
original implementation.
The reason is that db creation occurs during sync exchange, but before
this we try to ask for info and the code that checks for info raises an
error that will be used to signal the client that a database will be
created and that it must save the replica uid returned by server, after
sync exchange (where we send/fetch documents).
|
|
|
|
- bug: we were dumping the received secrets locally to disk *before*
setting the received property for the active secret, and therefore the
'active_secret' was always marked as null.
- refactor common code into an utility method.
|
|
|
|
We are getting "too many files open" while running tests with 1024 max
files open. This is a leak from close methods. Some of them should be fixed
on this commit, but further investigation may be necessary.
|
|
|
|
in this way we use the reactor pattern to dispatch the events, instead
of having the overhead of running a separate client thread.
- Resolves: #7274
|
|
Code is trying to close a closed threadpool. This raises errors on
Twisted 15.4.
|
|
- Resolves: #7412
|
|
The http_target.py refactor started in 8adf2dedb74941352520d8de42326b0c59818728
forgot to remove the old file.
|
|
|
|
From:
[refactor] removing getters and setters from couch.py
_couch_url was a private variable with getter and setter doing the same
as a public variable. It is accessed all over the code, so being private
with getters and setters didnt make sense. This commit fixes the tests
to also follow this style from now on.
|
|
Preparing many docs is useful for batching only. As we are sendind one
by one I will leave prepare_one_doc method to do the encrypt as the docs
goes to upload.
Also fixes method name as kaliy suggested.
|
|
isinstance is better, as kaliy pointed out, and the constructor is also
in a safer place on __init__.py to be explicit.
Also re-apply a change from last rebase;
|
|
There were some missing or on incorrect format (sphinx) as drebs and
kaliy pointed out.
|
|
* file headers
* variable names
* missing docstrings
* prune_conflicts
** extra: tests failed on a 1-based index bug
|
|
SoledadHTTPSyncTarget is composed of 4 main groups of responsibility:
* api.py - Public and main methods of a SyncTarget
* fetch.py - Document fetching logic
* send.py - Document sending logic
* support.py - Support functions and patches
Previous single file had ~600 lines with those 4 logic groups mixed,
making it harder to read and understand.
|
|
Just extracted some common logic to create u1db formatted requests
into RequestBody class and created new methods to represent operations
done during send_docs. This also removes send_one_doc, but does not add
batching yet. The single send behavior still the same, represented by the
parameter passed into RequestBody 'remove' method.
|
|
_prepare was being used to concatenate and prepare request body to send
or receive data on the format expected by the server. This behavior
wasnt clear, so I added a new class to abstract this out.
Content type and auth headers was being copied around methods. Now the
request method accepts a content_type parameter to remove this
duplication.
|
|
Creating a message, emitting an event and logging afterwards is a single
operation outside of of those method's responsabilities.
|
|
|
|
|
|
|
|
|
|
|
|
After we receive one document from the target database, we have to update the
target metadata or else we will not be able to succesfully return the new
generation and transaction id of the target when receiving exactly one
document during a sync.
|
|
Tests actually expect a tuple instead of a list on the return value of
get_sync_info().
|
|
|
|
Previous to this modification, the initialization of the sync decrypter pool
could happen concurrently with other database operations. That could cause the
pool to hang because it could be waiting for something that was mistakenly
deleted because of the wrong order of database operations.
This commit implements a standard which we already use in leap.keymanager and
leap.mail which makes some methods wait for the initialization operation
before they are actually called.
Closes: #7386
|
|
The old strategy for having Soledad sync running asynchronously with other
API calls was to have the sync running in its own threadpool. This is not
needed now that all sync code uses deferreds and will not block. This commit
removes what's left from the old threadpool.
|
|
Previous to this modification, the post-sync hooks were run after the sync
lock was released. In that scenario, the next sync process could start before
the previous sync's post-sync hooks were run. In general, we want the hooks to
run while the current sync lock is still locked, even though for some plugins
this might not make a difference.
|
|
|
|
|
|
|
|
When you tried to start a local sqlcipher that was created
before, with the wrong passphrase, the code was raising
a sqlcipher DatabaseError, there were tests covering this
but they were expecting a WrongMacError that was never raised.
I added code to wrap the DatabaseError and raise a new exception
DatabaseAccessError that is specific to soledad and adapted the
tests to expect it
|
|
multiprocessing.Queue is suitable for process communication, but its not
the ideal for a reactor model. This commit changes it to DeferredQueue,
where consumers and producers doesnt block and Twisted can handle them
better.
|
|
When we started to use the twisted http agent, we forgot to intercept http
response and raise the appropriate u1db errors based on the response status
code and messages. This commit implements that by redefining the http body
reader used by the http agent.
|
|
The encryption pool could be stopped twice and would break
on the second attempt because it deletes the encryption queue
variable. Added a condition to make sure it only deletes the
encryption queue if it exists, making it more idempotent
|
|
Change locking to be class based and each lock generated by db file
path.
|
|
Changes threading.lock to DeferredLock and checks syncing attribute by
looking into the lock state.
Also, applies more of startTwistedServer on tests that relies on
HTTP/1.1.
Fixes mock for events
|
