| Age | Commit message (Collapse) | Author |
|---|
|
Now each backend object will be retrieved from cache for sync.py and
values will live for 3600 by default. That is changed via parameter if
needed.
|
|
Before this change, we used a complicated update handler for storing the sync
state on the couchdb backend. That update handler was implemented as an
attempt to make couchdb take care of some validation for the update of the
sync log during the sync exchange, mainly to allow concurrent received
documents insertion during a sync.
Right now we rely on the remote sending one document at a time and do not
support concurrent insertions in the remote database backed by couch. Because
of that, the code removed by this commit was unneeded. And more: it was a
bottleneck of the sync process because we were writing to an unique file and
using unnecessary couch design docs processing for that. So this commit both
simplifies the storage of remote sync and removes a bottleneck of the sync
process.
Conflicts:
common/src/leap/soledad/common/couch.py
common/src/leap/soledad/common/tests/test_couch.py
|
|
The CouchDB backend implementation was accessing CouchDB too many times
for the same values. Those values are known inside the same sync_id,
which is the id of current sync session.
This commit adds caching for all redundant calls to Couch inside the
same sync_id for each replica.
Refactoring is still needed, but for now couch.py works normally as if
caching is not present, while sync.py injects the cache as a attribute
to enable it. This needs a simpler implementation.
|
|
There are two functions in couch.py used to save and retrieve the last
know gen and trans id for the syncing replica. The get function is
called very often, but is only set on one point. Added a simple caching
to avoid queying couch for a value that we already have.
If cache is empty, it just query as usual and fills it.
|
|
Python has a native ThreadPool implementation that fits our needs.
Changing it to use this instead and making some calls simpler.
|
|
_put_doc_if_newer is implemented on CommonBackend already. This was
copied over to CouchBackend just to add ensure conflicts. We can do this
before calling the super method instead.
|
|
This commit changes sync_state to be in memory, with all tests passing.
The memory variable for now is a dict with each key composed by
source_replica_uid and sync_id, replicating CouchDB implementation. Next
steps includes migrating this to Beaker and refactor/clean up code.
Changed the module's INFO dict to use Beaker's caching and adapted
methods to get and save from it. Still needs refactoring, all tests
passes.
Beaker is now using memory as default; It is configurable, but we aren't
opening the possibility of config now for security. We need to check
what can be misconfigured first.
We are not sure if beaker will be the definitive solution for server
side caching. This change isolates it with more granularity.
In order to replace it, just change get_cache_for to return the proper
caching object using another implementation. This caching object is
supposed to behave as a dict.
|
|
Soledad server will use Beaker as cache provider, starting with
sync_state being in memory.
|
|
|
|
This tests the previous fix on ensuring a db that is missing a doc other
than 'docs'.
|
|
This code only checks for 'docs' presence, while we have 3 design
documents. If one of them is missing, but 'docs' is not, then it will
not ensure the others.
This is needed to properly ensure ddocs on create command line script.
|
|
|
|
|
|
|
|
|
|
This was used during db isolation to make sure that everything created
was destroyed, but it fails with -j (multiprocess). Removing it allows
parallelism.
|
|
We are getting "too many files open" while running tests with 1024 max
files open. This is a leak from close methods. Some of them should be fixed
on this commit, but further investigation may be necessary.
|
|
|
|
in this way we use the reactor pattern to dispatch the events, instead
of having the overhead of running a separate client thread.
- Resolves: #7274
|
|
|
|
Removing unused code as this test case does not need a server.
|
|
Big tearDown logic can be replaced by a simple addCleanup. Also remove
unused imports and fix a small typo on a database cleanup check.
|
|
This test only defines a set of different scenarios, all other methods
are the same as this subclass.
|
|
self.db3 is closed on tearDown. This test was creating it as a local
variable, making close possibly fail.
|
|
It was hardcoded for 'test', but the database name is now random. What
is useful for test code is the replica_uid, database name for the
SyncTarget is now coming from database name.
|
|
'shared' has to be used as a DB name just because of a constant, but it
is used on only one point. This changes mock this point to have unique
names for better tests isolation. 'tokens' was removed as unnecessary.
|
|
This was a duplicate, but also was getting on the way to improve
isolation. With this small refactor it should be cleaner and have unique
names.
|
|
Test case has a addCleanup method, which provides a way to clean
resources up and express this need as soon as you create. We are now
using it to simplify some logic on database deletion during the test and
to make sure that as soon as it gets created a cleanup is there to
delete after.
|
|
We are using a single CouchDB install, which may cause tests to overlap
since many of them uses the same database name, hurting isolation. This
change tries to use uuid on most of it. Also changes for couch_url and
couch_port introduced by removal of CouchDB process.
|
|
This process per test gives a lot of headache, this is why we are
removing it. With it we would need to try to start and stop properly on
each test case. This fails badly when a test fail and, depending on how
it fails, it freezes my pc. Also, it is very heavy for a CI to run a
database process for each test case.
|
|
|
|
|
|
Code is trying to close a closed threadpool. This raises errors on
Twisted 15.4.
|
|
|
|
- Resolves: #7412
|
|
The http_target.py refactor started in 8adf2dedb74941352520d8de42326b0c59818728
forgot to remove the old file.
|
|
|
|
From:
[refactor] removing getters and setters from couch.py
_couch_url was a private variable with getter and setter doing the same
as a public variable. It is accessed all over the code, so being private
with getters and setters didnt make sense. This commit fixes the tests
to also follow this style from now on.
|
|
Preparing many docs is useful for batching only. As we are sendind one
by one I will leave prepare_one_doc method to do the encrypt as the docs
goes to upload.
Also fixes method name as kaliy suggested.
|
|
isinstance is better, as kaliy pointed out, and the constructor is also
in a safer place on __init__.py to be explicit.
Also re-apply a change from last rebase;
|
|
There were some missing or on incorrect format (sphinx) as drebs and
kaliy pointed out.
|
|
* file headers
* variable names
* missing docstrings
* prune_conflicts
** extra: tests failed on a 1-based index bug
|
|
Necessary methods are on CouchDocument, but we accept a Document as
well, in this case self._factory is needed.
Will be simpler soon.
|
|
This was being calculated inside CouchDatabase, but it is not a
persistence responsibility. It clearly doesn't belong to this
persistence layer and seeing both sides separated allow us to work
better on both parts.
|
|
|
|
Adding, removing and checking conflicts is an operation done by the
model, the Database shouldn't be aware of that. Fetching and saving also
is not model's responsability. Repetition remove as well.
|
|
|
|
This is not needed, the behavior under them is the same as an
assignment.
|
|
SoledadHTTPSyncTarget is composed of 4 main groups of responsibility:
* api.py - Public and main methods of a SyncTarget
* fetch.py - Document fetching logic
* send.py - Document sending logic
* support.py - Support functions and patches
Previous single file had ~600 lines with those 4 logic groups mixed,
making it harder to read and understand.
|
|
Just extracted some common logic to create u1db formatted requests
into RequestBody class and created new methods to represent operations
done during send_docs. This also removes send_one_doc, but does not add
batching yet. The single send behavior still the same, represented by the
parameter passed into RequestBody 'remove' method.
|
