| Age | Commit message (Collapse) | Author |
|---|
|
Moved out magic numbers into a constant and simplified logic during doc
upload.
|
|
|
|
|
|
|
|
batch is slower than usual insert for a single doc, so, if a document
exceeds the buffer, commit the batch (if any) and put the huge load by
traditional insert.
refactor coming.
|
|
This allow different paths for raw data and metadata, avoiding
unnecessary json parsing.
|
|
We enabled chunking, which means that a use can upload his entire db on
a single request. This commit makes server enable this and throttle
download as Twisted cant control the payload producer code as its
synchronous and blocking code.
|
|
1) enable HTTP 1.1 chunked upload on server
2) make the client sync.py generate a list of function calls instead of
a list of full docs
3) disable encryption pool
4) make the doc encryption a list of function calls
5) create a twisted protocol for sending
6) make a producer that calls the doc generation as necessary
|
|
This commit finishes reversion into u1db original streaming protocol for
downloads.
|
|
Will put a file object on doc json string if read_content is False,
otherwise it will fetch and fill as usual. This is useful for improving
server througput on sync download stream by receiving a bulk-get without
attachments and consume the file-objects as they come.
|
|
If a doc doesnt have a content it means it was deleted. Sync stream was
unable to represent this state.
|
|
couchdb lib returns a file object representing the attachment. This
commit dumps the read() call into the wsgi write() call. Doc
representation uses 2 lines also, separating metadata from content.
|
|
Instead of concurrent download, we are going to download a stream. This
commit modifies server to support it.
|
|
The couch state checks if all user databases have the correct schema
version on initialization, and may log errors and raise exceptions if
that is not the case. Because we are currently using `twistd web
--wsgi`, if those errors are logged too early the reactor may have not
been started and the twistd logging facilities may not have been
initialized.
This commit delays the state initialization until the reactor has been
started, to make sure any errors raised and logged in that stage will
actually reach the logfile.
|
|
Conflicts:
server/src/leap/soledad/server/__init__.py
testing/tests/conftest.py
|
|
When importing server, couch_state will load itself against couch_db url
configured on server. This fails when running on Docker as couchdb is in
another node.
|
|
CouchServerState is spread across test codebase and this option is
intended to be used only on server startup. This commit makes it default
to False and explicitly set it to True on where it's necessary.
|
|
|
|
|
|
|
|
From this moment on, we embed a fork of u1db called l2db.
|
|
to ease debugging of local servers w/o neededing the Token machinery in
place.
this needs still some extra changes to be fully functional:
- adapt the create-userdb script to work with no auth info.
|
|
|
|
|
|
|
|
While parsing the configuration file, if a key doesnt exist we need to
skip it.
|
|
All batching code has no effect by default with this commit. Since we
know that this is a dangerous new feature we will enable them only on
our test servers and check them manually before setting it as default
or adding more configuration features.
Use SyncTarget and server conf file to enable it for testing.
|
|
Batch support is optional. This commit adds a 'batching' configuration
option to disable it.
|
|
Created two methods on the backend to start and finish a batch. A dict of
callbacks is available to defer actions for the last document, allowing
temporary (changing often) metadata to be recorded only once.
Using those methods we will also be able to put all docs in one go on
the CouchDatabase implementation, but that is another step.
|
|
Creating a resource from a path to use get_json causes a lot of dirty
code and unexplained things like response[2]. This commit extracts that
logic into a helper to let it more clear about what is happening.
|
|
Added tests for this token verification as it wasn't covered. Then moved
it to the new couch module that implements a couch storage.
The ServerState was chosen to hold the verify_token method.
CouchServerState holds the current implementation, which is called on
authentication middleware as the new test shows.
|
|
First step of splitting classes across files on common.
backend.py holds SoledadBackend (generic backend logic)
couch/ is now a directory with old code inside __init__.py and
CouchServerState on state.py
Also removed mock IndexedSoledadBackend, since Soledad does not support
indexing due to encryption on server side.
Also fixed DesignDocUnknownError to show up what is the message of the
original exception. It was being lost.
|
|
CouchDatabase was renamed to SoledadBackend and a new class
CouchDatabase was created to hold all couchdb code. This should make
SoledadBackend less tied to database implementation. A few more
separations are needed to split into modules.
|
|
This was added for a Twisted 12 bug that should be gone by now.
|
|
|
|
LEAP Platform needs to granularly allow access on user database for
other services, like mx. This is now possible by editing
soledad-server.conf file. A new section 'database-security' was added
and it is parsed during 'create-user-db' to be set on security design
document, present on every per-user database.
|
|
couchdb library relies on garbage collector to close remaining
connections. Somehow, caching the instance is avoiding gc to call
__del__ on underlying couchdb sessions.
|
|
It was 3600s, but closing connections seems to yet depend on garbage
collection and now causes server to leak file handlers. 120s should be
enough to a sync session finish. Also, lowering this value will only
make very long syncs use more of couch every 2 minutes, while raising
this value will keep memory busy for useless time.
|
|
- Resolves: #7509
|
|
netrc file was hardcoded inside create-user-db. Now it reads the path
from /etc/leap/soledad-server.conf as done on server process.
The new configuration property is called 'admin_netrc'.
|
|
We can now use a custom script to create databases by setting a
parameter 'create_cmd' on soledad configuration.
This will set CouchServerState to use it on ensure_database.
|
|
|
|
As meskio found commented, setting this attribute directly is ugly,
CouchDatabase now has a init_caching method for setting up cache
instance.
|
|
Now each backend object will be retrieved from cache for sync.py and
values will live for 3600 by default. That is changed via parameter if
needed.
|
|
The CouchDB backend implementation was accessing CouchDB too many times
for the same values. Those values are known inside the same sync_id,
which is the id of current sync session.
This commit adds caching for all redundant calls to Couch inside the
same sync_id for each replica.
Refactoring is still needed, but for now couch.py works normally as if
caching is not present, while sync.py injects the cache as a attribute
to enable it. This needs a simpler implementation.
|
|
There are two functions in couch.py used to save and retrieve the last
know gen and trans id for the syncing replica. The get function is
called very often, but is only set on one point. Added a simple caching
to avoid queying couch for a value that we already have.
If cache is empty, it just query as usual and fills it.
|
|
This commit changes sync_state to be in memory, with all tests passing.
The memory variable for now is a dict with each key composed by
source_replica_uid and sync_id, replicating CouchDB implementation. Next
steps includes migrating this to Beaker and refactor/clean up code.
Changed the module's INFO dict to use Beaker's caching and adapted
methods to get and save from it. Still needs refactoring, all tests
passes.
Beaker is now using memory as default; It is configurable, but we aren't
opening the possibility of config now for security. We need to check
what can be misconfigured first.
We are not sure if beaker will be the definitive solution for server
side caching. This change isolates it with more granularity.
In order to replace it, just change get_cache_for to return the proper
caching object using another implementation. This caching object is
supposed to behave as a dict.
|
|
If we check for a BadRequest after calling meth_put we will end up on a
scenario where the server replies with an error, but everything got
processed.
|
|
|
|
|
