summaryrefslogtreecommitdiff
path: root/server/src/leap/soledad
AgeCommit message (Collapse)Author
2016-12-12[refactor] simplify server insertVictor Shyba
Moved out magic numbers into a constant and simplified logic during doc upload.
2016-12-12[bug] remove print debug statementsKali Kaneko
2016-12-12[bug] fix bad merge in imports blockKali Kaneko
2016-12-12[refactor] adapt fetcher to decryptorKali Kaneko
2016-12-12[feature] batch based on payload sizeVictor Shyba
batch is slower than usual insert for a single doc, so, if a document exceeds the buffer, commit the batch (if any) and put the huge load by traditional insert. refactor coming.
2016-12-12[feature] stream content in a separate lineVictor Shyba
This allow different paths for raw data and metadata, avoiding unnecessary json parsing.
2016-12-12[feature] make the test accept large uploadsVictor Shyba
We enabled chunking, which means that a use can upload his entire db on a single request. This commit makes server enable this and throttle download as Twisted cant control the payload producer code as its synchronous and blocking code.
2016-12-12[feature] upload streamingVictor Shyba
1) enable HTTP 1.1 chunked upload on server 2) make the client sync.py generate a list of function calls instead of a list of full docs 3) disable encryption pool 4) make the doc encryption a list of function calls 5) create a twisted protocol for sending 6) make a producer that calls the doc generation as necessary
2016-12-12[feature] streaming download protocolVictor Shyba
This commit finishes reversion into u1db original streaming protocol for downloads.
2016-12-12[feature] make reading attachments optionalVictor Shyba
Will put a file object on doc json string if read_content is False, otherwise it will fetch and fill as usual. This is useful for improving server througput on sync download stream by receiving a bulk-get without attachments and consume the file-objects as they come.
2016-12-12[bug] use an empty string to represent tumbstonesVictor Shyba
If a doc doesnt have a content it means it was deleted. Sync stream was unable to represent this state.
2016-12-12[feature] server download stream from file objectVictor Shyba
couchdb lib returns a file object representing the attachment. This commit dumps the read() call into the wsgi write() call. Doc representation uses 2 lines also, separating metadata from content.
2016-12-12[feature] revert sync download into straming (server)Victor Shyba
Instead of concurrent download, we are going to download a stream. This commit modifies server to support it.
2016-11-27[bug] delay couch state initializationdrebs
The couch state checks if all user databases have the correct schema version on initialization, and may log errors and raise exceptions if that is not the case. Because we are currently using `twistd web --wsgi`, if those errors are logged too early the reactor may have not been started and the twistd logging facilities may not have been initialized. This commit delays the state initialization until the reactor has been started, to make sure any errors raised and logged in that stage will actually reach the logfile.
2016-11-22[refactor] separate server application into another filedrebs
Conflicts: server/src/leap/soledad/server/__init__.py testing/tests/conftest.py
2016-10-21[tests] fix server importVictor Shyba
When importing server, couch_state will load itself against couch_db url configured on server. This fails when running on Docker as couchdb is in another node.
2016-10-21[tests] make check_schema_versions default to FalseVictor Shyba
CouchServerState is spread across test codebase and this option is intended to be used only on server startup. This commit makes it default to False and explicitly set it to True on where it's necessary.
2016-10-03[feature] check for user dbs couch schema versionsdrebs
2016-10-03[refactor] move configuration loading to its own moduledrebs
2016-09-22[feat] centralize logging and use twisted.logger by defaultdrebs
2016-07-12[refactor] make tests use l2db submoduleKali Kaneko
From this moment on, we embed a fork of u1db called l2db.
2016-06-06[feature] debug-mode server with dummy authenticationKali Kaneko
to ease debugging of local servers w/o neededing the Token machinery in place. this needs still some extra changes to be fully functional: - adapt the create-userdb script to work with no auth info.
2016-05-16[style] pep8Kali Kaneko
2016-04-26[refactor] remove shared db locking from serverdrebs
2016-04-01[pkg] updated to versioneer 0.16 (patched)Kali Kaneko
2015-12-04[bug] skip missing keys on existing sectionsVictor Shyba
While parsing the configuration file, if a key doesnt exist we need to skip it.
2015-12-03[feat] set default to False on batching for nowVictor Shyba
All batching code has no effect by default with this commit. Since we know that this is a dangerous new feature we will enable them only on our test servers and check them manually before setting it as default or adding more configuration features. Use SyncTarget and server conf file to enable it for testing.
2015-12-03[feat] add configuration to disable batchingVictor Shyba
Batch support is optional. This commit adds a 'batching' configuration option to disable it.
2015-12-03[feat] prepare server to handle batchesVictor Shyba
Created two methods on the backend to start and finish a batch. A dict of callbacks is available to defer actions for the last document, allowing temporary (changing often) metadata to be recorded only once. Using those methods we will also be able to put all docs in one go on the CouchDatabase implementation, but that is another step.
2015-11-03[refactor] resource logic encapsulationVictor Shyba
Creating a resource from a path to use get_json causes a lot of dirty code and unexplained things like response[2]. This commit extracts that logic into a helper to let it more clear about what is happening.
2015-11-03[refactor] token verification moved to couch module + testsVictor Shyba
Added tests for this token verification as it wasn't covered. Then moved it to the new couch module that implements a couch storage. The ServerState was chosen to hold the verify_token method. CouchServerState holds the current implementation, which is called on authentication middleware as the new test shows.
2015-11-03[refactor] split out backend from couch databaseVictor Shyba
First step of splitting classes across files on common. backend.py holds SoledadBackend (generic backend logic) couch/ is now a directory with old code inside __init__.py and CouchServerState on state.py Also removed mock IndexedSoledadBackend, since Soledad does not support indexing due to encryption on server side. Also fixed DesignDocUnknownError to show up what is the message of the original exception. It was being lost.
2015-11-03[refactor] separate SoledadBackend from CouchDatabaseVictor Shyba
CouchDatabase was renamed to SoledadBackend and a new class CouchDatabase was created to hold all couchdb code. This should make SoledadBackend less tied to database implementation. A few more separations are needed to split into modules.
2015-10-29[feat] remove SSL tsafe monkeypatchVictor Shyba
This was added for a Twisted 12 bug that should be gone by now.
2015-10-28[style] fix pep8 warnigsFolker Bernitt
2015-10-28[feat] read security doc from configurationVictor Shyba
LEAP Platform needs to granularly allow access on user database for other services, like mx. This is now possible by editing soledad-server.conf file. A new section 'database-security' was added and it is parsed during 'create-user-db' to be set on security design document, present on every per-user database.
2015-10-19[bug] remove instance cachingVictor Shyba
couchdb library relies on garbage collector to close remaining connections. Somehow, caching the instance is avoiding gc to call __del__ on underlying couchdb sessions.
2015-10-14[bug] reduce overall sync session caching to 120sVictor Shyba
It was 3600s, but closing connections seems to yet depend on garbage collection and now causes server to leak file handlers. 120s should be enough to a sync session finish. Also, lowering this value will only make very long syncs use more of couch every 2 minutes, while raising this value will keep memory busy for useless time.
2015-10-05[feat] Move config dir to /etc/soledadvarac
- Resolves: #7509
2015-10-01[feat] read netrc path from configuration fileVictor Shyba
netrc file was hardcoded inside create-user-db. Now it reads the path from /etc/leap/soledad-server.conf as done on server process. The new configuration property is called 'admin_netrc'.
2015-09-28[feat] conf for enabling db creation via custom shVictor Shyba
We can now use a custom script to create databases by setting a parameter 'create_cmd' on soledad configuration. This will set CouchServerState to use it on ensure_database.
2015-09-24[style] pep8Victor Shyba
2015-09-24[refactor] init_caching instead of setting attrVictor Shyba
As meskio found commented, setting this attribute directly is ugly, CouchDatabase now has a init_caching method for setting up cache instance.
2015-09-24[feat] adds cache expirationVictor Shyba
Now each backend object will be retrieved from cache for sync.py and values will live for 3600 by default. That is changed via parameter if needed.
2015-09-24[feat] full caching for each sync_id sessionVictor Shyba
The CouchDB backend implementation was accessing CouchDB too many times for the same values. Those values are known inside the same sync_id, which is the id of current sync session. This commit adds caching for all redundant calls to Couch inside the same sync_id for each replica. Refactoring is still needed, but for now couch.py works normally as if caching is not present, while sync.py injects the cache as a attribute to enable it. This needs a simpler implementation.
2015-09-24[feat] adds caching for other gen and trans idVictor Shyba
There are two functions in couch.py used to save and retrieve the last know gen and trans id for the syncing replica. The get function is called very often, but is only set on one point. Added a simple caching to avoid queying couch for a value that we already have. If cache is empty, it just query as usual and fills it.
2015-09-24[feat] first draft of sync_state in memoryVictor Shyba
This commit changes sync_state to be in memory, with all tests passing. The memory variable for now is a dict with each key composed by source_replica_uid and sync_id, replicating CouchDB implementation. Next steps includes migrating this to Beaker and refactor/clean up code. Changed the module's INFO dict to use Beaker's caching and adapted methods to get and save from it. Still needs refactoring, all tests passes. Beaker is now using memory as default; It is configurable, but we aren't opening the possibility of config now for security. We need to check what can be misconfigured first. We are not sure if beaker will be the definitive solution for server side caching. This change isolates it with more granularity. In order to replace it, just change get_cache_for to return the proper caching object using another implementation. This caching object is supposed to behave as a dict.
2015-08-26[bug] process put after last BadRequest checkVictor Shyba
If we check for a BadRequest after calling meth_put we will end up on a scenario where the server replies with an error, but everything got processed.
2015-08-24[refactor] remove simplejson dep on serverdrebs
2015-08-17[bug] add missing parameter to unauthorized errordrebs