summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2016-12-12[feature] stream content in a separate lineVictor Shyba
This allow different paths for raw data and metadata, avoiding unnecessary json parsing.
2016-12-12[bug] handle 400Victor Shyba
There was an if without an else on error handler that avoided handling errors that falled back current logic. Added a generic one to the tail so we dont miss it.
2016-12-12[feature] make the test accept large uploadsVictor Shyba
We enabled chunking, which means that a use can upload his entire db on a single request. This commit makes server enable this and throttle download as Twisted cant control the payload producer code as its synchronous and blocking code.
2016-12-12[feature] upload streamingVictor Shyba
1) enable HTTP 1.1 chunked upload on server 2) make the client sync.py generate a list of function calls instead of a list of full docs 3) disable encryption pool 4) make the doc encryption a list of function calls 5) create a twisted protocol for sending 6) make a producer that calls the doc generation as necessary
2016-12-12[feature] streaming download protocolVictor Shyba
This commit finishes reversion into u1db original streaming protocol for downloads.
2016-12-12[refactor] remove decpoolVictor Shyba
It's not being used
2016-12-12[feature] make reading attachments optionalVictor Shyba
Will put a file object on doc json string if read_content is False, otherwise it will fetch and fill as usual. This is useful for improving server througput on sync download stream by receiving a bulk-get without attachments and consume the file-objects as they come.
2016-12-12[bug] use an empty string to represent tumbstonesVictor Shyba
If a doc doesnt have a content it means it was deleted. Sync stream was unable to represent this state.
2016-12-12[test] commit isnt part of the backend apiVictor Shyba
Check if the backend provides a commit method before calling or we will break the tests with InMemoryDatabase
2016-12-12[feature] simple adaptation to let the client runVictor Shyba
Make the client parse a 2-line doc on sync download stream.
2016-12-12[feature] server download stream from file objectVictor Shyba
couchdb lib returns a file object representing the attachment. This commit dumps the read() call into the wsgi write() call. Doc representation uses 2 lines also, separating metadata from content.
2016-12-12[bug] disable decpoolVictor Shyba
Temporary fix for server streaming
2016-12-12[feature] use transactions on syncVictor Shyba
We were using 1 transaction per doc, which is bad. Reference: http://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite Code now uses 1 transaction for the whole sync.
2016-12-12[feature] get attachments as generator runsVictor Shyba
Instead of getting the attachments as the generator runs, get_docs will now get as needed. Also, deepcopy solves a memory issue where we were feeding the couchdb lib view with blobs while modifying it unintentionally.
2016-12-12[feature] revert sync download into straming (server)Victor Shyba
Instead of concurrent download, we are going to download a stream. This commit modifies server to support it.
2016-11-27Merge tag '0.9.1' into developdrebs
Tag version 0.9.1
2016-11-27[pkg] update changelog0.9.1drebs
2016-11-27[bug] fix import on create-user-dbVictor Shyba
2016-11-27[bug] patch twisted logger so it works with twistd --syslogdrebs
2016-11-27[bug] delay couch state initializationdrebs
The couch state checks if all user databases have the correct schema version on initialization, and may log errors and raise exceptions if that is not the case. Because we are currently using `twistd web --wsgi`, if those errors are logged too early the reactor may have not been started and the twistd logging facilities may not have been initialized. This commit delays the state initialization until the reactor has been started, to make sure any errors raised and logged in that stage will actually reach the logfile.
2016-11-22[feat] improve missing couch config doc error loggingdrebs
2016-11-22[refactor] separate server application into another filedrebs
Conflicts: server/src/leap/soledad/server/__init__.py testing/tests/conftest.py
2016-11-10Merge tag '0.9.0' into developdrebs
Tag version 0.9.0
2016-11-09[pkg] update leap requirements files0.9.0rc10.9.0drebs
2016-11-09[pkg] update changelog to 0.9.0drebs
2016-11-09[pkg] use correct folder name for migrate scriptdrebs
2016-10-21[tests] cache tox folderVictor Shyba
This should avoid tox virtualenv recreation.
2016-10-21[tests] fix server importVictor Shyba
When importing server, couch_state will load itself against couch_db url configured on server. This fails when running on Docker as couchdb is in another node.
2016-10-21[feature] adds libsqlite3-dev on docker imageVictor Shyba
This is necessary for keymanager and this image is shared, thus adding here with a comment explaining why. Also explained why using jessie-backports.
2016-10-21[docs] explain CouchServerState parametersVictor Shyba
create_cmd lacked an explanation and check_schema_versions lacked reasoning on why it defaults to False.
2016-10-21[tests] make check_schema_versions default to FalseVictor Shyba
CouchServerState is spread across test codebase and this option is intended to be used only on server startup. This commit makes it default to False and explicitly set it to True on where it's necessary.
2016-10-21[tests] specify basepython on root envVictor Shyba
code-check is running with py3 randomly on CI, this commit should pin it.
2016-10-21[feature] use latest imageVictor Shyba
Instead of hardcoding a version. This should give us the flexibility of changing images without changing code.
2016-10-21[bug] adds libsqlcipher to dockerVictor Shyba
Current docker image is broken due missing libsqlcipher. This commit adds it and jessie-backports due package needs. Resolves: #8508
2016-10-12[bug] remove finalClose from SQLCipherU1DBSyncTulio Casagrande
We discovered that class was registering a `finalClose` to be executed on reactor shutdown. On the multiuser scenario, a logout destroys Soledad and should properly terminate everything related to it. That SQLCipherU1DBSync instance was being held even after logout by the reactor so it could call that `finalClose` on shutdown. The `finalClose` only set running to False and set a `shutdownID` that was not used anywhere else, so we removed it and moved setting running to False to the `close` function method. That way we preserve the functionality but let the instance be properly garbage collected on logout.
2016-10-03[feature] check for user dbs couch schema versionsdrebs
2016-10-03[refactor] move configuration loading to its own moduledrebs
2016-09-30[bug] use % for formattingVictor Shyba
Otherwise it will put the exception as an additional parameter.
2016-09-30[test] add flake8 code check and generalize name of tox envdrebs
2016-09-30[test] use pytest tmpdir fixture in all testsdrebs
Tests that were imported from u1db or created on top of that structure were leaving temporary directories behind. This could cause problems in test servers, either by filling the partition or by extrapolating the maximum amount of files in a directory. This commit replaces all usages of temporary directories in the old test structure by pytest tmpdir fixture, which properly cares for removing temporary directories.
2016-09-22[style] standardize log messagesdrebs
2016-09-22[feat] centralize logging and use twisted.logger by defaultdrebs
2016-09-09[test] add tagged perf job for gitlab-cidrebs
In order to configure performance tests to run in a specific machine we need to add a tagged job to .gitlab-ci.yml file. That job will only execute the perf tests, and then we can have runners that will only run those jobs.
2016-09-06[test] add tox env for perf testsdrebs
2016-09-06[style] pep8Kali Kaneko
2016-09-02[tests] add pep8 in main tox run from within the gitlab scriptKali Kaneko
add coverage reports too. (hereby we swear not to write stupid tests just because it feels good to have an increased coverage metric). - Resolves: #8416
2016-09-02[bug] fail gracefully if dbsyncer has not been initializedKali Kaneko
this is needed for some mail tests.
2016-08-29[pkg] remove deprecated requirements-testing.pipdrebs
2016-08-29[pkg] remove leftover simplejson imports from l2dbdrebs
2016-08-29[test] randomize payloadVictor Shyba
We were using 'x'*size as payload, but on real usage the payload will be random. This commit randomizes the payload using a predefined seed, so the random payload will be the same across benchmarks. Using random payloads also improves accuracy of compression or encoding impacts and we will be evaluating those changes for resouce usage issues. Also note that base64 is used on payload. That was needed for utf8 safety, but overhead was removed to leave payloads as defined by benchmarks. Base64 was chosen also due its popular usage on MIME encoding, which is used on mail attachments (our current scenario).