| Age | Commit message (Collapse) | Author |
|---|
|
Moved out magic numbers into a constant and simplified logic during doc
upload.
|
|
We have benchmarks now to test sync limits and 100mb is too far from
current needs.
|
|
If we create all at once we cant test higher loads because it will try
to hold all in memory at the same time. Also, this code is smaller and
more readable.
|
|
A dict was used to store references for the synchronizers based on a
URL. This commit removes it as it doesnt make sense with current code.
|
|
This test was using pytest, but it was making other trial based tests
fail. I couldn't figure out why, but falling back to TestCase solved it.
|
|
We aren't using leap.common.http implementation and we need specific
features from original Twisted Web Agent. This commit implements it on
HTTP Targer.
|
|
Parsing from metadata we can store the total of docs and handle it for
the doc parser in order to be able to keep consistent events info.
|
|
|
|
Some code were duplicated, got removed. Additional comments added for
documenting such a critical and complex part as a protocol.
|
|
Both classes holds u1db error handling. Making DocStreamReceiver a
subclass reduces the error handling to a single place thus removing
duplicated code.
|
|
Insertion is synchronous and blocks the reactor. That's a temporary
solution as we used to have on decpool.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
batch is slower than usual insert for a single doc, so, if a document
exceeds the buffer, commit the batch (if any) and put the huge load by
traditional insert.
refactor coming.
|
|
Batching is now decided by server, this commits enables it.
|
|
This allow different paths for raw data and metadata, avoiding
unnecessary json parsing.
|
|
There was an if without an else on error handler that avoided handling
errors that falled back current logic. Added a generic one to the tail
so we dont miss it.
|
|
We enabled chunking, which means that a use can upload his entire db on
a single request. This commit makes server enable this and throttle
download as Twisted cant control the payload producer code as its
synchronous and blocking code.
|
|
1) enable HTTP 1.1 chunked upload on server
2) make the client sync.py generate a list of function calls instead of
a list of full docs
3) disable encryption pool
4) make the doc encryption a list of function calls
5) create a twisted protocol for sending
6) make a producer that calls the doc generation as necessary
|
|
This commit finishes reversion into u1db original streaming protocol for
downloads.
|
|
It's not being used
|
|
Will put a file object on doc json string if read_content is False,
otherwise it will fetch and fill as usual. This is useful for improving
server througput on sync download stream by receiving a bulk-get without
attachments and consume the file-objects as they come.
|
|
If a doc doesnt have a content it means it was deleted. Sync stream was
unable to represent this state.
|
|
Check if the backend provides a commit method before calling or we will
break the tests with InMemoryDatabase
|
|
Make the client parse a 2-line doc on sync download stream.
|
|
couchdb lib returns a file object representing the attachment. This
commit dumps the read() call into the wsgi write() call. Doc
representation uses 2 lines also, separating metadata from content.
|
|
Temporary fix for server streaming
|
|
We were using 1 transaction per doc, which is bad.
Reference:
http://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite
Code now uses 1 transaction for the whole sync.
|
|
Instead of getting the attachments as the generator runs, get_docs will
now get as needed. Also, deepcopy solves a memory issue where we were
feeding the couchdb lib view with blobs while modifying it
unintentionally.
|
|
Instead of concurrent download, we are going to download a stream. This
commit modifies server to support it.
|
|
Tag version 0.9.1
|
|
|
|
|
|
|
|
The couch state checks if all user databases have the correct schema
version on initialization, and may log errors and raise exceptions if
that is not the case. Because we are currently using `twistd web
--wsgi`, if those errors are logged too early the reactor may have not
been started and the twistd logging facilities may not have been
initialized.
This commit delays the state initialization until the reactor has been
started, to make sure any errors raised and logged in that stage will
actually reach the logfile.
|
|
|
