soledad.git - [soledad]

Age	Commit message (Collapse)	Author
2016-12-12	[style] improve comments for sync.py	Victor Shyba

2016-12-12	[refactor] simplify content as a new line	Victor Shyba
	Code was complex and raised a flag during review.
2016-12-12	[refactor] improve readability of stream producer	Victor Shyba

2016-12-12	[style] add license header	Victor Shyba

2016-12-12	[refactor] remove assert logic from fetch_protocol	Victor Shyba
	Asserts aren't a good solution for stream parsing, its cleaner to check and raise in place. Also, asserts can be ignored.
2016-12-12	[feature] Adds back support to deprecated crypto	Victor Shyba
	Will be removed when we have the proper tool to migrate data.
2016-12-12	[feature] Adds deprecated crypto module	Victor Shyba
	This is supposed to be used only for temporary backwards compatibility, while we develop a proper migration tool.
2016-12-12	[refactor] simplify server insert	Victor Shyba
	Moved out magic numbers into a constant and simplified logic during doc upload.
2016-12-12	[tests] remove test_sync_very_large_files	Victor Shyba
	We have benchmarks now to test sync limits and 100mb is too far from current needs.
2016-12-12	[tests] improve doc creation on benchmarks	Victor Shyba
	If we create all at once we cant test higher loads because it will try to hold all in memory at the same time. Also, this code is smaller and more readable.
2016-12-12	[refactor] stop using a dict for syncers	Victor Shyba
	A dict was used to store references for the synchronizers based on a URL. This commit removes it as it doesnt make sense with current code.
2016-12-12	[tests] fall back to TestCase classes	Victor Shyba
	This test was using pytest, but it was making other trial based tests fail. I couldn't figure out why, but falling back to TestCase solved it.
2016-12-12	[refactor] stop using leap.common.http	Victor Shyba
	We aren't using leap.common.http implementation and we need specific features from original Twisted Web Agent. This commit implements it on HTTP Targer.
2016-12-12	[bug] total number of documents	Victor Shyba
	Parsing from metadata we can store the total of docs and handle it for the doc parser in order to be able to keep consistent events info.
2016-12-12	[refactor] Issues from code review	Victor Shyba

2016-12-12	[refactor] more comments, less code	Victor Shyba
	Some code were duplicated, got removed. Additional comments added for documenting such a critical and complex part as a protocol.
2016-12-12	[refactor] DocStreamReceiver <- ReadBodyProtocol	Victor Shyba
	Both classes holds u1db error handling. Making DocStreamReceiver a subclass reduces the error handling to a single place thus removing duplicated code.
2016-12-12	[bug] defer insertion to threads during download	Victor Shyba
	Insertion is synchronous and blocks the reactor. That's a temporary solution as we used to have on decpool.
2016-12-12	[style] remove unused imports (pep8)	Victor Shyba

2016-12-12	[test] fix test and remove leftovers defer_encryption	Victor Shyba

2016-12-12	[bug] fix import	Kali Kaneko

2016-12-12	[bug] remove print debug statements	Kali Kaneko

2016-12-12	[style] pep8	Kali Kaneko

2016-12-12	[bug] fix bad merge in imports block	Kali Kaneko

2016-12-12	[refactor] remove lingering refs to syncdb	Kali Kaneko

2016-12-12	[refactor] remove legacy crypto implementation	Kali Kaneko

2016-12-12	[tests] adapt tests	Kali Kaneko

2016-12-12	[refactor] remove encdecpool, finally	Kali Kaneko

2016-12-12	[bug] fix wrong content parsing	Kali Kaneko

2016-12-12	[refactor] add SoledadCrypto interface	Kali Kaneko

2016-12-12	[refactor] adapt fetcher to decryptor	Kali Kaneko

2016-12-12	[feature] blob encryptor / decryptor	Kali Kaneko

2016-12-12	[refactor] remove encryption pool	Kali Kaneko

2016-12-12	[feature] streaming crypto implementation	Kali Kaneko

2016-12-12	[feature] batch based on payload size	Victor Shyba
	batch is slower than usual insert for a single doc, so, if a document exceeds the buffer, commit the batch (if any) and put the huge load by traditional insert. refactor coming.
2016-12-12	[feature] fix and enable batch	Victor Shyba
	Batching is now decided by server, this commits enables it.
2016-12-12	[feature] stream content in a separate line	Victor Shyba
	This allow different paths for raw data and metadata, avoiding unnecessary json parsing.
2016-12-12	[bug] handle 400	Victor Shyba
	There was an if without an else on error handler that avoided handling errors that falled back current logic. Added a generic one to the tail so we dont miss it.
2016-12-12	[feature] make the test accept large uploads	Victor Shyba
	We enabled chunking, which means that a use can upload his entire db on a single request. This commit makes server enable this and throttle download as Twisted cant control the payload producer code as its synchronous and blocking code.
2016-12-12	[feature] upload streaming	Victor Shyba
	1) enable HTTP 1.1 chunked upload on server 2) make the client sync.py generate a list of function calls instead of a list of full docs 3) disable encryption pool 4) make the doc encryption a list of function calls 5) create a twisted protocol for sending 6) make a producer that calls the doc generation as necessary
2016-12-12	[feature] streaming download protocol	Victor Shyba
	This commit finishes reversion into u1db original streaming protocol for downloads.
2016-12-12	[refactor] remove decpool	Victor Shyba
	It's not being used
2016-12-12	[feature] make reading attachments optional	Victor Shyba
	Will put a file object on doc json string if read_content is False, otherwise it will fetch and fill as usual. This is useful for improving server througput on sync download stream by receiving a bulk-get without attachments and consume the file-objects as they come.
2016-12-12	[bug] use an empty string to represent tumbstones	Victor Shyba
	If a doc doesnt have a content it means it was deleted. Sync stream was unable to represent this state.
2016-12-12	[test] commit isnt part of the backend api	Victor Shyba
	Check if the backend provides a commit method before calling or we will break the tests with InMemoryDatabase
2016-12-12	[feature] simple adaptation to let the client run	Victor Shyba
	Make the client parse a 2-line doc on sync download stream.
2016-12-12	[feature] server download stream from file object	Victor Shyba
	couchdb lib returns a file object representing the attachment. This commit dumps the read() call into the wsgi write() call. Doc representation uses 2 lines also, separating metadata from content.
2016-12-12	[bug] disable decpool	Victor Shyba
	Temporary fix for server streaming
2016-12-12	[feature] use transactions on sync	Victor Shyba
	We were using 1 transaction per doc, which is bad. Reference: http://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite Code now uses 1 transaction for the whole sync.
2016-12-12	[feature] get attachments as generator runs	Victor Shyba
	Instead of getting the attachments as the generator runs, get_docs will now get as needed. Also, deepcopy solves a memory issue where we were feeding the couchdb lib view with blobs while modifying it unintentionally.