| Age | Commit message (Collapse) | Author |
|---|
|
After adding the streaming decrypt, some classes were doing almost the
same thing. Unified them.
Also fixed some module level variables to upper case and some class name
to camel case.
|
|
Unfortunately, if a doc finishes decryption before the previous one we
will still have an issue while inserting. This commits solves it by
adding the parse and decrypt inside of the semaphore.
|
|
We are already doing this on encryption, now we can stream also from
decryption. This unblocks the reactor and will be valuable for blobs-io.
|
|
We now encode preamble and ciphertext+hmac in two distinct payloads
separated by a space. This allows metadata to be extracted and used
before decoding the whole document.
It also introduces a single packer for packing and unpacking of data
instead of reads and writes. Downside: doc_id and rev are limited to 255
chars now.
|
|
IV was being set during tests and this required some defensive coding to
avoid IV being set in production. This commits makes the test use the
generated IV and "hides" it using a read-only property to let it clear
this should never happen.
Also refactored out some parameters that are generated automatically to
reduce some lines of code and enhance readability.
|
|
|
|
Document sending happens after encryption, so the last sent document
needs to be signalled after request end.
|
|
encrypt returns a deferred and needs the adapted benchmark runner.
|
|
When we use marks the new pytests from benchmarks folder are collected
and ignored, but this causes trial to fail sometimes. Using --ignore
avoids it from being loaded while --benchmark-only will properly select
the benchmarks for tox, as intended.
|
|
test_deprecated_crypto was using pytest, which unfortunately doesnt work
when mixed with trial. Migrated back.
Also added norecursedirs option back, as it is necessary for parallel
testing mode.
|
|
|
|
|
|
|
|
|
|
Fixes setup.cfg, adding current exclude rules, simplified tox.ini to use
setup.cfg and fixed all.
|
|
Also refactored tests and code to stop relying on old parameters which
included docs instead of get_doc calls.
|
|
Deferred encryption option is gone.
|
|
Will be removed when we have the proper tool to migrate data.
|
|
We have benchmarks now to test sync limits and 100mb is too far from
current needs.
|
|
If we create all at once we cant test higher loads because it will try
to hold all in memory at the same time. Also, this code is smaller and
more readable.
|
|
This test was using pytest, but it was making other trial based tests
fail. I couldn't figure out why, but falling back to TestCase solved it.
|
|
We aren't using leap.common.http implementation and we need specific
features from original Twisted Web Agent. This commit implements it on
HTTP Targer.
|
|
Parsing from metadata we can store the total of docs and handle it for
the doc parser in order to be able to keep consistent events info.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1) enable HTTP 1.1 chunked upload on server
2) make the client sync.py generate a list of function calls instead of
a list of full docs
3) disable encryption pool
4) make the doc encryption a list of function calls
5) create a twisted protocol for sending
6) make a producer that calls the doc generation as necessary
|
|
This commit finishes reversion into u1db original streaming protocol for
downloads.
|
|
It's not being used
|
|
Temporary fix for server streaming
|
|
Conflicts:
server/src/leap/soledad/server/__init__.py
testing/tests/conftest.py
|
|
CouchServerState is spread across test codebase and this option is
intended to be used only on server startup. This commit makes it default
to False and explicitly set it to True on where it's necessary.
|
|
|
|
Tests that were imported from u1db or created on top of that structure
were leaving temporary directories behind. This could cause problems in
test servers, either by filling the partition or by extrapolating the
maximum amount of files in a directory.
This commit replaces all usages of temporary directories in the old test
structure by pytest tmpdir fixture, which properly cares for removing
temporary directories.
|
|
We were using 'x'*size as payload, but on real usage the payload will be
random. This commit randomizes the payload using a predefined seed, so
the random payload will be the same across benchmarks.
Using random payloads also improves accuracy of compression or encoding
impacts and we will be evaluating those changes for resouce usage
issues.
Also note that base64 is used on payload. That was needed for utf8
safety, but overhead was removed to leave payloads as defined by
benchmarks.
Base64 was chosen also due its popular usage on MIME encoding, which is
used on mail attachments (our current scenario).
|
|
TestSyncEncrypterPool.test_encrypt_doc_and_get_it_back was trying to do
an operation and asserting the number of attempts. This test is about
putting a doc on encrypter pool and getting it encrypted. If we dont
wait for the encryption operation to succeed, then complex
trial-and-error happens, but if we just ask twisted to wait for one
operation before going to the other, this is not needed.
-- Resolves: #8398
|
|
This isnt a test, but a benchmark. Initialization sounds more like an
operation while instance is just something.
|
|
We are using lower values on test_encdecpool due high memory usage,
described in #7370. Added a comment to explain it.
|
|
defer parameter wasnt clear
|
|
Otherwise it will add unrelated overhead to results.
|
|
|
|
They arent used so far and using empty dicts to make them work is ugly.
Removing it leaves the return function on setup code clean and readable.
|
|
1000 docs at 100k~500k are exploding memory (4Gb+4Gb swap).
Changed for 100 docs in order to be able to get measures on higher
loads. Now its 10k, 100k and 500k
|
|
Hypothesis: raw vs doc
Added the same sizes set (10k, 100k, 500k, 1M, 10M, 50M) as the document
crypto test, so we can compare how close to raw the higher level
operation is.
|
|
10k, 100k, 500k, 1m, 10m and 50m for encryption and decryption of a
whole document.
|
|
Most of them are commented as memory usage is going out of control for
now.
|
