| Age | Commit message (Collapse) | Author |
|---|
|
Also refactored tests and code to stop relying on old parameters which
included docs instead of get_doc calls.
|
|
Deferred encryption option is gone.
|
|
Will be removed when we have the proper tool to migrate data.
|
|
We have benchmarks now to test sync limits and 100mb is too far from
current needs.
|
|
If we create all at once we cant test higher loads because it will try
to hold all in memory at the same time. Also, this code is smaller and
more readable.
|
|
This test was using pytest, but it was making other trial based tests
fail. I couldn't figure out why, but falling back to TestCase solved it.
|
|
We aren't using leap.common.http implementation and we need specific
features from original Twisted Web Agent. This commit implements it on
HTTP Targer.
|
|
Parsing from metadata we can store the total of docs and handle it for
the doc parser in order to be able to keep consistent events info.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1) enable HTTP 1.1 chunked upload on server
2) make the client sync.py generate a list of function calls instead of
a list of full docs
3) disable encryption pool
4) make the doc encryption a list of function calls
5) create a twisted protocol for sending
6) make a producer that calls the doc generation as necessary
|
|
This commit finishes reversion into u1db original streaming protocol for
downloads.
|
|
It's not being used
|
|
Temporary fix for server streaming
|
|
Conflicts:
server/src/leap/soledad/server/__init__.py
testing/tests/conftest.py
|
|
CouchServerState is spread across test codebase and this option is
intended to be used only on server startup. This commit makes it default
to False and explicitly set it to True on where it's necessary.
|
|
|
|
Tests that were imported from u1db or created on top of that structure
were leaving temporary directories behind. This could cause problems in
test servers, either by filling the partition or by extrapolating the
maximum amount of files in a directory.
This commit replaces all usages of temporary directories in the old test
structure by pytest tmpdir fixture, which properly cares for removing
temporary directories.
|
|
We were using 'x'*size as payload, but on real usage the payload will be
random. This commit randomizes the payload using a predefined seed, so
the random payload will be the same across benchmarks.
Using random payloads also improves accuracy of compression or encoding
impacts and we will be evaluating those changes for resouce usage
issues.
Also note that base64 is used on payload. That was needed for utf8
safety, but overhead was removed to leave payloads as defined by
benchmarks.
Base64 was chosen also due its popular usage on MIME encoding, which is
used on mail attachments (our current scenario).
|
|
TestSyncEncrypterPool.test_encrypt_doc_and_get_it_back was trying to do
an operation and asserting the number of attempts. This test is about
putting a doc on encrypter pool and getting it encrypted. If we dont
wait for the encryption operation to succeed, then complex
trial-and-error happens, but if we just ask twisted to wait for one
operation before going to the other, this is not needed.
-- Resolves: #8398
|
|
This isnt a test, but a benchmark. Initialization sounds more like an
operation while instance is just something.
|
|
We are using lower values on test_encdecpool due high memory usage,
described in #7370. Added a comment to explain it.
|
|
defer parameter wasnt clear
|
|
Otherwise it will add unrelated overhead to results.
|
|
|
|
They arent used so far and using empty dicts to make them work is ugly.
Removing it leaves the return function on setup code clean and readable.
|
|
1000 docs at 100k~500k are exploding memory (4Gb+4Gb swap).
Changed for 100 docs in order to be able to get measures on higher
loads. Now its 10k, 100k and 500k
|
|
Hypothesis: raw vs doc
Added the same sizes set (10k, 100k, 500k, 1M, 10M, 50M) as the document
crypto test, so we can compare how close to raw the higher level
operation is.
|
|
10k, 100k, 500k, 1m, 10m and 50m for encryption and decryption of a
whole document.
|
|
Most of them are commented as memory usage is going out of control for
now.
|
|
It has a heavy scrypt hashing processing with room for improvement.
|
|
Syncing without any changes was reported as slow. This benchmark will
help measure it.
|
|
Use a new one to avoid reusing the same database.
|
|
function is the default scope, so there is no need to pass this
parameter. Previously, one of the scopes was 'module', but it is a
nested function that fires on demand, so it should clean up itself from
test to test in order to avoid conflict while putting.
|
|
|
|
Creating 20/500k, 100/100k and 1000/10k.
|
|
If we have many scenarios (like 20/500k, 100/100k, 1000,10k) then making
a nested function to generate tests based on scenario parameters
simplifies the code a lot.
|
|
Adapted pytest-benchmark to Twisted as it's synchronous and added
fixtures for benchmarking.
|
|
|
|
tox was configured to change to the testing/tests directory before
executing pytest, by using tox's "changedir" configuration option. The
reason why this was the case is that we wanted to discover tests inside
the testing/tests directory only.
The problem with that approach is that if we wanted to point to a
specific test file, for example "tests/perf/test_sync.py", we would have
to omit the "tests" part and write "tox perf/test_sync.py" because the
argument would be understood as relative to the changed dir. That is not
practical as doesn't allow to use the shell autocomplete, and is also
not the only way to achieve what we want.
Actually, pytest has a configuration option called "testpaths" where you
can indicate where it should discover tests. This commit changes one
approach by the other and allows to user shell autocomplete for easyness
of testing during development.
|
|
test_processing_order aims to check that unordered docs wont be
processed, but if we let the pool start and advance Twisted LoopingCall
clock right before calling the processing method manually, the process
method will run concurrently and cause a race condition issue.
|
|
`pytest_addoption` was declared twice making the second declaration
replace the first, thus removing couch url parameter.
|
|
|
|
Currently the perf tests use pytest-twisted plugin, and this has some
implications in the old tests adapted from u1db that now use trial
classes. Because of that, we exclude perf tests from usual tox calls,
but you can still run them by explicitelly calling `tox perf`.
|
|
|
