| Age | Commit message (Collapse) | Author |
|---|
|
this is needed for some mail tests.
|
|
|
|
|
|
This was discovered during load tests: Trying to process more than 999
docs triggers an error on SQLite due a select query not supporting 999
values to query.
|
|
test_processing_order aims to check that unordered docs wont be
processed, but if we let the pool start and advance Twisted LoopingCall
clock right before calling the processing method manually, the process
method will run concurrently and cause a race condition issue.
|
|
modifying original PR [0] by cristoph to account for the recent
vendoring of l2db code, which means we no longer depend on u1db/dirspec.
I expect the whole mess about the venv setup to be further simplified
pretty soon, since we are going to merge most of the leap.* packages
into a couple of repos.
[0] https://github.com/leapcode/soledad/pull/327
|
|
|
|
|
|
|
|
SQLCipher database access errors can raise Soledad exceptions. Database access
and multithreading resources are allocated in different places, so we have to
be careful to close all multithreading mechanismis in case of database access
errors. If we don't, zombie threads may haunt the reactor.
This commit adds SQLCipher exception trapping and Soledad exception raising
for database access errors, while properly shutting down multithreading
resources.
|
|
|
|
|
|
|
|
From this moment on, we embed a fork of u1db called l2db.
|
|
|
|
It was breaking E126 and E202 before
|
|
|
|
Do not initialize the openssl context on each call to decrypt.
I'm not 100% sure of the causal chain, but it seems that the
initialization of the osrandom engine that openssl backend does might be
breaking havoc when sqlcipher is calling rand_bytes concurrently.
further testing is needed to confirm this is the ultimate cause, but in
my tests this change avoids the occurrence of the dreaded Floating Point
Exception in soledad/sqlcipher.
- Resolves: #8180
|
|
Old versions of pip do not accept the --trusted-host option and will complain
when trying to upgrade pip from wheel. To fix that we upgrade pip from usual
location instead of doing it from wheel.
|
|
|
|
|
|
|
|
For the case where the user already has data synced, this commit will
migrate the docs_received table to have the column sync_id.
That is required by the refactoring in the previous commits.
|
|
Docs created from one failed sync would be there for the next one,
possibly causing a lot of hard to find errors. This commit adds a
sync_id field to track each sync documents isolated and cleans up the
pool on start instead of constructor.
|
|
This commit adds tests for doc ordering and encdecpool control
(start/stop). Also optimizes by deleting in batch and checking for a
sequence in memory before asking the local staging for documents.
|
|
This commit removes the multiprocessing pool and gives a step closer to
make encdecpool simpler. Download speed is now at a constant rate, CPU
usage lower and reactor responding fast when running with a HTTP server
like Pixelated.
|
|
|
|
Theoretically (until now), Soledad inherits from U1DB the behaviour of only
accepting valid JSON for documents contents. JSON documents only allow for
unicode strings. Despite that, until now we had implemented lossy convertion
to unicode to avoid encoding errors when dumping/loading JSON content. This
allowed for API users to pass non-unicode to Soledad, but caused the
application to take more time because of conversion.
There were 2 problem with this: (1) conversion may take a long time and a lot
of memory when convertin large payloads; and (2) conversion was being made
before deferring to the adbapi, and this was blocking the reactor.
This commit completelly removes the conversion to unicode, thus leaving the
responsibility of unicode conversion to users of the Soledad API.
|
|
The constructor method of Soledad was receiving two arguments for user
id. One of them was optional with None as default. It could cause an
inconsistent state with uuid set but userid unset.
This change remove the optional user_id argument from initialization
method and return the uuid if anyone call Soledad.userid method.
|
|
|
|
|
|
|
|
|
|
|
|
Shared db locking was used to avoid the case in which two different devices
try to store/modify remotelly stored secrets at the same time. We want to
avoid remote locks because of the problems they create, and prefer to crash
locally.
For the record, we are currently using the user's password to encrypt the
secrets stored in the server, and while we continue to do this we will have to
re-encrypt the secrets and update the remote storage whenever the user changes
her password.
|
|
|
|
|
|
for some reason, available_backends does not work inside a frozen
PyInstaller binary.
- Resolves: #7952
|
|
|
|
cryptography comes from OpenSSL and Twisted dependencies, so it's
already installed.
This commit removes a compiled dependency, also possibly making it
easier to use on Windows.
|
|
- Move them to a thread so reactor can continue
processing e.g. http requests
|
|
|
|
|
|
this allows to switch the online/offline mode on a running soledad
instance.
|
|
|
|
for the moment, userid has to be passed to constructor.
eventually, we might drop support for passing uuid, since it will be
mapped in the service tree
|
|
- Resolves: #7656
- Releases: 0.8.0
|
|
All batching code has no effect by default with this commit. Since we
know that this is a dangerous new feature we will enable them only on
our test servers and check them manually before setting it as default
or adding more configuration features.
Use SyncTarget and server conf file to enable it for testing.
|
|
Created two methods on the backend to start and finish a batch. A dict of
callbacks is available to defer actions for the last document, allowing
temporary (changing often) metadata to be recorded only once.
Using those methods we will also be able to put all docs in one go on
the CouchDatabase implementation, but that is another step.
|
|
u1db provides batching by default. Current Soledad HTTPS Sync Target was
stuck at 1 doc per request. This commit adds batching capability,
limiting the size to a predefined value.
Default limit size: 500kB
|
