<feed xmlns='http://www.w3.org/2005/Atom'>
<title>soledad.git/src/leap, branch 8978</title>
<subtitle>[soledad]
</subtitle>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/'/>
<entry>
<title>[bug] limit number of concurrent requests to local db</title>
<updated>2017-10-12T16:39:00+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@riseup.net</email>
</author>
<published>2017-10-12T14:26:30+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=fdfb55ad4d0110356393bd018bd9ed1c66bbbbcf'/>
<id>fdfb55ad4d0110356393bd018bd9ed1c66bbbbcf</id>
<content type='text'>
When running stress tests on blobs storage, we get weird errors when
hundreds of requests are made concurrently to the sqlite backend. This
commit adds a limit so only 10 requests will be delivered to the backend
at a time.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
When running stress tests on blobs storage, we get weird errors when
hundreds of requests are made concurrently to the sqlite backend. This
commit adds a limit so only 10 requests will be delivered to the backend
at a time.
</pre>
</div>
</content>
</entry>
<entry>
<title>[feature] make concurrent blob writes configurable</title>
<updated>2017-10-12T15:06:33+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@riseup.net</email>
</author>
<published>2017-10-11T14:20:22+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=07f94a9a6f281069a0441cafce3f8a92e6d03e8b'/>
<id>07f94a9a6f281069a0441cafce3f8a92e6d03e8b</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[bug] limit concurrent blob writes in server</title>
<updated>2017-10-11T14:11:14+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@riseup.net</email>
</author>
<published>2017-10-11T14:06:34+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=1cde86239f68ca8fa896ecf4ffc6b891eeb837e3'/>
<id>1cde86239f68ca8fa896ecf4ffc6b891eeb837e3</id>
<content type='text'>
If there's no limit to the number of concurrent blob writes in the
server, the maximum limit of open files will eventually be reached, and
the processing of requests will start crashing. This commit adds
a semaphore to limit the number of concurrent writes in the server.

Related: #8973
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
If there's no limit to the number of concurrent blob writes in the
server, the maximum limit of open files will eventually be reached, and
the processing of requests will start crashing. This commit adds
a semaphore to limit the number of concurrent writes in the server.

Related: #8973
</pre>
</div>
</content>
</entry>
<entry>
<title>[feature] log OS errors when writing blobs</title>
<updated>2017-10-10T10:53:40+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@riseup.net</email>
</author>
<published>2017-10-10T10:53:29+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=bf50414cc84f40449c3d469c893554de76b52c87'/>
<id>bf50414cc84f40449c3d469c893554de76b52c87</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[bug] handle put errors in the incoming blobs api</title>
<updated>2017-10-10T10:52:28+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@riseup.net</email>
</author>
<published>2017-10-10T10:43:19+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=850a0e32f996cdcc6724e3900a40325aae719828'/>
<id>850a0e32f996cdcc6724e3900a40325aae719828</id>
<content type='text'>
An errback was missing in the PUT renderer method of the incoming API.
Because of that, requests to that endpoint were not being correctly
finished in case of errors when writing blobs. That was causing delivery
requests to hang until timeout.

Closes: #8977
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
An errback was missing in the PUT renderer method of the incoming API.
Because of that, requests to that endpoint were not being correctly
finished in case of errors when writing blobs. That was causing delivery
requests to hang until timeout.

Closes: #8977
</pre>
</div>
</content>
</entry>
<entry>
<title>[bug] fix retries for blobs download</title>
<updated>2017-10-05T19:25:51+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@riseup.net</email>
</author>
<published>2017-10-05T18:24:46+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=2772d76d0d66d9e094a435e110fd9f945d9e105c'/>
<id>2772d76d0d66d9e094a435e110fd9f945d9e105c</id>
<content type='text'>
- add a MaximumRetriesError exception to encapsulate other exceptions.
- record the pending status before trying to download
- modify update_sync_status to insert or update
- modify retry tests to check number of retries
- add a test for download retry limit
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
- add a MaximumRetriesError exception to encapsulate other exceptions.
- record the pending status before trying to download
- modify update_sync_status to insert or update
- modify retry tests to check number of retries
- add a test for download retry limit
</pre>
</div>
</content>
</entry>
<entry>
<title>[bug] retry blob download for all retriable errors</title>
<updated>2017-10-05T12:43:41+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@riseup.net</email>
</author>
<published>2017-10-05T12:03:39+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=e98dd6b7b7366c9ae6ca18c6e94866fbf8641afe'/>
<id>e98dd6b7b7366c9ae6ca18c6e94866fbf8641afe</id>
<content type='text'>
Because the exception catching was being made inside
_download_and_decrypt() and only accounted for InvalidBlob exceptions,
not all retriable errors would lead to an actual retry.

This commit moves the exception catching to one level up and catches any
kind of exception, as is done in the upload part. This allows for
retrying on all retriable errors.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Because the exception catching was being made inside
_download_and_decrypt() and only accounted for InvalidBlob exceptions,
not all retriable errors would lead to an actual retry.

This commit moves the exception catching to one level up and catches any
kind of exception, as is done in the upload part. This allows for
retrying on all retriable errors.
</pre>
</div>
</content>
</entry>
<entry>
<title>[bug] improve error message on blob download error</title>
<updated>2017-10-05T12:43:40+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@riseup.net</email>
</author>
<published>2017-10-04T19:13:19+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=61a7ab212a1f7ed02b5d1020ab01ae825ba3696f'/>
<id>61a7ab212a1f7ed02b5d1020ab01ae825ba3696f</id>
<content type='text'>
The previous error message had some problems:

- the connection should not be a problem, as this is going over TCP. If
  the HTTP request was succesful, there's no reason to think its
  contents could have been corrupted by a connection problem.

- I am not sure what's the best communication strategy here, but the
  real problem is either a bug or actual tampering, so i make this
  explicit.

- A problem like this should be reported always, not only when the
  problem persists.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The previous error message had some problems:

- the connection should not be a problem, as this is going over TCP. If
  the HTTP request was succesful, there's no reason to think its
  contents could have been corrupted by a connection problem.

- I am not sure what's the best communication strategy here, but the
  real problem is either a bug or actual tampering, so i make this
  explicit.

- A problem like this should be reported always, not only when the
  problem persists.
</pre>
</div>
</content>
</entry>
<entry>
<title>[bug] log the exception on blob download error</title>
<updated>2017-10-05T12:43:40+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@riseup.net</email>
</author>
<published>2017-10-04T19:07:06+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=c6d41cf4fcac293a0e7ff09593d2a3c53c052174'/>
<id>c6d41cf4fcac293a0e7ff09593d2a3c53c052174</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[bug] don't use hardcoded number of retries when downloading blobs</title>
<updated>2017-10-05T12:43:40+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@riseup.net</email>
</author>
<published>2017-10-04T19:00:06+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=3b3652a915032609dcf458676b4bb85762e6b687'/>
<id>3b3652a915032609dcf458676b4bb85762e6b687</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>
