<feed xmlns='http://www.w3.org/2005/Atom'>
<title>soledad.git/common/src/leap, branch 0.9.0rc1</title>
<subtitle>[soledad]
</subtitle>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/'/>
<entry>
<title>[docs] explain CouchServerState parameters</title>
<updated>2016-10-21T17:09:54+00:00</updated>
<author>
<name>Victor Shyba</name>
<email>victor.shyba@gmail.com</email>
</author>
<published>2016-10-06T21:55:20+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=ee4dc679c8ae1a87a9b5ef3b2757a3113218e4c6'/>
<id>ee4dc679c8ae1a87a9b5ef3b2757a3113218e4c6</id>
<content type='text'>
create_cmd lacked an explanation and check_schema_versions lacked
reasoning on why it defaults to False.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
create_cmd lacked an explanation and check_schema_versions lacked
reasoning on why it defaults to False.
</pre>
</div>
</content>
</entry>
<entry>
<title>[tests] make check_schema_versions default to False</title>
<updated>2016-10-21T17:09:54+00:00</updated>
<author>
<name>Victor Shyba</name>
<email>victor.shyba@gmail.com</email>
</author>
<published>2016-10-05T22:52:58+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=09a62dd1d6b076fcc7ac001d0b998ebb119feaad'/>
<id>09a62dd1d6b076fcc7ac001d0b998ebb119feaad</id>
<content type='text'>
CouchServerState is spread across test codebase and this option is
intended to be used only on server startup. This commit makes it default
to False and explicitly set it to True on where it's necessary.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
CouchServerState is spread across test codebase and this option is
intended to be used only on server startup. This commit makes it default
to False and explicitly set it to True on where it's necessary.
</pre>
</div>
</content>
</entry>
<entry>
<title>[feature] check for user dbs couch schema versions</title>
<updated>2016-10-03T22:27:42+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@leap.se</email>
</author>
<published>2016-10-03T22:27:42+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=4e06eb370b99f2d343e96f774a3ad9b8b77c9548'/>
<id>4e06eb370b99f2d343e96f774a3ad9b8b77c9548</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[test] add flake8 code check and generalize name of tox env</title>
<updated>2016-09-30T12:08:47+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@leap.se</email>
</author>
<published>2016-09-24T18:57:54+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=887fd917a19654aa6a7c6c54be3f22c3b6c79b92'/>
<id>887fd917a19654aa6a7c6c54be3f22c3b6c79b92</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[feat] centralize logging and use twisted.logger by default</title>
<updated>2016-09-22T20:57:40+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@leap.se</email>
</author>
<published>2016-09-22T17:44:46+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=b7340a962bfeae9af28c4b514d0eb077f41dd832'/>
<id>b7340a962bfeae9af28c4b514d0eb077f41dd832</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[pkg] remove leftover simplejson imports from l2db</title>
<updated>2016-08-29T15:56:00+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@leap.se</email>
</author>
<published>2016-08-25T23:25:30+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=0690eb5338953dadc68c53b8d4010ca40adc3b0b'/>
<id>0690eb5338953dadc68c53b8d4010ca40adc3b0b</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>[bug] remove misleading ensure_ddoc</title>
<updated>2016-08-18T02:05:21+00:00</updated>
<author>
<name>Victor Shyba</name>
<email>victor.shyba@gmail.com</email>
</author>
<published>2016-08-18T02:00:34+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=dc0bae8b6025a060297b55520674cd7238f0186b'/>
<id>dc0bae8b6025a060297b55520674cd7238f0186b</id>
<content type='text'>
ensure_ddoc doesnt make sense anymore as we dont have any ddoc other
than _security, which has its own method for setting. 'ensure_security'
is explicit and is set internally when user is creating a database,
otherwise it will be False as it's only used during creation. This isn't
exposed externally (of couch module) to avoid confusion.
This confusion was making create-user-db fail to create a security ddoc
as it wasn't passing ensure_ddocs=True.

-- Resolves: #8388
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
ensure_ddoc doesnt make sense anymore as we dont have any ddoc other
than _security, which has its own method for setting. 'ensure_security'
is explicit and is set internally when user is creating a database,
otherwise it will be False as it's only used during creation. This isn't
exposed externally (of couch module) to avoid confusion.
This confusion was making create-user-db fail to create a security ddoc
as it wasn't passing ensure_ddocs=True.

-- Resolves: #8388
</pre>
</div>
</content>
</entry>
<entry>
<title>[bug] create gen document after saving the actual document in couch</title>
<updated>2016-08-05T15:34:13+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@leap.se</email>
</author>
<published>2016-08-05T15:34:13+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=18e4ffe5562be61efe9cd206494b9853e063a897'/>
<id>18e4ffe5562be61efe9cd206494b9853e063a897</id>
<content type='text'>
If we create the gen document before saving the actual document in
couch, we may run into problems if more than one client is syncing and
trying to save documents with the same id at the same time.

By moving the gen document creation to after the actual document save in
couch, we rely on couch/u1db resolution of conflicts before actually
allocating a new generation, and the problem above doesn't occur.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
If we create the gen document before saving the actual document in
couch, we may run into problems if more than one client is syncing and
trying to save documents with the same id at the same time.

By moving the gen document creation to after the actual document save in
couch, we rely on couch/u1db resolution of conflicts before actually
allocating a new generation, and the problem above doesn't occur.
</pre>
</div>
</content>
</entry>
<entry>
<title>[bug] retry allocation of gen instead of using a lock</title>
<updated>2016-08-02T00:09:05+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@leap.se</email>
</author>
<published>2016-07-31T21:25:48+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=49cd07b909f2185b116bda5b30cfcfe0095291e0'/>
<id>49cd07b909f2185b116bda5b30cfcfe0095291e0</id>
<content type='text'>
The use of a lock to allocate the next generation of a change in couch
backend suffers from at least 2 problems:

1. all modification to the couch database would have to be made through
   a soledad server entrypoint, otherwise the lock would have no effect.

2. introducing a lock makes code uglier, harder to debug, and prone to
   undesired blocks.

The solution implemented by this commit is not so elegant, but works for
what we need right now. Now, concurrent threads updating the couch
database will race for the allocation of a new generation, and retry
when they fail to do so.

There's no high risk of getting blocked for too much time in the while
loop because (1) there's always one thread that wins (what makes the
expected number of retries to be N/2 if N is the number of concurrent
threads), and (2) the number of concurrent attempts to update the user
database is limited by the number of devices syncing at the same time.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The use of a lock to allocate the next generation of a change in couch
backend suffers from at least 2 problems:

1. all modification to the couch database would have to be made through
   a soledad server entrypoint, otherwise the lock would have no effect.

2. introducing a lock makes code uglier, harder to debug, and prone to
   undesired blocks.

The solution implemented by this commit is not so elegant, but works for
what we need right now. Now, concurrent threads updating the couch
database will race for the allocation of a new generation, and retry
when they fail to do so.

There's no high risk of getting blocked for too much time in the while
loop because (1) there's always one thread that wins (what makes the
expected number of retries to be N/2 if N is the number of concurrent
threads), and (2) the number of concurrent attempts to update the user
database is limited by the number of devices syncing at the same time.
</pre>
</div>
</content>
</entry>
<entry>
<title>[feat] use couch _all_docs for get_docs() and get_all_docs()</title>
<updated>2016-08-02T00:09:05+00:00</updated>
<author>
<name>drebs</name>
<email>drebs@leap.se</email>
</author>
<published>2016-07-31T13:37:35+00:00</published>
<link rel='alternate' type='text/html' href='https://leap.se/git/soledad.git/commit/?id=3b237bb46743a93feed4bb6f3c839d72fc28df48'/>
<id>3b237bb46743a93feed4bb6f3c839d72fc28df48</id>
<content type='text'>
The previous solution would make use of concurrent get's to couch
backend in a pool of threads to implement the get_docs() and
get_all_docs() CouchDatabase backend methods.

This commit replaces those by a simpler implementation use the
`_all_docs` couchdb view api. It passes all needed IDs to the view and
r etrieves all documents with content in the same request.

A comparison between both implementations shows an improvement of at
least 15 times for large number of documents. The table below shows the
time for different implementations of get_all_docs() for different
number of documents and threads versus _all_docs implementation:

  +-------+-----------------+------------------+-------------+
  |       |     threads     |    _all_docs     | improvement |
  +-------+-----------------+------------------+-------------+
  |    10 | 0.0728030204773 | 0.00782012939453 |      9.3    |
  |   100 | 0.609349966049  | 0.0377721786499  |     16.1    |
  |  1000 | 5.86522197723   | 0.370730876923   |     15.8    |
  | 10000 | 66.1713931561   | 3.61764383316    |     18.3    |
  +-------+-----------------+------------------+-------------+
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The previous solution would make use of concurrent get's to couch
backend in a pool of threads to implement the get_docs() and
get_all_docs() CouchDatabase backend methods.

This commit replaces those by a simpler implementation use the
`_all_docs` couchdb view api. It passes all needed IDs to the view and
r etrieves all documents with content in the same request.

A comparison between both implementations shows an improvement of at
least 15 times for large number of documents. The table below shows the
time for different implementations of get_all_docs() for different
number of documents and threads versus _all_docs implementation:

  +-------+-----------------+------------------+-------------+
  |       |     threads     |    _all_docs     | improvement |
  +-------+-----------------+------------------+-------------+
  |    10 | 0.0728030204773 | 0.00782012939453 |      9.3    |
  |   100 | 0.609349966049  | 0.0377721786499  |     16.1    |
  |  1000 | 5.86522197723   | 0.370730876923   |     15.8    |
  | 10000 | 66.1713931561   | 3.61764383316    |     18.3    |
  +-------+-----------------+------------------+-------------+
</pre>
</div>
</content>
</entry>
</feed>
