From 672a6d4e62376a834378aa70c91ab8612b15094c Mon Sep 17 00:00:00 2001 From: drebs Date: Thu, 28 Sep 2017 14:47:04 -0300 Subject: [doc] move development docs to a subdir --- docs/benchmarks.rst | 106 -------------------------------------- docs/contributing.rst | 48 ----------------- docs/deprecation.rst | 34 ------------ docs/development.rst | 8 +-- docs/development/benchmarks.rst | 106 ++++++++++++++++++++++++++++++++++++++ docs/development/contributing.rst | 48 +++++++++++++++++ docs/development/deprecation.rst | 34 ++++++++++++ docs/development/tests.rst | 33 ++++++++++++ docs/tests.rst | 33 ------------ 9 files changed, 225 insertions(+), 225 deletions(-) delete mode 100644 docs/benchmarks.rst delete mode 100644 docs/contributing.rst delete mode 100644 docs/deprecation.rst create mode 100644 docs/development/benchmarks.rst create mode 100644 docs/development/contributing.rst create mode 100644 docs/development/deprecation.rst create mode 100644 docs/development/tests.rst delete mode 100644 docs/tests.rst (limited to 'docs') diff --git a/docs/benchmarks.rst b/docs/benchmarks.rst deleted file mode 100644 index 25e39ae7..00000000 --- a/docs/benchmarks.rst +++ /dev/null @@ -1,106 +0,0 @@ -.. _benchmarks: - -Benchmarks -========== - -We currently use `pytest-benchmark `_ -to write tests to assess the time and resources taken by various tasks. - -To run benchmark tests, once inside a cloned Soledad repository, do the -following:: - - tox -e benchmark - -Results of automated benchmarking for each commit in the repository can be seen -in: https://benchmarks.leap.se/. - -Benchmark tests also depend on `tox` and `CouchDB`. See the :ref:`tests` page -for more information on how to setup the test environment. - -Test repetition ---------------- - -``pytest-benchmark`` runs tests multiple times so it can provide meaningful -statistics for the time taken for a tipical run of a test function. The number -of times that the test is run can be manually or automatically configured. - -When automatically configured, the number of runs is decided by taking into -account multiple ``pytest-benchmark`` configuration parameters. See the `the -corresponding documenation -`_ for more -details on how automatic calibration works. - -To achieve a reasonable number of repetitions and a reasonable amount of time -at the same time, we let ``pytest-benchmark`` choose the number of repetitions -for faster tests, and manually limit the number of repetitions for slower tests. - -Currently, tests for `synchronization` and `sqlcipher asynchronous document -creation` are fixed to run 4 times each. All the other tests are left for -``pytest-benchmark`` to decide how many times to run each one. With this setup, -the benchmark suite is taking approximatelly 7 minutes to run in our CI server. -As the benchmark suite is run twice (once for time and cpu stats and a second -time for memory stats), the whole benchmarks run takes around 15 minutes. - -The actual number of times a test is run when calibration is done automatically -by ``pytest-benchmark`` depends on many parameters: the time taken for a sample -run and the configuration of the minimum number of rounds and maximum time -allowed for a benchmark. For a snapshot of the number of rounds for each test -function see `the soledad benchmarks wiki page -`_. - -Sync size statistics --------------------- - -Currenly, the main use of Soledad is to synchronize client-encrypted email -data. Because of that, it makes sense to measure the time and resources taken -to synchronize an amount of data that is realistically comparable to a user's -email box. - -In order to determine what is a good example of dataset for synchronization -tests, we used the size of messages of one week of incoming and outgoing email -flow of a friendly provider. The statistics that came out from that are (all -sizes are in KB): - -+--------+-----------+-----------+ -| | outgoing | incoming | -+========+===========+===========+ -| min | 0.675 | 0.461 | -+--------+-----------+-----------+ -| max | 25531.361 | 25571.748 | -+--------+-----------+-----------+ -| mean | 252.411 | 110.626 | -+--------+-----------+-----------+ -| median | 5.320 | 14.974 | -+--------+-----------+-----------+ -| mode | 1.404 | 1.411 | -+--------+-----------+-----------+ -| stddev | 1376.930 | 732.933 | -+--------+-----------+-----------+ - -Sync test scenarios -------------------- - -Ideally, we would want to run tests for a big data set (i.e. a high number of -documents and a big payload size), but that may be infeasible given time and -resource limitations. Because of that, we choose a smaller data set and suppose -that the behaviour is somewhat linear to get an idea for larger sets. - -Supposing a data set total size of 10MB, some possibilities for number of -documents and document sizes for testing download and upload can be seen below. -Scenarios marked in bold are the ones that are actually run in the current sync -benchmark tests, and you can see the current graphs for each one by following -the corresponding links: - - -* 10 x 1M -* **20 x 500K** (`upload `_, `download `_) -* **100 x 100K** (`upload `_, `download `_) -* 200 x 50K -* **1000 x 10K** (`upload `_, `download `_) - -In each of the above scenarios all the documents are of the same size. If we -want to account for some variability on document sizes, it is sufficient to -come up with a simple scenario where the average, minimum and maximum sizes are -somehow coherent with the above statistics, like the following one: - -* 60 x 15KB + 1 x 1MB diff --git a/docs/contributing.rst b/docs/contributing.rst deleted file mode 100644 index 76e7450c..00000000 --- a/docs/contributing.rst +++ /dev/null @@ -1,48 +0,0 @@ -Contributing -============ - -Thank you for your interest in contributing to Soledad! - -Filing bug reports ------------------- - -Bug reports are very welcome. Please file them on the `0xacab issue tracker -`_. Please include an extensive -description of the error, the steps to reproduce it, the version of Soledad -used and the provider used (if applicable). - -Patches -------- - -All patches to Soledad should be submitted in the form of pull requests to the -`main Soledad repository `_. These pull -requests should satisfy the following properties: - -* The pull request should focus on one particular improvement to Soledad. - Create different pull requests for unrelated features or bugfixes. -* Code should follow `PEP 8 `_, - especially in the "do what code around you does" sense. -* Pull requests that introduce code must test all new behavior they introduce - as well as for previously untested or poorly tested behavior that they touch. -* Pull requests are not allowed to break existing tests. We will consider pull - requests that are breaking the CI as work in progress. -* When introducing new functionality, please remember to write documentation. -* Please sign all of your commits. If you are merging code from other - contributors, you should sign their commits. - -Review -~~~~~~ - -Pull requests must be reviewed before merging. The final responsibility for the -reviewing of merged code lies with the person merging it. Exceptions to this -rule are modifications to documentation, packaging, or tests when the need of -agility of merging without review has little or no impact on the security and -functionality of working code. - -Getting help -~~~~~~~~~~~~ - -If you need help you can reach us through one of the following means: - -* IRC: ``#leap`` at ``irc.freenode.org``. -* Mailing list: `leap-discuss@lists.riseup.net `_ diff --git a/docs/deprecation.rst b/docs/deprecation.rst deleted file mode 100644 index d0454d5a..00000000 --- a/docs/deprecation.rst +++ /dev/null @@ -1,34 +0,0 @@ -Backwards-compatibility and deprecation policy -============================================== - -Since Soledad has not reached a stable `1.0` release yet, no guarantees are made -about the stability of its API or the backwards-compatibility of any given -version. - -Currently, the internal storage representation is experimenting changes that -will take some time to mature and settle up. For the moment, no given SOLEDAD -release is offering any backwards-compatibility guarantees. - -Although serious efforts are being made to ensure no data is corrupted or lost -while upgrading soledad versions, it's not advised to use SOLEDAD for any -critical storage at the moment, or to upgrade versions without any external data -backup (for instance, an email application that uses SOLEDAD should allow to -export mail data or PGP keys in a convertible format before upgrading). - -Deprecation Policy ------------------- - -The points above standing, the development team behind SOLEDAD will strive to -provide clear migration paths between any two given, consecutive **minor -releases**, in an automated form wherever possible. - -This means, for example, that a migration script will be provided with the -``0.10`` release, to migrate data stored by any of the ``0.9.x`` soledad -versions. Another script will be provided to migrate from ``0.10`` to ``0.11``, -etc (but not, for instance, from ``0.8`` to ``0.10``). - -At the same time, there's a backwards-compatibility policy of **deprecating APIs -after 2 minor releases**. This means that a feature will start to be marked as -deprecated in ``0.10``, with a warning being raised for 2 minor releases, and -the API will disappear completely no sooner than in ``0.12``. - diff --git a/docs/development.rst b/docs/development.rst index 1f9044e2..a7ff15aa 100644 --- a/docs/development.rst +++ b/docs/development.rst @@ -7,8 +7,8 @@ should help you get started with development for Soledad. .. toctree:: :maxdepth: 1 - contributing - deprecation - tests - benchmarks + development/contributing + development/deprecation + development/tests + development/benchmarks changelog diff --git a/docs/development/benchmarks.rst b/docs/development/benchmarks.rst new file mode 100644 index 00000000..25e39ae7 --- /dev/null +++ b/docs/development/benchmarks.rst @@ -0,0 +1,106 @@ +.. _benchmarks: + +Benchmarks +========== + +We currently use `pytest-benchmark `_ +to write tests to assess the time and resources taken by various tasks. + +To run benchmark tests, once inside a cloned Soledad repository, do the +following:: + + tox -e benchmark + +Results of automated benchmarking for each commit in the repository can be seen +in: https://benchmarks.leap.se/. + +Benchmark tests also depend on `tox` and `CouchDB`. See the :ref:`tests` page +for more information on how to setup the test environment. + +Test repetition +--------------- + +``pytest-benchmark`` runs tests multiple times so it can provide meaningful +statistics for the time taken for a tipical run of a test function. The number +of times that the test is run can be manually or automatically configured. + +When automatically configured, the number of runs is decided by taking into +account multiple ``pytest-benchmark`` configuration parameters. See the `the +corresponding documenation +`_ for more +details on how automatic calibration works. + +To achieve a reasonable number of repetitions and a reasonable amount of time +at the same time, we let ``pytest-benchmark`` choose the number of repetitions +for faster tests, and manually limit the number of repetitions for slower tests. + +Currently, tests for `synchronization` and `sqlcipher asynchronous document +creation` are fixed to run 4 times each. All the other tests are left for +``pytest-benchmark`` to decide how many times to run each one. With this setup, +the benchmark suite is taking approximatelly 7 minutes to run in our CI server. +As the benchmark suite is run twice (once for time and cpu stats and a second +time for memory stats), the whole benchmarks run takes around 15 minutes. + +The actual number of times a test is run when calibration is done automatically +by ``pytest-benchmark`` depends on many parameters: the time taken for a sample +run and the configuration of the minimum number of rounds and maximum time +allowed for a benchmark. For a snapshot of the number of rounds for each test +function see `the soledad benchmarks wiki page +`_. + +Sync size statistics +-------------------- + +Currenly, the main use of Soledad is to synchronize client-encrypted email +data. Because of that, it makes sense to measure the time and resources taken +to synchronize an amount of data that is realistically comparable to a user's +email box. + +In order to determine what is a good example of dataset for synchronization +tests, we used the size of messages of one week of incoming and outgoing email +flow of a friendly provider. The statistics that came out from that are (all +sizes are in KB): + ++--------+-----------+-----------+ +| | outgoing | incoming | ++========+===========+===========+ +| min | 0.675 | 0.461 | ++--------+-----------+-----------+ +| max | 25531.361 | 25571.748 | ++--------+-----------+-----------+ +| mean | 252.411 | 110.626 | ++--------+-----------+-----------+ +| median | 5.320 | 14.974 | ++--------+-----------+-----------+ +| mode | 1.404 | 1.411 | ++--------+-----------+-----------+ +| stddev | 1376.930 | 732.933 | ++--------+-----------+-----------+ + +Sync test scenarios +------------------- + +Ideally, we would want to run tests for a big data set (i.e. a high number of +documents and a big payload size), but that may be infeasible given time and +resource limitations. Because of that, we choose a smaller data set and suppose +that the behaviour is somewhat linear to get an idea for larger sets. + +Supposing a data set total size of 10MB, some possibilities for number of +documents and document sizes for testing download and upload can be seen below. +Scenarios marked in bold are the ones that are actually run in the current sync +benchmark tests, and you can see the current graphs for each one by following +the corresponding links: + + +* 10 x 1M +* **20 x 500K** (`upload `_, `download `_) +* **100 x 100K** (`upload `_, `download `_) +* 200 x 50K +* **1000 x 10K** (`upload `_, `download `_) + +In each of the above scenarios all the documents are of the same size. If we +want to account for some variability on document sizes, it is sufficient to +come up with a simple scenario where the average, minimum and maximum sizes are +somehow coherent with the above statistics, like the following one: + +* 60 x 15KB + 1 x 1MB diff --git a/docs/development/contributing.rst b/docs/development/contributing.rst new file mode 100644 index 00000000..76e7450c --- /dev/null +++ b/docs/development/contributing.rst @@ -0,0 +1,48 @@ +Contributing +============ + +Thank you for your interest in contributing to Soledad! + +Filing bug reports +------------------ + +Bug reports are very welcome. Please file them on the `0xacab issue tracker +`_. Please include an extensive +description of the error, the steps to reproduce it, the version of Soledad +used and the provider used (if applicable). + +Patches +------- + +All patches to Soledad should be submitted in the form of pull requests to the +`main Soledad repository `_. These pull +requests should satisfy the following properties: + +* The pull request should focus on one particular improvement to Soledad. + Create different pull requests for unrelated features or bugfixes. +* Code should follow `PEP 8 `_, + especially in the "do what code around you does" sense. +* Pull requests that introduce code must test all new behavior they introduce + as well as for previously untested or poorly tested behavior that they touch. +* Pull requests are not allowed to break existing tests. We will consider pull + requests that are breaking the CI as work in progress. +* When introducing new functionality, please remember to write documentation. +* Please sign all of your commits. If you are merging code from other + contributors, you should sign their commits. + +Review +~~~~~~ + +Pull requests must be reviewed before merging. The final responsibility for the +reviewing of merged code lies with the person merging it. Exceptions to this +rule are modifications to documentation, packaging, or tests when the need of +agility of merging without review has little or no impact on the security and +functionality of working code. + +Getting help +~~~~~~~~~~~~ + +If you need help you can reach us through one of the following means: + +* IRC: ``#leap`` at ``irc.freenode.org``. +* Mailing list: `leap-discuss@lists.riseup.net `_ diff --git a/docs/development/deprecation.rst b/docs/development/deprecation.rst new file mode 100644 index 00000000..d0454d5a --- /dev/null +++ b/docs/development/deprecation.rst @@ -0,0 +1,34 @@ +Backwards-compatibility and deprecation policy +============================================== + +Since Soledad has not reached a stable `1.0` release yet, no guarantees are made +about the stability of its API or the backwards-compatibility of any given +version. + +Currently, the internal storage representation is experimenting changes that +will take some time to mature and settle up. For the moment, no given SOLEDAD +release is offering any backwards-compatibility guarantees. + +Although serious efforts are being made to ensure no data is corrupted or lost +while upgrading soledad versions, it's not advised to use SOLEDAD for any +critical storage at the moment, or to upgrade versions without any external data +backup (for instance, an email application that uses SOLEDAD should allow to +export mail data or PGP keys in a convertible format before upgrading). + +Deprecation Policy +------------------ + +The points above standing, the development team behind SOLEDAD will strive to +provide clear migration paths between any two given, consecutive **minor +releases**, in an automated form wherever possible. + +This means, for example, that a migration script will be provided with the +``0.10`` release, to migrate data stored by any of the ``0.9.x`` soledad +versions. Another script will be provided to migrate from ``0.10`` to ``0.11``, +etc (but not, for instance, from ``0.8`` to ``0.10``). + +At the same time, there's a backwards-compatibility policy of **deprecating APIs +after 2 minor releases**. This means that a feature will start to be marked as +deprecated in ``0.10``, with a warning being raised for 2 minor releases, and +the API will disappear completely no sooner than in ``0.12``. + diff --git a/docs/development/tests.rst b/docs/development/tests.rst new file mode 100644 index 00000000..72e2d087 --- /dev/null +++ b/docs/development/tests.rst @@ -0,0 +1,33 @@ +.. _tests: + +Tests +===== + +We use `pytest `_ as a testing framework +and `Tox `_ as a test environment manager. +Currently, tests reside in the `testing/` folder and some of them need a +couchdb server to be run against. + +If you do have a couchdb server running on localhost on default port, the +following command should be enough to run tests:: + + tox + +CouchDB dependency +------------------ + +In case you want to use a couchdb on another host or port, use the +`--couch-url` parameter for `pytest`:: + + tox -- --couch-url=http://couch_host:5984 + +If you want to exclude all tests that depend on couchdb, deselect tests marked +with `needs_couch`:: + + tox -- -m 'not needs_couch' + +Benchmark tests +--------------- + +A set of benchmark tests is provided to measure the time and resources taken to +perform some actions. See the :ref:`documentation for benchmarks `. diff --git a/docs/tests.rst b/docs/tests.rst deleted file mode 100644 index 72e2d087..00000000 --- a/docs/tests.rst +++ /dev/null @@ -1,33 +0,0 @@ -.. _tests: - -Tests -===== - -We use `pytest `_ as a testing framework -and `Tox `_ as a test environment manager. -Currently, tests reside in the `testing/` folder and some of them need a -couchdb server to be run against. - -If you do have a couchdb server running on localhost on default port, the -following command should be enough to run tests:: - - tox - -CouchDB dependency ------------------- - -In case you want to use a couchdb on another host or port, use the -`--couch-url` parameter for `pytest`:: - - tox -- --couch-url=http://couch_host:5984 - -If you want to exclude all tests that depend on couchdb, deselect tests marked -with `needs_couch`:: - - tox -- -m 'not needs_couch' - -Benchmark tests ---------------- - -A set of benchmark tests is provided to measure the time and resources taken to -perform some actions. See the :ref:`documentation for benchmarks `. -- cgit v1.2.3