soledad.git
9 months agoFix Grammar master
CyberDrudge [Fri, 18 Jan 2019 14:10:49 +0000 (19:40 +0530)]
Fix Grammar

11 months agoRemove benchmark templates and jobs from CI
drebs [Thu, 15 Nov 2018 10:44:02 +0000 (08:44 -0200)]
Remove benchmark templates and jobs from CI

11 months agoDisable benchmarks
drebs [Thu, 15 Nov 2018 10:41:13 +0000 (08:41 -0200)]
Disable benchmarks

Benchmarks post results to elasticsearch, but the TLS certificates of
https://benchmarks.leap.se and https://moose.leap.se are currently
expired. Automated LetsEncrypt certificate renewal must be configured
before benchmarks can be enabled again.

16 months ago[pkg] bump debian changelog
Kali Kaneko [Tue, 3 Jul 2018 19:44:17 +0000 (21:44 +0200)]
[pkg] bump debian changelog

16 months ago[docs] bump version in docs
Kali Kaneko [Tue, 3 Jul 2018 19:35:53 +0000 (21:35 +0200)]
[docs] bump version in docs

16 months ago[pkg] fix author in debian changelog
Kali Kaneko [Tue, 3 Jul 2018 19:16:28 +0000 (21:16 +0200)]
[pkg] fix author in debian changelog

16 months ago[pkg] fix upload error in pypi
Kali Kaneko [Tue, 3 Jul 2018 19:15:11 +0000 (21:15 +0200)]
[pkg] fix upload error in pypi

16 months ago[pkg] bump version to 0.10.6
Kali Kaneko [Sat, 30 Jun 2018 11:31:56 +0000 (13:31 +0200)]
[pkg] bump version to 0.10.6

16 months ago[tests] disable e2e tests for now
drebs [Mon, 18 Jun 2018 13:54:08 +0000 (10:54 -0300)]
[tests] disable e2e tests for now

e2e tests currently depend on client dev environment running a platform
with blobs enabled. As the platform currently doesn't integrate with
blobs, any deploy to cdev environment breaks blobs configuration and
thus breaks e2e tests.

I am disabling these tests for now so we don't depend on them to have
packages built for soledad.

16 months ago[package] remove jessie, and zesty, add bionic and sid
Micah Anderson [Thu, 14 Jun 2018 18:58:08 +0000 (14:58 -0400)]
[package] remove jessie, and zesty, add bionic and sid

17 months agoClarify storage secrets use case and backups in documentation
drebs [Sat, 2 Jun 2018 12:57:27 +0000 (09:57 -0300)]
Clarify storage secrets use case and backups in documentation

17 months agoAllow passing empty server url for testing purposes
drebs [Sat, 2 Jun 2018 12:56:37 +0000 (09:56 -0300)]
Allow passing empty server url for testing purposes

17 months agoMark tests that need couch
drebs [Sat, 2 Jun 2018 12:55:44 +0000 (09:55 -0300)]
Mark tests that need couch

21 months ago[doc] improve blobs streaming documentation
drebs [Sun, 21 Jan 2018 17:15:31 +0000 (15:15 -0200)]
[doc] improve blobs streaming documentation

- move stream api doc to api section
- refactor section structure of blobs sync doc page
- improve wording a bit here and there

21 months ago[doc] add stream format to docs
Victor Shyba [Mon, 8 Jan 2018 19:40:17 +0000 (16:40 -0300)]
[doc] add stream format to docs

- Relates: #9003

22 months ago[bug] fix error catching in blobs resource
drebs [Thu, 4 Jan 2018 21:05:03 +0000 (19:05 -0200)]
[bug] fix error catching in blobs resource

22 months ago[style] fix typos and add comments from review
Victor Shyba [Thu, 4 Jan 2018 02:31:00 +0000 (23:31 -0300)]
[style] fix typos and add comments from review

22 months ago[tests] add a test for incomplete downloaded blob
Victor Shyba [Mon, 1 Jan 2018 00:16:04 +0000 (21:16 -0300)]
[tests] add a test for incomplete downloaded blob

22 months ago[bug] handle incomplete preamble as a retriable error
Victor Shyba [Mon, 1 Jan 2018 00:14:49 +0000 (21:14 -0300)]
[bug] handle incomplete preamble as a retriable error

22 months ago[feature] turn fs lock into a decorator
drebs [Fri, 29 Dec 2017 21:54:57 +0000 (19:54 -0200)]
[feature] turn fs lock into a decorator

22 months ago[bug] isolate operations in blobs file system backend
drebs [Fri, 29 Dec 2017 20:32:24 +0000 (18:32 -0200)]
[bug] isolate operations in blobs file system backend

Closes: #9025

22 months ago[test] decrease the number of blobs in fs backend benchmarks
drebs [Thu, 28 Dec 2017 10:17:48 +0000 (08:17 -0200)]
[test] decrease the number of blobs in fs backend benchmarks

22 months ago[test] fix blobs fs backend benchmarks to account for multiple rounds
drebs [Wed, 27 Dec 2017 16:47:27 +0000 (14:47 -0200)]
[test] fix blobs fs backend benchmarks to account for multiple rounds

22 months ago[test] use pull producer in blobs fs backend benchmarks
drebs [Wed, 27 Dec 2017 16:07:52 +0000 (14:07 -0200)]
[test] use pull producer in blobs fs backend benchmarks

22 months ago[doc] add documentation about the blobs range header
drebs [Tue, 26 Dec 2017 11:14:40 +0000 (09:14 -0200)]
[doc] add documentation about the blobs range header

22 months ago[bug] add BlobNotFound exception to methods interface
drebs [Fri, 22 Dec 2017 20:28:55 +0000 (18:28 -0200)]
[bug] add BlobNotFound exception to methods interface

22 months ago[feature] add ranges to blobs backend
drebs [Tue, 19 Dec 2017 20:00:23 +0000 (18:00 -0200)]
[feature] add ranges to blobs backend

22 months ago[doc] improve documentation of ciphertext size estimation function
drebs [Tue, 26 Dec 2017 10:48:18 +0000 (08:48 -0200)]
[doc] improve documentation of ciphertext size estimation function

22 months ago[doc] add comment to clarify order of results
drebs [Tue, 26 Dec 2017 10:38:30 +0000 (08:38 -0200)]
[doc] add comment to clarify order of results

22 months ago[refactor] use explicit stream direction argument
Victor Shyba [Sun, 24 Dec 2017 23:25:31 +0000 (20:25 -0300)]
[refactor] use explicit stream direction argument

download was being inferred. It's now being set on each and will raise
an error if node is provided. Also removed a duplication on params
variable.

22 months ago[bug] add code to obey protocol pause and stop
Victor Shyba [Sun, 24 Dec 2017 23:24:35 +0000 (20:24 -0300)]
[bug] add code to obey protocol pause and stop

Stream production wasn't pausing or stopping as asked by protocol.

22 months ago[test] fix checked range for size
Victor Shyba [Sun, 24 Dec 2017 23:21:30 +0000 (20:21 -0300)]
[test] fix checked range for size

Was too high due a left over from some bug hunting. 20 is enough to
check both zero sized and under/equal/above tag size.

22 months ago[refactor] move preamble constant to its module
Victor Shyba [Sun, 24 Dec 2017 23:20:06 +0000 (20:20 -0300)]
[refactor] move preamble constant to its module

22 months ago[feature] consume received stream in a thread
Victor Shyba [Wed, 20 Dec 2017 16:59:47 +0000 (13:59 -0300)]
[feature] consume received stream in a thread

As it's blocking and Twisted already stored everything on a file.

-- Related: #8809

22 months ago[feature] add a producer protocol for upstream
Victor Shyba [Wed, 20 Dec 2017 16:58:21 +0000 (13:58 -0300)]
[feature] add a producer protocol for upstream

-- Resolves: #8809

22 months ago[feature] add a resource for streaming
Victor Shyba [Thu, 14 Dec 2017 17:33:21 +0000 (14:33 -0300)]
[feature] add a resource for streaming

-- Related: #8809

22 months ago[feature] client implementation for upstream
Victor Shyba [Thu, 14 Dec 2017 17:30:52 +0000 (14:30 -0300)]
[feature] client implementation for upstream

-- Resolves #8773

22 months ago[test] test upstream from namespace
Victor Shyba [Thu, 14 Dec 2017 17:30:20 +0000 (14:30 -0300)]
[test] test upstream from namespace

22 months ago[feature] get_unarmored_ciphertext_size on client
Victor Shyba [Thu, 14 Dec 2017 17:29:14 +0000 (14:29 -0300)]
[feature] get_unarmored_ciphertext_size on client

22 months ago[test] ciphertext size function tests
Victor Shyba [Thu, 14 Dec 2017 17:28:05 +0000 (14:28 -0300)]
[test] ciphertext size function tests

22 months ago[feature] get_size_list for listing blobs sizes
Victor Shyba [Thu, 14 Dec 2017 17:27:02 +0000 (14:27 -0300)]
[feature] get_size_list for listing blobs sizes

22 months ago[test] add tests to blobs size list
Victor Shyba [Thu, 14 Dec 2017 17:25:45 +0000 (14:25 -0300)]
[test] add tests to blobs size list

22 months ago[pkg] override newer-standards-version lintian error
drebs [Thu, 21 Dec 2017 16:51:23 +0000 (14:51 -0200)]
[pkg] override newer-standards-version lintian error

22 months ago[pkg] bump debian standards version to 4.1.2
drebs [Wed, 20 Dec 2017 18:39:12 +0000 (16:39 -0200)]
[pkg] bump debian standards version to 4.1.2

22 months ago[pkg] avoid complaining about dependency on python2
drebs [Wed, 20 Dec 2017 18:38:32 +0000 (16:38 -0200)]
[pkg] avoid complaining about dependency on python2

22 months ago[doc] fix docstring wording
drebs [Tue, 19 Dec 2017 18:48:59 +0000 (16:48 -0200)]
[doc] fix docstring wording

22 months ago[refactor] move blobs state to its own submodule
drebs [Tue, 19 Dec 2017 18:39:23 +0000 (16:39 -0200)]
[refactor] move blobs state to its own submodule

22 months ago[refactor] move blobs fs backend and resource to their own submodules
drebs [Tue, 19 Dec 2017 18:10:27 +0000 (16:10 -0200)]
[refactor] move blobs fs backend and resource to their own submodules

22 months ago[refactor] move blobs errors to its own submodule
drebs [Tue, 19 Dec 2017 17:59:35 +0000 (15:59 -0200)]
[refactor] move blobs errors to its own submodule

22 months ago[refactor] move server blobs to its own submodule
drebs [Tue, 19 Dec 2017 17:31:13 +0000 (15:31 -0200)]
[refactor] move server blobs to its own submodule

22 months ago[test] account for first timers in outlier calculation
drebs [Tue, 19 Dec 2017 17:34:45 +0000 (15:34 -0200)]
[test] account for first timers in outlier calculation

When it's the first time a test is run, there'll not be previous results
to compare against. The script that calculates outliers was not prepared
for this situation. This commit just fixes that.

22 months ago[doc] small improvements in server doc
drebs [Tue, 19 Dec 2017 17:22:07 +0000 (15:22 -0200)]
[doc] small improvements in server doc

22 months ago[test] use producer in all fs backend tests
drebs [Mon, 18 Dec 2017 19:06:23 +0000 (17:06 -0200)]
[test] use producer in all fs backend tests

22 months ago[benchmarks] multiply by 10 the number of blobs in fs backend benchmarks
drebs [Mon, 18 Dec 2017 14:15:19 +0000 (12:15 -0200)]
[benchmarks] multiply by 10 the number of blobs in fs backend benchmarks

22 months ago[doc] add write_blob exceptions to interface docstring
drebs [Mon, 18 Dec 2017 14:10:45 +0000 (12:10 -0200)]
[doc] add write_blob exceptions to interface docstring

22 months ago[feature] cache user quota on blobs filesystem backend
drebs [Mon, 18 Dec 2017 14:10:01 +0000 (12:10 -0200)]
[feature] cache user quota on blobs filesystem backend

Closes: 9016

22 months ago[refactor] use producer/consumer on write/read_blob respectivelly
drebs [Tue, 12 Dec 2017 22:11:13 +0000 (20:11 -0200)]
[refactor] use producer/consumer on write/read_blob respectivelly

22 months ago[test] fix incoming test with the new async blobs backend
drebs [Sat, 9 Dec 2017 15:58:11 +0000 (13:58 -0200)]
[test] fix incoming test with the new async blobs backend

22 months ago[bug] handle path exceptions using twisted failures
drebs [Sat, 9 Dec 2017 15:44:29 +0000 (13:44 -0200)]
[bug] handle path exceptions using twisted failures

22 months ago[bug] fix exception catching with new blobs backend interface
drebs [Sat, 9 Dec 2017 15:35:15 +0000 (13:35 -0200)]
[bug] fix exception catching with new blobs backend interface

22 months ago[doc] improve documentation on IBlobsBackend
drebs [Sat, 9 Dec 2017 10:48:53 +0000 (08:48 -0200)]
[doc] improve documentation on IBlobsBackend

22 months ago[refactor] make read_blob() return a deferred
drebs [Sat, 9 Dec 2017 10:09:34 +0000 (08:09 -0200)]
[refactor] make read_blob() return a deferred

22 months ago[refactor] make set_flags() return a deferred
drebs [Sat, 9 Dec 2017 00:54:50 +0000 (22:54 -0200)]
[refactor] make set_flags() return a deferred

22 months ago[refactor] make get_flags() return a deferred
drebs [Sat, 9 Dec 2017 00:48:02 +0000 (22:48 -0200)]
[refactor] make get_flags() return a deferred

22 months ago[refactor] make get_tag() return a deferred
drebs [Sat, 9 Dec 2017 00:41:59 +0000 (22:41 -0200)]
[refactor] make get_tag() return a deferred

22 months ago[refactor] make get_total_storage() return a deferred
drebs [Sat, 9 Dec 2017 00:36:24 +0000 (22:36 -0200)]
[refactor] make get_total_storage() return a deferred

22 months ago[refactor] make list_blobs() return a deferred
drebs [Sat, 9 Dec 2017 00:34:34 +0000 (22:34 -0200)]
[refactor] make list_blobs() return a deferred

22 months ago[refactor] make count() return a deferred
drebs [Sat, 9 Dec 2017 00:16:11 +0000 (22:16 -0200)]
[refactor] make count() return a deferred

22 months ago[refactor] make get_blob_size() return a deferred
drebs [Fri, 8 Dec 2017 23:55:45 +0000 (21:55 -0200)]
[refactor] make get_blob_size() return a deferred

22 months ago[refactor] make delete_blob() return a deferred
drebs [Fri, 8 Dec 2017 23:54:13 +0000 (21:54 -0200)]
[refactor] make delete_blob() return a deferred

22 months ago[refactor] make blobs backend delete_blob() agnostic of twisted.web requests
drebs [Wed, 6 Dec 2017 22:37:55 +0000 (20:37 -0200)]
[refactor] make blobs backend delete_blob() agnostic of twisted.web requests

22 months ago[refactor] make blobs backend write_blob() agnostic of twisted.web requests
drebs [Wed, 6 Dec 2017 22:29:44 +0000 (20:29 -0200)]
[refactor] make blobs backend write_blob() agnostic of twisted.web requests

22 months ago[refactor] make blobs backend set_flags() agnostic of twisted.web requests
drebs [Wed, 6 Dec 2017 21:48:53 +0000 (19:48 -0200)]
[refactor] make blobs backend set_flags() agnostic of twisted.web requests

22 months ago[refactor] make blobs backend read_blob() agnostic of twisted.web
drebs [Wed, 6 Dec 2017 21:34:32 +0000 (19:34 -0200)]
[refactor] make blobs backend read_blob() agnostic of twisted.web

22 months ago[refactor] make blobs backend list_blobs() agnostic of twisted.web requests
drebs [Wed, 6 Dec 2017 21:24:39 +0000 (19:24 -0200)]
[refactor] make blobs backend list_blobs() agnostic of twisted.web requests

22 months ago[refactor] make blobs backend get_flags() agnostic of twisted.web requests
drebs [Wed, 6 Dec 2017 20:16:33 +0000 (18:16 -0200)]
[refactor] make blobs backend get_flags() agnostic of twisted.web requests

22 months ago[refactor] make blobs backend count() agnostic of twisted.web requests
drebs [Wed, 6 Dec 2017 20:09:40 +0000 (18:09 -0200)]
[refactor] make blobs backend count() agnostic of twisted.web requests

22 months ago[refactor] make blobs backend get_tag() agnostic of twisted.web requests
drebs [Wed, 6 Dec 2017 20:01:31 +0000 (18:01 -0200)]
[refactor] make blobs backend get_tag() agnostic of twisted.web requests

22 months ago[feature] add an exception for blobs not found
drebs [Wed, 6 Dec 2017 20:47:24 +0000 (18:47 -0200)]
[feature] add an exception for blobs not found

22 months ago[refactor] remove unneeded class
drebs [Wed, 6 Dec 2017 19:52:51 +0000 (17:52 -0200)]
[refactor] remove unneeded class

22 months ago[doc] add api doc for blobs backend
drebs [Wed, 6 Dec 2017 19:48:32 +0000 (17:48 -0200)]
[doc] add api doc for blobs backend

23 months ago[bug] use namespace when uploading blobs
drebs [Sun, 10 Dec 2017 23:22:13 +0000 (21:22 -0200)]
[bug] use namespace when uploading blobs

23 months ago[test] fix bug in blobs fs backend benchmark test
drebs [Tue, 12 Dec 2017 15:16:18 +0000 (13:16 -0200)]
[test] fix bug in blobs fs backend benchmark test

The new benchmark test for blobs fs backend was rebased from a merge
request in which it was written using the new interface for blobs
backend. After rebase, the test was not reverted to old interface, and
was just failing bad. This commit fixes it.

23 months ago[benchmark] add read/write benchmark for blobs fs backend
drebs [Sat, 9 Dec 2017 12:57:21 +0000 (10:57 -0200)]
[benchmark] add read/write benchmark for blobs fs backend

23 months ago[bug] transfer maximum allowed number of blobs
drebs [Tue, 5 Dec 2017 13:48:20 +0000 (11:48 -0200)]
[bug] transfer maximum allowed number of blobs

-- Closes: #9004

23 months ago[bug] escape tag and size
Victor Shyba [Mon, 4 Dec 2017 18:52:56 +0000 (15:52 -0300)]
[bug] escape tag and size

It was generating spaces, causing split to fail sometimes.

23 months ago[style] fixes from code review
Victor Shyba [Mon, 4 Dec 2017 14:24:35 +0000 (11:24 -0300)]
[style] fixes from code review

23 months ago[feature] stream blobs from a CooperativeTask
Victor Shyba [Tue, 28 Nov 2017 00:08:12 +0000 (21:08 -0300)]
[feature] stream blobs from a CooperativeTask

Adds a CooperativeTask to stream blobs to client as the transport is
able to handle it, pausing and resuming as necessary.

-- Related: #8809

23 months ago[feature] adds a stream downloader
Victor Shyba [Mon, 27 Nov 2017 16:05:33 +0000 (13:05 -0300)]
[feature] adds a stream downloader

First version, still missing consumer/producer model and some tweaks,
but working.

-- Related: #8809

23 months ago[feature] add implementation for get_blob_size
Victor Shyba [Mon, 27 Nov 2017 12:44:51 +0000 (09:44 -0300)]
[feature] add implementation for get_blob_size

23 months ago[test] init StreamingResource on test_blobs_server
Victor Shyba [Mon, 27 Nov 2017 12:36:04 +0000 (09:36 -0300)]
[test] init StreamingResource on test_blobs_server

-- Related: #8809

23 months ago[refactor] get namespace as an optional query arg
Victor Shyba [Mon, 27 Nov 2017 12:34:41 +0000 (09:34 -0300)]
[refactor] get namespace as an optional query arg

23 months ago[bug] missing self parameter on class method
Victor Shyba [Mon, 27 Nov 2017 12:33:21 +0000 (09:33 -0300)]
[bug] missing self parameter on class method

23 months ago[feature] add a streaming resource
Victor Shyba [Mon, 27 Nov 2017 11:47:12 +0000 (08:47 -0300)]
[feature] add a streaming resource

-- Related: #8809

23 months ago[style] fix typo
Victor Shyba [Mon, 27 Nov 2017 02:11:21 +0000 (23:11 -0300)]
[style] fix typo

23 months ago[style] remove public get/set blob priority methods
drebs [Thu, 30 Nov 2017 22:32:53 +0000 (20:32 -0200)]
[style] remove public get/set blob priority methods

23 months ago[bug] avoid overwriting priorities on sync status batch update
drebs [Thu, 30 Nov 2017 21:14:47 +0000 (19:14 -0200)]
[bug] avoid overwriting priorities on sync status batch update

23 months ago[style] some sugar for setting the priority
Kali Kaneko [Thu, 30 Nov 2017 15:20:18 +0000 (16:20 +0100)]
[style] some sugar for setting the priority

23 months ago[feature] add priorities for blob transfers
drebs [Wed, 29 Nov 2017 13:42:29 +0000 (11:42 -0200)]
[feature] add priorities for blob transfers

Closes: #8691

23 months ago[feature] add bandwidth throttling to the client
drebs [Thu, 23 Nov 2017 18:05:51 +0000 (16:05 -0200)]
[feature] add bandwidth throttling to the client

Intercept the creation of the protocol factory in the HTTP connection
pool to use twisted.protocols.policies.ThrottlingFactory and control the
incoming and outgoing bandwidth.

The factory only controls one connection, so when throttling we limit
the number of connections of the pool to one per host. This way,
throttling happens in a per-host basis.

Closes: #8931