From 0a09a6e6f247729457d15480f8d2b9bb0b89ae5e Mon Sep 17 00:00:00 2001 From: elijah Date: Mon, 29 Aug 2016 22:55:41 -0700 Subject: Updated (very out of date) docs and README.md --- docs/en/services/couchdb.html | 328 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 328 insertions(+) create mode 100644 docs/en/services/couchdb.html (limited to 'docs/en/services/couchdb.html') diff --git a/docs/en/services/couchdb.html b/docs/en/services/couchdb.html new file mode 100644 index 00000000..3dde5a3c --- /dev/null +++ b/docs/en/services/couchdb.html @@ -0,0 +1,328 @@ + + + + +couchdb - LEAP Platform Documentation + + + + + + + + +
+
+

couchdb

+ +
Data storage for all user data.
+
+
+ + +

Topology

+ +

Required:

+ +
    +
  • Nodes with couchdb service must also have soledad service, if email is enabled.
  • +
+ + +

Suggested:

+ +
    +
  • Nodes with couchdb service communicate heavily with webapp and mx.
  • +
+ + +

couchdb nodes do not need to be reachable from the public internet, although the soledad service does require this.

+ +

Configuration

+ +

Nighly dumps

+ +

You can do a nightly couchdb data dump by adding this to your node config:

+ +
"couch": {
+  "backup": true
+}
+
+ +

Data will get dumped to /var/backups/couchdb.

+ +

Plain CouchDB

+ +

BigCouch is not supported on Platform version 0.8 and higher: only plain CouchDB is possible. For earlier versions, you must do this in order to use plain CouchDB:

+ +
"couch": {
+  "master": true,
+  "pwhash_alg": "pbkdf2"
+}
+
+ +

Various Tasks

+ +

Re-enabling blocked account

+ +

When a user account gets destroyed from the webapp, there’s still a leftover doc in the identities db so other people can’t claim that account without an admin’s intervention. You can remove this username reservation through the webapp.

+ +

However, here is how you could do it manually, if you wanted to:

+ +

grep the identities db for the email address:

+ +
curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET http://127.0.0.1:5984/identities/_all_docs?include_docs=true|grep test_127@bitmask.net
+
+ +

lookup “id” and “rev” to delete the doc:

+ +
curl -s --netrc-file /etc/couchdb/couchdb.netrc -X DELETE 'http://127.0.0.1:5984/identities/b25cf10f935b58088f0d547fca823265?rev=2-715a9beba597a2ab01851676f12c3e4a'
+
+ +

How to find out which userstore belongs to which identity?

+ +
/usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep testuser
+
+{"id":"665e004870ee17aa4c94331ff3ecb173","key":"665e004870ee17aa4c94331ff3ecb173","value":{"rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b"},"doc":{"_id":"665e004870ee17aa4c94331ff3ecb173","_rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b","user_id":"665e004870ee17aa4c94331ff3cd59eb","address":"testuser@example.org","destination":"testuser@example.org","keys": ...
+
+ +
    +
  • search for the “user_id” field
  • +
  • in this example testuser@example.org uses the database user-665e004870ee17aa4c94331ff3cd59eb
  • +
+ + +

How much disk space is used by a userstore

+ +

Beware that this returns the uncompacted disk size (see http://wiki.apache.org/couchdb/Compaction)

+ +
echo "`curl --netrc -s -X GET 'http://127.0.0.1:5984/user-dcd6492d74b90967b6b874100b7dbfcf'|json_pp|grep disk_size|cut -d: -f 2`/1024"|bc
+
+ +

Migrating from BigCouch to plain CouchDB

+ +

At the end of this process, you will have just one node with services property equal to couchdb. If you had a BigCouch cluster before, you will be removing all but one of those machines to consolidate them into one CouchDB machine.

+ +
    +
  1. if you have multiple nodes with the couchdb service on them, pick one of them to be your CouchDB server, and remove the service from the others. If these machines were only doing BigCouch before, you can remove the nodes completely with leap node rm <nodename> and then you can decommission the servers

  2. +
  3. put the webapp into maintenance mode

  4. +
  5. turn off daemons that access the database. For example:

    + +
     workstation$ leap ssh <each soledad-node>
    + server# /etc/init.d/soledad-server stop
    +
    + workstation$ leap ssh <mx-node>
    + server# /etc/init.d/postfix stop
    + server# /etc/init.d/leap-mx stop
    +
    + workstation$ leap ssh <webapp-node>
    + server# /etc/init.d/nickserver stop
    +
    + +

    Alternately, you can create a temporary firewall rule to block access (run on couchdb server):

    + +
     server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT
    +
  6. +
  7. remove orphaned databases and do a backup of all remaining, active databases. This can take some time and will place several hundred megabytes of data into /var/backups/couchdb. The size and time depends on how many users there are on your system. For example, 15k users took approximately 25 minutes and 308M of space:

    + +
     workstation$ leap ssh <couchdb-node>
    + server# cd /srv/leap/couchdb/scripts
    + server# ./cleanup-user-dbs
    + server# time ./couchdb_dumpall.sh
    +
  8. +
  9. stop bigcouch:

    + +
     server# /etc/init.d/bigcouch stop
    + server# pkill epmd
    +
  10. +
  11. remove bigcouch:

    + +
     server# apt-get remove bigcouch
    +
  12. +
  13. configure your couch node to use plain couchdb instead of bigcouch, you can do this by editing nodes/.json, look for this section:

    + +
     "couch": {
    +   "mode": "plain"
    + }
    +
    + +

    change it, so it looks like this instead:

    + +
      "couch": {
    +    "mode": "plain",
    +    "pwhash_alg": "pbkdf2"
    +  }
    +
  14. +
+ +

+ +

    +
  1. restore the backup, this will take approximately the same amount of time as the backup took above:

    + +
     server# cd /srv/leap/couchdb/scripts
    + server# time ./couchdb_restoreall.sh
    +
  2. +
  3. start services again that were stopped in the beginning:

    + +
     workstation$ leap ssh soledad-nodes
    + server# /etc/init.d/soledad-server start
    +
    + workstation$ leap ssh mx-node
    + server# /etc/init.d/postfix start
    + server# /etc/init.d/leap-mx start
    +
    + workstation$ leap ssh webapp
    + server# /etc/init.d/nickserver start
    +
    + +

    Or, alternately, if you set up the firewall rule instead, now remove it:

    + +
     server# iptables -D INPUT -p tcp --dport 5984 --jump REJECT
    +
  4. +
+ +

+ +

    +
  1. check if everything is working, including running the test on your deployment machine:

    + +
     workstation$ leap test
    +
  2. +
  3. Remove old bigcouch data dir /opt after you double checked everything is in place

  4. +
  5. Relax, enjoy a refreshing beverage.

  6. +
+ +

+ +
+
+ + -- cgit v1.2.3 From 811deee9e5b8cc42a3ea424ef873e9d69eb50cba Mon Sep 17 00:00:00 2001 From: elijah Date: Wed, 14 Sep 2016 15:09:32 -0700 Subject: updated docs --- docs/en/services/couchdb.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs/en/services/couchdb.html') diff --git a/docs/en/services/couchdb.html b/docs/en/services/couchdb.html index 3dde5a3c..d8bc8553 100644 --- a/docs/en/services/couchdb.html +++ b/docs/en/services/couchdb.html @@ -215,7 +215,7 @@ couchdb - LEAP Platform Documentation -- cgit v1.2.3 From 238b0853711b91422eff5b97397b549a66370f72 Mon Sep 17 00:00:00 2001 From: elijah Date: Wed, 14 Sep 2016 15:31:42 -0700 Subject: refresh /docs/ --- docs/en/services/couchdb.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs/en/services/couchdb.html') diff --git a/docs/en/services/couchdb.html b/docs/en/services/couchdb.html index d8bc8553..6de6455c 100644 --- a/docs/en/services/couchdb.html +++ b/docs/en/services/couchdb.html @@ -215,7 +215,7 @@ couchdb - LEAP Platform Documentation
  • search for the “user_id” field
  • -
  • in this example testuser@example.org uses the database user-665e004870ee17aa4c94331ff3cd59eb
  • +
  • in this example testuser@example.org uses the database user-665e004870ee17aa4c94331ff3cd59eb
-- cgit v1.2.3