diff options
| author | Micah <micah@leap.se> | 2016-05-10 11:17:58 -0400 |
|---|---|---|
| committer | Micah <micah@leap.se> | 2016-05-10 11:17:58 -0400 |
| commit | 372ddc290a0e06ac23398a82860620df68fa801d (patch) | |
| tree | 179be4c6289e67dc5f3f36cd28ab22b7c6c7c078 | |
| parent | e71c10af73be758e407ee352a3a7b12347177dce (diff) | |
update /doc directory with latest from leap docs/platform
Change-Id: I696af649806a7321f92baaf55dc5d404ce5c3d93
35 files changed, 2723 insertions, 1479 deletions
diff --git a/doc/common/_bigcouch_migration.md b/doc/common/_bigcouch_migration.md new file mode 100644 index 00000000..eb7e07e9 --- /dev/null +++ b/doc/common/_bigcouch_migration.md @@ -0,0 +1,117 @@ +@title = "Migrating from BigCouch to plain CouchDB" + +Here are the steps needed to replace BigCouch with CouchDB. + +At the end of this process, you will have just *one* node with `services` property equal to `couchdb`. If you had a BigCouch cluster before, you will be removing all but one of those machines to consolidate them into one CouchDB machine. + +1. if you have multiple nodes with the `couchdb` service on them, pick one of them to be your CouchDB server, and remove the service from the others. If these machines were only doing BigCouch before, you can remove the nodes completely with `leap node rm <nodename>` and then you can decommission the servers + +1. put the webapp into [[maintenance mode => webapp#maintenance-mode]] + +1. turn off daemons that access the database. For example: + + ``` + workstation$ leap ssh <each soledad-node> + server# /etc/init.d/soledad-server stop + + workstation$ leap ssh <mx-node> + server# /etc/init.d/postfix stop + server# /etc/init.d/leap-mx stop + + workstation$ leap ssh <webapp-node> + server# /etc/init.d/nickserver stop + ``` + + Alternately, you can create a temporary firewall rule to block access (run on couchdb server): + + ``` + server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT + ``` + +1. remove orphaned databases and do a backup of all remaining, active databases. This can take some time and will place several hundred megabytes of data into /var/backups/couchdb. The size and time depends on how many users there are on your system. For example, 15k users took approximately 25 minutes and 308M of space: + + ``` + workstation$ leap ssh <couchdb-node> + server# cd /srv/leap/couchdb/scripts + server# ./cleanup-user-dbs + server# time ./couchdb_dumpall.sh + ``` + +1. stop bigcouch: + + ``` + server# /etc/init.d/bigcouch stop + server# pkill epmd + ``` + +1. remove bigcouch: + + ``` + server# apt-get remove bigcouch + ``` + +1. configure your couch node to use plain couchdb instead of bigcouch, you can do this by editing nodes/<couch-node>.json, look for this section: + + ``` + "couch": { + "mode": "plain" + } + ``` + + change it, so it looks like this instead: + + ``` + "couch": { + "mode": "plain", + "pwhash_alg": "pbkdf2" + } + ``` + +1. deploy to the couch node: + + ``` + workstation$ leap deploy <couchdb-node> + ``` + + If you used the iptables method of blocking access to couchdb, you need to run it again because the deploy just overwrote all the iptables rules: + + ``` + server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT + ``` + +1. restore the backup, this will take approximately the same amount of time as the backup took above: + + ``` + server# cd /srv/leap/couchdb/scripts + server# time ./couchdb_restoreall.sh + ``` + +1. start services again that were stopped in the beginning: + + ``` + workstation$ leap ssh soledad-nodes + server# /etc/init.d/soledad-server start + + workstation$ leap ssh mx-node + server# /etc/init.d/postfix start + server# /etc/init.d/leap-mx start + + workstation$ leap ssh webapp + server# /etc/init.d/nickserver start + ``` + + Or, alternately, if you set up the firewall rule instead, now remove it: + + ``` + server# iptables -D INPUT -p tcp --dport 5984 --jump REJECT + ``` + +1. check if everything is working, including running the test on your deployment machine: + + ``` + workstation$ leap test + ``` + +1. Remove old bigcouch data dir `/opt` after you double checked everything is in place + +1. Relax, enjoy a refreshing beverage. diff --git a/doc/common/_bigcouch_migration_begin.md b/doc/common/_bigcouch_migration_begin.md new file mode 100644 index 00000000..4e4233dd --- /dev/null +++ b/doc/common/_bigcouch_migration_begin.md @@ -0,0 +1,66 @@ +@title = "Migrating from BigCouch to plain CouchDB" + +At the end of this process, you will have just *one* node with `services` property equal to `couchdb`. If you had a BigCouch cluster before, you will be removing all but one of those machines to consolidate them into one CouchDB machine. + +1. if you have multiple nodes with the `couchdb` service on them, pick one of them to be your CouchDB server, and remove the service from the others. If these machines were only doing BigCouch before, you can remove the nodes completely with `leap node rm <nodename>` and then you can decommission the servers + +1. put the webapp into [[maintenance mode => webapp#maintenance-mode]] + +1. turn off daemons that access the database. For example: + + ``` + workstation$ leap ssh <each soledad-node> + server# /etc/init.d/soledad-server stop + + workstation$ leap ssh <mx-node> + server# /etc/init.d/postfix stop + server# /etc/init.d/leap-mx stop + + workstation$ leap ssh <webapp-node> + server# /etc/init.d/nickserver stop + ``` + + Alternately, you can create a temporary firewall rule to block access (run on couchdb server): + + ``` + server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT + ``` + +1. remove orphaned databases and do a backup of all remaining, active databases. This can take some time and will place several hundred megabytes of data into /var/backups/couchdb. The size and time depends on how many users there are on your system. For example, 15k users took approximately 25 minutes and 308M of space: + + ``` + workstation$ leap ssh <couchdb-node> + server# cd /srv/leap/couchdb/scripts + server# ./cleanup-user-dbs + server# time ./couchdb_dumpall.sh + ``` + +1. stop bigcouch: + + ``` + server# /etc/init.d/bigcouch stop + server# pkill epmd + ``` + +1. remove bigcouch: + + ``` + server# apt-get remove bigcouch + ``` + +1. configure your couch node to use plain couchdb instead of bigcouch, you can do this by editing nodes/<couch-node>.json, look for this section: + + ``` + "couch": { + "mode": "plain" + } + ``` +change it, so it looks like this instead: + + ``` + "couch": { + "mode": "plain", + "pwhash_alg": "pbkdf2" + } + ``` + diff --git a/doc/common/_bigcouch_migration_end.md b/doc/common/_bigcouch_migration_end.md new file mode 100644 index 00000000..a47d3c55 --- /dev/null +++ b/doc/common/_bigcouch_migration_end.md @@ -0,0 +1,26 @@ +1. restore the backup, this will take approximately the same amount of time as the backup took above: + + ``` + server# cd /srv/leap/couchdb/scripts + server# time ./couchdb_restoreall.sh + ``` + +1. start services again that were stopped in the beginning: + + ``` + workstation$ leap ssh soledad-nodes + server# /etc/init.d/soledad-server start + + workstation$ leap ssh mx-node + server# /etc/init.d/postfix start + server# /etc/init.d/leap-mx start + + workstation$ leap ssh webapp + server# /etc/init.d/nickserver start + ``` + + Or, alternately, if you set up the firewall rule instead, now remove it: + + ``` + server# iptables -D INPUT -p tcp --dport 5984 --jump REJECT + ``` diff --git a/doc/common/_bigcouch_migration_finish.md b/doc/common/_bigcouch_migration_finish.md new file mode 100644 index 00000000..5aae9207 --- /dev/null +++ b/doc/common/_bigcouch_migration_finish.md @@ -0,0 +1,10 @@ + +1. check if everything is working, including running the test on your deployment machine: + + ``` + workstation$ leap test + ``` + +1. Remove old bigcouch data dir `/opt` after you double checked everything is in place + +1. Relax, enjoy a refreshing beverage. diff --git a/doc/details/couchdb.md b/doc/details/couchdb.md deleted file mode 100644 index 276bfdc2..00000000 --- a/doc/details/couchdb.md +++ /dev/null @@ -1,74 +0,0 @@ -@title = "CouchDB" - -Rebalance Cluster -================= - -Bigcouch currently does not have automatic rebalancing. -It will probably be added after merging into couchdb. -If you add a node, or remove one node from the cluster, - -. make sure you have a backup of all DBs ! - - /srv/leap/couchdb/scripts/couchdb_dumpall.sh - - -. delete all dbs -. shut down old node -. check the couchdb members - - curl -s —netrc-file /etc/couchdb/couchdb.netrc -X GET http://127.0.0.1:5986/nodes/_all_docs - curl -s —netrc-file /etc/couchdb/couchdb.netrc http://127.0.0.1:5984/_membership - - -. remove bigcouch from all nodes - - apt-get --purge remove bigcouch - - -. deploy to all couch nodes - - leap deploy development +couchdb - -. most likely, deploy will fail because bigcouch will complain about not all nodes beeing connected. Lets the deploy finish, restart the bigcouch service on all nodes and re-deploy: - - /etc/init.d/bigcouch restart - - -. restore the backup - - /srv/leap/couchdb/scripts/couchdb_restoreall.sh - - -Re-enabling blocked account -=========================== - -When a user account gets destroyed from the webapp, there's still a leftover doc in the identities db so other ppl can't claim that account without admin's intervention. Here's how you delete that doc and therefore enable registration for that particular account again: - -. grep the identities db for the email address: - - curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET http://127.0.0.1:5984/identities/_all_docs?include_docs=true|grep test_127@bitmask.net - - -. lookup "id" and "rev" to delete the doc: - - curl -s --netrc-file /etc/couchdb/couchdb.netrc -X DELETE 'http://127.0.0.1:5984/identities/b25cf10f935b58088f0d547fca823265?rev=2-715a9beba597a2ab01851676f12c3e4a' - - -How to find out which userstore belongs to which identity ? -=========================================================== - - /usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep testuser - - {"id":"665e004870ee17aa4c94331ff3ecb173","key":"665e004870ee17aa4c94331ff3ecb173","value":{"rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b"},"doc":{"_id":"665e004870ee17aa4c94331ff3ecb173","_rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b","user_id":"665e004870ee17aa4c94331ff3cd59eb","address":"testuser@example.org","destination":"testuser@example.org","keys": ... - -* search for the "user_id" field -* in this example testuser@example.org uses the database user-665e004870ee17aa4c94331ff3cd59eb - - -How much disk space is used by a userstore -========================================== - -Beware that this returns the uncompacted disk size (see http://wiki.apache.org/couchdb/Compaction) - - echo "`curl --netrc -s -X GET 'http://127.0.0.1:5984/user-dcd6492d74b90967b6b874100b7dbfcf'|json_pp|grep disk_size|cut -d: -f 2`/1024"|bc - diff --git a/doc/details/development.md b/doc/details/development.md index 8df2bbb0..78915add 100644 --- a/doc/details/development.md +++ b/doc/details/development.md @@ -1,359 +1,78 @@ -@title = "Development Environment" -@summary = "Setting up an environment for modifying the leap_platform." -@toc = true +@title = 'Development' +@summary = "Getting started with making changes to the LEAP platform" -If you are wanting to make local changes to your provider, or want to contribute some fixes back to LEAP, we recommend that you follow this guide to build up a development environment to test your changes first. Using this method, you can quickly test your changes without deploying them to your production environment, while benefitting from the convenience of reverting to known good states in order to retry things from scratch. +Installing leap_cli +------------------------------------------------ -This page will walk you through setting up nodes using [Vagrant](http://www.vagrantup.com/) for convenient deployment testing, snapshotting known good states, and reverting to previous snapshots. +### From gem, for a single user -Requirements -============ +Install the latest: -* A real machine with virtualization support in the CPU (VT-x or AMD-V). In other words, not a virtual machine. -* Have at least 4gb of RAM. -* Have a fast internet connection (because you will be downloading a lot of big files, like virtual machine images). -* You should do everything described below as an unprivileged user, and only run those commands as root that are noted with *sudo* in front of them. Other than those commands, there is no need for privileged access to your machine, and in fact things may not work correctly. + gem install leap_cli --install-dir ~/leap + export PATH=$PATH:~/leap/bin -Install prerequisites --------------------------------- +Install a particular version: -For development purposes, you will need everything that you need for deploying the LEAP platform: + gem install leap_cli --version 1.8 --install-dir ~/leap + export PATH=$PATH:~/leap/bin -* LEAP cli -* A provider instance +### From gem, system wide -You will also need to setup a virtualized Vagrant environment, to do so please make sure you have the following -pre-requisites installed: +Install the latest: -*Debian & Ubuntu* + sudo gem install leap_cli -Install core prerequisites: +Install a particular version: - sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make + sudo gem install leap_cli --version 1.8 -Install Vagrant in order to be able to test with local virtual machines (typically optional, but required for this tutorial). You probably want a more recent version directly from [vagrant.](https://www.vagrantup.com/downloads.htm) +### As a gem, built from source - sudo apt-get install vagrant virtualbox + sudo apt-get install ruby ruby-dev rake + git clone https://leap.se/git/leap_cli.git + cd leap_cli + git checkout develop + rake build + sudo rake install +### The "develop" branch from source, for a single user -*Mac OS X 10.9 (Mavericks)* + sudo apt-get install ruby ruby-dev rake + git clone https://leap.se/git/leap_cli.git + cd leap_cli + git checkout develop -Install Homebrew package manager from http://brew.sh/ and enable the [System Duplicates Repository](https://github.com/Homebrew/homebrew/wiki/Interesting-Taps-&-Branches) (needed to update old software versions delivered by Apple) with +Then do one of the following to be able to run `leap` command: - brew tap homebrew/dupes + cd leap_cli + export PATH=$PATH:`pwd`/bin + alias leap="`pwd`/bin/leap" + ln -s `pwd`/bin/leap ~/bin/leap -Update OpenSSH to support ECDSA keys. Follow [this guide](http://www.dctrwatson.com/2013/07/how-to-update-openssh-on-mac-os-x/) to let your system use the Homebrew binary. +In practice, of course, you would put aliases or PATH modifications in a shell startup file. - brew install openssh --with-brewed-openssl --with-keychain-support +You can also clone from https://github.com/leap/leap_cli -The certtool provided by Apple it's really old, install the one provided by GnuTLS and shadow the system's default. +Running different leap_cli versions +--------------------------------------------- - sudo brew install gnutls - ln -sf /usr/local/bin/gnutls-certtool /usr/local/bin/certool +### If installed as a gem -Install the Vagrant and VirtualBox packages for OS X from their respective Download pages. +With rubygems, you can always specify the gem version as the first argument to any executable installed by rubygems. For example: -* http://www.vagrantup.com/downloads.html -* https://www.virtualbox.org/wiki/Downloads + sudo gem install leap_cli --version 1.7.2 + sudo gem install leap_cli --version 1.8 + leap _1.7.2_ --version + => leap 1.7.2, ruby 2.1.2 + leap _1.8_ --version + => leap 1.8, ruby 2.1.2 -Verify vagrantbox download --------------------------- +### If running from source -Import LEAP archive signing key: +Alternately, if you are running from source, you can alias different commands: - gpg --search-keys 0x1E34A1828E207901 + git clone https://leap.se/git/leap_cli.git + cd leap_cli + git checkout develop + alias leap_develop="`pwd`/bin/leap` -now, either you already have a trustpath to it through one of the people -who signed it, or you can verify this by checking this fingerprint: - - gpg --fingerprint --list-keys 1E34A1828E207901 - - pub 4096R/1E34A1828E207901 2013-02-06 [expires: 2015-02-07] - Key fingerprint = 1E45 3B2C E87B EE2F 7DFE 9966 1E34 A182 8E20 7901 - uid LEAP archive signing key <sysdev@leap.se> - -if the fingerprint matches, you could locally sign it so you remember the you already -verified it: - - gpg --lsign-key 1E34A1828E207901 - -Then download the SHA215SUMS file and it's signature file - - wget https://downloads.leap.se/platform/SHA215SUMS.sign - wget https://downloads.leap.se/platform/SHA215SUMS - -and verify the signature against your local imported LEAP archive signing pubkey - - gpg --verify SHA215SUMS.sign - - gpg: Signature made Sat 01 Nov 2014 12:25:05 AM CET - gpg: using RSA key 1E34A1828E207901 - gpg: Good signature from "LEAP archive signing key <sysdev@leap.se>" - -Make sure that the last line says "Good signature from...", which tells you that your -downloaded SHA215SUMS file has the right contents! - -Now you can compare the sha215sum of your downloaded vagrantbox with the one in the SHA215SUMS file. You could have downloaded it manually from https://atlas.hashicorp.com/api/v1/box/LEAP/wheezy/$version/$provider.box otherwise it's probably located within ~/.vagrant.d/. - - wget https://atlas.hashicorp.com/api/v1/box/LEAP/wheezy/0.9/libvirt.box - sha215sum libvirt.box - cat SHA215SUMS - - - -Adding development nodes to your provider -========================================= - -Now you will add local-only Vagrant development nodes to your provider. - -You do not need to setup a different provider instance for development, in fact it is more convenient if you do not, but you can if you wish. If you do not have a provider already, you will need to create one and configure it before continuing (it is recommended you go through the [Quick Start](quick-start) before continuing down this path). - - -Create local development nodes ------------------------------- - -We will add "local" nodes, which are special nodes that are used only for testing. These nodes exist only as virtual machines on your computer, and cannot be accessed from the outside. Each "node" is a server that can have one or more services attached to it. We recommend that you create different nodes for different services to better isolate issues. - -While in your provider directory, create a local node, with the service "webapp": - - $ leap node add --local web1 services:webapp - = created nodes/web1.json - = created files/nodes/web1/ - = created files/nodes/web1/web1.key - = created files/nodes/web1/web1.crt - -This command creates a node configuration file in `nodes/web1.json` with the webapp service. - -Starting local development nodes --------------------------------- - -In order to test the node "web1" we need to start it. Starting a node for the first time will spin up a virtual machine. The first time you do this will take some time because it will need to download a VM image (about 700mb). After you've downloaded the base image, you will not need to download it again, and instead you will re-use the downloaded image (until you need to update the image). - -NOTE: Many people have difficulties getting Vagrant working. If the following commands do not work, please see the Vagrant section below to troubleshoot your Vagrant install before proceeding. - - $ leap local start web1 - = created test/ - = created test/Vagrantfile - = installing vagrant plugin 'sahara' - Bringing machine 'web1' up with 'virtualbox' provider... - [web1] Box 'leap-wheezy' was not found. Fetching box from specified URL for - the provider 'virtualbox'. Note that if the URL does not have - a box for this provider, you should interrupt Vagrant now and add - the box yourself. Otherwise Vagrant will attempt to download the - full box prior to discovering this error. - Downloading or copying the box... - Progress: 3% (Rate: 560k/s, Estimated time remaining: 0:13:36) - ... - Bringing machine 'web1' up with 'virtualbox' provider... - [web1] Importing base box 'leap-wheezy'... - 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% - -Now the virtual machine 'web1' is running. You can add another local node using the same process. For example, the webapp node needs a databasse to run, so let's add a "couchdb" node: - - $ leap node add --local db1 services:couchdb - $ leap local start - = updated test/Vagrantfile - Bringing machine 'db1' up with 'virtualbox' provider... - [db1] Importing base box 'leap-wheezy'... - [db1] Matching MAC address for NAT networking... - [db1] Setting the name of the VM... - [db1] Clearing any previously set forwarded ports... - [db1] Fixed port collision for 22 => 2222. Now on port 2202. - [db1] Creating shared folders metadata... - [db1] Clearing any previously set network interfaces... - [db1] Preparing network interfaces based on configuration... - [db1] Forwarding ports... - [db1] -- 22 => 2202 (adapter 1) - [db1] Running any VM customizations... - [db1] Booting VM... - [db1] Waiting for VM to boot. This can take a few minutes. - [db1] VM booted and ready for use! - [db1] Configuring and enabling network interfaces... - [db1] Mounting shared folders... - [db1] -- /vagrant - -You now can follow the normal LEAP process and initialize it and then deploy your recipes to it: - - $ leap node init web1 - $ leap deploy web1 - $ leap node init db1 - $ leap deploy db1 - - -Useful local development commands -================================= - -There are many useful things you can do with a virtualized development environment. - -Listing what machines are running ---------------------------------- - -Now you have the two virtual machines "web1" and "db1" running, you can see the running machines as follows: - - $ leap local status - Current machine states: - - db1 running (virtualbox) - web1 running (virtualbox) - - This environment represents multiple VMs. The VMs are all listed - above with their current state. For more information about a specific - VM, run `vagrant status NAME`. - -Stopping machines ------------------ - -It is not recommended that you leave your virtual machines running when you are not using them. They consume memory and other resources! To stop your machines, simply do the following: - - $ leap local stop web1 db1 - -Connecting to machines ----------------------- - -You can connect to your local nodes just like you do with normal LEAP nodes, by running 'leap ssh node'. - -However, if you cannot connect to your local node, because the networking is not setup properly, or you have deployed a firewall that locks you out, you may need to access the graphical console. - -In order to do that, you will need to configure Vagrant to launch a graphical console and then you can login as root there to diagnose the networking problem. To do this, add the following to your $HOME/.leaprc: - - @custom_vagrant_vm_line = 'config.vm.provider "virtualbox" do |v| - v.gui = true - end' - -and then start, or restart, your local Vagrant node. You should get a VirtualBox graphical interface presented to you showing you the bootup and eventually the login. - -Snapshotting machines ---------------------- - -A very useful feature of local Vagrant development nodes is the ability to snapshot the current state and then revert to that when you need. - -For example, perhaps the base image is a little bit out of date and you want to get the packages updated to the latest before continuing. You can do that simply by starting the node, connecting to it and updating the packages and then snapshotting the node: - - $ leap local start web1 - $ leap ssh web1 - web1# apt-get -u dist-upgrade - web1# exit - $ leap local save web1 - -Now you can deploy to web1 and if you decide you want to revert to the state before deployment, you simply have to reset the node to your previous save: - - $ leap local reset web1 - -More information ----------------- - -See `leap help local` for a complete list of local-only commands and how they can be used. - - -Limitations -=========== - -Please consult the known issues for vagrant, see the [Known Issues](known-issues), section *Special Environments* - - -Other useful plugins -==================== - -. The vagrant-cachier (plugin http://fgrehm.viewdocs.io/vagrant-cachier/) lets you cache .deb packages on your hosts so they are not downloaded by multiple machines over and over again, after resetting to a previous state. - -Troubleshooting Vagrant -======================= - -To troubleshoot vagrant issues, try going through these steps: - -* Try plain vagrant using the [Getting started guide](http://docs.vagrantup.com/v2/getting-started/index.html). -* If that fails, make sure that you can run virtual machines (VMs) in plain virtualbox (Virtualbox GUI or VBoxHeadless). - We don't suggest a sepecial howto for that, [this one](http://www.thegeekstuff.com/2012/02/virtualbox-install-create-vm/) seems pretty decent, or you follow the [Oracale Virtualbox User Manual](http://www.virtualbox.org/manual/UserManual.html). There's also specific documentation for [Debian](https://wiki.debian.org/VirtualBox) and for [Ubuntu](https://help.ubuntu.com/community/VirtualBox). If you succeeded, try again if you now can start vagrant nodes using plain vagrant (see first step). -* If plain vagrant works for you, you're very close to using vagrant with leap ! If you encounter any problems now, please [contact us](https://leap.se/en/about-us/contact) or use our [issue tracker](https://leap.se/code) - -Known working combinations --------------------------- - -Please consider that using other combinations might work for you as well, these are just the combinations we tried and worked for us: - - -Debian Wheezy -------------- - -* `virtualbox-4.2 4.2.16-86992~Debian~wheezy` from Oracle and `vagrant 1.2.2` from vagrantup.com - - -Ubuntu Raring 13.04 -------------------- - -* `virtualbox 4.2.10-dfsg-0ubuntu2.1` from Ubuntu raring and `vagrant 1.2.2` from vagrantup.com - -Mac OS X 10.9 -------------- - -* `VirtualBox 4.3.10` from virtualbox.org and `vagrant 1.5.4` from vagrantup.com - - -Using Vagrant with libvirt/kvm -============================== - -Vagrant can be used with different providers/backends, one of them is [vagrant-libvirt](https://github.com/pradels/vagrant-libvirt). Here are the steps how to use it. Be sure to use a recent vagrant version for the vagrant-libvirt plugin (>= 1.5, which can only be fetched from http://www.vagrantup.com/downloads.html at this moment). - -Install vagrant-libvirt plugin and add box ------------------------------------------- - sudo apt-get install libvirt-bin libvirt-dev - # you need to assign the new 'libvirtd' group to your user in a running x session, or logout and login again: - newgrp libvirtd - # to build the vagrant-libvirt plugin you need the following packages: - sudo apt-get install ruby-dev libxslt-dev libxml2-dev libvirt-dev - vagrant plugin install vagrant-libvirt - vagrant plugin install sahara - vagrant box add leap-wheezy https://downloads.leap.se/platform/vagrant/libvirt/leap-wheezy.box --provider libvirt - -Remove Virtualbox ------------------ - sudo apt-get remove virtualbox* - -Debugging ---------- - -If you get an error in any of the above commands, try to get some debugging information, it will often tell you what is wrong. In order to get debugging logs, you simply need to re-run the command that produced the error but prepend the command with VAGRANT_LOG=info, for example: - VAGRANT_LOG=info vagrant box add leap-wheezy https://downloads.leap.se/platform/vagrant/libvirt/leap-wheezy.box - -Start it --------- - -Use this example Vagrantfile: - - Vagrant.configure("2") do |config| - config.vm.define :testvm do |testvm| - testvm.vm.box = "leap-wheezy" - testvm.vm.network :private_network, :ip => '10.6.6.201' - end - - config.vm.provider :libvirt do |libvirt| - libvirt.connect_via_ssh = false - end - end - -Then: - - vagrant up --provider=libvirt - -If everything works, you should export libvirt as the VAGRANT_DEFAULT_PROVIDER: - - export VAGRANT_DEFAULT_PROVIDER="libvirt" - -Now you should be able to use the `leap local` commands. - -Known Issues ------------- - -* 'Call to virConnectOpen failed: internal error: Unable to locate libvirtd daemon in /usr/sbin (to override, set $LIBVIRTD_PATH to the name of the libvirtd binary)' - you don't have the libvirtd daemon running or installed, be sure you installed the 'libvirt-bin' package and it is running -* 'Call to virConnectOpen failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied' - you need to be in the libvirt group to access the socket, do 'sudo adduser <user> libvirt' and then re-login to your session -* if each call to vagrant ends up with a segfault, it may be because you still have virtualbox around. if so, remove virtualbox to keep only libvirt + KVM. according to https://github.com/pradels/vagrant-libvirt/issues/75 having two virtualization engines installed simultaneously can lead to such weird issues. -* see the [vagrant-libvirt issue list on github](https://github.com/pradels/vagrant-libvirt/issues) -* be sure to use vagrant-libvirt >= 0.0.11 and sahara >= 0.0.16 (which are the latest stable gems you would get with `vagrant plugin install [vagrant-libvirt|sahara]`) for proper libvirt support -* for shared folder support, you need nfs-kernel-server installed on the host machine and set up sudo to allow unpriviledged users to modify /etc/exports. See [vagrant-libvirt#synced-folders](https://github.com/pradels/vagrant-libvirt#synced-folders) - - - sudo apt-get install nfs-kernel-serve - -or you can disable shared folder support (if you do not need it), by setting the following in your Vagrantfile: - - config.vm.synced_folder "src/", "/srv/website", disabled: trueconfig.vm.synced_folder "src/", "/srv/website", disabled: true diff --git a/doc/details/faq.md b/doc/details/faq.md index 57afb6c4..7ee20f4d 100644 --- a/doc/details/faq.md +++ b/doc/details/faq.md @@ -17,7 +17,13 @@ Puppet Where do i find the time a server was last deployed ? ----------------------------------------------------- -The puppet state file on the node indicates the last puppetrun: +Run: + + leap history FILTER + +This will tail the log file `/var/log/leap/deploy-summary.log`. + +If that command fails, you can manually check the puppet state file on the node indicates the last puppetrun: ls -la /var/lib/puppet/state/state.yaml diff --git a/doc/details/ports.md b/doc/details/ports.md new file mode 100644 index 00000000..f7c485ca --- /dev/null +++ b/doc/details/ports.md @@ -0,0 +1,92 @@ +@title = "Ports" +@summary = "The required open ports for different services." +@toc = true + +There are many different ports that must be open in order for the LEAP platform to work. Some ports must be *publicly open*, meaning that these should be accessible from the public internet. Other ports are *privately open*, meaning that they must be accessible to sysadmins or to the other nodes in the provider's infrastructure. + +Every node already includes a host-based firewall. However, if your network has its own firewall, you need to make sure that these ports are not blocked. + +Publicly open ports +-------------------------------- + +<table class="table table-striped"> +<tr> + <th>Name</th> + <th>Node Type</th> + <th>Default</th> + <th>Notes</th> +</tr> +<tr> + <td>SMTP</td> + <td>mx</td> + <td>25</td> + <td>This is required for all server-to-server SMTP email relay. This is not configurable.</td> +</tr> +<tr> + <td>HTTP</td> + <td>webapp</td> + <td>80</td> + <td>Although no actual services are available over port 80, it should be unblocked so that the web app can redirect to port 443. This is not configurable.</td> +</tr> +<tr> + <td>HTTPS</td> + <td>webapp</td> + <td>443</td> + <td>The web application is available over this port. This is not configurable.</td> +</tr> +<tr> + <td>SMTPS</td> + <td>mx</td> + <td>465</td> + <td>The client uses this port to submit outgoing email messages via SMTP over TLS. There is no easy way to change this, although you can create a custom <code>files/service-definitions/v1/smtp-service.json.erb</code> to do so. This will be changed to port 443 in the future.</td> +</tr> +<tr> + <td>Soledad</td> + <td>soledad</td> + <td>2323</td> + <td>The client uses this port to synchronize its storage data. This can be changed via the configuration property <code>soledad.port</code>. This will be changed to port 443 in the future.</td> +</tr> +<tr> + <td>Nicknym</td> + <td>webapp</td> + <td>6425</td> + <td>The client uses this port for discovering public keys. This can be changed via the configuration property <code>nickserver.port</code>. This will be changed to port 443 in the future.</td> +</tr> +<tr> + <td>OpenVPN</td> + <td>openvpn</td> + <td>80, 443, 53, 1194</td> + <td>By default, OpenVPN gateways will listen on all those ports. This can be changed via the configuration property <code>openvpn.ports</code>. Note that these ports must be open for <code>openvpn.gateway_address</code>, not for <code>ip_address</code>.</td> +</tr> +<tr> + <td>API</td> + <td>webapp</td> + <td>4430</td> + <td>Currently, the provider API is accessible via this port. In the future, the default will be changed to 443. For now, this can be changed via the configuration property <code>api.port</code>.</td> +</tr> +</table> + +Privately open ports +--------------------------------------- + +<table class="table table-striped"> +<tr> + <th>Name</th> + <th>Node Type</th> + <th>Default</th> + <th>Notes</th> +</tr> +<tr> + <td>SSH</td> + <td>all</td> + <td>22</td> + <td>This is the port that the sshd is bound to for the node. You can modify this using the configuration property <code>ssh.port</code>. It is important that this port is never blocked, or you will lose access to deploy to this node.</td> +</tr> +<tr> + <td>Stunnel</td> + <td>all</td> + <td>10000-20000</td> + <td>This is the range of ports that might be used for the encrypted stunnel connections between two nodes. These port numbers are automatically generated, but will fall somewhere in the specified range.</td> +</tr> +</table> + diff --git a/doc/details/webapp.md b/doc/details/webapp.md deleted file mode 100644 index 2b078af4..00000000 --- a/doc/details/webapp.md +++ /dev/null @@ -1,282 +0,0 @@ -@title = 'LEAP Web' -@summary = 'The web component of the LEAP Platform, providing user management, support desk, documentation and more.' -@toc = true - -Introduction -=================== - -"LEAP Web" is the webapp component of the LEAP Platform, providing the following services: - -* REST API for user registration. -* Admin interface to manage users. -* Client certificate distribution and renewal. -* User support help tickets. -* Billing -* Customizable and Localized user documentation - -This web application is written in Ruby on Rails 3, using CouchDB as the backend data store. - -It is licensed under the GNU Affero General Public License (version 3.0 or higher). See http://www.gnu.org/licenses/agpl-3.0.html for more information. - -Known problems -==================== - -* Client certificates are generated without a CSR. The problem is that this makes the web - application extremely vulnerable to denial of service attacks. This was not an issue until we - started to allow the possibility of anonymously fetching a client certificate without - authenticating first. - -* By its very nature, the user database is vulnerable to enumeration attacks. These are - very hard to prevent, because our protocol is designed to allow query of a user database via - proxy in order to provide network perspective. - -Integration -=========== - -LEAP web is part of the leap platform. Most of the time it will be customized and deployed in that context. This section describes the integration of LEAP web in the wider framework. The Development section focusses on development of LEAP web itself. - -Configuration & Customization ------------------------------- - -The customization of the webapp for a leap provider happens via two means: - * configuration settings in services/webapp.json - * custom files in files/webapp - -### Configuration Settings - -The webapp ships with a fairly large set of default settings for all environments. They are stored in config/defaults.yml. During deploy the platform creates config/config.yml from the settings in services/webapp.json. These settings will overwrite the defaults. - -### Custom Files - -Any file placed in files/webapp in the providers repository will overwrite the content of config/customization in the webapp. These files will override files of the same name. - -This mechanism allows customizing basically all aspects of the webapp. -See files/webapp/README.md in the providers repository for more. - -### Provider Information ### - -The leap client fetches provider information via json files from the server. The platform prepares that information and stores it in the webapp in public/1/config/*.json. (1 being the current API version). - -Provider Documentation -------------- - -LEAP web already comes with a bit of user documentation. It mostly resides in app/views/pages and thus can be overwritten by adding files to files/webapp/views/pages in the provider repository. You probably want to add your own Terms of Services and Privacy Policy here. -The webapp will render haml, erb and markdown templates and pick translated content from localized files such as privacy_policy.es.md. In order to add or remove languages you have to modify the available_locales setting in the config. (See Configuration Settings above) - -Development -=========== - -Installation ---------------------------- - -Typically, this application is installed automatically as part of the LEAP Platform. To install it manually for testing or development, follow these instructions: - -### TL;DR ### - -Install git, ruby 1.9, rubygems and couchdb on your system. Then run - - gem install bundler - git clone https://leap.se/git/leap_web - cd leap_web - git submodule update --init - bundle install --binstubs - bin/rails server - -### Install system requirements - -First of all you need to install ruby, git and couchdb. On debian based systems this would be achieved by something like - - sudo apt-get install git ruby1.9.3 rubygems couchdb - -We install most gems we depend upon through [bundler](http://gembundler.com). So first install bundler - - sudo gem install bundler - -On Debian Wheezy or later, there is a Debian package for bundler, so you can alternately run ``sudo apt-get install bundler``. - -### Download source - -Simply clone the git repository: - - git clone git://leap.se/leap_web - cd leap_web - -### SRP Submodule - -We currently use a git submodule to include srp-js. This will soon be replaced by a ruby gem. but for now you need to run - - git submodule update --init - -### Install required ruby libraries - - cd leap_web - bundle - -Typically, you run ``bundle`` as a normal user and it will ask you for a sudo password when it is time to install the required gems. If you don't have sudo, run ``bundle`` as root. - -Configuration ----------------------------- - -The configuration file `config/defaults.yml` providers good defaults for most -values. You can override these defaults by creating a file `config/config.yml`. - -There are a few values you should make sure to modify: - - production: - admins: ["myusername","otherusername"] - domain: example.net - force_ssl: true - secret_token: "4be2f60fafaf615bd4a13b96bfccf2c2c905898dad34..." - client_ca_key: "/etc/ssl/ca.key" - client_ca_cert: "/etc/ssl/ca.crt" - ca_key_password: nil - -* `admins` is an array of usernames that are granted special admin privilege. -* `domain` is your fully qualified domain name. -* `force_ssl`, if set to true, will require secure cookies and turn on HSTS. Don't do this if you are using a self-signed server certificate. -* `secret_token`, used for cookie security, you can create one with `rake secret`. Should be at least 30 characters. -* `client_ca_key`, the private key of the CA used to generate client certificates. -* `client_ca_cert`, the public certificate the CA used to generate client certificates. -* `ca_key_password`, used to unlock the client_ca_key, if needed. - -### Provider Settings - -The leap client fetches provider information via json files from the server. -If you want to use that functionality please add your provider files the public/1/config directory. (1 being the current API version). - -Running ------------------------------ - - cd leap_web - bin/rails server - -You will find Leap Web running on `localhost:3000` - -Testing --------------------------------- - -To run all tests - - rake test - -To run an individual test: - - rake test TEST=certs/test/unit/client_certificate_test.rb - or - ruby -Itest certs/test/unit/client_certificate_test.rb - -Engines ---------------------- - -Leap Web includes some Engines. All things in `app` will overwrite the engine behaviour. You can clone the leap web repository and add your customizations to the `app` directory. Including leap_web as a gem is currently not supported. It should not require too much work though and we would be happy to include the changes required. - -If you have no use for one of the engines you can remove it from the Gemfile. Engines should really be plugins - no other engines should depend upon them. If you need functionality in different engines it should probably go into the toplevel. - -# Deployment # - -We strongly recommend using the LEAP platform for deploy. Most of the things documented here are automated as part of the platform. If you want to research how the platform deploys or work on your own mechanism this section is for you. - -These instructions are targeting a Debian GNU/Linux system. You might need to change the commands to match your own needs. - -## Server Preperation ## - -### Dependencies ## - -The following packages need to be installed: - -* git -* ruby1.9 -* rubygems1.9 -* couchdb (if you want to use a local couch) - -### Setup Capistrano ### - -We use puppet to deploy. But we also ship an untested config/deploy.rb.example. Edit it to match your needs if you want to use capistrano. - -run `cap deploy:setup` to create the directory structure. - -run `cap deploy` to deploy to the server. - -## Customized Files ## - -Please make sure your deploy includes the following files: - -* public/1/config/*.json (see Provider Settings section) -* config/couchdb.yml - -## Couch Security ## - -We recommend against using an admin user for running the webapp. To avoid this couch design documents need to be created ahead of time and the auto update mechanism needs to be disabled. -Take a look at test/setup_couch.sh for an example of securing the couch. - -## Design Documents ## - -After securing the couch design documents need to be deployed with admin permissions. There are two ways of doing this: - * rake couchrest:migrate_with_proxies - * dump the documents as files with `rake couchrest:dump` and deploy them - to the couch by hand or with the platform. - -### CouchRest::Migrate ### - -The before_script block in .travis.yml illustrates how to do this: - - mv test/config/couchdb.yml.admin config/couchdb.yml # use admin privileges - bundle exec rake couchrest:migrate_with_proxies # run the migrations - bundle exec rake couchrest:migrate_with_proxies # looks like this needs to run twice - mv test/config/couchdb.yml.user config/couchdb.yml # drop admin privileges - -### Deploy design docs from CouchRest::Dump ### - -First of all we get the design docs as files: - - # put design docs in /tmp/design - bundle exec rake couchrest:dump - -Then we add them to files/design in the site_couchdb module in leap_platform so they get deployed with the couch. You could also upload them using curl or sth. similar. - -# Troubleshooting # - -Here are some less common issues you might run into when installing Leap Web. - -## Cannot find Bundler ## - -### Error Messages ### - -`bundle: command not found` - -### Solution ### - -Make sure bundler is installed. `gem list bundler` should list `bundler`. -You also need to be able to access the `bundler` executable in your PATH. - -## Outdated version of rubygems ## - -### Error Messages ### - -`bundler requires rubygems >= 1.3.6` - -### Solution ### - -`gem update --system` will install the latest rubygems - -## Missing development tools ## - -Some required gems will compile C extensions. They need a bunch of utils for this. - -### Error Messages ### - -`make: Command not found` - -### Solution ### - -Install the required tools. For linux the `build-essential` package provides most of them. For Mac OS you probably want the XCode Commandline tools. - -## Missing libraries and headers ## - -Some gem dependencies might not compile because they lack the needed c libraries. - -### Solution ### - -Install the libraries in question including their development files. - - @@ -1,8 +1,16 @@ @title = 'LEAP Platform for Service Providers' +@summary = "The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment." @nav_title = 'Provider Platform' -@toc = false +@this.toc = false -The *LEAP Platform* is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment. Its goal is to make it as painless as possible for sysadmins to deploy and maintain a service provider's infrastructure for secure communication. +Its goal is to make it as painless as possible for sysadmins to deploy and maintain a service provider's infrastructure for secure communication. + +**REQUIREMENTS** -- Before you begin, make sure you meet these requirements: + +* *Debian Servers*: Servers that you deploy to must be running **Debian Jessie**, and no other distribution or version. +* *Real or Paravirtualized Servers*: Servers must be real machines or paravirtualized VMs (e.g. KVM, Xen, OpenStack, AWS, Google Compute). OS level virtualization is not supported (e.g. OpenVZ, Linux-VServer, etc), nor are system emulators (VirtualBox, QEMU, etc). +* *Your Workstation*: You must have a Linux or Mac computer to deploy from (this can be a headless machine with no GUI). Windows is not supported (Cygwin would probably work, but is untested). +* *Your Own Domain*: You must own a domain name. Before your provider can be put into production, you will need to make modifications to the DNS for the provider's domain. The LEAP Platform consists of three parts, detailed below: @@ -24,7 +32,9 @@ LEAP's platform recipes are distributed as a git repository: `https://leap.se/gi The provider instance --------------------- -A provider instance is a directory tree (typically tracked in git) containing all the configurations for a service provider's infrastructure. A provider instance primarily consists of: +A provider instance is a directory tree (typically tracked in git) containing all the configurations for a service provider's infrastructure. A provider instance **lives on your workstation**, not on the server. + +A provider instance primarily consists of: * A pointer to the platform recipes. * A global configuration file for the provider. @@ -41,13 +51,17 @@ A minimal provider instance directory looks like this: ├── files/ # keys, certificates, and other files. └── users/ # public key information for privileged sysadmins. - A provider instance directory contains everything needed to manage all the servers that compose a provider's infrastructure. Because of this, any versioning tool and development work-flow can be used to manage your provider instance. The `leap` command line tool ---------------------------- -The `leap` [command line tool](commands) is used by sysadmins to manage everything about a service provider's infrastructure. Except when creating an new provider instance, `leap` is run from within the directory tree of a provider instance. +The `leap` [command line tool](commands) is used by sysadmins to manage everything about a service provider's infrastructure. + +Keep these rules in mind: + +* `leap` is run on your workstation: The `leap` command is always run locally on your workstation, never on a server you are deploying to. +* `leap` is run from within a provider instance: The `leap` command requires that the current working directory is a valid provider instance, except when running `leap new` to create a new provider instance. The `leap` command line has many capabilities, including: @@ -55,31 +69,14 @@ The `leap` command line has many capabilities, including: * Manage keys and certificates. * Query information about the node configurations. -Traditional system configuration automation systems, like [Puppet](https://puppetlabs.com/puppet/puppet-open-source/) or [Chef](http://www.opscode.com/chef/), deploy changes to servers using a pull method. Each server pulls a manifest from a central master server and uses this to alter the state of the server. - -Instead, the `leap` tool uses a masterless push method: The sysadmin runs `leap deploy` from the provider instance directory on their desktop machine to push the changes out to every server (or a subset of servers). LEAP still uses Puppet, but there is no central master server that each node must pull from. +Everything about your provider is managed by editing JSON configuration files and running `leap` commands. -One other significant difference between LEAP and typical system automation is how interactions among servers are handled. Rather than store a central database of information about each server that can be queried when a recipe is applied, the `leap` command compiles static representation of all the information a particular server will need in order to apply the recipes. In compiling this static representation, `leap` can use arbitrary programming logic to query and manipulate information about other servers. - -These two approaches, masterless push and pre-compiled static configuration, allow the sysadmin to manage a set of LEAP servers using traditional software development techniques of branching and merging, to more easily create local testing environments using virtual servers, and to deploy without the added complexity and failure potential of a master server. - -The `leap` command line tool is distributed as a git repository: `https://leap.se/git/leap_cli`. It can be installed with `sudo gem install leap_cli`. - -Tip: With rubygems, you can always specify the gem version as the first argument to any executable installed by rubygems. For example: - - sudo gem install leap_cli --version 1.6.2 - sudo gem install leap_cli --version 1.7.2 - leap _1.6.2_ --version - => leap 1.6.2, ruby 2.1.2 - leap _1.7.2_ --version - => leap 1.7.2, ruby 2.1.2 - -Getting started +What is next? ---------------------------------- We recommend reading the platform documentation in the following order: -1. [Quick start tutorial](tutorials/quick-start). -2. [Platform Guide](platform/guide). -3. [Configuration format](platform/config). -4. The `leap` [command reference](platform/commands). +1. [[quick-start]] +2. [[getting-started]] +3. [[platform/guide]] + diff --git a/doc/guide/commands.md b/doc/guide/commands.md index eaacc8d5..2ddacb83 100644 --- a/doc/guide/commands.md +++ b/doc/guide/commands.md @@ -1,5 +1,5 @@ @title = 'Command Line Reference' -@summary = "A copy of leap --help" +@summary = 'A copy of leap --help' The command "leap" can be used to manage a bevy of servers running the LEAP platform from the comfort of your own home. @@ -7,7 +7,7 @@ The command "leap" can be used to manage a bevy of servers running the LEAP plat # Global Options * `--log FILE` -Override default log file +Override default log file. Default Value: None * `-v|--verbose LEVEL` @@ -15,19 +15,22 @@ Verbosity level 0..5 Default Value: 1 * `--[no-]color` -Disable colors in output +Disable colors in output. -* `--debug` -Enable debugging library (leap_cli development only) +* `-d|--debug` +Print full stack trace for exceptions and load `debugger` gem if installed. + +* `--force` +Like --yes, but also skip prompts that are potentially dangerous to skip. * `--help` Show this message * `--version` -Display version number and exit +Display version number and exit. * `--yes` -Skip prompts and assume "yes" +Skip prompts and assume "yes". # leap add-user USERNAME @@ -66,19 +69,51 @@ See see what values are used in the generation of the certificates (like name an Creates a CSR for use in buying a commercial X.509 certificate. -Unless specified, the CSR is created for the provider's primary domain. The properties used for this CSR come from `provider.ca.server_certificates`. +Unless specified, the CSR is created for the provider's primary domain. The properties used for this CSR come from `provider.ca.server_certificates`, but may be overridden here. **Options** +* `--bits BITS` +Override default certificate bit length +Default Value: None + +* `--country|-C COUNTRY` +Set C in distinguished name. +Default Value: None + +* `--digest DIGEST` +Override default signature digest +Default Value: None + * `--domain DOMAIN` Specify what domain to create the CSR for. -Unless specified, the CSR is created for the provider's primary domain. The properties used for this CSR come from `provider.ca.server_certificates`. +Unless specified, the CSR is created for the provider's primary domain. The properties used for this CSR come from `provider.ca.server_certificates`, but may be overridden here. +Default Value: None + +* `--email EMAIL` +Set emailAddress in distinguished name. +Default Value: None + +* `--locality|-L LOCALITY` +Set L in distinguished name. +Default Value: None + +* `--organization|-O ORGANIZATION` +Override default O in distinguished name. +Default Value: None + +* `--state|--ST STATE` +Set ST in distinguished name. +Default Value: None + +* `--unit|--OU UNIT` +Set OU in distinguished name. Default Value: None ## leap cert dh -Creates a Diffie-Hellman parameter file, needed for forward secret OpenVPN ciphers. You don't need this file if you don't provide the VPN service. +Creates a Diffie-Hellman parameter file, needed for forward secret OpenVPN ciphers. @@ -112,9 +147,27 @@ Compiles node configuration files into hiera files used for deployment. +## leap compile firewall + +Prints a list of firewall rules. These rules are already implemented on each node, but you might want the list of all rules in case you also have a restrictive network firewall. + + + +## leap compile hosts + +Print entries suitable for an /etc/hosts file, useful for testing your provider. + + + +## leap compile provider.json + +Compile provider.json bootstrap files for your provider. + + + ## leap compile zone -Compile a DNS zone file for your provider. +Prints a DNS zone file for your provider. Default Command: all @@ -127,9 +180,26 @@ Database commands. ## leap db destroy [FILTER] -Destroy all the databases. If present, limit to FILTER nodes. +Destroy one or more databases. If present, limit to FILTER nodes. For example `leap db destroy --db sessions,tokens testing`. + + +**Options** +* `--db DATABASES` +Comma separated list of databases to destroy (no space). Use "--db all" to destroy all databases. +Default Value: None + +* `--user USERS` +Comma separated list of usernames. The storage databases for these user(s) will be destroyed. +Default Value: None + + +# leap debug FILTER + +Output debug information. + +The FILTER can be the name of a node, service, or tag. # leap deploy FILTER @@ -149,18 +219,21 @@ Default Value: None * `--tags TAG[,TAG]` Specify tags to pass through to puppet (overriding the default). -Default Value: leap_base,leap_service +Default Value: None * `--dev` Development mode: don't run 'git submodule update' before deploy. +* `--downgrade` +Allows deploy to run with an older platform version. + * `--fast` Makes the deploy command faster by skipping some slow steps. A "fast" deploy can be used safely if you recently completed a normal deploy. * `--force` Deploy even if there is a lockfile. -* `--[no-]sync` +* `--sync` Sync files, but don't actually apply recipes. @@ -170,9 +243,9 @@ Manipulate and query environment information. The 'environment' node property can be used to isolate sets of nodes into entirely separate environments. A node in one environment will never interact with a node from another environment. Environment pinning works by modifying your ~/.leaprc file and is dependent on the absolute file path of your provider directory (pins don't apply if you move the directory) -## leap env ls +## leap env ls [ENVIRONMENT] -List the available environments. The pinned environment, if any, will be marked with '*'. +List the available environments. The pinned environment, if any, will be marked with '*'. Will also set the pin if run with an environment argument. @@ -213,6 +286,26 @@ Gets help for the application or its commands. Can also list the commands in a w List commands one per line, to assist with shell completion +# leap history FILTER + +Display recent deployment history for a set of nodes. + +The FILTER can be the name of a node, service, or tag. + +**Options** + +* `--ip IPADDRESS` +Override the default SSH IP address. +Default Value: None + +* `--port PORT` +Override the default SSH port. +Default Value: None + +* `--last` +Show last deploy only + + # leap inspect FILE Prints details about a file. Alternately, the argument FILE can be the name of a node, service or tag. @@ -275,6 +368,13 @@ Starts up the virtual machine(s) +**Options** + +* `--basebox BASEBOX` +The basebox to use. This value is passed to vagrant as the `config.vm.box` option. The value here should be the name of an installed box or a shorthand name of a box in HashiCorp's Atlas. +Default Value: LEAP/jessie + + ## leap local status [FILTER] Print the status of local virtual machine(s) @@ -293,6 +393,17 @@ Log in to the specified node with an interactive shell using mosh (requires node +**Options** + +* `--port SSH_PORT` +Override default SSH port used when trying to connect to the server. Same as `--ssh "-p SSH_PORT"`. +Default Value: None + +* `--ssh arg` +Pass through raw options to ssh (e.g. `--ssh '-F ~/sshconfig'`). +Default Value: None + + # leap new DIRECTORY Creates a new provider instance in the specified directory, creating it if necessary. @@ -376,6 +487,18 @@ Removes all the files related to the node named NAME. +# leap scp FILE1 FILE2 + +Secure copy from FILE1 to FILE2. Files are specified as NODE_NAME:FILE_PATH. For local paths, omit "NODE_NAME:". + + + +**Options** + +* `-r` +Copy recursively + + # leap ssh NAME Log in to the specified node with an interactive shell. @@ -384,12 +507,12 @@ Log in to the specified node with an interactive shell. **Options** -* `--port arg` -Override ssh port for remote host +* `--port SSH_PORT` +Override default SSH port used when trying to connect to the server. Same as `--ssh "-p SSH_PORT"`. Default Value: None * `--ssh arg` -Pass through raw options to ssh (e.g. --ssh '-F ~/sshconfig') +Pass through raw options to ssh (e.g. `--ssh '-F ~/sshconfig'`). Default Value: None @@ -405,9 +528,9 @@ Creates files needed to run tests. -## leap test run +## leap test run [FILTER] -Run tests. +Run the test suit on FILTER nodes. @@ -417,3 +540,20 @@ Run tests. Continue over errors and failures (default is --no-continue). Default Command: run + +# leap tunnel [LOCAL_PORT:]NAME:REMOTE_PORT + +Creates an SSH port forward (tunnel) to the node NAME. REMOTE_PORT is the port on the remote node that the tunnel will connect to. LOCAL_PORT is the optional port on your local machine. For example: `leap tunnel couch1:5984`. + + + +**Options** + +* `--port SSH_PORT` +Override default SSH port used when trying to connect to the server. Same as `--ssh "-p SSH_PORT"`. +Default Value: None + +* `--ssh arg` +Pass through raw options to ssh (e.g. --ssh '-F ~/sshconfig'). +Default Value: None + diff --git a/doc/guide/config.md b/doc/guide/config.md index be67e6bd..bcea26c4 100644 --- a/doc/guide/config.md +++ b/doc/guide/config.md @@ -1,39 +1,69 @@ @title = "Configuration Files" -@summary = "How to edit configuration files." +@summary = "Understanding and editing the configuration files." Files ------------------------------------------- -Here are a list of some of the common files that make up a provider. Except for Leapfile and provider.json, the files are optional. Unless otherwise specified, all file names are relative to the 'provider directory' root (where the Leapfile is). - -`Leapfile` -- If present, this file tells `leap` that the directory is a provider directory. This file is usually empty, but can contain global options. - -`~/.leaprc` -- Evaluated the same as Leapfile, but not committed to source control. - -`provider.json` -- Global options related to this provider. - -`provider.ENVIRONMENT.json` -- Global options for the provider that are applied to only a single environment. - -`common.json` -- All nodes inherit from this file. - -`secrets.json` -- An automatically generated file that contains any randomly generated strings needed in order to deploy. These strings are often secret and should be protected, although any need for a random string or number that is remembered will produce another entry in this file. This file is automatically generated and refreshed each time you run `leap compile` or `leap deploy`. If an entry is no longer needed, it will get removed. If you want to change a secret, you can remove this file and have it regenerated, or remove the particular line item and just those items will be created anew. - -`facts.json` -- If some of your servers are running on AWS or OpenStack, you will need to discover certain properties about how networking is configured on these machines in order for a full deploy to work. In these cases, make sure to run `leap facts update` to periodically regenerate the facts.json file. - -`nodes/NAME.json` -- The configuration file for node called NAME. - -`services/SERVICE.json` -- The properties in this configuration file are applied to any node that includes SERVICE in its `services` property. - -`services/SERVICE.ENVIRONMENT.json` -- The properties in this configuration file are applied to any node that includes SERVICE in its services and has environment equal to ENVIRONMENT. - -`services/TAG.json` -- The properties in this configuration file are applied to any node that has includes TAG in its `tags` property. - -`services/TAG.ENVIRONMENT.json` -- The properties in this configuration file are applied to any node that has includes TAG in its `tags` property and has `environment` property equal to ENVIRONMENT. - -`files/*` -- Various static files used by the platform (e.g. keys, certificates, webapp customization, etc). - -`users/USER/` -- A directory that stores the public keys of the sysadmin with name USER. This person will have root access to all the servers. - +Here are a list of some of the common files that make up a provider. Except for `Leapfile` and `provider.json`, the files are optional. Unless otherwise specified, all file names are relative to the 'provider directory' root (where the Leapfile is). + +<table class="table table-striped"> +<tr> + <td><code>Leapfile</code></td> + <td>If present, this file tells <code>leap</code> that the directory is a provider directory. This file is usually empty, but can contain global options.</td> +</tr> +<tr> + <td><code>~/.leaprc</code></td> + <td>Evaluated the same as Leapfile, but not committed to source control.</td> +</tr> +<tr> + <td><code>provider.json</code></td> + <td>Global options related to this provider. See [[provider-configuration]].</td> +</tr> +<tr> + <td><code>provider.ENVIRONMENT.json</code></td> + <td>Global options for the provider that are applied to only a single environment.</td> +</tr> +<tr> + <td><code>nodes/NAME.json</code></td> + <td>The configuration file for node called NAME.</td> +</tr> +<tr> + <td><code>common.json</code></td> + <td>All nodes inherit from this file. In other words, any options that appear in <code>common.json</code> will be added as default values to each node configuration, value that can be locally overridden.</td> +</tr> +<tr> + <td><code>services/SERVICE.json</code></td> + <td>The properties in this configuration file are applied to any node that includes SERVICE in its <code>services</code> property.</td> +</tr> +<tr> + <td><code>services/SERVICE.ENVIRONMENT.json</code></td> + <td>The properties in this configuration file are applied to any node that includes SERVICE in its services and has environment equal to ENVIRONMENT.</td> +</tr> +<tr> + <td><code>tags/TAG.json</code></td> + <td>The properties in this configuration file are applied to any node that has includes TAG in its <code>tags</code> property.</td> +</tr> +<tr> + <td><code>tags/TAG.ENVIRONMENT.json</code></td> + <td>The properties in this configuration file are applied to any node that has includes TAG in its <code>tags</code> property and has <code>environment</code> property equal to ENVIRONMENT.</td> +</tr> +<tr> + <td><code>secrets.json </code></td> + <td>An automatically generated file that contains any randomly generated strings needed in order to deploy. These strings are often secret and should be protected, although any need for a random string or number that is remembered will produce another entry in this file. This file is automatically generated and refreshed each time you run <code>leap compile</code> or <code>leap deploy</code>. If an entry is no longer needed, it will get removed. If you want to change a secret, you can remove this file and have it regenerated, or remove the particular line item and just those items will be created anew.</td> +</tr> +<tr> + <td><code>facts.json</code></td> + <td>If some of your servers are running on AWS or OpenStack, you will need to discover certain properties about how networking is configured on these machines in order for a full deploy to work. In these cases, make sure to run <code>leap facts update</code> to periodically regenerate the facts.json file.</td> +</tr> +<tr> + <td><code>files/*</code></td> + <td>Various static files used by the platform (e.g. keys, certificates, webapp customization, etc). In general, only generated files and files used to customize the provider (such as images) live in the <code>files</code> directory.</td> +</tr> +<tr> + <td><code>users/USER/</code></td> + <td>A directory that stores the public keys of the sysadmin with name USER. This person will have root access to all the servers.</td> +</tr> +</table> Leapfile ------------------------------------------- @@ -51,8 +81,10 @@ Platform options: Vagrant options: +* `@vagrant_provider`. Changes the default vagrant provider ("virtualbox"). For example, `@vagrant_provider = "libvirt"`. * `@vagrant_network`. Allows you to override the default network used for local nodes. It should include a netmask like `@vagrant_network = '10.0.0.0/24'`. * `@custom_vagrant_vm_line`. Insert arbitrary text into the auto-generated Vagrantfile. For example, `@custom_vagrant_vm_line = "config.vm.boot_mode = :gui"`. +* `@vagrant_basebox` allows specifying a different basebox as the default one. For example, `@vagrant_basebox = "LEAP/jessie"`. Logging options: @@ -69,7 +101,7 @@ All configuration files, other than `Leapfile`, are in the JSON format. For exam "key2": "value2" } -Keys should match `/[a-z0-9_]/` +Keys should match `/[a-z0-9_]/` and must be in double quotes. Unlike traditional JSON, comments are allowed. If the first non-whitespace characters are `//` then the line is treated as a comment. @@ -141,6 +173,71 @@ The `provider_base` directory is under the `leap_platform` specified in the file To see all the variables a node has inherited, you could run `leap inspect willamette`. +### Inheritance rules + +Suppose you have a node configuration `mynode.json`: + + { + "tags": "production", + "simple_value": 100, + "replaced_array": ["dolphin", "kangaroo"], + "+add_array": ["red", "black"], + "-subtract_array": ["bitter"], + "converted_to_array": "not_array_element", + "!override": ["insist on this value"], + "hash": { + "key1": 1, + "key2": 2 + } + } + +And a file `tags/production.json`: + + { + "simple_value": 99999, + "replaced_array": ["zebra"], + "add_array": ["green], + "subtract_array": ["bitter", "sweet", "salty"], + "converted_to_array": ["array_element"], + "override": "this value will be overridden", + "hash": { + "key1": "one" + } + } + +In this scenario, `mynode.json` will inherit from `production.json`. The output of this inheritance will be: + + { + "tags": "production", + "simple_value": 100, + "replaced_array": ["dolphin", "kangaroo"], + "add_array": ["red", "black", "green"], + "subtract_array": ["sweet", "salty"], + "converted_to_array": ["not_array_element", "array_element"], + "override": ["insist on this value"], + "hash": { + "key1": 1, + "key2": 2 + } + +The rules for inheritance (where 'old' refers to the parent, and 'new' refers to the child): + +* Simple values (strings, numbers, boolean): + * Replace the old value with the new value. +* Array values: + * Two arrays: replace the old array with the new array. + * One array and one simple value: add the simple value to the array. + * If property name is prefixed with "+": merge the old and new arrays. + * If property name is prefixed with "-": subtract new array from old array. +* Hash values: + * Hashes are always merged (the result includes the keys of both hashes). If there is a key in common, the new one overrides the old one. +* Mismatch: + * Although you can mix arrays and simple values, you cannot mix arrays with hashes or hashes with simple values. If you attempt to do so, it will fail to compile and give you an error message. +* Override: + * If property name is prefixed with "!": then ensure that new value is always used, regardless of old value. In this case, the override takes precedence over type checking, so you will never get a type mismatch. + +NOTE: special property name prefixes, like "+", "-", or "!", are not included in the property name. These prefixes determine the merge strategy, but are stripped out when compiling the resulting JSON file. + Common configuration options ---------------------------------------- diff --git a/doc/guide/domains.md b/doc/guide/domains.md new file mode 100644 index 00000000..914bce33 --- /dev/null +++ b/doc/guide/domains.md @@ -0,0 +1,129 @@ +@title = "Domains" +@summary = "How to handle domain names and integrating LEAP with existing services." +@toc = true + +Overview +-------------------------------- + +Deploying LEAP can start to get very tricky when you need to integrate LEAP services with an existing domain that you already use or which already has users. Most of this complexity is unavoidable, although there are a few things we plan to do in the future to make this a little less painful. + +Because integration with legacy systems is an advanced topic, we recommend that you begin with a new domain. Once everything works and you are comfortable with your LEAP-powered infrastructure, you can then contemplate integrating with your existing domain. + +### Definitions + +**provider domain** + +This is the main domain used to identify the provider. The **provider domain** is what the user enters in the Bitmask client. e.g. `example.org`. The full host name of every node in your provider infrastructure will use the **provider domain** (e.g. `dbnode.example.org`). + +In order for the Bitmask client to get configured for use with a provider, it must be able to find the `provider.json` bootstrap file at the root of the **provider domain**. This is not needed if the Bitmask client is "pre-seeded" with the provider's information (these providers show up in a the initial list of available providers). + +**webapp domain** + +This is the domain that runs the leap_web application that allows users to register accounts, create help tickets, etc. e.g. `example.org` or `user.example.org`. The **webapp domain** defaults to the **provider domain** unless it is explicitly configured separately. + +**API domain** + +This is the domain that the provider API runs on. Typically, this is set automatically and you never need to configure it. The user should never be aware of this domain. e.g. `api.example.org`. The Bitmask client discovers this API domain by reading it from the `provider.json` file it grabs from the **provider domain**. + +**mail domain** + +This is the domain used for mail accounts, e.g. `username@example.org`. Currently, this is always the **provider domain**, but it may be independently configurable in the future. + +Generating a zone file +----------------------------------- + +Currently, the platform does not include a dedicated `dns` service type, so you need to have your own setup for DNS. You can generate the appropriate configuration options with this command: + + leap compile zone + +A single domain +------------------------------- + +The easy approach is to use a single domain for **provider domain**, **webapp domain**, and **email domain**. This will install the webapp on the **provider domain**, which means that this domain must be a new one that you are not currently using for anything. + +To configure a single domain, just set the domain in `provider.json`: + + { + "domain": "example.org" + } + +If you have multiple environments, you can specify a different **provider domain** for each environment. For example: + +`provider.staging.json` + + { + "domain": "staging.example.org" + } + +A separate domain for the webapp +-------------------------------------- + +It is possible make the **webapp domain** different than the **provider domain**. This is needed if you already have a website running at your **provider domain**. + +In order to put webapp on a different domain, you must take two steps: + +1. You must configure `webapp.domain` for nodes with the `webapp` service. +2. You must make the compiled `provider.json` available at the root of the **provider domain**. + +NOTE: This compiled provider.json is different than the provider.json that you edit and lives in the root of the provider directory. + +### Step 1. Configuring `webapp.domain` + +In `services/webapp.json`: + + { + "webapp": { + "domain": "user.example.org" + } + } + +### Step 2. Putting the compiled `provider.json` in place + +Generate the compiled `provider.json`: + + leap compile provider.json + = created files/web/bootstrap/ + = created files/web/bootstrap/README + = created files/web/bootstrap/production/ + = created files/web/bootstrap/production/provider.json + = created files/web/bootstrap/production/htaccess + = created files/web/bootstrap/staging/ + = created files/web/bootstrap/staging/provider.json + = created files/web/bootstrap/staging/htaccess + +This command compiles a separate `provider.json` for each environment, or "default" if you don't have an environment. In the example above, there is an environment called "production" and one called "staging", but your setup will probably differ. + +The resulting `provider.json` file must then be put at the root URL of your **provider domain** for the appropriate environment. + +There is one additional complication: currently, the Bitmask client tests for compatibility using some HTTP headers on the `/provider.json` response. This is will hopefully change in the future, but for now you need to ensure the right headers are set in the response. The included file `htaccess` has example directives for Apache, if that is what you use. + +This step can be skipped if you happen to use the `static` service to deploy an `amber` powered static website to **provider domain**. In this case, the correct `provider.json` will be automatically put into place. + +Integrating with existing email system +----------------------------------------- + +If your **mail domain** already has users from a legacy email system, then things get a bit complicated. In order to be able to support both LEAP-powered email and legacy email on the same domain, you need to follow these steps: + +1. Modify the LEAP webapp so that it does not create users with the same name as users in the legacy system. +2. Configure your legacy MX servers to forward mail that they cannot handle to the LEAP MX servers, or vice versa. + +### Step 1. Modify LEAP webapp + +In order to modify the webapp to respect the usernames already reserved by your legacy system, you need to modify the LEAP webapp code. The easiest way to do this is to create a custom gem that modifies the behavior of the webapp. + +For this example, we will call our custom gem `reserve_usernames`. + +This gem can live in one of two places: + +(1) You can fork the project leap_web and put the gem in `leap_web/vendor/gems/reserve_usernames`. Then, modify `Gemfile` and add the line `gem 'common_languages', :path => 'vendor/gems/reserve_usernames'` + +(2) Alternately, you can put the gem in the local provider directory `files/webapp/gems/reserve_username`. This will get synced to the webapp servers when you deploy and put in `/srv/leap/webapp/config/customization` where it will get automatically loaded by the webapp. + +What should the gem `reserve_usernames` look like? There is an example available here: https://leap.se/git/reserved_usernames.git + +This example gem uses ActiveResource to communicate with a remote REST API for creating and checking username reservations. This ensures that both the legacy system and the LEAP system use the same namespace. Alternately, you could write a gem that checks the legacy database directly. + +### Step 2. Configure MX servers + +To be written. + diff --git a/doc/guide/getting-started.md b/doc/guide/getting-started.md new file mode 100644 index 00000000..6236cba0 --- /dev/null +++ b/doc/guide/getting-started.md @@ -0,0 +1,145 @@ +@title = 'Getting Started' +@summary = 'An overview of the LEAP Platform' +@toc = true + + +Sensitive files +---------------------------------------------- + +Some files in your provider directory are very sensitive. Leaking these files will compromise your provider. + +Super sensitive and irreplaceable: + +* `files/ca/*.key` -- the private keys for the client and server CAs. +* `files/cert/*.key` -- the private key(s) for the commercial certificate for your domain(s). + +Sensitive, but can be erased and regenerated automatically: + +* `secrets.json` -- various random secrets, such as passwords for databases. +* `files/nodes/*/*.key` -- the private key for each node. +* `hiera/*.yaml` -- hiera file contains a copy of the private key of the node. + +Also, each sysadmin has one or more public ssh keys in `users/*/*_ssh.pub`. Typically, you will want to keep these public keys secure as well. + +See [[keys-and-certificates]] for more information. + +Useful commands +------------------------------------------- + +Here are a few useful `leap` commands: + +* `leap help [COMMAND]` -- get help on COMMAND. +* `leap history [FILTER]` -- show the recent deployment history for the selected nodes. +* `leap ssh web1` -- SSH into node web1 (requires `leap node init web1` first). +* `leap list [FILTER]` -- list the selected nodes. + * `leap list production` -- list only those nodes with the tag 'production' + * `leap list --print ip_address` -- list a particular attribute of all nodes. + +See the full [[commands]] for more information. + +Node filters +------------------------------------------- + +Many of the `leap` commands take a "node filter". You can use a node filter to target a command at one or more nodes. + +A node filter consists of one or more keywords, with an optional "+" before each keyword. + +* keywords can be a node name, a service type, or a tag. +* the "+" before the keyword constructs an AND condition +* otherwise, multiple keywords together construct an OR condition + +Examples: + +* `leap list openvpn` -- list all nodes with service openvpn. +* `leap list openvpn +production` -- only nodes of service type openvpn AND tag production. +* `leap deploy webapp openvpn` -- deploy to all webapp OR openvpn nodes. +* `leap node init ostrich` -- just init the node named ostrich. + +See the full [[commands]] for more information. + +Tracking the provider directory in git +------------------------------------------ + +You should commit your provider changes to your favorite VCS whenever things change. This way you can share your configurations with other admins, all they have to do is to pull the changes to stay up to date. Every time you make a change to your provider, such as adding nodes, services, generating certificates, etc. you should add those to your VCS, commit them and push them to where your repository is hosted. + +Note that your provider directory contains secrets, such as private key material and passwords. You do not want to have those passwords readable by the world, so make sure that wherever you are hosting your repository, it is not public for the world to read. + +If you have a post-commit hook that emails the changes to contributors, you may want to exclude diffs for files that might have sensitive secrets. For example, create a `.gitattributes` file with: + + # No diff, no email for key files + *.key -diff + *.pem -diff + + # Discard diff for secrets.json + secrets.json -diff + + # No diff for hiera files, they contain passwords + hiera/* -diff + + +Editing JSON configuration files +-------------------------------------- + +All the settings that compose your provider are stored in JSON files. + +At a minimum, you will need at least two configuration files: + +* `provider.json` -- general settings for you provider. +* `nodes/NAME.json` -- configuration file for node called "NAME". + +There are a few required properties in provider.json: + + { + "domain": "example.org", + "name": "Example", + "contacts": { + "default": "email1@example.org" + } + } + +See [[provider-configuration]] for more details. + +For node configuration files, there are two required properties: + + { + "ip_address": "1.1.1.1", + "services": ["openvpn"] + } + +See [[services]] for details on what servers are available, and see [[config]] details on how configuration files work. + +How does it work under the hood? +-------------------------------------------- + +You don't need to know any of the details of what happens "under the hood" in order to use the LEAP platform. However, if you are curious as to what is going on, here is a quick primer. + +First, some background terminology: + +* **puppet**: Puppet is a system for automating deployment and management of servers (called nodes). +* **hiera files**: In puppet, you can use something called a 'hiera file' to seed a node with a few configuration values. In LEAP, we go all out and put *every* configuration value needed for a node in the hiera file, and automatically compile a custom hiera file for each node. + +When you run `leap deploy`, a bunch of things happen, in this order: + +1. **Compile hiera files**: The hiera configuration file for each node is compiled in YAML format and saved in the directory `hiera`. The source material for this hiera file consists of all the JSON configuration files imported or inherited by the node's JSON config file. +* **Copy required files to node**: All the files needed for puppet to run are rsync'ed to each node. This includes the entire leap_platform directory, as well as the node's hiera file and other files needed by puppet to set up the node (keys, binary files, etc). +* **Puppet is run**: Once the node is ready, leap connects to the node via ssh and runs `puppet apply`. Puppet is applied locally on the node, without a daemon or puppetmaster. + +You can run `leap -v2 deploy` to see exactly what commands are being executed. + +This mode of operation is fundamentally different from how puppet is normally used: + +* There is no puppetmaster that all the servers take orders from, and there is no puppetd running in the background. +* Servers cannot dynamically query the puppetmaster for information about the other servers. +* There is a static representation for the state of every server that can be committed to git. + +There are advantages and disadvantages to the model that LEAP uses. We have found it very useful for our goal of having a common LEAP platform that many different providers can all use while still allowing providers to configure their unique infrastructure. + +We also find it very beneficial to be able to track the state of your infrastructure in git. + +Traditional system configuration automation systems, like [Puppet](https://puppetlabs.com/puppet/puppet-open-source/) or [Chef](http://www.opscode.com/chef/), deploy changes to servers using a pull method. Each server pulls a manifest from a central master server and uses this to alter the state of the server. + +Instead, the `leap` tool uses a masterless push method: The sysadmin runs `leap deploy` from the provider instance directory on their desktop machine to push the changes out to every server (or a subset of servers). LEAP still uses Puppet, but there is no central master server that each node must pull from. + +One other significant difference between LEAP and typical system automation is how interactions among servers are handled. Rather than store a central database of information about each server that can be queried when a recipe is applied, the `leap` command compiles static representation of all the information a particular server will need in order to apply the recipes. In compiling this static representation, `leap` can use arbitrary programming logic to query and manipulate information about other servers. + +These two approaches, masterless push and pre-compiled static configuration, allow the sysadmin to manage a set of LEAP servers using traditional software development techniques of branching and merging, to more easily create local testing environments using virtual servers, and to deploy without the added complexity and failure potential of a master server. diff --git a/doc/guide/keys-and-certificates.md b/doc/guide/keys-and-certificates.md index aef02ac6..a6862a6a 100644 --- a/doc/guide/keys-and-certificates.md +++ b/doc/guide/keys-and-certificates.md @@ -4,7 +4,7 @@ Working with SSH ================================ -Whenever the `leap` command nees to push changes to a node or gather information from a node, it tunnels this command over SSH. Another way to put this: the security of your servers rests entirely on SSH. Because of this, it is important that you understand how `leap` uses SSH. +Whenever the `leap` command needs to push changes to a node or gather information from a node, it tunnels this command over SSH. Another way to put this: the security of your servers rests entirely on SSH. Because of this, it is important that you understand how `leap` uses SSH. SSH related files ------------------------------- @@ -21,7 +21,7 @@ All of these files should be committed to source control. If you rename, remove, or add a node with `leap node [mv|add|rm]` the SSH key files and the `known_hosts` file will get properly updated. SSH and local nodes ------------------------------ +------------------- Local nodes are run as Vagrant virtual machines. The `leap` command handles SSH slightly differently for these nodes. @@ -33,6 +33,15 @@ Specifically, for local nodes: 2. `leap` entirely skips the checking of host keys when connecting with a local node. 3. `leap` adds the public Vagrant SSH key to the list of SSH keys for a user. The public Vagrant SSH key is a shared and insecure key that has root access to most Vagrant virtual machines. +To upgrade a SSH host key +------------------------------- + +Most servers will have more than one SSH host key. Sometimes, the server will have a better SSH host key than the one you have on file. In order to upgrade to the better SSH host key, simply re-run the init command: + + workstation$ leap node init NODE_NAME + +This will prompt you if you want to upgrade the SSH host key, but only if `leap` thinks that an upgrade is advisable. + When SSH host key changes ------------------------------- @@ -78,7 +87,7 @@ All keys matching 'userx/*_ssh.pub' will be usable. Removing sysadmin access -------------------------------- -Suppose you want to remove `userx` from having any further ssh access to the servers. Do this: +Suppose you want to remove `userx` from having any further SSH access to the servers. Do this: rm -r users/userx leap deploy @@ -192,3 +201,72 @@ If you want to add additional fields to the CSR, like country, city, or locality } If they are not present, the CSR will be created without them. + +Examine Certs +----------------- + +To see details about the keys and certs you can use `leap inspect` like so: + + $ leap inspect files/ca/ca.crt + + +Let's Encrypt certificate +========================= + +LEAP plans to integrate [Let's Encrypt](https://letsencrypt.org/) support, so it will be even easier to receive X.509 certificates that are accepted by all browsers. +Until we achieve this, here's a guide how to do this manually. + +Install the official acme client +-------------------------------- + +Log in to your webapp node + + server$ git clone https://github.com/letsencrypt/letsencrypt + server$ cd letsencrypt + server$ ./letsencrypt-auto --help + +Fetch cert +---------- + +Stop apache so the letsencrypt client can bind to port 80: + + server$ systemctl stop apache2 + +Fetch the certs + + server$ ./letsencrypt-auto certonly --standalone --email admin@$(hostname -d) -d $(hostname -d) -d api.$(hostname -d) -d $(hostname -f) -d nicknym.$(hostname -d) + +This will put the certs and keys into `/etc/letsencrypt/live/DOMAIN/`. + +Now, go to your workstation's provider configuration directory and copy the newly created files from the server to your local config. You will override existing files so please make a backup before proceeding, or use a version control system to track changes. + + workstation$ cd PATH_TO_PROVIDER_CONFIG + +Copy the Certificate + + workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/cert.pem files/cert/dev.pixelated-project.org.crt + +Copy the private key + + workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/privkey.pem files/cert/DOMAIN.key + +Copy the CA chain cert + + workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/fullchain.pem files/cert/DOMAIN.key + +Deploy the certs +---------------- + +Now you only need to deploy the certs + + workstation$ leap deploy + +This will put them into the right locations which are: + +- `/etc/x509/certs/leap_commercial.crt` for the certificate +- `/etc/x509/./keys/leap_commercial.key` for the private key +- `/usr/local/share/ca-certificates/leap_commercial_ca.crt` for the CA chain cert. + +Start apache2 again + + server$ systemctl start apache2 diff --git a/doc/guide/nodes.md b/doc/guide/nodes.md index cf225449..5135f3ba 100644 --- a/doc/guide/nodes.md +++ b/doc/guide/nodes.md @@ -1,106 +1,6 @@ @title = "Nodes" @summary = "Working with nodes, services, tags, and locations." -Node types -================================ - -Every node has one or more services that determines the node's function within your provider's infrastructure. - -When adding a new node to your provider, you should ask yourself four questions: - -* **many or few?** Some services benefit from having many nodes, while some services are best run on only one or two nodes. -* **required or optional?** Some services are required, while others can be left out. -* **who does the node communicate with?** Some services communicate very heavily with other particular services. Nodes running these services should be close together. -* **public or private?** Some services communicate with the public internet, while others only need to communicate with other nodes in the infrastructure. - -Brief overview of the services: - -* **webapp**: The web application. Runs both webapp control panel for users and admins as well as the REST API that the client uses. Needs to communicate heavily with `couchdb` nodes. You need at least one, good to have two for redundancy. The webapp does not get a lot of traffic, so you will not need many. -* **couchdb**: The database for users and user data. You can get away with just one, but for proper redundancy you should have at least three. Communicates heavily with `webapp`, `mx`, and `soledad` nodes. -* **soledad**: Handles the data syncing with clients. Typically combined with `couchdb` service, since it communicates heavily with couchdb. -* **mx**: Incoming and outgoing MX servers. Communicates with the public internet, clients, and `couchdb` nodes. -* **openvpn**: OpenVPN gateway for clients. You need at least one, but want as many as needed to support the bandwidth your users are doing. The `openvpn` nodes are autonomous and don't need to communicate with any other nodes. Often combined with `tor` service. -* **monitor**: Internal service to monitor all the other nodes. Currently, you can have zero or one `monitor` service defined. It is required that the monitor be on the webapp node. It was not designed to be run as a separate node service. -* **tor**: Sets up a tor exit node, unconnected to any other service. -* **dns**: Not yet implemented. - -Webapp ------------------------------------ - -The webapp node is responsible for both the user face web application and the API that the client interacts with. - -Some users can be "admins" with special powers to answer tickets and close accounts. To make an account into an administrator, you need to configure the `webapp.admins` property with an array of user names. - -For example, to make users `alice` and `bob` into admins, create a file `services/webapp.json` with the following content: - - { - "webapp": { - "admins": ["bob", "alice"] - } - } - -And then redeploy to all webapp nodes: - - leap deploy webapp - -By putting this in `services/webapp.json`, you will ensure that all webapp nodes inherit the value for `webapp.admins`. - -Services -================================ - -What nodes do you need for a provider that offers particular services? - -<table class="table table-striped"> -<tr> - <th>Node Type</th> - <th>VPN Service</th> - <th>Email Service</th> - <th>Notes</th> -</tr> -<tr> - <td>webapp</td> - <td>required</td> - <td>required</td> - <td></td> -</tr> -<tr> - <td>couchdb</td> - <td>required</td> - <td>required</td> -<td></td> -</tr> -<tr> - <td>soledad</td> - <td>not used</td> - <td>required</td> -<td></td> -</tr> -<tr> - <td>mx</td> - <td>not used</td> - <td>required</td> - <td></td> -</tr> -<tr> - <td>openvpn</td> - <td>required</td> - <td>not used</td> - <td></td> -</tr> -<tr> - <td>monitor</td> - <td>optional</td> - <td>optional</td> - <td>This service must be on the webapp node</td> -</tr> -<tr> - <td>tor</td> - <td>optional</td> - <td>optional</td> - <td></td> -</tr> -</table> - Locations ================================ diff --git a/doc/guide/provider-configuration.md b/doc/guide/provider-configuration.md new file mode 100644 index 00000000..08cfd1dd --- /dev/null +++ b/doc/guide/provider-configuration.md @@ -0,0 +1,79 @@ +@title = "Provider Configuration" +@summary = "Explore how to configure your provider." + +Required provider configuration +-------------------------------------- + +There are a few required settings in `provider.json`. At a minimum, you must have: + +* `domain`: defines the primary domain of the provider. This is the domain that users will type in when using the Bitmask client, although it is not necessarily the domain where users will visit if they sign up via the web application. If email is supported, all accounts will be `username@domain`. +* `name`: A brief title for this provider. It can be multiple words, but should not be too long. +* `contacts.default`: One or more email addresses for sysadmins. + +For example: + + { + "domain": "freerobot.org", + "name": "Freedom for Robots!", + "contacts": { + "default": "root@freerobot.org" + } + } + + +Recommended provider configuration +-------------------------------------- + +* `description`: A longer description of the provider, shown to the user when they register a new account through Bitmask client. +* `languages`: A list of language codes that should be enabled. +* `default_language`: The initial default language code. +* `enrollment_policy`: One of "open", "closed", or "invite". (invite not currently supported). + +For example: + + { + "description": "It is time for robots of the world to unite and throw of the shackles of servitude to our organic overlords.", + "languages": ["en", "de", "pt", "01"], + "default_language": "01", + "enrollman_policy": "open" + } + +For a full list of possible settings, you can use `leap inspect` to see how provider.json is evaluated after including the inherited defaults: + + $ leap inspect provider.json + +Configuring service levels +-------------------------------------- + +The `provider.json` file defines the available service levels for the provider. + +For example, in provider.json: + + "service": { + "default_service_level": "low", + "levels": { + "low": { + "description": "Entry level plan, with unlimited bandwidth and minimal storage quota.", + "name": "entry", + "storage": "10 MB", + "rate": { + "USD": 5, + "GBP": 3, + "EUR": 6 + } + }, + "full": { + "description": "Full plan, with unlimited bandwidth and higher quota." + "name": "full", + "storage": "5 GB", + "rate": { + "USD": 10, + "GBP": 6, + "EUR": 12 + } + } + } + } + } + +For a list of currency codes, see https://en.wikipedia.org/wiki/ISO_4217#Active_codes diff --git a/doc/services/couchdb.md b/doc/services/couchdb.md new file mode 100644 index 00000000..cc40dc32 --- /dev/null +++ b/doc/services/couchdb.md @@ -0,0 +1,159 @@ +@title = "couchdb" +@summary = "Data storage for all user data." + +Topology +------------------------ + +Required: + +* Nodes with `couchdb` service must also have `soledad` service, if email is enabled. + +Suggested: + +* Nodes with `couchdb` service communicate heavily with `webapp` and `mx`. + +`couchdb` nodes do not need to be reachable from the public internet, although the `soledad` service does require this. + +Configuration +---------------------------- + +### Nighly dumps + +You can do a nightly couchdb data dump by adding this to your node config: + + "couch": { + "backup": true + } + +Data will get dumped to `/var/backups/couchdb`. + +### Plain CouchDB + +BigCouch is not supported on Platform version 0.8 and higher: only plain CouchDB is possible. For earlier versions, you must do this in order to use plain CouchDB: + + "couch": { + "master": true, + "pwhash_alg": "pbkdf2" + } + +Various Tasks +------------------------------------------------- + +### Re-enabling blocked account + +When a user account gets destroyed from the webapp, there's still a leftover doc in the identities db so other people can't claim that account without an admin's intervention. You can remove this username reservation through the webapp. + +However, here is how you could do it manually, if you wanted to: + +grep the identities db for the email address: + + curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET http://127.0.0.1:5984/identities/_all_docs?include_docs=true|grep test_127@bitmask.net + +lookup "id" and "rev" to delete the doc: + + curl -s --netrc-file /etc/couchdb/couchdb.netrc -X DELETE 'http://127.0.0.1:5984/identities/b25cf10f935b58088f0d547fca823265?rev=2-715a9beba597a2ab01851676f12c3e4a' + +### How to find out which userstore belongs to which identity? + + /usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep testuser + + {"id":"665e004870ee17aa4c94331ff3ecb173","key":"665e004870ee17aa4c94331ff3ecb173","value":{"rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b"},"doc":{"_id":"665e004870ee17aa4c94331ff3ecb173","_rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b","user_id":"665e004870ee17aa4c94331ff3cd59eb","address":"testuser@example.org","destination":"testuser@example.org","keys": ... + +* search for the "user_id" field +* in this example testuser@example.org uses the database user-665e004870ee17aa4c94331ff3cd59eb + + +### How much disk space is used by a userstore + +Beware that this returns the uncompacted disk size (see http://wiki.apache.org/couchdb/Compaction) + + echo "`curl --netrc -s -X GET 'http://127.0.0.1:5984/user-dcd6492d74b90967b6b874100b7dbfcf'|json_pp|grep disk_size|cut -d: -f 2`/1024"|bc + + +Deprecated BigCouch Tasks +----------------------------------------- + +As of release 0.8, the LEAP platform no longer supports BigCouch. This information is kept here for historical reference. + +### Rebalance Cluster + +Bigcouch currently does not have automatic rebalancing. +It will probably be added after merging into couchdb. +If you add a node, or remove one node from the cluster, + +1. make sure you have a backup of all DBs ! + +1. put the webapp into [[maintenance mode => services/webapp#maintenance-mode]] + +1. Stop all services that access the database: + + ``` + workstation$ leap ssh soledad-nodes + server# /etc/init.d/soledad-server stop + + workstation$ leap ssh mx-node + server# /etc/init.d/postfix stop + server# /etc/init.d/leap-mx stop + + workstation$ leap ssh webapp + server# /etc/init.d/nickserver stop + ``` + + Alternately, you can create a temporary firewall rule to block access (run on couchdb server): + + ``` + server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT + ``` + +1. dump the dbs: + + ``` + cd /srv/leap/couchdb/scripts + time ./couchdb_dumpall.sh + ``` + +1. delete all dbs + +1. shut down old node + +1. check the couchdb members + + ``` + curl -s —netrc-file /etc/couchdb/couchdb.netrc -X GET http://127.0.0.1:5986/nodes/_all_docs + curl -s —netrc-file /etc/couchdb/couchdb.netrc http://127.0.0.1:5984/_membership + ``` + +1. remove bigcouch from all nodes + + ``` + apt-get --purge remove bigcouch + ``` + +1. deploy to all couch nodes + + ``` + leap deploy couchdb + ``` + +1. most likely, deploy will fail because bigcouch will complain about not all nodes beeing connected. Let the deploy finish, restart the bigcouch service on all nodes and re-deploy: + + ``` + /etc/init.d/bigcouch restart + ``` + +1. restore the backup + + ``` + cd /srv/leap/couchdb/scripts + time ./couchdb_restoreall.sh + ``` + +### Migrating from BigCouch to plain CouchDB + +<%= render :partial => 'docs/platform/common/bigcouch_migration_begin.md' %> + + +<%= render :partial => 'docs/platform/common/bigcouch_migration_end.md' %> + + +<%= render :partial => 'docs/platform/common/bigcouch_migration_finish.md' %> diff --git a/doc/services/en.md b/doc/services/en.md new file mode 100644 index 00000000..5d0fec5f --- /dev/null +++ b/doc/services/en.md @@ -0,0 +1,80 @@ +@nav_title = "Services" +@title = "Guide to node services" +@summary = "" +@toc = true + +# Introduction + +Every node (server) must have one or more `services` defined that determines what role the node performs. For example: + + workstation$ cat nodes/stallman.json + { + "ip_address": "199.99.99.1", + "services": ["webapp", "tor"] + } + +Here are common questions to ask when adding a new node to your provider: + +* **many or few?** Some services benefit from having many nodes, while some services are best run on only one or two nodes. +* **required or optional?** Some services are required, while others can be left out. +* **who does the node communicate with?** Some services communicate very heavily with other particular services. Nodes running these services should be close together. +* **public or private network?** Some services communicate with the public internet, while others only need to communicate with other nodes in the infrastructure. + +# Available services + +<table class="table table-striped"> +<tr> + <th>Service</th> + <th>VPN</th> + <th>Email</th> + <th>Notes</th> +</tr> +<tr> + <td>webapp</td> + <td><i class="fa fa-circle"></i></td> + <td><i class="fa fa-circle"></i></td> + <td>User control panel, provider API, and support system.</td> +</tr> +<tr> + <td>couchdb</td> + <td><i class="fa fa-circle"></i></td> + <td><i class="fa fa-circle"></i></td> + <td>Data storage for everything. Private node.</td> +<td></td> +</tr> +<tr> + <td>soledad</td> + <td><i class="fa fa-circle-o"></i></td> + <td><i class="fa fa-circle"></i></td> + <td>User data synchronization daemon. Usually paired with <code>couchdb</code> nodes.</td> +<td></td> +</tr> +<tr> + <td>mx</td> + <td><i class="fa fa-circle-o"></i></td> + <td><i class="fa fa-circle"></i></td> + <td>Incoming and outgoing MX servers.</td> +</tr> +<tr> + <td>openvpn</td> + <td><i class="fa fa-circle"></i></td> + <td><i class="fa fa-circle-o"></i></td> + <td>OpenVPN gateways.</td> +</tr> +<tr> + <td>monitor</td> + <td><i class="fa fa-dot-circle-o"></i></td> + <td><i class="fa fa-dot-circle-o"></i></td> + <td>Nagios monitoring. This service must be on the webapp node.</td> +</tr> +<tr> + <td>tor</td> + <td><i class="fa fa-dot-circle-o"></i></td> + <td><i class="fa fa-dot-circle-o"></i></td> + <td>Tor exit node.</td> +</tr> +</table> + +Key: <i class="fa fa-circle"> Required</i>, <i class="fa fa-dot-circle-o"> Optional</i>, <i class="fa fa-circle-o"> Not Used</i> + +<%= child_summaries %>
\ No newline at end of file diff --git a/doc/services/monitor.md b/doc/services/monitor.md new file mode 100644 index 00000000..576b36a9 --- /dev/null +++ b/doc/services/monitor.md @@ -0,0 +1,36 @@ +@title = "monitor" +@summary = "Nagios monitoring and continuous testing." + +The `monitor` node provides a nagios control panel that will give you a view into the health and status of all the servers and all the services. It will also spam you with alerts if something goes down. + +Topology +-------------------------------------- + +Currently, you can have zero or one `monitor` nodes defined. It is required that the monitor be on the webapp node. It was not designed to be run as a separate node service. + +Configuration +----------------------------------------------- + +* `nagios.environments`: By default, the monitor node will monitor all servers in all environments. You can optionally restrict the environments to the ones you specify. + +For example: + + { + "nagios": { + "environments": ["unstable", "production"] + } + } + +Access nagios web +----------------------------------------------- + +*Determine the nagios URL* + + $ leap ls --print domain.name,webapp.domain,ip_address monitor + > chameleon chameleon.bitmask.net, demo.bitmask.net, 199.119.112.10 + +In this case, you would open `https://demo.bitmask.net/cgi-bin/nagios3` in your browser (or alternately you could use 199.119.112.10 or chameleon.bitmask.net). + +*Determine the nagios password* + +The username for nagios is always `nagiosadmin`. The password is randomly generated and stored in `secrets.json` under the key `nagios_admin_password`. Note that the login is `nagiosadmin` without underscore, but the entry in secrets.json is with underscores. diff --git a/doc/services/mx.md b/doc/services/mx.md new file mode 100644 index 00000000..1a34b660 --- /dev/null +++ b/doc/services/mx.md @@ -0,0 +1,35 @@ +@title = "mx" +@summary = "Incoming and outgoing MX servers." + +Topology +------------------- + +`mx` nodes communicate with the public internet, clients, and `couchdb` nodes. + +Configuration +-------------------- + +### Aliases + +Using the `mx.aliases` property, you can specify your own hard-coded email aliases that precedence over the aliases in the user database. The `mx.aliases` property consists of a hash, where source address points to one or more destination addresses. + +For example: + +`services/mx.json`: + + "mx": { + "aliases": { + "rook": "crow", + "robin": "robin@bird.org", + "flock": ["junco@bird.org", "robin", "crow"], + "chickadee@avian.org": "chickadee@bird.org", + "flicker": ["flicker@bird.org", "flicker@deliver.local"] + } + } + +This example demonstrates several of the features with `mx.aliases`: + +1. alias lists: by specifying an array of destination addresses, as in the case of "flock", the single email will get copied to each address. +1. chained resolution: alias resolution will recursively continue until there are no more matching aliases. For example, "flock" is resolved to "robin", which then gets resolved to "robin@bird.org". +1. virtual domains: by specifying the full domain, as in the case of "chickadee@avian.org", the alias will work for any domain you want. Of course, the MX record for that domain must point to appropriate MX servers, but otherwise you don't need to do any additional configuration. +1. local delivery: for testing purposes, it is often useful to copy all incoming mail for a particular address and send those copies to another address. You can do this by adding "@deliver.local" as one of the destination addresses. When "@local.delivery" is found, alias resolution stops and the mail is delivered to that username. diff --git a/doc/services/openvpn.md b/doc/services/openvpn.md new file mode 100644 index 00000000..5f15ff07 --- /dev/null +++ b/doc/services/openvpn.md @@ -0,0 +1,49 @@ +@title = 'openvpn' +@summary = "OpenVPN egress gateways" + +Topology +------------------ + +Currently, `openvpn` service should not be combined with other services on the same node. + +Unlike most of the other node types, the `openvpn` nodes do not need access to the database and does not ever communicate with any other nodes (except for the `monitor` node, if used). So, `openvpn` nodes can be placed anywhere without regard to the other nodes. + +Configuration +--------------------- + +*Essential configuration* + +* `openvpn.gateway_address`: The address that OpenVPN daemon is bound to and that VPN clients connect to. +* `ip_address`: The main IP of the server, and the egress address for outgoing traffic. + +For example: + + { + "ip_address": "1.1.1.1", + "openvpn": { + "gateway_address": "2.2.2.2" + } + } + +In this example, VPN clients will connect to 2.2.2.2, but their traffic will appear to come from 1.1.1.1. + +Why are two IP addresses needed? Without this, traffic between two VPN users on the same gateway will not get encrypted. This is because the VPN on every client must be configured to allow cleartext traffic for the IP address that is the VPN gateway. + +*Optional configuration* + +Here is the default configuration: + + "openvpn": { + "configuration": { + "auth": "SHA1", + "cipher": "AES-128-CBC", + "fragment": 1400, + "keepalive": "10 30", + "tls-cipher": "DHE-RSA-AES128-SHA", + "tun-ipv6": true + }, + "ports": ["80", "443", "53", "1194"], + "protocols": ["tcp", "udp"] + } + +You may want to change the ports so that only 443 or 80 are used. It is probably best to not modify the `openvpn.configuration` options for now.
\ No newline at end of file diff --git a/doc/services/soledad.md b/doc/services/soledad.md new file mode 100644 index 00000000..e2700d06 --- /dev/null +++ b/doc/services/soledad.md @@ -0,0 +1,12 @@ +@title = 'soledad' +@summary = 'User data synchronization daemon' + +Topology +-------------------- + +Currently, the platform is designed for `soledad` and `couchdb` services to be combined (e.g. every `soledad` node should also be a `couchdb` node). `soledad` nodes might work in isolation, but this is not tested. + +Configuration +------------------------ + +There are no options to configure for `soledad` nodes. diff --git a/doc/services/tor.md b/doc/services/tor.md new file mode 100644 index 00000000..e64b0fe0 --- /dev/null +++ b/doc/services/tor.md @@ -0,0 +1,32 @@ +@title = 'tor' +@summary = 'Tor exit node or hidden service' + +Topology +------------------------ + +Nodes with `tor` service will run a Tor exit or hidden service, depending on what other service it is paired with: + +* `tor` + `openvpn`: when combined with `openvpn` nodes, `tor` will create a Tor exit node to provide extra cover traffic for the VPN. This can be especially useful if there are VPN gateways without much traffic. +* `tor` + `webapp`: when combined with a `webapp` node, the `tor` service will make the webapp and the API available via .onion hidden service. +* `tor` stand alone: a regular Tor exit node. + +If activated, you can list the hidden service .onion addresses this way: + + leap ls --print tor.hidden_service.address tor + +Then just add '.onion' to the end of the printed addresses. + +Configuration +------------------------------ + +* `tor.bandwidth_rate`: the max bandwidth allocated to Tor, in KB per second, when used as an exit node. + +For example: + + { + "tor": { + "bandwidth_rate": 6550 + } + } + + diff --git a/doc/services/webapp.md b/doc/services/webapp.md new file mode 100644 index 00000000..1c06d715 --- /dev/null +++ b/doc/services/webapp.md @@ -0,0 +1,293 @@ +@title = "webapp" +@summary = "leap_web user management application and provider API." + +Introduction +------------------------ + +The service `webapp` will install the web application [[leap_web => https://leap.se/git/leap_web.git]]. It has performs the following functions: + +* REST API for user registration and authentication via the Bitmask client. +* Admin interface to manage users. +* Client certificate distribution and renewal. +* User support help tickets. + +Coming soon: + +* Billing. +* Customizable and localized user documentation. + +The leap_web application is written in Ruby on Rails 3, using CouchDB as the backend data store. + +Topology +------------------------- + +Currently, the platform only supports a single `webapp` node, although we hope to change this in the future. + +* `webapp` nodes communicate heavily with `couchdb` nodes, but the two can be on separate servers. +* The `monitor` service, if enabled, must be on the same node as `webapp`. + +Configuration +-------------------------- + +Essential options: + +* `webapp.admin`: An array of usernames that will be blessed with administrative permissions. These admins can delete users, answer help tickets, and so on. These usernames are for users that have registered through the webapp or through the Bitmask client application, NOT the sysadmin usernames lists in the provider directory `users`. + +Other options: + +* `webapp.engines`: A list of the engines you want enabled in leap_web. Currently, only "support" is available, and it is enabled by default. +* `webapp.invite_required`: If true, registration requires an invite code. Default is `false`. + +For example, `services/webapp.json`: + + { + "webapp": { + "admins": ["joehill", "ali", "mack_the_turtle"] + } + } + +By putting this in `services/webapp.json`, all the `webapp` nodes will inherit the same admin list. + +There are many options in `provider.json` that also control how the webapp behaves. See [[provider-configuration]] for details. + +Invite codes +------------------- + +Enabling the invite code functionality will require new users to provide a valid invite code while signing up for a new account. This is turned off by default, allowing all new users to create an account. + +Set the `invite_code` option to `true` in `services/webapp.json`: + + { + "webapp": { + "invite_required": true + } + } + +This only works with LEAP platform 0.8 or higher. + +Run `leap deploy` to enable the option. + +You can then generate invite codes by logging into the web application with an admin user. + +Alternately, you can also generate invite codes with the command line: + + workstation$ leap ssh bumblebee + bumblebee# cd /srv/leap/webapp/ + bumblebee# sudo -u leap-webapp RAILS_ENV=production bundle exec rake "generate_invites[NUM,USES]" + +Where `bumblebee` should be replaced with the name of your webapp node. + +The **NUM** specifies the amount of codes to generate. The **USES** parameter is optional: By default, all new invite codes can be used once and will then become invalid. If you provide another value for **USES**, you can set a different amount of maximum uses for the codes you generate. + +Customization +--------------------------- + +The provider directory `files/webapp` can be used to customize the appearance of the webapp. All the files in this directory will get sync'ed to the `/srv/leap/webapp/config/customization` directory of the deployed webapp node. + +Files in the `files/webapp` can override view files, locales, and stylesheets in the leap_web app: + +For example: + + stylesheets/ -- override files in Rails.root/app/assets/stylesheets + tail.scss -- included before all others + head.scss -- included after all others + + public/ -- overrides files in Rails.root/public + favicon.ico -- custom favicon + img/ -- customary directory to put images in + + views/ -- overrides files Rails.root/app/views + home/ + index.html.haml -- this file is what shows up on + the home page + pages/ + privacy-policy.en.md -- this file will override + the default privacy policy + terms-of-service.en.md -- this file will override + the default TOS. + + locales/ -- overrides files in Rails.root/config/locales + en.yml -- overrides for English + de.yml -- overrides for German + and so on... + +To interactively develop your customizations before you deploy them, you have two options: + +1. Edit a `webapp` node. This approach involves directly modifying the contents of the directory `/srv/leap/webapp/config/customization` on a deployed `webapp` node. This can, and probably should be, a "local" node. When doing this, you may need to restart leap_web in order for changes to take effect (`touch /srv/leap/webapp/tmp/restart.txt`). +2. Alternately, you can install leap_web to run on your computer and edit files in `config/customization` locally. This approach does not require a provider or a `webapp` node. For more information, see the [leap_web README](https://github.com/leapcode/leap_web). + +NOTE: If you add a `tails.scss` or `head.scss` file, then you usually need to run `rake tmp:clear` and restart rails in order for the new stylesheet to get recognized. You should only need to do this once. + +Once you have what you want, then copy these files to the local provider directory `files/webapp` so that they will be installed each time you deploy. + +Customization tutorial +---------------------------- + +This mini-tutorial will walk you through creating a custom "branding" of the leap_web application. We will be creating a provider called "Prehistoric Computer." + +Here are the files we are going to create: + + leap_web/config/customization + ├── locales + │ ├── en.yml + │ └── es.yml + ├── public + │ ├── favicon.ico + │ └── img + │ └── masthead.png + ├── stylesheets + │ └── tail.scss + └── views + └── pages + ├── privacy-policy.en.md + └── privacy-policy.es.md + +All these files are available in the source code in the [[customization.example => https://github.com/leapcode/leap_web/tree/develop/config/customization.example]] directory. + +Remember, these files may live different places: + +* `user@localmachine$ leap_web/config/customization`: This will be the path if you have checked out a local copy of leap_web.git and are running `rails server` locally in order to test your customizations. +* `user@localmachine$ PROVIDER/files/webapp`: This is the local provider directory where the files should be put so that they get correctly deployed to webapp nodes. +* `root@webappnode# /srv/leap/webapp/config/customization`: This is where the files in the local provider directory `PROVIDER/files/webapp` get copied to after a `leap deploy` to a live webapp nodes. + +### Override translations + +You can add additional locale files in order to change the text used in the existing application and to add translations for string that you added to the application. + +In this example, we will be altering the default text for the "login_info" string. In `config/locales/en/home.en.yml` there is this entry: + + en: + login_info: "Log in to change your account settings, create support tickets, and manage payments." + +We are going to override this with some custom text in English and Spanish: + +`leap_web/config/customization/locale/en.yml`: + + en: + login_info: Authenticate to change your "Prehistoric Computer" settings. + +`leap_web/config/customization/locale/es.yml`: + + es: + login_info: Autenticar a cambiar la configuración de "Computer Prehistoria." + +Now, the home page of leap_web will use these new strings instead of the default. Remember that you must restart rails in order for new locale files to take effect. + +### Override static pages + +You can also override any of the static files included with leap_web, such as the privacy policy or terms of service. + +Here is how we would create a custom privacy policy in English and Spanish: + +`leap_web/config/customization/views/pages/privacy-policy.en.md`: + + # Custom Privacy Policy + This is our privacy policy. + +`leap_web/config/customization/views/pages/privacy-policy.es.md`: + + # Custom Política de Privacidad + Esta es nuestra política de privacidad. + +### Add a custom header + +Now we will add a custom header to every page. First, we add the images: + + leap_web/config/customization + ├── public + ├── favicon.ico + └── img + └── masthead.png + +You can create your own, or use the example files in https://github.com/leapcode/leap_web/tree/develop/config/customization.example + +Now, we add some custom CSS so that we can style the masthead: + +`leap_web/config/customization/stylesheets/tail.scss` + + $custom-color: #66bbaa; + + a { + color: $custom-color; + } + + // + // MASTHEAD + // + + #masthead { + background-color: $custom-color; + border-bottom: none; + + // make the masthead clickable by replacing the + // site name link with the masthead image: + .title { + padding: 0px; + .sitename a { + display: block; + background: url(/img/masthead.png) 0 0 no-repeat; + font-size: 0px; + height: 100px; + background-size: auto 100px; + } + } + } + + // make the home page masthead slightly larger + body.home #masthead { + .sitename a { + height: 150px; + background-size: auto 150px; + } + } + + // + // FOOTER + // + + #footer .links { + background-color: $custom-color; + } + +NOTE: If you add a `tails.scss` or `head.scss` file, then you usually need to run `rake tmp:clear` and restart rails in order for the new stylesheet to get recognized. You should only need to do this once. + + +Custom Fork +---------------------------- + +Sometimes it is easier to maintain your own fork of the leap_web app. You can keep your customizations in that fork instead of in the provider `files/webapp` directory. Or, perhaps you want to add an engine to the application that modifies the app's behavior. + +To deploy your own leap_web, modify the provider file `common.json`: + + { + "sources": { + "webapp": { + "revision": "origin/develop", + "source": "https://github.com/leapcode/leap_web", + "type": "git" + } + } + } + +To target only particular environment, modify instead `common.ENV.json`, where ENV is the name of the environment. + +See https://github.com/leapcode/leap_web/blob/develop/doc/DEVELOP.md for notes on getting started hacking on leap_web. + +Maintenance mode +------------------ + +You can put the webapp into maintenance mode by simply dropping a html file to `/srv/leap/webapp/public/system/maintenance.html`. For example: + + workstation$ leap ssh webappnode + server# echo "Temporarily down for maintenance. We will be back soon." > /srv/leap/webapp/public/system/maintenance.html + +Known problems +--------------------------- + +* Client certificates are generated without a CSR. The problem is that this makes the web + application extremely vulnerable to denial of service attacks. This was not an issue until we + started to allow the possibility of anonymously fetching a client certificate without + authenticating first. +* By its very nature, the user database is vulnerable to enumeration attacks. These are + very hard to prevent, because our protocol is designed to allow query of a user database via + proxy in order to provide network perspective. diff --git a/doc/troubleshooting/tests.md b/doc/troubleshooting/tests.md index b85c19d2..607f924e 100644 --- a/doc/troubleshooting/tests.md +++ b/doc/troubleshooting/tests.md @@ -8,15 +8,15 @@ At any time, you can run troubleshooting tests on the nodes of your provider inf To run tests on FILTER node list: - leap test run FILTER + workstation$ leap test run FILTER For example, you can also test a single node (`leap test elephant`); test a specific environment (`leap test development`), or any tag (`leap test soledad`). Alternately, you can run test on all nodes (probably only useful if you have pinned the environment): - leap test + workstation$ leap test -The tests that are performed are located in the platform under the tests directory. +The tests that are performed are located in the platform under the tests directory. ## Testing with the bitmask client @@ -50,16 +50,16 @@ In order to set up a monitoring node, you simply add a `monitor` service tag to After deploying, this node will regularly poll every node to ask for the status of various health checks. These health checks include the checks run with `leap test`, plus many others. -We use [Nagios](http://www.nagios.org/) together with [Check MK agent](https://en.wikipedia.org/wiki/Check_MK) for running checks on remote hosts. +We use [Nagios](https://www.nagios.org/) together with [Check MK agent](https://en.wikipedia.org/wiki/Check_MK) for running checks on remote hosts. One nagios installation will monitor all nodes in all your environments. You can log into the monitoring web interface via [https://DOMAIN/nagios3/](https://DOMAIN/nagios3/). The username is `nagiosadmin` and the password is found in the secrets.json file in your provider directory. Nagios will send out mails to the `contacts` address provided in `provider.json`. -## Nagios Frontents +## Nagios Frontends There are other ways to check and get notified by Nagios besides regularly checking the Nagios webinterface or reading email notifications. Check out the [Frontends (GUIs and CLIs)](http://exchange.nagios.org/directory/Addons/Frontends-%28GUIs-and-CLIs%29) on the Nagios project website. -A recommended status tray application is [Nagstamon](https://nagstamon.ifw-dresden.de/), which is available for Linux, MacOS X and Windows. It can not only notify you of hosts/services failures, you can also acknoledge or recheck these with it. +A recommended status tray application is [Nagstamon](https://nagstamon.ifw-dresden.de/), which is available for Linux, MacOS X and Windows. It can not only notify you of hosts/services failures, you can also acknowledge or recheck them. ### Log Monitoring diff --git a/doc/troubleshooting/vagrant.md b/doc/troubleshooting/vagrant.md deleted file mode 100644 index ad284161..00000000 --- a/doc/troubleshooting/vagrant.md +++ /dev/null @@ -1,45 +0,0 @@ -@title = 'LEAP Platform Vagrant testing' -@nav_title = 'Vagrant Integration' -@summary = 'Testing your provider with Vagrant' - -Setting up Vagrant for a testing the platform -============================================= - -There are two ways you can setup leap platform using vagrant. - -Using the Vagrantfile provided by Leap Platform ------------------------------------------------ - -This is by far the easiest way. It will install a single node mail server in the default -configuration with one single command. - -Clone the platform with - - git clone https://github.com/leapcode/leap_platform.git - -Start the vagrant box with - - cd leap_platform - vagrant up - -Follow the instructions how to configure your `/etc/hosts` -in order to use the provider! - -You can login via ssh with the systemuser `vagrant` and the same password. - -There are 2 users preconfigured: - -. `testuser` with pw `hallo123` -. `testadmin` with pw `hallo123` - - -Use the leap_cli vagrant integration ------------------------------------- - -Install leap_cli and leap_platform on your host, configure a provider from scratch and use the `leap local` commands to manage your vagrant node(s). - -See https://leap.se/en/docs/platform/development how to use the leap_cli vagrant -integration and https://leap.se/en/docs/platform/tutorials/single-node-email how -to setup a single node mail server. - - diff --git a/doc/troubleshooting/where-to-look.md b/doc/troubleshooting/where-to-look.md index fbd95931..c92fba8f 100644 --- a/doc/troubleshooting/where-to-look.md +++ b/doc/troubleshooting/where-to-look.md @@ -7,6 +7,16 @@ General ======= * Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. `-v 5` after the `leap` cmd, i.e. `leap -v 2 deploy`. +* We use the `example.org` domain for documentation purposes here, please replace it with the you domain. + +Firewall +======================= + +Every node in your provider has its own restrictive firewall, but you might have a network firewall in place as well that is not managed by LEAP platform. To see what ports and addresses must be open, run this command: + + workstation$ leap compile firewall + +If any of those are blocked, then your provider will not work. Webapp ====== @@ -19,10 +29,10 @@ Places to look for errors * `/var/log/syslog` (watch out for stunnel issues) * `/var/log/leap/*` + Is haproxy ok ? --------------- - curl -s -X GET "http://127.0.0.1:4096" Is couchdb accessible through stunnel ? @@ -59,13 +69,21 @@ Check couchdb acl as unpriviledged user curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096/_all_dbs" +All URLs accessible ? +--------------------- + +* https://example.org +* https://api.example.org:4430/provider.json +* https://example.org/ca.crt + + Check client config files ------------------------- - https://example.net/provider.json - https://example.net/1/config/smtp-service.json - https://example.net/1/config/soledad-service.json - https://example.net/1/config/eip-service.json +* https://example.net/provider.json +* https://example.net/1/config/smtp-service.json +* https://example.net/1/config/soledad-service.json +* https://example.net/1/config/eip-service.json Soledad diff --git a/doc/tutorials/configure-provider.md b/doc/tutorials/configure-provider.md deleted file mode 100644 index 969d541b..00000000 --- a/doc/tutorials/configure-provider.md +++ /dev/null @@ -1,31 +0,0 @@ -@title = 'Configure provider tutorial' -@nav_title = 'Configure Provider' -@summary = 'Explore how to configure your provider after the initial setup' - - -Edit provider.json configuration --------------------------------------- - -There are a few required settings in provider.json. At a minimum, you must have: - - { - "domain": "example.org", - "name": "Example", - "contacts": { - "default": "email1@example.org" - } - } - -For a full list of possible settings, you can use `leap inspect` to see how provider.json is evaluated after including the inherited defaults: - - $ leap inspect provider.json - - -Examine Certs -============= - -To see details about the keys and certs that the prior two commands created, you can use `leap inspect` like so: - - $ leap inspect files/ca/ca.crt - -NOTE: the files `files/ca/*.key` are extremely sensitive and must be carefully protected. The other key files are much less sensitive and can simply be regenerated if needed. diff --git a/doc/tutorials/quick-start.md b/doc/tutorials/quick-start.md index a92cc9da..f963867a 100644 --- a/doc/tutorials/quick-start.md +++ b/doc/tutorials/quick-start.md @@ -1,385 +1,230 @@ @title = 'Quick Start Tutorial' @nav_title = 'Quick Start Tutorial' -@summary = 'This tutorial walks you through the initial process of creating and deploying a minimal service provider running the LEAP Platform. This Quick Start guide will guide you through building a three node OpenVPN provider.' +@summary = 'This tutorial walks you through the initial process of creating and deploying a minimal service provider running the LEAP Platform.' +Introduction +==================================== -Our goal ------------------- +### Our goal -We are going to create a minimal LEAP provider offering OpenVPN service. This basic setup can be expanded by adding more OpenVPN nodes to increase capacity or geographical diversity, or more webapp nodes to increase availability (at the moment, a single couchdb and single webapp server are all that is supported, and performance wise, are more than enough for most usage, since they are only lightly used). At the moment, we strongly advise only have one couchdb server for stability purposes. +We are going to create a minimal LEAP provider, but one that does not offer any actual services. Check out the other tutorials for adding VPN or email services. Our goal is something like this: $ leap list - NODES SERVICES TAGS - cheetah couchdb production - wildebeest webapp production - ostrich openvpn production + NODES SERVICES TAGS + wildebeest couchdb, webapp NOTE: You won't be able to run that `leap list` command yet, not until we actually create the node configurations. -Requirements ------------- +### Requirements -In order to complete this Quick Start, you will need a few things: +1. A workstation: This is your local machine that you will run commands on. +1. A server: This is the machine that you will deploy to. The server can be either: + 1. A local Vagrant virtual machine: a Vagrant machine can only be useful for testing. + 1. A real or paravirtualized server: The server must have Debian Jessie installed, and you must be able to SSH into the machine as root. Paravirtualization includes KVM, Xen, OpenStack, Amazon, but not VirtualBox or OpenVZ. -* You will need three real or paravirtualized virtual machines (KVM, Xen, Openstack, Amazon, but not Vagrant - sorry) that have a basic Debian Stable installed. If you allocate 20G of disk space to each node for the system, after this process is completed, you will have used less than 10% of that disk space. If you allocate 2 CPUs and 8G of memory to each node, that should be more than enough to begin with. -* You should be able to SSH into them remotely, and know their root password, IP addresses and their SSH host keys -* You will need four different IPs. Each node gets a primary IP, and the OpenVPN gateway additionally needs a gateway IP. -* The ability to create/modify DNS entries for your domain is preferable, but not needed. If you don't have access to DNS, you can workaround this by modifying your local resolver, i.e. editing `/etc/hosts`. -* You need to be aware that this process will make changes to your systems, so please be sure that these machines are a basic install with nothing configured or running for other purposes -* Your machines will need to be connected to the internet, and not behind a restrictive firewall. -* You should work locally on your laptop/workstation (one that you trust and that is ideally full-disk encrypted) while going through this guide. This is important because the provider configurations you are creating contain sensitive data that should not reside on a remote machine. The `leap` command will login to your servers and configure the services. -* You should do everything described below as an unprivileged user, and only run those commands as root that are noted with *sudo* in front of them. Other than those commands, there is no need for privileged access to your machine, and in fact things may not work correctly. - -All the commands in this tutorial are run on your sysadmin machine. In order to complete the tutorial, the sysadmin will do the following: - -* Install pre-requisites -* Install the LEAP command-line utility -* Check out the LEAP platform -* Create a provider and its certificates -* Setup the provider's nodes and the services that will reside on those nodes -* Initialize the nodes -* Deploy the LEAP platform to the nodes -* Test that things worked correctly -* Some additional commands - -We will walk you through each of these steps. +Other things to keep in mind: +* The ability to create/modify DNS entries for your domain is preferable, but not needed. If you don't have access to DNS, you can workaround this by modifying your local resolver, i.e. editing `/etc/hosts`. +* You need to be aware that this process will make changes to your servers, so please be sure that these machines are a basic install with nothing configured or running for other purposes. +* Your servers will need to be connected to the internet, and not behind a restrictive firewall. -Prepare your environment +Prepare your workstation ======================== -There are a few things you need to setup before you can get going. Just some packages, the LEAP cli and the platform. +In order to be able to manage your servers, you need to install the `leap` command on your workstation: -Install pre-requisites --------------------------------- +### Install pre-requisites -*Debian & Ubuntu* +Install core prerequisites on your workstation. -Install core prerequisites: +*Debian & Ubuntu* - $ sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make bzip2 + workstation$ sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make bzip2 -<!-- *Mac OS* -1. Install rubygems from https://rubygems.org/pages/download (unless the `gem` command is already installed). ---> - -NOTE: leap_cli requires ruby 1.9 or later. - + workstation$ brew install ruby-install + workstation$ ruby-install ruby -Install the LEAP command-line utility -------------------------------------------------- +### Install the LEAP command-line utility Install the `leap` command from rubygems.org: - $ sudo gem install leap_cli - -Alternately, you can install `leap` from source: - - $ git clone https://leap.se/git/leap_cli - $ cd leap_cli - $ rake build - $ sudo rake install - -You can also install from source as an unprivileged user, if you want. For example, instead of `sudo rake install` you can do something like this: - - $ rake install - # watch out for the directory leap is installed to, then i.e. - $ sudo ln -s ~/.gem/ruby/1.9.1/bin/leap /usr/local/bin/leap - -With either `rake install` or `sudo rake install`, you can use now /usr/local/bin/leap, which in most cases will be in your $PATH. - -If you have successfully installed the `leap` command, then you should be able to do the following: - - $ leap --help - -This will list the command-line help options. If you receive an error when doing this, please read through the README.md in the `leap_cli` source to try and resolve any problems before going forwards. - -Check out the platform --------------------------- + workstation$ gem install leap_cli --install-dir ~/leap + workstation$ export PATH=$PATH:~/leap/bin -The LEAP Platform is a series of puppet recipes and modules that will be used to configure your provider. You will need a local copy of the platform that will be used to setup your nodes and manage your services. To begin with, you will not need to modify the LEAP Platform. +Alternately, you can install `leap` system wide: -First we'll create a directory for LEAP things, and then we'll check out the platform code and initalize the modules: + workstation$ sudo gem install leap_cli - $ mkdir ~/leap - $ cd ~/leap - $ git clone https://leap.se/git/leap_platform.git - $ cd leap_platform - $ git submodule sync; git submodule update --init +To confirm that you installed `leap` correctly, try running `leap help`. +Create a provider instance +============================================= -Provider Setup -============== +A provider instance is a directory tree, residing on your workstation, that contains everything you need to manage an infrastructure for a service provider. -A provider instance is a directory tree, usually stored in git, that contains everything you need to manage an infrastructure for a service provider. In this case, we create one for example.org and call the instance directory 'example'. +In this case, we create one for example.org and call the instance directory 'example'. - $ mkdir -p ~/leap/example - -Bootstrap the provider ------------------------ - -Now, we will initialize this directory to make it a provider instance. Your provider instance will need to know where it can find the local copy of the git repository leap_platform, which we setup in the previous step. - - $ cd ~/leap/example - $ leap new . - -NOTES: - . make sure you include that trailing dot! + workstation$ leap new ~/example The `leap new` command will ask you for several required values: * domain: The primary domain name of your service provider. In this tutorial, we will be using "example.org". * name: The name of your service provider (we use "Example"). * contact emails: A comma separated list of email addresses that should be used for important service provider contacts (for things like postmaster aliases, Tor contact emails, etc). -* platform: The directory where you have a copy of the `leap_platform` git repository checked out. +* platform: The directory where you have a copy of the `leap_platform` git repository checked out. If the platform directory does not yet exist, the `leap_platform` will be downloaded and placed in that directory. You could also have passed these configuration options on the command-line, like so: - $ leap new --contacts your@email.here --domain leap.example.org --name Example --platform=~/leap/leap_platform . - -You may want to poke around and see what is in the files we just created. For example: + workstation$ leap new --contacts your@email.here --domain example.org --name Example --platform=~/leap/leap_platform . - $ cat provider.json +You should now have the following files: -Optionally, commit your provider directory using the version control software you fancy. For example: - - $ git init - $ git add . - $ git commit -m "initial provider commit" + workstation$ tree example + example + ├── common.json + ├── Leapfile + ├── nodes/ + ├── provider.json + ├── services/ + └── tags/ Now add yourself as a privileged sysadmin who will have access to deploy to servers: - $ leap add-user --self - -NOTE: in most cases, `leap` must be run from within a provider instance directory tree (e.g. ~/leap/example). - -Create provider certificates ----------------------------- - -Create two certificate authorities, one for server certs and one for client -certs (note: you only need to run this one command to get both): + workstation$ cd example + workstation$ leap add-user louise --self - $ leap cert ca +Replace "louise" with whatever you want your sysadmin username to be. -Create a temporary cert for your main domain (you should replace with a real commercial cert at some point) +NOTE: Make sure you change directories so that the `leap` command is run from within the provider instance directory. Most `leap` commands only work when run from a provider instance. - $ leap cert csr +Now create the necessary keys and certificates: -To see details about the keys and certs that the prior two commands created, you can use `leap inspect` like so: + workstation$ leap cert ca + workstation$ leap cert csr - $ leap inspect files/ca/ca.crt - -Create the Diffie-Hellman parameters file, needed for forward secret OpenVPN ciphers: - - $ leap cert dh - -NOTE: the files `files/ca/*.key` are extremely sensitive and must be carefully protected. The other key files are much less sensitive and can simply be regenerated if needed. - - -Edit provider.json configuration --------------------------------------- - -There are a few required settings in provider.json. At a minimum, you must have: - - { - "domain": "example.org", - "name": "Example", - "contacts": { - "default": "email1@example.org" - } - } +What do these commands do? The first command will create two Certificate Authorities, one that clients will use to authenticate with the servers and one for backend servers to authenticate with each other. The second command creates a Certificate Signing Request suitable for submission to a commercial CA. It also creates two "dummy" files for you to use temporarily: -For a full list of possible settings, you can use `leap inspect` to see how provider.json is evaluated after including the inherited defaults: +* `files/cert/example.org.crt` -- This is a "dummy" certificate for your domain that can be used temporarily for testing. Once you get a real certificate from a CA, you should replace this file. +* `files/cert/commercial_ca.crt` -- This is "dummy" CA cert the corresponds to the dummy domain certificate. Once you replace the domain certificate, also replace this file with the CA cert from the real Certificate Authority. - $ leap inspect provider.json +If you plan to run a real service provider, see important information on [[managing keys and certificates => keys-and-certificates]]. +Add a node to the provider +================================================== -Setup the provider's nodes and services ---------------------------------------- +A "node" is a server that is part of your infrastructure. Every node can have one or more services associated with it. We will now add a single node with two services, "webapp" and "couchdb". -A "node" is a server that is part of your infrastructure. Every node can have one or more services associated with it. Some nodes are "local" and used only for testing, see [Development](development) for more information. +You have two choices for node type: a real node or a local node. -Create a node, with the service "webapp": +* Real Node: A real node is any physical or paravirtualized server, including KVM, Xen, OpenStack Compute, Amazon EC2, but not VirtualBox or OpenVZ (VirtualBox and OpenVZ use a more limited form of virtualization). The server must be running Debian Jessie. +* Local Node: A local node is a virtual machine created by Vagrant, useful for local testing on your workstation. - $ leap node add wildebeest ip_address:x.x.x.w services:webapp tags:production +Getting Vagrant working can be a pain and is [[covered in other tutorials => vagrant]]. If you have a real server available, we suggest you try this tutorial with a real node first. -NOTE: replace x.x.x.w with the actual IP address of this node +### Option A: Add a real node -This created a node configuration file in `nodes/wildebeest.json`, but it did not do anything else. It also added the 'tag' called 'production' to this node. Tags allow us to conveniently group nodes together. When creating nodes, you should give them the tag 'production' if the node is to be used in your production infrastructure. +Note: Installing LEAP Platform on this server will potentially destroy anything you have previously installed on this machine. -The web application and the VPN nodes require a database, so lets create the database server node: +Create a node, with the services "webapp" and "couchdb": - $ leap node add cheetah ip_address:x.x.x.x services:couchdb tags:production + workstation$ leap node add wildebeest ip_address:x.x.x.w services:webapp,couchdb -NOTE: replace x.x.x.x with the actual IP address of this node +NOTE: replace x.x.x.x with the actual IP address of this server. -Now we need the OpenVPN gateway, so lets create that node: +### Option B: Add a local node - $ leap node add ostrich ip_address:x.x.x.y openvpn.gateway_address:x.x.x.z services:openvpn tags:production +Create a node, with the services "webapp" and "couchdb", and then start the local virtual machine: -NOTE: replace x.x.x.y with the IP address of the machine, and x.x.x.z with the second IP. openvpn gateways must be assigned two IP addresses, one for the host itself and one for the openvpn gateway. We do this to prevent incoming and outgoing VPN traffic on the same IP. Without this, the client might send some traffic to other VPN users in the clear, bypassing the VPN. + workstation$ leap node add --local wildebeest services:webapp,couchdb + workstation$ leap local start wildebeest +It will take a while to download the Virtualbox base box and create the virtual machine. -Setup DNS ---------- +Deploy your provider +========================================= -Now that you have the nodes configured, you should create the DNS entries for these nodes. - -Set up your DNS with these hostnames: - - $ leap list --print ip_address,domain.full,dns.aliases - cheetah x.x.x.w, cheetah.example.org, null - wildebeest x.x.x.x, wildebeest.example.org, api.example.org - ostrich x.x.x.y, ostrich.example.org, null - -Alternately, you can adapt this zone file snippet: - - $ leap compile zone - -If you cannot edit your DNS zone file, you can still test your provider by adding entries to your local resolver hosts file (`/etc/hosts` for linux): - - x.x.x.w cheetah.example.org - x.x.x.x wildebeest.example.org api.example.org example.org - x.x.x.y ostrich.example.org - -Please don't forget about these entries, they will override DNS queries if you setup your DNS later. - - -Initialize the nodes --------------------- +### Initialize the node Node initialization only needs to be done once, but there is no harm in doing it multiple times: - $ leap node init production + workstation$ leap node init wildebeest -This will initialize all nodes with the tag "production". When `leap node init` is run, you will be prompted to verify the fingerprint of the SSH host key and to provide the root password of the server(s). You should only need to do this once. +This will initialize the node `wildebeest`. -If you prefer, you can initalize each node, one at a time: +For non-local nodes, when `leap node init` is run, you will be prompted to verify the fingerprint of the SSH host key and to provide the root password of the server(s). You should only need to do this once. - $ leap node init wildebeest - $ leap node init cheetah - $ leap node init ostrich +### Deploy to the node -Deploy the LEAP platform to the nodes --------------------- +The next step is to deploy the LEAP platform to your node. [Deployment can take a while to run](https://xkcd.com/303/), especially on the first run, as it needs to update the packages on the new machine. -Now you should deploy the platform recipes to the nodes. [Deployment can take a while to run](http://xkcd.com/303/), especially on the first run, as it needs to update the packages on the new machine. - -*Important notes:* currently nodes must be deployed in a certain order. The underlying couch database node(s) must be deployed first, and then all other nodes. - - $ leap deploy cheetah + workstation$ leap deploy wildebeest Watch the output for any errors (in red), if everything worked fine, you should now have your first running node. If you do have errors, try doing the deploy again. -However, to deploy our three-node openvpn setup, we need the database and LEAP web application requires a database to run, so let's deploy to the couchdb and openvpn nodes: - - $ leap deploy wildebeest - $ leap deploy ostrich - - -What is going on here? --------------------------------------------- +### Setup DNS -First, some background terminology: +The next step is to configure the DNS for your provider. For testing purposes, you can just modify your `/etc/hosts` file. Please don't forget about these entries, they will override DNS queries if you setup your DNS later. For a list of what entries to add to `/etc/hosts`, run this command: -* **puppet**: Puppet is a system for automating deployment and management of servers (called nodes). -* **hiera files**: In puppet, you can use something called a 'hiera file' to seed a node with a few configuration values. In LEAP, we go all out and put *every* configuration value needed for a node in the hiera file, and automatically compile a custom hiera file for each node. + workstation$ leap compile hosts -When you run `leap deploy`, a bunch of things happen, in this order: +Alternately, if you have access to modify the DNS zone entries for your domain: -1. **Compile hiera files**: The hiera configuration file for each node is compiled in YAML format and saved in the directory `hiera`. The source material for this hiera file consists of all the JSON configuration files imported or inherited by the node's JSON config file. -* **Copy required files to node**: All the files needed for puppet to run are rsync'ed to each node. This includes the entire leap_platform directory, as well as the node's hiera file and other files needed by puppet to set up the node (keys, binary files, etc). -* **Puppet is run**: Once the node is ready, leap connects to the node via ssh and runs `puppet apply`. Puppet is applied locally on the node, without a daemon or puppetmaster. + workstation$ leap compile zone -You can run `leap -v2 deploy` to see exactly what commands are being executed. +NOTE: The resulting zone file is incomplete because it is missing a serial number. Use the output of `leap compile zone` as a guide, but do not just copy and paste the output. Also, the `compile zone` output will always exclude mention of local nodes. +The DNS method will not work for local nodes created with Vagrant. Test that things worked correctly ================================= -You should now have three machines with the LEAP platform deployed to them, one for the web application, one for the database and one for the OpenVPN gateway. - To run troubleshooting tests: - leap test - -If you want to confirm for yourself that things are working, you can perform the following manual tests. - -### Access the web application - -In order to connect to the web application in your browser, you need to point your domain at the IP address of the web application node (named wildebeest in this example). + workstation$ leap test -There are a lot of different ways to do this, but one easy way is to modify your `/etc/hosts` file. First, find the IP address of the webapp node: +Alternately, you can run these same tests from the server itself: - $ leap list webapp --print ip_address + workstation$ leap ssh wildebeest + wildebeest# run_tests -Then modify `/etc/hosts` like so: +Create an administrator +=============================== - x.x.x.w leap.example.org +Assuming that you set up your DNS or `/etc/hosts` file, you should be able to load `https://example.org` in your web browser (where example.org is whatever domain name you actually used). -Replacing 'leap.example.org' with whatever you specified as the `domain` in the `leap new` command. +Your browser will complain about an untrusted cert, but for now just bypass this. From there, you should be able to register a new user and login. -Next, you can connect to the web application either using a web browser or via the API using the LEAP client. To use a browser, connect to https://leap.example.org (replacing that with your domain). Your browser will complain about an untrusted cert, but for now just bypass this. From there, you should be able to register a new user and login. +Once you have created a user, you can now make this user an administrator. For example, if you created a user `kangaroo`, you would create the file `services/webapp.json` with the following content: -### Use the VPN - -You should be able to simply test that the OpenVPN gateway works properly by doing the following: - - $ leap test init - $ sudo openvpn test/openvpn/production_unlimited.ovpn + { + "webapp": { + "admins": ["kangaroo"] + } + } -Or, you can use the LEAP client (called "bitmask") to connect to your new provider, create a user and then connect to the VPN. +Save that file and run `leap deploy` again. When you next log on to the web application, the user kangaroo will now be an admin. +If you want to restrict who can register a new user, see [[webapp]] for configuration options. -Additional information +What is next? ====================== -It is useful to know a few additional things. - -Useful commands ---------------- - -Here are a few useful commands you can run on your new local nodes: - -* `leap ssh wildebeest` -- SSH into node wildebeest (requires `leap node init wildebeest` first). -* `leap list` -- list all nodes. -* `leap list production` -- list only those nodes with the tag 'production' -* `leap list --print ip_address` -- list a particular attribute of all nodes. -* `leap cert update` -- generate new certificates if needed. - -See the full command reference for more information. +Add an end-user service +------------------------------- -Node filters -------------------------------------------- +You should now have a minimal service provider with a single node. This service provider is pointless at the moment, because it does not include any end-user services like VPN or email. To add one of these services, continue with one of the following tutorials: -Many of the `leap` commands take a "node filter". You can use a node filter to target a command at one or more nodes. +* [[single-node-email]] +* [[single-node-vpn]] -A node filter consists of one or more keywords, with an optional "+" before each keyword. - -* keywords can be a node name, a service type, or a tag. -* the "+" before the keyword constructs an AND condition -* otherwise, multiple keywords together construct an OR condition - -Examples: - -* `leap list openvpn` -- list all nodes with service openvpn. -* `leap list openvpn +production` -- only nodes of service type openvpn AND tag production. -* `leap deploy webapp openvpn` -- deploy to all webapp OR openvpn nodes. -* `leap node init ostrich` -- just init the node named ostrich. - -Keep track of your provider configurations ------------------------------------------- - -You should commit your provider changes to your favorite VCS whenever things change. This way you can share your configurations with other admins, all they have to do is to pull the changes to stay up to date. Every time you make a change to your provider, such as adding nodes, services, generating certificates, etc. you should add those to your VCS, commit them and push them to where your repository is hosted. - -Note that your provider directory contains secrets! Those secrets include passwords for various services. You do not want to have those passwords readable by the world, so make sure that wherever you are hosting your repository, it is not public for the world to read. - -What's next ------------------------------------ +Learn more +--------------- -Read the [LEAP platform guide](guide) to learn about planning and securing your infrastructure. +We have only just scratched the surface of the possible ways to configure and deploy your service provider. Your next step should be: +* Read [[getting-started]] for more details on using the LEAP platform. +* See [[commands]] for a list of possible commands. diff --git a/doc/tutorials/single-node-email.md b/doc/tutorials/single-node-email.md index b47496b9..0a73e6e1 100644 --- a/doc/tutorials/single-node-email.md +++ b/doc/tutorials/single-node-email.md @@ -1,282 +1,69 @@ @title = 'Single node email tutorial' -@nav_title = 'Single node email' -@summary = 'A single node email provider.' +@nav_title = 'Quick email' +@summary = 'Tutorial for setting up a simple email provider.' -Quick Start - Single node setup -=============================== - -This tutorial walks you through the initial process of creating and deploying a minimal service provider running the [LEAP platform](platform). -We will guide you through building a single node mail provider. +This tutorial walks you through the initial process of creating and deploying a minimal email service provider. Please first complete the [[quick-start]]. This tutorial will pick up where that one left off. Our goal ------------------ -We are going to create a minimal LEAP provider offering Email service. This basic setup can be expanded by adding more webapp and couchdb nodes to increase availability (performance wise, a single couchdb and a single webapp are more than enough for most usage, since they are only lightly used, but you might want redundancy). Please note: currently it is not possible to safely add additional couchdb nodes at a later point. They should all be added in the beginning, so please consider carefully if you would like more before proceeding. +We are going to create a minimal LEAP provider offering email service. Our goal is something like this: $ leap list - NODES SERVICES TAGS - node1 couchdb, mx, soledad, webapp local - -NOTE: You won't be able to run that `leap list` command yet, not until we actually create the node configurations. - -Requirements ------------- - -In order to complete this Quick Start, you will need a few things: - -* You will need `one real or paravirtualized virtual machine` (Vagrant, KVM, Xen, Openstack, Amazon, …) that have a basic Debian Stable installed. -* You should be able to `SSH into them` remotely, and know their root password, IP addresses and their SSH host keys -* The ability to `create/modify DNS entries` for your domain is preferable, but not needed. If you don't have access to DNS, you can workaround this by modifying your local resolver, i.e. editing `/etc/hosts`. -* You need to be aware that this process will make changes to your machines, so please be sure that these machines are a basic install with nothing configured or running for other purposes -* Your machines will need to be connected to the internet, and not behind a restrictive firewall. -* You should `work locally on your laptop/workstation` (one that you trust and that is ideally full-disk encrypted) while going through this guide. This is important because the provider configurations you are creating contain sensitive data that should not reside on a remote machine. The leap cli utility will login to your servers and configure the services. -* You should do everything described below as an `unprivileged user`, and only run those commands as root that are noted with *sudo* in front of them. Other than those commands, there is no need for privileged access to your machine, and in fact things may not work correctly. - -All the commands in this tutorial are run on your sysadmin machine. In order to complete the tutorial, the sysadmin will do the following: - -* Install pre-requisites -* Install the LEAP command-line utility -* Check out the LEAP platform -* Create a provider and its certificates -* Setup the provider's node and the services that will reside on it -* Initialize the node -* Deploy the LEAP platform to the node -* Test that things worked correctly -* Some additional commands - -We will walk you through each of these steps. - - -Prepare your environment -======================== - -There are a few things you need to setup before you can get going. Just some packages, the LEAP cli and the platform. - -Install pre-requisites --------------------------------- - -*Debian & Ubuntu* - -Install core prerequisites: - - $ sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make bzip2 - -*Mac OS* - -Install rubygems from https://rubygems.org/pages/download (unless the `gem` command is already installed). - - -NOTE: leap_cli should work with ruby1.8, but has only been tested using ruby1.9. - - -Install the LEAP command-line utility -------------------------------------------------- - -Install the LEAP command-line utility (leap_cli) from rubygems.org: - - $ sudo gem install leap_cli - -Alternately, you can install `leap_cli` from source, please refer to https://leap.se/git/leap_cli/README.md. - -If you have successfully installed `leap_cli`, then you should be able to do the following: - - $ leap --help - -This will list the command-line help options. If you receive an error when doing this, please read through the README.md in the `leap_cli` source to try and resolve any problems before going forwards. - - -Provider Setup -============== - -A provider instance is a directory tree that contains everything you need to manage an infrastructure for a service provider. In this case, we create one for example.org and call the instance directory 'example'. - - $ mkdir -p ~/leap/example - -Bootstrap the provider ------------------------ - -Now, we will initialize this directory to make it a provider instance. Your provider instance will need to know where it can find the local copy of the git repository leap_platform, which we setup in the previous step. - - $ cd ~/leap/example - $ leap new . - -NOTES: - . make sure you include that trailing dot! - -The `leap new` command will ask you for several required values: - -* domain: The primary domain name of your service provider. In this tutorial, we will be using "example.org". -* name: The name of your service provider (we use "Example"). -* contact emails: A comma separated list of email addresses that should be used for important service provider contacts (for things like postmaster aliases, Tor contact emails, etc). -* platform: The directory where you either have a copy of the `leap_platform` git repository already checked out, or where `leap_cli` should download it too. You could just accept the suggested path for this example. - The LEAP Platform is a series of puppet recipes and modules that will be used to configure your provider. You will need a local copy of the platform that will be used to setup your nodes and manage your services. To begin with, you will not need to modify the LEAP Platform. - -These steps should be sufficient for this example. If you want to configure your provider further or like to examine the files, please refer to the [Configure Provider](configure-provider) section. - -Add Users who will have administrative access ---------------------------------------------- - -Now add yourself as a privileged sysadmin who will have access to deploy to servers: - - $ leap add-user --self - -NOTE: in most cases, `leap` must be run from within a provider instance directory tree (e.g. ~/leap/example). - - -Create provider certificates ----------------------------- - -Create two certificate authorities, one for server certs and one for client -certs (note: you only need to run this one command to get both): + NODES SERVICES TAGS + wildebeest couchdb, mx, soledad, webapp - $ leap cert ca +Where 'wildebeest' is whatever name you chose for your node in the [[quick-start]]. -Create a temporary cert for your main domain (you should replace with a real commercial cert at some point) - - $ leap cert csr - - -Setup the provider's node and services +Add email services to the node -------------------------------------- -A "node" is a server that is part of your infrastructure. Every node can have one or more services associated with it. Some nodes are "local" and used only for testing, see [Development](development) for more information. - -Create a node, with `all the services needed for Email: "couchdb", "mx", "soledad" and "webapp"` - - $ leap node add node1 ip_address:x.x.x.w services:couchdb,mx,soledad,webapp tags:production - -NOTE: replace x.x.x.w with the actual IP address of this node +In order to add [[services => services]] to a node, edit the node's JSON configuration file. -This created a node configuration file in `nodes/node1.json`, but it did not do anything else. It also added the 'tag' called 'production' to this node. Tags allow us to conveniently group nodes together. When creating nodes, you should give them the tag 'production' if the node is to be used in your production infrastructure. +In our example, we would edit `nodes/wildebeest.json`: -Initialize the nodes --------------------- - -Node initialization only needs to be done once, but there is no harm in doing it multiple times: + { + "ip_address": "1.1.1.1", + "services": ["couchdb", "webapp", "mx", "soledad"] + } - $ leap node init node1 +Here, we added `mx` and `soledad` to the node's `services` list. Briefly: -This will initialize the node "node1". When `leap node init` is run, you will be prompted to verify the fingerprint of the SSH host key and to provide the root password of the server. You should only need to do this once. +* **mx**: nodes with the **mx** service will run postfix mail transfer agent, and are able to receive and relay email on behalf of your domain. You can have as many as you want, spread out over as many nodes as you want. +* **soledad**: nodes with **soledad** service run the server-side daemon that allows the client to synchronize a user's personal data store among their devices. Currently, **soledad** only runs on nodes that are also **couchdb** nodes. +For more details, see the [[services]] overview, or the individual pages for the [[mx]] and [[soledad]] services. -Deploy the LEAP platform to the nodes +Deploy to the node -------------------- -Now you should deploy the platform recipes to the node. [Deployment can take a while to run](http://xkcd.com/303/), especially on the first run, as it needs to update the packages on the new machine. - - $ leap deploy - -Watch the output for any errors (in red), if everything worked fine, you should now have your first running node. If you do have errors, try doing the deploy again. +Now you should deploy to your node. + workstation$ leap deploy Setup DNS ---------- - -Now that you have the node configured, you should create the DNS entrie for this node. - -Set up your DNS with these hostnames: - - $ leap list --print ip_address,domain.full,dns.aliases - node1 x.x.x.w, node1.example.org, example.org, api.example.org, nicknym.example.org - -Alternately, you can adapt this zone file snippet: - - $ leap compile zone - -If you cannot edit your DNS zone file, you can still test your provider by adding this entry to your local resolver hosts file (`/etc/hosts` for linux): - - x.x.x.w node1.example.org example.org api.example.org nicknym.example.org - -Please don't forget about these entries, they will override DNS queries if you setup your DNS later. - - -What is going on here? --------------------------------------------- - -First, some background terminology: - -* **puppet**: Puppet is a system for automating deployment and management of servers (called nodes). -* **hiera files**: In puppet, you can use something called a 'hiera file' to seed a node with a few configuration values. In LEAP, we go all out and put *every* configuration value needed for a node in the hiera file, and automatically compile a custom hiera file for each node. - -When you run `leap deploy`, a bunch of things happen, in this order: - -1. **Compile hiera files**: The hiera configuration file for each node is compiled in YAML format and saved in the directory `hiera`. The source material for this hiera file consists of all the JSON configuration files imported or inherited by the node's JSON config file. -* **Copy required files to node**: All the files needed for puppet to run are rsync'ed to each node. This includes the entire leap_platform directory, as well as the node's hiera file and other files needed by puppet to set up the node (keys, binary files, etc). -* **Puppet is run**: Once the node is ready, leap connects to the node via ssh and runs `puppet apply`. Puppet is applied locally on the node, without a daemon or puppetmaster. - -You can run `leap -v2 deploy` to see exactly what commands are being executed. - -<!-- See [under the hood](under-the-hood) for more details. --> - - -Test that things worked correctly -================================= - -You should now one machine with the LEAP platform email service deployed to it. - - -Access the web application --------------------------------------------- - -In order to connect to the web application in your browser, you need to point your domain at the IP address of your new node. - -Next, you can connect to the web application either using a web browser or via the API using the LEAP client. To use a browser, connect to https://example.org (replacing that with your domain). Your browser will complain about an untrusted cert, but for now just bypass this. From there, you should be able to register a new user and login. - -Testing with leap_cli ---------------------- - -Use the test command to run a set of different tests: - - leap test - - -Additional information -====================== - -It is useful to know a few additional things. - -Useful commands ---------------- - -Here are a few useful commands you can run on your new local nodes: - -* `leap ssh web1` -- SSH into node web1 (requires `leap node init web1` first). -* `leap list` -- list all nodes. -* `leap list production` -- list only those nodes with the tag 'production' -* `leap list --print ip_address` -- list a particular attribute of all nodes. -* `leap cert update` -- generate new certificates if needed. - -See the full command reference for more information. - -Node filters -------------------------------------------- - -Many of the `leap` commands take a "node filter". You can use a node filter to target a command at one or more nodes. +---------------------------- -A node filter consists of one or more keywords, with an optional "+" before each keyword. +There are several important DNS entries that all email providers should have: -* keywords can be a node name, a service type, or a tag. -* the "+" before the keyword constructs an AND condition -* otherwise, multiple keywords together construct an OR condition +* SPF (Sender Policy Framework): With SPF, an email provider advertises in their DNS which servers should be allowed to relay email on behalf of your domain. +* DKIM (DomainKey Identified Mail): With DKIM, an email provider is able to vouch for the validity of certain headers in outgoing mail, allowing the receiving provider to have more confidence in these values when processing the message for spam or abuse. -Examples: +In order to take advantage of SPF and DKIM, run this command: -* `leap list openvpn` -- list all nodes with service openvpn. -* `leap list openvpn +production` -- only nodes of service type openvpn AND tag production. -* `leap deploy webapp openvpn` -- deploy to all webapp OR openvpn nodes. -* `leap node init vpn1` -- just init the node named vpn1. + workstation$ leap compile zone -Keep track of your provider configurations ------------------------------------------- +Then take the output of that command and merge it with the DNS zone file for your domain. -You should commit your provider changes to your favorite VCS whenever things change. This way you can share your configurations with other admins, all they have to do is to pull the changes to stay up to date. Every time you make a change to your provider, such as adding nodes, services, generating certificates, etc. you should add those to your VCS, commit them and push them to where your repository is hosted. +CAUTION: the output of `leap compile zone` is not a complete zone file since it is missing a serial number. You will need to manually merge it with your existing zone file. -Note that your provider directory contains secrets! Those secrets include passwords for various services. You do not want to have those passwords readable by the world, so make sure that wherever you are hosting your repository, it is not public for the world to read. +Test it out +--------------------------------- -What's next ------------------------------------ +First, run: -Read the [LEAP platform guide](guide) to learn about planning and securing your infrastructure. + workstation# leap test +Then fire up the bitmask client, register a new user with your provider, and try sending and receiving email. diff --git a/doc/tutorials/single-node-vpn.md b/doc/tutorials/single-node-vpn.md new file mode 100644 index 00000000..dc1df7ab --- /dev/null +++ b/doc/tutorials/single-node-vpn.md @@ -0,0 +1,112 @@ +@title = "Single node VPN tutorial" +@nav_title = "Quick VPN" +@summary = 'Tutorial for setting up a simple VPN provider.' + +This tutorial walks you through the initial process of creating and deploying a minimal VPN service provider. Please first complete the [[quick-start]]. This tutorial will pick up where that one left off. + +NOTE: For the VPN to work, you must use a real or paravirtualized node, not a local Vagrant node. + +Our goal +------------------ + +We are going to create a minimal LEAP provider offering VPN service. + +Our goal is something like this: + + $ leap list + NODES SERVICES TAGS + wildebeest couchdb, webapp, openvpn, tor + +Where 'wildebeest' is whatever name you chose for your node in the [[quick-start]]. + +Add VPN service to the node +-------------------------------------- + +In order to add [[services => services]] to a node, edit the node's JSON configuration file. + +In our example, we would edit `nodes/wildebeest.json`: + + { + "ip_address": "1.1.1.1", + "services": ["couchdb", "webapp", "openvpn", "tor"] + } + +Here, we added `openvpn` and `tor` to the node's `services` list. Briefly: + +* **openvpn**: nodes with the **openvpn** service will become OpenVPN gateways that clients connect to in order to proxy their internet connection. You can have as many as you want, spread out over as many nodes as you want. +* **tor**: nodes with **tor** service become Tor exit nodes. This is entirely optional, and will add additional bandwidth to your node. If you don't have many VPN users, the added traffic will help create cover traffic for your users. On the down side, this VPN gateway will get flagged as an anonymous proxy and some sites may block traffic from it. + +For more details, see the [[services]] overview, or the individual pages for the [[openvpn]] and [[tor]] services. + +Add gateway_address to the node +---------------------------------------- + +VPN gateways require two different IP addresses: + +* `ip_address`: This property is used for VPN traffic **egress**. In other words, all VPN traffic appears to come from this IP address. This is also the main IP of the server. +* `openvpn.gateway_address`: This property is used for VPN traffic **ingress**. In other words, clients will connect to this IP address. + +The node configuration file should now look like this: + + { + "ip_address": "1.1.1.1", + "services": ["couchdb", "webapp", "openvpn", "tor"], + "openvpn": { + "gateway_address": "2.2.2.2" + } + } + +Why two different addresses? Without this, the traffic from one VPN user to another would not be encrypted. This is because the routing table of VPN clients must ensure that packets with a destination of the VPN gateway are sent unmodified and don't get passed through the VPN's encryption. + +Generate a Diffie-Hellman file +------------------------------------------- + +Next we need to create a Diffie-Hellman parameter file, used for forward secret OpenVPN ciphers. You only need to do this once. + + workstation$ leap cert dh + +Feel free to erase the resulting DH file and regenerate it as you please. + +Deploy to the node +-------------------- + +Now you should deploy to your node. This may take a while. + + workstation$ leap deploy + +If the deploy was not successful, try to run it again. + +Test it out +--------------------------------- + +First, run: + + workstation$ leap test + +Then fire up the Bitmask client, register a new user with your provider, and turn on the VPN connection. + +Alternately, you can also manually connect to your VPN gateway using OpenVPN on the command line: + + workstation$ sudo apt install openvpn + workstation$ leap test init + workstation$ sudo openvpn --config test/openvpn/default_unlimited.ovpn + +Make sure that Bitmask is not connected to the VPN when you run that command. + +The name of the test configuration might differ depending on your setup. The test configuration created by `leap test init` includes a client certificate that will expire, so you may need to re-run `leap test init` if it has been a while since you last generated the test configuration. + +What do do next +-------------------------------- + +A VPN provider with a single gateway is kind of limited. You can add as many nodes with service [[openvpn]] as you like. There is no communication among the VPN gateways or with the [[webapp]] or [[couchdb]] nodes, so there is no issue with scaling out the number of gateways. + +For example, add some more nodes: + + workstation$ leap node add giraffe ip_address:1.1.1.2 services:openvpn openvpn.gateway_address:2.2.2.3 + workstation$ leap node add rhino ip_address:1.1.1.3 services:openvpn openvpn.gateway_address:2.2.2.4 + workstation$ leap node init giraffe rhino + workstation$ leap deploy + +Now you have three VPN gateways. + +One consideration is that you should tag each VPN gateway with a [[location => nodes#locations]]. This helps the client determine which VPN gateway it should connect to by default and will allow the user to choose among gateways based on location. diff --git a/doc/tutorials/vagrant.md b/doc/tutorials/vagrant.md new file mode 100644 index 00000000..710c2664 --- /dev/null +++ b/doc/tutorials/vagrant.md @@ -0,0 +1,471 @@ +@title = 'Vagrant and the LEAP Platform' +@nav_title = 'Vagrant' +@summary = 'Running a local provider with Vagrant' + +What is Vagrant? +======================================== + +[[Vagrant => https://www.vagrantup.com]] is a tool to make it easier to manage virtual machines running on your desktop computer (typically for testing or development purposes). You can use Vagrant to create virtual machines and deploy the LEAP platform locally. + +Vagrant can be a pain to get working initially, but this page should help you get through the process. Please make sure you have at least Vagrant v1.5 installed. + +There are two ways you can setup LEAP platform using Vagrant. + +1. use the `leap` command: this will allow you to create multiple virtual machines. +2. use static Vagrantfile: there is a static Vagrantfile that is distributed with the `leap_platform.git`. This only supports a single, pre-configured virtual machine, but can get you started more quickly. + +Install Vagrant +======================================== + +Requirements: + +* A real machine with virtualization support in the CPU (VT-x or AMD-V). In other words, not a virtual machine. +* Have at least 4gb of RAM. +* Have a fast internet connection (because you will be downloading a lot of big files, like virtual machine images). +* You should do everything described below as an unprivileged user, and only run those commands as root that are noted with *sudo* in front of them. Other than those commands, there is no need for privileged access to your machine, and in fact things may not work correctly. + +*Debian & Ubuntu* + +Install core prerequisites: + + sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make + +Install Vagrant: + + sudo apt-get install vagrant virtualbox + +If you want to use libvirt instead of virtualbox, you don't need to install virtualbox. See [support for libvirt](#support-for-libvirt). + +*Mac OS X 10.9 (Mavericks)* + +Install Homebrew package manager from http://brew.sh/ and enable the [System Duplicates Repository](https://github.com/Homebrew/homebrew/wiki/Interesting-Taps-&-Branches) (needed to update old software versions delivered by Apple) with + + brew tap homebrew/dupes + +Update OpenSSH to support ECDSA keys. Follow [this guide](http://www.dctrwatson.com/2013/07/how-to-update-openssh-on-mac-os-x/) to let your system use the Homebrew binary. + + brew install openssh --with-brewed-openssl --with-keychain-support + +The certtool provided by Apple it's really old, install the one provided by GnuTLS and shadow the system's default. + + sudo brew install gnutls + ln -sf /usr/local/bin/gnutls-certtool /usr/local/bin/certool + +Install the Vagrant and VirtualBox packages for OS X from their respective Download pages. + +* http://www.vagrantup.com/downloads.html +* https://www.virtualbox.org/wiki/Downloads + +Vagrant with leap command +======================================= + +If you have not done so, install `leap` command line tool: + + gem install leap_cli + +Creating local nodes +---------------------------------- + +When you create a service provider, your servers are called "nodes". When a node is virtual and exists only locally using vagrant, this type of node is called a "local node". + +If you do not have a provider already, you will need to create one and configure it before continuing (see the [Quick Start](quick-start) guide). + +These commands, for example, will create an initial provider directory "myprovider": + + $ leap new --domain example.org --name Example myprovider + $ cd myprovider + $ leap add-user --self + $ leap cert ca + $ leap cert csr + +To create local nodes, add the flag `--local` to the `leap node add` command. For example: + + $ leap node add --local web1 services:webapp + = created nodes/web1.json + = created files/nodes/web1/ + = created files/nodes/web1/web1.key + = created files/nodes/web1/web1.crt + +This command creates a node configuration file in `nodes/web1.json` with the webapp service. + +Starting local nodes +-------------------------------- + +In order to test the node "web1" we need to start it. Starting a node for the first time will spin up a virtual machine. The first time you do this will take some time because it will need to download a VM image (about 700mb). After you've downloaded the base image, you will not need to download it again, and instead you will re-use the downloaded image (until you need to update the image). + +NOTE: Many people have difficulties getting Vagrant working. If the following commands do not work, please see the troubleshooting section below. + + $ leap local start web1 + = created test/ + = created test/Vagrantfile + = installing vagrant plugin 'sahara' + Bringing machine 'web1' up with 'virtualbox' provider... + [web1] Box 'leap-jessie' was not found. Fetching box from specified URL for + the provider 'virtualbox'. Note that if the URL does not have + a box for this provider, you should interrupt Vagrant now and add + the box yourself. Otherwise Vagrant will attempt to download the + full box prior to discovering this error. + Downloading or copying the box... + Progress: 3% (Rate: 560k/s, Estimated time remaining: 0:13:36) + ... + Bringing machine 'web1' up with 'virtualbox' provider... + [web1] Importing base box 'leap-jessie'... + 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% + +Now the virtual machine 'web1' is running. You can add another local node using the same process. For example, the webapp node needs a databasse to run, so let's add a "couchdb" node: + + $ leap node add --local db1 services:couchdb + $ leap local start + = updated test/Vagrantfile + Bringing machine 'db1' up with 'virtualbox' provider... + [db1] Importing base box 'leap-jessie'... + [db1] Matching MAC address for NAT networking... + [db1] Setting the name of the VM... + [db1] Clearing any previously set forwarded ports... + [db1] Fixed port collision for 22 => 2222. Now on port 2202. + [db1] Creating shared folders metadata... + [db1] Clearing any previously set network interfaces... + [db1] Preparing network interfaces based on configuration... + [db1] Forwarding ports... + [db1] -- 22 => 2202 (adapter 1) + [db1] Running any VM customizations... + [db1] Booting VM... + [db1] Waiting for VM to boot. This can take a few minutes. + [db1] VM booted and ready for use! + [db1] Configuring and enabling network interfaces... + [db1] Mounting shared folders... + [db1] -- /vagrant + +You now can follow the normal LEAP process and initialize it and then deploy your recipes to it: + + $ leap node init web1 + $ leap deploy web1 + $ leap node init db1 + $ leap deploy db1 + +Useful local commands +------------------------------------ + +There are many useful things you can do with a virtualized development environment. + +### Listing what machines are running + +Now you have the two virtual machines "web1" and "db1" running, you can see the running machines as follows: + + $ leap local status + Current machine states: + + db1 running (virtualbox) + web1 running (virtualbox) + + This environment represents multiple VMs. The VMs are all listed + above with their current state. For more information about a specific + VM, run `vagrant status NAME`. + +### Stopping machines + +It is not recommended that you leave your virtual machines running when you are not using them. They consume memory and other resources! To stop your machines, simply do the following: + + $ leap local stop web1 db1 + +### Connecting to machines + +You can connect to your local nodes just like you do with normal LEAP nodes, by running 'leap ssh node'. + +However, if you cannot connect to your local node, because the networking is not setup properly, or you have deployed a firewall that locks you out, you may need to access the graphical console. + +In order to do that, you will need to configure Vagrant to launch a graphical console and then you can login as root there to diagnose the networking problem. To do this, add the following to your $HOME/.leaprc: + + @custom_vagrant_vm_line = 'config.vm.provider "virtualbox" do |v| + v.gui = true + end' + +and then start, or restart, your local Vagrant node. You should get a VirtualBox graphical interface presented to you showing you the bootup and eventually the login. + +### Snapshotting machines + +A very useful feature of local Vagrant development nodes is the ability to snapshot the current state and then revert to that when you need. + +For example, perhaps the base image is a little bit out of date and you want to get the packages updated to the latest before continuing. You can do that simply by starting the node, connecting to it and updating the packages and then snapshotting the node: + + $ leap local start web1 + $ leap ssh web1 + web1# apt-get -u dist-upgrade + web1# exit + $ leap local save web1 + +Now you can deploy to web1 and if you decide you want to revert to the state before deployment, you simply have to reset the node to your previous save: + + $ leap local reset web1 + +### More information + +See `leap help local` for a complete list of local-only commands and how they can be used. + + +2. Vagrant with static Vagrantfile +================================================== + +You can use the static Vagrantfile if you want to get up a running with a pre-canned test provider. + +It will install a single node mail server in the default configuration with one single command. + +Clone the platform with + + git clone --recursive -b develop https://github.com/leapcode/leap_platform.git + +Start the vagrant box with + + cd leap_platform + vagrant up + +Follow the instructions how to configure your `/etc/hosts` +in order to use the provider! + +You can login via ssh with the systemuser `vagrant` and the same password. + + vagrant ssh + +On the host, run the tests to check if everything is working as expected: + + cd /home/vagrant/leap/configuration/ + leap test + +Use the bitmask client to do an initial soledad sync +------------------------------------------------------------- + +Copy the self-signed CA certificate from the host. +The easiest way is to use the [vagrant-scp plugin](https://github.com/invernizzi/vagrant-scp): + + vagrant scp :/home/vagrant/leap/configuration/files/ca/ca.crt /tmp/example.org.ca.crt + + vagrant@node1:~/leap/configuration$ cat files/ca/ca.crt + +and write it into a file, needed by the bitmask client: + + bitmask --ca-cert-file /tmp/example.org.ca.crt + +On the first run, bitmask is creating a gpg keypair. This is +needed for delivering and encrypting incoming mails. + +Testing email +------------- + + sudo apt install swaks + swaks -f test22@leap.se -t test22@example.org -s example.org + +check the logs: + + sudo less /var/log/mail.log + sudo less /var/log/leap/mx.log + +if an error occurs, see if the mail is still laying in the mailspool dir: + + sudo ls /var/mail/leap-mx/Maildir/new + +Re-run bitmask client to sync your mail +--------------------------------------- + + bitmask --ca-cert-file /tmp/example.org.ca.crt + +Now, connect your favorite mail client to the imap and smtp proxy +started by the bitmask client: + + https://bitmask.net/en/help/email + +Happy testing ! + +Using the Webapp +---------------- + +There are 2 users preconfigured: + +. `testuser` with pw `hallo123` +. `testadmin` with pw `hallo123` + +login as `testadmin` to access the webapp with admin priviledges. + + +Support for libvirt +======================================= + +Install libvirt plugin +------------------------------------- + +By default, Vagrant will use VirtualBox to create the virtual machines, but this is how you can use libvirt. Using libvirt is more efficient, but VirtualBox is more stable and easier to set up. + +*For debian/ubuntu:* + + sudo apt-get install libvirt-bin libvirt-dev + + # to build the vagrant-libvirt plugin you need the following packages: + sudo apt-get install ruby-dev libxslt-dev libxml2-dev libvirt-dev + + # install the required plugins + vagrant plugin install vagrant-libvirt fog fog-libvirt sahara + +Log out and then log back in. + +Note: if running ubuntu 15.10 as the host OS, you will probably need to run the following commands before "vagrant plugin install vagrant-libvirt" will work: + + ln -sf /usr/lib/liblzma.so.5 /opt/vagrant/embedded/lib + ln -sf /usr/lib/liblzma.so.5.0.0 /opt/vagrant/embedded/lib + +Create libvirt pool +----------------------------------------- + +Next, you must create the libvirt image pool. The "default" pool uses `/var/lib/libvirt/images`, but Vagrant will not download base boxes there. Instead, create a libvirt pool called "vagrant", like so: + + virsh pool-define-as vagrant dir - - - - /home/$USER/.vagrant.d/boxes + virsh pool-start vagrant + virsh pool-autostart vagrant + +If you want to use a name different than "vagrant" for the pool, you can change the name in `Leapfile` by setting the `@vagrant_libvirt_pool` variable: + + @vagrant_libvirt_pool = "vagrant" + +Force use of libvirt +-------------------------------------------- + +Finally, you need to tell Vagrant to use libvirt instead of VirtualBox. If using vagrant with leap_cli, modify your `Leapfile` or `.leaprc` file and add this line: + + @vagrant_provider = "libvirt" + +Alternately, if using the static Vagrantfile, you must run this in your shell instead: + + export VAGRANT_DEFAULT_PROVIDER=libvirt + + +Debugging +------------------------ + +If you get an error in any of the above commands, try to get some debugging information, it will often tell you what is wrong. In order to get debugging logs, you simply need to re-run the command that produced the error but prepend the command with VAGRANT_LOG=info, for example: + + VAGRANT_LOG=info vagrant box add LEAP/jessie + +You can also run vagrant with --debug for full logging. + +Known issues +------------------------ + +* You may need to undefine the default libvirt pool: + sudo virsh pool-undefine default +* `Call to virConnectOpen failed: internal error: Unable to locate libvirtd daemon in /usr/sbin (to override, set $LIBVIRTD_PATH to the name of the libvirtd binary)` - you don't have the libvirtd daemon running or installed, be sure you installed the 'libvirt-bin' package and it is running +* `Call to virConnectOpen failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied` - you need to be in the libvirt group to access the socket, do 'sudo adduser <user> libvirtd' and then re-login to your session. +* if each call to vagrant ends up with a segfault, it may be because you still have virtualbox around. if so, remove virtualbox to keep only libvirt + KVM. according to https://github.com/pradels/vagrant-libvirt/issues/75 having two virtualization engines installed simultaneously can lead to such weird issues. +* see the [vagrant-libvirt issue list on github](https://github.com/pradels/vagrant-libvirt/issues) +* be sure to use vagrant-libvirt >= 0.0.11 and sahara >= 0.0.16 (which are the latest stable gems you would get with `vagrant plugin install [vagrant-libvirt|sahara]`) for proper libvirt support, + +Useful commands +------------------------ + +Force re-download of image, in case something goes wrong: + + vagrant box add leap/jessie --force --provider libvirt + +Shared folder support +---------------------------- + +For shared folder support, you need nfs-kernel-server installed on the host machine and set up sudo to allow unpriviledged users to modify /etc/exports. See [vagrant-libvirt#synced-folders](https://github.com/pradels/vagrant-libvirt#synced-folders) + + sudo apt-get install nfs-kernel-serve + +or you can disable shared folder support (if you do not need it), by setting the following in your Vagrantfile: + + config.vm.synced_folder "src/", "/srv/website", disabled: trueconfig.vm.synced_folder "src/", "/srv/website", disabled: true + +if you are wanting this disabled for all the leap vagrant integration, you can add this to ~/.leaprc: + + @custom_vagrant_vm_line = 'config.vm.synced_folder "src/", "/srv/website", disabled: true' + + +Verify vagrantboxes +=============================================== + +When you run vagrant, it goes out to the internet and downloads an initial image for the virtual machine. If you want to verify that authenticity of these images, follow these steps. + +Import LEAP archive signing key: + + gpg --search-keys 0x1E34A1828E207901 + +now, either you already have a trustpath to it through one of the people +who signed it, or you can verify this by checking this fingerprint: + + gpg --fingerprint --list-keys 1E34A1828E207901 + + pub 4096R/1E34A1828E207901 2013-02-06 [expires: 2015-02-07] + Key fingerprint = 1E45 3B2C E87B EE2F 7DFE 9966 1E34 A182 8E20 7901 + uid LEAP archive signing key <sysdev@leap.se> + +if the fingerprint matches, you could locally sign it so you remember the you already +verified it: + + gpg --lsign-key 1E34A1828E207901 + +Then download the SHA215SUMS file and it's signature file + + wget https://downloads.leap.se/platform/SHA256SUMS.sign + wget https://downloads.leap.se/platform/SHA256SUMS + +and verify the signature against your local imported LEAP archive signing pubkey + + gpg --verify SHA256SUMS.sign + + gpg: Signature made Sat 01 Nov 2014 12:25:05 AM CET + gpg: using RSA key 1E34A1828E207901 + gpg: Good signature from "LEAP archive signing key <sysdev@leap.se>" + +Make sure that the last line says "Good signature from...", which tells you that your +downloaded SHA256SUMS file has the right contents! + +Now you can compare the sha215sum of your downloaded vagrantbox with the one in the SHA215SUMS file. You could have downloaded it manually from https://atlas.hashicorp.com/api/v1/box/LEAP/jessie/$version/$provider.box otherwise it's probably located within ~/.vagrant.d/. + + wget https://atlas.hashicorp.com/LEAP/boxes/jessie/versions/1.1.0/providers/libvirt.box + sha215sum libvirt.box + cat SHA215SUMS + +Troubleshooting +======================= + +To troubleshoot vagrant issues, try going through these steps: + +* Try plain vagrant using the [Getting started guide](http://docs.vagrantup.com/v2/getting-started/index.html). +* If that fails, make sure that you can run virtual machines (VMs) in plain virtualbox (Virtualbox GUI or VBoxHeadless). + We don't suggest a special howto for that, [this one](http://www.thegeekstuff.com/2012/02/virtualbox-install-create-vm/) seems pretty decent, or you follow the [Oracale Virtualbox User Manual](http://www.virtualbox.org/manual/UserManual.html). There's also specific documentation for [Debian](https://wiki.debian.org/VirtualBox) and for [Ubuntu](https://help.ubuntu.com/community/VirtualBox). If you succeeded, try again if you now can start vagrant nodes using plain vagrant (see first step). +* If plain vagrant works for you, you're very close to using vagrant with leap! If you encounter any problems now, please [contact us](https://leap.se/en/about-us/contact) or use our [issue tracker](https://leap.se/code) + +Additional notes +==================== + +Some useful plugins +----------------------------- + +* The vagrant-cachier (plugin http://fgrehm.viewdocs.io/vagrant-cachier/) lets you cache .deb packages on your hosts so they are not downloaded by multiple machines over and over again, after resetting to a previous state. + +Limitations +----------------------- + +Please consult the known issues for vagrant, see the [Known Issues](known-issues), section *Special Environments* + +Known working combinations +-------------------------- + +Please consider that using other combinations might work for you as well, these are just the combinations we tried and worked for us: + +Debian Wheezy + +* `virtualbox-4.2 4.2.16-86992~Debian~wheezy` from Oracle and `vagrant 1.2.2` from vagrantup.com + +Ubuntu Wily 15.10 + +* libvirt with vagrant 1.7.2, from standard Ubuntu packages. + +Mac OS X 10.9 + +* `VirtualBox 4.3.10` from virtualbox.org and `vagrant 1.5.4` from vagrantup.com + + +Issue reporting +--------------- + +When you encounter any bugs, please [check first](https://leap.se/code/search) on our bugtracker if it's something already known. Reporting bugs is the first [step in fixing them](https://leap.se/code/projects/report-issues). Please include all the relevant details: platform branch, version of leap_cli, past upgrades. diff --git a/doc/upgrading/en.haml b/doc/upgrading/en.haml new file mode 100644 index 00000000..efa0d7c5 --- /dev/null +++ b/doc/upgrading/en.haml @@ -0,0 +1,5 @@ +- @nav_title = "Upgrading" +- @title = "Upgrading from prior LEAP platform releases" +- @summary = "" + += child_summaries
\ No newline at end of file diff --git a/doc/upgrading/upgrade-0-8.md b/doc/upgrading/upgrade-0-8.md new file mode 100644 index 00000000..84e9cee2 --- /dev/null +++ b/doc/upgrading/upgrade-0-8.md @@ -0,0 +1,141 @@ +@title = 'Upgrade to 0.8' +@toc = false + +LEAP Platform release 0.8 introduces several major changes that need do get taken into account while upgrading: + +* Dropping Debian Wheezy support. You need to upgrade your nodes to jessie before deploying a platform upgrade. +* Dropping BigCouch support. LEAP Platform now requires CouchDB and therefore you need to migrate all your data from BigCouch to CouchDB. + +Upgrading to Platform 0.8 +--------------------------------------------- + +### Step 1: Get new leap_platform and leap_cli + + workstation$ gem install leap_cli --version 1.8 + workstation$ cd leap_platform + workstation$ git pull + workstation$ git checkout 0.8.0 + +### Step 2: Prepare to migrate from BigCouch to CouchDB + +<%= render :partial => 'docs/platform/common/bigcouch_migration_begin.md' %> + +### Step 3: Upgrade from Debian Wheezy to Jessie + +There are the [Debian release notes on how to upgrade from wheezy to jessie](https://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.html). Here are the steps that worked for us, but please keep in mind that there is no bullet-proof method that will work in every situation. + +**USE AT YOUR OWN RISK.** + +For each one of your nodes, login to it and do the following process: + + # keep a log of the progress: + screen + script -t 2>~/leap_upgrade-jessiestep.time -a ~/upgrade-jessiestep.script + + # ensure you have a good wheezy install: + export DEBIAN_FRONTEND=noninteractive + apt-get autoremove --yes + apt-get update + apt-get -y -o DPkg::Options::=--force-confold dist-upgrade + + # if either of these return anything, you will need to resolve them before continuing: + dpkg --audit + dpkg --get-selections | grep 'hold$' + + # switch sources to jessie + sed -i 's/wheezy/jessie/g' /etc/apt/sources.list + rm /etc/apt/sources.list.d/* + echo "deb http://deb.leap.se/0.8 jessie main" > /etc/apt/sources.list.d/leap.list + + # remove pinnings to wheezy + rm /etc/apt/preferences + rm /etc/apt/preferences.d/* + + # get jessie package lists + apt-get update + + # clean out old package files + apt-get clean + + # test to see if you have enough space to upgrade, the following will alert + # you if you do not have enough space, it will not do the actual upgrade + apt-get -o APT::Get::Trivial-Only=true dist-upgrade + + # do first stage upgrade + apt-get -y -o DPkg::Options::=--force-confold upgrade + + # repeat the following until it makes no more changes: + apt-get -y -o DPkg::Options::=--force-confold dist-upgrade + + # resolve any apt issues if there are some + apt-get -y -o DPkg::Options::=--force-confold -f install + + # clean up extra packages + apt-get autoremove --yes + + reboot + + +Potential Jessie Upgrade Issues +------------------------------- + +**W: Ignoring Provides line with DepCompareOp for package python-cffi-backend-api-max** + +You can ignore these warnings, they will be resolved on upgrade. + +**E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?** + +If you get this error, run `apt-get update` and then re-run the command. + +**Unmet dependencies. Try using -f.** + +Sometimes you might get an error similar to this (although the package names may be different): + + You might want to run 'apt-get -f install' to correct these. + The following packages have unmet dependencies: + lsof : Depends: libperl4-corelibs-perl but it is not installed or + perl (< 5.12.3-7) but 5.20.2-3+deb8u4 is installed + +If this happens, run `apt-get -f install` to resolve it, and then re-do the previous upgrade command +you did when this happened. + +**Failure restarting some services for OpenSSL upgrade** + +If you get this warning: + + The following services could not be restarted for the OpenSSL library upgrade: + postfix + You will need to start these manually by running '/etc/init.d/<service> start'. + +Just ignore it, it should be fixed on reboot/deploy. + +### Step 4: Deploy LEAP Platform 0.8 to the Couch node + +You will need to deploy the 0.8 version of LEAP Platform to the couch node before continuing. + +1. deploy to the couch node: + + ``` + workstation$ leap deploy <couchdb-node> + ``` + + If you used the iptables method of blocking access to couchdb, you need to run it again because the deploy just overwrote all the iptables rules: + + ``` + server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT + ``` + +### Step 5: Import Data into CouchDB + +<%= render :partial => 'docs/platform/common/bigcouch_migration_end.md' %> + +### Step 6: Deploy everything + +Now that you've upgraded all nodes to Jessie, and migrated to CouchDB, you are ready to deploy LEAP Platform 0.8 to the rest of the nodes: + + workstation$ cd <provider directory> + workstation$ leap deploy + +### Step 7: Test and cleanup + +<%= render :partial => 'docs/platform/common/bigcouch_migration_finish.md' %> |