From 0a09a6e6f247729457d15480f8d2b9bb0b89ae5e Mon Sep 17 00:00:00 2001 From: elijah Date: Mon, 29 Aug 2016 22:55:41 -0700 Subject: Updated (very out of date) docs and README.md --- README.md | 117 ++-- doc/common/_bigcouch_migration.md | 117 ---- doc/common/_bigcouch_migration_begin.md | 66 -- doc/common/_bigcouch_migration_end.md | 26 - doc/common/_bigcouch_migration_finish.md | 10 - doc/details/development.md | 78 --- doc/details/en.haml | 4 - doc/details/faq.md | 71 -- doc/details/ports.md | 92 --- doc/details/under-the-hood.md | 40 -- doc/en.md | 82 --- doc/guide/commands.md | 559 ---------------- doc/guide/config.md | 360 ----------- doc/guide/domains.md | 129 ---- doc/guide/en.haml | 4 - doc/guide/environments.md | 75 --- doc/guide/getting-started.md | 145 ----- doc/guide/keys-and-certificates.md | 272 -------- doc/guide/miscellaneous.md | 14 - doc/guide/nodes.md | 87 --- doc/guide/provider-configuration.md | 79 --- doc/service-diagram.odg | Bin 12131 -> 0 bytes doc/service-diagram.png | Bin 25988 -> 0 bytes doc/services/couchdb.md | 159 ----- doc/services/en.md | 80 --- doc/services/monitor.md | 36 -- doc/services/mx.md | 35 - doc/services/openvpn.md | 49 -- doc/services/soledad.md | 12 - doc/services/tor.md | 32 - doc/services/webapp.md | 293 --------- doc/troubleshooting/en.haml | 3 - doc/troubleshooting/known-issues.md | 115 ---- doc/troubleshooting/tests.md | 70 -- doc/troubleshooting/where-to-look.md | 267 -------- doc/tutorials/en.haml | 4 - doc/tutorials/quick-start.md | 230 ------- doc/tutorials/single-node-email.md | 69 -- doc/tutorials/single-node-vpn.md | 112 ---- doc/tutorials/vagrant.md | 471 -------------- doc/upgrading/en.haml | 5 - doc/upgrading/upgrade-0-8.md | 141 ---- docs/en/commands.html | 0 docs/en/details.html | 138 ++++ docs/en/details/development.html | 226 +++++++ docs/en/details/development/index.html | 226 +++++++ docs/en/details/faq.html | 213 ++++++ docs/en/details/faq/index.html | 213 ++++++ docs/en/details/ports.html | 209 ++++++ docs/en/details/ports/index.html | 209 ++++++ docs/en/details/under-the-hood.html | 168 +++++ docs/en/details/under-the-hood/index.html | 168 +++++ docs/en/guide.html | 192 ++++++ docs/en/guide/commands.html | 784 +++++++++++++++++++++++ docs/en/guide/commands/index.html | 784 +++++++++++++++++++++++ docs/en/guide/config.html | 584 +++++++++++++++++ docs/en/guide/config/index.html | 584 +++++++++++++++++ docs/en/guide/domains.html | 298 +++++++++ docs/en/guide/domains/index.html | 298 +++++++++ docs/en/guide/environments.html | 228 +++++++ docs/en/guide/environments/index.html | 228 +++++++ docs/en/guide/getting-started.html | 317 +++++++++ docs/en/guide/getting-started/index.html | 317 +++++++++ docs/en/guide/keys-and-certificates.html | 480 ++++++++++++++ docs/en/guide/keys-and-certificates/index.html | 480 ++++++++++++++ docs/en/guide/miscellaneous.html | 145 +++++ docs/en/guide/miscellaneous/index.html | 145 +++++ docs/en/guide/nodes.html | 231 +++++++ docs/en/guide/nodes/index.html | 231 +++++++ docs/en/guide/provider-configuration.html | 223 +++++++ docs/en/guide/provider-configuration/index.html | 223 +++++++ docs/en/guide/virtual-machines.html | 299 +++++++++ docs/en/guide/virtual-machines/index.html | 299 +++++++++ docs/en/services.html | 251 ++++++++ docs/en/services/couchdb.html | 328 ++++++++++ docs/en/services/couchdb/index.html | 328 ++++++++++ docs/en/services/index.html | 251 ++++++++ docs/en/services/monitor.html | 186 ++++++ docs/en/services/monitor/index.html | 186 ++++++ docs/en/services/mx.html | 168 +++++ docs/en/services/mx/index.html | 168 +++++ docs/en/services/openvpn.html | 178 +++++ docs/en/services/openvpn/index.html | 178 +++++ docs/en/services/soledad.html | 136 ++++ docs/en/services/soledad/index.html | 136 ++++ docs/en/services/tor.html | 161 +++++ docs/en/services/tor/index.html | 161 +++++ docs/en/services/webapp.html | 479 ++++++++++++++ docs/en/services/webapp/index.html | 479 ++++++++++++++ docs/en/troubleshooting.html | 129 ++++ docs/en/troubleshooting/known-issues.html | 238 +++++++ docs/en/troubleshooting/known-issues/index.html | 238 +++++++ docs/en/troubleshooting/tests.html | 201 ++++++ docs/en/troubleshooting/tests/index.html | 201 ++++++ docs/en/troubleshooting/where-to-look.html | 451 +++++++++++++ docs/en/troubleshooting/where-to-look/index.html | 451 +++++++++++++ docs/en/tutorials.html | 138 ++++ docs/en/tutorials/quick-start.html | 446 +++++++++++++ docs/en/tutorials/quick-start/index.html | 446 +++++++++++++ docs/en/tutorials/single-node-email.html | 200 ++++++ docs/en/tutorials/single-node-email/index.html | 200 ++++++ docs/en/tutorials/single-node-vpn.html | 250 ++++++++ docs/en/tutorials/single-node-vpn/index.html | 250 ++++++++ docs/en/tutorials/vagrant.html | 724 +++++++++++++++++++++ docs/en/tutorials/vagrant/index.html | 724 +++++++++++++++++++++ docs/en/tutorials/vagrant/known-issues.html | 0 docs/en/tutorials/vagrant/quick-start.html | 0 docs/en/upgrading.html | 111 ++++ docs/en/upgrading/upgrade-0-8.html | 334 ++++++++++ docs/index.html | 190 ++++++ docs/robots.txt.html | 0 111 files changed, 18890 insertions(+), 4556 deletions(-) delete mode 100644 doc/common/_bigcouch_migration.md delete mode 100644 doc/common/_bigcouch_migration_begin.md delete mode 100644 doc/common/_bigcouch_migration_end.md delete mode 100644 doc/common/_bigcouch_migration_finish.md delete mode 100644 doc/details/development.md delete mode 100644 doc/details/en.haml delete mode 100644 doc/details/faq.md delete mode 100644 doc/details/ports.md delete mode 100644 doc/details/under-the-hood.md delete mode 100644 doc/en.md delete mode 100644 doc/guide/commands.md delete mode 100644 doc/guide/config.md delete mode 100644 doc/guide/domains.md delete mode 100644 doc/guide/en.haml delete mode 100644 doc/guide/environments.md delete mode 100644 doc/guide/getting-started.md delete mode 100644 doc/guide/keys-and-certificates.md delete mode 100644 doc/guide/miscellaneous.md delete mode 100644 doc/guide/nodes.md delete mode 100644 doc/guide/provider-configuration.md delete mode 100644 doc/service-diagram.odg delete mode 100644 doc/service-diagram.png delete mode 100644 doc/services/couchdb.md delete mode 100644 doc/services/en.md delete mode 100644 doc/services/monitor.md delete mode 100644 doc/services/mx.md delete mode 100644 doc/services/openvpn.md delete mode 100644 doc/services/soledad.md delete mode 100644 doc/services/tor.md delete mode 100644 doc/services/webapp.md delete mode 100644 doc/troubleshooting/en.haml delete mode 100644 doc/troubleshooting/known-issues.md delete mode 100644 doc/troubleshooting/tests.md delete mode 100644 doc/troubleshooting/where-to-look.md delete mode 100644 doc/tutorials/en.haml delete mode 100644 doc/tutorials/quick-start.md delete mode 100644 doc/tutorials/single-node-email.md delete mode 100644 doc/tutorials/single-node-vpn.md delete mode 100644 doc/tutorials/vagrant.md delete mode 100644 doc/upgrading/en.haml delete mode 100644 doc/upgrading/upgrade-0-8.md create mode 100644 docs/en/commands.html create mode 100644 docs/en/details.html create mode 100644 docs/en/details/development.html create mode 100644 docs/en/details/development/index.html create mode 100644 docs/en/details/faq.html create mode 100644 docs/en/details/faq/index.html create mode 100644 docs/en/details/ports.html create mode 100644 docs/en/details/ports/index.html create mode 100644 docs/en/details/under-the-hood.html create mode 100644 docs/en/details/under-the-hood/index.html create mode 100644 docs/en/guide.html create mode 100644 docs/en/guide/commands.html create mode 100644 docs/en/guide/commands/index.html create mode 100644 docs/en/guide/config.html create mode 100644 docs/en/guide/config/index.html create mode 100644 docs/en/guide/domains.html create mode 100644 docs/en/guide/domains/index.html create mode 100644 docs/en/guide/environments.html create mode 100644 docs/en/guide/environments/index.html create mode 100644 docs/en/guide/getting-started.html create mode 100644 docs/en/guide/getting-started/index.html create mode 100644 docs/en/guide/keys-and-certificates.html create mode 100644 docs/en/guide/keys-and-certificates/index.html create mode 100644 docs/en/guide/miscellaneous.html create mode 100644 docs/en/guide/miscellaneous/index.html create mode 100644 docs/en/guide/nodes.html create mode 100644 docs/en/guide/nodes/index.html create mode 100644 docs/en/guide/provider-configuration.html create mode 100644 docs/en/guide/provider-configuration/index.html create mode 100644 docs/en/guide/virtual-machines.html create mode 100644 docs/en/guide/virtual-machines/index.html create mode 100644 docs/en/services.html create mode 100644 docs/en/services/couchdb.html create mode 100644 docs/en/services/couchdb/index.html create mode 100644 docs/en/services/index.html create mode 100644 docs/en/services/monitor.html create mode 100644 docs/en/services/monitor/index.html create mode 100644 docs/en/services/mx.html create mode 100644 docs/en/services/mx/index.html create mode 100644 docs/en/services/openvpn.html create mode 100644 docs/en/services/openvpn/index.html create mode 100644 docs/en/services/soledad.html create mode 100644 docs/en/services/soledad/index.html create mode 100644 docs/en/services/tor.html create mode 100644 docs/en/services/tor/index.html create mode 100644 docs/en/services/webapp.html create mode 100644 docs/en/services/webapp/index.html create mode 100644 docs/en/troubleshooting.html create mode 100644 docs/en/troubleshooting/known-issues.html create mode 100644 docs/en/troubleshooting/known-issues/index.html create mode 100644 docs/en/troubleshooting/tests.html create mode 100644 docs/en/troubleshooting/tests/index.html create mode 100644 docs/en/troubleshooting/where-to-look.html create mode 100644 docs/en/troubleshooting/where-to-look/index.html create mode 100644 docs/en/tutorials.html create mode 100644 docs/en/tutorials/quick-start.html create mode 100644 docs/en/tutorials/quick-start/index.html create mode 100644 docs/en/tutorials/single-node-email.html create mode 100644 docs/en/tutorials/single-node-email/index.html create mode 100644 docs/en/tutorials/single-node-vpn.html create mode 100644 docs/en/tutorials/single-node-vpn/index.html create mode 100644 docs/en/tutorials/vagrant.html create mode 100644 docs/en/tutorials/vagrant/index.html create mode 100644 docs/en/tutorials/vagrant/known-issues.html create mode 100644 docs/en/tutorials/vagrant/quick-start.html create mode 100644 docs/en/upgrading.html create mode 100644 docs/en/upgrading/upgrade-0-8.html create mode 100644 docs/index.html create mode 100644 docs/robots.txt.html diff --git a/README.md b/README.md index a8e46edb..999e0087 100644 --- a/README.md +++ b/README.md @@ -3,102 +3,93 @@ Leap Platform [![Build Status](https://0xacab.org/leap/platform/badges/develop/build.svg)](https://0xacab.org/leap/platform/commits/develop) -The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment. Its goal is to make it as painless as possible for sysadmins to deploy and maintain a service provider's infrastructure for secure communication. These recipes define an abstract service provider. It is a set of Puppet modules designed to work together to provide to sysadmins everything they need to manage a service provider infrastructure that provides secure communication services. +The LEAP Platform is set of complementary packages and server recipes to +automate the maintenance of LEAP services in a hardened Debian environment. Its +goal is to make it as painless as possible for sysadmins to deploy and maintain +a service provider's infrastructure for secure communication. These recipes +define an abstract service provider. It is a set of Puppet modules designed to +work together to provide to sysadmins everything they need to manage a service +provider infrastructure that provides secure communication services. Getting started ============================= -It is highly recommended that you start by reading the overview of the [LEAP Platform](https://leap.se/docs/platform) and then begin with the [Quick Start tutorial](https://leap.se/en/docs/platform/tutorials/quick-start) to walk through a test environment setup to get familiar with how things work before deploying to live servers. +It is highly recommended that you start by reading the overview of the [LEAP +Platform](https://leap.se/docs/platform) and then begin with the [Quick Start +tutorial](https://leap.se/en/docs/platform/tutorials/quick-start) to walk +through a test environment setup to get familiar with how things work before +deploying to live servers. -An offline copy of this documentation is contained in the `doc` subdirectory. For more current updates to the documentation, visit the website. +An offline copy of this documentation is contained in the `docs` subdirectory: + + cd leap_platform + gnome-open docs/index.html Requirements ------------------- +----------------------------- -For testing a virtual deployment simulated on your computer, you will need a fairly recent computer x86_64 with hardware virtualization features (AMD-V or VT-x) and plenty of RAM. If you follow the "Quick Start" documentation we will walk you through using Vagrant to setup a test deployment. +For testing a virtual deployment simulated on your computer, you will need a +fairly recent computer x86_64 with hardware virtualization features (AMD-V or +VT-x) and plenty of RAM. If you follow the "Quick Start" documentation we will +walk you through using Vagrant to setup a test deployment. -For a live deployment of the platform, the number of servers that is required depends on your needs and which services you want to deploy. At the moment, the LEAP Platform supports servers with a base Debian Wheezy installation. +For a live deployment of the platform, the number of servers that is required +depends on your needs and which services you want to deploy. At the moment, the +LEAP Platform supports servers with a base Debian Jessie installation. Troubleshooting ============================= If you have a problem, we are interested in fixing it! -If you have a problem, be sure to have a look at the [Known Issues](https://leap.se/docs/platform/known-issues) to see if your issue is detailed there. +If you have a problem, be sure to have a look at the [Known +Issues](https://leap.se/docs/platform/known-issues) to see if your issue is +detailed there. -If not, the best way for us to solve your problem is if you provide to us the complete log of what you did, and the output that was produced. Please don't cut out what appears to be useless information and only include the error that you received, instead copy and paste the complete log so that we can better determine the overall situation. If you can run the same command that produced the error with a raised verbosity level (such as -v2), that provides us with more useful debugging information. +If not, the best way for us to solve your problem is if you provide to us the +complete log of what you did, and the output that was produced. Please don't +cut out what appears to be useless information and only include the error that +you received, instead copy and paste the complete log so that we can better +determine the overall situation. If you can run the same command that produced +the error with a raised verbosity level (such as -v2), that provides us with +more useful debugging information. -To capture the log, you can copy from the console, or run `leap --log FILE` or edit Leapfile to include `@log = '/tmp/leap.log'`. +To capture the log, you can copy from the console, or run `leap --log FILE` or +edit Leapfile to include `@log = '/tmp/leap.log'`. -Visit https://leap.se/en/docs/get-involved/communication for details on how to contact the developers. +Visit https://leap.se/en/docs/get-involved/communication for details on how to +contact the developers. Known issues -============ - -The following issues are known to exist in 0.5.2 and later: - -CouchDB Sync ------------- -You can't deploy new couchdb nodes after one or more have been deployed. Make *sure* that you configure and deploy all your couchdb nodes when first creating your provider. The problem is that we dont not have a clean way of adding couch nodes after initial creation of the databases, so any nodes added after result in improperly synchronized data. See Bug [#5601](https://leap.se/code/issues/5601) for more information. - -User setup and ssh ------------------- - -. if you aren't using a single ssh key, but have different ones, you will need to define the following at the top of your ~/.ssh/config: - HostName - IdentityFile - - (see: https://leap.se/code/issues/2946 and https://leap.se/code/issues/3002) - -. If the ssh host key changes, you need to run node init again (see: https://leap.se/en/docs/platform/guide#Working.with.SSH) - -. At the moment, only ECDSA ssh host keys are supported. If you get the following error: `= FAILED ssh-keyscan: no hostkey alg (must be missing an ecdsa public host key)` then you should confirm that you have the following line defined in your server's **/etc/ssh/sshd_config**: `HostKey /etc/ssh/ssh_host_ecdsa_key`. If that file doesn't exist, run `ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ""` in order to create it. If you made a change to your sshd_config, then you need to run `/etc/init.d/ssh restart` (see: https://leap.se/code/issues/2373) - -. To remove an admin's access to your servers, please remove the directory for that user under the `users/` subdirectory in your provider directory and then remove that user's ssh keys from files/ssh/authorized_keys. When finished you *must* run a `leap deploy` to update that information on the servers. - -. At the moment, it is only possible to add an admin who will have access to all LEAP servers (see: https://leap.se/code/issues/2280) +============================== -. leap add-user --self allows only one key - if you run that command twice with different keys, you will just replace the key with the second key. To add a second key, add it manually to files/ssh/authorized_keys (see: https://leap.se/code/issues/866) +ssh +------------------------------ +* At the moment, it is only possible to add an admin who will have access to + all LEAP servers (see: https://leap.se/code/issues/2280) Deploying ---------- - -. If you have any errors during a run, please try to deploy again as this often solves non-deterministic issues that were not uncovered in our testing. Please re-deploy with `leap -v2 deploy` to get more verbose logs and capture the complete output to provide to us for debugging. - -. If when deploying your debian mirror fails for some reason, network anomoly or the mirror itself is out of date, then platform deployment will not succeed properly. Check the mirror is up and try to deploy again when it is resolved (see: https://leap.se/code/issues/1091) - -. Deployment gives 'error: in `%`: too few arguments (ArgumentError)' - this is because you attempted to do a deploy before initializing a node, please initialize the node first and then do a deploy afterwards (see: https://leap.se/code/issues/2550) - -. This release has no ability to custom configure apt sources or proxies (see: https://leap.se/code/issues/1971) - -. When running a deploy at a verbosity level of 2 and above, you will notice puppet deprecation warnings, these are known and we are working on fixing them - -Special Environments --------------------- - -. When deploying to OpenStack release "nova" or newer, you will need to do an initial deploy, then when it has finished run `leap facts update` and then deploy again (see: https://leap.se/code/issues/3020) - -leap-mx -------- - -. see https://github.com/leapcode/leap_mx#070 for issues regarding leap_mx +------------------------------- +* If you have any errors during a run, please try to deploy again as this often + solves non-deterministic issues that were not uncovered in our testing. + Please re-deploy with `leap -v2 deploy` to get more verbose logs and capture + the complete output to provide to us for debugging. Contributing -============ - -In order to validate the syntax and style guide compliance -before you commit, see https://github.com/pixelated-project/puppet-git-hooks#installation +================================ +In order to validate the syntax and style guide compliance before you commit, +see https://github.com/pixelated-project/puppet-git-hooks#installation Changes -========= +================================ Read CHANGES.md or run `git log`. Authors and Credits -=================== +================================ See contributors: @@ -106,6 +97,6 @@ See contributors: Copyright/License -================= +================================ Read LICENSE diff --git a/doc/common/_bigcouch_migration.md b/doc/common/_bigcouch_migration.md deleted file mode 100644 index eb7e07e9..00000000 --- a/doc/common/_bigcouch_migration.md +++ /dev/null @@ -1,117 +0,0 @@ -@title = "Migrating from BigCouch to plain CouchDB" - -Here are the steps needed to replace BigCouch with CouchDB. - -At the end of this process, you will have just *one* node with `services` property equal to `couchdb`. If you had a BigCouch cluster before, you will be removing all but one of those machines to consolidate them into one CouchDB machine. - -1. if you have multiple nodes with the `couchdb` service on them, pick one of them to be your CouchDB server, and remove the service from the others. If these machines were only doing BigCouch before, you can remove the nodes completely with `leap node rm ` and then you can decommission the servers - -1. put the webapp into [[maintenance mode => webapp#maintenance-mode]] - -1. turn off daemons that access the database. For example: - - ``` - workstation$ leap ssh - server# /etc/init.d/soledad-server stop - - workstation$ leap ssh - server# /etc/init.d/postfix stop - server# /etc/init.d/leap-mx stop - - workstation$ leap ssh - server# /etc/init.d/nickserver stop - ``` - - Alternately, you can create a temporary firewall rule to block access (run on couchdb server): - - ``` - server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT - ``` - -1. remove orphaned databases and do a backup of all remaining, active databases. This can take some time and will place several hundred megabytes of data into /var/backups/couchdb. The size and time depends on how many users there are on your system. For example, 15k users took approximately 25 minutes and 308M of space: - - ``` - workstation$ leap ssh - server# cd /srv/leap/couchdb/scripts - server# ./cleanup-user-dbs - server# time ./couchdb_dumpall.sh - ``` - -1. stop bigcouch: - - ``` - server# /etc/init.d/bigcouch stop - server# pkill epmd - ``` - -1. remove bigcouch: - - ``` - server# apt-get remove bigcouch - ``` - -1. configure your couch node to use plain couchdb instead of bigcouch, you can do this by editing nodes/.json, look for this section: - - ``` - "couch": { - "mode": "plain" - } - ``` - - change it, so it looks like this instead: - - ``` - "couch": { - "mode": "plain", - "pwhash_alg": "pbkdf2" - } - ``` - -1. deploy to the couch node: - - ``` - workstation$ leap deploy - ``` - - If you used the iptables method of blocking access to couchdb, you need to run it again because the deploy just overwrote all the iptables rules: - - ``` - server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT - ``` - -1. restore the backup, this will take approximately the same amount of time as the backup took above: - - ``` - server# cd /srv/leap/couchdb/scripts - server# time ./couchdb_restoreall.sh - ``` - -1. start services again that were stopped in the beginning: - - ``` - workstation$ leap ssh soledad-nodes - server# /etc/init.d/soledad-server start - - workstation$ leap ssh mx-node - server# /etc/init.d/postfix start - server# /etc/init.d/leap-mx start - - workstation$ leap ssh webapp - server# /etc/init.d/nickserver start - ``` - - Or, alternately, if you set up the firewall rule instead, now remove it: - - ``` - server# iptables -D INPUT -p tcp --dport 5984 --jump REJECT - ``` - -1. check if everything is working, including running the test on your deployment machine: - - ``` - workstation$ leap test - ``` - -1. Remove old bigcouch data dir `/opt` after you double checked everything is in place - -1. Relax, enjoy a refreshing beverage. diff --git a/doc/common/_bigcouch_migration_begin.md b/doc/common/_bigcouch_migration_begin.md deleted file mode 100644 index 4e4233dd..00000000 --- a/doc/common/_bigcouch_migration_begin.md +++ /dev/null @@ -1,66 +0,0 @@ -@title = "Migrating from BigCouch to plain CouchDB" - -At the end of this process, you will have just *one* node with `services` property equal to `couchdb`. If you had a BigCouch cluster before, you will be removing all but one of those machines to consolidate them into one CouchDB machine. - -1. if you have multiple nodes with the `couchdb` service on them, pick one of them to be your CouchDB server, and remove the service from the others. If these machines were only doing BigCouch before, you can remove the nodes completely with `leap node rm ` and then you can decommission the servers - -1. put the webapp into [[maintenance mode => webapp#maintenance-mode]] - -1. turn off daemons that access the database. For example: - - ``` - workstation$ leap ssh - server# /etc/init.d/soledad-server stop - - workstation$ leap ssh - server# /etc/init.d/postfix stop - server# /etc/init.d/leap-mx stop - - workstation$ leap ssh - server# /etc/init.d/nickserver stop - ``` - - Alternately, you can create a temporary firewall rule to block access (run on couchdb server): - - ``` - server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT - ``` - -1. remove orphaned databases and do a backup of all remaining, active databases. This can take some time and will place several hundred megabytes of data into /var/backups/couchdb. The size and time depends on how many users there are on your system. For example, 15k users took approximately 25 minutes and 308M of space: - - ``` - workstation$ leap ssh - server# cd /srv/leap/couchdb/scripts - server# ./cleanup-user-dbs - server# time ./couchdb_dumpall.sh - ``` - -1. stop bigcouch: - - ``` - server# /etc/init.d/bigcouch stop - server# pkill epmd - ``` - -1. remove bigcouch: - - ``` - server# apt-get remove bigcouch - ``` - -1. configure your couch node to use plain couchdb instead of bigcouch, you can do this by editing nodes/.json, look for this section: - - ``` - "couch": { - "mode": "plain" - } - ``` -change it, so it looks like this instead: - - ``` - "couch": { - "mode": "plain", - "pwhash_alg": "pbkdf2" - } - ``` - diff --git a/doc/common/_bigcouch_migration_end.md b/doc/common/_bigcouch_migration_end.md deleted file mode 100644 index a47d3c55..00000000 --- a/doc/common/_bigcouch_migration_end.md +++ /dev/null @@ -1,26 +0,0 @@ -1. restore the backup, this will take approximately the same amount of time as the backup took above: - - ``` - server# cd /srv/leap/couchdb/scripts - server# time ./couchdb_restoreall.sh - ``` - -1. start services again that were stopped in the beginning: - - ``` - workstation$ leap ssh soledad-nodes - server# /etc/init.d/soledad-server start - - workstation$ leap ssh mx-node - server# /etc/init.d/postfix start - server# /etc/init.d/leap-mx start - - workstation$ leap ssh webapp - server# /etc/init.d/nickserver start - ``` - - Or, alternately, if you set up the firewall rule instead, now remove it: - - ``` - server# iptables -D INPUT -p tcp --dport 5984 --jump REJECT - ``` diff --git a/doc/common/_bigcouch_migration_finish.md b/doc/common/_bigcouch_migration_finish.md deleted file mode 100644 index 5aae9207..00000000 --- a/doc/common/_bigcouch_migration_finish.md +++ /dev/null @@ -1,10 +0,0 @@ - -1. check if everything is working, including running the test on your deployment machine: - - ``` - workstation$ leap test - ``` - -1. Remove old bigcouch data dir `/opt` after you double checked everything is in place - -1. Relax, enjoy a refreshing beverage. diff --git a/doc/details/development.md b/doc/details/development.md deleted file mode 100644 index 78915add..00000000 --- a/doc/details/development.md +++ /dev/null @@ -1,78 +0,0 @@ -@title = 'Development' -@summary = "Getting started with making changes to the LEAP platform" - -Installing leap_cli ------------------------------------------------- - -### From gem, for a single user - -Install the latest: - - gem install leap_cli --install-dir ~/leap - export PATH=$PATH:~/leap/bin - -Install a particular version: - - gem install leap_cli --version 1.8 --install-dir ~/leap - export PATH=$PATH:~/leap/bin - -### From gem, system wide - -Install the latest: - - sudo gem install leap_cli - -Install a particular version: - - sudo gem install leap_cli --version 1.8 - -### As a gem, built from source - - sudo apt-get install ruby ruby-dev rake - git clone https://leap.se/git/leap_cli.git - cd leap_cli - git checkout develop - rake build - sudo rake install - -### The "develop" branch from source, for a single user - - sudo apt-get install ruby ruby-dev rake - git clone https://leap.se/git/leap_cli.git - cd leap_cli - git checkout develop - -Then do one of the following to be able to run `leap` command: - - cd leap_cli - export PATH=$PATH:`pwd`/bin - alias leap="`pwd`/bin/leap" - ln -s `pwd`/bin/leap ~/bin/leap - -In practice, of course, you would put aliases or PATH modifications in a shell startup file. - -You can also clone from https://github.com/leap/leap_cli - -Running different leap_cli versions ---------------------------------------------- - -### If installed as a gem - -With rubygems, you can always specify the gem version as the first argument to any executable installed by rubygems. For example: - - sudo gem install leap_cli --version 1.7.2 - sudo gem install leap_cli --version 1.8 - leap _1.7.2_ --version - => leap 1.7.2, ruby 2.1.2 - leap _1.8_ --version - => leap 1.8, ruby 2.1.2 - -### If running from source - -Alternately, if you are running from source, you can alias different commands: - - git clone https://leap.se/git/leap_cli.git - cd leap_cli - git checkout develop - alias leap_develop="`pwd`/bin/leap` - diff --git a/doc/details/en.haml b/doc/details/en.haml deleted file mode 100644 index fe7a4c84..00000000 --- a/doc/details/en.haml +++ /dev/null @@ -1,4 +0,0 @@ -- @nav_title = "Details" -- @title = 'Platform Details' - -= child_summaries \ No newline at end of file diff --git a/doc/details/faq.md b/doc/details/faq.md deleted file mode 100644 index 7ee20f4d..00000000 --- a/doc/details/faq.md +++ /dev/null @@ -1,71 +0,0 @@ -@title = 'Frequently asked questions' -@nav_title = 'FAQ' -@summary = "Frequently Asked Questions" -@toc = true - -APT -=============== - -What do I do when unattended upgrades fail? --------------------------------------------------- - -When you receive notification e-mails with a subject of 'unattended-upgrades result for $machinename', that means that some package couldn't be automatically upgraded and needs manual interaction. The reasons vary, so you have to be careful. Most often you can simply login to the affected machine and run `apt-get dist-upgrade`. - -Puppet -====== - -Where do i find the time a server was last deployed ? ------------------------------------------------------ - -Run: - - leap history FILTER - -This will tail the log file `/var/log/leap/deploy-summary.log`. - -If that command fails, you can manually check the puppet state file on the node indicates the last puppetrun: - - ls -la /var/lib/puppet/state/state.yaml - -What resources are touched by puppet/leap_platform (services/packages/files etc.) ? ------------------------------------------------------------------------------------ - -Log into your server and issue: - - grep -v '!ruby/sym' /var/lib/puppet/state/state.yaml | sed 's/\"//' | sort - - -How can i customize the leap_platform puppet manifests ? --------------------------------------------------------- - -You can create custom puppet modules under `files/puppet`. -The custom puppet entry point is in class 'custom' which can be put into -`files/puppet/modules/custom/manifests/init.pp`. This class gets automatically included -by site_config::default, which is applied to all nodes. - -Of cause you can also create a different git branch and change whatever you want, if you are -familiar wit git. - -Facter -====== - -How can i see custom facts distributed by leap_platform on a node ? -------------------------------------------------------------------- - -On the server, export the FACTERLIB env. variable to include the path of the custom fact in question: - - export FACTERLIB=/var/lib/puppet/lib/facter:/srv/leap/puppet/modules/stdlib/lib/facter/ - facter - - -Etc -=== - -How do i change the domain of my provider ? -------------------------------------------- - -* First of all, you need to have access to the nameserver config of your new domain. -* Update domain in provider.json -* remove all ca and cert files: `rm files/cert/* files/ca/*` -* create ca, csr and certs : `leap cert ca; leap cert csr; leap cert dh; leap cert update` -* deploy diff --git a/doc/details/ports.md b/doc/details/ports.md deleted file mode 100644 index f7c485ca..00000000 --- a/doc/details/ports.md +++ /dev/null @@ -1,92 +0,0 @@ -@title = "Ports" -@summary = "The required open ports for different services." -@toc = true - -There are many different ports that must be open in order for the LEAP platform to work. Some ports must be *publicly open*, meaning that these should be accessible from the public internet. Other ports are *privately open*, meaning that they must be accessible to sysadmins or to the other nodes in the provider's infrastructure. - -Every node already includes a host-based firewall. However, if your network has its own firewall, you need to make sure that these ports are not blocked. - -Publicly open ports --------------------------------- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NameNode TypeDefaultNotes
SMTPmx25This is required for all server-to-server SMTP email relay. This is not configurable.
HTTPwebapp80Although no actual services are available over port 80, it should be unblocked so that the web app can redirect to port 443. This is not configurable.
HTTPSwebapp443The web application is available over this port. This is not configurable.
SMTPSmx465The client uses this port to submit outgoing email messages via SMTP over TLS. There is no easy way to change this, although you can create a custom files/service-definitions/v1/smtp-service.json.erb to do so. This will be changed to port 443 in the future.
Soledadsoledad2323The client uses this port to synchronize its storage data. This can be changed via the configuration property soledad.port. This will be changed to port 443 in the future.
Nicknymwebapp6425The client uses this port for discovering public keys. This can be changed via the configuration property nickserver.port. This will be changed to port 443 in the future.
OpenVPNopenvpn80, 443, 53, 1194By default, OpenVPN gateways will listen on all those ports. This can be changed via the configuration property openvpn.ports. Note that these ports must be open for openvpn.gateway_address, not for ip_address.
APIwebapp4430Currently, the provider API is accessible via this port. In the future, the default will be changed to 443. For now, this can be changed via the configuration property api.port.
- -Privately open ports ---------------------------------------- - - - - - - - - - - - - - - - - - - - - -
NameNode TypeDefaultNotes
SSHall22This is the port that the sshd is bound to for the node. You can modify this using the configuration property ssh.port. It is important that this port is never blocked, or you will lose access to deploy to this node.
Stunnelall10000-20000This is the range of ports that might be used for the encrypted stunnel connections between two nodes. These port numbers are automatically generated, but will fall somewhere in the specified range.
- diff --git a/doc/details/under-the-hood.md b/doc/details/under-the-hood.md deleted file mode 100644 index 0bc4fe77..00000000 --- a/doc/details/under-the-hood.md +++ /dev/null @@ -1,40 +0,0 @@ -@title = "Under the hood" -@summary = "Various implementation details." - -This page contains various details on the how the platform is implemented. You can safely ignore this page, although it may be useful if you plan to make modifications to the platform. - -Puppet Details -====================================== - -Tags ----- - -Tags are beeing used to deploy different classes. - -* leap_base: site_config::default (configure hostname + resolver, sshd, ) -* leap_slow: site_config::slow (slow: apt-get update, apt-get dist-upgrade) -* leap_service: cofigure platform service (openvpn, couchdb, etc.) - -You can pass any combination of tags, i.e. use - -* "--tags leap_base,leap_slow,leap_service" (DEFAULT): Deploy all -* "--tags leap_service": Only deploy service(s) (useful for debugging/development) -* "--tags leap_base": Only deploy basic configuration (again, useful for debugging/development) - - -### Doing faster partial deploys - -If you only change a tiny bit on the platform puppet recipes, you could achieve a -*much* faster deploy specifying the resource tag you changed. -i.e. you changed the way rsyslog config snippets for LEAP logfiles are created -in `puppet/modules/leap/manifests/logfile.pp`. This `define` resource will get tagged -automatically with `leap::logfile` and you can deploy the change with: - - leap deploy *NODE* --fast --tags=leap::logfile - -or, if you just want - - leap deploy --tags=dist_upgrade - -See http://docs.puppetlabs.com/puppet/2.7/reference/lang_tags.html for puppet tag usage. - diff --git a/doc/en.md b/doc/en.md deleted file mode 100644 index 098aacb1..00000000 --- a/doc/en.md +++ /dev/null @@ -1,82 +0,0 @@ -@title = 'LEAP Platform for Service Providers' -@summary = "The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment." -@nav_title = 'Provider Platform' -@this.toc = false - -Its goal is to make it as painless as possible for sysadmins to deploy and maintain a service provider's infrastructure for secure communication. - -**REQUIREMENTS** -- Before you begin, make sure you meet these requirements: - -* *Debian Servers*: Servers that you deploy to must be running **Debian Jessie**, and no other distribution or version. -* *Real or Paravirtualized Servers*: Servers must be real machines or paravirtualized VMs (e.g. KVM, Xen, OpenStack, AWS, Google Compute). OS level virtualization is not supported (e.g. OpenVZ, Linux-VServer, etc), nor are system emulators (VirtualBox, QEMU, etc). -* *Your Workstation*: You must have a Linux or Mac computer to deploy from (this can be a headless machine with no GUI). Windows is not supported (Cygwin would probably work, but is untested). -* *Your Own Domain*: You must own a domain name. Before your provider can be put into production, you will need to make modifications to the DNS for the provider's domain. - -The LEAP Platform consists of three parts, detailed below: - -1. [The platform recipes.](#the-platform-recipes) -2. [The provider instance.](#the-provider-instance) -3. [The `leap` command line tool.](#the-leap-command-line-tool) - -The platform recipes --------------------- - -The LEAP platform recipes define an abstract service provider. It is a set of [Puppet](https://puppetlabs.com/puppet/puppet-open-source/) modules designed to work together to provide to sysadmins everything they need to manage a service provider infrastructure that provides secure communication services. - -LEAP maintains a repository of platform recipes, which typically do not need to be modified, although it can be forked and merged as desired. Most service providers using the LEAP platform can use the same set of platform recipes. - -As these recipes consist in abstract definitions, in order to configure settings for a particular service provider a system administrator has to create a provider instance (see below). - -LEAP's platform recipes are distributed as a git repository: `https://leap.se/git/leap_platform` - -The provider instance ---------------------- - -A provider instance is a directory tree (typically tracked in git) containing all the configurations for a service provider's infrastructure. A provider instance **lives on your workstation**, not on the server. - -A provider instance primarily consists of: - -* A pointer to the platform recipes. -* A global configuration file for the provider. -* A configuration file for each server (node) in the provider's infrastructure. -* Additional files, such as certificates and keys. - -A minimal provider instance directory looks like this: - - └── bitmask # provider instance directory. - ├── Leapfile # settings for the `leap` command line tool. - ├── provider.json # global settings of the provider. - ├── common.json # settings common to all nodes. - ├── nodes/ # a directory for node configurations. - ├── files/ # keys, certificates, and other files. - └── users/ # public key information for privileged sysadmins. - -A provider instance directory contains everything needed to manage all the servers that compose a provider's infrastructure. Because of this, any versioning tool and development work-flow can be used to manage your provider instance. - -The `leap` command line tool ----------------------------- - -The `leap` [command line tool](commands) is used by sysadmins to manage everything about a service provider's infrastructure. - -Keep these rules in mind: - -* `leap` is run on your workstation: The `leap` command is always run locally on your workstation, never on a server you are deploying to. -* `leap` is run from within a provider instance: The `leap` command requires that the current working directory is a valid provider instance, except when running `leap new` to create a new provider instance. - -The `leap` command line has many capabilities, including: - -* Create, initialize, and deploy nodes. -* Manage keys and certificates. -* Query information about the node configurations. - -Everything about your provider is managed by editing JSON configuration files and running `leap` commands. - -What is next? ----------------------------------- - -We recommend reading the platform documentation in the following order: - -1. [[quick-start]] -2. [[getting-started]] -3. [[platform/guide]] - diff --git a/doc/guide/commands.md b/doc/guide/commands.md deleted file mode 100644 index 2ddacb83..00000000 --- a/doc/guide/commands.md +++ /dev/null @@ -1,559 +0,0 @@ -@title = 'Command Line Reference' -@summary = 'A copy of leap --help' - -The command "leap" can be used to manage a bevy of servers running the LEAP platform from the comfort of your own home. - - -# Global Options - -* `--log FILE` -Override default log file. -Default Value: None - -* `-v|--verbose LEVEL` -Verbosity level 0..5 -Default Value: 1 - -* `--[no-]color` -Disable colors in output. - -* `-d|--debug` -Print full stack trace for exceptions and load `debugger` gem if installed. - -* `--force` -Like --yes, but also skip prompts that are potentially dangerous to skip. - -* `--help` -Show this message - -* `--version` -Display version number and exit. - -* `--yes` -Skip prompts and assume "yes". - - -# leap add-user USERNAME - -Adds a new trusted sysadmin by adding public keys to the "users" directory. - - - -**Options** - -* `--pgp-pub-key arg` -OpenPGP public key file for this new user -Default Value: None - -* `--ssh-pub-key arg` -SSH public key file for this new user -Default Value: None - -* `--self` -Add yourself as a trusted sysadmin by choosing among the public keys available for the current user. - - -# leap cert - -Manage X.509 certificates - - - -## leap cert ca - -Creates two Certificate Authorities (one for validating servers and one for validating clients). - -See see what values are used in the generation of the certificates (like name and key size), run `leap inspect provider` and look for the "ca" property. To see the details of the created certs, run `leap inspect `. - -## leap cert csr - -Creates a CSR for use in buying a commercial X.509 certificate. - -Unless specified, the CSR is created for the provider's primary domain. The properties used for this CSR come from `provider.ca.server_certificates`, but may be overridden here. - -**Options** - -* `--bits BITS` -Override default certificate bit length -Default Value: None - -* `--country|-C COUNTRY` -Set C in distinguished name. -Default Value: None - -* `--digest DIGEST` -Override default signature digest -Default Value: None - -* `--domain DOMAIN` -Specify what domain to create the CSR for. -Unless specified, the CSR is created for the provider's primary domain. The properties used for this CSR come from `provider.ca.server_certificates`, but may be overridden here. -Default Value: None - -* `--email EMAIL` -Set emailAddress in distinguished name. -Default Value: None - -* `--locality|-L LOCALITY` -Set L in distinguished name. -Default Value: None - -* `--organization|-O ORGANIZATION` -Override default O in distinguished name. -Default Value: None - -* `--state|--ST STATE` -Set ST in distinguished name. -Default Value: None - -* `--unit|--OU UNIT` -Set OU in distinguished name. -Default Value: None - - -## leap cert dh - -Creates a Diffie-Hellman parameter file, needed for forward secret OpenVPN ciphers. - - - -## leap cert update FILTER - -Creates or renews a X.509 certificate/key pair for a single node or all nodes, but only if needed. - -This command will a generate new certificate for a node if some value in the node has changed that is included in the certificate (like hostname or IP address), or if the old certificate will be expiring soon. Sometimes, you might want to force the generation of a new certificate, such as in the cases where you have changed a CA parameter for server certificates, like bit size or digest hash. In this case, use --force. If is empty, this command will apply to all nodes. - -**Options** - -* `--force` -Always generate new certificates - - -# leap clean - -Removes all files generated with the "compile" command. - - - -# leap compile - -Compile generated files. - - - -## leap compile all [ENVIRONMENT] - -Compiles node configuration files into hiera files used for deployment. - - - -## leap compile firewall - -Prints a list of firewall rules. These rules are already implemented on each node, but you might want the list of all rules in case you also have a restrictive network firewall. - - - -## leap compile hosts - -Print entries suitable for an /etc/hosts file, useful for testing your provider. - - - -## leap compile provider.json - -Compile provider.json bootstrap files for your provider. - - - -## leap compile zone - -Prints a DNS zone file for your provider. - - -Default Command: all - -# leap db - -Database commands. - - - -## leap db destroy [FILTER] - -Destroy one or more databases. If present, limit to FILTER nodes. For example `leap db destroy --db sessions,tokens testing`. - - - -**Options** - -* `--db DATABASES` -Comma separated list of databases to destroy (no space). Use "--db all" to destroy all databases. -Default Value: None - -* `--user USERS` -Comma separated list of usernames. The storage databases for these user(s) will be destroyed. -Default Value: None - - -# leap debug FILTER - -Output debug information. - -The FILTER can be the name of a node, service, or tag. - -# leap deploy FILTER - -Apply recipes to a node or set of nodes. - -The FILTER can be the name of a node, service, or tag. - -**Options** - -* `--ip IPADDRESS` -Override the default SSH IP address. -Default Value: None - -* `--port PORT` -Override the default SSH port. -Default Value: None - -* `--tags TAG[,TAG]` -Specify tags to pass through to puppet (overriding the default). -Default Value: None - -* `--dev` -Development mode: don't run 'git submodule update' before deploy. - -* `--downgrade` -Allows deploy to run with an older platform version. - -* `--fast` -Makes the deploy command faster by skipping some slow steps. A "fast" deploy can be used safely if you recently completed a normal deploy. - -* `--force` -Deploy even if there is a lockfile. - -* `--sync` -Sync files, but don't actually apply recipes. - - -# leap env - -Manipulate and query environment information. - -The 'environment' node property can be used to isolate sets of nodes into entirely separate environments. A node in one environment will never interact with a node from another environment. Environment pinning works by modifying your ~/.leaprc file and is dependent on the absolute file path of your provider directory (pins don't apply if you move the directory) - -## leap env ls [ENVIRONMENT] - -List the available environments. The pinned environment, if any, will be marked with '*'. Will also set the pin if run with an environment argument. - - - -## leap env pin ENVIRONMENT - -Pin the environment to ENVIRONMENT. All subsequent commands will only apply to nodes in this environment. - - - -## leap env unpin - -Unpin the environment. All subsequent commands will apply to all nodes. - - -Default Command: ls - -# leap facts - -Gather information on nodes. - - - -## leap facts update FILTER - -Query servers to update facts.json. - -Queries every node included in FILTER and saves the important information to facts.json - -# leap help command - -Shows a list of commands or help for one command - -Gets help for the application or its commands. Can also list the commands in a way helpful to creating a bash-style completion function - -**Options** - -* `-c` -List commands one per line, to assist with shell completion - - -# leap history FILTER - -Display recent deployment history for a set of nodes. - -The FILTER can be the name of a node, service, or tag. - -**Options** - -* `--ip IPADDRESS` -Override the default SSH IP address. -Default Value: None - -* `--port PORT` -Override the default SSH port. -Default Value: None - -* `--last` -Show last deploy only - - -# leap inspect FILE - -Prints details about a file. Alternately, the argument FILE can be the name of a node, service or tag. - - - -**Options** - -* `--base` -Inspect the FILE from the provider_base (i.e. without local inheritance). - - -# leap list [FILTER] - -List nodes and their classifications - -Prints out a listing of nodes, services, or tags. If present, the FILTER can be a list of names of nodes, services, or tags. If the name is prefixed with +, this acts like an AND condition. For example: - -`leap list node1 node2` matches all nodes named "node1" OR "node2" - -`leap list openvpn +local` matches all nodes with service "openvpn" AND tag "local" - -**Options** - -* `--print arg` -What attributes to print (optional) -Default Value: None - -* `--disabled` -Include disabled nodes in the list. - - -# leap local - -Manage local virtual machines. - -This command provides a convient way to manage Vagrant-based virtual machines. If FILTER argument is missing, the command runs on all local virtual machines. The Vagrantfile is automatically generated in 'test/Vagrantfile'. If you want to run vagrant commands manually, cd to 'test'. - -## leap local destroy [FILTER] - -Destroys the virtual machine(s), reclaiming the disk space - - - -## leap local reset [FILTER] - -Resets virtual machine(s) to the last saved snapshot - - - -## leap local save [FILTER] - -Saves the current state of the virtual machine as a new snapshot - - - -## leap local start [FILTER] - -Starts up the virtual machine(s) - - - -**Options** - -* `--basebox BASEBOX` -The basebox to use. This value is passed to vagrant as the `config.vm.box` option. The value here should be the name of an installed box or a shorthand name of a box in HashiCorp's Atlas. -Default Value: LEAP/jessie - - -## leap local status [FILTER] - -Print the status of local virtual machine(s) - - - -## leap local stop [FILTER] - -Shuts down the virtual machine(s) - - - -# leap mosh NAME - -Log in to the specified node with an interactive shell using mosh (requires node to have mosh.enabled set to true). - - - -**Options** - -* `--port SSH_PORT` -Override default SSH port used when trying to connect to the server. Same as `--ssh "-p SSH_PORT"`. -Default Value: None - -* `--ssh arg` -Pass through raw options to ssh (e.g. `--ssh '-F ~/sshconfig'`). -Default Value: None - - -# leap new DIRECTORY - -Creates a new provider instance in the specified directory, creating it if necessary. - - - -**Options** - -* `--contacts arg` -Default email address contacts. -Default Value: None - -* `--domain arg` -The primary domain of the provider. -Default Value: None - -* `--name arg` -The name of the provider. -Default Value: None - -* `--platform arg` -File path of the leap_platform directory. -Default Value: None - - -# leap node - -Node management - - - -## leap node add NAME [SEED] - -Create a new configuration file for a node named NAME. - -If specified, the optional argument SEED can be used to seed values in the node configuration file. - -The format is property_name:value. - -For example: `leap node add web1 ip_address:1.2.3.4 services:webapp`. - -To set nested properties, property name can contain '.', like so: `leap node add web1 ssh.port:44` - -Separeate multiple values for a single property with a comma, like so: `leap node add mynode services:webapp,dns` - -**Options** - -* `--local` -Make a local testing node (by automatically assigning the next available local IP address). Local nodes are run as virtual machines on your computer. - - -## leap node init FILTER - -Bootstraps a node or nodes, setting up SSH keys and installing prerequisite packages - -This command prepares a server to be used with the LEAP Platform by saving the server's SSH host key, copying the authorized_keys file, installing packages that are required for deploying, and registering important facts. Node init must be run before deploying to a server, and the server must be running and available via the network. This command only needs to be run once, but there is no harm in running it multiple times. - -**Options** - -* `--ip IPADDRESS` -Override the default SSH IP address. -Default Value: None - -* `--port PORT` -Override the default SSH port. -Default Value: None - -* `--echo` -If set, passwords are visible as you type them (default is hidden) - - -## leap node mv OLD_NAME NEW_NAME - -Renames a node file, and all its related files. - - - -## leap node rm NAME - -Removes all the files related to the node named NAME. - - - -# leap scp FILE1 FILE2 - -Secure copy from FILE1 to FILE2. Files are specified as NODE_NAME:FILE_PATH. For local paths, omit "NODE_NAME:". - - - -**Options** - -* `-r` -Copy recursively - - -# leap ssh NAME - -Log in to the specified node with an interactive shell. - - - -**Options** - -* `--port SSH_PORT` -Override default SSH port used when trying to connect to the server. Same as `--ssh "-p SSH_PORT"`. -Default Value: None - -* `--ssh arg` -Pass through raw options to ssh (e.g. `--ssh '-F ~/sshconfig'`). -Default Value: None - - -# leap test - -Run tests. - - - -## leap test init - -Creates files needed to run tests. - - - -## leap test run [FILTER] - -Run the test suit on FILTER nodes. - - - -**Options** - -* `--[no-]continue` -Continue over errors and failures (default is --no-continue). - -Default Command: run - -# leap tunnel [LOCAL_PORT:]NAME:REMOTE_PORT - -Creates an SSH port forward (tunnel) to the node NAME. REMOTE_PORT is the port on the remote node that the tunnel will connect to. LOCAL_PORT is the optional port on your local machine. For example: `leap tunnel couch1:5984`. - - - -**Options** - -* `--port SSH_PORT` -Override default SSH port used when trying to connect to the server. Same as `--ssh "-p SSH_PORT"`. -Default Value: None - -* `--ssh arg` -Pass through raw options to ssh (e.g. --ssh '-F ~/sshconfig'). -Default Value: None - diff --git a/doc/guide/config.md b/doc/guide/config.md deleted file mode 100644 index bcea26c4..00000000 --- a/doc/guide/config.md +++ /dev/null @@ -1,360 +0,0 @@ -@title = "Configuration Files" -@summary = "Understanding and editing the configuration files." - -Files -------------------------------------------- - -Here are a list of some of the common files that make up a provider. Except for `Leapfile` and `provider.json`, the files are optional. Unless otherwise specified, all file names are relative to the 'provider directory' root (where the Leapfile is). - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
LeapfileIf present, this file tells leap that the directory is a provider directory. This file is usually empty, but can contain global options.
~/.leaprcEvaluated the same as Leapfile, but not committed to source control.
provider.jsonGlobal options related to this provider. See [[provider-configuration]].
provider.ENVIRONMENT.jsonGlobal options for the provider that are applied to only a single environment.
nodes/NAME.jsonThe configuration file for node called NAME.
common.jsonAll nodes inherit from this file. In other words, any options that appear in common.json will be added as default values to each node configuration, value that can be locally overridden.
services/SERVICE.jsonThe properties in this configuration file are applied to any node that includes SERVICE in its services property.
services/SERVICE.ENVIRONMENT.jsonThe properties in this configuration file are applied to any node that includes SERVICE in its services and has environment equal to ENVIRONMENT.
tags/TAG.jsonThe properties in this configuration file are applied to any node that has includes TAG in its tags property.
tags/TAG.ENVIRONMENT.jsonThe properties in this configuration file are applied to any node that has includes TAG in its tags property and has environment property equal to ENVIRONMENT.
secrets.json An automatically generated file that contains any randomly generated strings needed in order to deploy. These strings are often secret and should be protected, although any need for a random string or number that is remembered will produce another entry in this file. This file is automatically generated and refreshed each time you run leap compile or leap deploy. If an entry is no longer needed, it will get removed. If you want to change a secret, you can remove this file and have it regenerated, or remove the particular line item and just those items will be created anew.
facts.jsonIf some of your servers are running on AWS or OpenStack, you will need to discover certain properties about how networking is configured on these machines in order for a full deploy to work. In these cases, make sure to run leap facts update to periodically regenerate the facts.json file.
files/*Various static files used by the platform (e.g. keys, certificates, webapp customization, etc). In general, only generated files and files used to customize the provider (such as images) live in the files directory.
users/USER/A directory that stores the public keys of the sysadmin with name USER. This person will have root access to all the servers.
- -Leapfile -------------------------------------------- - -A `Leapfile` defines options for the `leap` command and lives at the root of your provider directory. `Leapfile` is evaluated as ruby, so you can include whatever weird logic you want in this file. In particular, there are several variables you can set that modify the behavior of leap. For example: - - @platform_directory_path = '../leap_platform' - @log = '/var/log/leap.log' - -Additionally, you can create a `~/.leaprc` file that is loaded after `Leapfile` and is evaluated the same way. - -Platform options: - -* `@platform_directory_path` (required). This must be set to the path where `leap_platform` lives. The path may be relative. - -Vagrant options: - -* `@vagrant_provider`. Changes the default vagrant provider ("virtualbox"). For example, `@vagrant_provider = "libvirt"`. -* `@vagrant_network`. Allows you to override the default network used for local nodes. It should include a netmask like `@vagrant_network = '10.0.0.0/24'`. -* `@custom_vagrant_vm_line`. Insert arbitrary text into the auto-generated Vagrantfile. For example, `@custom_vagrant_vm_line = "config.vm.boot_mode = :gui"`. -* `@vagrant_basebox` allows specifying a different basebox as the default one. For example, `@vagrant_basebox = "LEAP/jessie"`. - -Logging options: - -* `@log`. If set, all command invocation and results are logged to the specified file. This is the same as the switch `--log FILE`, except that the command line switch will override the value in the Leapfile. - - -JSON format -------------------------------------------- - -All configuration files, other than `Leapfile`, are in the JSON format. For example: - - { - "key1": "value1", - "key2": "value2" - } - -Keys should match `/[a-z0-9_]/` and must be in double quotes. - -Unlike traditional JSON, comments are allowed. If the first non-whitespace characters are `//` then the line is treated as a comment. - - // this is a comment - { - // this is a comment - "key": "value" // this is an error - } - -Options in the configuration files might be nested hashes, arrays, numbers, strings, or boolean. Numbers and boolean values should **not** be quoted. For example: - - { - "openvpn": { - "ip_address": "1.1.1.1", - "protocols": ["tcp", "udp"], - "ports": [80, 53], - "options": { - "public_ip": false, - "adblock": true - } - } - } - -If the value string is prefixed with an '=' character, the result is evaluated as ruby. For example: - - { - "domain": { - "public": "domain.org" - } - "api_domain": "= 'api.' + domain.public" - } - -In this case, the property "api_domain" will be set to "api.domain.org". So long as you do not create unresolvable circular dependencies, you can reference other properties in evaluated ruby that are themselves evaluated ruby. - -See "Macros" below for information on the special macros available to the evaluated ruby. - -TIP: In rare cases, you might want to force the evaluation of a value to happen in a later pass after most of the other properties have been evaluated. To do this, prefix the value string with "=>" instead of "=". - -Node inheritance ----------------------------------------- - -Every node inherits from common.json and also any of the services or tags attached to the node. Additionally, the `leap_platform` contains a directory `provider_base` that defines the default values for tags, services and common.json. - -Suppose you have a node configuration for `bitmask/nodes/willamette.json` like so: - - { - "services": "webapp", - "tags": ["production", "northwest-us"], - "ip_address": "1.1.1.1" - } - -This node will have hostname "willamette" and it will inherit from the following files (in this order): - -1. common.json - - load defaults: `provider_base/common.json` - - load provider: `bitmask/common.json` -2. service "webapp" - - load defaults: `provider_base/services/webapp.json` - - load provider: `bitmask/services/webapp.json` -3. tag "production" - - load defaults: `provider_base/tags/production.json` - - load provider: `bitmask/tags/production.json` -4. tag "northwest-us" - - load: `bitmask/tags/northwest-us.json` -5. finally, load node "willamette" - - load: `bitmask/nodes/willamette.json` - -The `provider_base` directory is under the `leap_platform` specified in the file `Leapfile`. - -To see all the variables a node has inherited, you could run `leap inspect willamette`. - -### Inheritance rules - -Suppose you have a node configuration `mynode.json`: - - { - "tags": "production", - "simple_value": 100, - "replaced_array": ["dolphin", "kangaroo"], - "+add_array": ["red", "black"], - "-subtract_array": ["bitter"], - "converted_to_array": "not_array_element", - "!override": ["insist on this value"], - "hash": { - "key1": 1, - "key2": 2 - } - } - -And a file `tags/production.json`: - - { - "simple_value": 99999, - "replaced_array": ["zebra"], - "add_array": ["green], - "subtract_array": ["bitter", "sweet", "salty"], - "converted_to_array": ["array_element"], - "override": "this value will be overridden", - "hash": { - "key1": "one" - } - } - -In this scenario, `mynode.json` will inherit from `production.json`. The output of this inheritance will be: - - { - "tags": "production", - "simple_value": 100, - "replaced_array": ["dolphin", "kangaroo"], - "add_array": ["red", "black", "green"], - "subtract_array": ["sweet", "salty"], - "converted_to_array": ["not_array_element", "array_element"], - "override": ["insist on this value"], - "hash": { - "key1": 1, - "key2": 2 - } - -The rules for inheritance (where 'old' refers to the parent, and 'new' refers to the child): - -* Simple values (strings, numbers, boolean): - * Replace the old value with the new value. -* Array values: - * Two arrays: replace the old array with the new array. - * One array and one simple value: add the simple value to the array. - * If property name is prefixed with "+": merge the old and new arrays. - * If property name is prefixed with "-": subtract new array from old array. -* Hash values: - * Hashes are always merged (the result includes the keys of both hashes). If there is a key in common, the new one overrides the old one. -* Mismatch: - * Although you can mix arrays and simple values, you cannot mix arrays with hashes or hashes with simple values. If you attempt to do so, it will fail to compile and give you an error message. -* Override: - * If property name is prefixed with "!": then ensure that new value is always used, regardless of old value. In this case, the override takes precedence over type checking, so you will never get a type mismatch. - -NOTE: special property name prefixes, like "+", "-", or "!", are not included in the property name. These prefixes determine the merge strategy, but are stripped out when compiling the resulting JSON file. - -Common configuration options ----------------------------------------- - -You can use the command `leap inspect` to see what options are available for a provider, node, service, or tag configuration. For example: - -* `leap inspect common` -- show the options inherited by all nodes. -* `leap inspect --base common` -- show the common.json from `provider_base` without the local `common.json` inheritance applied. -* `leap inspect webapp` -- show all the options available for the service `webapp`. - -Here are some of the more important options you should be aware of: - -* `ip_address` -- Required for all nodes, no default. -* `ssh.port` -- The SSH port you want the node's OpenSSH server to bind to. This is also the default when trying to connect to a node, but if the node currently has OpenSSH running on a different port then run deploy with `--port` to override the `ssh.port` configuration value. -* `mosh.enabled` -- If set to `true`, then mosh will be installed on the server. The default is `false`. - -Macros ----------------------------------------- - -When using evaluated ruby in a JSON configuration file, there are several special macros that are available. These are evaluated in the context of a node (available as the variable `self`). - -The following methods are available to the evaluated ruby: - -`variable.variable` - - > Any variable defined or inherited by a particular node configuration is available by just referencing it using either hash notation or object field notation (e.g. `['domain']['public']` or `domain.public`). Circular references are not allowed, but otherwise it is OK to nest evaluated values in other evaluated values. If a value has not been defined, the hash notation will return nil but the field notation will raise an exception. Properties of services, tags, and the global provider can all be referenced the same way. For example, `global.services['openvpn'].x509.dh`. - -`nodes` - - > A hash of all nodes. This list can be filtered. - -`nodes_like_me` - - > A hash of nodes that have the same deployment tags as the current node (e.g. 'production' or 'local'). - -`global.services` - - > A hash of all services, e.g. `global.services['openvpn']` would return the "openvpn" service. - -`global.tags` - - > A hash of all tags, e.g. `global.tags['production']` would return the "production" tag. - - `global.provider` - - > Can be used to access variables defined in `provider.json`, e.g. `global.provider.contacts.default`. - -`file(filename)` - - > Inserts the full contents of the file. If the file is an erb template, it is rendered. The filename can either be one of the pre-defined file symbols, or it can be a path relative to the "files" directory in your provider instance. E.g, `file :ca_cert` or `files 'ca/ca.crt'`. - -`file_path(filename)` - - > Ensures that the file will get rsynced to the node as an individual file. The value returned by `file_path` is the full path where this file will ultimately live when deploy to the node. e.g. `file_path :ca_cert` or `file_path 'branding/images/logo.png'`. - -`secret(:symbol)` - - > Returns the value of a secret in secrets.json (or creates it if necessary). E.g. `secret :couch_admin_password` - -`hosts_file` - - > Returns a data structure that puppet will use to generate /etc/hosts. Care is taken to use the local IP of other hosts when needed. - -`known_hosts_file` - - > Returns the lines needed in a SSH `known_hosts` file. - -`stunnel_client(node_list, port, options={})` - - > Returns a stunnel configuration data structure for the client side. Argument `node_list` is an `ObjectList` of nodes running stunnel servers. Argument `port` is the real port of the ultimate service running on the servers that the client wants to connect to. - -`stunnel_server(port)` - - > Generates a stunnel server entry. The `port` is the real port targeted service. - -Hash tables ------------------------------------------ - -The macros `nodes`, `nodes_like_me`, `global.services`, and `global.tags` all return a hash table of configuration objects (either nodes, services, or tags). There are several ways to filter and process these hash tables: - -Access an element by name: - - nodes['vpn1'] # returns node named 'vpn1' - global.services['openvpn'] # returns service named 'openvpn' - -Create a new hash table by applying filters: - - nodes[:public_dns => true] # all nodes where public_dns == true - nodes[:services => 'openvpn', 'location.country_code' => 'US'] # openvpn service OR in the US. - nodes[[:services, 'openvpn'], [:services, 'tor']] # services equal to openvpn OR tor - nodes[:services => 'openvpn'][:tags => 'production'] # openvpn AND production - nodes[:name => "!bob"] # all nodes that are NOT named "bob" - -Create an array of values by selecting a single field: - - nodes.field('location.name') - ==> ['seattle', 'istanbul'] - -Create an array of hashes by selecting multiple fields: - - nodes.fields('domain.full', 'ip_address') - ==> [ - {'domain_full' => 'red.bitmask.net', 'ip_address' => '1.1.1.1'}, - {'domain_full' => 'blue.bitmask.net', 'ip_address' => '1.1.1.2'}, - ] - -Create a new hash table of hashes, with only certain fields: - - nodes.pick_fields('domain.full', 'ip_address') - ==> { - "red" => {'domain_full' => 'red.bitmask.net', 'ip_address' => '1.1.1.1'}, - "blue => {'domain_full' => 'blue.bitmask.net', 'ip_address' => '1.1.1.2'}, - } - -With `pick_fields`, if there is only one field, it will generate a simple hash table: - - nodes.pick_fields('ip_address') - ==> { - "red" => '1.1.1.1', - "blue => '1.1.1.2', - } diff --git a/doc/guide/domains.md b/doc/guide/domains.md deleted file mode 100644 index 914bce33..00000000 --- a/doc/guide/domains.md +++ /dev/null @@ -1,129 +0,0 @@ -@title = "Domains" -@summary = "How to handle domain names and integrating LEAP with existing services." -@toc = true - -Overview --------------------------------- - -Deploying LEAP can start to get very tricky when you need to integrate LEAP services with an existing domain that you already use or which already has users. Most of this complexity is unavoidable, although there are a few things we plan to do in the future to make this a little less painful. - -Because integration with legacy systems is an advanced topic, we recommend that you begin with a new domain. Once everything works and you are comfortable with your LEAP-powered infrastructure, you can then contemplate integrating with your existing domain. - -### Definitions - -**provider domain** - -This is the main domain used to identify the provider. The **provider domain** is what the user enters in the Bitmask client. e.g. `example.org`. The full host name of every node in your provider infrastructure will use the **provider domain** (e.g. `dbnode.example.org`). - -In order for the Bitmask client to get configured for use with a provider, it must be able to find the `provider.json` bootstrap file at the root of the **provider domain**. This is not needed if the Bitmask client is "pre-seeded" with the provider's information (these providers show up in a the initial list of available providers). - -**webapp domain** - -This is the domain that runs the leap_web application that allows users to register accounts, create help tickets, etc. e.g. `example.org` or `user.example.org`. The **webapp domain** defaults to the **provider domain** unless it is explicitly configured separately. - -**API domain** - -This is the domain that the provider API runs on. Typically, this is set automatically and you never need to configure it. The user should never be aware of this domain. e.g. `api.example.org`. The Bitmask client discovers this API domain by reading it from the `provider.json` file it grabs from the **provider domain**. - -**mail domain** - -This is the domain used for mail accounts, e.g. `username@example.org`. Currently, this is always the **provider domain**, but it may be independently configurable in the future. - -Generating a zone file ------------------------------------ - -Currently, the platform does not include a dedicated `dns` service type, so you need to have your own setup for DNS. You can generate the appropriate configuration options with this command: - - leap compile zone - -A single domain -------------------------------- - -The easy approach is to use a single domain for **provider domain**, **webapp domain**, and **email domain**. This will install the webapp on the **provider domain**, which means that this domain must be a new one that you are not currently using for anything. - -To configure a single domain, just set the domain in `provider.json`: - - { - "domain": "example.org" - } - -If you have multiple environments, you can specify a different **provider domain** for each environment. For example: - -`provider.staging.json` - - { - "domain": "staging.example.org" - } - -A separate domain for the webapp --------------------------------------- - -It is possible make the **webapp domain** different than the **provider domain**. This is needed if you already have a website running at your **provider domain**. - -In order to put webapp on a different domain, you must take two steps: - -1. You must configure `webapp.domain` for nodes with the `webapp` service. -2. You must make the compiled `provider.json` available at the root of the **provider domain**. - -NOTE: This compiled provider.json is different than the provider.json that you edit and lives in the root of the provider directory. - -### Step 1. Configuring `webapp.domain` - -In `services/webapp.json`: - - { - "webapp": { - "domain": "user.example.org" - } - } - -### Step 2. Putting the compiled `provider.json` in place - -Generate the compiled `provider.json`: - - leap compile provider.json - = created files/web/bootstrap/ - = created files/web/bootstrap/README - = created files/web/bootstrap/production/ - = created files/web/bootstrap/production/provider.json - = created files/web/bootstrap/production/htaccess - = created files/web/bootstrap/staging/ - = created files/web/bootstrap/staging/provider.json - = created files/web/bootstrap/staging/htaccess - -This command compiles a separate `provider.json` for each environment, or "default" if you don't have an environment. In the example above, there is an environment called "production" and one called "staging", but your setup will probably differ. - -The resulting `provider.json` file must then be put at the root URL of your **provider domain** for the appropriate environment. - -There is one additional complication: currently, the Bitmask client tests for compatibility using some HTTP headers on the `/provider.json` response. This is will hopefully change in the future, but for now you need to ensure the right headers are set in the response. The included file `htaccess` has example directives for Apache, if that is what you use. - -This step can be skipped if you happen to use the `static` service to deploy an `amber` powered static website to **provider domain**. In this case, the correct `provider.json` will be automatically put into place. - -Integrating with existing email system ------------------------------------------ - -If your **mail domain** already has users from a legacy email system, then things get a bit complicated. In order to be able to support both LEAP-powered email and legacy email on the same domain, you need to follow these steps: - -1. Modify the LEAP webapp so that it does not create users with the same name as users in the legacy system. -2. Configure your legacy MX servers to forward mail that they cannot handle to the LEAP MX servers, or vice versa. - -### Step 1. Modify LEAP webapp - -In order to modify the webapp to respect the usernames already reserved by your legacy system, you need to modify the LEAP webapp code. The easiest way to do this is to create a custom gem that modifies the behavior of the webapp. - -For this example, we will call our custom gem `reserve_usernames`. - -This gem can live in one of two places: - -(1) You can fork the project leap_web and put the gem in `leap_web/vendor/gems/reserve_usernames`. Then, modify `Gemfile` and add the line `gem 'common_languages', :path => 'vendor/gems/reserve_usernames'` - -(2) Alternately, you can put the gem in the local provider directory `files/webapp/gems/reserve_username`. This will get synced to the webapp servers when you deploy and put in `/srv/leap/webapp/config/customization` where it will get automatically loaded by the webapp. - -What should the gem `reserve_usernames` look like? There is an example available here: https://leap.se/git/reserved_usernames.git - -This example gem uses ActiveResource to communicate with a remote REST API for creating and checking username reservations. This ensures that both the legacy system and the LEAP system use the same namespace. Alternately, you could write a gem that checks the legacy database directly. - -### Step 2. Configure MX servers - -To be written. - diff --git a/doc/guide/en.haml b/doc/guide/en.haml deleted file mode 100644 index 61c24ea8..00000000 --- a/doc/guide/en.haml +++ /dev/null @@ -1,4 +0,0 @@ -- @nav_title = "Guide" -- @title = "Platform Guide" - -= child_summaries \ No newline at end of file diff --git a/doc/guide/environments.md b/doc/guide/environments.md deleted file mode 100644 index 752e0608..00000000 --- a/doc/guide/environments.md +++ /dev/null @@ -1,75 +0,0 @@ -@title = "Working with environments" -@nav_title = "Environments" -@summary = "How to partition the nodes into separate environments." - -With environments, you can divide your nodes into different and entirely separate sets. For example, you might have sets of nodes for 'testing', 'staging' and 'production'. - -Typically, the nodes in one environment are totally isolated from the nodes in a different environment. Each environment will have its own separate database, for example. - -There are a few exceptions to this rule: backup nodes, for example, will by default attempt to back up data from all the environments (excluding local). - -## Assign an environment - -To assign an environment to a node, you just set the `environment` node property. This is typically done with tags, although it is not necessary. For example: - -`tags/production.json` - - { - "environment": "production" - } - -`nodes/mynode.json` - - { - "tags": ["production"] - } - -There are several built-in tags that will apply a value for the environment: - -* `production`: An environment for nodes that are in use by end users. -* `development`: An environment to be used for nodes that are being used for experiments or staging. -* `local`: This environment gets automatically applied to all nodes that run only on local VMs. Nodes with a `local` environment are treated special and excluded from certain calculations. - -You don't need to use these and you can add your own. - -## Environment commands - -* `leap env` -- List the available environments and disply which one is active. -* `leap env pin ENV` -- Pin the current environment to ENV. -* `leap env unpin` -- Remove the environment pin. - -The environment pin is only active for your local machine: it is not recorded in the provider directory and not shared with other users. - -## Environment specific JSON files - -You can add JSON configuration files that are only applied when a specific environment is active. For example, if you create a file `provider.production.json`, these values will only get applied to the `provider.json` file for the `production` environment. - -This will also work for services and tags. For example: - - provider.local.json - services/webapp.development.json - tags/seattle.production.json - -In this example, `local`, `development`, and `production` are the names of environments. - -## Bind an environment to a Platform version - -If you want to ensure that a particular environment is bound to a particular version of the LEAP Platform, you can add a `platform` section to the `provider.ENV.json` file (where ENV is the name of the environment in question). - -The available options are `platform.version`, `platform.branch`, or `platform.commit`. For example: - - { - "platform": { - "version": "1.6.1", - "branch": "develop", - "commit": "5df867fbd3a78ca4160eb54d708d55a7d047bdb2" - } - } - -You can use any combination of `version`, `branch`, and `commit` to specify the binding. The values for `branch` and `commit` only work if the `leap_platform` directory is a git repository. - -The value for `commit` is passed directly through to `git log` to query for a list of acceptable commits. See [[man gitrevisions => https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html#_specifying_ranges]] to see how to specify ranges. For example: - -* `HEAD^..HEAD` - current commit must be head of the branch. -* `3172444652af71bd771609d6b80258e70cc82ce9..HEAD` - current commit must be after 3172444652af71bd771609d6b80258e70cc82ce9. -* `refs/tags/0.6.0rc1..refs/tags/0.6.0rc2` - current commit must be after tag 0.6.0rc1 and before or including tag 0.6.0rc2. \ No newline at end of file diff --git a/doc/guide/getting-started.md b/doc/guide/getting-started.md deleted file mode 100644 index 6236cba0..00000000 --- a/doc/guide/getting-started.md +++ /dev/null @@ -1,145 +0,0 @@ -@title = 'Getting Started' -@summary = 'An overview of the LEAP Platform' -@toc = true - - -Sensitive files ----------------------------------------------- - -Some files in your provider directory are very sensitive. Leaking these files will compromise your provider. - -Super sensitive and irreplaceable: - -* `files/ca/*.key` -- the private keys for the client and server CAs. -* `files/cert/*.key` -- the private key(s) for the commercial certificate for your domain(s). - -Sensitive, but can be erased and regenerated automatically: - -* `secrets.json` -- various random secrets, such as passwords for databases. -* `files/nodes/*/*.key` -- the private key for each node. -* `hiera/*.yaml` -- hiera file contains a copy of the private key of the node. - -Also, each sysadmin has one or more public ssh keys in `users/*/*_ssh.pub`. Typically, you will want to keep these public keys secure as well. - -See [[keys-and-certificates]] for more information. - -Useful commands -------------------------------------------- - -Here are a few useful `leap` commands: - -* `leap help [COMMAND]` -- get help on COMMAND. -* `leap history [FILTER]` -- show the recent deployment history for the selected nodes. -* `leap ssh web1` -- SSH into node web1 (requires `leap node init web1` first). -* `leap list [FILTER]` -- list the selected nodes. - * `leap list production` -- list only those nodes with the tag 'production' - * `leap list --print ip_address` -- list a particular attribute of all nodes. - -See the full [[commands]] for more information. - -Node filters -------------------------------------------- - -Many of the `leap` commands take a "node filter". You can use a node filter to target a command at one or more nodes. - -A node filter consists of one or more keywords, with an optional "+" before each keyword. - -* keywords can be a node name, a service type, or a tag. -* the "+" before the keyword constructs an AND condition -* otherwise, multiple keywords together construct an OR condition - -Examples: - -* `leap list openvpn` -- list all nodes with service openvpn. -* `leap list openvpn +production` -- only nodes of service type openvpn AND tag production. -* `leap deploy webapp openvpn` -- deploy to all webapp OR openvpn nodes. -* `leap node init ostrich` -- just init the node named ostrich. - -See the full [[commands]] for more information. - -Tracking the provider directory in git ------------------------------------------- - -You should commit your provider changes to your favorite VCS whenever things change. This way you can share your configurations with other admins, all they have to do is to pull the changes to stay up to date. Every time you make a change to your provider, such as adding nodes, services, generating certificates, etc. you should add those to your VCS, commit them and push them to where your repository is hosted. - -Note that your provider directory contains secrets, such as private key material and passwords. You do not want to have those passwords readable by the world, so make sure that wherever you are hosting your repository, it is not public for the world to read. - -If you have a post-commit hook that emails the changes to contributors, you may want to exclude diffs for files that might have sensitive secrets. For example, create a `.gitattributes` file with: - - # No diff, no email for key files - *.key -diff - *.pem -diff - - # Discard diff for secrets.json - secrets.json -diff - - # No diff for hiera files, they contain passwords - hiera/* -diff - - -Editing JSON configuration files --------------------------------------- - -All the settings that compose your provider are stored in JSON files. - -At a minimum, you will need at least two configuration files: - -* `provider.json` -- general settings for you provider. -* `nodes/NAME.json` -- configuration file for node called "NAME". - -There are a few required properties in provider.json: - - { - "domain": "example.org", - "name": "Example", - "contacts": { - "default": "email1@example.org" - } - } - -See [[provider-configuration]] for more details. - -For node configuration files, there are two required properties: - - { - "ip_address": "1.1.1.1", - "services": ["openvpn"] - } - -See [[services]] for details on what servers are available, and see [[config]] details on how configuration files work. - -How does it work under the hood? --------------------------------------------- - -You don't need to know any of the details of what happens "under the hood" in order to use the LEAP platform. However, if you are curious as to what is going on, here is a quick primer. - -First, some background terminology: - -* **puppet**: Puppet is a system for automating deployment and management of servers (called nodes). -* **hiera files**: In puppet, you can use something called a 'hiera file' to seed a node with a few configuration values. In LEAP, we go all out and put *every* configuration value needed for a node in the hiera file, and automatically compile a custom hiera file for each node. - -When you run `leap deploy`, a bunch of things happen, in this order: - -1. **Compile hiera files**: The hiera configuration file for each node is compiled in YAML format and saved in the directory `hiera`. The source material for this hiera file consists of all the JSON configuration files imported or inherited by the node's JSON config file. -* **Copy required files to node**: All the files needed for puppet to run are rsync'ed to each node. This includes the entire leap_platform directory, as well as the node's hiera file and other files needed by puppet to set up the node (keys, binary files, etc). -* **Puppet is run**: Once the node is ready, leap connects to the node via ssh and runs `puppet apply`. Puppet is applied locally on the node, without a daemon or puppetmaster. - -You can run `leap -v2 deploy` to see exactly what commands are being executed. - -This mode of operation is fundamentally different from how puppet is normally used: - -* There is no puppetmaster that all the servers take orders from, and there is no puppetd running in the background. -* Servers cannot dynamically query the puppetmaster for information about the other servers. -* There is a static representation for the state of every server that can be committed to git. - -There are advantages and disadvantages to the model that LEAP uses. We have found it very useful for our goal of having a common LEAP platform that many different providers can all use while still allowing providers to configure their unique infrastructure. - -We also find it very beneficial to be able to track the state of your infrastructure in git. - -Traditional system configuration automation systems, like [Puppet](https://puppetlabs.com/puppet/puppet-open-source/) or [Chef](http://www.opscode.com/chef/), deploy changes to servers using a pull method. Each server pulls a manifest from a central master server and uses this to alter the state of the server. - -Instead, the `leap` tool uses a masterless push method: The sysadmin runs `leap deploy` from the provider instance directory on their desktop machine to push the changes out to every server (or a subset of servers). LEAP still uses Puppet, but there is no central master server that each node must pull from. - -One other significant difference between LEAP and typical system automation is how interactions among servers are handled. Rather than store a central database of information about each server that can be queried when a recipe is applied, the `leap` command compiles static representation of all the information a particular server will need in order to apply the recipes. In compiling this static representation, `leap` can use arbitrary programming logic to query and manipulate information about other servers. - -These two approaches, masterless push and pre-compiled static configuration, allow the sysadmin to manage a set of LEAP servers using traditional software development techniques of branching and merging, to more easily create local testing environments using virtual servers, and to deploy without the added complexity and failure potential of a master server. diff --git a/doc/guide/keys-and-certificates.md b/doc/guide/keys-and-certificates.md deleted file mode 100644 index a6862a6a..00000000 --- a/doc/guide/keys-and-certificates.md +++ /dev/null @@ -1,272 +0,0 @@ -@title = "Keys and Certificates" -@summary = "Working with SSH keys, secrets, and X.509 certificates." - -Working with SSH -================================ - -Whenever the `leap` command needs to push changes to a node or gather information from a node, it tunnels this command over SSH. Another way to put this: the security of your servers rests entirely on SSH. Because of this, it is important that you understand how `leap` uses SSH. - -SSH related files -------------------------------- - -Assuming your provider directory is called 'provider': - -* `provider/nodes/crow/crow_ssh.pub` -- The public SSH host key for node 'crow'. -* `provider/users/alice/alice_ssh.pub` -- The public SSH user key for user 'alice'. Anyone with the private key that corresponds to this public key will have root access to all nodes. -* `provider/files/ssh/known_hosts` -- An autogenerated known_hosts, built from combining `provider/nodes/*/*_ssh.pub`. You must not edit this file directly. If you need to change it, remove or change one of the files that is used to generate `known_hosts` and then run `leap compile`. -* `provider/files/ssh/authorized_keys` -- An autogenerated list of all the user SSH keys with root access to the notes. It is created from `provider/users/*/*_ssh.pub`. You must not edit this file directly. If you need to change it, remove or change one of the files that is used to generate `authorized_keys` and then run `leap compile`. - -All of these files should be committed to source control. - -If you rename, remove, or add a node with `leap node [mv|add|rm]` the SSH key files and the `known_hosts` file will get properly updated. - -SSH and local nodes -------------------- - -Local nodes are run as Vagrant virtual machines. The `leap` command handles SSH slightly differently for these nodes. - -Basically, all the SSH security is turned off for local nodes. Since local nodes only exist for a short time on your computer and can't be reached from the internet, this is not a problem. - -Specifically, for local nodes: - -1. `known_hosts` is never updated with local node keys, since the SSH public key of a local node is different for each user. -2. `leap` entirely skips the checking of host keys when connecting with a local node. -3. `leap` adds the public Vagrant SSH key to the list of SSH keys for a user. The public Vagrant SSH key is a shared and insecure key that has root access to most Vagrant virtual machines. - -To upgrade a SSH host key -------------------------------- - -Most servers will have more than one SSH host key. Sometimes, the server will have a better SSH host key than the one you have on file. In order to upgrade to the better SSH host key, simply re-run the init command: - - workstation$ leap node init NODE_NAME - -This will prompt you if you want to upgrade the SSH host key, but only if `leap` thinks that an upgrade is advisable. - -When SSH host key changes -------------------------------- - -If the host key for a node has changed, you will get an error "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED". - -To fix this, you need to remove the file `files/nodes/stompy/stompy_ssh.pub` and run `leap node init stompy`, where the node's name is 'stompy'. **Only do this if you are ABSOLUTELY CERTAIN that the node's SSH host key has changed**. - -Changing the SSH port --------------------------------- - -Suppose you have a node `blinky` that has SSH listening on port 22 and you want to make it port 2200. - -First, modify the configuration for `blinky` to specify the variable `ssh.port` as 2200. Usually, this is done in `common.json` or in a tag file. - -For example, you could put this in `tags/production.json`: - - { - "ssh": { - "port": 2200 - } - } - -Run `leap compile` and open `hiera/blinky.yaml` to confirm that `ssh.port` is set to 2200. The port number must be specified as a number, not a string (no quotes). - -Then, you need to deploy this change so that SSH will bind to 2200. You cannot simply run `leap deploy blinky` because this command will default to using the variable `ssh.port` which is now `2200` but SSH on the node is still bound to 22. - -So, you manually override the port in the deploy command, using the old port: - - leap deploy --port 22 blinky - -Afterwards, SSH on `blinky` should be listening on port 2200 and you can just run `leap deploy blinky` from then on. - -Sysadmins with multiple SSH keys ------------------------------------ - -The command `leap add-user --self` allows only one SSH key. If you want to specify more than one key for a user, you can do it manually: - - users/userx/userx_ssh.pub - users/userx/otherkey_ssh.pub - -All keys matching 'userx/*_ssh.pub' will be usable. - -Removing sysadmin access --------------------------------- - -Suppose you want to remove `userx` from having any further SSH access to the servers. Do this: - - rm -r users/userx - leap deploy - -X.509 Certificates -================================ - -Configuration options -------------------------------------------- - -The `ca` option in provider.json provides settings used when generating CAs and certificates. The defaults are as follows: - - { - "ca": { - "name": "= global.provider.ca.organization + ' Root CA'", - "organization": "= global.provider.name[global.provider.default_language]", - "organizational_unit": "= 'https://' + global.provider.domain", - "bit_size": 4096, - "digest": "SHA256", - "life_span": "10y", - "server_certificates": { - "bit_size": 2048, - "digest": "SHA256", - "life_span": "1y" - }, - "client_certificates": { - "bit_size": 2048, - "digest": "SHA256", - "life_span": "2m", - "limited_prefix": "LIMITED", - "unlimited_prefix": "UNLIMITED" - } - } - } - -You should not need to override these defaults in your own provider.json, but you can if you want to. To see what values are used for your provider, run `leap inspect provider.json`. - -NOTE: A certificate `bit_size` greater than 2048 will probably not be recognized by most commercial CAs. - -Certificate Authorities ------------------------------------------ - -There are three x.509 certificate authorities (CA) associated with your provider: - -1. **Commercial CA:** It is strongly recommended that you purchase a commercial cert for your primary domain. The goal of platform is to not depend on the commercial CA system, but it does increase security and usability if you purchase a certificate. The cert for the commercial CA must live at `files/cert/commercial_ca.crt`. -2. **Server CA:** This is a self-signed CA responsible for signing all the **server** certificates. The private key lives at `files/ca/ca.key` and the public cert lives at `files/ca/ca.crt`. The key is very sensitive information and must be kept private. The public cert is distributed publicly. -3. **Client CA:** This is a self-signed CA responsible for signing all the **client** certificates. The private key lives at `files/ca/client_ca.key` and the public cert lives at `files/ca/client_ca.crt`. Neither file is distribute publicly. It is not a big deal if the private key for the client CA is compromised, you can just generate a new one and re-deploy. - -To generate both the Server CA and the Client CA, run the command: - - leap cert ca - -Server certificates ------------------------------------ - -Most every server in your service provider will have a x.509 certificate, generated by the `leap` command using the Server CA. Whenever you modify any settings of a node that might affect it's certificate (like changing the IP address, hostname, or settings in provider.json), you can magically regenerate all the certs that need to be regenerated with this command: - - leap cert update - -Run `leap help cert update` for notes on usage options. - -Because the server certificates are generated locally on your personal machine, the private key for the Server CA need never be put on any server. It is up to you to keep this file secure. - -Client certificates --------------------------------- - -Every leap client gets its own time-limited client certificate. This cert is use to connect to the OpenVPN gateway (and probably other things in the future). It is generated on the fly by the webapp using the Client CA. - -To make this work, the private key of the Client CA is made available to the webapp. This might seem bad, but compromise of the Client CA simply allows the attacker to use the OpenVPN gateways without paying. In the future, we plan to add a command to automatically regenerate the Client CA periodically. - -There are two types of client certificates: limited and unlimited. A client using a limited cert will have its bandwidth limited to the rate specified by `provider.service.bandwidth_limit` (in Bytes per second). An unlimited cert is given to the user if they authenticate and the user's service level matches one configured in `provider.service.levels` without bandwidth limits. Otherwise, the user is given a limited client cert. - -Commercial certificates ------------------------------------ - -We strongly recommend that you use a commercial signed server certificate for your primary domain (in other words, a certificate with a common name matching whatever you have configured for `provider.domain`). This provides several benefits: - -1. When users visit your website, they don't get a scary notice that something is wrong. -2. When a user runs the LEAP client, selecting your service provider will not cause a warning message. -3. When other providers first discover your provider, they are more likely to trust your provider key if it is fetched over a commercially verified link. - -The LEAP platform is designed so that it assumes you are using a commercial cert for the primary domain of your provider, but all other servers are assumed to use non-commercial certs signed by the Server CA you create. - -To generate a CSR, run: - - leap cert csr - -This command will generate the CSR and private key matching `provider.domain` (you can change the domain with `--domain=DOMAIN` switch). It also generates a server certificate signed with the Server CA. You should delete this certificate and replace it with a real one once it is created by your commercial CA. - -The related commercial cert files are: - - files/ - cert/ - domain.org.crt # Server certificate for domain.org, obtained by commercial CA. - domain.org.csr # Certificate signing request - domain.org.key # Private key for you certificate - commercial_ca.crt # The CA cert obtained from the commercial CA. - -The private key file is extremely sensitive and care should be taken with its provenance. - -If your commercial CA has a chained CA cert, you should be OK if you just put the **last** cert in the chain into the `commercial_ca.crt` file. This only works if the other CAs in the chain have certs in the debian package `ca-certificates`, which is the case for almost all CAs. - -If you want to add additional fields to the CSR, like country, city, or locality, you can configure these values in provider.json like so: - - "ca": { - "server_certificates": { - "country": "US", - "state": "Washington", - "locality": "Seattle" - } - } - -If they are not present, the CSR will be created without them. - -Examine Certs ------------------ - -To see details about the keys and certs you can use `leap inspect` like so: - - $ leap inspect files/ca/ca.crt - - -Let's Encrypt certificate -========================= - -LEAP plans to integrate [Let's Encrypt](https://letsencrypt.org/) support, so it will be even easier to receive X.509 certificates that are accepted by all browsers. -Until we achieve this, here's a guide how to do this manually. - -Install the official acme client --------------------------------- - -Log in to your webapp node - - server$ git clone https://github.com/letsencrypt/letsencrypt - server$ cd letsencrypt - server$ ./letsencrypt-auto --help - -Fetch cert ----------- - -Stop apache so the letsencrypt client can bind to port 80: - - server$ systemctl stop apache2 - -Fetch the certs - - server$ ./letsencrypt-auto certonly --standalone --email admin@$(hostname -d) -d $(hostname -d) -d api.$(hostname -d) -d $(hostname -f) -d nicknym.$(hostname -d) - -This will put the certs and keys into `/etc/letsencrypt/live/DOMAIN/`. - -Now, go to your workstation's provider configuration directory and copy the newly created files from the server to your local config. You will override existing files so please make a backup before proceeding, or use a version control system to track changes. - - workstation$ cd PATH_TO_PROVIDER_CONFIG - -Copy the Certificate - - workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/cert.pem files/cert/dev.pixelated-project.org.crt - -Copy the private key - - workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/privkey.pem files/cert/DOMAIN.key - -Copy the CA chain cert - - workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/fullchain.pem files/cert/DOMAIN.key - -Deploy the certs ----------------- - -Now you only need to deploy the certs - - workstation$ leap deploy - -This will put them into the right locations which are: - -- `/etc/x509/certs/leap_commercial.crt` for the certificate -- `/etc/x509/./keys/leap_commercial.key` for the private key -- `/usr/local/share/ca-certificates/leap_commercial_ca.crt` for the CA chain cert. - -Start apache2 again - - server$ systemctl start apache2 diff --git a/doc/guide/miscellaneous.md b/doc/guide/miscellaneous.md deleted file mode 100644 index c38c007c..00000000 --- a/doc/guide/miscellaneous.md +++ /dev/null @@ -1,14 +0,0 @@ -@title = "Miscellaneous" -@summary = "Miscellaneous commands you may need to know." - -Facts -============================== - -There are a few cases when we must gather internal data from a node before we can successfully deploy to other nodes. This is what `facts.json` is for. It stores a snapshot of certain facts about each node, as needed. Entries in `facts.json` are updated automatically when you initialize, rename, or remove a node. To manually force a full update of `facts.json`, run: - - leap facts update FILTER - -Run `leap help facts update` for more information. - -The file `facts.json` should be committed to source control. You might not have a `facts.json` if one is not required for your provider. - diff --git a/doc/guide/nodes.md b/doc/guide/nodes.md deleted file mode 100644 index 5135f3ba..00000000 --- a/doc/guide/nodes.md +++ /dev/null @@ -1,87 +0,0 @@ -@title = "Nodes" -@summary = "Working with nodes, services, tags, and locations." - -Locations -================================ - -All nodes should have a `location.name` specified, and optionally additional information about the location, like the time zone. This location information is used for two things: - -* Determine which nodes can, or must, communicate with one another via a local network. The way some virtualization environments work, like OpenStack, requires that nodes communicate via the local network if they are on the same network. -* Allows the client to prefer connections to nodes that are closer in physical proximity to the user. This is particularly important for OpenVPN nodes. - -The location stanza in a node's config file looks like this: - - { - "location": { - "id": "ankara", - "name": "Ankara", - "country_code": "TR", - "timezone": "+2", - "hemisphere": "N" - } - } - -The fields: - -* `id`: An internal handle to use for this location. If two nodes have match `location.id`, then they are treated as being on a local network with one another. This value defaults to downcase and underscore of `location.name`. -* `name`: Can be anything, might be displayed to the user in the client if they choose to manually select a gateway. -* `country_code`: The [ISO 3166-1](https://en.wikipedia.org/wiki/ISO_3166-1) two letter country code. -* `timezone`: The timezone expressed as an offset from UTC (in standard time, not daylight savings). You can look up the timezone using this [handy map](http://www.timeanddate.com/time/map/). -* `hemisphere`: This should be "S" for all servers in South America, Africa, or Australia. Otherwise, this should be "N". - -These location options are very imprecise, but good enough for most usage. The client often does not know its own location precisely either. Instead, the client makes an educated guess at location based on the OS's timezone and locale. - -If you have multiple nodes in a single location, it is best to use a tag for the location. For example: - -`tags/ankara.json`: - - { - "location": { - "name": "Ankara", - "country_code": "TR", - "timezone": "+2", - "hemisphere": "N" - } - } - -`nodes/vpngateway.json`: - - { - "services": "openvpn", - "tags": ["production", "ankara"], - "ip_address": "1.1.1.1", - "openvpn": { - "gateway_address": "1.1.1.2" - } - } - -Unless you are using OpenStack or AWS, setting `location` for nodes is not required. It is, however, highly recommended. - -Disabling Nodes -===================================== - -There are two ways to temporarily disable a node: - -**Option 1: disabled environment** - -You can assign an environment to the node that marks it as disabled. Then, if you use environment pinning, the node will be ignored when you deploy. For example: - - { - "environment": "disabled" - } - -Then use `leap env pin ENV` to pin the environment to something other than 'disabled'. This only works if all the other nodes are also assigned to some environment. - -**Option 2: enabled == false** - -If a node has a property `enabled` set to false, then the `leap` command will skip over the node and pretend that it does not exist. For example: - - { - "ip_address": "1.1.1.1", - "services": ["openvpn"], - "enabled": false - } - -**Options 3: no-deploy** - -If the file `/etc/leap/no-deploy` exists on a node, then when you run the commmand `leap deploy` it will halt and prevent a deploy from going through (if the node was going to be included in the deploy). diff --git a/doc/guide/provider-configuration.md b/doc/guide/provider-configuration.md deleted file mode 100644 index 08cfd1dd..00000000 --- a/doc/guide/provider-configuration.md +++ /dev/null @@ -1,79 +0,0 @@ -@title = "Provider Configuration" -@summary = "Explore how to configure your provider." - -Required provider configuration --------------------------------------- - -There are a few required settings in `provider.json`. At a minimum, you must have: - -* `domain`: defines the primary domain of the provider. This is the domain that users will type in when using the Bitmask client, although it is not necessarily the domain where users will visit if they sign up via the web application. If email is supported, all accounts will be `username@domain`. -* `name`: A brief title for this provider. It can be multiple words, but should not be too long. -* `contacts.default`: One or more email addresses for sysadmins. - -For example: - - { - "domain": "freerobot.org", - "name": "Freedom for Robots!", - "contacts": { - "default": "root@freerobot.org" - } - } - - -Recommended provider configuration --------------------------------------- - -* `description`: A longer description of the provider, shown to the user when they register a new account through Bitmask client. -* `languages`: A list of language codes that should be enabled. -* `default_language`: The initial default language code. -* `enrollment_policy`: One of "open", "closed", or "invite". (invite not currently supported). - -For example: - - { - "description": "It is time for robots of the world to unite and throw of the shackles of servitude to our organic overlords.", - "languages": ["en", "de", "pt", "01"], - "default_language": "01", - "enrollman_policy": "open" - } - -For a full list of possible settings, you can use `leap inspect` to see how provider.json is evaluated after including the inherited defaults: - - $ leap inspect provider.json - -Configuring service levels --------------------------------------- - -The `provider.json` file defines the available service levels for the provider. - -For example, in provider.json: - - "service": { - "default_service_level": "low", - "levels": { - "low": { - "description": "Entry level plan, with unlimited bandwidth and minimal storage quota.", - "name": "entry", - "storage": "10 MB", - "rate": { - "USD": 5, - "GBP": 3, - "EUR": 6 - } - }, - "full": { - "description": "Full plan, with unlimited bandwidth and higher quota." - "name": "full", - "storage": "5 GB", - "rate": { - "USD": 10, - "GBP": 6, - "EUR": 12 - } - } - } - } - } - -For a list of currency codes, see https://en.wikipedia.org/wiki/ISO_4217#Active_codes diff --git a/doc/service-diagram.odg b/doc/service-diagram.odg deleted file mode 100644 index 09265c2d..00000000 Binary files a/doc/service-diagram.odg and /dev/null differ diff --git a/doc/service-diagram.png b/doc/service-diagram.png deleted file mode 100644 index 85e62436..00000000 Binary files a/doc/service-diagram.png and /dev/null differ diff --git a/doc/services/couchdb.md b/doc/services/couchdb.md deleted file mode 100644 index cc40dc32..00000000 --- a/doc/services/couchdb.md +++ /dev/null @@ -1,159 +0,0 @@ -@title = "couchdb" -@summary = "Data storage for all user data." - -Topology ------------------------- - -Required: - -* Nodes with `couchdb` service must also have `soledad` service, if email is enabled. - -Suggested: - -* Nodes with `couchdb` service communicate heavily with `webapp` and `mx`. - -`couchdb` nodes do not need to be reachable from the public internet, although the `soledad` service does require this. - -Configuration ----------------------------- - -### Nighly dumps - -You can do a nightly couchdb data dump by adding this to your node config: - - "couch": { - "backup": true - } - -Data will get dumped to `/var/backups/couchdb`. - -### Plain CouchDB - -BigCouch is not supported on Platform version 0.8 and higher: only plain CouchDB is possible. For earlier versions, you must do this in order to use plain CouchDB: - - "couch": { - "master": true, - "pwhash_alg": "pbkdf2" - } - -Various Tasks -------------------------------------------------- - -### Re-enabling blocked account - -When a user account gets destroyed from the webapp, there's still a leftover doc in the identities db so other people can't claim that account without an admin's intervention. You can remove this username reservation through the webapp. - -However, here is how you could do it manually, if you wanted to: - -grep the identities db for the email address: - - curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET http://127.0.0.1:5984/identities/_all_docs?include_docs=true|grep test_127@bitmask.net - -lookup "id" and "rev" to delete the doc: - - curl -s --netrc-file /etc/couchdb/couchdb.netrc -X DELETE 'http://127.0.0.1:5984/identities/b25cf10f935b58088f0d547fca823265?rev=2-715a9beba597a2ab01851676f12c3e4a' - -### How to find out which userstore belongs to which identity? - - /usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep testuser - - {"id":"665e004870ee17aa4c94331ff3ecb173","key":"665e004870ee17aa4c94331ff3ecb173","value":{"rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b"},"doc":{"_id":"665e004870ee17aa4c94331ff3ecb173","_rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b","user_id":"665e004870ee17aa4c94331ff3cd59eb","address":"testuser@example.org","destination":"testuser@example.org","keys": ... - -* search for the "user_id" field -* in this example testuser@example.org uses the database user-665e004870ee17aa4c94331ff3cd59eb - - -### How much disk space is used by a userstore - -Beware that this returns the uncompacted disk size (see http://wiki.apache.org/couchdb/Compaction) - - echo "`curl --netrc -s -X GET 'http://127.0.0.1:5984/user-dcd6492d74b90967b6b874100b7dbfcf'|json_pp|grep disk_size|cut -d: -f 2`/1024"|bc - - -Deprecated BigCouch Tasks ------------------------------------------ - -As of release 0.8, the LEAP platform no longer supports BigCouch. This information is kept here for historical reference. - -### Rebalance Cluster - -Bigcouch currently does not have automatic rebalancing. -It will probably be added after merging into couchdb. -If you add a node, or remove one node from the cluster, - -1. make sure you have a backup of all DBs ! - -1. put the webapp into [[maintenance mode => services/webapp#maintenance-mode]] - -1. Stop all services that access the database: - - ``` - workstation$ leap ssh soledad-nodes - server# /etc/init.d/soledad-server stop - - workstation$ leap ssh mx-node - server# /etc/init.d/postfix stop - server# /etc/init.d/leap-mx stop - - workstation$ leap ssh webapp - server# /etc/init.d/nickserver stop - ``` - - Alternately, you can create a temporary firewall rule to block access (run on couchdb server): - - ``` - server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT - ``` - -1. dump the dbs: - - ``` - cd /srv/leap/couchdb/scripts - time ./couchdb_dumpall.sh - ``` - -1. delete all dbs - -1. shut down old node - -1. check the couchdb members - - ``` - curl -s —netrc-file /etc/couchdb/couchdb.netrc -X GET http://127.0.0.1:5986/nodes/_all_docs - curl -s —netrc-file /etc/couchdb/couchdb.netrc http://127.0.0.1:5984/_membership - ``` - -1. remove bigcouch from all nodes - - ``` - apt-get --purge remove bigcouch - ``` - -1. deploy to all couch nodes - - ``` - leap deploy couchdb - ``` - -1. most likely, deploy will fail because bigcouch will complain about not all nodes beeing connected. Let the deploy finish, restart the bigcouch service on all nodes and re-deploy: - - ``` - /etc/init.d/bigcouch restart - ``` - -1. restore the backup - - ``` - cd /srv/leap/couchdb/scripts - time ./couchdb_restoreall.sh - ``` - -### Migrating from BigCouch to plain CouchDB - -<%= render :partial => 'docs/platform/common/bigcouch_migration_begin.md' %> - - -<%= render :partial => 'docs/platform/common/bigcouch_migration_end.md' %> - - -<%= render :partial => 'docs/platform/common/bigcouch_migration_finish.md' %> diff --git a/doc/services/en.md b/doc/services/en.md deleted file mode 100644 index 5d0fec5f..00000000 --- a/doc/services/en.md +++ /dev/null @@ -1,80 +0,0 @@ -@nav_title = "Services" -@title = "Guide to node services" -@summary = "" -@toc = true - -# Introduction - -Every node (server) must have one or more `services` defined that determines what role the node performs. For example: - - workstation$ cat nodes/stallman.json - { - "ip_address": "199.99.99.1", - "services": ["webapp", "tor"] - } - -Here are common questions to ask when adding a new node to your provider: - -* **many or few?** Some services benefit from having many nodes, while some services are best run on only one or two nodes. -* **required or optional?** Some services are required, while others can be left out. -* **who does the node communicate with?** Some services communicate very heavily with other particular services. Nodes running these services should be close together. -* **public or private network?** Some services communicate with the public internet, while others only need to communicate with other nodes in the infrastructure. - -# Available services - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ServiceVPNEmailNotes
webappUser control panel, provider API, and support system.
couchdbData storage for everything. Private node.
soledadUser data synchronization daemon. Usually paired with couchdb nodes.
mxIncoming and outgoing MX servers.
openvpnOpenVPN gateways.
monitorNagios monitoring. This service must be on the webapp node.
torTor exit node.
- -Key: Required, Optional, Not Used - -<%= child_summaries %> \ No newline at end of file diff --git a/doc/services/monitor.md b/doc/services/monitor.md deleted file mode 100644 index 576b36a9..00000000 --- a/doc/services/monitor.md +++ /dev/null @@ -1,36 +0,0 @@ -@title = "monitor" -@summary = "Nagios monitoring and continuous testing." - -The `monitor` node provides a nagios control panel that will give you a view into the health and status of all the servers and all the services. It will also spam you with alerts if something goes down. - -Topology --------------------------------------- - -Currently, you can have zero or one `monitor` nodes defined. It is required that the monitor be on the webapp node. It was not designed to be run as a separate node service. - -Configuration ------------------------------------------------ - -* `nagios.environments`: By default, the monitor node will monitor all servers in all environments. You can optionally restrict the environments to the ones you specify. - -For example: - - { - "nagios": { - "environments": ["unstable", "production"] - } - } - -Access nagios web ------------------------------------------------ - -*Determine the nagios URL* - - $ leap ls --print domain.name,webapp.domain,ip_address monitor - > chameleon chameleon.bitmask.net, demo.bitmask.net, 199.119.112.10 - -In this case, you would open `https://demo.bitmask.net/cgi-bin/nagios3` in your browser (or alternately you could use 199.119.112.10 or chameleon.bitmask.net). - -*Determine the nagios password* - -The username for nagios is always `nagiosadmin`. The password is randomly generated and stored in `secrets.json` under the key `nagios_admin_password`. Note that the login is `nagiosadmin` without underscore, but the entry in secrets.json is with underscores. diff --git a/doc/services/mx.md b/doc/services/mx.md deleted file mode 100644 index 1a34b660..00000000 --- a/doc/services/mx.md +++ /dev/null @@ -1,35 +0,0 @@ -@title = "mx" -@summary = "Incoming and outgoing MX servers." - -Topology -------------------- - -`mx` nodes communicate with the public internet, clients, and `couchdb` nodes. - -Configuration --------------------- - -### Aliases - -Using the `mx.aliases` property, you can specify your own hard-coded email aliases that precedence over the aliases in the user database. The `mx.aliases` property consists of a hash, where source address points to one or more destination addresses. - -For example: - -`services/mx.json`: - - "mx": { - "aliases": { - "rook": "crow", - "robin": "robin@bird.org", - "flock": ["junco@bird.org", "robin", "crow"], - "chickadee@avian.org": "chickadee@bird.org", - "flicker": ["flicker@bird.org", "flicker@deliver.local"] - } - } - -This example demonstrates several of the features with `mx.aliases`: - -1. alias lists: by specifying an array of destination addresses, as in the case of "flock", the single email will get copied to each address. -1. chained resolution: alias resolution will recursively continue until there are no more matching aliases. For example, "flock" is resolved to "robin", which then gets resolved to "robin@bird.org". -1. virtual domains: by specifying the full domain, as in the case of "chickadee@avian.org", the alias will work for any domain you want. Of course, the MX record for that domain must point to appropriate MX servers, but otherwise you don't need to do any additional configuration. -1. local delivery: for testing purposes, it is often useful to copy all incoming mail for a particular address and send those copies to another address. You can do this by adding "@deliver.local" as one of the destination addresses. When "@local.delivery" is found, alias resolution stops and the mail is delivered to that username. diff --git a/doc/services/openvpn.md b/doc/services/openvpn.md deleted file mode 100644 index 5f15ff07..00000000 --- a/doc/services/openvpn.md +++ /dev/null @@ -1,49 +0,0 @@ -@title = 'openvpn' -@summary = "OpenVPN egress gateways" - -Topology ------------------- - -Currently, `openvpn` service should not be combined with other services on the same node. - -Unlike most of the other node types, the `openvpn` nodes do not need access to the database and does not ever communicate with any other nodes (except for the `monitor` node, if used). So, `openvpn` nodes can be placed anywhere without regard to the other nodes. - -Configuration ---------------------- - -*Essential configuration* - -* `openvpn.gateway_address`: The address that OpenVPN daemon is bound to and that VPN clients connect to. -* `ip_address`: The main IP of the server, and the egress address for outgoing traffic. - -For example: - - { - "ip_address": "1.1.1.1", - "openvpn": { - "gateway_address": "2.2.2.2" - } - } - -In this example, VPN clients will connect to 2.2.2.2, but their traffic will appear to come from 1.1.1.1. - -Why are two IP addresses needed? Without this, traffic between two VPN users on the same gateway will not get encrypted. This is because the VPN on every client must be configured to allow cleartext traffic for the IP address that is the VPN gateway. - -*Optional configuration* - -Here is the default configuration: - - "openvpn": { - "configuration": { - "auth": "SHA1", - "cipher": "AES-128-CBC", - "fragment": 1400, - "keepalive": "10 30", - "tls-cipher": "DHE-RSA-AES128-SHA", - "tun-ipv6": true - }, - "ports": ["80", "443", "53", "1194"], - "protocols": ["tcp", "udp"] - } - -You may want to change the ports so that only 443 or 80 are used. It is probably best to not modify the `openvpn.configuration` options for now. \ No newline at end of file diff --git a/doc/services/soledad.md b/doc/services/soledad.md deleted file mode 100644 index e2700d06..00000000 --- a/doc/services/soledad.md +++ /dev/null @@ -1,12 +0,0 @@ -@title = 'soledad' -@summary = 'User data synchronization daemon' - -Topology --------------------- - -Currently, the platform is designed for `soledad` and `couchdb` services to be combined (e.g. every `soledad` node should also be a `couchdb` node). `soledad` nodes might work in isolation, but this is not tested. - -Configuration ------------------------- - -There are no options to configure for `soledad` nodes. diff --git a/doc/services/tor.md b/doc/services/tor.md deleted file mode 100644 index e64b0fe0..00000000 --- a/doc/services/tor.md +++ /dev/null @@ -1,32 +0,0 @@ -@title = 'tor' -@summary = 'Tor exit node or hidden service' - -Topology ------------------------- - -Nodes with `tor` service will run a Tor exit or hidden service, depending on what other service it is paired with: - -* `tor` + `openvpn`: when combined with `openvpn` nodes, `tor` will create a Tor exit node to provide extra cover traffic for the VPN. This can be especially useful if there are VPN gateways without much traffic. -* `tor` + `webapp`: when combined with a `webapp` node, the `tor` service will make the webapp and the API available via .onion hidden service. -* `tor` stand alone: a regular Tor exit node. - -If activated, you can list the hidden service .onion addresses this way: - - leap ls --print tor.hidden_service.address tor - -Then just add '.onion' to the end of the printed addresses. - -Configuration ------------------------------- - -* `tor.bandwidth_rate`: the max bandwidth allocated to Tor, in KB per second, when used as an exit node. - -For example: - - { - "tor": { - "bandwidth_rate": 6550 - } - } - - diff --git a/doc/services/webapp.md b/doc/services/webapp.md deleted file mode 100644 index 1c06d715..00000000 --- a/doc/services/webapp.md +++ /dev/null @@ -1,293 +0,0 @@ -@title = "webapp" -@summary = "leap_web user management application and provider API." - -Introduction ------------------------- - -The service `webapp` will install the web application [[leap_web => https://leap.se/git/leap_web.git]]. It has performs the following functions: - -* REST API for user registration and authentication via the Bitmask client. -* Admin interface to manage users. -* Client certificate distribution and renewal. -* User support help tickets. - -Coming soon: - -* Billing. -* Customizable and localized user documentation. - -The leap_web application is written in Ruby on Rails 3, using CouchDB as the backend data store. - -Topology -------------------------- - -Currently, the platform only supports a single `webapp` node, although we hope to change this in the future. - -* `webapp` nodes communicate heavily with `couchdb` nodes, but the two can be on separate servers. -* The `monitor` service, if enabled, must be on the same node as `webapp`. - -Configuration --------------------------- - -Essential options: - -* `webapp.admin`: An array of usernames that will be blessed with administrative permissions. These admins can delete users, answer help tickets, and so on. These usernames are for users that have registered through the webapp or through the Bitmask client application, NOT the sysadmin usernames lists in the provider directory `users`. - -Other options: - -* `webapp.engines`: A list of the engines you want enabled in leap_web. Currently, only "support" is available, and it is enabled by default. -* `webapp.invite_required`: If true, registration requires an invite code. Default is `false`. - -For example, `services/webapp.json`: - - { - "webapp": { - "admins": ["joehill", "ali", "mack_the_turtle"] - } - } - -By putting this in `services/webapp.json`, all the `webapp` nodes will inherit the same admin list. - -There are many options in `provider.json` that also control how the webapp behaves. See [[provider-configuration]] for details. - -Invite codes -------------------- - -Enabling the invite code functionality will require new users to provide a valid invite code while signing up for a new account. This is turned off by default, allowing all new users to create an account. - -Set the `invite_code` option to `true` in `services/webapp.json`: - - { - "webapp": { - "invite_required": true - } - } - -This only works with LEAP platform 0.8 or higher. - -Run `leap deploy` to enable the option. - -You can then generate invite codes by logging into the web application with an admin user. - -Alternately, you can also generate invite codes with the command line: - - workstation$ leap ssh bumblebee - bumblebee# cd /srv/leap/webapp/ - bumblebee# sudo -u leap-webapp RAILS_ENV=production bundle exec rake "generate_invites[NUM,USES]" - -Where `bumblebee` should be replaced with the name of your webapp node. - -The **NUM** specifies the amount of codes to generate. The **USES** parameter is optional: By default, all new invite codes can be used once and will then become invalid. If you provide another value for **USES**, you can set a different amount of maximum uses for the codes you generate. - -Customization ---------------------------- - -The provider directory `files/webapp` can be used to customize the appearance of the webapp. All the files in this directory will get sync'ed to the `/srv/leap/webapp/config/customization` directory of the deployed webapp node. - -Files in the `files/webapp` can override view files, locales, and stylesheets in the leap_web app: - -For example: - - stylesheets/ -- override files in Rails.root/app/assets/stylesheets - tail.scss -- included before all others - head.scss -- included after all others - - public/ -- overrides files in Rails.root/public - favicon.ico -- custom favicon - img/ -- customary directory to put images in - - views/ -- overrides files Rails.root/app/views - home/ - index.html.haml -- this file is what shows up on - the home page - pages/ - privacy-policy.en.md -- this file will override - the default privacy policy - terms-of-service.en.md -- this file will override - the default TOS. - - locales/ -- overrides files in Rails.root/config/locales - en.yml -- overrides for English - de.yml -- overrides for German - and so on... - -To interactively develop your customizations before you deploy them, you have two options: - -1. Edit a `webapp` node. This approach involves directly modifying the contents of the directory `/srv/leap/webapp/config/customization` on a deployed `webapp` node. This can, and probably should be, a "local" node. When doing this, you may need to restart leap_web in order for changes to take effect (`touch /srv/leap/webapp/tmp/restart.txt`). -2. Alternately, you can install leap_web to run on your computer and edit files in `config/customization` locally. This approach does not require a provider or a `webapp` node. For more information, see the [leap_web README](https://github.com/leapcode/leap_web). - -NOTE: If you add a `tails.scss` or `head.scss` file, then you usually need to run `rake tmp:clear` and restart rails in order for the new stylesheet to get recognized. You should only need to do this once. - -Once you have what you want, then copy these files to the local provider directory `files/webapp` so that they will be installed each time you deploy. - -Customization tutorial ----------------------------- - -This mini-tutorial will walk you through creating a custom "branding" of the leap_web application. We will be creating a provider called "Prehistoric Computer." - -Here are the files we are going to create: - - leap_web/config/customization - ├── locales - │   ├── en.yml - │   └── es.yml - ├── public - │   ├── favicon.ico - │   └── img - │   └── masthead.png - ├── stylesheets - │   └── tail.scss - └── views - └── pages - ├── privacy-policy.en.md - └── privacy-policy.es.md - -All these files are available in the source code in the [[customization.example => https://github.com/leapcode/leap_web/tree/develop/config/customization.example]] directory. - -Remember, these files may live different places: - -* `user@localmachine$ leap_web/config/customization`: This will be the path if you have checked out a local copy of leap_web.git and are running `rails server` locally in order to test your customizations. -* `user@localmachine$ PROVIDER/files/webapp`: This is the local provider directory where the files should be put so that they get correctly deployed to webapp nodes. -* `root@webappnode# /srv/leap/webapp/config/customization`: This is where the files in the local provider directory `PROVIDER/files/webapp` get copied to after a `leap deploy` to a live webapp nodes. - -### Override translations - -You can add additional locale files in order to change the text used in the existing application and to add translations for string that you added to the application. - -In this example, we will be altering the default text for the "login_info" string. In `config/locales/en/home.en.yml` there is this entry: - - en: - login_info: "Log in to change your account settings, create support tickets, and manage payments." - -We are going to override this with some custom text in English and Spanish: - -`leap_web/config/customization/locale/en.yml`: - - en: - login_info: Authenticate to change your "Prehistoric Computer" settings. - -`leap_web/config/customization/locale/es.yml`: - - es: - login_info: Autenticar a cambiar la configuración de "Computer Prehistoria." - -Now, the home page of leap_web will use these new strings instead of the default. Remember that you must restart rails in order for new locale files to take effect. - -### Override static pages - -You can also override any of the static files included with leap_web, such as the privacy policy or terms of service. - -Here is how we would create a custom privacy policy in English and Spanish: - -`leap_web/config/customization/views/pages/privacy-policy.en.md`: - - # Custom Privacy Policy - This is our privacy policy. - -`leap_web/config/customization/views/pages/privacy-policy.es.md`: - - # Custom Política de Privacidad - Esta es nuestra política de privacidad. - -### Add a custom header - -Now we will add a custom header to every page. First, we add the images: - - leap_web/config/customization - ├── public - ├── favicon.ico - └── img - └── masthead.png - -You can create your own, or use the example files in https://github.com/leapcode/leap_web/tree/develop/config/customization.example - -Now, we add some custom CSS so that we can style the masthead: - -`leap_web/config/customization/stylesheets/tail.scss` - - $custom-color: #66bbaa; - - a { - color: $custom-color; - } - - // - // MASTHEAD - // - - #masthead { - background-color: $custom-color; - border-bottom: none; - - // make the masthead clickable by replacing the - // site name link with the masthead image: - .title { - padding: 0px; - .sitename a { - display: block; - background: url(/img/masthead.png) 0 0 no-repeat; - font-size: 0px; - height: 100px; - background-size: auto 100px; - } - } - } - - // make the home page masthead slightly larger - body.home #masthead { - .sitename a { - height: 150px; - background-size: auto 150px; - } - } - - // - // FOOTER - // - - #footer .links { - background-color: $custom-color; - } - -NOTE: If you add a `tails.scss` or `head.scss` file, then you usually need to run `rake tmp:clear` and restart rails in order for the new stylesheet to get recognized. You should only need to do this once. - - -Custom Fork ----------------------------- - -Sometimes it is easier to maintain your own fork of the leap_web app. You can keep your customizations in that fork instead of in the provider `files/webapp` directory. Or, perhaps you want to add an engine to the application that modifies the app's behavior. - -To deploy your own leap_web, modify the provider file `common.json`: - - { - "sources": { - "webapp": { - "revision": "origin/develop", - "source": "https://github.com/leapcode/leap_web", - "type": "git" - } - } - } - -To target only particular environment, modify instead `common.ENV.json`, where ENV is the name of the environment. - -See https://github.com/leapcode/leap_web/blob/develop/doc/DEVELOP.md for notes on getting started hacking on leap_web. - -Maintenance mode ------------------- - -You can put the webapp into maintenance mode by simply dropping a html file to `/srv/leap/webapp/public/system/maintenance.html`. For example: - - workstation$ leap ssh webappnode - server# echo "Temporarily down for maintenance. We will be back soon." > /srv/leap/webapp/public/system/maintenance.html - -Known problems ---------------------------- - -* Client certificates are generated without a CSR. The problem is that this makes the web - application extremely vulnerable to denial of service attacks. This was not an issue until we - started to allow the possibility of anonymously fetching a client certificate without - authenticating first. -* By its very nature, the user database is vulnerable to enumeration attacks. These are - very hard to prevent, because our protocol is designed to allow query of a user database via - proxy in order to provide network perspective. diff --git a/doc/troubleshooting/en.haml b/doc/troubleshooting/en.haml deleted file mode 100644 index f0f1359c..00000000 --- a/doc/troubleshooting/en.haml +++ /dev/null @@ -1,3 +0,0 @@ -- @title = "Troubleshooting" - -= child_summaries \ No newline at end of file diff --git a/doc/troubleshooting/known-issues.md b/doc/troubleshooting/known-issues.md deleted file mode 100644 index 4defc886..00000000 --- a/doc/troubleshooting/known-issues.md +++ /dev/null @@ -1,115 +0,0 @@ -@title = 'Leap Platform Release Notes' -@nav_title = 'Known issues' -@summary = 'Known issues in the Leap Platform.' -@toc = true - -Here you can find documentation about known issues and potential work-arounds in the current Leap Platform release. - -0.6.0 -============== - -Upgrading ------------------- - -Upgrade your leap_platform to 0.6 and make sure you have the latest leap_cli. - -**Update leap_platform:** - - cd leap_platform - git pull - git checkout -b 0.6.0 0.6.0 - -**Update leap_cli:** - -If it is installed as a gem from rubygems: - - sudo gem update leap_cli - -If it is installed as a gem from source: - - cd leap_cli - git pull - git checkout master - rake build - sudo rake install - -If it is run directly from source: - - cd leap_cli - git pull - git checkout master - -To upgrade: - - leap --version # must be at least 1.6.2 - leap cert update - leap deploy - leap test - -If the tests fail, try deploying again. If a test fails because there are two tapicero daemons running, you need to ssh into the server, kill all the tapicero daemons manually, and then try deploying again (sometimes the daemon from platform 0.5 would put its PID file in an odd place). - -OpenVPN ------------------- - -On deployment to a openvpn node, if the following happens: - - - err: /Stage[main]/Site_openvpn/Service[openvpn]/ensure: change from stopped to running failed: Could not start Service[openvpn]: Execution of '/etc/init.d/openvpn start' returned 1: at /srv/leap/puppet/modules/site_openvpn/manifests/init.pp:189 - -this is likely the result of a kernel upgrade that happened during the deployment, requiring that the machine be restarted before this service can start. To confirm this, login to the node (leap ssh ) and look at the end of the /var/log/daemon.log: - - # tail /var/log/daemon.log - Nov 22 19:04:15 snail ovpn-udp_config[16173]: ERROR: Cannot open TUN/TAP dev /dev/net/tun: No such device (errno=19) - Nov 22 19:04:15 snail ovpn-udp_config[16173]: Exiting due to fatal error - -if you see this error, simply restart the node. - -CouchDB ---------------------- - -At the moment, we strongly advise only have one bigcouch server for stability purposes. - -With multiple couch nodes (not recommended at this time), in some scenarios, such as when certain components are unavailable, the couchdb syncing will be broken. When things are brought back to normal, shortly after restart, the nodes will attempt to resync all their data, and can fail to complete this process because they run out of file descriptors. A symptom of this is the webapp wont allow you to register or login, the /opt/bigcouch/var/log/bigcouch.log is huge with a lot of errors that include (over multiple lines): {error, emfile}}. We have raised the limits for available file descriptors to bigcouch to try and accommodate for this situation, but if you still experience it, you may need to increase your /etc/sv/bigcouch/run ulimit values and restart bigcouch while monitoring the open file descriptors. We hope that in the next platform release, a newer couchdb will be better at handling these resources. - -You can also see the number of file descriptors in use by doing: - - # watch -n1 -d lsof -p `pidof beam`|wc -l - -The command `leap db destroy` will not automatically recreate new databases. You must run `leap deploy` afterwards for this. - -User setup and ssh ------------------- - -At the moment, it is only possible to add an admin who will have access to all LEAP servers (see: https://leap.se/code/issues/2280) - -The command `leap add-user --self` allows only one SSH key. If you want to specify more than one key for a user, you can do it manually: - - users/userx/userx_ssh.pub - users/userx/otherkey_ssh.pub - -All keys matching 'userx/*_ssh.pub' will be used for that user. - -Deploying ---------- - -If you have any errors during a run, please try to deploy again as this often solves non-deterministic issues that were not uncovered in our testing. Please re-deploy with `leap -v2 deploy` to get more verbose logs and capture the complete output to provide to us for debugging. - -If when deploying your debian mirror fails for some reason, network anomoly or the mirror itself is out of date, then platform deployment will not succeed properly. Check the mirror is up and try to deploy again when it is resolved (see: https://leap.se/code/issues/1091) - -Deployment gives 'error: in `%`: too few arguments (ArgumentError)' - this is because you attempted to do a deploy before initializing a node, please initialize the node first and then do a deploy afterwards (see: https://leap.se/code/issues/2550) - -This release has no ability to custom configure apt sources or proxies (see: https://leap.se/code/issues/1971) - -When running a deploy at a verbosity level of 2 and above, you will notice puppet deprecation warnings, these are known and we are working on fixing them - -IPv6 ----- - -As of this release, IPv6 is not supported by the VPN configuration. If IPv6 is detected on your network as a client, it is blocked and instead it should revert to IPv4. We plan on adding IPv6 support in an upcoming release. - - -Special Environments --------------------- - -When deploying to OpenStack release "nova" or newer, you will need to do an initial deploy, then when it has finished run `leap facts update` and then deploy again (see: https://leap.se/code/issues/3020) - -It is not possible to actually use the EIP openvpn server on vagrant nodes (see: https://leap.se/code/issues/2401) diff --git a/doc/troubleshooting/tests.md b/doc/troubleshooting/tests.md deleted file mode 100644 index 607f924e..00000000 --- a/doc/troubleshooting/tests.md +++ /dev/null @@ -1,70 +0,0 @@ -@title = 'Tests and Monitoring' -@summary = 'Testing and monitoring your infrastructure.' -@toc = true - -## Troubleshooting Tests - -At any time, you can run troubleshooting tests on the nodes of your provider infrastructure to check to see if things seem to be working correctly. If there is a problem, these tests should help you narrow down precisely where the problem is. - -To run tests on FILTER node list: - - workstation$ leap test run FILTER - -For example, you can also test a single node (`leap test elephant`); test a specific environment (`leap test development`), or any tag (`leap test soledad`). - -Alternately, you can run test on all nodes (probably only useful if you have pinned the environment): - - workstation$ leap test - -The tests that are performed are located in the platform under the tests directory. - -## Testing with the bitmask client - -Download the provider ca: - - wget --no-check-certificate https://example.org/ca.crt -O /tmp/ca.crt - -Start bitmask: - - bitmask --ca-cert-file /tmp/ca.crt - -## Testing Recieving Mail - -Use i.e. swaks to send a testmail - - swaks -f noone@example.org -t testuser@example.org -s example.org - -and use your favorite mail client to examine your inbox. - -You can also use [offlineimap](http://offlineimap.org/) to fetch mails: - - offlineimap -c vagrant/.offlineimaprc.example.org - -WARNING: Use offlineimap *only* for testing/debugging, -because it will save the mails *decrypted* locally to -your disk ! - -## Monitoring - -In order to set up a monitoring node, you simply add a `monitor` service tag to the node configuration file. It could be combined with any other service, but we propose that you add it to the webapp node, as this already is public accessible via HTTPS. - -After deploying, this node will regularly poll every node to ask for the status of various health checks. These health checks include the checks run with `leap test`, plus many others. - -We use [Nagios](https://www.nagios.org/) together with [Check MK agent](https://en.wikipedia.org/wiki/Check_MK) for running checks on remote hosts. - -One nagios installation will monitor all nodes in all your environments. You can log into the monitoring web interface via [https://DOMAIN/nagios3/](https://DOMAIN/nagios3/). The username is `nagiosadmin` and the password is found in the secrets.json file in your provider directory. -Nagios will send out mails to the `contacts` address provided in `provider.json`. - - -## Nagios Frontends - -There are other ways to check and get notified by Nagios besides regularly checking the Nagios webinterface or reading email notifications. Check out the [Frontends (GUIs and CLIs)](http://exchange.nagios.org/directory/Addons/Frontends-%28GUIs-and-CLIs%29) on the Nagios project website. -A recommended status tray application is [Nagstamon](https://nagstamon.ifw-dresden.de/), which is available for Linux, MacOS X and Windows. It can not only notify you of hosts/services failures, you can also acknowledge or recheck them. - -### Log Monitoring - -At the moment, we use [check-mk-agent-logwatch](https://mathias-kettner.de/checkmk_check_logwatch.html) for searching logs for irregularities. -Logs are parsed for patterns using a blacklist, and are stored in `/var/lib/check_mk/logwatch/`. - -In order to "acknowledge" a log warning, you need to log in to the monitoring server, and delete the corresponding file in `/var/lib/check_mk/logwatch/`. This should be done via the nagios webinterface in the future. - diff --git a/doc/troubleshooting/where-to-look.md b/doc/troubleshooting/where-to-look.md deleted file mode 100644 index c92fba8f..00000000 --- a/doc/troubleshooting/where-to-look.md +++ /dev/null @@ -1,267 +0,0 @@ -@title = 'Where to look for errors' -@nav_title = 'Where to look' -@toc = true - - -General -======= - -* Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. `-v 5` after the `leap` cmd, i.e. `leap -v 2 deploy`. -* We use the `example.org` domain for documentation purposes here, please replace it with the you domain. - -Firewall -======================= - -Every node in your provider has its own restrictive firewall, but you might have a network firewall in place as well that is not managed by LEAP platform. To see what ports and addresses must be open, run this command: - - workstation$ leap compile firewall - -If any of those are blocked, then your provider will not work. - -Webapp -====== - -Places to look for errors -------------------------- - -* `/var/log/apache2/error.log` -* `/srv/leap/webapp/log/production.log` -* `/var/log/syslog` (watch out for stunnel issues) -* `/var/log/leap/*` - - -Is haproxy ok ? ---------------- - - curl -s -X GET "http://127.0.0.1:4096" - -Is couchdb accessible through stunnel ? ---------------------------------------- - -* Depending on how many couch nodes you have, increase the port for every test - (see /etc/haproxy/haproxy.cfg for the server/port mapping): - - - curl -s -X GET "http://127.0.0.1:4000" - curl -s -X GET "http://127.0.0.1:4001" - ... - - -Check couchdb acl as admin --------------------------- - - mkdir /etc/couchdb - cat /srv/leap/webapp/config/couchdb.yml.admin # see username and password - echo "machine 127.0.0.1 login admin password " > /etc/couchdb/couchdb-admin.netrc - chmod 600 /etc/couchdb/couchdb-admin.netrc - - curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096" - curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096/_all_dbs" - -Check couchdb acl as unpriviledged user ---------------------------------------- - - cat /srv/leap/webapp/config/couchdb.yml # see username and password - echo "machine 127.0.0.1 login webapp password " > /etc/couchdb/couchdb-webapp.netrc - chmod 600 /etc/couchdb/couchdb-webapp.netrc - - curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096" - curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096/_all_dbs" - - -All URLs accessible ? ---------------------- - -* https://example.org -* https://api.example.org:4430/provider.json -* https://example.org/ca.crt - - -Check client config files -------------------------- - -* https://example.net/provider.json -* https://example.net/1/config/smtp-service.json -* https://example.net/1/config/soledad-service.json -* https://example.net/1/config/eip-service.json - - -Soledad -======= - - /var/log/soledad.log - - -Couchdb -======= - -Places to look for errors -------------------------- - -* `/opt/bigcouch/var/log/bigcouch.log` -* `/var/log/syslog` (watch out for stunnel issues) - - - -Bigcouch membership -------------------- - -* All nodes configured for the provider should appear here: - -
-    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5986/nodes/_all_docs'
-
- -* All configured nodes should show up under "cluster_nodes", and the ones online and communicating with each other should appear under "all_nodes". This example output shows the configured cluster nodes `couch1.bitmask.net` and `couch2.bitmask.net`, but `couch2.bitmask.net` is currently not accessible from `couch1.bitmask.net` - - -
-    curl -s --netrc-file /etc/couchdb/couchdb.netrc 'http://127.0.0.1:5984/_membership'
-    {"all_nodes":["bigcouch@couch1.bitmask.net"],"cluster_nodes":["bigcouch@couch1.bitmask.net","bigcouch@couch2.bitmask.net"]}
-
- -* Sometimes a `/etc/init.d/bigcouch restart` on all nodes is needed, to register new nodes - -Databases ---------- - -* Following output shows all neccessary DBs that should be present. Note that the `user-0123456....` DBs are the data stores for a particular user. - -
-    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_all_dbs'
-    ["customers","identities","sessions","shared","tickets","tokens","user-0","user-9d34680b01074c75c2ec58c7321f540c","user-9d34680b01074c75c2ec58c7325fb7ff","users"]
-
- - - - -Design Documents ----------------- - -* Is User `_design doc` available ? - - -
-    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X  GET "http://127.0.0.1:5984/users/_design/User"
-
- -Is couchdb cluster backend accessible through stunnel ? -------------------------------------------------------- - -* Find out how many connections are set up for the couchdb cluster backend: - -
-    grep "accept = 127.0.0.1" /etc/stunnel/*
-
- - -* Now connect to all of those local endpoints to see if they up. All these tests should return "localhost [127.0.0.1] 4000 (?) open" - -
-    nc -v 127.0.0.1 4000
-    nc -v 127.0.0.1 4001
-    ...
-
- - -MX -== - -Places to look for errors -------------------------- - -* `/var/log/mail.log` -* `/var/log/leap_mx.log` -* `/var/log/syslog` (watch out for stunnel issues) - -Is couchdb accessible through stunnel ? ---------------------------------------- - -* Depending on how many couch nodes you have, increase the port for every test - (see /etc/haproxy/haproxy.cfg for the server/port mapping): - - - curl -s -X GET "http://127.0.0.1:4000" - curl -s -X GET "http://127.0.0.1:4001" - ... - -Query leap-mx -------------- - -* for useraccount - - -
-    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:2244
-    ...
-    postmap: dict_tcp_lookup: send: get jow@dev.bitmask.net
-    postmap: dict_tcp_lookup: recv: 200
-    ...
-
- -* for mailalias - - -
-    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:4242
-    ...
-    postmap: dict_tcp_lookup: send: get joe@dev.bitmask.net
-    postmap: dict_tcp_lookup: recv: 200 f01bc1c70de7d7d80bc1ad77d987e73a
-    postmap: dict_tcp_lookup: found: f01bc1c70de7d7d80bc1ad77d987e73a
-    f01bc1c70de7d7d80bc1ad77d987e73a
-    ...
-
- - -Check couchdb acl as unpriviledged user ---------------------------------------- - - - - cat /etc/leap/mx.conf # see username and password - echo "machine 127.0.0.1 login leap_mx password " > /etc/couchdb/couchdb-leap_mx.netrc - chmod 600 /etc/couchdb/couchdb-leap_mx.netrc - - curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/_all_dbs" # pick one "user-" db - curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/user-de9c77a3d7efbc779c6c20da88e8fb9c" - - -* you may check multiple times, cause 127.0.0.1:4096 is haproxy load-balancing the different couchdb nodes - - -Mailspool ---------- - -* Any file in the leap_mx mailspool longer for a few seconds ? - - - -
-    ls -la /var/mail/vmail/Maildir/cur/
-
- -* Any mails in postfix mailspool longer than a few seconds ? - -
-    mailq
-
- - - -Testing mail delivery ---------------------- - - swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 25 - swaks -f varac@cdev.bitmask.net -t varac@cdev.bitmask.net -s chipmonk.cdev.bitmask.net --port 465 --tlsc - swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 587 --tls - - -VPN -=== - -Places to look for errors -------------------------- - -* `/var/log/syslog` (watch out for openvpn issues) - - diff --git a/doc/tutorials/en.haml b/doc/tutorials/en.haml deleted file mode 100644 index 1c73fc0f..00000000 --- a/doc/tutorials/en.haml +++ /dev/null @@ -1,4 +0,0 @@ -- @nav_title = "Tutorials" -- @title = "Platform Tutorials" - -= child_summaries \ No newline at end of file diff --git a/doc/tutorials/quick-start.md b/doc/tutorials/quick-start.md deleted file mode 100644 index f963867a..00000000 --- a/doc/tutorials/quick-start.md +++ /dev/null @@ -1,230 +0,0 @@ -@title = 'Quick Start Tutorial' -@nav_title = 'Quick Start Tutorial' -@summary = 'This tutorial walks you through the initial process of creating and deploying a minimal service provider running the LEAP Platform.' - -Introduction -==================================== - -### Our goal - -We are going to create a minimal LEAP provider, but one that does not offer any actual services. Check out the other tutorials for adding VPN or email services. - -Our goal is something like this: - - $ leap list - NODES SERVICES TAGS - wildebeest couchdb, webapp - -NOTE: You won't be able to run that `leap list` command yet, not until we actually create the node configurations. - -### Requirements - -1. A workstation: This is your local machine that you will run commands on. -1. A server: This is the machine that you will deploy to. The server can be either: - 1. A local Vagrant virtual machine: a Vagrant machine can only be useful for testing. - 1. A real or paravirtualized server: The server must have Debian Jessie installed, and you must be able to SSH into the machine as root. Paravirtualization includes KVM, Xen, OpenStack, Amazon, but not VirtualBox or OpenVZ. - -Other things to keep in mind: - -* The ability to create/modify DNS entries for your domain is preferable, but not needed. If you don't have access to DNS, you can workaround this by modifying your local resolver, i.e. editing `/etc/hosts`. -* You need to be aware that this process will make changes to your servers, so please be sure that these machines are a basic install with nothing configured or running for other purposes. -* Your servers will need to be connected to the internet, and not behind a restrictive firewall. - -Prepare your workstation -======================== - -In order to be able to manage your servers, you need to install the `leap` command on your workstation: - -### Install pre-requisites - -Install core prerequisites on your workstation. - -*Debian & Ubuntu* - - workstation$ sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make bzip2 - -*Mac OS* - - workstation$ brew install ruby-install - workstation$ ruby-install ruby - -### Install the LEAP command-line utility - -Install the `leap` command from rubygems.org: - - workstation$ gem install leap_cli --install-dir ~/leap - workstation$ export PATH=$PATH:~/leap/bin - -Alternately, you can install `leap` system wide: - - workstation$ sudo gem install leap_cli - -To confirm that you installed `leap` correctly, try running `leap help`. - -Create a provider instance -============================================= - -A provider instance is a directory tree, residing on your workstation, that contains everything you need to manage an infrastructure for a service provider. - -In this case, we create one for example.org and call the instance directory 'example'. - - workstation$ leap new ~/example - -The `leap new` command will ask you for several required values: - -* domain: The primary domain name of your service provider. In this tutorial, we will be using "example.org". -* name: The name of your service provider (we use "Example"). -* contact emails: A comma separated list of email addresses that should be used for important service provider contacts (for things like postmaster aliases, Tor contact emails, etc). -* platform: The directory where you have a copy of the `leap_platform` git repository checked out. If the platform directory does not yet exist, the `leap_platform` will be downloaded and placed in that directory. - -You could also have passed these configuration options on the command-line, like so: - - workstation$ leap new --contacts your@email.here --domain example.org --name Example --platform=~/leap/leap_platform . - -You should now have the following files: - - workstation$ tree example - example - ├── common.json - ├── Leapfile - ├── nodes/ - ├── provider.json - ├── services/ - └── tags/ - -Now add yourself as a privileged sysadmin who will have access to deploy to servers: - - workstation$ cd example - workstation$ leap add-user louise --self - -Replace "louise" with whatever you want your sysadmin username to be. - -NOTE: Make sure you change directories so that the `leap` command is run from within the provider instance directory. Most `leap` commands only work when run from a provider instance. - -Now create the necessary keys and certificates: - - workstation$ leap cert ca - workstation$ leap cert csr - -What do these commands do? The first command will create two Certificate Authorities, one that clients will use to authenticate with the servers and one for backend servers to authenticate with each other. The second command creates a Certificate Signing Request suitable for submission to a commercial CA. It also creates two "dummy" files for you to use temporarily: - -* `files/cert/example.org.crt` -- This is a "dummy" certificate for your domain that can be used temporarily for testing. Once you get a real certificate from a CA, you should replace this file. -* `files/cert/commercial_ca.crt` -- This is "dummy" CA cert the corresponds to the dummy domain certificate. Once you replace the domain certificate, also replace this file with the CA cert from the real Certificate Authority. - -If you plan to run a real service provider, see important information on [[managing keys and certificates => keys-and-certificates]]. - -Add a node to the provider -================================================== - -A "node" is a server that is part of your infrastructure. Every node can have one or more services associated with it. We will now add a single node with two services, "webapp" and "couchdb". - -You have two choices for node type: a real node or a local node. - -* Real Node: A real node is any physical or paravirtualized server, including KVM, Xen, OpenStack Compute, Amazon EC2, but not VirtualBox or OpenVZ (VirtualBox and OpenVZ use a more limited form of virtualization). The server must be running Debian Jessie. -* Local Node: A local node is a virtual machine created by Vagrant, useful for local testing on your workstation. - -Getting Vagrant working can be a pain and is [[covered in other tutorials => vagrant]]. If you have a real server available, we suggest you try this tutorial with a real node first. - -### Option A: Add a real node - -Note: Installing LEAP Platform on this server will potentially destroy anything you have previously installed on this machine. - -Create a node, with the services "webapp" and "couchdb": - - workstation$ leap node add wildebeest ip_address:x.x.x.w services:webapp,couchdb - -NOTE: replace x.x.x.x with the actual IP address of this server. - -### Option B: Add a local node - -Create a node, with the services "webapp" and "couchdb", and then start the local virtual machine: - - workstation$ leap node add --local wildebeest services:webapp,couchdb - workstation$ leap local start wildebeest - -It will take a while to download the Virtualbox base box and create the virtual machine. - -Deploy your provider -========================================= - -### Initialize the node - -Node initialization only needs to be done once, but there is no harm in doing it multiple times: - - workstation$ leap node init wildebeest - -This will initialize the node `wildebeest`. - -For non-local nodes, when `leap node init` is run, you will be prompted to verify the fingerprint of the SSH host key and to provide the root password of the server(s). You should only need to do this once. - -### Deploy to the node - -The next step is to deploy the LEAP platform to your node. [Deployment can take a while to run](https://xkcd.com/303/), especially on the first run, as it needs to update the packages on the new machine. - - workstation$ leap deploy wildebeest - -Watch the output for any errors (in red), if everything worked fine, you should now have your first running node. If you do have errors, try doing the deploy again. - -### Setup DNS - -The next step is to configure the DNS for your provider. For testing purposes, you can just modify your `/etc/hosts` file. Please don't forget about these entries, they will override DNS queries if you setup your DNS later. For a list of what entries to add to `/etc/hosts`, run this command: - - workstation$ leap compile hosts - -Alternately, if you have access to modify the DNS zone entries for your domain: - - workstation$ leap compile zone - -NOTE: The resulting zone file is incomplete because it is missing a serial number. Use the output of `leap compile zone` as a guide, but do not just copy and paste the output. Also, the `compile zone` output will always exclude mention of local nodes. - -The DNS method will not work for local nodes created with Vagrant. - -Test that things worked correctly -================================= - -To run troubleshooting tests: - - workstation$ leap test - -Alternately, you can run these same tests from the server itself: - - workstation$ leap ssh wildebeest - wildebeest# run_tests - -Create an administrator -=============================== - -Assuming that you set up your DNS or `/etc/hosts` file, you should be able to load `https://example.org` in your web browser (where example.org is whatever domain name you actually used). - -Your browser will complain about an untrusted cert, but for now just bypass this. From there, you should be able to register a new user and login. - -Once you have created a user, you can now make this user an administrator. For example, if you created a user `kangaroo`, you would create the file `services/webapp.json` with the following content: - - { - "webapp": { - "admins": ["kangaroo"] - } - } - -Save that file and run `leap deploy` again. When you next log on to the web application, the user kangaroo will now be an admin. - -If you want to restrict who can register a new user, see [[webapp]] for configuration options. - -What is next? -====================== - -Add an end-user service -------------------------------- - -You should now have a minimal service provider with a single node. This service provider is pointless at the moment, because it does not include any end-user services like VPN or email. To add one of these services, continue with one of the following tutorials: - -* [[single-node-email]] -* [[single-node-vpn]] - -Learn more ---------------- - -We have only just scratched the surface of the possible ways to configure and deploy your service provider. Your next step should be: - -* Read [[getting-started]] for more details on using the LEAP platform. -* See [[commands]] for a list of possible commands. diff --git a/doc/tutorials/single-node-email.md b/doc/tutorials/single-node-email.md deleted file mode 100644 index 0a73e6e1..00000000 --- a/doc/tutorials/single-node-email.md +++ /dev/null @@ -1,69 +0,0 @@ -@title = 'Single node email tutorial' -@nav_title = 'Quick email' -@summary = 'Tutorial for setting up a simple email provider.' - -This tutorial walks you through the initial process of creating and deploying a minimal email service provider. Please first complete the [[quick-start]]. This tutorial will pick up where that one left off. - -Our goal ------------------- - -We are going to create a minimal LEAP provider offering email service. - -Our goal is something like this: - - $ leap list - NODES SERVICES TAGS - wildebeest couchdb, mx, soledad, webapp - -Where 'wildebeest' is whatever name you chose for your node in the [[quick-start]]. - -Add email services to the node --------------------------------------- - -In order to add [[services => services]] to a node, edit the node's JSON configuration file. - -In our example, we would edit `nodes/wildebeest.json`: - - { - "ip_address": "1.1.1.1", - "services": ["couchdb", "webapp", "mx", "soledad"] - } - -Here, we added `mx` and `soledad` to the node's `services` list. Briefly: - -* **mx**: nodes with the **mx** service will run postfix mail transfer agent, and are able to receive and relay email on behalf of your domain. You can have as many as you want, spread out over as many nodes as you want. -* **soledad**: nodes with **soledad** service run the server-side daemon that allows the client to synchronize a user's personal data store among their devices. Currently, **soledad** only runs on nodes that are also **couchdb** nodes. - -For more details, see the [[services]] overview, or the individual pages for the [[mx]] and [[soledad]] services. - -Deploy to the node --------------------- - -Now you should deploy to your node. - - workstation$ leap deploy - -Setup DNS ----------------------------- - -There are several important DNS entries that all email providers should have: - -* SPF (Sender Policy Framework): With SPF, an email provider advertises in their DNS which servers should be allowed to relay email on behalf of your domain. -* DKIM (DomainKey Identified Mail): With DKIM, an email provider is able to vouch for the validity of certain headers in outgoing mail, allowing the receiving provider to have more confidence in these values when processing the message for spam or abuse. - -In order to take advantage of SPF and DKIM, run this command: - - workstation$ leap compile zone - -Then take the output of that command and merge it with the DNS zone file for your domain. - -CAUTION: the output of `leap compile zone` is not a complete zone file since it is missing a serial number. You will need to manually merge it with your existing zone file. - -Test it out ---------------------------------- - -First, run: - - workstation# leap test - -Then fire up the bitmask client, register a new user with your provider, and try sending and receiving email. diff --git a/doc/tutorials/single-node-vpn.md b/doc/tutorials/single-node-vpn.md deleted file mode 100644 index dc1df7ab..00000000 --- a/doc/tutorials/single-node-vpn.md +++ /dev/null @@ -1,112 +0,0 @@ -@title = "Single node VPN tutorial" -@nav_title = "Quick VPN" -@summary = 'Tutorial for setting up a simple VPN provider.' - -This tutorial walks you through the initial process of creating and deploying a minimal VPN service provider. Please first complete the [[quick-start]]. This tutorial will pick up where that one left off. - -NOTE: For the VPN to work, you must use a real or paravirtualized node, not a local Vagrant node. - -Our goal ------------------- - -We are going to create a minimal LEAP provider offering VPN service. - -Our goal is something like this: - - $ leap list - NODES SERVICES TAGS - wildebeest couchdb, webapp, openvpn, tor - -Where 'wildebeest' is whatever name you chose for your node in the [[quick-start]]. - -Add VPN service to the node --------------------------------------- - -In order to add [[services => services]] to a node, edit the node's JSON configuration file. - -In our example, we would edit `nodes/wildebeest.json`: - - { - "ip_address": "1.1.1.1", - "services": ["couchdb", "webapp", "openvpn", "tor"] - } - -Here, we added `openvpn` and `tor` to the node's `services` list. Briefly: - -* **openvpn**: nodes with the **openvpn** service will become OpenVPN gateways that clients connect to in order to proxy their internet connection. You can have as many as you want, spread out over as many nodes as you want. -* **tor**: nodes with **tor** service become Tor exit nodes. This is entirely optional, and will add additional bandwidth to your node. If you don't have many VPN users, the added traffic will help create cover traffic for your users. On the down side, this VPN gateway will get flagged as an anonymous proxy and some sites may block traffic from it. - -For more details, see the [[services]] overview, or the individual pages for the [[openvpn]] and [[tor]] services. - -Add gateway_address to the node ----------------------------------------- - -VPN gateways require two different IP addresses: - -* `ip_address`: This property is used for VPN traffic **egress**. In other words, all VPN traffic appears to come from this IP address. This is also the main IP of the server. -* `openvpn.gateway_address`: This property is used for VPN traffic **ingress**. In other words, clients will connect to this IP address. - -The node configuration file should now look like this: - - { - "ip_address": "1.1.1.1", - "services": ["couchdb", "webapp", "openvpn", "tor"], - "openvpn": { - "gateway_address": "2.2.2.2" - } - } - -Why two different addresses? Without this, the traffic from one VPN user to another would not be encrypted. This is because the routing table of VPN clients must ensure that packets with a destination of the VPN gateway are sent unmodified and don't get passed through the VPN's encryption. - -Generate a Diffie-Hellman file -------------------------------------------- - -Next we need to create a Diffie-Hellman parameter file, used for forward secret OpenVPN ciphers. You only need to do this once. - - workstation$ leap cert dh - -Feel free to erase the resulting DH file and regenerate it as you please. - -Deploy to the node --------------------- - -Now you should deploy to your node. This may take a while. - - workstation$ leap deploy - -If the deploy was not successful, try to run it again. - -Test it out ---------------------------------- - -First, run: - - workstation$ leap test - -Then fire up the Bitmask client, register a new user with your provider, and turn on the VPN connection. - -Alternately, you can also manually connect to your VPN gateway using OpenVPN on the command line: - - workstation$ sudo apt install openvpn - workstation$ leap test init - workstation$ sudo openvpn --config test/openvpn/default_unlimited.ovpn - -Make sure that Bitmask is not connected to the VPN when you run that command. - -The name of the test configuration might differ depending on your setup. The test configuration created by `leap test init` includes a client certificate that will expire, so you may need to re-run `leap test init` if it has been a while since you last generated the test configuration. - -What do do next --------------------------------- - -A VPN provider with a single gateway is kind of limited. You can add as many nodes with service [[openvpn]] as you like. There is no communication among the VPN gateways or with the [[webapp]] or [[couchdb]] nodes, so there is no issue with scaling out the number of gateways. - -For example, add some more nodes: - - workstation$ leap node add giraffe ip_address:1.1.1.2 services:openvpn openvpn.gateway_address:2.2.2.3 - workstation$ leap node add rhino ip_address:1.1.1.3 services:openvpn openvpn.gateway_address:2.2.2.4 - workstation$ leap node init giraffe rhino - workstation$ leap deploy - -Now you have three VPN gateways. - -One consideration is that you should tag each VPN gateway with a [[location => nodes#locations]]. This helps the client determine which VPN gateway it should connect to by default and will allow the user to choose among gateways based on location. diff --git a/doc/tutorials/vagrant.md b/doc/tutorials/vagrant.md deleted file mode 100644 index 710c2664..00000000 --- a/doc/tutorials/vagrant.md +++ /dev/null @@ -1,471 +0,0 @@ -@title = 'Vagrant and the LEAP Platform' -@nav_title = 'Vagrant' -@summary = 'Running a local provider with Vagrant' - -What is Vagrant? -======================================== - -[[Vagrant => https://www.vagrantup.com]] is a tool to make it easier to manage virtual machines running on your desktop computer (typically for testing or development purposes). You can use Vagrant to create virtual machines and deploy the LEAP platform locally. - -Vagrant can be a pain to get working initially, but this page should help you get through the process. Please make sure you have at least Vagrant v1.5 installed. - -There are two ways you can setup LEAP platform using Vagrant. - -1. use the `leap` command: this will allow you to create multiple virtual machines. -2. use static Vagrantfile: there is a static Vagrantfile that is distributed with the `leap_platform.git`. This only supports a single, pre-configured virtual machine, but can get you started more quickly. - -Install Vagrant -======================================== - -Requirements: - -* A real machine with virtualization support in the CPU (VT-x or AMD-V). In other words, not a virtual machine. -* Have at least 4gb of RAM. -* Have a fast internet connection (because you will be downloading a lot of big files, like virtual machine images). -* You should do everything described below as an unprivileged user, and only run those commands as root that are noted with *sudo* in front of them. Other than those commands, there is no need for privileged access to your machine, and in fact things may not work correctly. - -*Debian & Ubuntu* - -Install core prerequisites: - - sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make - -Install Vagrant: - - sudo apt-get install vagrant virtualbox - -If you want to use libvirt instead of virtualbox, you don't need to install virtualbox. See [support for libvirt](#support-for-libvirt). - -*Mac OS X 10.9 (Mavericks)* - -Install Homebrew package manager from http://brew.sh/ and enable the [System Duplicates Repository](https://github.com/Homebrew/homebrew/wiki/Interesting-Taps-&-Branches) (needed to update old software versions delivered by Apple) with - - brew tap homebrew/dupes - -Update OpenSSH to support ECDSA keys. Follow [this guide](http://www.dctrwatson.com/2013/07/how-to-update-openssh-on-mac-os-x/) to let your system use the Homebrew binary. - - brew install openssh --with-brewed-openssl --with-keychain-support - -The certtool provided by Apple it's really old, install the one provided by GnuTLS and shadow the system's default. - - sudo brew install gnutls - ln -sf /usr/local/bin/gnutls-certtool /usr/local/bin/certool - -Install the Vagrant and VirtualBox packages for OS X from their respective Download pages. - -* http://www.vagrantup.com/downloads.html -* https://www.virtualbox.org/wiki/Downloads - -Vagrant with leap command -======================================= - -If you have not done so, install `leap` command line tool: - - gem install leap_cli - -Creating local nodes ----------------------------------- - -When you create a service provider, your servers are called "nodes". When a node is virtual and exists only locally using vagrant, this type of node is called a "local node". - -If you do not have a provider already, you will need to create one and configure it before continuing (see the [Quick Start](quick-start) guide). - -These commands, for example, will create an initial provider directory "myprovider": - - $ leap new --domain example.org --name Example myprovider - $ cd myprovider - $ leap add-user --self - $ leap cert ca - $ leap cert csr - -To create local nodes, add the flag `--local` to the `leap node add` command. For example: - - $ leap node add --local web1 services:webapp - = created nodes/web1.json - = created files/nodes/web1/ - = created files/nodes/web1/web1.key - = created files/nodes/web1/web1.crt - -This command creates a node configuration file in `nodes/web1.json` with the webapp service. - -Starting local nodes --------------------------------- - -In order to test the node "web1" we need to start it. Starting a node for the first time will spin up a virtual machine. The first time you do this will take some time because it will need to download a VM image (about 700mb). After you've downloaded the base image, you will not need to download it again, and instead you will re-use the downloaded image (until you need to update the image). - -NOTE: Many people have difficulties getting Vagrant working. If the following commands do not work, please see the troubleshooting section below. - - $ leap local start web1 - = created test/ - = created test/Vagrantfile - = installing vagrant plugin 'sahara' - Bringing machine 'web1' up with 'virtualbox' provider... - [web1] Box 'leap-jessie' was not found. Fetching box from specified URL for - the provider 'virtualbox'. Note that if the URL does not have - a box for this provider, you should interrupt Vagrant now and add - the box yourself. Otherwise Vagrant will attempt to download the - full box prior to discovering this error. - Downloading or copying the box... - Progress: 3% (Rate: 560k/s, Estimated time remaining: 0:13:36) - ... - Bringing machine 'web1' up with 'virtualbox' provider... - [web1] Importing base box 'leap-jessie'... - 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% - -Now the virtual machine 'web1' is running. You can add another local node using the same process. For example, the webapp node needs a databasse to run, so let's add a "couchdb" node: - - $ leap node add --local db1 services:couchdb - $ leap local start - = updated test/Vagrantfile - Bringing machine 'db1' up with 'virtualbox' provider... - [db1] Importing base box 'leap-jessie'... - [db1] Matching MAC address for NAT networking... - [db1] Setting the name of the VM... - [db1] Clearing any previously set forwarded ports... - [db1] Fixed port collision for 22 => 2222. Now on port 2202. - [db1] Creating shared folders metadata... - [db1] Clearing any previously set network interfaces... - [db1] Preparing network interfaces based on configuration... - [db1] Forwarding ports... - [db1] -- 22 => 2202 (adapter 1) - [db1] Running any VM customizations... - [db1] Booting VM... - [db1] Waiting for VM to boot. This can take a few minutes. - [db1] VM booted and ready for use! - [db1] Configuring and enabling network interfaces... - [db1] Mounting shared folders... - [db1] -- /vagrant - -You now can follow the normal LEAP process and initialize it and then deploy your recipes to it: - - $ leap node init web1 - $ leap deploy web1 - $ leap node init db1 - $ leap deploy db1 - -Useful local commands ------------------------------------- - -There are many useful things you can do with a virtualized development environment. - -### Listing what machines are running - -Now you have the two virtual machines "web1" and "db1" running, you can see the running machines as follows: - - $ leap local status - Current machine states: - - db1 running (virtualbox) - web1 running (virtualbox) - - This environment represents multiple VMs. The VMs are all listed - above with their current state. For more information about a specific - VM, run `vagrant status NAME`. - -### Stopping machines - -It is not recommended that you leave your virtual machines running when you are not using them. They consume memory and other resources! To stop your machines, simply do the following: - - $ leap local stop web1 db1 - -### Connecting to machines - -You can connect to your local nodes just like you do with normal LEAP nodes, by running 'leap ssh node'. - -However, if you cannot connect to your local node, because the networking is not setup properly, or you have deployed a firewall that locks you out, you may need to access the graphical console. - -In order to do that, you will need to configure Vagrant to launch a graphical console and then you can login as root there to diagnose the networking problem. To do this, add the following to your $HOME/.leaprc: - - @custom_vagrant_vm_line = 'config.vm.provider "virtualbox" do |v| - v.gui = true - end' - -and then start, or restart, your local Vagrant node. You should get a VirtualBox graphical interface presented to you showing you the bootup and eventually the login. - -### Snapshotting machines - -A very useful feature of local Vagrant development nodes is the ability to snapshot the current state and then revert to that when you need. - -For example, perhaps the base image is a little bit out of date and you want to get the packages updated to the latest before continuing. You can do that simply by starting the node, connecting to it and updating the packages and then snapshotting the node: - - $ leap local start web1 - $ leap ssh web1 - web1# apt-get -u dist-upgrade - web1# exit - $ leap local save web1 - -Now you can deploy to web1 and if you decide you want to revert to the state before deployment, you simply have to reset the node to your previous save: - - $ leap local reset web1 - -### More information - -See `leap help local` for a complete list of local-only commands and how they can be used. - - -2. Vagrant with static Vagrantfile -================================================== - -You can use the static Vagrantfile if you want to get up a running with a pre-canned test provider. - -It will install a single node mail server in the default configuration with one single command. - -Clone the platform with - - git clone --recursive -b develop https://github.com/leapcode/leap_platform.git - -Start the vagrant box with - - cd leap_platform - vagrant up - -Follow the instructions how to configure your `/etc/hosts` -in order to use the provider! - -You can login via ssh with the systemuser `vagrant` and the same password. - - vagrant ssh - -On the host, run the tests to check if everything is working as expected: - - cd /home/vagrant/leap/configuration/ - leap test - -Use the bitmask client to do an initial soledad sync -------------------------------------------------------------- - -Copy the self-signed CA certificate from the host. -The easiest way is to use the [vagrant-scp plugin](https://github.com/invernizzi/vagrant-scp): - - vagrant scp :/home/vagrant/leap/configuration/files/ca/ca.crt /tmp/example.org.ca.crt - - vagrant@node1:~/leap/configuration$ cat files/ca/ca.crt - -and write it into a file, needed by the bitmask client: - - bitmask --ca-cert-file /tmp/example.org.ca.crt - -On the first run, bitmask is creating a gpg keypair. This is -needed for delivering and encrypting incoming mails. - -Testing email -------------- - - sudo apt install swaks - swaks -f test22@leap.se -t test22@example.org -s example.org - -check the logs: - - sudo less /var/log/mail.log - sudo less /var/log/leap/mx.log - -if an error occurs, see if the mail is still laying in the mailspool dir: - - sudo ls /var/mail/leap-mx/Maildir/new - -Re-run bitmask client to sync your mail ---------------------------------------- - - bitmask --ca-cert-file /tmp/example.org.ca.crt - -Now, connect your favorite mail client to the imap and smtp proxy -started by the bitmask client: - - https://bitmask.net/en/help/email - -Happy testing ! - -Using the Webapp ----------------- - -There are 2 users preconfigured: - -. `testuser` with pw `hallo123` -. `testadmin` with pw `hallo123` - -login as `testadmin` to access the webapp with admin priviledges. - - -Support for libvirt -======================================= - -Install libvirt plugin -------------------------------------- - -By default, Vagrant will use VirtualBox to create the virtual machines, but this is how you can use libvirt. Using libvirt is more efficient, but VirtualBox is more stable and easier to set up. - -*For debian/ubuntu:* - - sudo apt-get install libvirt-bin libvirt-dev - - # to build the vagrant-libvirt plugin you need the following packages: - sudo apt-get install ruby-dev libxslt-dev libxml2-dev libvirt-dev - - # install the required plugins - vagrant plugin install vagrant-libvirt fog fog-libvirt sahara - -Log out and then log back in. - -Note: if running ubuntu 15.10 as the host OS, you will probably need to run the following commands before "vagrant plugin install vagrant-libvirt" will work: - - ln -sf /usr/lib/liblzma.so.5 /opt/vagrant/embedded/lib - ln -sf /usr/lib/liblzma.so.5.0.0 /opt/vagrant/embedded/lib - -Create libvirt pool ------------------------------------------ - -Next, you must create the libvirt image pool. The "default" pool uses `/var/lib/libvirt/images`, but Vagrant will not download base boxes there. Instead, create a libvirt pool called "vagrant", like so: - - virsh pool-define-as vagrant dir - - - - /home/$USER/.vagrant.d/boxes - virsh pool-start vagrant - virsh pool-autostart vagrant - -If you want to use a name different than "vagrant" for the pool, you can change the name in `Leapfile` by setting the `@vagrant_libvirt_pool` variable: - - @vagrant_libvirt_pool = "vagrant" - -Force use of libvirt --------------------------------------------- - -Finally, you need to tell Vagrant to use libvirt instead of VirtualBox. If using vagrant with leap_cli, modify your `Leapfile` or `.leaprc` file and add this line: - - @vagrant_provider = "libvirt" - -Alternately, if using the static Vagrantfile, you must run this in your shell instead: - - export VAGRANT_DEFAULT_PROVIDER=libvirt - - -Debugging ------------------------- - -If you get an error in any of the above commands, try to get some debugging information, it will often tell you what is wrong. In order to get debugging logs, you simply need to re-run the command that produced the error but prepend the command with VAGRANT_LOG=info, for example: - - VAGRANT_LOG=info vagrant box add LEAP/jessie - -You can also run vagrant with --debug for full logging. - -Known issues ------------------------- - -* You may need to undefine the default libvirt pool: - sudo virsh pool-undefine default -* `Call to virConnectOpen failed: internal error: Unable to locate libvirtd daemon in /usr/sbin (to override, set $LIBVIRTD_PATH to the name of the libvirtd binary)` - you don't have the libvirtd daemon running or installed, be sure you installed the 'libvirt-bin' package and it is running -* `Call to virConnectOpen failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied` - you need to be in the libvirt group to access the socket, do 'sudo adduser libvirtd' and then re-login to your session. -* if each call to vagrant ends up with a segfault, it may be because you still have virtualbox around. if so, remove virtualbox to keep only libvirt + KVM. according to https://github.com/pradels/vagrant-libvirt/issues/75 having two virtualization engines installed simultaneously can lead to such weird issues. -* see the [vagrant-libvirt issue list on github](https://github.com/pradels/vagrant-libvirt/issues) -* be sure to use vagrant-libvirt >= 0.0.11 and sahara >= 0.0.16 (which are the latest stable gems you would get with `vagrant plugin install [vagrant-libvirt|sahara]`) for proper libvirt support, - -Useful commands ------------------------- - -Force re-download of image, in case something goes wrong: - - vagrant box add leap/jessie --force --provider libvirt - -Shared folder support ----------------------------- - -For shared folder support, you need nfs-kernel-server installed on the host machine and set up sudo to allow unpriviledged users to modify /etc/exports. See [vagrant-libvirt#synced-folders](https://github.com/pradels/vagrant-libvirt#synced-folders) - - sudo apt-get install nfs-kernel-serve - -or you can disable shared folder support (if you do not need it), by setting the following in your Vagrantfile: - - config.vm.synced_folder "src/", "/srv/website", disabled: trueconfig.vm.synced_folder "src/", "/srv/website", disabled: true - -if you are wanting this disabled for all the leap vagrant integration, you can add this to ~/.leaprc: - - @custom_vagrant_vm_line = 'config.vm.synced_folder "src/", "/srv/website", disabled: true' - - -Verify vagrantboxes -=============================================== - -When you run vagrant, it goes out to the internet and downloads an initial image for the virtual machine. If you want to verify that authenticity of these images, follow these steps. - -Import LEAP archive signing key: - - gpg --search-keys 0x1E34A1828E207901 - -now, either you already have a trustpath to it through one of the people -who signed it, or you can verify this by checking this fingerprint: - - gpg --fingerprint --list-keys 1E34A1828E207901 - - pub 4096R/1E34A1828E207901 2013-02-06 [expires: 2015-02-07] - Key fingerprint = 1E45 3B2C E87B EE2F 7DFE 9966 1E34 A182 8E20 7901 - uid LEAP archive signing key - -if the fingerprint matches, you could locally sign it so you remember the you already -verified it: - - gpg --lsign-key 1E34A1828E207901 - -Then download the SHA215SUMS file and it's signature file - - wget https://downloads.leap.se/platform/SHA256SUMS.sign - wget https://downloads.leap.se/platform/SHA256SUMS - -and verify the signature against your local imported LEAP archive signing pubkey - - gpg --verify SHA256SUMS.sign - - gpg: Signature made Sat 01 Nov 2014 12:25:05 AM CET - gpg: using RSA key 1E34A1828E207901 - gpg: Good signature from "LEAP archive signing key " - -Make sure that the last line says "Good signature from...", which tells you that your -downloaded SHA256SUMS file has the right contents! - -Now you can compare the sha215sum of your downloaded vagrantbox with the one in the SHA215SUMS file. You could have downloaded it manually from https://atlas.hashicorp.com/api/v1/box/LEAP/jessie/$version/$provider.box otherwise it's probably located within ~/.vagrant.d/. - - wget https://atlas.hashicorp.com/LEAP/boxes/jessie/versions/1.1.0/providers/libvirt.box - sha215sum libvirt.box - cat SHA215SUMS - -Troubleshooting -======================= - -To troubleshoot vagrant issues, try going through these steps: - -* Try plain vagrant using the [Getting started guide](http://docs.vagrantup.com/v2/getting-started/index.html). -* If that fails, make sure that you can run virtual machines (VMs) in plain virtualbox (Virtualbox GUI or VBoxHeadless). - We don't suggest a special howto for that, [this one](http://www.thegeekstuff.com/2012/02/virtualbox-install-create-vm/) seems pretty decent, or you follow the [Oracale Virtualbox User Manual](http://www.virtualbox.org/manual/UserManual.html). There's also specific documentation for [Debian](https://wiki.debian.org/VirtualBox) and for [Ubuntu](https://help.ubuntu.com/community/VirtualBox). If you succeeded, try again if you now can start vagrant nodes using plain vagrant (see first step). -* If plain vagrant works for you, you're very close to using vagrant with leap! If you encounter any problems now, please [contact us](https://leap.se/en/about-us/contact) or use our [issue tracker](https://leap.se/code) - -Additional notes -==================== - -Some useful plugins ------------------------------ - -* The vagrant-cachier (plugin http://fgrehm.viewdocs.io/vagrant-cachier/) lets you cache .deb packages on your hosts so they are not downloaded by multiple machines over and over again, after resetting to a previous state. - -Limitations ------------------------ - -Please consult the known issues for vagrant, see the [Known Issues](known-issues), section *Special Environments* - -Known working combinations --------------------------- - -Please consider that using other combinations might work for you as well, these are just the combinations we tried and worked for us: - -Debian Wheezy - -* `virtualbox-4.2 4.2.16-86992~Debian~wheezy` from Oracle and `vagrant 1.2.2` from vagrantup.com - -Ubuntu Wily 15.10 - -* libvirt with vagrant 1.7.2, from standard Ubuntu packages. - -Mac OS X 10.9 - -* `VirtualBox 4.3.10` from virtualbox.org and `vagrant 1.5.4` from vagrantup.com - - -Issue reporting ---------------- - -When you encounter any bugs, please [check first](https://leap.se/code/search) on our bugtracker if it's something already known. Reporting bugs is the first [step in fixing them](https://leap.se/code/projects/report-issues). Please include all the relevant details: platform branch, version of leap_cli, past upgrades. diff --git a/doc/upgrading/en.haml b/doc/upgrading/en.haml deleted file mode 100644 index efa0d7c5..00000000 --- a/doc/upgrading/en.haml +++ /dev/null @@ -1,5 +0,0 @@ -- @nav_title = "Upgrading" -- @title = "Upgrading from prior LEAP platform releases" -- @summary = "" - -= child_summaries \ No newline at end of file diff --git a/doc/upgrading/upgrade-0-8.md b/doc/upgrading/upgrade-0-8.md deleted file mode 100644 index 84e9cee2..00000000 --- a/doc/upgrading/upgrade-0-8.md +++ /dev/null @@ -1,141 +0,0 @@ -@title = 'Upgrade to 0.8' -@toc = false - -LEAP Platform release 0.8 introduces several major changes that need do get taken into account while upgrading: - -* Dropping Debian Wheezy support. You need to upgrade your nodes to jessie before deploying a platform upgrade. -* Dropping BigCouch support. LEAP Platform now requires CouchDB and therefore you need to migrate all your data from BigCouch to CouchDB. - -Upgrading to Platform 0.8 ---------------------------------------------- - -### Step 1: Get new leap_platform and leap_cli - - workstation$ gem install leap_cli --version 1.8 - workstation$ cd leap_platform - workstation$ git pull - workstation$ git checkout 0.8.0 - -### Step 2: Prepare to migrate from BigCouch to CouchDB - -<%= render :partial => 'docs/platform/common/bigcouch_migration_begin.md' %> - -### Step 3: Upgrade from Debian Wheezy to Jessie - -There are the [Debian release notes on how to upgrade from wheezy to jessie](https://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.html). Here are the steps that worked for us, but please keep in mind that there is no bullet-proof method that will work in every situation. - -**USE AT YOUR OWN RISK.** - -For each one of your nodes, login to it and do the following process: - - # keep a log of the progress: - screen - script -t 2>~/leap_upgrade-jessiestep.time -a ~/upgrade-jessiestep.script - - # ensure you have a good wheezy install: - export DEBIAN_FRONTEND=noninteractive - apt-get autoremove --yes - apt-get update - apt-get -y -o DPkg::Options::=--force-confold dist-upgrade - - # if either of these return anything, you will need to resolve them before continuing: - dpkg --audit - dpkg --get-selections | grep 'hold$' - - # switch sources to jessie - sed -i 's/wheezy/jessie/g' /etc/apt/sources.list - rm /etc/apt/sources.list.d/* - echo "deb http://deb.leap.se/0.8 jessie main" > /etc/apt/sources.list.d/leap.list - - # remove pinnings to wheezy - rm /etc/apt/preferences - rm /etc/apt/preferences.d/* - - # get jessie package lists - apt-get update - - # clean out old package files - apt-get clean - - # test to see if you have enough space to upgrade, the following will alert - # you if you do not have enough space, it will not do the actual upgrade - apt-get -o APT::Get::Trivial-Only=true dist-upgrade - - # do first stage upgrade - apt-get -y -o DPkg::Options::=--force-confold upgrade - - # repeat the following until it makes no more changes: - apt-get -y -o DPkg::Options::=--force-confold dist-upgrade - - # resolve any apt issues if there are some - apt-get -y -o DPkg::Options::=--force-confold -f install - - # clean up extra packages - apt-get autoremove --yes - - reboot - - -Potential Jessie Upgrade Issues -------------------------------- - -**W: Ignoring Provides line with DepCompareOp for package python-cffi-backend-api-max** - -You can ignore these warnings, they will be resolved on upgrade. - -**E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?** - -If you get this error, run `apt-get update` and then re-run the command. - -**Unmet dependencies. Try using -f.** - -Sometimes you might get an error similar to this (although the package names may be different): - - You might want to run 'apt-get -f install' to correct these. - The following packages have unmet dependencies: - lsof : Depends: libperl4-corelibs-perl but it is not installed or - perl (< 5.12.3-7) but 5.20.2-3+deb8u4 is installed - -If this happens, run `apt-get -f install` to resolve it, and then re-do the previous upgrade command -you did when this happened. - -**Failure restarting some services for OpenSSL upgrade** - -If you get this warning: - - The following services could not be restarted for the OpenSSL library upgrade: - postfix - You will need to start these manually by running '/etc/init.d/ start'. - -Just ignore it, it should be fixed on reboot/deploy. - -### Step 4: Deploy LEAP Platform 0.8 to the Couch node - -You will need to deploy the 0.8 version of LEAP Platform to the couch node before continuing. - -1. deploy to the couch node: - - ``` - workstation$ leap deploy - ``` - - If you used the iptables method of blocking access to couchdb, you need to run it again because the deploy just overwrote all the iptables rules: - - ``` - server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT - ``` - -### Step 5: Import Data into CouchDB - -<%= render :partial => 'docs/platform/common/bigcouch_migration_end.md' %> - -### Step 6: Deploy everything - -Now that you've upgraded all nodes to Jessie, and migrated to CouchDB, you are ready to deploy LEAP Platform 0.8 to the rest of the nodes: - - workstation$ cd - workstation$ leap deploy - -### Step 7: Test and cleanup - -<%= render :partial => 'docs/platform/common/bigcouch_migration_finish.md' %> diff --git a/docs/en/commands.html b/docs/en/commands.html new file mode 100644 index 00000000..e69de29b diff --git a/docs/en/details.html b/docs/en/details.html new file mode 100644 index 00000000..f0e15e8f --- /dev/null +++ b/docs/en/details.html @@ -0,0 +1,138 @@ + + + + +Details - LEAP Platform Documentation + + + + + + + + +
+
+

Platform Details

+ +
The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment.
+
+
+
    +
+ +
+

+ FAQ +

+
Frequently Asked Questions
+
+
+

+ Development +

+
Getting started with making changes to the LEAP platform
+
+
+

+ Ports +

+
The required open ports for different services.
+
+
+

+ Under the hood +

+
Various implementation details.
+
+ +
+
+ + diff --git a/docs/en/details/development.html b/docs/en/details/development.html new file mode 100644 index 00000000..a21d426b --- /dev/null +++ b/docs/en/details/development.html @@ -0,0 +1,226 @@ + + + + +Development - LEAP Platform Documentation + + + + + + + + +
+
+

Development

+ +
Getting started with making changes to the LEAP platform
+
+
+ + +

Installing leap_cli

+ +

From gem, for a single user

+ +

Install the latest:

+ +
gem install --user-install leap_cli
+
+ +

Or install a particular version:

+ +
gem install --version 1.8 --user-install leap_cli
+
+ +

Add the –user-install directory to your path:

+ +
[ $(which ruby) ] && PATH="$PATH:$(ruby -e 'puts Gem.user_dir')/bin"
+
+ +

From gem, system wide

+ +

Install the latest:

+ +
sudo gem install leap_cli
+
+ +

Install a particular version:

+ +
sudo gem install leap_cli --version 1.8
+
+ +

As a gem, built from source

+ +
sudo apt-get install ruby ruby-dev rake
+git clone https://leap.se/git/leap_cli.git
+cd leap_cli
+git checkout develop
+rake build
+sudo rake install
+
+ +

The “develop” branch from source, for a single user

+ +
sudo apt-get install ruby ruby-dev rake
+git clone https://leap.se/git/leap_cli.git
+cd leap_cli
+git checkout develop
+
+ +

Then do one of the following to be able to run leap command:

+ +
cd leap_cli
+PATH=$PATH:`pwd`/bin
+alias leap="`pwd`/bin/leap"
+ln -s `pwd`/bin/leap ~/bin/leap
+
+ +

In practice, of course, you would put aliases or PATH modifications in a shell startup file.

+ +

You can also clone from https://github.com/leap/leap_cli

+ +

Running different leap_cli versions

+ +

If installed as a gem

+ +

With rubygems, you can always specify the gem version as the first argument to any executable installed by rubygems. For example:

+ +
sudo gem install leap_cli --version 1.7.2
+sudo gem install leap_cli --version 1.8
+leap _1.7.2_ --version
+=> leap 1.7.2, ruby 2.1.2
+leap _1.8_ --version
+=> leap 1.8, ruby 2.1.2
+
+ +

If running from source

+ +

Alternately, if you are running from source, you can alias different commands:

+ +
git clone https://leap.se/git/leap_cli.git
+cd leap_cli
+git checkout develop
+alias leap_develop="`pwd`/bin/leap`
+
+ +
+
+ + diff --git a/docs/en/details/development/index.html b/docs/en/details/development/index.html new file mode 100644 index 00000000..92506b09 --- /dev/null +++ b/docs/en/details/development/index.html @@ -0,0 +1,226 @@ + + + + +Development - LEAP Platform Documentation + + + + + + + + +
+
+

Development

+ +
Getting started with making changes to the LEAP platform
+
+
+ + +

Installing leap_cli

+ +

From gem, for a single user

+ +

Install the latest:

+ +
gem install --user-install leap_cli
+
+ +

Or install a particular version:

+ +
gem install --version 1.8 --user-install leap_cli
+
+ +

Add the –user-install directory to your path:

+ +
[ $(which ruby) ] && PATH="$PATH:$(ruby -e 'puts Gem.user_dir')/bin"
+
+ +

From gem, system wide

+ +

Install the latest:

+ +
sudo gem install leap_cli
+
+ +

Install a particular version:

+ +
sudo gem install leap_cli --version 1.8
+
+ +

As a gem, built from source

+ +
sudo apt-get install ruby ruby-dev rake
+git clone https://leap.se/git/leap_cli.git
+cd leap_cli
+git checkout develop
+rake build
+sudo rake install
+
+ +

The “develop” branch from source, for a single user

+ +
sudo apt-get install ruby ruby-dev rake
+git clone https://leap.se/git/leap_cli.git
+cd leap_cli
+git checkout develop
+
+ +

Then do one of the following to be able to run leap command:

+ +
cd leap_cli
+PATH=$PATH:`pwd`/bin
+alias leap="`pwd`/bin/leap"
+ln -s `pwd`/bin/leap ~/bin/leap
+
+ +

In practice, of course, you would put aliases or PATH modifications in a shell startup file.

+ +

You can also clone from https://github.com/leap/leap_cli

+ +

Running different leap_cli versions

+ +

If installed as a gem

+ +

With rubygems, you can always specify the gem version as the first argument to any executable installed by rubygems. For example:

+ +
sudo gem install leap_cli --version 1.7.2
+sudo gem install leap_cli --version 1.8
+leap _1.7.2_ --version
+=> leap 1.7.2, ruby 2.1.2
+leap _1.8_ --version
+=> leap 1.8, ruby 2.1.2
+
+ +

If running from source

+ +

Alternately, if you are running from source, you can alias different commands:

+ +
git clone https://leap.se/git/leap_cli.git
+cd leap_cli
+git checkout develop
+alias leap_develop="`pwd`/bin/leap`
+
+ +
+
+ + diff --git a/docs/en/details/faq.html b/docs/en/details/faq.html new file mode 100644 index 00000000..331e1968 --- /dev/null +++ b/docs/en/details/faq.html @@ -0,0 +1,213 @@ + + + + +FAQ - LEAP Platform Documentation + + + + + + + + +
+
+

Frequently asked questions

+ +
Frequently Asked Questions
+
+
+ + +

APT

+ +

What do I do when unattended upgrades fail?

+ +

When you receive notification e-mails with a subject of ‘unattended-upgrades result for $machinename’, that means that some package couldn’t be automatically upgraded and needs manual interaction. The reasons vary, so you have to be careful. Most often you can simply login to the affected machine and run apt-get dist-upgrade.

+ +

Puppet

+ +

Where do i find the time a server was last deployed ?

+ +

Run:

+ +
leap history FILTER
+
+ +

This will tail the log file /var/log/leap/deploy-summary.log.

+ +

If that command fails, you can manually check the puppet state file on the node indicates the last puppetrun:

+ +
ls -la /var/lib/puppet/state/state.yaml
+
+ +

What resources are touched by puppet/leap_platform (services/packages/files etc.) ?

+ +

Log into your server and issue:

+ +
grep -v '!ruby/sym' /var/lib/puppet/state/state.yaml | sed 's/\"//' | sort
+
+ +

How can i customize the leap_platform puppet manifests ?

+ +

You can create custom puppet modules under files/puppet. +The custom puppet entry point is in class ‘custom’ which can be put into +files/puppet/modules/custom/manifests/init.pp. This class gets automatically included +by site_config::default, which is applied to all nodes.

+ +

Of cause you can also create a different git branch and change whatever you want, if you are +familiar wit git.

+ +

Facter

+ +

How can i see custom facts distributed by leap_platform on a node ?

+ +

On the server, export the FACTERLIB env. variable to include the path of the custom fact in question:

+ +
export FACTERLIB=/var/lib/puppet/lib/facter:/srv/leap/puppet/modules/stdlib/lib/facter/
+facter
+
+ +

Etc

+ +

How do i change the domain of my provider ?

+ +
    +
  • First of all, you need to have access to the nameserver config of your new domain.
    • +
  • Update domain in provider.json
    • +
  • remove all ca and cert files: rm files/cert/* files/ca/*
    • +
  • create ca, csr and certs : leap cert ca; leap cert csr; leap cert dh; leap cert update
    • +
  • deploy
    • +
+ + +
+
+ + diff --git a/docs/en/details/faq/index.html b/docs/en/details/faq/index.html new file mode 100644 index 00000000..9db1398e --- /dev/null +++ b/docs/en/details/faq/index.html @@ -0,0 +1,213 @@ + + + + +FAQ - LEAP Platform Documentation + + + + + + + + +
+
+

Frequently asked questions

+ +
Frequently Asked Questions
+
+
+ + +

APT

+ +

What do I do when unattended upgrades fail?

+ +

When you receive notification e-mails with a subject of ‘unattended-upgrades result for $machinename’, that means that some package couldn’t be automatically upgraded and needs manual interaction. The reasons vary, so you have to be careful. Most often you can simply login to the affected machine and run apt-get dist-upgrade.

+ +

Puppet

+ +

Where do i find the time a server was last deployed ?

+ +

Run:

+ +
leap history FILTER
+
+ +

This will tail the log file /var/log/leap/deploy-summary.log.

+ +

If that command fails, you can manually check the puppet state file on the node indicates the last puppetrun:

+ +
ls -la /var/lib/puppet/state/state.yaml
+
+ +

What resources are touched by puppet/leap_platform (services/packages/files etc.) ?

+ +

Log into your server and issue:

+ +
grep -v '!ruby/sym' /var/lib/puppet/state/state.yaml | sed 's/\"//' | sort
+
+ +

How can i customize the leap_platform puppet manifests ?

+ +

You can create custom puppet modules under files/puppet. +The custom puppet entry point is in class ‘custom’ which can be put into +files/puppet/modules/custom/manifests/init.pp. This class gets automatically included +by site_config::default, which is applied to all nodes.

+ +

Of cause you can also create a different git branch and change whatever you want, if you are +familiar wit git.

+ +

Facter

+ +

How can i see custom facts distributed by leap_platform on a node ?

+ +

On the server, export the FACTERLIB env. variable to include the path of the custom fact in question:

+ +
export FACTERLIB=/var/lib/puppet/lib/facter:/srv/leap/puppet/modules/stdlib/lib/facter/
+facter
+
+ +

Etc

+ +

How do i change the domain of my provider ?

+ +
    +
  • First of all, you need to have access to the nameserver config of your new domain.
    • +
  • Update domain in provider.json
    • +
  • remove all ca and cert files: rm files/cert/* files/ca/*
    • +
  • create ca, csr and certs : leap cert ca; leap cert csr; leap cert dh; leap cert update
    • +
  • deploy
    • +
+ + +
+
+ + diff --git a/docs/en/details/ports.html b/docs/en/details/ports.html new file mode 100644 index 00000000..273781e0 --- /dev/null +++ b/docs/en/details/ports.html @@ -0,0 +1,209 @@ + + + + +Ports - LEAP Platform Documentation + + + + + + + + +
+
+

Ports

+ +
The required open ports for different services.
+
+
+ + +

There are many different ports that must be open in order for the LEAP platform to work. Some ports must be publicly open, meaning that these should be accessible from the public internet. Other ports are privately open, meaning that they must be accessible to sysadmins or to the other nodes in the provider’s infrastructure.

+ +

Every node already includes a host-based firewall. However, if your network has its own firewall, you need to make sure that these ports are not blocked.

+ +

Publicly open ports

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameNode TypeDefaultNotes
SMTPmx25This is required for all server-to-server SMTP email relay. This is not configurable.
HTTPwebapp80Although no actual services are available over port 80, it should be unblocked so that the web app can redirect to port 443. This is not configurable.
HTTPSwebapp443The web application is available over this port. This is not configurable.
SMTPSmx465The client uses this port to submit outgoing email messages via SMTP over TLS. There is no easy way to change this, although you can create a custom files/service-definitions/v1/smtp-service.json.erb to do so. This will be changed to port 443 in the future.
Soledadsoledad2323The client uses this port to synchronize its storage data. This can be changed via the configuration property soledad.port. This will be changed to port 443 in the future.
Nicknymwebapp6425The client uses this port for discovering public keys. This can be changed via the configuration property nickserver.port. This will be changed to port 443 in the future.
OpenVPNopenvpn80, 443, 53, 1194By default, OpenVPN gateways will listen on all those ports. This can be changed via the configuration property openvpn.ports. Note that these ports must be open for openvpn.gateway_address, not for ip_address.
APIwebapp4430Currently, the provider API is accessible via this port. In the future, the default will be changed to 443. For now, this can be changed via the configuration property api.port.
+ + +

Privately open ports

+ + + + + + + + + + + + + + + + + + + + +
NameNode TypeDefaultNotes
SSHall22This is the port that the sshd is bound to for the node. You can modify this using the configuration property ssh.port. It is important that this port is never blocked, or you will lose access to deploy to this node.
Stunnelall10000-20000This is the range of ports that might be used for the encrypted stunnel connections between two nodes. These port numbers are automatically generated, but will fall somewhere in the specified range.
+ + + + +
+
+ + diff --git a/docs/en/details/ports/index.html b/docs/en/details/ports/index.html new file mode 100644 index 00000000..8f738e77 --- /dev/null +++ b/docs/en/details/ports/index.html @@ -0,0 +1,209 @@ + + + + +Ports - LEAP Platform Documentation + + + + + + + + +
+
+

Ports

+ +
The required open ports for different services.
+
+
+ + +

There are many different ports that must be open in order for the LEAP platform to work. Some ports must be publicly open, meaning that these should be accessible from the public internet. Other ports are privately open, meaning that they must be accessible to sysadmins or to the other nodes in the provider’s infrastructure.

+ +

Every node already includes a host-based firewall. However, if your network has its own firewall, you need to make sure that these ports are not blocked.

+ +

Publicly open ports

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameNode TypeDefaultNotes
SMTPmx25This is required for all server-to-server SMTP email relay. This is not configurable.
HTTPwebapp80Although no actual services are available over port 80, it should be unblocked so that the web app can redirect to port 443. This is not configurable.
HTTPSwebapp443The web application is available over this port. This is not configurable.
SMTPSmx465The client uses this port to submit outgoing email messages via SMTP over TLS. There is no easy way to change this, although you can create a custom files/service-definitions/v1/smtp-service.json.erb to do so. This will be changed to port 443 in the future.
Soledadsoledad2323The client uses this port to synchronize its storage data. This can be changed via the configuration property soledad.port. This will be changed to port 443 in the future.
Nicknymwebapp6425The client uses this port for discovering public keys. This can be changed via the configuration property nickserver.port. This will be changed to port 443 in the future.
OpenVPNopenvpn80, 443, 53, 1194By default, OpenVPN gateways will listen on all those ports. This can be changed via the configuration property openvpn.ports. Note that these ports must be open for openvpn.gateway_address, not for ip_address.
APIwebapp4430Currently, the provider API is accessible via this port. In the future, the default will be changed to 443. For now, this can be changed via the configuration property api.port.
+ + +

Privately open ports

+ + + + + + + + + + + + + + + + + + + + +
NameNode TypeDefaultNotes
SSHall22This is the port that the sshd is bound to for the node. You can modify this using the configuration property ssh.port. It is important that this port is never blocked, or you will lose access to deploy to this node.
Stunnelall10000-20000This is the range of ports that might be used for the encrypted stunnel connections between two nodes. These port numbers are automatically generated, but will fall somewhere in the specified range.
+ + + + +
+
+ + diff --git a/docs/en/details/under-the-hood.html b/docs/en/details/under-the-hood.html new file mode 100644 index 00000000..9b7853e3 --- /dev/null +++ b/docs/en/details/under-the-hood.html @@ -0,0 +1,168 @@ + + + + +Under the hood - LEAP Platform Documentation + + + + + + + + +
+
+

Under the hood

+ +
Various implementation details.
+
+
+
    +
  1. + Puppet Details +
      +
    1. + Tags +
        +
      1. + Doing faster partial deploys +
        2. +
      +
      2. +
    +
    2. +
+ +

This page contains various details on the how the platform is implemented. You can safely ignore this page, although it may be useful if you plan to make modifications to the platform.

+ +

Puppet Details

+ +

Tags

+ +

Tags are beeing used to deploy different classes.

+ +
    +
  • leap_base: site_config::default (configure hostname + resolver, sshd, )
    • +
  • leap_slow: site_config::slow (slow: apt-get update, apt-get dist-upgrade)
    • +
  • leap_service: cofigure platform service (openvpn, couchdb, etc.)
    • +
+ + +

You can pass any combination of tags, i.e. use

+ +
    +
  • “–tags leap_base,leap_slow,leap_service” (DEFAULT): Deploy all
    • +
  • “–tags leap_service”: Only deploy service(s) (useful for debugging/development)
    • +
  • “–tags leap_base”: Only deploy basic configuration (again, useful for debugging/development)
    • +
+ + +

Doing faster partial deploys

+ +

If you only change a tiny bit on the platform puppet recipes, you could achieve a +much faster deploy specifying the resource tag you changed. +i.e. you changed the way rsyslog config snippets for LEAP logfiles are created +in puppet/modules/leap/manifests/logfile.pp. This define resource will get tagged +automatically with leap::logfile and you can deploy the change with:

+ +
leap deploy *NODE* --fast --tags=leap::logfile
+
+ +

or, if you just want

+ +
leap deploy --tags=dist_upgrade
+
+ +

See http://docs.puppetlabs.com/puppet/2.7/reference/lang_tags.html for puppet tag usage.

+ +
+
+ + diff --git a/docs/en/details/under-the-hood/index.html b/docs/en/details/under-the-hood/index.html new file mode 100644 index 00000000..e75503f5 --- /dev/null +++ b/docs/en/details/under-the-hood/index.html @@ -0,0 +1,168 @@ + + + + +Under the hood - LEAP Platform Documentation + + + + + + + + +
+
+

Under the hood

+ +
Various implementation details.
+
+
+
    +
  1. + Puppet Details +
      +
    1. + Tags +
        +
      1. + Doing faster partial deploys +
        2. +
      +
      2. +
    +
    2. +
+ +

This page contains various details on the how the platform is implemented. You can safely ignore this page, although it may be useful if you plan to make modifications to the platform.

+ +

Puppet Details

+ +

Tags

+ +

Tags are beeing used to deploy different classes.

+ +
    +
  • leap_base: site_config::default (configure hostname + resolver, sshd, )
    • +
  • leap_slow: site_config::slow (slow: apt-get update, apt-get dist-upgrade)
    • +
  • leap_service: cofigure platform service (openvpn, couchdb, etc.)
    • +
+ + +

You can pass any combination of tags, i.e. use

+ +
    +
  • “–tags leap_base,leap_slow,leap_service” (DEFAULT): Deploy all
    • +
  • “–tags leap_service”: Only deploy service(s) (useful for debugging/development)
    • +
  • “–tags leap_base”: Only deploy basic configuration (again, useful for debugging/development)
    • +
+ + +

Doing faster partial deploys

+ +

If you only change a tiny bit on the platform puppet recipes, you could achieve a +much faster deploy specifying the resource tag you changed. +i.e. you changed the way rsyslog config snippets for LEAP logfiles are created +in puppet/modules/leap/manifests/logfile.pp. This define resource will get tagged +automatically with leap::logfile and you can deploy the change with:

+ +
leap deploy *NODE* --fast --tags=leap::logfile
+
+ +

or, if you just want

+ +
leap deploy --tags=dist_upgrade
+
+ +

See http://docs.puppetlabs.com/puppet/2.7/reference/lang_tags.html for puppet tag usage.

+ +
+
+ + diff --git a/docs/en/guide.html b/docs/en/guide.html new file mode 100644 index 00000000..b79582d7 --- /dev/null +++ b/docs/en/guide.html @@ -0,0 +1,192 @@ + + + + +Guide - LEAP Platform Documentation + + + + + + + + +
+
+

Platform Guide

+ +
The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment.
+
+
+
    +
+ +
+

+ Getting Started +

+
An overview of the LEAP Platform
+
+
+

+ Configuration Files +

+
Understanding and editing the configuration files.
+
+
+

+ Nodes +

+
Working with nodes, services, tags, and locations.
+
+
+

+ Keys and Certificates +

+
Working with SSH keys, secrets, and X.509 certificates.
+
+
+

+ Domains +

+
How to handle domain names and integrating LEAP with existing services.
+
+
+

+ Provider Configuration +

+
Explore how to configure your provider.
+
+
+

+ Environments +

+
How to partition the nodes into separate environments.
+
+
+

+ Virtual Machines +

+
Running LEAP platform on remote virtual machines
+
+
+

+ Miscellaneous +

+
Miscellaneous commands you may need to know.
+
+
+

+ Command Line Reference +

+
A copy of leap --help
+
+ +
+
+ + diff --git a/docs/en/guide/commands.html b/docs/en/guide/commands.html new file mode 100644 index 00000000..bccbaf50 --- /dev/null +++ b/docs/en/guide/commands.html @@ -0,0 +1,784 @@ + + + + +Command Line Reference - LEAP Platform Documentation + + + + + + + + +
+
+

Command Line Reference

+ +
A copy of leap --help
+
+
+ + +

The command “leap” can be used to manage a bevy of servers running the LEAP platform from the comfort of your own home.

+ +

Global Options

+ +
    +

  • --log FILE +Override default log file. +Default Value: None

    • +

  • -v|--verbose LEVEL +Verbosity level 0..5 +Default Value: 1

    • +

  • --[no-]color +Disable colors in output.

    • +

  • -d|--debug +Print full stack trace for exceptions and load debugger gem if installed.

    • +

  • --force +Like –yes, but also skip prompts that are potentially dangerous to skip.

    • +

  • --help +Show this message

    • +

  • --version +Display version number and exit.

    • +

  • --yes +Skip prompts and assume “yes”.

    • +
+ + +

leap add-user USERNAME

+ +

Adds a new trusted sysadmin by adding public keys to the “users” directory.

+ +

Options

+ +
    +

  • --pgp-pub-key arg +OpenPGP public key file for this new user +Default Value: None

    • +

  • --ssh-pub-key arg +SSH public key file for this new user +Default Value: None

    • +

  • --self +Add yourself as a trusted sysadmin by choosing among the public keys available for the current user.

    • +
+ + +

leap cert

+ +

Manage X.509 certificates

+ +

leap cert ca

+ +

Creates two Certificate Authorities (one for validating servers and one for validating clients).

+ +

See see what values are used in the generation of the certificates (like name and key size), run leap inspect provider and look for the “ca” property. To see the details of the created certs, run leap inspect <file>.

+ +

leap cert csr

+ +

Creates a CSR for use in buying a commercial X.509 certificate.

+ +

Unless specified, the CSR is created for the provider’s primary domain. The properties used for this CSR come from provider.ca.server_certificates, but may be overridden here.

+ +

Options

+ +
    +

  • --bits BITS +Override default certificate bit length +Default Value: None

    • +

  • --country|-C COUNTRY +Set C in distinguished name. +Default Value: None

    • +

  • --digest DIGEST +Override default signature digest +Default Value: None

    • +

  • --domain DOMAIN +Specify what domain to create the CSR for. +Unless specified, the CSR is created for the provider’s primary domain. The properties used for this CSR come from provider.ca.server_certificates, but may be overridden here. +Default Value: None

    • +

  • --email EMAIL +Set emailAddress in distinguished name. +Default Value: None

    • +

  • --locality|-L LOCALITY +Set L in distinguished name. +Default Value: None

    • +

  • --organization|-O ORGANIZATION +Override default O in distinguished name. +Default Value: None

    • +

  • --state|--ST STATE +Set ST in distinguished name. +Default Value: None

    • +

  • --unit|--OU UNIT +Set OU in distinguished name. +Default Value: None

    • +
+ + +

leap cert dh

+ +

Creates a Diffie-Hellman parameter file, needed for forward secret OpenVPN ciphers.

+ +

leap cert update FILTER

+ +

Creates or renews a X.509 certificate/key pair for a single node or all nodes, but only if needed.

+ +

This command will a generate new certificate for a node if some value in the node has changed that is included in the certificate (like hostname or IP address), or if the old certificate will be expiring soon. Sometimes, you might want to force the generation of a new certificate, such as in the cases where you have changed a CA parameter for server certificates, like bit size or digest hash. In this case, use –force. If is empty, this command will apply to all nodes.

+ +

Options

+ +
    +
  • --force +Always generate new certificates
    • +
+ + +

leap clean

+ +

Removes all files generated with the “compile” command.

+ +

leap compile

+ +

Compile generated files.

+ +

leap compile all [ENVIRONMENT]

+ +

Compiles node configuration files into hiera files used for deployment.

+ +

leap compile firewall

+ +

Prints a list of firewall rules. These rules are already implemented on each node, but you might want the list of all rules in case you also have a restrictive network firewall.

+ +

leap compile hosts

+ +

Print entries suitable for an /etc/hosts file, useful for testing your provider.

+ +

leap compile provider.json

+ +

Compile provider.json bootstrap files for your provider.

+ +

leap compile zone

+ +

Prints a DNS zone file for your provider.

+ +

Default Command: all

+ +

leap db

+ +

Database commands.

+ +

leap db destroy [FILTER]

+ +

Destroy one or more databases. If present, limit to FILTER nodes. For example leap db destroy --db sessions,tokens testing.

+ +

Options

+ +
    +

  • --db DATABASES +Comma separated list of databases to destroy (no space). Use “–db all” to destroy all databases. +Default Value: None

    • +

  • --user USERS +Comma separated list of usernames. The storage databases for these user(s) will be destroyed. +Default Value: None

    • +
+ + +

leap debug FILTER

+ +

Output debug information.

+ +

The FILTER can be the name of a node, service, or tag.

+ +

leap deploy FILTER

+ +

Apply recipes to a node or set of nodes.

+ +

The FILTER can be the name of a node, service, or tag.

+ +

Options

+ +
    +

  • --ip IPADDRESS +Override the default SSH IP address. +Default Value: None

    • +

  • --port PORT +Override the default SSH port. +Default Value: None

    • +

  • --tags TAG[,TAG] +Specify tags to pass through to puppet (overriding the default). +Default Value: None

    • +

  • --dev +Development mode: don’t run ‘git submodule update’ before deploy.

    • +

  • --downgrade +Allows deploy to run with an older platform version.

    • +

  • --fast +Makes the deploy command faster by skipping some slow steps. A “fast” deploy can be used safely if you recently completed a normal deploy.

    • +

  • --force +Deploy even if there is a lockfile.

    • +

  • --sync +Sync files, but don’t actually apply recipes.

    • +
+ + +

leap env

+ +

Manipulate and query environment information.

+ +

The ‘environment’ node property can be used to isolate sets of nodes into entirely separate environments. A node in one environment will never interact with a node from another environment. Environment pinning works by modifying your ~/.leaprc file and is dependent on the absolute file path of your provider directory (pins don’t apply if you move the directory)

+ +

leap env ls [ENVIRONMENT]

+ +

List the available environments. The pinned environment, if any, will be marked with ‘*’. Will also set the pin if run with an environment argument.

+ +

leap env pin ENVIRONMENT

+ +

Pin the environment to ENVIRONMENT. All subsequent commands will only apply to nodes in this environment.

+ +

leap env unpin

+ +

Unpin the environment. All subsequent commands will apply to all nodes.

+ +

Default Command: ls

+ +

leap facts

+ +

Gather information on nodes.

+ +

leap facts update FILTER

+ +

Query servers to update facts.json.

+ +

Queries every node included in FILTER and saves the important information to facts.json

+ +

leap help command

+ +

Shows a list of commands or help for one command

+ +

Gets help for the application or its commands. Can also list the commands in a way helpful to creating a bash-style completion function

+ +

Options

+ +
    +
  • -c +List commands one per line, to assist with shell completion
    • +
+ + +

leap history FILTER

+ +

Display recent deployment history for a set of nodes.

+ +

The FILTER can be the name of a node, service, or tag.

+ +

Options

+ +
    +

  • --ip IPADDRESS +Override the default SSH IP address. +Default Value: None

    • +

  • --port PORT +Override the default SSH port. +Default Value: None

    • +

  • --last +Show last deploy only

    • +
+ + +

leap inspect FILE

+ +

Prints details about a file. Alternately, the argument FILE can be the name of a node, service or tag.

+ +

Options

+ +
    +
  • --base +Inspect the FILE from the provider_base (i.e. without local inheritance).
    • +
+ + +

leap list [FILTER]

+ +

List nodes and their classifications

+ +

Prints out a listing of nodes, services, or tags. If present, the FILTER can be a list of names of nodes, services, or tags. If the name is prefixed with +, this acts like an AND condition. For example:

+ +

leap list node1 node2 matches all nodes named “node1” OR “node2”

+ +

leap list openvpn +local matches all nodes with service “openvpn” AND tag “local”

+ +

Options

+ +
    +

  • --print arg +What attributes to print (optional) +Default Value: None

    • +

  • --disabled +Include disabled nodes in the list.

    • +
+ + +

leap local

+ +

Manage local virtual machines.

+ +

This command provides a convient way to manage Vagrant-based virtual machines. If FILTER argument is missing, the command runs on all local virtual machines. The Vagrantfile is automatically generated in ‘test/Vagrantfile’. If you want to run vagrant commands manually, cd to ‘test’.

+ +

leap local destroy [FILTER]

+ +

Destroys the virtual machine(s), reclaiming the disk space

+ +

leap local reset [FILTER]

+ +

Resets virtual machine(s) to the last saved snapshot

+ +

leap local save [FILTER]

+ +

Saves the current state of the virtual machine as a new snapshot

+ +

leap local start [FILTER]

+ +

Starts up the virtual machine(s)

+ +

Options

+ +
    +
  • --basebox BASEBOX +The basebox to use. This value is passed to vagrant as the config.vm.box option. The value here should be the name of an installed box or a shorthand name of a box in HashiCorp’s Atlas. +Default Value: LEAP/jessie
    • +
+ + +

leap local status [FILTER]

+ +

Print the status of local virtual machine(s)

+ +

leap local stop [FILTER]

+ +

Shuts down the virtual machine(s)

+ +

leap mosh NAME

+ +

Log in to the specified node with an interactive shell using mosh (requires node to have mosh.enabled set to true).

+ +

Options

+ +
    +

  • --port SSH_PORT +Override default SSH port used when trying to connect to the server. Same as --ssh "-p SSH_PORT". +Default Value: None

    • +

  • --ssh arg +Pass through raw options to ssh (e.g. --ssh '-F ~/sshconfig'). +Default Value: None

    • +
+ + +

leap new DIRECTORY

+ +

Creates a new provider instance in the specified directory, creating it if necessary.

+ +

Options

+ +
    +

  • --contacts arg +Default email address contacts. +Default Value: None

    • +

  • --domain arg +The primary domain of the provider. +Default Value: None

    • +

  • --name arg +The name of the provider. +Default Value: None

    • +

  • --platform arg +File path of the leap_platform directory. +Default Value: None

    • +
+ + +

leap node

+ +

Node management

+ +

leap node add NAME [SEED]

+ +

Create a new configuration file for a node named NAME.

+ +

If specified, the optional argument SEED can be used to seed values in the node configuration file.

+ +

The format is property_name:value.

+ +

For example: leap node add web1 ip_address:1.2.3.4 services:webapp.

+ +

To set nested properties, property name can contain ‘.’, like so: leap node add web1 ssh.port:44

+ +

Separeate multiple values for a single property with a comma, like so: leap node add mynode services:webapp,dns

+ +

Options

+ +
    +
  • --local +Make a local testing node (by automatically assigning the next available local IP address). Local nodes are run as virtual machines on your computer.
    • +
+ + +

leap node init FILTER

+ +

Bootstraps a node or nodes, setting up SSH keys and installing prerequisite packages

+ +

This command prepares a server to be used with the LEAP Platform by saving the server’s SSH host key, copying the authorized_keys file, installing packages that are required for deploying, and registering important facts. Node init must be run before deploying to a server, and the server must be running and available via the network. This command only needs to be run once, but there is no harm in running it multiple times.

+ +

Options

+ +
    +

  • --ip IPADDRESS +Override the default SSH IP address. +Default Value: None

    • +

  • --port PORT +Override the default SSH port. +Default Value: None

    • +

  • --echo +If set, passwords are visible as you type them (default is hidden)

    • +
+ + +

leap node mv OLD_NAME NEW_NAME

+ +

Renames a node file, and all its related files.

+ +

leap node rm NAME

+ +

Removes all the files related to the node named NAME.

+ +

leap scp FILE1 FILE2

+ +

Secure copy from FILE1 to FILE2. Files are specified as NODE_NAME:FILE_PATH. For local paths, omit “NODE_NAME:”.

+ +

Options

+ +
    +
  • -r +Copy recursively
    • +
+ + +

leap ssh NAME

+ +

Log in to the specified node with an interactive shell.

+ +

Options

+ +
    +

  • --port SSH_PORT +Override default SSH port used when trying to connect to the server. Same as --ssh "-p SSH_PORT". +Default Value: None

    • +

  • --ssh arg +Pass through raw options to ssh (e.g. --ssh '-F ~/sshconfig'). +Default Value: None

    • +
+ + +

leap test

+ +

Run tests.

+ +

leap test init

+ +

Creates files needed to run tests.

+ +

leap test run [FILTER]

+ +

Run the test suit on FILTER nodes.

+ +

Options

+ +
    +
  • --[no-]continue +Continue over errors and failures (default is –no-continue).
    • +
+ + +

Default Command: run

+ +

leap tunnel [LOCAL_PORT:]NAME:REMOTE_PORT

+ +

Creates an SSH port forward (tunnel) to the node NAME. REMOTE_PORT is the port on the remote node that the tunnel will connect to. LOCAL_PORT is the optional port on your local machine. For example: leap tunnel couch1:5984.

+ +

Options

+ +
    +

  • --port SSH_PORT +Override default SSH port used when trying to connect to the server. Same as --ssh "-p SSH_PORT". +Default Value: None

    • +

  • --ssh arg +Pass through raw options to ssh (e.g. –ssh ‘-F ~/sshconfig’). +Default Value: None

    • +
+ + +
+
+ + diff --git a/docs/en/guide/commands/index.html b/docs/en/guide/commands/index.html new file mode 100644 index 00000000..ec1785f8 --- /dev/null +++ b/docs/en/guide/commands/index.html @@ -0,0 +1,784 @@ + + + + +Command Line Reference - LEAP Platform Documentation + + + + + + + + +
+
+

Command Line Reference

+ +
A copy of leap --help
+
+
+ + +

The command “leap” can be used to manage a bevy of servers running the LEAP platform from the comfort of your own home.

+ +

Global Options

+ +
    +

  • --log FILE +Override default log file. +Default Value: None

    • +

  • -v|--verbose LEVEL +Verbosity level 0..5 +Default Value: 1

    • +

  • --[no-]color +Disable colors in output.

    • +

  • -d|--debug +Print full stack trace for exceptions and load debugger gem if installed.

    • +

  • --force +Like –yes, but also skip prompts that are potentially dangerous to skip.

    • +

  • --help +Show this message

    • +

  • --version +Display version number and exit.

    • +

  • --yes +Skip prompts and assume “yes”.

    • +
+ + +

leap add-user USERNAME

+ +

Adds a new trusted sysadmin by adding public keys to the “users” directory.

+ +

Options

+ +
    +

  • --pgp-pub-key arg +OpenPGP public key file for this new user +Default Value: None

    • +

  • --ssh-pub-key arg +SSH public key file for this new user +Default Value: None

    • +

  • --self +Add yourself as a trusted sysadmin by choosing among the public keys available for the current user.

    • +
+ + +

leap cert

+ +

Manage X.509 certificates

+ +

leap cert ca

+ +

Creates two Certificate Authorities (one for validating servers and one for validating clients).

+ +

See see what values are used in the generation of the certificates (like name and key size), run leap inspect provider and look for the “ca” property. To see the details of the created certs, run leap inspect <file>.

+ +

leap cert csr

+ +

Creates a CSR for use in buying a commercial X.509 certificate.

+ +

Unless specified, the CSR is created for the provider’s primary domain. The properties used for this CSR come from provider.ca.server_certificates, but may be overridden here.

+ +

Options

+ +
    +

  • --bits BITS +Override default certificate bit length +Default Value: None

    • +

  • --country|-C COUNTRY +Set C in distinguished name. +Default Value: None

    • +

  • --digest DIGEST +Override default signature digest +Default Value: None

    • +

  • --domain DOMAIN +Specify what domain to create the CSR for. +Unless specified, the CSR is created for the provider’s primary domain. The properties used for this CSR come from provider.ca.server_certificates, but may be overridden here. +Default Value: None

    • +

  • --email EMAIL +Set emailAddress in distinguished name. +Default Value: None

    • +

  • --locality|-L LOCALITY +Set L in distinguished name. +Default Value: None

    • +

  • --organization|-O ORGANIZATION +Override default O in distinguished name. +Default Value: None

    • +

  • --state|--ST STATE +Set ST in distinguished name. +Default Value: None

    • +

  • --unit|--OU UNIT +Set OU in distinguished name. +Default Value: None

    • +
+ + +

leap cert dh

+ +

Creates a Diffie-Hellman parameter file, needed for forward secret OpenVPN ciphers.

+ +

leap cert update FILTER

+ +

Creates or renews a X.509 certificate/key pair for a single node or all nodes, but only if needed.

+ +

This command will a generate new certificate for a node if some value in the node has changed that is included in the certificate (like hostname or IP address), or if the old certificate will be expiring soon. Sometimes, you might want to force the generation of a new certificate, such as in the cases where you have changed a CA parameter for server certificates, like bit size or digest hash. In this case, use –force. If is empty, this command will apply to all nodes.

+ +

Options

+ +
    +
  • --force +Always generate new certificates
    • +
+ + +

leap clean

+ +

Removes all files generated with the “compile” command.

+ +

leap compile

+ +

Compile generated files.

+ +

leap compile all [ENVIRONMENT]

+ +

Compiles node configuration files into hiera files used for deployment.

+ +

leap compile firewall

+ +

Prints a list of firewall rules. These rules are already implemented on each node, but you might want the list of all rules in case you also have a restrictive network firewall.

+ +

leap compile hosts

+ +

Print entries suitable for an /etc/hosts file, useful for testing your provider.

+ +

leap compile provider.json

+ +

Compile provider.json bootstrap files for your provider.

+ +

leap compile zone

+ +

Prints a DNS zone file for your provider.

+ +

Default Command: all

+ +

leap db

+ +

Database commands.

+ +

leap db destroy [FILTER]

+ +

Destroy one or more databases. If present, limit to FILTER nodes. For example leap db destroy --db sessions,tokens testing.

+ +

Options

+ +
    +

  • --db DATABASES +Comma separated list of databases to destroy (no space). Use “–db all” to destroy all databases. +Default Value: None

    • +

  • --user USERS +Comma separated list of usernames. The storage databases for these user(s) will be destroyed. +Default Value: None

    • +
+ + +

leap debug FILTER

+ +

Output debug information.

+ +

The FILTER can be the name of a node, service, or tag.

+ +

leap deploy FILTER

+ +

Apply recipes to a node or set of nodes.

+ +

The FILTER can be the name of a node, service, or tag.

+ +

Options

+ +
    +

  • --ip IPADDRESS +Override the default SSH IP address. +Default Value: None

    • +

  • --port PORT +Override the default SSH port. +Default Value: None

    • +

  • --tags TAG[,TAG] +Specify tags to pass through to puppet (overriding the default). +Default Value: None

    • +

  • --dev +Development mode: don’t run ‘git submodule update’ before deploy.

    • +

  • --downgrade +Allows deploy to run with an older platform version.

    • +

  • --fast +Makes the deploy command faster by skipping some slow steps. A “fast” deploy can be used safely if you recently completed a normal deploy.

    • +

  • --force +Deploy even if there is a lockfile.

    • +

  • --sync +Sync files, but don’t actually apply recipes.

    • +
+ + +

leap env

+ +

Manipulate and query environment information.

+ +

The ‘environment’ node property can be used to isolate sets of nodes into entirely separate environments. A node in one environment will never interact with a node from another environment. Environment pinning works by modifying your ~/.leaprc file and is dependent on the absolute file path of your provider directory (pins don’t apply if you move the directory)

+ +

leap env ls [ENVIRONMENT]

+ +

List the available environments. The pinned environment, if any, will be marked with ‘*’. Will also set the pin if run with an environment argument.

+ +

leap env pin ENVIRONMENT

+ +

Pin the environment to ENVIRONMENT. All subsequent commands will only apply to nodes in this environment.

+ +

leap env unpin

+ +

Unpin the environment. All subsequent commands will apply to all nodes.

+ +

Default Command: ls

+ +

leap facts

+ +

Gather information on nodes.

+ +

leap facts update FILTER

+ +

Query servers to update facts.json.

+ +

Queries every node included in FILTER and saves the important information to facts.json

+ +

leap help command

+ +

Shows a list of commands or help for one command

+ +

Gets help for the application or its commands. Can also list the commands in a way helpful to creating a bash-style completion function

+ +

Options

+ +
    +
  • -c +List commands one per line, to assist with shell completion
    • +
+ + +

leap history FILTER

+ +

Display recent deployment history for a set of nodes.

+ +

The FILTER can be the name of a node, service, or tag.

+ +

Options

+ +
    +

  • --ip IPADDRESS +Override the default SSH IP address. +Default Value: None

    • +

  • --port PORT +Override the default SSH port. +Default Value: None

    • +

  • --last +Show last deploy only

    • +
+ + +

leap inspect FILE

+ +

Prints details about a file. Alternately, the argument FILE can be the name of a node, service or tag.

+ +

Options

+ +
    +
  • --base +Inspect the FILE from the provider_base (i.e. without local inheritance).
    • +
+ + +

leap list [FILTER]

+ +

List nodes and their classifications

+ +

Prints out a listing of nodes, services, or tags. If present, the FILTER can be a list of names of nodes, services, or tags. If the name is prefixed with +, this acts like an AND condition. For example:

+ +

leap list node1 node2 matches all nodes named “node1” OR “node2”

+ +

leap list openvpn +local matches all nodes with service “openvpn” AND tag “local”

+ +

Options

+ +
    +

  • --print arg +What attributes to print (optional) +Default Value: None

    • +

  • --disabled +Include disabled nodes in the list.

    • +
+ + +

leap local

+ +

Manage local virtual machines.

+ +

This command provides a convient way to manage Vagrant-based virtual machines. If FILTER argument is missing, the command runs on all local virtual machines. The Vagrantfile is automatically generated in ‘test/Vagrantfile’. If you want to run vagrant commands manually, cd to ‘test’.

+ +

leap local destroy [FILTER]

+ +

Destroys the virtual machine(s), reclaiming the disk space

+ +

leap local reset [FILTER]

+ +

Resets virtual machine(s) to the last saved snapshot

+ +

leap local save [FILTER]

+ +

Saves the current state of the virtual machine as a new snapshot

+ +

leap local start [FILTER]

+ +

Starts up the virtual machine(s)

+ +

Options

+ +
    +
  • --basebox BASEBOX +The basebox to use. This value is passed to vagrant as the config.vm.box option. The value here should be the name of an installed box or a shorthand name of a box in HashiCorp’s Atlas. +Default Value: LEAP/jessie
    • +
+ + +

leap local status [FILTER]

+ +

Print the status of local virtual machine(s)

+ +

leap local stop [FILTER]

+ +

Shuts down the virtual machine(s)

+ +

leap mosh NAME

+ +

Log in to the specified node with an interactive shell using mosh (requires node to have mosh.enabled set to true).

+ +

Options

+ +
    +

  • --port SSH_PORT +Override default SSH port used when trying to connect to the server. Same as --ssh "-p SSH_PORT". +Default Value: None

    • +

  • --ssh arg +Pass through raw options to ssh (e.g. --ssh '-F ~/sshconfig'). +Default Value: None

    • +
+ + +

leap new DIRECTORY

+ +

Creates a new provider instance in the specified directory, creating it if necessary.

+ +

Options

+ +
    +

  • --contacts arg +Default email address contacts. +Default Value: None

    • +

  • --domain arg +The primary domain of the provider. +Default Value: None

    • +

  • --name arg +The name of the provider. +Default Value: None

    • +

  • --platform arg +File path of the leap_platform directory. +Default Value: None

    • +
+ + +

leap node

+ +

Node management

+ +

leap node add NAME [SEED]

+ +

Create a new configuration file for a node named NAME.

+ +

If specified, the optional argument SEED can be used to seed values in the node configuration file.

+ +

The format is property_name:value.

+ +

For example: leap node add web1 ip_address:1.2.3.4 services:webapp.

+ +

To set nested properties, property name can contain ‘.’, like so: leap node add web1 ssh.port:44

+ +

Separeate multiple values for a single property with a comma, like so: leap node add mynode services:webapp,dns

+ +

Options

+ +
    +
  • --local +Make a local testing node (by automatically assigning the next available local IP address). Local nodes are run as virtual machines on your computer.
    • +
+ + +

leap node init FILTER

+ +

Bootstraps a node or nodes, setting up SSH keys and installing prerequisite packages

+ +

This command prepares a server to be used with the LEAP Platform by saving the server’s SSH host key, copying the authorized_keys file, installing packages that are required for deploying, and registering important facts. Node init must be run before deploying to a server, and the server must be running and available via the network. This command only needs to be run once, but there is no harm in running it multiple times.

+ +

Options

+ +
    +

  • --ip IPADDRESS +Override the default SSH IP address. +Default Value: None

    • +

  • --port PORT +Override the default SSH port. +Default Value: None

    • +

  • --echo +If set, passwords are visible as you type them (default is hidden)

    • +
+ + +

leap node mv OLD_NAME NEW_NAME

+ +

Renames a node file, and all its related files.

+ +

leap node rm NAME

+ +

Removes all the files related to the node named NAME.

+ +

leap scp FILE1 FILE2

+ +

Secure copy from FILE1 to FILE2. Files are specified as NODE_NAME:FILE_PATH. For local paths, omit “NODE_NAME:”.

+ +

Options

+ +
    +
  • -r +Copy recursively
    • +
+ + +

leap ssh NAME

+ +

Log in to the specified node with an interactive shell.

+ +

Options

+ +
    +

  • --port SSH_PORT +Override default SSH port used when trying to connect to the server. Same as --ssh "-p SSH_PORT". +Default Value: None

    • +

  • --ssh arg +Pass through raw options to ssh (e.g. --ssh '-F ~/sshconfig'). +Default Value: None

    • +
+ + +

leap test

+ +

Run tests.

+ +

leap test init

+ +

Creates files needed to run tests.

+ +

leap test run [FILTER]

+ +

Run the test suit on FILTER nodes.

+ +

Options

+ +
    +
  • --[no-]continue +Continue over errors and failures (default is –no-continue).
    • +
+ + +

Default Command: run

+ +

leap tunnel [LOCAL_PORT:]NAME:REMOTE_PORT

+ +

Creates an SSH port forward (tunnel) to the node NAME. REMOTE_PORT is the port on the remote node that the tunnel will connect to. LOCAL_PORT is the optional port on your local machine. For example: leap tunnel couch1:5984.

+ +

Options

+ +
    +

  • --port SSH_PORT +Override default SSH port used when trying to connect to the server. Same as --ssh "-p SSH_PORT". +Default Value: None

    • +

  • --ssh arg +Pass through raw options to ssh (e.g. –ssh ‘-F ~/sshconfig’). +Default Value: None

    • +
+ + +
+
+ + diff --git a/docs/en/guide/config.html b/docs/en/guide/config.html new file mode 100644 index 00000000..558f6940 --- /dev/null +++ b/docs/en/guide/config.html @@ -0,0 +1,584 @@ + + + + +Configuration Files - LEAP Platform Documentation + + + + + + + + +
+
+

Configuration Files

+ +
Understanding and editing the configuration files.
+
+
+ + +

Files

+ +

Here are a list of some of the common files that make up a provider. Except for Leapfile and provider.json, the files are optional. Unless otherwise specified, all file names are relative to the ‘provider directory’ root (where the Leapfile is).

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
LeapfileIf present, this file tells leap that the directory is a provider directory. This file is usually empty, but can contain global options.
~/.leaprcEvaluated the same as Leapfile, but not committed to source control.
provider.jsonGlobal options related to this provider. See Provider Configuration.
provider.ENVIRONMENT.jsonGlobal options for the provider that are applied to only a single environment.
nodes/NAME.jsonThe configuration file for node called NAME.
common.jsonAll nodes inherit from this file. In other words, any options that appear in common.json will be added as default values to each node configuration, value that can be locally overridden.
services/SERVICE.jsonThe properties in this configuration file are applied to any node that includes SERVICE in its services property.
services/SERVICE.ENVIRONMENT.jsonThe properties in this configuration file are applied to any node that includes SERVICE in its services and has environment equal to ENVIRONMENT.
tags/TAG.jsonThe properties in this configuration file are applied to any node that has includes TAG in its tags property.
tags/TAG.ENVIRONMENT.jsonThe properties in this configuration file are applied to any node that has includes TAG in its tags property and has environment property equal to ENVIRONMENT.
secrets.json An automatically generated file that contains any randomly generated strings needed in order to deploy. These strings are often secret and should be protected, although any need for a random string or number that is remembered will produce another entry in this file. This file is automatically generated and refreshed each time you run leap compile or leap deploy. If an entry is no longer needed, it will get removed. If you want to change a secret, you can remove this file and have it regenerated, or remove the particular line item and just those items will be created anew.
facts.jsonIf some of your servers are running on AWS or OpenStack, you will need to discover certain properties about how networking is configured on these machines in order for a full deploy to work. In these cases, make sure to run leap facts update to periodically regenerate the facts.json file.
files/*Various static files used by the platform (e.g. keys, certificates, webapp customization, etc). In general, only generated files and files used to customize the provider (such as images) live in the files directory.
users/USER/A directory that stores the public keys of the sysadmin with name USER. This person will have root access to all the servers.
+ + +

Leapfile

+ +

A Leapfile defines options for the leap command and lives at the root of your provider directory. Leapfile is evaluated as ruby, so you can include whatever weird logic you want in this file. In particular, there are several variables you can set that modify the behavior of leap. For example:

+ +
@platform_directory_path = '../leap_platform'
+@log = '/var/log/leap.log'
+
+ +

Additionally, you can create a ~/.leaprc file that is loaded after Leapfile and is evaluated the same way.

+ +

Platform options:

+ +
    +
  • @platform_directory_path (required). This must be set to the path where leap_platform lives. The path may be relative.
    • +
+ + +

Vagrant options:

+ +
    +
  • @vagrant_provider. Changes the default vagrant provider (“virtualbox”). For example, @vagrant_provider = "libvirt".
    • +
  • @vagrant_network. Allows you to override the default network used for local nodes. It should include a netmask like @vagrant_network = '10.0.0.0/24'.
    • +
  • @custom_vagrant_vm_line. Insert arbitrary text into the auto-generated Vagrantfile. For example, @custom_vagrant_vm_line = "config.vm.boot_mode = :gui".
    • +
  • @vagrant_basebox allows specifying a different basebox as the default one. For example, @vagrant_basebox = "LEAP/jessie".
    • +
+ + +

Logging options:

+ +
    +
  • @log. If set, all command invocation and results are logged to the specified file. This is the same as the switch --log FILE, except that the command line switch will override the value in the Leapfile.
    • +
+ + +

JSON format

+ +

All configuration files, other than Leapfile, are in the JSON format. For example:

+ +
{
+  "key1": "value1",
+  "key2": "value2"
+}
+
+ +

Keys should match /[a-z0-9_]/ and must be in double quotes.

+ +

Unlike traditional JSON, comments are allowed. If the first non-whitespace characters are // then the line is treated as a comment.

+ +
// this is a comment
+{
+  // this is a comment
+  "key": "value"  // this is an error
+}
+
+ +

Options in the configuration files might be nested hashes, arrays, numbers, strings, or boolean. Numbers and boolean values should not be quoted. For example:

+ +
{
+  "openvpn": {
+    "ip_address": "1.1.1.1",
+    "protocols": ["tcp", "udp"],
+    "ports": [80, 53],
+    "options": {
+      "public_ip": false,
+      "adblock": true
+    }
+  }
+}
+
+ +

If the value string is prefixed with an ‘=’ character, the result is evaluated as ruby. For example:

+ +
{
+  "domain": {
+    "public": "domain.org"
+  }
+  "api_domain": "= 'api.' + domain.public"
+}
+
+ +

In this case, the property “api_domain” will be set to “api.domain.org”. So long as you do not create unresolvable circular dependencies, you can reference other properties in evaluated ruby that are themselves evaluated ruby.

+ +

See “Macros” below for information on the special macros available to the evaluated ruby.

+ +

TIP: In rare cases, you might want to force the evaluation of a value to happen in a later pass after most of the other properties have been evaluated. To do this, prefix the value string with “=>” instead of “=”.

+ +

Node inheritance

+ +

Every node inherits from common.json and also any of the services or tags attached to the node. Additionally, the leap_platform contains a directory provider_base that defines the default values for tags, services and common.json.

+ +

Suppose you have a node configuration for bitmask/nodes/willamette.json like so:

+ +
{
+   "services": "webapp",
+   "tags": ["production", "northwest-us"],
+   "ip_address": "1.1.1.1"
+}
+
+ +

This node will have hostname “willamette” and it will inherit from the following files (in this order):

+ +
    +
  1. common.json + +
      +
    • load defaults: provider_base/common.json
      • +
    • load provider: bitmask/common.json
      • +
    +
    2. +
  2. service “webapp” + +
      +
    • load defaults: provider_base/services/webapp.json
      • +
    • load provider: bitmask/services/webapp.json
      • +
    +
    3. +
  3. tag “production” + +
      +
    • load defaults: provider_base/tags/production.json
      • +
    • load provider: bitmask/tags/production.json
      • +
    +
    4. +
  4. tag “northwest-us” + +
      +
    • load: bitmask/tags/northwest-us.json
      • +
    +
    5. +
  5. finally, load node “willamette” + +
      +
    • load: bitmask/nodes/willamette.json
      • +
    +
    6. +
+ + +

The provider_base directory is under the leap_platform specified in the file Leapfile.

+ +

To see all the variables a node has inherited, you could run leap inspect willamette.

+ +

Inheritance rules

+ +

Suppose you have a node configuration mynode.json:

+ +
{
+  "tags": "production",
+  "simple_value": 100,
+  "replaced_array": ["dolphin", "kangaroo"],
+  "+add_array": ["red", "black"],
+  "-subtract_array": ["bitter"],
+  "converted_to_array": "not_array_element",
+  "!override": ["insist on this value"],
+  "hash": {
+    "key1": 1,
+    "key2": 2
+  }
+}
+
+ +

And a file tags/production.json:

+ +
{
+  "simple_value": 99999,
+  "replaced_array": ["zebra"],
+  "add_array": ["green],
+  "subtract_array": ["bitter", "sweet", "salty"],
+  "converted_to_array": ["array_element"],
+  "override": "this value will be overridden",
+  "hash": {
+    "key1": "one"
+  }
+}
+
+ +

In this scenario, mynode.json will inherit from production.json. The output of this inheritance will be:

+ +
{
+  "tags": "production",
+  "simple_value": 100,
+  "replaced_array": ["dolphin", "kangaroo"],
+  "add_array": ["red", "black", "green"],
+  "subtract_array": ["sweet", "salty"],
+  "converted_to_array": ["not_array_element", "array_element"],
+  "override": ["insist on this value"],
+  "hash": {
+    "key1": 1,
+    "key2": 2
+  }
+
+ +

The rules for inheritance (where ‘old’ refers to the parent, and ‘new’ refers to the child):

+ +
    +
  • Simple values (strings, numbers, boolean): + +
      +
    • Replace the old value with the new value.
      • +
    +
    • +
  • Array values: + +
      +
    • Two arrays: replace the old array with the new array.
      • +
    • One array and one simple value: add the simple value to the array.
      • +
    • If property name is prefixed with “+”: merge the old and new arrays.
      • +
    • If property name is prefixed with “-”: subtract new array from old array.
      • +
    +
    • +
  • Hash values: + +
      +
    • Hashes are always merged (the result includes the keys of both hashes). If there is a key in common, the new one overrides the old one.
      • +
    +
    • +
  • Mismatch: + +
      +
    • Although you can mix arrays and simple values, you cannot mix arrays with hashes or hashes with simple values. If you attempt to do so, it will fail to compile and give you an error message.
      • +
    +
    • +
  • Override: + +
      +
    • If property name is prefixed with “!”: then ensure that new value is always used, regardless of old value. In this case, the override takes precedence over type checking, so you will never get a type mismatch.
      • +
    +
    • +
+ + +

NOTE: special property name prefixes, like “+”, “-”, or “!”, are not included in the property name. These prefixes determine the merge strategy, but are stripped out when compiling the resulting JSON file.

+ +

Common configuration options

+ +

You can use the command leap inspect to see what options are available for a provider, node, service, or tag configuration. For example:

+ +
    +
  • leap inspect common – show the options inherited by all nodes.
    • +
  • leap inspect --base common – show the common.json from provider_base without the local common.json inheritance applied.
    • +
  • leap inspect webapp – show all the options available for the service webapp.
    • +
+ + +

Here are some of the more important options you should be aware of:

+ +
    +
  • ip_address – Required for all nodes, no default.
    • +
  • ssh.port – The SSH port you want the node’s OpenSSH server to bind to. This is also the default when trying to connect to a node, but if the node currently has OpenSSH running on a different port then run deploy with --port to override the ssh.port configuration value.
    • +
  • mosh.enabled – If set to true, then mosh will be installed on the server. The default is false.
    • +
+ + +

Macros

+ +

When using evaluated ruby in a JSON configuration file, there are several special macros that are available. These are evaluated in the context of a node (available as the variable self).

+ +

The following methods are available to the evaluated ruby:

+ +

variable.variable

+ +

Any variable defined or inherited by a particular node configuration is available by just referencing it using either hash notation or object field notation (e.g. ['domain']['public'] or domain.public). Circular references are not allowed, but otherwise it is OK to nest evaluated values in other evaluated values. If a value has not been defined, the hash notation will return nil but the field notation will raise an exception. Properties of services, tags, and the global provider can all be referenced the same way. For example, global.services['openvpn'].x509.dh.

+ +

nodes

+ +

A hash of all nodes. This list can be filtered.

+ +

nodes_like_me

+ +

A hash of nodes that have the same deployment tags as the current node (e.g. ‘production’ or ‘local’).

+ +

global.services

+ +

A hash of all services, e.g. global.services['openvpn'] would return the “openvpn” service.

+ +

global.tags

+ +

A hash of all tags, e.g. global.tags['production'] would return the “production” tag.

+ +

global.provider

+ +

Can be used to access variables defined in provider.json, e.g. global.provider.contacts.default.

+ +

file(filename)

+ +

Inserts the full contents of the file. If the file is an erb template, it is rendered. The filename can either be one of the pre-defined file symbols, or it can be a path relative to the “files” directory in your provider instance. E.g, file :ca_cert or files 'ca/ca.crt'.

+ +

file_path(filename)

+ +

Ensures that the file will get rsynced to the node as an individual file. The value returned by file_path is the full path where this file will ultimately live when deploy to the node. e.g. file_path :ca_cert or file_path 'branding/images/logo.png'.

+ +

secret(:symbol)

+ +

Returns the value of a secret in secrets.json (or creates it if necessary). E.g. secret :couch_admin_password

+ +

hosts_file

+ +

Returns a data structure that puppet will use to generate /etc/hosts. Care is taken to use the local IP of other hosts when needed.

+ +

known_hosts_file

+ +

Returns the lines needed in a SSH known_hosts file.

+ +

stunnel_client(node_list, port, options={})

+ +

Returns a stunnel configuration data structure for the client side. Argument node_list is an ObjectList of nodes running stunnel servers. Argument port is the real port of the ultimate service running on the servers that the client wants to connect to.

+ +

stunnel_server(port)

+ +

Generates a stunnel server entry. The port is the real port targeted service.

+ +

Hash tables

+ +

The macros nodes, nodes_like_me, global.services, and global.tags all return a hash table of configuration objects (either nodes, services, or tags). There are several ways to filter and process these hash tables:

+ +

Access an element by name:

+ +
nodes['vpn1']                # returns node named 'vpn1'
+global.services['openvpn']   # returns service named 'openvpn'
+
+ +

Create a new hash table by applying filters:

+ +
nodes[:public_dns => true] # all nodes where public_dns == true
+nodes[:services => 'openvpn', 'location.country_code' => 'US'] # openvpn service OR in the US.
+nodes[[:services, 'openvpn'], [:services, 'tor']]  # services equal to openvpn OR tor
+nodes[:services => 'openvpn'][:tags => 'production']  # openvpn AND production
+nodes[:name => "!bob"] # all nodes that are NOT named "bob"
+
+ +

Create an array of values by selecting a single field:

+ +
nodes.field('location.name')
+==> ['seattle', 'istanbul']
+
+ +

Create an array of hashes by selecting multiple fields:

+ +
nodes.fields('domain.full', 'ip_address')
+==> [
+  {'domain_full' => 'red.bitmask.net', 'ip_address' => '1.1.1.1'},
+  {'domain_full' => 'blue.bitmask.net', 'ip_address' => '1.1.1.2'},
+]
+
+ +

Create a new hash table of hashes, with only certain fields:

+ +
nodes.pick_fields('domain.full', 'ip_address')
+==> {
+  "red" => {'domain_full' => 'red.bitmask.net', 'ip_address' => '1.1.1.1'},
+  "blue => {'domain_full' => 'blue.bitmask.net', 'ip_address' => '1.1.1.2'},
+}
+
+ +

With pick_fields, if there is only one field, it will generate a simple hash table:

+ +
nodes.pick_fields('ip_address')
+==> {
+  "red" => '1.1.1.1',
+  "blue => '1.1.1.2',
+}
+
+ +
+
+ + diff --git a/docs/en/guide/config/index.html b/docs/en/guide/config/index.html new file mode 100644 index 00000000..23e162d0 --- /dev/null +++ b/docs/en/guide/config/index.html @@ -0,0 +1,584 @@ + + + + +Configuration Files - LEAP Platform Documentation + + + + + + + + +
+
+

Configuration Files

+ +
Understanding and editing the configuration files.
+
+
+ + +

Files

+ +

Here are a list of some of the common files that make up a provider. Except for Leapfile and provider.json, the files are optional. Unless otherwise specified, all file names are relative to the ‘provider directory’ root (where the Leapfile is).

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
LeapfileIf present, this file tells leap that the directory is a provider directory. This file is usually empty, but can contain global options.
~/.leaprcEvaluated the same as Leapfile, but not committed to source control.
provider.jsonGlobal options related to this provider. See Provider Configuration.
provider.ENVIRONMENT.jsonGlobal options for the provider that are applied to only a single environment.
nodes/NAME.jsonThe configuration file for node called NAME.
common.jsonAll nodes inherit from this file. In other words, any options that appear in common.json will be added as default values to each node configuration, value that can be locally overridden.
services/SERVICE.jsonThe properties in this configuration file are applied to any node that includes SERVICE in its services property.
services/SERVICE.ENVIRONMENT.jsonThe properties in this configuration file are applied to any node that includes SERVICE in its services and has environment equal to ENVIRONMENT.
tags/TAG.jsonThe properties in this configuration file are applied to any node that has includes TAG in its tags property.
tags/TAG.ENVIRONMENT.jsonThe properties in this configuration file are applied to any node that has includes TAG in its tags property and has environment property equal to ENVIRONMENT.
secrets.json An automatically generated file that contains any randomly generated strings needed in order to deploy. These strings are often secret and should be protected, although any need for a random string or number that is remembered will produce another entry in this file. This file is automatically generated and refreshed each time you run leap compile or leap deploy. If an entry is no longer needed, it will get removed. If you want to change a secret, you can remove this file and have it regenerated, or remove the particular line item and just those items will be created anew.
facts.jsonIf some of your servers are running on AWS or OpenStack, you will need to discover certain properties about how networking is configured on these machines in order for a full deploy to work. In these cases, make sure to run leap facts update to periodically regenerate the facts.json file.
files/*Various static files used by the platform (e.g. keys, certificates, webapp customization, etc). In general, only generated files and files used to customize the provider (such as images) live in the files directory.
users/USER/A directory that stores the public keys of the sysadmin with name USER. This person will have root access to all the servers.
+ + +

Leapfile

+ +

A Leapfile defines options for the leap command and lives at the root of your provider directory. Leapfile is evaluated as ruby, so you can include whatever weird logic you want in this file. In particular, there are several variables you can set that modify the behavior of leap. For example:

+ +
@platform_directory_path = '../leap_platform'
+@log = '/var/log/leap.log'
+
+ +

Additionally, you can create a ~/.leaprc file that is loaded after Leapfile and is evaluated the same way.

+ +

Platform options:

+ +
    +
  • @platform_directory_path (required). This must be set to the path where leap_platform lives. The path may be relative.
    • +
+ + +

Vagrant options:

+ +
    +
  • @vagrant_provider. Changes the default vagrant provider (“virtualbox”). For example, @vagrant_provider = "libvirt".
    • +
  • @vagrant_network. Allows you to override the default network used for local nodes. It should include a netmask like @vagrant_network = '10.0.0.0/24'.
    • +
  • @custom_vagrant_vm_line. Insert arbitrary text into the auto-generated Vagrantfile. For example, @custom_vagrant_vm_line = "config.vm.boot_mode = :gui".
    • +
  • @vagrant_basebox allows specifying a different basebox as the default one. For example, @vagrant_basebox = "LEAP/jessie".
    • +
+ + +

Logging options:

+ +
    +
  • @log. If set, all command invocation and results are logged to the specified file. This is the same as the switch --log FILE, except that the command line switch will override the value in the Leapfile.
    • +
+ + +

JSON format

+ +

All configuration files, other than Leapfile, are in the JSON format. For example:

+ +
{
+  "key1": "value1",
+  "key2": "value2"
+}
+
+ +

Keys should match /[a-z0-9_]/ and must be in double quotes.

+ +

Unlike traditional JSON, comments are allowed. If the first non-whitespace characters are // then the line is treated as a comment.

+ +
// this is a comment
+{
+  // this is a comment
+  "key": "value"  // this is an error
+}
+
+ +

Options in the configuration files might be nested hashes, arrays, numbers, strings, or boolean. Numbers and boolean values should not be quoted. For example:

+ +
{
+  "openvpn": {
+    "ip_address": "1.1.1.1",
+    "protocols": ["tcp", "udp"],
+    "ports": [80, 53],
+    "options": {
+      "public_ip": false,
+      "adblock": true
+    }
+  }
+}
+
+ +

If the value string is prefixed with an ‘=’ character, the result is evaluated as ruby. For example:

+ +
{
+  "domain": {
+    "public": "domain.org"
+  }
+  "api_domain": "= 'api.' + domain.public"
+}
+
+ +

In this case, the property “api_domain” will be set to “api.domain.org”. So long as you do not create unresolvable circular dependencies, you can reference other properties in evaluated ruby that are themselves evaluated ruby.

+ +

See “Macros” below for information on the special macros available to the evaluated ruby.

+ +

TIP: In rare cases, you might want to force the evaluation of a value to happen in a later pass after most of the other properties have been evaluated. To do this, prefix the value string with “=>” instead of “=”.

+ +

Node inheritance

+ +

Every node inherits from common.json and also any of the services or tags attached to the node. Additionally, the leap_platform contains a directory provider_base that defines the default values for tags, services and common.json.

+ +

Suppose you have a node configuration for bitmask/nodes/willamette.json like so:

+ +
{
+   "services": "webapp",
+   "tags": ["production", "northwest-us"],
+   "ip_address": "1.1.1.1"
+}
+
+ +

This node will have hostname “willamette” and it will inherit from the following files (in this order):

+ +
    +
  1. common.json + +
      +
    • load defaults: provider_base/common.json
      • +
    • load provider: bitmask/common.json
      • +
    +
    2. +
  2. service “webapp” + +
      +
    • load defaults: provider_base/services/webapp.json
      • +
    • load provider: bitmask/services/webapp.json
      • +
    +
    3. +
  3. tag “production” + +
      +
    • load defaults: provider_base/tags/production.json
      • +
    • load provider: bitmask/tags/production.json
      • +
    +
    4. +
  4. tag “northwest-us” + +
      +
    • load: bitmask/tags/northwest-us.json
      • +
    +
    5. +
  5. finally, load node “willamette” + +
      +
    • load: bitmask/nodes/willamette.json
      • +
    +
    6. +
+ + +

The provider_base directory is under the leap_platform specified in the file Leapfile.

+ +

To see all the variables a node has inherited, you could run leap inspect willamette.

+ +

Inheritance rules

+ +

Suppose you have a node configuration mynode.json:

+ +
{
+  "tags": "production",
+  "simple_value": 100,
+  "replaced_array": ["dolphin", "kangaroo"],
+  "+add_array": ["red", "black"],
+  "-subtract_array": ["bitter"],
+  "converted_to_array": "not_array_element",
+  "!override": ["insist on this value"],
+  "hash": {
+    "key1": 1,
+    "key2": 2
+  }
+}
+
+ +

And a file tags/production.json:

+ +
{
+  "simple_value": 99999,
+  "replaced_array": ["zebra"],
+  "add_array": ["green],
+  "subtract_array": ["bitter", "sweet", "salty"],
+  "converted_to_array": ["array_element"],
+  "override": "this value will be overridden",
+  "hash": {
+    "key1": "one"
+  }
+}
+
+ +

In this scenario, mynode.json will inherit from production.json. The output of this inheritance will be:

+ +
{
+  "tags": "production",
+  "simple_value": 100,
+  "replaced_array": ["dolphin", "kangaroo"],
+  "add_array": ["red", "black", "green"],
+  "subtract_array": ["sweet", "salty"],
+  "converted_to_array": ["not_array_element", "array_element"],
+  "override": ["insist on this value"],
+  "hash": {
+    "key1": 1,
+    "key2": 2
+  }
+
+ +

The rules for inheritance (where ‘old’ refers to the parent, and ‘new’ refers to the child):

+ +
    +
  • Simple values (strings, numbers, boolean): + +
      +
    • Replace the old value with the new value.
      • +
    +
    • +
  • Array values: + +
      +
    • Two arrays: replace the old array with the new array.
      • +
    • One array and one simple value: add the simple value to the array.
      • +
    • If property name is prefixed with “+”: merge the old and new arrays.
      • +
    • If property name is prefixed with “-”: subtract new array from old array.
      • +
    +
    • +
  • Hash values: + +
      +
    • Hashes are always merged (the result includes the keys of both hashes). If there is a key in common, the new one overrides the old one.
      • +
    +
    • +
  • Mismatch: + +
      +
    • Although you can mix arrays and simple values, you cannot mix arrays with hashes or hashes with simple values. If you attempt to do so, it will fail to compile and give you an error message.
      • +
    +
    • +
  • Override: + +
      +
    • If property name is prefixed with “!”: then ensure that new value is always used, regardless of old value. In this case, the override takes precedence over type checking, so you will never get a type mismatch.
      • +
    +
    • +
+ + +

NOTE: special property name prefixes, like “+”, “-”, or “!”, are not included in the property name. These prefixes determine the merge strategy, but are stripped out when compiling the resulting JSON file.

+ +

Common configuration options

+ +

You can use the command leap inspect to see what options are available for a provider, node, service, or tag configuration. For example:

+ +
    +
  • leap inspect common – show the options inherited by all nodes.
    • +
  • leap inspect --base common – show the common.json from provider_base without the local common.json inheritance applied.
    • +
  • leap inspect webapp – show all the options available for the service webapp.
    • +
+ + +

Here are some of the more important options you should be aware of:

+ +
    +
  • ip_address – Required for all nodes, no default.
    • +
  • ssh.port – The SSH port you want the node’s OpenSSH server to bind to. This is also the default when trying to connect to a node, but if the node currently has OpenSSH running on a different port then run deploy with --port to override the ssh.port configuration value.
    • +
  • mosh.enabled – If set to true, then mosh will be installed on the server. The default is false.
    • +
+ + +

Macros

+ +

When using evaluated ruby in a JSON configuration file, there are several special macros that are available. These are evaluated in the context of a node (available as the variable self).

+ +

The following methods are available to the evaluated ruby:

+ +

variable.variable

+ +

Any variable defined or inherited by a particular node configuration is available by just referencing it using either hash notation or object field notation (e.g. ['domain']['public'] or domain.public). Circular references are not allowed, but otherwise it is OK to nest evaluated values in other evaluated values. If a value has not been defined, the hash notation will return nil but the field notation will raise an exception. Properties of services, tags, and the global provider can all be referenced the same way. For example, global.services['openvpn'].x509.dh.

+ +

nodes

+ +

A hash of all nodes. This list can be filtered.

+ +

nodes_like_me

+ +

A hash of nodes that have the same deployment tags as the current node (e.g. ‘production’ or ‘local’).

+ +

global.services

+ +

A hash of all services, e.g. global.services['openvpn'] would return the “openvpn” service.

+ +

global.tags

+ +

A hash of all tags, e.g. global.tags['production'] would return the “production” tag.

+ +

global.provider

+ +

Can be used to access variables defined in provider.json, e.g. global.provider.contacts.default.

+ +

file(filename)

+ +

Inserts the full contents of the file. If the file is an erb template, it is rendered. The filename can either be one of the pre-defined file symbols, or it can be a path relative to the “files” directory in your provider instance. E.g, file :ca_cert or files 'ca/ca.crt'.

+ +

file_path(filename)

+ +

Ensures that the file will get rsynced to the node as an individual file. The value returned by file_path is the full path where this file will ultimately live when deploy to the node. e.g. file_path :ca_cert or file_path 'branding/images/logo.png'.

+ +

secret(:symbol)

+ +

Returns the value of a secret in secrets.json (or creates it if necessary). E.g. secret :couch_admin_password

+ +

hosts_file

+ +

Returns a data structure that puppet will use to generate /etc/hosts. Care is taken to use the local IP of other hosts when needed.

+ +

known_hosts_file

+ +

Returns the lines needed in a SSH known_hosts file.

+ +

stunnel_client(node_list, port, options={})

+ +

Returns a stunnel configuration data structure for the client side. Argument node_list is an ObjectList of nodes running stunnel servers. Argument port is the real port of the ultimate service running on the servers that the client wants to connect to.

+ +

stunnel_server(port)

+ +

Generates a stunnel server entry. The port is the real port targeted service.

+ +

Hash tables

+ +

The macros nodes, nodes_like_me, global.services, and global.tags all return a hash table of configuration objects (either nodes, services, or tags). There are several ways to filter and process these hash tables:

+ +

Access an element by name:

+ +
nodes['vpn1']                # returns node named 'vpn1'
+global.services['openvpn']   # returns service named 'openvpn'
+
+ +

Create a new hash table by applying filters:

+ +
nodes[:public_dns => true] # all nodes where public_dns == true
+nodes[:services => 'openvpn', 'location.country_code' => 'US'] # openvpn service OR in the US.
+nodes[[:services, 'openvpn'], [:services, 'tor']]  # services equal to openvpn OR tor
+nodes[:services => 'openvpn'][:tags => 'production']  # openvpn AND production
+nodes[:name => "!bob"] # all nodes that are NOT named "bob"
+
+ +

Create an array of values by selecting a single field:

+ +
nodes.field('location.name')
+==> ['seattle', 'istanbul']
+
+ +

Create an array of hashes by selecting multiple fields:

+ +
nodes.fields('domain.full', 'ip_address')
+==> [
+  {'domain_full' => 'red.bitmask.net', 'ip_address' => '1.1.1.1'},
+  {'domain_full' => 'blue.bitmask.net', 'ip_address' => '1.1.1.2'},
+]
+
+ +

Create a new hash table of hashes, with only certain fields:

+ +
nodes.pick_fields('domain.full', 'ip_address')
+==> {
+  "red" => {'domain_full' => 'red.bitmask.net', 'ip_address' => '1.1.1.1'},
+  "blue => {'domain_full' => 'blue.bitmask.net', 'ip_address' => '1.1.1.2'},
+}
+
+ +

With pick_fields, if there is only one field, it will generate a simple hash table:

+ +
nodes.pick_fields('ip_address')
+==> {
+  "red" => '1.1.1.1',
+  "blue => '1.1.1.2',
+}
+
+ +
+
+ + diff --git a/docs/en/guide/domains.html b/docs/en/guide/domains.html new file mode 100644 index 00000000..eb3331ff --- /dev/null +++ b/docs/en/guide/domains.html @@ -0,0 +1,298 @@ + + + + +Domains - LEAP Platform Documentation + + + + + + + + +
+
+

Domains

+ +
How to handle domain names and integrating LEAP with existing services.
+
+
+ + +

Overview

+ +

Deploying LEAP can start to get very tricky when you need to integrate LEAP services with an existing domain that you already use or which already has users. Most of this complexity is unavoidable, although there are a few things we plan to do in the future to make this a little less painful.

+ +

Because integration with legacy systems is an advanced topic, we recommend that you begin with a new domain. Once everything works and you are comfortable with your LEAP-powered infrastructure, you can then contemplate integrating with your existing domain.

+ +

Definitions

+ +

provider domain

+ +

This is the main domain used to identify the provider. The provider domain is what the user enters in the Bitmask client. e.g. example.org. The full host name of every node in your provider infrastructure will use the provider domain (e.g. dbnode.example.org).

+ +

In order for the Bitmask client to get configured for use with a provider, it must be able to find the provider.json bootstrap file at the root of the provider domain. This is not needed if the Bitmask client is “pre-seeded” with the provider’s information (these providers show up in a the initial list of available providers).

+ +

webapp domain

+ +

This is the domain that runs the leap_web application that allows users to register accounts, create help tickets, etc. e.g. example.org or user.example.org. The webapp domain defaults to the provider domain unless it is explicitly configured separately.

+ +

API domain

+ +

This is the domain that the provider API runs on. Typically, this is set automatically and you never need to configure it. The user should never be aware of this domain. e.g. api.example.org. The Bitmask client discovers this API domain by reading it from the provider.json file it grabs from the provider domain.

+ +

mail domain

+ +

This is the domain used for mail accounts, e.g. username@example.org. Currently, this is always the provider domain, but it may be independently configurable in the future.

+ +

Generating a zone file

+ +

Currently, the platform does not include a dedicated dns service type, so you need to have your own setup for DNS. You can generate the appropriate configuration options with this command:

+ +
leap compile zone
+
+ +

A single domain

+ +

The easy approach is to use a single domain for provider domain, webapp domain, and email domain. This will install the webapp on the provider domain, which means that this domain must be a new one that you are not currently using for anything.

+ +

To configure a single domain, just set the domain in provider.json:

+ +
{
+  "domain": "example.org"
+}
+
+ +

If you have multiple environments, you can specify a different provider domain for each environment. For example:

+ +

provider.staging.json

+ +
{
+  "domain": "staging.example.org"
+}
+
+ +

A separate domain for the webapp

+ +

It is possible make the webapp domain different than the provider domain. This is needed if you already have a website running at your provider domain.

+ +

In order to put webapp on a different domain, you must take two steps:

+ +
    +
  1. You must configure webapp.domain for nodes with the webapp service.
    2. +
  2. You must make the compiled provider.json available at the root of the provider domain.
    3. +
+ + +

NOTE: This compiled provider.json is different than the provider.json that you edit and lives in the root of the provider directory.

+ +

Step 1. Configuring webapp.domain

+ +

In services/webapp.json:

+ +
{
+  "webapp": {
+    "domain": "user.example.org"
+  }
+}
+
+ +

Step 2. Putting the compiled provider.json in place

+ +

Generate the compiled provider.json:

+ +
leap compile provider.json
+= created files/web/bootstrap/
+= created files/web/bootstrap/README
+= created files/web/bootstrap/production/
+= created files/web/bootstrap/production/provider.json
+= created files/web/bootstrap/production/htaccess
+= created files/web/bootstrap/staging/
+= created files/web/bootstrap/staging/provider.json
+= created files/web/bootstrap/staging/htaccess
+
+ +

This command compiles a separate provider.json for each environment, or “default” if you don’t have an environment. In the example above, there is an environment called “production” and one called “staging”, but your setup will probably differ.

+ +

The resulting provider.json file must then be put at the root URL of your provider domain for the appropriate environment.

+ +

There is one additional complication: currently, the Bitmask client tests for compatibility using some HTTP headers on the /provider.json response. This is will hopefully change in the future, but for now you need to ensure the right headers are set in the response. The included file htaccess has example directives for Apache, if that is what you use.

+ +

This step can be skipped if you happen to use the static service to deploy an amber powered static website to provider domain. In this case, the correct provider.json will be automatically put into place.

+ +

Integrating with existing email system

+ +

If your mail domain already has users from a legacy email system, then things get a bit complicated. In order to be able to support both LEAP-powered email and legacy email on the same domain, you need to follow these steps:

+ +
    +
  1. Modify the LEAP webapp so that it does not create users with the same name as users in the legacy system.
    2. +
  2. Configure your legacy MX servers to forward mail that they cannot handle to the LEAP MX servers, or vice versa.
    3. +
+ + +

Step 1. Modify LEAP webapp

+ +

In order to modify the webapp to respect the usernames already reserved by your legacy system, you need to modify the LEAP webapp code. The easiest way to do this is to create a custom gem that modifies the behavior of the webapp.

+ +

For this example, we will call our custom gem reserve_usernames.

+ +

This gem can live in one of two places:

+ +

(1) You can fork the project leap_web and put the gem in leap_web/vendor/gems/reserve_usernames. Then, modify Gemfile and add the line gem 'common_languages', :path => 'vendor/gems/reserve_usernames'

+ +

(2) Alternately, you can put the gem in the local provider directory files/webapp/gems/reserve_username. This will get synced to the webapp servers when you deploy and put in /srv/leap/webapp/config/customization where it will get automatically loaded by the webapp.

+ +

What should the gem reserve_usernames look like? There is an example available here: https://leap.se/git/reserved_usernames.git

+ +

This example gem uses ActiveResource to communicate with a remote REST API for creating and checking username reservations. This ensures that both the legacy system and the LEAP system use the same namespace. Alternately, you could write a gem that checks the legacy database directly.

+ +

Step 2. Configure MX servers

+ +

To be written.

+ +
+
+ + diff --git a/docs/en/guide/domains/index.html b/docs/en/guide/domains/index.html new file mode 100644 index 00000000..9ebf3b2c --- /dev/null +++ b/docs/en/guide/domains/index.html @@ -0,0 +1,298 @@ + + + + +Domains - LEAP Platform Documentation + + + + + + + + +
+
+

Domains

+ +
How to handle domain names and integrating LEAP with existing services.
+
+
+ + +

Overview

+ +

Deploying LEAP can start to get very tricky when you need to integrate LEAP services with an existing domain that you already use or which already has users. Most of this complexity is unavoidable, although there are a few things we plan to do in the future to make this a little less painful.

+ +

Because integration with legacy systems is an advanced topic, we recommend that you begin with a new domain. Once everything works and you are comfortable with your LEAP-powered infrastructure, you can then contemplate integrating with your existing domain.

+ +

Definitions

+ +

provider domain

+ +

This is the main domain used to identify the provider. The provider domain is what the user enters in the Bitmask client. e.g. example.org. The full host name of every node in your provider infrastructure will use the provider domain (e.g. dbnode.example.org).

+ +

In order for the Bitmask client to get configured for use with a provider, it must be able to find the provider.json bootstrap file at the root of the provider domain. This is not needed if the Bitmask client is “pre-seeded” with the provider’s information (these providers show up in a the initial list of available providers).

+ +

webapp domain

+ +

This is the domain that runs the leap_web application that allows users to register accounts, create help tickets, etc. e.g. example.org or user.example.org. The webapp domain defaults to the provider domain unless it is explicitly configured separately.

+ +

API domain

+ +

This is the domain that the provider API runs on. Typically, this is set automatically and you never need to configure it. The user should never be aware of this domain. e.g. api.example.org. The Bitmask client discovers this API domain by reading it from the provider.json file it grabs from the provider domain.

+ +

mail domain

+ +

This is the domain used for mail accounts, e.g. username@example.org. Currently, this is always the provider domain, but it may be independently configurable in the future.

+ +

Generating a zone file

+ +

Currently, the platform does not include a dedicated dns service type, so you need to have your own setup for DNS. You can generate the appropriate configuration options with this command:

+ +
leap compile zone
+
+ +

A single domain

+ +

The easy approach is to use a single domain for provider domain, webapp domain, and email domain. This will install the webapp on the provider domain, which means that this domain must be a new one that you are not currently using for anything.

+ +

To configure a single domain, just set the domain in provider.json:

+ +
{
+  "domain": "example.org"
+}
+
+ +

If you have multiple environments, you can specify a different provider domain for each environment. For example:

+ +

provider.staging.json

+ +
{
+  "domain": "staging.example.org"
+}
+
+ +

A separate domain for the webapp

+ +

It is possible make the webapp domain different than the provider domain. This is needed if you already have a website running at your provider domain.

+ +

In order to put webapp on a different domain, you must take two steps:

+ +
    +
  1. You must configure webapp.domain for nodes with the webapp service.
    2. +
  2. You must make the compiled provider.json available at the root of the provider domain.
    3. +
+ + +

NOTE: This compiled provider.json is different than the provider.json that you edit and lives in the root of the provider directory.

+ +

Step 1. Configuring webapp.domain

+ +

In services/webapp.json:

+ +
{
+  "webapp": {
+    "domain": "user.example.org"
+  }
+}
+
+ +

Step 2. Putting the compiled provider.json in place

+ +

Generate the compiled provider.json:

+ +
leap compile provider.json
+= created files/web/bootstrap/
+= created files/web/bootstrap/README
+= created files/web/bootstrap/production/
+= created files/web/bootstrap/production/provider.json
+= created files/web/bootstrap/production/htaccess
+= created files/web/bootstrap/staging/
+= created files/web/bootstrap/staging/provider.json
+= created files/web/bootstrap/staging/htaccess
+
+ +

This command compiles a separate provider.json for each environment, or “default” if you don’t have an environment. In the example above, there is an environment called “production” and one called “staging”, but your setup will probably differ.

+ +

The resulting provider.json file must then be put at the root URL of your provider domain for the appropriate environment.

+ +

There is one additional complication: currently, the Bitmask client tests for compatibility using some HTTP headers on the /provider.json response. This is will hopefully change in the future, but for now you need to ensure the right headers are set in the response. The included file htaccess has example directives for Apache, if that is what you use.

+ +

This step can be skipped if you happen to use the static service to deploy an amber powered static website to provider domain. In this case, the correct provider.json will be automatically put into place.

+ +

Integrating with existing email system

+ +

If your mail domain already has users from a legacy email system, then things get a bit complicated. In order to be able to support both LEAP-powered email and legacy email on the same domain, you need to follow these steps:

+ +
    +
  1. Modify the LEAP webapp so that it does not create users with the same name as users in the legacy system.
    2. +
  2. Configure your legacy MX servers to forward mail that they cannot handle to the LEAP MX servers, or vice versa.
    3. +
+ + +

Step 1. Modify LEAP webapp

+ +

In order to modify the webapp to respect the usernames already reserved by your legacy system, you need to modify the LEAP webapp code. The easiest way to do this is to create a custom gem that modifies the behavior of the webapp.

+ +

For this example, we will call our custom gem reserve_usernames.

+ +

This gem can live in one of two places:

+ +

(1) You can fork the project leap_web and put the gem in leap_web/vendor/gems/reserve_usernames. Then, modify Gemfile and add the line gem 'common_languages', :path => 'vendor/gems/reserve_usernames'

+ +

(2) Alternately, you can put the gem in the local provider directory files/webapp/gems/reserve_username. This will get synced to the webapp servers when you deploy and put in /srv/leap/webapp/config/customization where it will get automatically loaded by the webapp.

+ +

What should the gem reserve_usernames look like? There is an example available here: https://leap.se/git/reserved_usernames.git

+ +

This example gem uses ActiveResource to communicate with a remote REST API for creating and checking username reservations. This ensures that both the legacy system and the LEAP system use the same namespace. Alternately, you could write a gem that checks the legacy database directly.

+ +

Step 2. Configure MX servers

+ +

To be written.

+ +
+
+ + diff --git a/docs/en/guide/environments.html b/docs/en/guide/environments.html new file mode 100644 index 00000000..db82302d --- /dev/null +++ b/docs/en/guide/environments.html @@ -0,0 +1,228 @@ + + + + +Environments - LEAP Platform Documentation + + + + + + + + +
+
+

Working with environments

+ +
How to partition the nodes into separate environments.
+
+
+ + +

With environments, you can divide your nodes into different and entirely separate sets. For example, you might have sets of nodes for ‘testing’, ‘staging’ and ‘production’.

+ +

Typically, the nodes in one environment are totally isolated from the nodes in a different environment. Each environment will have its own separate database, for example.

+ +

There are a few exceptions to this rule: backup nodes, for example, will by default attempt to back up data from all the environments (excluding local).

+ +

Assign an environment

+ +

To assign an environment to a node, you just set the environment node property. This is typically done with tags, although it is not necessary. For example:

+ +

tags/production.json

+ +
{
+  "environment": "production"
+}
+
+ +

nodes/mynode.json

+ +
{
+  "tags": ["production"]
+}
+
+ +

There are several built-in tags that will apply a value for the environment:

+ +
    +
  • production: An environment for nodes that are in use by end users.
    • +
  • development: An environment to be used for nodes that are being used for experiments or staging.
    • +
  • local: This environment gets automatically applied to all nodes that run only on local VMs. Nodes with a local environment are treated special and excluded from certain calculations.
    • +
+ + +

You don’t need to use these and you can add your own.

+ +

Environment commands

+ +
    +
  • leap env – List the available environments and disply which one is active.
    • +
  • leap env pin ENV – Pin the current environment to ENV.
    • +
  • leap env unpin – Remove the environment pin.
    • +
+ + +

The environment pin is only active for your local machine: it is not recorded in the provider directory and not shared with other users.

+ +

Environment specific JSON files

+ +

You can add JSON configuration files that are only applied when a specific environment is active. For example, if you create a file provider.production.json, these values will only get applied to the provider.json file for the production environment.

+ +

This will also work for services and tags. For example:

+ +
provider.local.json
+services/webapp.development.json
+tags/seattle.production.json
+
+ +

In this example, local, development, and production are the names of environments.

+ +

Bind an environment to a Platform version

+ +

If you want to ensure that a particular environment is bound to a particular version of the LEAP Platform, you can add a platform section to the provider.ENV.json file (where ENV is the name of the environment in question).

+ +

The available options are platform.version, platform.branch, or platform.commit. For example:

+ +
{
+  "platform": {
+    "version": "1.6.1",
+    "branch": "develop",
+    "commit": "5df867fbd3a78ca4160eb54d708d55a7d047bdb2"
+  }
+}
+
+ +

You can use any combination of version, branch, and commit to specify the binding. The values for branch and commit only work if the leap_platform directory is a git repository.

+ +

The value for commit is passed directly through to git log to query for a list of acceptable commits. See man gitrevisions to see how to specify ranges. For example:

+ +
    +
  • HEAD^..HEAD - current commit must be head of the branch.
    • +
  • 3172444652af71bd771609d6b80258e70cc82ce9..HEAD - current commit must be after 3172444652af71bd771609d6b80258e70cc82ce9.
    • +
  • refs/tags/0.6.0rc1..refs/tags/0.6.0rc2 - current commit must be after tag 0.6.0rc1 and before or including tag 0.6.0rc2.
    • +
+ + +
+
+ + diff --git a/docs/en/guide/environments/index.html b/docs/en/guide/environments/index.html new file mode 100644 index 00000000..faeb6c6c --- /dev/null +++ b/docs/en/guide/environments/index.html @@ -0,0 +1,228 @@ + + + + +Environments - LEAP Platform Documentation + + + + + + + + +
+
+

Working with environments

+ +
How to partition the nodes into separate environments.
+
+
+ + +

With environments, you can divide your nodes into different and entirely separate sets. For example, you might have sets of nodes for ‘testing’, ‘staging’ and ‘production’.

+ +

Typically, the nodes in one environment are totally isolated from the nodes in a different environment. Each environment will have its own separate database, for example.

+ +

There are a few exceptions to this rule: backup nodes, for example, will by default attempt to back up data from all the environments (excluding local).

+ +

Assign an environment

+ +

To assign an environment to a node, you just set the environment node property. This is typically done with tags, although it is not necessary. For example:

+ +

tags/production.json

+ +
{
+  "environment": "production"
+}
+
+ +

nodes/mynode.json

+ +
{
+  "tags": ["production"]
+}
+
+ +

There are several built-in tags that will apply a value for the environment:

+ +
    +
  • production: An environment for nodes that are in use by end users.
    • +
  • development: An environment to be used for nodes that are being used for experiments or staging.
    • +
  • local: This environment gets automatically applied to all nodes that run only on local VMs. Nodes with a local environment are treated special and excluded from certain calculations.
    • +
+ + +

You don’t need to use these and you can add your own.

+ +

Environment commands

+ +
    +
  • leap env – List the available environments and disply which one is active.
    • +
  • leap env pin ENV – Pin the current environment to ENV.
    • +
  • leap env unpin – Remove the environment pin.
    • +
+ + +

The environment pin is only active for your local machine: it is not recorded in the provider directory and not shared with other users.

+ +

Environment specific JSON files

+ +

You can add JSON configuration files that are only applied when a specific environment is active. For example, if you create a file provider.production.json, these values will only get applied to the provider.json file for the production environment.

+ +

This will also work for services and tags. For example:

+ +
provider.local.json
+services/webapp.development.json
+tags/seattle.production.json
+
+ +

In this example, local, development, and production are the names of environments.

+ +

Bind an environment to a Platform version

+ +

If you want to ensure that a particular environment is bound to a particular version of the LEAP Platform, you can add a platform section to the provider.ENV.json file (where ENV is the name of the environment in question).

+ +

The available options are platform.version, platform.branch, or platform.commit. For example:

+ +
{
+  "platform": {
+    "version": "1.6.1",
+    "branch": "develop",
+    "commit": "5df867fbd3a78ca4160eb54d708d55a7d047bdb2"
+  }
+}
+
+ +

You can use any combination of version, branch, and commit to specify the binding. The values for branch and commit only work if the leap_platform directory is a git repository.

+ +

The value for commit is passed directly through to git log to query for a list of acceptable commits. See man gitrevisions to see how to specify ranges. For example:

+ +
    +
  • HEAD^..HEAD - current commit must be head of the branch.
    • +
  • 3172444652af71bd771609d6b80258e70cc82ce9..HEAD - current commit must be after 3172444652af71bd771609d6b80258e70cc82ce9.
    • +
  • refs/tags/0.6.0rc1..refs/tags/0.6.0rc2 - current commit must be after tag 0.6.0rc1 and before or including tag 0.6.0rc2.
    • +
+ + +
+
+ + diff --git a/docs/en/guide/getting-started.html b/docs/en/guide/getting-started.html new file mode 100644 index 00000000..b1274e14 --- /dev/null +++ b/docs/en/guide/getting-started.html @@ -0,0 +1,317 @@ + + + + +Getting Started - LEAP Platform Documentation + + + + + + + + +
+
+

Getting Started

+ +
An overview of the LEAP Platform
+
+
+ + +

Sensitive files

+ +

Some files in your provider directory are very sensitive. Leaking these files will compromise your provider.

+ +

Super sensitive and irreplaceable:

+ +
    +
  • files/ca/*.key – the private keys for the client and server CAs.
    • +
  • files/cert/*.key – the private key(s) for the commercial certificate for your domain(s).
    • +
+ + +

Sensitive, but can be erased and regenerated automatically:

+ +
    +
  • secrets.json – various random secrets, such as passwords for databases.
    • +
  • files/nodes/*/*.key – the private key for each node.
    • +
  • hiera/*.yaml – hiera file contains a copy of the private key of the node.
    • +
+ + +

Also, each sysadmin has one or more public ssh keys in users/*/*_ssh.pub. Typically, you will want to keep these public keys secure as well.

+ +

See Keys and Certificates for more information.

+ +

Useful commands

+ +

Here are a few useful leap commands:

+ +
    +
  • leap help [COMMAND] – get help on COMMAND.
    • +
  • leap history [FILTER] – show the recent deployment history for the selected nodes.
    • +
  • leap ssh web1 – SSH into node web1 (requires leap node init web1 first).
    • +
  • leap list [FILTER] – list the selected nodes. + +
      +
    • leap list production – list only those nodes with the tag ‘production’
      • +
    • leap list --print ip_address – list a particular attribute of all nodes.
      • +
    +
    • +
+ + +

See the full Command Line Reference for more information.

+ +

Node filters

+ +

Many of the leap commands take a “node filter”. You can use a node filter to target a command at one or more nodes.

+ +

A node filter consists of one or more keywords, with an optional “+” before each keyword.

+ +
    +
  • keywords can be a node name, a service type, or a tag.
    • +
  • the “+” before the keyword constructs an AND condition
    • +
  • otherwise, multiple keywords together construct an OR condition
    • +
+ + +

Examples:

+ +
    +
  • leap list openvpn – list all nodes with service openvpn.
    • +
  • leap list openvpn +production – only nodes of service type openvpn AND tag production.
    • +
  • leap deploy webapp openvpn – deploy to all webapp OR openvpn nodes.
    • +
  • leap node init ostrich – just init the node named ostrich.
    • +
+ + +

See the full Command Line Reference for more information.

+ +

Tracking the provider directory in git

+ +

You should commit your provider changes to your favorite VCS whenever things change. This way you can share your configurations with other admins, all they have to do is to pull the changes to stay up to date. Every time you make a change to your provider, such as adding nodes, services, generating certificates, etc. you should add those to your VCS, commit them and push them to where your repository is hosted.

+ +

Note that your provider directory contains secrets, such as private key material and passwords. You do not want to have those passwords readable by the world, so make sure that wherever you are hosting your repository, it is not public for the world to read.

+ +

If you have a post-commit hook that emails the changes to contributors, you may want to exclude diffs for files that might have sensitive secrets. For example, create a .gitattributes file with:

+ +
# No diff, no email for key files
+*.key -diff
+*.pem -diff
+
+# Discard diff for secrets.json
+secrets.json -diff
+
+# No diff for hiera files, they contain passwords
+hiera/* -diff
+
+ +

Editing JSON configuration files

+ +

All the settings that compose your provider are stored in JSON files.

+ +

At a minimum, you will need at least two configuration files:

+ +
    +
  • provider.json – general settings for you provider.
    • +
  • nodes/NAME.json – configuration file for node called “NAME”.
    • +
+ + +

There are a few required properties in provider.json:

+ +
{
+  "domain": "example.org",
+  "name": "Example",
+  "contacts": {
+    "default": "email1@example.org"
+  }
+}
+
+ +

See Provider Configuration for more details.

+ +

For node configuration files, there are two required properties:

+ +
{
+  "ip_address": "1.1.1.1",
+  "services": ["openvpn"]
+}
+
+ +

See Services for details on what servers are available, and see Configuration Files details on how configuration files work.

+ +

How does it work under the hood?

+ +

You don’t need to know any of the details of what happens “under the hood” in order to use the LEAP platform. However, if you are curious as to what is going on, here is a quick primer.

+ +

First, some background terminology:

+ +
    +
  • puppet: Puppet is a system for automating deployment and management of servers (called nodes).
    • +
  • hiera files: In puppet, you can use something called a ‘hiera file’ to seed a node with a few configuration values. In LEAP, we go all out and put every configuration value needed for a node in the hiera file, and automatically compile a custom hiera file for each node.
    • +
+ + +

When you run leap deploy, a bunch of things happen, in this order:

+ +
    +
  1. Compile hiera files: The hiera configuration file for each node is compiled in YAML format and saved in the directory hiera. The source material for this hiera file consists of all the JSON configuration files imported or inherited by the node’s JSON config file.
    2. +
  2. Copy required files to node: All the files needed for puppet to run are rsync'ed to each node. This includes the entire leap_platform directory, as well as the node’s hiera file and other files needed by puppet to set up the node (keys, binary files, etc).
    3. +
  3. Puppet is run: Once the node is ready, leap connects to the node via ssh and runs puppet apply. Puppet is applied locally on the node, without a daemon or puppetmaster.
    4. +
+ + +

You can run leap -v2 deploy to see exactly what commands are being executed.

+ +

This mode of operation is fundamentally different from how puppet is normally used:

+ +
    +
  • There is no puppetmaster that all the servers take orders from, and there is no puppetd running in the background.
    • +
  • Servers cannot dynamically query the puppetmaster for information about the other servers.
    • +
  • There is a static representation for the state of every server that can be committed to git.
    • +
+ + +

There are advantages and disadvantages to the model that LEAP uses. We have found it very useful for our goal of having a common LEAP platform that many different providers can all use while still allowing providers to configure their unique infrastructure.

+ +

We also find it very beneficial to be able to track the state of your infrastructure in git.

+ +

Traditional system configuration automation systems, like Puppet or Chef, deploy changes to servers using a pull method. Each server pulls a manifest from a central master server and uses this to alter the state of the server.

+ +

Instead, the leap tool uses a masterless push method: The sysadmin runs leap deploy from the provider instance directory on their desktop machine to push the changes out to every server (or a subset of servers). LEAP still uses Puppet, but there is no central master server that each node must pull from.

+ +

One other significant difference between LEAP and typical system automation is how interactions among servers are handled. Rather than store a central database of information about each server that can be queried when a recipe is applied, the leap command compiles static representation of all the information a particular server will need in order to apply the recipes. In compiling this static representation, leap can use arbitrary programming logic to query and manipulate information about other servers.

+ +

These two approaches, masterless push and pre-compiled static configuration, allow the sysadmin to manage a set of LEAP servers using traditional software development techniques of branching and merging, to more easily create local testing environments using virtual servers, and to deploy without the added complexity and failure potential of a master server.

+ +
+
+ + diff --git a/docs/en/guide/getting-started/index.html b/docs/en/guide/getting-started/index.html new file mode 100644 index 00000000..c9457ddd --- /dev/null +++ b/docs/en/guide/getting-started/index.html @@ -0,0 +1,317 @@ + + + + +Getting Started - LEAP Platform Documentation + + + + + + + + +
+
+

Getting Started

+ +
An overview of the LEAP Platform
+
+
+ + +

Sensitive files

+ +

Some files in your provider directory are very sensitive. Leaking these files will compromise your provider.

+ +

Super sensitive and irreplaceable:

+ +
    +
  • files/ca/*.key – the private keys for the client and server CAs.
    • +
  • files/cert/*.key – the private key(s) for the commercial certificate for your domain(s).
    • +
+ + +

Sensitive, but can be erased and regenerated automatically:

+ +
    +
  • secrets.json – various random secrets, such as passwords for databases.
    • +
  • files/nodes/*/*.key – the private key for each node.
    • +
  • hiera/*.yaml – hiera file contains a copy of the private key of the node.
    • +
+ + +

Also, each sysadmin has one or more public ssh keys in users/*/*_ssh.pub. Typically, you will want to keep these public keys secure as well.

+ +

See Keys and Certificates for more information.

+ +

Useful commands

+ +

Here are a few useful leap commands:

+ +
    +
  • leap help [COMMAND] – get help on COMMAND.
    • +
  • leap history [FILTER] – show the recent deployment history for the selected nodes.
    • +
  • leap ssh web1 – SSH into node web1 (requires leap node init web1 first).
    • +
  • leap list [FILTER] – list the selected nodes. + +
      +
    • leap list production – list only those nodes with the tag ‘production’
      • +
    • leap list --print ip_address – list a particular attribute of all nodes.
      • +
    +
    • +
+ + +

See the full Command Line Reference for more information.

+ +

Node filters

+ +

Many of the leap commands take a “node filter”. You can use a node filter to target a command at one or more nodes.

+ +

A node filter consists of one or more keywords, with an optional “+” before each keyword.

+ +
    +
  • keywords can be a node name, a service type, or a tag.
    • +
  • the “+” before the keyword constructs an AND condition
    • +
  • otherwise, multiple keywords together construct an OR condition
    • +
+ + +

Examples:

+ +
    +
  • leap list openvpn – list all nodes with service openvpn.
    • +
  • leap list openvpn +production – only nodes of service type openvpn AND tag production.
    • +
  • leap deploy webapp openvpn – deploy to all webapp OR openvpn nodes.
    • +
  • leap node init ostrich – just init the node named ostrich.
    • +
+ + +

See the full Command Line Reference for more information.

+ +

Tracking the provider directory in git

+ +

You should commit your provider changes to your favorite VCS whenever things change. This way you can share your configurations with other admins, all they have to do is to pull the changes to stay up to date. Every time you make a change to your provider, such as adding nodes, services, generating certificates, etc. you should add those to your VCS, commit them and push them to where your repository is hosted.

+ +

Note that your provider directory contains secrets, such as private key material and passwords. You do not want to have those passwords readable by the world, so make sure that wherever you are hosting your repository, it is not public for the world to read.

+ +

If you have a post-commit hook that emails the changes to contributors, you may want to exclude diffs for files that might have sensitive secrets. For example, create a .gitattributes file with:

+ +
# No diff, no email for key files
+*.key -diff
+*.pem -diff
+
+# Discard diff for secrets.json
+secrets.json -diff
+
+# No diff for hiera files, they contain passwords
+hiera/* -diff
+
+ +

Editing JSON configuration files

+ +

All the settings that compose your provider are stored in JSON files.

+ +

At a minimum, you will need at least two configuration files:

+ +
    +
  • provider.json – general settings for you provider.
    • +
  • nodes/NAME.json – configuration file for node called “NAME”.
    • +
+ + +

There are a few required properties in provider.json:

+ +
{
+  "domain": "example.org",
+  "name": "Example",
+  "contacts": {
+    "default": "email1@example.org"
+  }
+}
+
+ +

See Provider Configuration for more details.

+ +

For node configuration files, there are two required properties:

+ +
{
+  "ip_address": "1.1.1.1",
+  "services": ["openvpn"]
+}
+
+ +

See Services for details on what servers are available, and see Configuration Files details on how configuration files work.

+ +

How does it work under the hood?

+ +

You don’t need to know any of the details of what happens “under the hood” in order to use the LEAP platform. However, if you are curious as to what is going on, here is a quick primer.

+ +

First, some background terminology:

+ +
    +
  • puppet: Puppet is a system for automating deployment and management of servers (called nodes).
    • +
  • hiera files: In puppet, you can use something called a ‘hiera file’ to seed a node with a few configuration values. In LEAP, we go all out and put every configuration value needed for a node in the hiera file, and automatically compile a custom hiera file for each node.
    • +
+ + +

When you run leap deploy, a bunch of things happen, in this order:

+ +
    +
  1. Compile hiera files: The hiera configuration file for each node is compiled in YAML format and saved in the directory hiera. The source material for this hiera file consists of all the JSON configuration files imported or inherited by the node’s JSON config file.
    2. +
  2. Copy required files to node: All the files needed for puppet to run are rsync'ed to each node. This includes the entire leap_platform directory, as well as the node’s hiera file and other files needed by puppet to set up the node (keys, binary files, etc).
    3. +
  3. Puppet is run: Once the node is ready, leap connects to the node via ssh and runs puppet apply. Puppet is applied locally on the node, without a daemon or puppetmaster.
    4. +
+ + +

You can run leap -v2 deploy to see exactly what commands are being executed.

+ +

This mode of operation is fundamentally different from how puppet is normally used:

+ +
    +
  • There is no puppetmaster that all the servers take orders from, and there is no puppetd running in the background.
    • +
  • Servers cannot dynamically query the puppetmaster for information about the other servers.
    • +
  • There is a static representation for the state of every server that can be committed to git.
    • +
+ + +

There are advantages and disadvantages to the model that LEAP uses. We have found it very useful for our goal of having a common LEAP platform that many different providers can all use while still allowing providers to configure their unique infrastructure.

+ +

We also find it very beneficial to be able to track the state of your infrastructure in git.

+ +

Traditional system configuration automation systems, like Puppet or Chef, deploy changes to servers using a pull method. Each server pulls a manifest from a central master server and uses this to alter the state of the server.

+ +

Instead, the leap tool uses a masterless push method: The sysadmin runs leap deploy from the provider instance directory on their desktop machine to push the changes out to every server (or a subset of servers). LEAP still uses Puppet, but there is no central master server that each node must pull from.

+ +

One other significant difference between LEAP and typical system automation is how interactions among servers are handled. Rather than store a central database of information about each server that can be queried when a recipe is applied, the leap command compiles static representation of all the information a particular server will need in order to apply the recipes. In compiling this static representation, leap can use arbitrary programming logic to query and manipulate information about other servers.

+ +

These two approaches, masterless push and pre-compiled static configuration, allow the sysadmin to manage a set of LEAP servers using traditional software development techniques of branching and merging, to more easily create local testing environments using virtual servers, and to deploy without the added complexity and failure potential of a master server.

+ +
+
+ + diff --git a/docs/en/guide/keys-and-certificates.html b/docs/en/guide/keys-and-certificates.html new file mode 100644 index 00000000..ee7d2699 --- /dev/null +++ b/docs/en/guide/keys-and-certificates.html @@ -0,0 +1,480 @@ + + + + +Keys and Certificates - LEAP Platform Documentation + + + + + + + + +
+
+

Keys and Certificates

+ +
Working with SSH keys, secrets, and X.509 certificates.
+
+
+ + +

Working with SSH

+ +

Whenever the leap command needs to push changes to a node or gather information from a node, it tunnels this command over SSH. Another way to put this: the security of your servers rests entirely on SSH. Because of this, it is important that you understand how leap uses SSH.

+ +

SSH related files

+ +

Assuming your provider directory is called ‘provider’:

+ +
    +
  • provider/nodes/crow/crow_ssh.pub – The public SSH host key for node ‘crow’.
    • +
  • provider/users/alice/alice_ssh.pub – The public SSH user key for user ‘alice’. Anyone with the private key that corresponds to this public key will have root access to all nodes.
    • +
  • provider/files/ssh/known_hosts – An autogenerated known_hosts, built from combining provider/nodes/*/*_ssh.pub. You must not edit this file directly. If you need to change it, remove or change one of the files that is used to generate known_hosts and then run leap compile.
    • +
  • provider/files/ssh/authorized_keys – An autogenerated list of all the user SSH keys with root access to the notes. It is created from provider/users/*/*_ssh.pub. You must not edit this file directly. If you need to change it, remove or change one of the files that is used to generate authorized_keys and then run leap compile.
    • +
+ + +

All of these files should be committed to source control.

+ +

If you rename, remove, or add a node with leap node [mv|add|rm] the SSH key files and the known_hosts file will get properly updated.

+ +

SSH and local nodes

+ +

Local nodes are run as Vagrant virtual machines. The leap command handles SSH slightly differently for these nodes.

+ +

Basically, all the SSH security is turned off for local nodes. Since local nodes only exist for a short time on your computer and can’t be reached from the internet, this is not a problem.

+ +

Specifically, for local nodes:

+ +
    +
  1. known_hosts is never updated with local node keys, since the SSH public key of a local node is different for each user.
    2. +
  2. leap entirely skips the checking of host keys when connecting with a local node.
    3. +
  3. leap adds the public Vagrant SSH key to the list of SSH keys for a user. The public Vagrant SSH key is a shared and insecure key that has root access to most Vagrant virtual machines.
    4. +
+ + +

To upgrade a SSH host key

+ +

Most servers will have more than one SSH host key. Sometimes, the server will have a better SSH host key than the one you have on file. In order to upgrade to the better SSH host key, simply re-run the init command:

+ +
workstation$ leap node init NODE_NAME
+
+ +

This will prompt you if you want to upgrade the SSH host key, but only if leap thinks that an upgrade is advisable.

+ +

When SSH host key changes

+ +

If the host key for a node has changed, you will get an error “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED”.

+ +

To fix this, you need to remove the file files/nodes/stompy/stompy_ssh.pub and run leap node init stompy, where the node’s name is ‘stompy’. Only do this if you are ABSOLUTELY CERTAIN that the node’s SSH host key has changed.

+ +

Changing the SSH port

+ +

Suppose you have a node blinky that has SSH listening on port 22 and you want to make it port 2200.

+ +

First, modify the configuration for blinky to specify the variable ssh.port as 2200. Usually, this is done in common.json or in a tag file.

+ +

For example, you could put this in tags/production.json:

+ +
{
+  "ssh": {
+    "port": 2200
+  }
+}
+
+ +

Run leap compile and open hiera/blinky.yaml to confirm that ssh.port is set to 2200. The port number must be specified as a number, not a string (no quotes).

+ +

Then, you need to deploy this change so that SSH will bind to 2200. You cannot simply run leap deploy blinky because this command will default to using the variable ssh.port which is now 2200 but SSH on the node is still bound to 22.

+ +

So, you manually override the port in the deploy command, using the old port:

+ +
leap deploy --port 22 blinky
+
+ +

Afterwards, SSH on blinky should be listening on port 2200 and you can just run leap deploy blinky from then on.

+ +

Sysadmins with multiple SSH keys

+ +

The command leap add-user --self allows only one SSH key. If you want to specify more than one key for a user, you can do it manually:

+ +
users/userx/userx_ssh.pub
+users/userx/otherkey_ssh.pub
+
+ +

All keys matching ‘userx/*_ssh.pub’ will be usable.

+ +

Removing sysadmin access

+ +

Suppose you want to remove userx from having any further SSH access to the servers. Do this:

+ +
rm -r users/userx
+leap deploy
+
+ +

X.509 Certificates

+ +

Configuration options

+ +

The ca option in provider.json provides settings used when generating CAs and certificates. The defaults are as follows:

+ +
{
+  "ca": {
+    "name": "= global.provider.ca.organization + ' Root CA'",
+    "organization": "= global.provider.name[global.provider.default_language]",
+    "organizational_unit": "= 'https://' + global.provider.domain",
+    "bit_size": 4096,
+    "digest": "SHA256",
+    "life_span": "10y",
+    "server_certificates": {
+      "bit_size": 2048,
+      "digest": "SHA256",
+      "life_span": "1y"
+    },
+    "client_certificates": {
+      "bit_size": 2048,
+      "digest": "SHA256",
+      "life_span": "2m",
+      "limited_prefix": "LIMITED",
+      "unlimited_prefix": "UNLIMITED"
+    }
+  }
+}
+
+ +

You should not need to override these defaults in your own provider.json, but you can if you want to. To see what values are used for your provider, run leap inspect provider.json.

+ +

NOTE: A certificate bit_size greater than 2048 will probably not be recognized by most commercial CAs.

+ +

Certificate Authorities

+ +

There are three x.509 certificate authorities (CA) associated with your provider:

+ +
    +
  1. Commercial CA: It is strongly recommended that you purchase a commercial cert for your primary domain. The goal of platform is to not depend on the commercial CA system, but it does increase security and usability if you purchase a certificate. The cert for the commercial CA must live at files/cert/commercial_ca.crt.
    2. +
  2. Server CA: This is a self-signed CA responsible for signing all the server certificates. The private key lives at files/ca/ca.key and the public cert lives at files/ca/ca.crt. The key is very sensitive information and must be kept private. The public cert is distributed publicly.
    3. +
  3. Client CA: This is a self-signed CA responsible for signing all the client certificates. The private key lives at files/ca/client_ca.key and the public cert lives at files/ca/client_ca.crt. Neither file is distribute publicly. It is not a big deal if the private key for the client CA is compromised, you can just generate a new one and re-deploy.
    4. +
+ + +

To generate both the Server CA and the Client CA, run the command:

+ +
leap cert ca
+
+ +

Server certificates

+ +

Most every server in your service provider will have a x.509 certificate, generated by the leap command using the Server CA. Whenever you modify any settings of a node that might affect it’s certificate (like changing the IP address, hostname, or settings in provider.json), you can magically regenerate all the certs that need to be regenerated with this command:

+ +
leap cert update
+
+ +

Run leap help cert update for notes on usage options.

+ +

Because the server certificates are generated locally on your personal machine, the private key for the Server CA need never be put on any server. It is up to you to keep this file secure.

+ +

Client certificates

+ +

Every leap client gets its own time-limited client certificate. This cert is use to connect to the OpenVPN gateway (and probably other things in the future). It is generated on the fly by the webapp using the Client CA.

+ +

To make this work, the private key of the Client CA is made available to the webapp. This might seem bad, but compromise of the Client CA simply allows the attacker to use the OpenVPN gateways without paying. In the future, we plan to add a command to automatically regenerate the Client CA periodically.

+ +

There are two types of client certificates: limited and unlimited. A client using a limited cert will have its bandwidth limited to the rate specified by provider.service.bandwidth_limit (in Bytes per second). An unlimited cert is given to the user if they authenticate and the user’s service level matches one configured in provider.service.levels without bandwidth limits. Otherwise, the user is given a limited client cert.

+ +

Commercial certificates

+ +

We strongly recommend that you use a commercial signed server certificate for your primary domain (in other words, a certificate with a common name matching whatever you have configured for provider.domain). This provides several benefits:

+ +
    +
  1. When users visit your website, they don’t get a scary notice that something is wrong.
    2. +
  2. When a user runs the LEAP client, selecting your service provider will not cause a warning message.
    3. +
  3. When other providers first discover your provider, they are more likely to trust your provider key if it is fetched over a commercially verified link.
    4. +
+ + +

The LEAP platform is designed so that it assumes you are using a commercial cert for the primary domain of your provider, but all other servers are assumed to use non-commercial certs signed by the Server CA you create.

+ +

To generate a CSR, run:

+ +
leap cert csr
+
+ +

This command will generate the CSR and private key matching provider.domain (you can change the domain with --domain=DOMAIN switch). It also generates a server certificate signed with the Server CA. You should delete this certificate and replace it with a real one once it is created by your commercial CA.

+ +

The related commercial cert files are:

+ +
files/
+  cert/
+    domain.org.crt    # Server certificate for domain.org, obtained by commercial CA.
+    domain.org.csr    # Certificate signing request
+    domain.org.key    # Private key for you certificate
+    commercial_ca.crt # The CA cert obtained from the commercial CA.
+
+ +

The private key file is extremely sensitive and care should be taken with its provenance.

+ +

If your commercial CA has a chained CA cert, you should be OK if you just put the last cert in the chain into the commercial_ca.crt file. This only works if the other CAs in the chain have certs in the debian package ca-certificates, which is the case for almost all CAs.

+ +

If you want to add additional fields to the CSR, like country, city, or locality, you can configure these values in provider.json like so:

+ +
  "ca": {
+    "server_certificates": {
+      "country": "US",
+      "state": "Washington",
+      "locality": "Seattle"
+    }
+  }
+
+ +

If they are not present, the CSR will be created without them.

+ +

Examine Certs

+ +

To see details about the keys and certs you can use leap inspect like so:

+ +
$ leap inspect files/ca/ca.crt
+
+ +

Let’s Encrypt certificate

+ +

LEAP plans to integrate Let’s Encrypt support, so it will be even easier to receive X.509 certificates that are accepted by all browsers. +Until we achieve this, here’s a guide how to do this manually.

+ +

Install the official acme client

+ +

Log in to your webapp node

+ +
server$ git clone https://github.com/certbot/certbot
+server$ cd certbot
+server$ ./certbot-auto --help
+
+ +

Fetch cert

+ +

Stop apache so the letsencrypt client can bind to port 80:

+ +
server$ systemctl stop apache2
+
+ +

Fetch the certs

+ +
server$ ./certbot-auto certonly --standalone --email admin@$(hostname -d) -d $(hostname -d) -d api.$(hostname -d) -d $(hostname -f) -d nicknym.$(hostname -d)
+
+ +

This will put the certs and keys into /etc/letsencrypt/live/DOMAIN/.

+ +

Now, go to your workstation’s provider configuration directory and copy the newly created files from the server to your local config. You will override existing files so please make a backup before proceeding, or use a version control system to track changes.

+ +
workstation$ cd PATH_TO_PROVIDER_CONFIG
+
+ +

Copy the Certificate

+ +
workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/cert.pem files/cert/dev.pixelated-project.org.crt
+
+ +

Copy the private key

+ +
workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/privkey.pem files/cert/DOMAIN.key
+
+ +

Copy the CA chain cert

+ +
workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/fullchain.pem files/cert/commercial_ca.crt
+
+ +

Deploy the certs

+ +

Now you only need to deploy the certs

+ +
workstation$ leap deploy
+
+ +

This will put them into the right locations which are:

+ +
    +
  • /etc/x509/certs/leap_commercial.crt for the certificate
    • +
  • /etc/x509/./keys/leap_commercial.key for the private key
    • +
  • /usr/local/share/ca-certificates/leap_commercial_ca.crt for the CA chain cert.
    • +
+ + +

Start apache2 again

+ +
server$ systemctl start apache2
+
+ +
+
+ + diff --git a/docs/en/guide/keys-and-certificates/index.html b/docs/en/guide/keys-and-certificates/index.html new file mode 100644 index 00000000..4775de5a --- /dev/null +++ b/docs/en/guide/keys-and-certificates/index.html @@ -0,0 +1,480 @@ + + + + +Keys and Certificates - LEAP Platform Documentation + + + + + + + + +
+
+

Keys and Certificates

+ +
Working with SSH keys, secrets, and X.509 certificates.
+
+
+ + +

Working with SSH

+ +

Whenever the leap command needs to push changes to a node or gather information from a node, it tunnels this command over SSH. Another way to put this: the security of your servers rests entirely on SSH. Because of this, it is important that you understand how leap uses SSH.

+ +

SSH related files

+ +

Assuming your provider directory is called ‘provider’:

+ +
    +
  • provider/nodes/crow/crow_ssh.pub – The public SSH host key for node ‘crow’.
    • +
  • provider/users/alice/alice_ssh.pub – The public SSH user key for user ‘alice’. Anyone with the private key that corresponds to this public key will have root access to all nodes.
    • +
  • provider/files/ssh/known_hosts – An autogenerated known_hosts, built from combining provider/nodes/*/*_ssh.pub. You must not edit this file directly. If you need to change it, remove or change one of the files that is used to generate known_hosts and then run leap compile.
    • +
  • provider/files/ssh/authorized_keys – An autogenerated list of all the user SSH keys with root access to the notes. It is created from provider/users/*/*_ssh.pub. You must not edit this file directly. If you need to change it, remove or change one of the files that is used to generate authorized_keys and then run leap compile.
    • +
+ + +

All of these files should be committed to source control.

+ +

If you rename, remove, or add a node with leap node [mv|add|rm] the SSH key files and the known_hosts file will get properly updated.

+ +

SSH and local nodes

+ +

Local nodes are run as Vagrant virtual machines. The leap command handles SSH slightly differently for these nodes.

+ +

Basically, all the SSH security is turned off for local nodes. Since local nodes only exist for a short time on your computer and can’t be reached from the internet, this is not a problem.

+ +

Specifically, for local nodes:

+ +
    +
  1. known_hosts is never updated with local node keys, since the SSH public key of a local node is different for each user.
    2. +
  2. leap entirely skips the checking of host keys when connecting with a local node.
    3. +
  3. leap adds the public Vagrant SSH key to the list of SSH keys for a user. The public Vagrant SSH key is a shared and insecure key that has root access to most Vagrant virtual machines.
    4. +
+ + +

To upgrade a SSH host key

+ +

Most servers will have more than one SSH host key. Sometimes, the server will have a better SSH host key than the one you have on file. In order to upgrade to the better SSH host key, simply re-run the init command:

+ +
workstation$ leap node init NODE_NAME
+
+ +

This will prompt you if you want to upgrade the SSH host key, but only if leap thinks that an upgrade is advisable.

+ +

When SSH host key changes

+ +

If the host key for a node has changed, you will get an error “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED”.

+ +

To fix this, you need to remove the file files/nodes/stompy/stompy_ssh.pub and run leap node init stompy, where the node’s name is ‘stompy’. Only do this if you are ABSOLUTELY CERTAIN that the node’s SSH host key has changed.

+ +

Changing the SSH port

+ +

Suppose you have a node blinky that has SSH listening on port 22 and you want to make it port 2200.

+ +

First, modify the configuration for blinky to specify the variable ssh.port as 2200. Usually, this is done in common.json or in a tag file.

+ +

For example, you could put this in tags/production.json:

+ +
{
+  "ssh": {
+    "port": 2200
+  }
+}
+
+ +

Run leap compile and open hiera/blinky.yaml to confirm that ssh.port is set to 2200. The port number must be specified as a number, not a string (no quotes).

+ +

Then, you need to deploy this change so that SSH will bind to 2200. You cannot simply run leap deploy blinky because this command will default to using the variable ssh.port which is now 2200 but SSH on the node is still bound to 22.

+ +

So, you manually override the port in the deploy command, using the old port:

+ +
leap deploy --port 22 blinky
+
+ +

Afterwards, SSH on blinky should be listening on port 2200 and you can just run leap deploy blinky from then on.

+ +

Sysadmins with multiple SSH keys

+ +

The command leap add-user --self allows only one SSH key. If you want to specify more than one key for a user, you can do it manually:

+ +
users/userx/userx_ssh.pub
+users/userx/otherkey_ssh.pub
+
+ +

All keys matching ‘userx/*_ssh.pub’ will be usable.

+ +

Removing sysadmin access

+ +

Suppose you want to remove userx from having any further SSH access to the servers. Do this:

+ +
rm -r users/userx
+leap deploy
+
+ +

X.509 Certificates

+ +

Configuration options

+ +

The ca option in provider.json provides settings used when generating CAs and certificates. The defaults are as follows:

+ +
{
+  "ca": {
+    "name": "= global.provider.ca.organization + ' Root CA'",
+    "organization": "= global.provider.name[global.provider.default_language]",
+    "organizational_unit": "= 'https://' + global.provider.domain",
+    "bit_size": 4096,
+    "digest": "SHA256",
+    "life_span": "10y",
+    "server_certificates": {
+      "bit_size": 2048,
+      "digest": "SHA256",
+      "life_span": "1y"
+    },
+    "client_certificates": {
+      "bit_size": 2048,
+      "digest": "SHA256",
+      "life_span": "2m",
+      "limited_prefix": "LIMITED",
+      "unlimited_prefix": "UNLIMITED"
+    }
+  }
+}
+
+ +

You should not need to override these defaults in your own provider.json, but you can if you want to. To see what values are used for your provider, run leap inspect provider.json.

+ +

NOTE: A certificate bit_size greater than 2048 will probably not be recognized by most commercial CAs.

+ +

Certificate Authorities

+ +

There are three x.509 certificate authorities (CA) associated with your provider:

+ +
    +
  1. Commercial CA: It is strongly recommended that you purchase a commercial cert for your primary domain. The goal of platform is to not depend on the commercial CA system, but it does increase security and usability if you purchase a certificate. The cert for the commercial CA must live at files/cert/commercial_ca.crt.
    2. +
  2. Server CA: This is a self-signed CA responsible for signing all the server certificates. The private key lives at files/ca/ca.key and the public cert lives at files/ca/ca.crt. The key is very sensitive information and must be kept private. The public cert is distributed publicly.
    3. +
  3. Client CA: This is a self-signed CA responsible for signing all the client certificates. The private key lives at files/ca/client_ca.key and the public cert lives at files/ca/client_ca.crt. Neither file is distribute publicly. It is not a big deal if the private key for the client CA is compromised, you can just generate a new one and re-deploy.
    4. +
+ + +

To generate both the Server CA and the Client CA, run the command:

+ +
leap cert ca
+
+ +

Server certificates

+ +

Most every server in your service provider will have a x.509 certificate, generated by the leap command using the Server CA. Whenever you modify any settings of a node that might affect it’s certificate (like changing the IP address, hostname, or settings in provider.json), you can magically regenerate all the certs that need to be regenerated with this command:

+ +
leap cert update
+
+ +

Run leap help cert update for notes on usage options.

+ +

Because the server certificates are generated locally on your personal machine, the private key for the Server CA need never be put on any server. It is up to you to keep this file secure.

+ +

Client certificates

+ +

Every leap client gets its own time-limited client certificate. This cert is use to connect to the OpenVPN gateway (and probably other things in the future). It is generated on the fly by the webapp using the Client CA.

+ +

To make this work, the private key of the Client CA is made available to the webapp. This might seem bad, but compromise of the Client CA simply allows the attacker to use the OpenVPN gateways without paying. In the future, we plan to add a command to automatically regenerate the Client CA periodically.

+ +

There are two types of client certificates: limited and unlimited. A client using a limited cert will have its bandwidth limited to the rate specified by provider.service.bandwidth_limit (in Bytes per second). An unlimited cert is given to the user if they authenticate and the user’s service level matches one configured in provider.service.levels without bandwidth limits. Otherwise, the user is given a limited client cert.

+ +

Commercial certificates

+ +

We strongly recommend that you use a commercial signed server certificate for your primary domain (in other words, a certificate with a common name matching whatever you have configured for provider.domain). This provides several benefits:

+ +
    +
  1. When users visit your website, they don’t get a scary notice that something is wrong.
    2. +
  2. When a user runs the LEAP client, selecting your service provider will not cause a warning message.
    3. +
  3. When other providers first discover your provider, they are more likely to trust your provider key if it is fetched over a commercially verified link.
    4. +
+ + +

The LEAP platform is designed so that it assumes you are using a commercial cert for the primary domain of your provider, but all other servers are assumed to use non-commercial certs signed by the Server CA you create.

+ +

To generate a CSR, run:

+ +
leap cert csr
+
+ +

This command will generate the CSR and private key matching provider.domain (you can change the domain with --domain=DOMAIN switch). It also generates a server certificate signed with the Server CA. You should delete this certificate and replace it with a real one once it is created by your commercial CA.

+ +

The related commercial cert files are:

+ +
files/
+  cert/
+    domain.org.crt    # Server certificate for domain.org, obtained by commercial CA.
+    domain.org.csr    # Certificate signing request
+    domain.org.key    # Private key for you certificate
+    commercial_ca.crt # The CA cert obtained from the commercial CA.
+
+ +

The private key file is extremely sensitive and care should be taken with its provenance.

+ +

If your commercial CA has a chained CA cert, you should be OK if you just put the last cert in the chain into the commercial_ca.crt file. This only works if the other CAs in the chain have certs in the debian package ca-certificates, which is the case for almost all CAs.

+ +

If you want to add additional fields to the CSR, like country, city, or locality, you can configure these values in provider.json like so:

+ +
  "ca": {
+    "server_certificates": {
+      "country": "US",
+      "state": "Washington",
+      "locality": "Seattle"
+    }
+  }
+
+ +

If they are not present, the CSR will be created without them.

+ +

Examine Certs

+ +

To see details about the keys and certs you can use leap inspect like so:

+ +
$ leap inspect files/ca/ca.crt
+
+ +

Let’s Encrypt certificate

+ +

LEAP plans to integrate Let’s Encrypt support, so it will be even easier to receive X.509 certificates that are accepted by all browsers. +Until we achieve this, here’s a guide how to do this manually.

+ +

Install the official acme client

+ +

Log in to your webapp node

+ +
server$ git clone https://github.com/certbot/certbot
+server$ cd certbot
+server$ ./certbot-auto --help
+
+ +

Fetch cert

+ +

Stop apache so the letsencrypt client can bind to port 80:

+ +
server$ systemctl stop apache2
+
+ +

Fetch the certs

+ +
server$ ./certbot-auto certonly --standalone --email admin@$(hostname -d) -d $(hostname -d) -d api.$(hostname -d) -d $(hostname -f) -d nicknym.$(hostname -d)
+
+ +

This will put the certs and keys into /etc/letsencrypt/live/DOMAIN/.

+ +

Now, go to your workstation’s provider configuration directory and copy the newly created files from the server to your local config. You will override existing files so please make a backup before proceeding, or use a version control system to track changes.

+ +
workstation$ cd PATH_TO_PROVIDER_CONFIG
+
+ +

Copy the Certificate

+ +
workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/cert.pem files/cert/dev.pixelated-project.org.crt
+
+ +

Copy the private key

+ +
workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/privkey.pem files/cert/DOMAIN.key
+
+ +

Copy the CA chain cert

+ +
workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/fullchain.pem files/cert/commercial_ca.crt
+
+ +

Deploy the certs

+ +

Now you only need to deploy the certs

+ +
workstation$ leap deploy
+
+ +

This will put them into the right locations which are:

+ +
    +
  • /etc/x509/certs/leap_commercial.crt for the certificate
    • +
  • /etc/x509/./keys/leap_commercial.key for the private key
    • +
  • /usr/local/share/ca-certificates/leap_commercial_ca.crt for the CA chain cert.
    • +
+ + +

Start apache2 again

+ +
server$ systemctl start apache2
+
+ +
+
+ + diff --git a/docs/en/guide/miscellaneous.html b/docs/en/guide/miscellaneous.html new file mode 100644 index 00000000..03239410 --- /dev/null +++ b/docs/en/guide/miscellaneous.html @@ -0,0 +1,145 @@ + + + + +Miscellaneous - LEAP Platform Documentation + + + + + + + + +
+
+

Miscellaneous

+ +
Miscellaneous commands you may need to know.
+
+
+
    +
  1. + Facts +
    2. +
+ +

Facts

+ +

There are a few cases when we must gather internal data from a node before we can successfully deploy to other nodes. This is what facts.json is for. It stores a snapshot of certain facts about each node, as needed. Entries in facts.json are updated automatically when you initialize, rename, or remove a node. To manually force a full update of facts.json, run:

+ +
leap facts update FILTER
+
+ +

Run leap help facts update for more information.

+ +

The file facts.json should be committed to source control. You might not have a facts.json if one is not required for your provider.

+ +
+
+ + diff --git a/docs/en/guide/miscellaneous/index.html b/docs/en/guide/miscellaneous/index.html new file mode 100644 index 00000000..9f17df4e --- /dev/null +++ b/docs/en/guide/miscellaneous/index.html @@ -0,0 +1,145 @@ + + + + +Miscellaneous - LEAP Platform Documentation + + + + + + + + +
+
+

Miscellaneous

+ +
Miscellaneous commands you may need to know.
+
+
+
    +
  1. + Facts +
    2. +
+ +

Facts

+ +

There are a few cases when we must gather internal data from a node before we can successfully deploy to other nodes. This is what facts.json is for. It stores a snapshot of certain facts about each node, as needed. Entries in facts.json are updated automatically when you initialize, rename, or remove a node. To manually force a full update of facts.json, run:

+ +
leap facts update FILTER
+
+ +

Run leap help facts update for more information.

+ +

The file facts.json should be committed to source control. You might not have a facts.json if one is not required for your provider.

+ +
+
+ + diff --git a/docs/en/guide/nodes.html b/docs/en/guide/nodes.html new file mode 100644 index 00000000..c6238b5f --- /dev/null +++ b/docs/en/guide/nodes.html @@ -0,0 +1,231 @@ + + + + +Nodes - LEAP Platform Documentation + + + + + + + + +
+
+

Nodes

+ +
Working with nodes, services, tags, and locations.
+
+
+
    +
  1. + Locations +
    2. +
  2. + Disabling Nodes +
    3. +
+ +

Locations

+ +

All nodes should have a location.name specified, and optionally additional information about the location, like the time zone. This location information is used for two things:

+ +
    +
  • Determine which nodes can, or must, communicate with one another via a local network. The way some virtualization environments work, like OpenStack, requires that nodes communicate via the local network if they are on the same network.
    • +
  • Allows the client to prefer connections to nodes that are closer in physical proximity to the user. This is particularly important for OpenVPN nodes.
    • +
+ + +

The location stanza in a node’s config file looks like this:

+ +
{
+  "location": {
+    "id": "ankara",
+    "name": "Ankara",
+    "country_code": "TR",
+    "timezone": "+2",
+    "hemisphere": "N"
+  }
+}
+
+ +

The fields:

+ +
    +
  • id: An internal handle to use for this location. If two nodes have match location.id, then they are treated as being on a local network with one another. This value defaults to downcase and underscore of location.name.
    • +
  • name: Can be anything, might be displayed to the user in the client if they choose to manually select a gateway.
    • +
  • country_code: The ISO 3166-1 two letter country code.
    • +
  • timezone: The timezone expressed as an offset from UTC (in standard time, not daylight savings). You can look up the timezone using this handy map.
    • +
  • hemisphere: This should be “S” for all servers in South America, Africa, or Australia. Otherwise, this should be “N”.
    • +
+ + +

These location options are very imprecise, but good enough for most usage. The client often does not know its own location precisely either. Instead, the client makes an educated guess at location based on the OS’s timezone and locale.

+ +

If you have multiple nodes in a single location, it is best to use a tag for the location. For example:

+ +

tags/ankara.json:

+ +
{
+  "location": {
+    "name": "Ankara",
+    "country_code": "TR",
+    "timezone": "+2",
+    "hemisphere": "N"
+  }
+}
+
+ +

nodes/vpngateway.json:

+ +
{
+  "services": "openvpn",
+  "tags": ["production", "ankara"],
+  "ip_address": "1.1.1.1",
+  "openvpn": {
+    "gateway_address": "1.1.1.2"
+  }
+}
+
+ +

Unless you are using OpenStack or AWS, setting location for nodes is not required. It is, however, highly recommended.

+ +

Disabling Nodes

+ +

There are two ways to temporarily disable a node:

+ +

Option 1: disabled environment

+ +

You can assign an environment to the node that marks it as disabled. Then, if you use environment pinning, the node will be ignored when you deploy. For example:

+ +
{
+  "environment": "disabled"
+}
+
+ +

Then use leap env pin ENV to pin the environment to something other than ‘disabled’. This only works if all the other nodes are also assigned to some environment.

+ +

Option 2: enabled == false

+ +

If a node has a property enabled set to false, then the leap command will skip over the node and pretend that it does not exist. For example:

+ +
{
+  "ip_address": "1.1.1.1",
+  "services": ["openvpn"],
+  "enabled": false
+}
+
+ +

Options 3: no-deploy

+ +

If the file /etc/leap/no-deploy exists on a node, then when you run the commmand leap deploy it will halt and prevent a deploy from going through (if the node was going to be included in the deploy).

+ +
+
+ + diff --git a/docs/en/guide/nodes/index.html b/docs/en/guide/nodes/index.html new file mode 100644 index 00000000..7f1a60b3 --- /dev/null +++ b/docs/en/guide/nodes/index.html @@ -0,0 +1,231 @@ + + + + +Nodes - LEAP Platform Documentation + + + + + + + + +
+
+

Nodes

+ +
Working with nodes, services, tags, and locations.
+
+
+
    +
  1. + Locations +
    2. +
  2. + Disabling Nodes +
    3. +
+ +

Locations

+ +

All nodes should have a location.name specified, and optionally additional information about the location, like the time zone. This location information is used for two things:

+ +
    +
  • Determine which nodes can, or must, communicate with one another via a local network. The way some virtualization environments work, like OpenStack, requires that nodes communicate via the local network if they are on the same network.
    • +
  • Allows the client to prefer connections to nodes that are closer in physical proximity to the user. This is particularly important for OpenVPN nodes.
    • +
+ + +

The location stanza in a node’s config file looks like this:

+ +
{
+  "location": {
+    "id": "ankara",
+    "name": "Ankara",
+    "country_code": "TR",
+    "timezone": "+2",
+    "hemisphere": "N"
+  }
+}
+
+ +

The fields:

+ +
    +
  • id: An internal handle to use for this location. If two nodes have match location.id, then they are treated as being on a local network with one another. This value defaults to downcase and underscore of location.name.
    • +
  • name: Can be anything, might be displayed to the user in the client if they choose to manually select a gateway.
    • +
  • country_code: The ISO 3166-1 two letter country code.
    • +
  • timezone: The timezone expressed as an offset from UTC (in standard time, not daylight savings). You can look up the timezone using this handy map.
    • +
  • hemisphere: This should be “S” for all servers in South America, Africa, or Australia. Otherwise, this should be “N”.
    • +
+ + +

These location options are very imprecise, but good enough for most usage. The client often does not know its own location precisely either. Instead, the client makes an educated guess at location based on the OS’s timezone and locale.

+ +

If you have multiple nodes in a single location, it is best to use a tag for the location. For example:

+ +

tags/ankara.json:

+ +
{
+  "location": {
+    "name": "Ankara",
+    "country_code": "TR",
+    "timezone": "+2",
+    "hemisphere": "N"
+  }
+}
+
+ +

nodes/vpngateway.json:

+ +
{
+  "services": "openvpn",
+  "tags": ["production", "ankara"],
+  "ip_address": "1.1.1.1",
+  "openvpn": {
+    "gateway_address": "1.1.1.2"
+  }
+}
+
+ +

Unless you are using OpenStack or AWS, setting location for nodes is not required. It is, however, highly recommended.

+ +

Disabling Nodes

+ +

There are two ways to temporarily disable a node:

+ +

Option 1: disabled environment

+ +

You can assign an environment to the node that marks it as disabled. Then, if you use environment pinning, the node will be ignored when you deploy. For example:

+ +
{
+  "environment": "disabled"
+}
+
+ +

Then use leap env pin ENV to pin the environment to something other than ‘disabled’. This only works if all the other nodes are also assigned to some environment.

+ +

Option 2: enabled == false

+ +

If a node has a property enabled set to false, then the leap command will skip over the node and pretend that it does not exist. For example:

+ +
{
+  "ip_address": "1.1.1.1",
+  "services": ["openvpn"],
+  "enabled": false
+}
+
+ +

Options 3: no-deploy

+ +

If the file /etc/leap/no-deploy exists on a node, then when you run the commmand leap deploy it will halt and prevent a deploy from going through (if the node was going to be included in the deploy).

+ +
+
+ + diff --git a/docs/en/guide/provider-configuration.html b/docs/en/guide/provider-configuration.html new file mode 100644 index 00000000..5c98eb34 --- /dev/null +++ b/docs/en/guide/provider-configuration.html @@ -0,0 +1,223 @@ + + + + +Provider Configuration - LEAP Platform Documentation + + + + + + + + +
+
+

Provider Configuration

+ +
Explore how to configure your provider.
+
+
+ + +

Required provider configuration

+ +

There are a few required settings in provider.json. At a minimum, you must have:

+ +
    +
  • domain: defines the primary domain of the provider. This is the domain that users will type in when using the Bitmask client, although it is not necessarily the domain where users will visit if they sign up via the web application. If email is supported, all accounts will be username@domain.
    • +
  • name: A brief title for this provider. It can be multiple words, but should not be too long.
    • +
  • contacts.default: One or more email addresses for sysadmins.
    • +
+ + +

For example:

+ +
{
+  "domain": "freerobot.org",
+  "name": "Freedom for Robots!",
+  "contacts": {
+    "default": "root@freerobot.org"
+  }
+}
+
+ +

Recommended provider configuration

+ +
    +
  • description: A longer description of the provider, shown to the user when they register a new account through Bitmask client.
    • +
  • languages: A list of language codes that should be enabled.
    • +
  • default_language: The initial default language code.
    • +
  • enrollment_policy: One of “open”, “closed”, or “invite”. (invite not currently supported).
    • +
+ + +

For example:

+ +
{
+  "description": "It is time for robots of the world to unite and throw of the shackles of servitude to our organic overlords.",
+  "languages": ["en", "de", "pt", "01"],
+  "default_language": "01",
+  "enrollman_policy": "open"
+}
+
+ +

For a full list of possible settings, you can use leap inspect to see how provider.json is evaluated after including the inherited defaults:

+ +
$ leap inspect provider.json
+
+ +

Configuring service levels

+ +

The provider.json file defines the available service levels for the provider.

+ +

For example, in provider.json:

+ +
"service": {
+   "default_service_level": "low",
+   "levels": {
+      "low": {
+        "description": "Entry level plan, with unlimited bandwidth and minimal storage quota.",
+        "name": "entry",
+        "storage": "10 MB",
+        "rate": {
+          "USD": 5,
+          "GBP": 3,
+          "EUR": 6
+        }
+      },
+      "full": {
+        "description": "Full plan, with unlimited bandwidth and higher quota."
+        "name": "full",
+        "storage": "5 GB",
+        "rate": {
+          "USD": 10,
+          "GBP": 6,
+          "EUR": 12
+        }
+      }
+    }
+  }
+}
+
+ +

For a list of currency codes, see https://en.wikipedia.org/wiki/ISO_4217#Active_codes

+ +
+
+ + diff --git a/docs/en/guide/provider-configuration/index.html b/docs/en/guide/provider-configuration/index.html new file mode 100644 index 00000000..b710cb64 --- /dev/null +++ b/docs/en/guide/provider-configuration/index.html @@ -0,0 +1,223 @@ + + + + +Provider Configuration - LEAP Platform Documentation + + + + + + + + +
+
+

Provider Configuration

+ +
Explore how to configure your provider.
+
+
+ + +

Required provider configuration

+ +

There are a few required settings in provider.json. At a minimum, you must have:

+ +
    +
  • domain: defines the primary domain of the provider. This is the domain that users will type in when using the Bitmask client, although it is not necessarily the domain where users will visit if they sign up via the web application. If email is supported, all accounts will be username@domain.
    • +
  • name: A brief title for this provider. It can be multiple words, but should not be too long.
    • +
  • contacts.default: One or more email addresses for sysadmins.
    • +
+ + +

For example:

+ +
{
+  "domain": "freerobot.org",
+  "name": "Freedom for Robots!",
+  "contacts": {
+    "default": "root@freerobot.org"
+  }
+}
+
+ +

Recommended provider configuration

+ +
    +
  • description: A longer description of the provider, shown to the user when they register a new account through Bitmask client.
    • +
  • languages: A list of language codes that should be enabled.
    • +
  • default_language: The initial default language code.
    • +
  • enrollment_policy: One of “open”, “closed”, or “invite”. (invite not currently supported).
    • +
+ + +

For example:

+ +
{
+  "description": "It is time for robots of the world to unite and throw of the shackles of servitude to our organic overlords.",
+  "languages": ["en", "de", "pt", "01"],
+  "default_language": "01",
+  "enrollman_policy": "open"
+}
+
+ +

For a full list of possible settings, you can use leap inspect to see how provider.json is evaluated after including the inherited defaults:

+ +
$ leap inspect provider.json
+
+ +

Configuring service levels

+ +

The provider.json file defines the available service levels for the provider.

+ +

For example, in provider.json:

+ +
"service": {
+   "default_service_level": "low",
+   "levels": {
+      "low": {
+        "description": "Entry level plan, with unlimited bandwidth and minimal storage quota.",
+        "name": "entry",
+        "storage": "10 MB",
+        "rate": {
+          "USD": 5,
+          "GBP": 3,
+          "EUR": 6
+        }
+      },
+      "full": {
+        "description": "Full plan, with unlimited bandwidth and higher quota."
+        "name": "full",
+        "storage": "5 GB",
+        "rate": {
+          "USD": 10,
+          "GBP": 6,
+          "EUR": 12
+        }
+      }
+    }
+  }
+}
+
+ +

For a list of currency codes, see https://en.wikipedia.org/wiki/ISO_4217#Active_codes

+ +
+
+ + diff --git a/docs/en/guide/virtual-machines.html b/docs/en/guide/virtual-machines.html new file mode 100644 index 00000000..5cee9a40 --- /dev/null +++ b/docs/en/guide/virtual-machines.html @@ -0,0 +1,299 @@ + + + + +Virtual Machines - LEAP Platform Documentation + + + + + + + + +
+
+

Virtual Machines

+ +
Running LEAP platform on remote virtual machines
+
+
+ + +

Introduction

+ +

You can use the leap command line to easily remote virtual machines.

+ +

Note: there are two types of virtual machines that leap can handle:

+ +
    +
  • Local virtual machines running with vagrant, for use in testing.
    • +
  • Remote virtual machines hosted by a cloud provider like AWS or Rackspace.
    • +
+ + +

This guide is for “remote virtual machines”. For “local virtual machines” see Vagrant.

+ +

Currently, only Amazon AWS is supported as a cloud provider.

+ +

Configuration

+ +

To get started with virtual machines, you must configure a cloud.json file with your API credentials for the virtual machine vendor. This file lives in the root of your provider directory.

+ +

For example:

+ +
{
+  "my_aws": {
+    "api": "aws",
+    "vendor": "aws",
+    "auth": {
+      "region": "us-west-2",
+      "aws_access_key_id": "xxxx my key id xxxx",
+      "aws_secret_access_key": "xxxx my access key xxxx"
+    }
+  }
+}
+
+ +

This will configure a cloud “authentication profile” called “my_aws”. This profile will be used by default if there is only one. See below for managing multiple authentication profiles.

+ +

Required cloud.json properties

+ +
    +
  • $profile: In this case, ‘my_aws’.
    • +
  • $profile.api: For now, must always be “aws”.
    • +
  • $profile.vendor: For now, must always be “aws”.
    • +
  • $profile.auth: API specific authentication configuration for this profile. In the case of AWS, it must include auth.region, auth.aws_access_key_id, and aws_secret_access_key.
    • +
+ + +

Additional cloud.json properties

+ +

In addition to required configuration properties, these are optional:

+ +
    +
  • $profile.default_image: What image to use for new nodes by default. Generally, you should not specify this, because it will automatically select the right Debian image for your region. A node can override this with the property vm.image.
    • +
  • $profile.default_options: This is passed directly to the cloud API, and so is specific to whichever API you are using. The node can override this with the property vm.options.
    • +
+ + +

A more complete example cloud.json:

+ +
{
+  "my_aws": {
+    "api": "aws",
+    "vendor": "aws",
+    "auth": {
+      "region": "us-west-2",
+      "aws_access_key_id": "xxxx my key id xxxx",
+      "aws_secret_access_key": "xxxx my access key xxxx"
+    },
+    "default_image": "ami-98e114f8",
+    "default_options": {
+      "InstanceType": "t2.nano"
+    }
+  }
+}
+
+ +

See also:

+ + + + +

Usage

+ +

See leap help vm for a description of all the possible commands.

+ +

In order to be able to create new virtual machine instances, you need to register your SSH key with the VM vendor.

+ +
leap vm key-register
+
+ +

You only have to do this once, and only people who will be creating new VM instances need to do this.

+ +

Once you have done that, you just leap vm add to create the virtual machine and then leap vm start to actually boot it.

+ +
leap vm add mynode
+leap vm start mynode
+
+ +

You can specify seed values to leap vm add. For example:

+ +
leap vm add mynode services:webapp tags:seattle vm.options.InstanceType:t2.small
+
+ +

Check to see what the status is of all VMs:

+ +
leap vm status
+
+ +

If it looks good, you can now deploy to the new server:

+ +
leap node init mynode
+leap deploy mynode
+
+ +

To stop the VM:

+ +
leap vm stop mynode
+
+ +

To destroy the VM and clean up its storage space:

+ +
leap vm rm mynode
+
+ +

In general, you should remove VMs instead of stopping them, unless you plan on stopping the VM for a short amount of time. A stopped VM will still use disk space and still incur charges.

+ +

Keeping State Synchronized

+ +

The LEAP platform stores all its state information in flat static files. The virtual machine vendor, however, also has its own state.

+ +

On the provider side, VM state is stored in node configuration files in nodes/*.json. Of particular importance are the properties ip_address and vm.id.

+ +

Most of the time, you should not have any trouble: the leap vm commands will keep things in sync. However, if the state of your configuration files gets out of sync with the state of the virtual machines, it can cause problems.

+ +

The command leap vm status will warn you whenever it detects a problem and it will usually propose a fix.

+ +

Typically, the fix is to manually update the binding between a local node configuration and the running remote virtual machine, like so:

+ +
leap vm bind NODE_NAME VM_ID
+
+ +

Multiple authentication profiles

+ +

If you have multiple profiles configured in cloud.json, you can specify which one you want to use:

+ +
    +
  • Set the vm.auth property in the node configuration to match the name of the authentication profile.
    • +
  • Or, pass --auth PROFILE_NAME on the command line.
    • +
+ + +
+
+ + diff --git a/docs/en/guide/virtual-machines/index.html b/docs/en/guide/virtual-machines/index.html new file mode 100644 index 00000000..da0da107 --- /dev/null +++ b/docs/en/guide/virtual-machines/index.html @@ -0,0 +1,299 @@ + + + + +Virtual Machines - LEAP Platform Documentation + + + + + + + + +
+
+

Virtual Machines

+ +
Running LEAP platform on remote virtual machines
+
+
+ + +

Introduction

+ +

You can use the leap command line to easily remote virtual machines.

+ +

Note: there are two types of virtual machines that leap can handle:

+ +
    +
  • Local virtual machines running with vagrant, for use in testing.
    • +
  • Remote virtual machines hosted by a cloud provider like AWS or Rackspace.
    • +
+ + +

This guide is for “remote virtual machines”. For “local virtual machines” see Vagrant.

+ +

Currently, only Amazon AWS is supported as a cloud provider.

+ +

Configuration

+ +

To get started with virtual machines, you must configure a cloud.json file with your API credentials for the virtual machine vendor. This file lives in the root of your provider directory.

+ +

For example:

+ +
{
+  "my_aws": {
+    "api": "aws",
+    "vendor": "aws",
+    "auth": {
+      "region": "us-west-2",
+      "aws_access_key_id": "xxxx my key id xxxx",
+      "aws_secret_access_key": "xxxx my access key xxxx"
+    }
+  }
+}
+
+ +

This will configure a cloud “authentication profile” called “my_aws”. This profile will be used by default if there is only one. See below for managing multiple authentication profiles.

+ +

Required cloud.json properties

+ +
    +
  • $profile: In this case, ‘my_aws’.
    • +
  • $profile.api: For now, must always be “aws”.
    • +
  • $profile.vendor: For now, must always be “aws”.
    • +
  • $profile.auth: API specific authentication configuration for this profile. In the case of AWS, it must include auth.region, auth.aws_access_key_id, and aws_secret_access_key.
    • +
+ + +

Additional cloud.json properties

+ +

In addition to required configuration properties, these are optional:

+ +
    +
  • $profile.default_image: What image to use for new nodes by default. Generally, you should not specify this, because it will automatically select the right Debian image for your region. A node can override this with the property vm.image.
    • +
  • $profile.default_options: This is passed directly to the cloud API, and so is specific to whichever API you are using. The node can override this with the property vm.options.
    • +
+ + +

A more complete example cloud.json:

+ +
{
+  "my_aws": {
+    "api": "aws",
+    "vendor": "aws",
+    "auth": {
+      "region": "us-west-2",
+      "aws_access_key_id": "xxxx my key id xxxx",
+      "aws_secret_access_key": "xxxx my access key xxxx"
+    },
+    "default_image": "ami-98e114f8",
+    "default_options": {
+      "InstanceType": "t2.nano"
+    }
+  }
+}
+
+ +

See also:

+ + + + +

Usage

+ +

See leap help vm for a description of all the possible commands.

+ +

In order to be able to create new virtual machine instances, you need to register your SSH key with the VM vendor.

+ +
leap vm key-register
+
+ +

You only have to do this once, and only people who will be creating new VM instances need to do this.

+ +

Once you have done that, you just leap vm add to create the virtual machine and then leap vm start to actually boot it.

+ +
leap vm add mynode
+leap vm start mynode
+
+ +

You can specify seed values to leap vm add. For example:

+ +
leap vm add mynode services:webapp tags:seattle vm.options.InstanceType:t2.small
+
+ +

Check to see what the status is of all VMs:

+ +
leap vm status
+
+ +

If it looks good, you can now deploy to the new server:

+ +
leap node init mynode
+leap deploy mynode
+
+ +

To stop the VM:

+ +
leap vm stop mynode
+
+ +

To destroy the VM and clean up its storage space:

+ +
leap vm rm mynode
+
+ +

In general, you should remove VMs instead of stopping them, unless you plan on stopping the VM for a short amount of time. A stopped VM will still use disk space and still incur charges.

+ +

Keeping State Synchronized

+ +

The LEAP platform stores all its state information in flat static files. The virtual machine vendor, however, also has its own state.

+ +

On the provider side, VM state is stored in node configuration files in nodes/*.json. Of particular importance are the properties ip_address and vm.id.

+ +

Most of the time, you should not have any trouble: the leap vm commands will keep things in sync. However, if the state of your configuration files gets out of sync with the state of the virtual machines, it can cause problems.

+ +

The command leap vm status will warn you whenever it detects a problem and it will usually propose a fix.

+ +

Typically, the fix is to manually update the binding between a local node configuration and the running remote virtual machine, like so:

+ +
leap vm bind NODE_NAME VM_ID
+
+ +

Multiple authentication profiles

+ +

If you have multiple profiles configured in cloud.json, you can specify which one you want to use:

+ +
    +
  • Set the vm.auth property in the node configuration to match the name of the authentication profile.
    • +
  • Or, pass --auth PROFILE_NAME on the command line.
    • +
+ + +
+
+ + diff --git a/docs/en/services.html b/docs/en/services.html new file mode 100644 index 00000000..55211e64 --- /dev/null +++ b/docs/en/services.html @@ -0,0 +1,251 @@ + + + + +Services - LEAP Platform Documentation + + + + + + + + +
+
+

Guide to node services

+ +
+
+
+ + +

Introduction

+ +

Every node (server) must have one or more services defined that determines what role the node performs. For example:

+ +
workstation$ cat nodes/stallman.json
+{
+  "ip_address": "199.99.99.1",
+  "services": ["webapp", "tor"]
+}
+
+ +

Here are common questions to ask when adding a new node to your provider:

+ +
    +
  • many or few? Some services benefit from having many nodes, while some services are best run on only one or two nodes.
    • +
  • required or optional? Some services are required, while others can be left out.
    • +
  • who does the node communicate with? Some services communicate very heavily with other particular services. Nodes running these services should be close together.
    • +
  • public or private network? Some services communicate with the public internet, while others only need to communicate with other nodes in the infrastructure.
    • +
+ + +

Available services

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ServiceVPNEmailNotes
webappUser control panel, provider API, and support system.
couchdbData storage for everything. Private node.
soledadUser data synchronization daemon. Usually paired with couchdb nodes.
mxIncoming and outgoing MX servers.
openvpnOpenVPN gateways.
monitorNagios monitoring. This service must be on the webapp node.
torTor exit node.
+ + +

Key: Required, Optional, Not Used

+ +

+

+couchdb +

+
Data storage for all user data.
+
+
+

+openvpn +

+
OpenVPN egress gateways
+
+
+

+monitor +

+
Nagios monitoring and continuous testing.
+
+
+

+mx +

+
Incoming and outgoing MX servers.
+
+
+

+soledad +

+
User data synchronization daemon
+
+
+

+tor +

+
Tor exit node or hidden service
+
+
+

+webapp +

+
leap_web user management application and provider API.
+
+

+ +
+
+ + diff --git a/docs/en/services/couchdb.html b/docs/en/services/couchdb.html new file mode 100644 index 00000000..3dde5a3c --- /dev/null +++ b/docs/en/services/couchdb.html @@ -0,0 +1,328 @@ + + + + +couchdb - LEAP Platform Documentation + + + + + + + + +
+
+

couchdb

+ +
Data storage for all user data.
+
+
+ + +

Topology

+ +

Required:

+ +
    +
  • Nodes with couchdb service must also have soledad service, if email is enabled.
    • +
+ + +

Suggested:

+ +
    +
  • Nodes with couchdb service communicate heavily with webapp and mx.
    • +
+ + +

couchdb nodes do not need to be reachable from the public internet, although the soledad service does require this.

+ +

Configuration

+ +

Nighly dumps

+ +

You can do a nightly couchdb data dump by adding this to your node config:

+ +
"couch": {
+  "backup": true
+}
+
+ +

Data will get dumped to /var/backups/couchdb.

+ +

Plain CouchDB

+ +

BigCouch is not supported on Platform version 0.8 and higher: only plain CouchDB is possible. For earlier versions, you must do this in order to use plain CouchDB:

+ +
"couch": {
+  "master": true,
+  "pwhash_alg": "pbkdf2"
+}
+
+ +

Various Tasks

+ +

Re-enabling blocked account

+ +

When a user account gets destroyed from the webapp, there’s still a leftover doc in the identities db so other people can’t claim that account without an admin’s intervention. You can remove this username reservation through the webapp.

+ +

However, here is how you could do it manually, if you wanted to:

+ +

grep the identities db for the email address:

+ +
curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET http://127.0.0.1:5984/identities/_all_docs?include_docs=true|grep test_127@bitmask.net
+
+ +

lookup “id” and “rev” to delete the doc:

+ +
curl -s --netrc-file /etc/couchdb/couchdb.netrc -X DELETE 'http://127.0.0.1:5984/identities/b25cf10f935b58088f0d547fca823265?rev=2-715a9beba597a2ab01851676f12c3e4a'
+
+ +

How to find out which userstore belongs to which identity?

+ +
/usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep testuser
+
+{"id":"665e004870ee17aa4c94331ff3ecb173","key":"665e004870ee17aa4c94331ff3ecb173","value":{"rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b"},"doc":{"_id":"665e004870ee17aa4c94331ff3ecb173","_rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b","user_id":"665e004870ee17aa4c94331ff3cd59eb","address":"testuser@example.org","destination":"testuser@example.org","keys": ...
+
+ +
    +
  • search for the “user_id” field
    • +
  • in this example testuser@example.org uses the database user-665e004870ee17aa4c94331ff3cd59eb
    • +
+ + +

How much disk space is used by a userstore

+ +

Beware that this returns the uncompacted disk size (see http://wiki.apache.org/couchdb/Compaction)

+ +
echo "`curl --netrc -s -X GET 'http://127.0.0.1:5984/user-dcd6492d74b90967b6b874100b7dbfcf'|json_pp|grep disk_size|cut -d: -f 2`/1024"|bc
+
+ +

Migrating from BigCouch to plain CouchDB

+ +

At the end of this process, you will have just one node with services property equal to couchdb. If you had a BigCouch cluster before, you will be removing all but one of those machines to consolidate them into one CouchDB machine.

+ +
    +

  1. if you have multiple nodes with the couchdb service on them, pick one of them to be your CouchDB server, and remove the service from the others. If these machines were only doing BigCouch before, you can remove the nodes completely with leap node rm <nodename> and then you can decommission the servers

    2. +

  2. put the webapp into maintenance mode

    3. +

  3. turn off daemons that access the database. For example:

    + +
     workstation$ leap ssh <each soledad-node>
+ server# /etc/init.d/soledad-server stop
+
+ workstation$ leap ssh <mx-node>
+ server# /etc/init.d/postfix stop
+ server# /etc/init.d/leap-mx stop
+
+ workstation$ leap ssh <webapp-node>
+ server# /etc/init.d/nickserver stop
+
    + +

    Alternately, you can create a temporary firewall rule to block access (run on couchdb server):

    + +
     server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT
+
    4. +

  4. remove orphaned databases and do a backup of all remaining, active databases. This can take some time and will place several hundred megabytes of data into /var/backups/couchdb. The size and time depends on how many users there are on your system. For example, 15k users took approximately 25 minutes and 308M of space:

    + +
     workstation$ leap ssh <couchdb-node>
+ server# cd /srv/leap/couchdb/scripts
+ server# ./cleanup-user-dbs
+ server# time ./couchdb_dumpall.sh
+
    5. +

  5. stop bigcouch:

    + +
     server# /etc/init.d/bigcouch stop
+ server# pkill epmd
+
    6. +

  6. remove bigcouch:

    + +
     server# apt-get remove bigcouch
+
    7. +

  7. configure your couch node to use plain couchdb instead of bigcouch, you can do this by editing nodes/.json, look for this section:

    + +
     "couch": {
+   "mode": "plain"
+ }
+
    + +

    change it, so it looks like this instead:

    + +
      "couch": {
+    "mode": "plain",
+    "pwhash_alg": "pbkdf2"
+  }
+
    8. +
+ +

+ +

    +

  1. restore the backup, this will take approximately the same amount of time as the backup took above:

    + +
     server# cd /srv/leap/couchdb/scripts
+ server# time ./couchdb_restoreall.sh
+
    2. +

  2. start services again that were stopped in the beginning:

    + +
     workstation$ leap ssh soledad-nodes
+ server# /etc/init.d/soledad-server start
+
+ workstation$ leap ssh mx-node
+ server# /etc/init.d/postfix start
+ server# /etc/init.d/leap-mx start
+
+ workstation$ leap ssh webapp
+ server# /etc/init.d/nickserver start
+
    + +

    Or, alternately, if you set up the firewall rule instead, now remove it:

    + +
     server# iptables -D INPUT -p tcp --dport 5984 --jump REJECT
+
    3. +
+ +

+ +

    +

  1. check if everything is working, including running the test on your deployment machine:

    + +
     workstation$ leap test
+
    2. +

  2. Remove old bigcouch data dir /opt after you double checked everything is in place

    3. +

  3. Relax, enjoy a refreshing beverage.

    4. +
+ +

+ +
+
+ + diff --git a/docs/en/services/couchdb/index.html b/docs/en/services/couchdb/index.html new file mode 100644 index 00000000..7fa6b951 --- /dev/null +++ b/docs/en/services/couchdb/index.html @@ -0,0 +1,328 @@ + + + + +couchdb - LEAP Platform Documentation + + + + + + + + +
+
+

couchdb

+ +
Data storage for all user data.
+
+
+ + +

Topology

+ +

Required:

+ +
    +
  • Nodes with couchdb service must also have soledad service, if email is enabled.
    • +
+ + +

Suggested:

+ +
    +
  • Nodes with couchdb service communicate heavily with webapp and mx.
    • +
+ + +

couchdb nodes do not need to be reachable from the public internet, although the soledad service does require this.

+ +

Configuration

+ +

Nighly dumps

+ +

You can do a nightly couchdb data dump by adding this to your node config:

+ +
"couch": {
+  "backup": true
+}
+
+ +

Data will get dumped to /var/backups/couchdb.

+ +

Plain CouchDB

+ +

BigCouch is not supported on Platform version 0.8 and higher: only plain CouchDB is possible. For earlier versions, you must do this in order to use plain CouchDB:

+ +
"couch": {
+  "master": true,
+  "pwhash_alg": "pbkdf2"
+}
+
+ +

Various Tasks

+ +

Re-enabling blocked account

+ +

When a user account gets destroyed from the webapp, there’s still a leftover doc in the identities db so other people can’t claim that account without an admin’s intervention. You can remove this username reservation through the webapp.

+ +

However, here is how you could do it manually, if you wanted to:

+ +

grep the identities db for the email address:

+ +
curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET http://127.0.0.1:5984/identities/_all_docs?include_docs=true|grep test_127@bitmask.net
+
+ +

lookup “id” and “rev” to delete the doc:

+ +
curl -s --netrc-file /etc/couchdb/couchdb.netrc -X DELETE 'http://127.0.0.1:5984/identities/b25cf10f935b58088f0d547fca823265?rev=2-715a9beba597a2ab01851676f12c3e4a'
+
+ +

How to find out which userstore belongs to which identity?

+ +
/usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep testuser
+
+{"id":"665e004870ee17aa4c94331ff3ecb173","key":"665e004870ee17aa4c94331ff3ecb173","value":{"rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b"},"doc":{"_id":"665e004870ee17aa4c94331ff3ecb173","_rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b","user_id":"665e004870ee17aa4c94331ff3cd59eb","address":"testuser@example.org","destination":"testuser@example.org","keys": ...
+
+ +
    +
  • search for the “user_id” field
    • +
  • in this example testuser@example.org uses the database user-665e004870ee17aa4c94331ff3cd59eb
    • +
+ + +

How much disk space is used by a userstore

+ +

Beware that this returns the uncompacted disk size (see http://wiki.apache.org/couchdb/Compaction)

+ +
echo "`curl --netrc -s -X GET 'http://127.0.0.1:5984/user-dcd6492d74b90967b6b874100b7dbfcf'|json_pp|grep disk_size|cut -d: -f 2`/1024"|bc
+
+ +

Migrating from BigCouch to plain CouchDB

+ +

At the end of this process, you will have just one node with services property equal to couchdb. If you had a BigCouch cluster before, you will be removing all but one of those machines to consolidate them into one CouchDB machine.

+ +
    +

  1. if you have multiple nodes with the couchdb service on them, pick one of them to be your CouchDB server, and remove the service from the others. If these machines were only doing BigCouch before, you can remove the nodes completely with leap node rm <nodename> and then you can decommission the servers

    2. +

  2. put the webapp into maintenance mode

    3. +

  3. turn off daemons that access the database. For example:

    + +
     workstation$ leap ssh <each soledad-node>
+ server# /etc/init.d/soledad-server stop
+
+ workstation$ leap ssh <mx-node>
+ server# /etc/init.d/postfix stop
+ server# /etc/init.d/leap-mx stop
+
+ workstation$ leap ssh <webapp-node>
+ server# /etc/init.d/nickserver stop
+
    + +

    Alternately, you can create a temporary firewall rule to block access (run on couchdb server):

    + +
     server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT
+
    4. +

  4. remove orphaned databases and do a backup of all remaining, active databases. This can take some time and will place several hundred megabytes of data into /var/backups/couchdb. The size and time depends on how many users there are on your system. For example, 15k users took approximately 25 minutes and 308M of space:

    + +
     workstation$ leap ssh <couchdb-node>
+ server# cd /srv/leap/couchdb/scripts
+ server# ./cleanup-user-dbs
+ server# time ./couchdb_dumpall.sh
+
    5. +

  5. stop bigcouch:

    + +
     server# /etc/init.d/bigcouch stop
+ server# pkill epmd
+
    6. +

  6. remove bigcouch:

    + +
     server# apt-get remove bigcouch
+
    7. +

  7. configure your couch node to use plain couchdb instead of bigcouch, you can do this by editing nodes/.json, look for this section:

    + +
     "couch": {
+   "mode": "plain"
+ }
+
    + +

    change it, so it looks like this instead:

    + +
      "couch": {
+    "mode": "plain",
+    "pwhash_alg": "pbkdf2"
+  }
+
    8. +
+ +

+ +

    +

  1. restore the backup, this will take approximately the same amount of time as the backup took above:

    + +
     server# cd /srv/leap/couchdb/scripts
+ server# time ./couchdb_restoreall.sh
+
    2. +

  2. start services again that were stopped in the beginning:

    + +
     workstation$ leap ssh soledad-nodes
+ server# /etc/init.d/soledad-server start
+
+ workstation$ leap ssh mx-node
+ server# /etc/init.d/postfix start
+ server# /etc/init.d/leap-mx start
+
+ workstation$ leap ssh webapp
+ server# /etc/init.d/nickserver start
+
    + +

    Or, alternately, if you set up the firewall rule instead, now remove it:

    + +
     server# iptables -D INPUT -p tcp --dport 5984 --jump REJECT
+
    3. +
+ +

+ +

    +

  1. check if everything is working, including running the test on your deployment machine:

    + +
     workstation$ leap test
+
    2. +

  2. Remove old bigcouch data dir /opt after you double checked everything is in place

    3. +

  3. Relax, enjoy a refreshing beverage.

    4. +
+ +

+ +
+
+ + diff --git a/docs/en/services/index.html b/docs/en/services/index.html new file mode 100644 index 00000000..6d5c68e1 --- /dev/null +++ b/docs/en/services/index.html @@ -0,0 +1,251 @@ + + + + +Services - LEAP Platform Documentation + + + + + + + + +
+
+

Guide to node services

+ +
+
+
+ + +

Introduction

+ +

Every node (server) must have one or more services defined that determines what role the node performs. For example:

+ +
workstation$ cat nodes/stallman.json
+{
+  "ip_address": "199.99.99.1",
+  "services": ["webapp", "tor"]
+}
+
+ +

Here are common questions to ask when adding a new node to your provider:

+ +
    +
  • many or few? Some services benefit from having many nodes, while some services are best run on only one or two nodes.
    • +
  • required or optional? Some services are required, while others can be left out.
    • +
  • who does the node communicate with? Some services communicate very heavily with other particular services. Nodes running these services should be close together.
    • +
  • public or private network? Some services communicate with the public internet, while others only need to communicate with other nodes in the infrastructure.
    • +
+ + +

Available services

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ServiceVPNEmailNotes
webappUser control panel, provider API, and support system.
couchdbData storage for everything. Private node.
soledadUser data synchronization daemon. Usually paired with couchdb nodes.
mxIncoming and outgoing MX servers.
openvpnOpenVPN gateways.
monitorNagios monitoring. This service must be on the webapp node.
torTor exit node.
+ + +

Key: Required, Optional, Not Used

+ +

+

+couchdb +

+
Data storage for all user data.
+
+
+

+openvpn +

+
OpenVPN egress gateways
+
+
+

+monitor +

+
Nagios monitoring and continuous testing.
+
+
+

+mx +

+
Incoming and outgoing MX servers.
+
+
+

+soledad +

+
User data synchronization daemon
+
+
+

+tor +

+
Tor exit node or hidden service
+
+
+

+webapp +

+
leap_web user management application and provider API.
+
+

+ +
+
+ + diff --git a/docs/en/services/monitor.html b/docs/en/services/monitor.html new file mode 100644 index 00000000..5ed2e2fc --- /dev/null +++ b/docs/en/services/monitor.html @@ -0,0 +1,186 @@ + + + + +monitor - LEAP Platform Documentation + + + + + + + + +
+
+

monitor

+ +
Nagios monitoring and continuous testing.
+
+
+ + +

The monitor node provides a nagios control panel that will give you a view into the health and status of all the servers and all the services. It will also spam you with alerts if something goes down.

+ +

Topology

+ +

Currently, you can have zero or one monitor nodes defined. It is required that the monitor be on the webapp node. It was not designed to be run as a separate node service.

+ +

Configuration

+ +
    +
  • nagios.environments: By default, the monitor node will monitor all servers in all environments. You can optionally restrict the environments to the ones you specify.
    • +
+ + +

For example:

+ +
{
+  "nagios": {
+    "environments": ["unstable", "production"]
+  }
+}
+
+ +

Access nagios web

+ +

To open the nagios control panel:

+ +
workstation$ leap open monitor
+
+ +

This will open a web browser window with the appropriate URL, including the nagios username and password.

+ +

If the URL does not open because of HSTS or DNS problems, pass the --ip option to leap.

+ +

If you are using an older version of leap command that doesn’t include leap open, you can determine the nagio parameters manually:

+ +

Step 1. find the domain:

+ +
workstation$ export DOMAIN=$(leap ls --print webapp.domain monitor | grep . | cut -f3 -d' ')
+
+ +

Step 2. find the username:

+ +
workstation$ export USERNAME="nagiosadmin"
+
+ +

Step 3. find the password:

+ +
workstation$ export PASSWORD=$(grep nagios_admin_password secrets.json | cut -f4 -d\")
+
+ +

Step 4. put it all together:

+ +
workstation$ sensible-browser "https://$USERNAME:$PASSWORD@$DOMAIN/nagios3"
+
+ +
+
+ + diff --git a/docs/en/services/monitor/index.html b/docs/en/services/monitor/index.html new file mode 100644 index 00000000..f6a16cdf --- /dev/null +++ b/docs/en/services/monitor/index.html @@ -0,0 +1,186 @@ + + + + +monitor - LEAP Platform Documentation + + + + + + + + +
+
+

monitor

+ +
Nagios monitoring and continuous testing.
+
+
+ + +

The monitor node provides a nagios control panel that will give you a view into the health and status of all the servers and all the services. It will also spam you with alerts if something goes down.

+ +

Topology

+ +

Currently, you can have zero or one monitor nodes defined. It is required that the monitor be on the webapp node. It was not designed to be run as a separate node service.

+ +

Configuration

+ +
    +
  • nagios.environments: By default, the monitor node will monitor all servers in all environments. You can optionally restrict the environments to the ones you specify.
    • +
+ + +

For example:

+ +
{
+  "nagios": {
+    "environments": ["unstable", "production"]
+  }
+}
+
+ +

Access nagios web

+ +

To open the nagios control panel:

+ +
workstation$ leap open monitor
+
+ +

This will open a web browser window with the appropriate URL, including the nagios username and password.

+ +

If the URL does not open because of HSTS or DNS problems, pass the --ip option to leap.

+ +

If you are using an older version of leap command that doesn’t include leap open, you can determine the nagio parameters manually:

+ +

Step 1. find the domain:

+ +
workstation$ export DOMAIN=$(leap ls --print webapp.domain monitor | grep . | cut -f3 -d' ')
+
+ +

Step 2. find the username:

+ +
workstation$ export USERNAME="nagiosadmin"
+
+ +

Step 3. find the password:

+ +
workstation$ export PASSWORD=$(grep nagios_admin_password secrets.json | cut -f4 -d\")
+
+ +

Step 4. put it all together:

+ +
workstation$ sensible-browser "https://$USERNAME:$PASSWORD@$DOMAIN/nagios3"
+
+ +
+
+ + diff --git a/docs/en/services/mx.html b/docs/en/services/mx.html new file mode 100644 index 00000000..0d693204 --- /dev/null +++ b/docs/en/services/mx.html @@ -0,0 +1,168 @@ + + + + +mx - LEAP Platform Documentation + + + + + + + + +
+
+

mx

+ +
Incoming and outgoing MX servers.
+
+
+
    +
  1. + Topology +
    2. +
  2. + Configuration +
      +
    1. + Aliases +
      2. +
    +
    3. +
+ +

Topology

+ +

mx nodes communicate with the public internet, clients, and couchdb nodes.

+ +

Configuration

+ +

Aliases

+ +

Using the mx.aliases property, you can specify your own hard-coded email aliases that precedence over the aliases in the user database. The mx.aliases property consists of a hash, where source address points to one or more destination addresses.

+ +

For example:

+ +

services/mx.json:

+ +
"mx": {
+  "aliases": {
+    "rook": "crow",
+    "robin": "robin@bird.org",
+    "flock": ["junco@bird.org", "robin", "crow"],
+    "chickadee@avian.org": "chickadee@bird.org",
+    "flicker": ["flicker@bird.org", "flicker@deliver.local"]
+  }
+}
+
+ +

This example demonstrates several of the features with mx.aliases:

+ +
    +
  1. alias lists: by specifying an array of destination addresses, as in the case of “flock”, the single email will get copied to each address.
    2. +
  2. chained resolution: alias resolution will recursively continue until there are no more matching aliases. For example, “flock” is resolved to “robin”, which then gets resolved to “robin@bird.org”.
    3. +
  3. virtual domains: by specifying the full domain, as in the case of “chickadee@avian.org”, the alias will work for any domain you want. Of course, the MX record for that domain must point to appropriate MX servers, but otherwise you don’t need to do any additional configuration.
    4. +
  4. local delivery: for testing purposes, it is often useful to copy all incoming mail for a particular address and send those copies to another address. You can do this by adding “@deliver.local” as one of the destination addresses. When “@local.delivery” is found, alias resolution stops and the mail is delivered to that username.
    5. +
+ + +
+
+ + diff --git a/docs/en/services/mx/index.html b/docs/en/services/mx/index.html new file mode 100644 index 00000000..639d9039 --- /dev/null +++ b/docs/en/services/mx/index.html @@ -0,0 +1,168 @@ + + + + +mx - LEAP Platform Documentation + + + + + + + + +
+
+

mx

+ +
Incoming and outgoing MX servers.
+
+
+
    +
  1. + Topology +
    2. +
  2. + Configuration +
      +
    1. + Aliases +
      2. +
    +
    3. +
+ +

Topology

+ +

mx nodes communicate with the public internet, clients, and couchdb nodes.

+ +

Configuration

+ +

Aliases

+ +

Using the mx.aliases property, you can specify your own hard-coded email aliases that precedence over the aliases in the user database. The mx.aliases property consists of a hash, where source address points to one or more destination addresses.

+ +

For example:

+ +

services/mx.json:

+ +
"mx": {
+  "aliases": {
+    "rook": "crow",
+    "robin": "robin@bird.org",
+    "flock": ["junco@bird.org", "robin", "crow"],
+    "chickadee@avian.org": "chickadee@bird.org",
+    "flicker": ["flicker@bird.org", "flicker@deliver.local"]
+  }
+}
+
+ +

This example demonstrates several of the features with mx.aliases:

+ +
    +
  1. alias lists: by specifying an array of destination addresses, as in the case of “flock”, the single email will get copied to each address.
    2. +
  2. chained resolution: alias resolution will recursively continue until there are no more matching aliases. For example, “flock” is resolved to “robin”, which then gets resolved to “robin@bird.org”.
    3. +
  3. virtual domains: by specifying the full domain, as in the case of “chickadee@avian.org”, the alias will work for any domain you want. Of course, the MX record for that domain must point to appropriate MX servers, but otherwise you don’t need to do any additional configuration.
    4. +
  4. local delivery: for testing purposes, it is often useful to copy all incoming mail for a particular address and send those copies to another address. You can do this by adding “@deliver.local” as one of the destination addresses. When “@local.delivery” is found, alias resolution stops and the mail is delivered to that username.
    5. +
+ + +
+
+ + diff --git a/docs/en/services/openvpn.html b/docs/en/services/openvpn.html new file mode 100644 index 00000000..e5fe1128 --- /dev/null +++ b/docs/en/services/openvpn.html @@ -0,0 +1,178 @@ + + + + +openvpn - LEAP Platform Documentation + + + + + + + + +
+
+

openvpn

+ +
OpenVPN egress gateways
+
+
+
    +
  1. + Topology +
    2. +
  2. + Configuration +
    3. +
+ +

Topology

+ +

Currently, openvpn service should not be combined with other services on the same node.

+ +

Unlike most of the other node types, the openvpn nodes do not need access to the database and does not ever communicate with any other nodes (except for the monitor node, if used). So, openvpn nodes can be placed anywhere without regard to the other nodes.

+ +

Configuration

+ +

Essential configuration

+ +
    +
  • openvpn.gateway_address: The address that OpenVPN daemon is bound to and that VPN clients connect to.
    • +
  • ip_address: The main IP of the server, and the egress address for outgoing traffic.
    • +
+ + +

For example:

+ +
{
+  "ip_address": "1.1.1.1",
+  "openvpn": {
+    "gateway_address": "2.2.2.2"
+  }
+}
+
+ +

In this example, VPN clients will connect to 2.2.2.2, but their traffic will appear to come from 1.1.1.1.

+ +

Why are two IP addresses needed? Without this, traffic between two VPN users on the same gateway will not get encrypted. This is because the VPN on every client must be configured to allow cleartext traffic for the IP address that is the VPN gateway.

+ +

Optional configuration

+ +

Here is the default configuration:

+ +
"openvpn": {
+  "configuration": {
+    "auth": "SHA1",
+    "cipher": "AES-128-CBC",
+    "fragment": 1400,
+    "keepalive": "10 30",
+    "tls-cipher": "DHE-RSA-AES128-SHA",
+    "tun-ipv6": true
+  },
+  "ports": ["80", "443", "53", "1194"],
+  "protocols": ["tcp", "udp"]
+}
+
+ +

You may want to change the ports so that only 443 or 80 are used. It is probably best to not modify the openvpn.configuration options for now.

+ +
+
+ + diff --git a/docs/en/services/openvpn/index.html b/docs/en/services/openvpn/index.html new file mode 100644 index 00000000..4a9dc993 --- /dev/null +++ b/docs/en/services/openvpn/index.html @@ -0,0 +1,178 @@ + + + + +openvpn - LEAP Platform Documentation + + + + + + + + +
+
+

openvpn

+ +
OpenVPN egress gateways
+
+
+
    +
  1. + Topology +
    2. +
  2. + Configuration +
    3. +
+ +

Topology

+ +

Currently, openvpn service should not be combined with other services on the same node.

+ +

Unlike most of the other node types, the openvpn nodes do not need access to the database and does not ever communicate with any other nodes (except for the monitor node, if used). So, openvpn nodes can be placed anywhere without regard to the other nodes.

+ +

Configuration

+ +

Essential configuration

+ +
    +
  • openvpn.gateway_address: The address that OpenVPN daemon is bound to and that VPN clients connect to.
    • +
  • ip_address: The main IP of the server, and the egress address for outgoing traffic.
    • +
+ + +

For example:

+ +
{
+  "ip_address": "1.1.1.1",
+  "openvpn": {
+    "gateway_address": "2.2.2.2"
+  }
+}
+
+ +

In this example, VPN clients will connect to 2.2.2.2, but their traffic will appear to come from 1.1.1.1.

+ +

Why are two IP addresses needed? Without this, traffic between two VPN users on the same gateway will not get encrypted. This is because the VPN on every client must be configured to allow cleartext traffic for the IP address that is the VPN gateway.

+ +

Optional configuration

+ +

Here is the default configuration:

+ +
"openvpn": {
+  "configuration": {
+    "auth": "SHA1",
+    "cipher": "AES-128-CBC",
+    "fragment": 1400,
+    "keepalive": "10 30",
+    "tls-cipher": "DHE-RSA-AES128-SHA",
+    "tun-ipv6": true
+  },
+  "ports": ["80", "443", "53", "1194"],
+  "protocols": ["tcp", "udp"]
+}
+
+ +

You may want to change the ports so that only 443 or 80 are used. It is probably best to not modify the openvpn.configuration options for now.

+ +
+
+ + diff --git a/docs/en/services/soledad.html b/docs/en/services/soledad.html new file mode 100644 index 00000000..be372401 --- /dev/null +++ b/docs/en/services/soledad.html @@ -0,0 +1,136 @@ + + + + +soledad - LEAP Platform Documentation + + + + + + + + +
+
+

soledad

+ +
User data synchronization daemon
+
+
+
    +
  1. + Topology +
    2. +
  2. + Configuration +
    3. +
+ +

Topology

+ +

Currently, the platform is designed for soledad and couchdb services to be combined (e.g. every soledad node should also be a couchdb node). soledad nodes might work in isolation, but this is not tested.

+ +

Configuration

+ +

There are no options to configure for soledad nodes.

+ +
+
+ + diff --git a/docs/en/services/soledad/index.html b/docs/en/services/soledad/index.html new file mode 100644 index 00000000..33e72046 --- /dev/null +++ b/docs/en/services/soledad/index.html @@ -0,0 +1,136 @@ + + + + +soledad - LEAP Platform Documentation + + + + + + + + +
+
+

soledad

+ +
User data synchronization daemon
+
+
+
    +
  1. + Topology +
    2. +
  2. + Configuration +
    3. +
+ +

Topology

+ +

Currently, the platform is designed for soledad and couchdb services to be combined (e.g. every soledad node should also be a couchdb node). soledad nodes might work in isolation, but this is not tested.

+ +

Configuration

+ +

There are no options to configure for soledad nodes.

+ +
+
+ + diff --git a/docs/en/services/tor.html b/docs/en/services/tor.html new file mode 100644 index 00000000..f649c086 --- /dev/null +++ b/docs/en/services/tor.html @@ -0,0 +1,161 @@ + + + + +tor - LEAP Platform Documentation + + + + + + + + +
+
+

tor

+ +
Tor exit node or hidden service
+
+
+
    +
  1. + Topology +
    2. +
  2. + Configuration +
    3. +
+ +

Topology

+ +

Nodes with tor service will run a Tor exit or hidden service, depending on what other service it is paired with:

+ +
    +
  • tor + openvpn: when combined with openvpn nodes, tor will create a Tor exit node to provide extra cover traffic for the VPN. This can be especially useful if there are VPN gateways without much traffic.
    • +
  • tor + webapp: when combined with a webapp node, the tor service will make the webapp and the API available via .onion hidden service.
    • +
  • tor stand alone: a regular Tor exit node.
    • +
+ + +

If activated, you can list the hidden service .onion addresses this way:

+ +

leap ls –print tor.hidden_service.address tor

+ +

Then just add ‘.onion’ to the end of the printed addresses.

+ +

Configuration

+ +
    +
  • tor.bandwidth_rate: the max bandwidth allocated to Tor, in KB per second, when used as an exit node.
    • +
+ + +

For example:

+ +
{
+  "tor": {
+    "bandwidth_rate": 6550
+  }
+}
+
+ +
+
+ + diff --git a/docs/en/services/tor/index.html b/docs/en/services/tor/index.html new file mode 100644 index 00000000..8fecf152 --- /dev/null +++ b/docs/en/services/tor/index.html @@ -0,0 +1,161 @@ + + + + +tor - LEAP Platform Documentation + + + + + + + + +
+
+

tor

+ +
Tor exit node or hidden service
+
+
+
    +
  1. + Topology +
    2. +
  2. + Configuration +
    3. +
+ +

Topology

+ +

Nodes with tor service will run a Tor exit or hidden service, depending on what other service it is paired with:

+ +
    +
  • tor + openvpn: when combined with openvpn nodes, tor will create a Tor exit node to provide extra cover traffic for the VPN. This can be especially useful if there are VPN gateways without much traffic.
    • +
  • tor + webapp: when combined with a webapp node, the tor service will make the webapp and the API available via .onion hidden service.
    • +
  • tor stand alone: a regular Tor exit node.
    • +
+ + +

If activated, you can list the hidden service .onion addresses this way:

+ +

leap ls –print tor.hidden_service.address tor

+ +

Then just add ‘.onion’ to the end of the printed addresses.

+ +

Configuration

+ +
    +
  • tor.bandwidth_rate: the max bandwidth allocated to Tor, in KB per second, when used as an exit node.
    • +
+ + +

For example:

+ +
{
+  "tor": {
+    "bandwidth_rate": 6550
+  }
+}
+
+ +
+
+ + diff --git a/docs/en/services/webapp.html b/docs/en/services/webapp.html new file mode 100644 index 00000000..6c853c22 --- /dev/null +++ b/docs/en/services/webapp.html @@ -0,0 +1,479 @@ + + + + +webapp - LEAP Platform Documentation + + + + + + + + +
+
+

webapp

+ +
leap_web user management application and provider API.
+
+
+ + +

Introduction

+ +

The service webapp will install the web application leap_web. It has performs the following functions:

+ +
    +
  • REST API for user registration and authentication via the Bitmask client.
    • +
  • Admin interface to manage users.
    • +
  • Client certificate distribution and renewal.
    • +
  • User support help tickets.
    • +
+ + +

Coming soon:

+ +
    +
  • Billing.
    • +
  • Customizable and localized user documentation.
    • +
+ + +

The leap_web application is written in Ruby on Rails 3, using CouchDB as the backend data store.

+ +

Topology

+ +

Currently, the platform only supports a single webapp node, although we hope to change this in the future.

+ +
    +
  • webapp nodes communicate heavily with couchdb nodes, but the two can be on separate servers.
    • +
  • The monitor service, if enabled, must be on the same node as webapp.
    • +
+ + +

Configuration

+ +

Essential options:

+ +
    +
  • webapp.admin: An array of usernames that will be blessed with administrative permissions. These admins can delete users, answer help tickets, and so on. These usernames are for users that have registered through the webapp or through the Bitmask client application, NOT the sysadmin usernames lists in the provider directory users.
    • +
+ + +

Other options:

+ +
    +
  • webapp.engines: A list of the engines you want enabled in leap_web. Currently, only “support” is available, and it is enabled by default.
    • +
  • webapp.invite_required: If true, registration requires an invite code. Default is false.
    • +
+ + +

For example, services/webapp.json:

+ +
{
+  "webapp": {
+    "admins": ["joehill", "ali", "mack_the_turtle"]
+  }
+}
+
+ +

By putting this in services/webapp.json, all the webapp nodes will inherit the same admin list.

+ +

There are many options in provider.json that also control how the webapp behaves. See Provider Configuration for details.

+ +

Invite codes

+ +

Enabling the invite code functionality will require new users to provide a valid invite code while signing up for a new account. This is turned off by default, allowing all new users to create an account.

+ +

Set the invite_code option to true in services/webapp.json:

+ +
{
+  "webapp": {
+    "invite_required": true
+  }
+}
+
+ +

This only works with LEAP platform 0.8 or higher.

+ +

Run leap deploy to enable the option.

+ +

You can then generate invite codes by logging into the web application with an admin user.

+ +

Alternately, you can also generate invite codes with the command line:

+ +
workstation$ leap ssh bumblebee
+bumblebee# cd /srv/leap/webapp/
+bumblebee# sudo -u leap-webapp RAILS_ENV=production bundle exec rake "generate_invites[NUM,USES]"
+
+ +

Where bumblebee should be replaced with the name of your webapp node.

+ +

The NUM specifies the amount of codes to generate. The USES parameter is optional: By default, all new invite codes can be used once and will then become invalid. If you provide another value for USES, you can set a different amount of maximum uses for the codes you generate.

+ +

Customization

+ +

The provider directory files/webapp can be used to customize the appearance of the webapp. All the files in this directory will get sync'ed to the /srv/leap/webapp/config/customization directory of the deployed webapp node.

+ +

Files in the files/webapp can override view files, locales, and stylesheets in the leap_web app:

+ +

For example:

+ +
stylesheets/ -- override files in Rails.root/app/assets/stylesheets
+  tail.scss -- included before all others
+  head.scss -- included after all others
+
+public/ -- overrides files in Rails.root/public
+  favicon.ico -- custom favicon
+  img/ -- customary directory to put images in
+
+views/ -- overrides files Rails.root/app/views
+  home/
+    index.html.haml -- this file is what shows up on
+                       the home page
+  pages/
+    privacy-policy.en.md -- this file will override
+                            the default privacy policy
+    terms-of-service.en.md -- this file will override
+                              the default TOS.
+
+locales/ -- overrides files in Rails.root/config/locales
+  en.yml -- overrides for English
+  de.yml -- overrides for German
+  and so on...
+
+ +

To interactively develop your customizations before you deploy them, you have two options:

+ +
    +
  1. Edit a webapp node. This approach involves directly modifying the contents of the directory /srv/leap/webapp/config/customization on a deployed webapp node. This can, and probably should be, a “local” node. When doing this, you may need to restart leap_web in order for changes to take effect (touch /srv/leap/webapp/tmp/restart.txt).
    2. +
  2. Alternately, you can install leap_web to run on your computer and edit files in config/customization locally. This approach does not require a provider or a webapp node. For more information, see the leap_web README.
    3. +
+ + +

NOTE: If you add a tails.scss or head.scss file, then you usually need to run rake tmp:clear and restart rails in order for the new stylesheet to get recognized. You should only need to do this once.

+ +

Once you have what you want, then copy these files to the local provider directory files/webapp so that they will be installed each time you deploy.

+ +

Customization tutorial

+ +

This mini-tutorial will walk you through creating a custom “branding” of the leap_web application. We will be creating a provider called “Prehistoric Computer.”

+ +

Here are the files we are going to create:

+ +
leap_web/config/customization
+├── locales
+│   ├── en.yml
+│   └── es.yml
+├── public
+│   ├── favicon.ico
+│   └── img
+│       └── masthead.png
+├── stylesheets
+│   └── tail.scss
+└── views
+    └── pages
+        ├── privacy-policy.en.md
+        └── privacy-policy.es.md
+
+ +

All these files are available in the source code in the customization.example directory.

+ +

Remember, these files may live different places:

+ +
    +
  • user@localmachine$ leap_web/config/customization: This will be the path if you have checked out a local copy of leap_web.git and are running rails server locally in order to test your customizations.
    • +
  • user@localmachine$ PROVIDER/files/webapp: This is the local provider directory where the files should be put so that they get correctly deployed to webapp nodes.
    • +
  • root@webappnode# /srv/leap/webapp/config/customization: This is where the files in the local provider directory PROVIDER/files/webapp get copied to after a leap deploy to a live webapp nodes.
    • +
+ + +

Override translations

+ +

You can add additional locale files in order to change the text used in the existing application and to add translations for string that you added to the application.

+ +

In this example, we will be altering the default text for the “login_info” string. In config/locales/en/home.en.yml there is this entry:

+ +
en:
+  login_info: "Log in to change your account settings, create support tickets, and manage payments."
+
+ +

We are going to override this with some custom text in English and Spanish:

+ +

leap_web/config/customization/locale/en.yml:

+ +
en:
+  login_info: Authenticate to change your "Prehistoric Computer" settings.
+
+ +

leap_web/config/customization/locale/es.yml:

+ +
es:
+  login_info: Autenticar a cambiar la configuración de "Computer Prehistoria."
+
+ +

Now, the home page of leap_web will use these new strings instead of the default. Remember that you must restart rails in order for new locale files to take effect.

+ +

Override static pages

+ +

You can also override any of the static files included with leap_web, such as the privacy policy or terms of service.

+ +

Here is how we would create a custom privacy policy in English and Spanish:

+ +

leap_web/config/customization/views/pages/privacy-policy.en.md:

+ +
# Custom Privacy Policy
+This is our privacy policy.
+
+ +

leap_web/config/customization/views/pages/privacy-policy.es.md:

+ +
# Custom Política de Privacidad
+Esta es nuestra política de privacidad.
+
+ +

Add a custom header

+ +

Now we will add a custom header to every page. First, we add the images:

+ +
leap_web/config/customization
+    ├── public
+        ├── favicon.ico
+        └── img
+            └── masthead.png
+
+ +

You can create your own, or use the example files in https://github.com/leapcode/leap_web/tree/develop/config/customization.example

+ +

Now, we add some custom CSS so that we can style the masthead:

+ +

leap_web/config/customization/stylesheets/tail.scss

+ +
$custom-color: #66bbaa;
+
+a {
+  color: $custom-color;
+}
+
+//
+// MASTHEAD
+//
+
+#masthead {
+  background-color: $custom-color;
+  border-bottom: none;
+
+  // make the masthead clickable by replacing the
+  // site name link with the masthead image:
+  .title {
+    padding: 0px;
+    .sitename a {
+      display: block;
+      background: url(/img/masthead.png) 0 0 no-repeat;
+      font-size: 0px;
+      height: 100px;
+      background-size: auto 100px;
+    }
+  }
+}
+
+// make the home page masthead slightly larger
+body.home #masthead {
+  .sitename a {
+    height: 150px;
+    background-size: auto 150px;
+  }
+}
+
+//
+// FOOTER
+//
+
+#footer .links {
+  background-color: $custom-color;
+}
+
+ +

NOTE: If you add a tails.scss or head.scss file, then you usually need to run rake tmp:clear and restart rails in order for the new stylesheet to get recognized. You should only need to do this once.

+ +

Custom Fork

+ +

Sometimes it is easier to maintain your own fork of the leap_web app. You can keep your customizations in that fork instead of in the provider files/webapp directory. Or, perhaps you want to add an engine to the application that modifies the app’s behavior.

+ +

To deploy your own leap_web, modify the provider file common.json:

+ +
{
+  "sources": {
+    "webapp": {
+      "revision": "origin/develop",
+      "source": "https://github.com/leapcode/leap_web",
+      "type": "git"
+    }
+  }
+}
+
+ +

To target only particular environment, modify instead common.ENV.json, where ENV is the name of the environment.

+ +

See https://github.com/leapcode/leap_web/blob/develop/doc/DEVELOP.md for notes on getting started hacking on leap_web.

+ +

Maintenance mode

+ +

You can put the webapp into maintenance mode by simply dropping a html file to /srv/leap/webapp/public/system/maintenance.html. For example:

+ +
workstation$ leap ssh webappnode
+server# echo "Temporarily down for maintenance. We will be back soon." > /srv/leap/webapp/public/system/maintenance.html
+
+ +

Known problems

+ +
    +
  • Client certificates are generated without a CSR. The problem is that this makes the web +application extremely vulnerable to denial of service attacks. This was not an issue until we +started to allow the possibility of anonymously fetching a client certificate without +authenticating first.
    • +
  • By its very nature, the user database is vulnerable to enumeration attacks. These are +very hard to prevent, because our protocol is designed to allow query of a user database via +proxy in order to provide network perspective.
    • +
+ + +
+
+ + diff --git a/docs/en/services/webapp/index.html b/docs/en/services/webapp/index.html new file mode 100644 index 00000000..acdc098c --- /dev/null +++ b/docs/en/services/webapp/index.html @@ -0,0 +1,479 @@ + + + + +webapp - LEAP Platform Documentation + + + + + + + + +
+
+

webapp

+ +
leap_web user management application and provider API.
+
+
+ + +

Introduction

+ +

The service webapp will install the web application leap_web. It has performs the following functions:

+ +
    +
  • REST API for user registration and authentication via the Bitmask client.
    • +
  • Admin interface to manage users.
    • +
  • Client certificate distribution and renewal.
    • +
  • User support help tickets.
    • +
+ + +

Coming soon:

+ +
    +
  • Billing.
    • +
  • Customizable and localized user documentation.
    • +
+ + +

The leap_web application is written in Ruby on Rails 3, using CouchDB as the backend data store.

+ +

Topology

+ +

Currently, the platform only supports a single webapp node, although we hope to change this in the future.

+ +
    +
  • webapp nodes communicate heavily with couchdb nodes, but the two can be on separate servers.
    • +
  • The monitor service, if enabled, must be on the same node as webapp.
    • +
+ + +

Configuration

+ +

Essential options:

+ +
    +
  • webapp.admin: An array of usernames that will be blessed with administrative permissions. These admins can delete users, answer help tickets, and so on. These usernames are for users that have registered through the webapp or through the Bitmask client application, NOT the sysadmin usernames lists in the provider directory users.
    • +
+ + +

Other options:

+ +
    +
  • webapp.engines: A list of the engines you want enabled in leap_web. Currently, only “support” is available, and it is enabled by default.
    • +
  • webapp.invite_required: If true, registration requires an invite code. Default is false.
    • +
+ + +

For example, services/webapp.json:

+ +
{
+  "webapp": {
+    "admins": ["joehill", "ali", "mack_the_turtle"]
+  }
+}
+
+ +

By putting this in services/webapp.json, all the webapp nodes will inherit the same admin list.

+ +

There are many options in provider.json that also control how the webapp behaves. See Provider Configuration for details.

+ +

Invite codes

+ +

Enabling the invite code functionality will require new users to provide a valid invite code while signing up for a new account. This is turned off by default, allowing all new users to create an account.

+ +

Set the invite_code option to true in services/webapp.json:

+ +
{
+  "webapp": {
+    "invite_required": true
+  }
+}
+
+ +

This only works with LEAP platform 0.8 or higher.

+ +

Run leap deploy to enable the option.

+ +

You can then generate invite codes by logging into the web application with an admin user.

+ +

Alternately, you can also generate invite codes with the command line:

+ +
workstation$ leap ssh bumblebee
+bumblebee# cd /srv/leap/webapp/
+bumblebee# sudo -u leap-webapp RAILS_ENV=production bundle exec rake "generate_invites[NUM,USES]"
+
+ +

Where bumblebee should be replaced with the name of your webapp node.

+ +

The NUM specifies the amount of codes to generate. The USES parameter is optional: By default, all new invite codes can be used once and will then become invalid. If you provide another value for USES, you can set a different amount of maximum uses for the codes you generate.

+ +

Customization

+ +

The provider directory files/webapp can be used to customize the appearance of the webapp. All the files in this directory will get sync'ed to the /srv/leap/webapp/config/customization directory of the deployed webapp node.

+ +

Files in the files/webapp can override view files, locales, and stylesheets in the leap_web app:

+ +

For example:

+ +
stylesheets/ -- override files in Rails.root/app/assets/stylesheets
+  tail.scss -- included before all others
+  head.scss -- included after all others
+
+public/ -- overrides files in Rails.root/public
+  favicon.ico -- custom favicon
+  img/ -- customary directory to put images in
+
+views/ -- overrides files Rails.root/app/views
+  home/
+    index.html.haml -- this file is what shows up on
+                       the home page
+  pages/
+    privacy-policy.en.md -- this file will override
+                            the default privacy policy
+    terms-of-service.en.md -- this file will override
+                              the default TOS.
+
+locales/ -- overrides files in Rails.root/config/locales
+  en.yml -- overrides for English
+  de.yml -- overrides for German
+  and so on...
+
+ +

To interactively develop your customizations before you deploy them, you have two options:

+ +
    +
  1. Edit a webapp node. This approach involves directly modifying the contents of the directory /srv/leap/webapp/config/customization on a deployed webapp node. This can, and probably should be, a “local” node. When doing this, you may need to restart leap_web in order for changes to take effect (touch /srv/leap/webapp/tmp/restart.txt).
    2. +
  2. Alternately, you can install leap_web to run on your computer and edit files in config/customization locally. This approach does not require a provider or a webapp node. For more information, see the leap_web README.
    3. +
+ + +

NOTE: If you add a tails.scss or head.scss file, then you usually need to run rake tmp:clear and restart rails in order for the new stylesheet to get recognized. You should only need to do this once.

+ +

Once you have what you want, then copy these files to the local provider directory files/webapp so that they will be installed each time you deploy.

+ +

Customization tutorial

+ +

This mini-tutorial will walk you through creating a custom “branding” of the leap_web application. We will be creating a provider called “Prehistoric Computer.”

+ +

Here are the files we are going to create:

+ +
leap_web/config/customization
+├── locales
+│   ├── en.yml
+│   └── es.yml
+├── public
+│   ├── favicon.ico
+│   └── img
+│       └── masthead.png
+├── stylesheets
+│   └── tail.scss
+└── views
+    └── pages
+        ├── privacy-policy.en.md
+        └── privacy-policy.es.md
+
+ +

All these files are available in the source code in the customization.example directory.

+ +

Remember, these files may live different places:

+ +
    +
  • user@localmachine$ leap_web/config/customization: This will be the path if you have checked out a local copy of leap_web.git and are running rails server locally in order to test your customizations.
    • +
  • user@localmachine$ PROVIDER/files/webapp: This is the local provider directory where the files should be put so that they get correctly deployed to webapp nodes.
    • +
  • root@webappnode# /srv/leap/webapp/config/customization: This is where the files in the local provider directory PROVIDER/files/webapp get copied to after a leap deploy to a live webapp nodes.
    • +
+ + +

Override translations

+ +

You can add additional locale files in order to change the text used in the existing application and to add translations for string that you added to the application.

+ +

In this example, we will be altering the default text for the “login_info” string. In config/locales/en/home.en.yml there is this entry:

+ +
en:
+  login_info: "Log in to change your account settings, create support tickets, and manage payments."
+
+ +

We are going to override this with some custom text in English and Spanish:

+ +

leap_web/config/customization/locale/en.yml:

+ +
en:
+  login_info: Authenticate to change your "Prehistoric Computer" settings.
+
+ +

leap_web/config/customization/locale/es.yml:

+ +
es:
+  login_info: Autenticar a cambiar la configuración de "Computer Prehistoria."
+
+ +

Now, the home page of leap_web will use these new strings instead of the default. Remember that you must restart rails in order for new locale files to take effect.

+ +

Override static pages

+ +

You can also override any of the static files included with leap_web, such as the privacy policy or terms of service.

+ +

Here is how we would create a custom privacy policy in English and Spanish:

+ +

leap_web/config/customization/views/pages/privacy-policy.en.md:

+ +
# Custom Privacy Policy
+This is our privacy policy.
+
+ +

leap_web/config/customization/views/pages/privacy-policy.es.md:

+ +
# Custom Política de Privacidad
+Esta es nuestra política de privacidad.
+
+ +

Add a custom header

+ +

Now we will add a custom header to every page. First, we add the images:

+ +
leap_web/config/customization
+    ├── public
+        ├── favicon.ico
+        └── img
+            └── masthead.png
+
+ +

You can create your own, or use the example files in https://github.com/leapcode/leap_web/tree/develop/config/customization.example

+ +

Now, we add some custom CSS so that we can style the masthead:

+ +

leap_web/config/customization/stylesheets/tail.scss

+ +
$custom-color: #66bbaa;
+
+a {
+  color: $custom-color;
+}
+
+//
+// MASTHEAD
+//
+
+#masthead {
+  background-color: $custom-color;
+  border-bottom: none;
+
+  // make the masthead clickable by replacing the
+  // site name link with the masthead image:
+  .title {
+    padding: 0px;
+    .sitename a {
+      display: block;
+      background: url(/img/masthead.png) 0 0 no-repeat;
+      font-size: 0px;
+      height: 100px;
+      background-size: auto 100px;
+    }
+  }
+}
+
+// make the home page masthead slightly larger
+body.home #masthead {
+  .sitename a {
+    height: 150px;
+    background-size: auto 150px;
+  }
+}
+
+//
+// FOOTER
+//
+
+#footer .links {
+  background-color: $custom-color;
+}
+
+ +

NOTE: If you add a tails.scss or head.scss file, then you usually need to run rake tmp:clear and restart rails in order for the new stylesheet to get recognized. You should only need to do this once.

+ +

Custom Fork

+ +

Sometimes it is easier to maintain your own fork of the leap_web app. You can keep your customizations in that fork instead of in the provider files/webapp directory. Or, perhaps you want to add an engine to the application that modifies the app’s behavior.

+ +

To deploy your own leap_web, modify the provider file common.json:

+ +
{
+  "sources": {
+    "webapp": {
+      "revision": "origin/develop",
+      "source": "https://github.com/leapcode/leap_web",
+      "type": "git"
+    }
+  }
+}
+
+ +

To target only particular environment, modify instead common.ENV.json, where ENV is the name of the environment.

+ +

See https://github.com/leapcode/leap_web/blob/develop/doc/DEVELOP.md for notes on getting started hacking on leap_web.

+ +

Maintenance mode

+ +

You can put the webapp into maintenance mode by simply dropping a html file to /srv/leap/webapp/public/system/maintenance.html. For example:

+ +
workstation$ leap ssh webappnode
+server# echo "Temporarily down for maintenance. We will be back soon." > /srv/leap/webapp/public/system/maintenance.html
+
+ +

Known problems

+ +
    +
  • Client certificates are generated without a CSR. The problem is that this makes the web +application extremely vulnerable to denial of service attacks. This was not an issue until we +started to allow the possibility of anonymously fetching a client certificate without +authenticating first.
    • +
  • By its very nature, the user database is vulnerable to enumeration attacks. These are +very hard to prevent, because our protocol is designed to allow query of a user database via +proxy in order to provide network perspective.
    • +
+ + +
+
+ + diff --git a/docs/en/troubleshooting.html b/docs/en/troubleshooting.html new file mode 100644 index 00000000..916cf229 --- /dev/null +++ b/docs/en/troubleshooting.html @@ -0,0 +1,129 @@ + + + + +Troubleshooting - LEAP Platform Documentation + + + + + + + + +
+
+

Troubleshooting

+ +
The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment.
+
+
+
    +
+ +
+

+ Tests and Monitoring +

+
Testing and monitoring your infrastructure.
+
+
+

+ Known issues +

+
Known issues in the Leap Platform.
+
+
+

+ Where to look +

+
The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment.
+
+ +
+
+ + diff --git a/docs/en/troubleshooting/known-issues.html b/docs/en/troubleshooting/known-issues.html new file mode 100644 index 00000000..607970b1 --- /dev/null +++ b/docs/en/troubleshooting/known-issues.html @@ -0,0 +1,238 @@ + + + + +Known issues - LEAP Platform Documentation + + + + + + + + +
+
+

Leap Platform Release Notes

+ +
Known issues in the Leap Platform.
+
+
+
    +
  1. + 0.6.0 +
      +
    1. + Upgrading +
      2. +
    2. + OpenVPN +
      3. +
    3. + CouchDB +
      4. +
    4. + User setup and ssh +
      5. +
    5. + Deploying +
      6. +
    6. + IPv6 +
      7. +
    7. + Special Environments +
      8. +
    +
    2. +
+ +

Here you can find documentation about known issues and potential work-arounds in the current Leap Platform release.

+ +

0.6.0

+ +

Upgrading

+ +

Upgrade your leap_platform to 0.6 and make sure you have the latest leap_cli.

+ +

Update leap_platform:

+ +
cd leap_platform
+git pull
+git checkout -b 0.6.0 0.6.0 
+
+ +

Update leap_cli:

+ +

If it is installed as a gem from rubygems:

+ +
sudo gem update leap_cli
+
+ +

If it is installed as a gem from source:

+ +
cd leap_cli
+git pull
+git checkout master
+rake build
+sudo rake install
+
+ +

If it is run directly from source:

+ +
cd leap_cli
+git pull
+git checkout master
+
+ +

To upgrade:

+ +
leap --version  # must be at least 1.6.2
+leap cert update
+leap deploy
+leap test
+
+ +

If the tests fail, try deploying again. If a test fails because there are two tapicero daemons running, you need to ssh into the server, kill all the tapicero daemons manually, and then try deploying again (sometimes the daemon from platform 0.5 would put its PID file in an odd place).

+ +

OpenVPN

+ +

On deployment to a openvpn node, if the following happens:

+ +
- err: /Stage[main]/Site_openvpn/Service[openvpn]/ensure: change from stopped to running failed: Could not start Service[openvpn]: Execution of '/etc/init.d/openvpn start' returned 1:  at /srv/leap/puppet/modules/site_openvpn/manifests/init.pp:189
+
+ +

this is likely the result of a kernel upgrade that happened during the deployment, requiring that the machine be restarted before this service can start. To confirm this, login to the node (leap ssh ) and look at the end of the /var/log/daemon.log:

+ +
# tail /var/log/daemon.log
+Nov 22 19:04:15 snail ovpn-udp_config[16173]: ERROR: Cannot open TUN/TAP dev /dev/net/tun: No such device (errno=19)
+Nov 22 19:04:15 snail ovpn-udp_config[16173]: Exiting due to fatal error
+
+ +

if you see this error, simply restart the node.

+ +

CouchDB

+ +

At the moment, we only support one couchdb server for stability purposes.

+ +

User setup and ssh

+ +

At the moment, it is only possible to add an admin who will have access to all LEAP servers (see: https://leap.se/code/issues/2280)

+ +

The command leap add-user --self allows only one SSH key. If you want to specify more than one key for a user, you can do it manually:

+ +
users/userx/userx_ssh.pub
+users/userx/otherkey_ssh.pub
+
+ +

All keys matching ‘userx/*_ssh.pub’ will be used for that user.

+ +

Deploying

+ +

If you have any errors during a run, please try to deploy again as this often solves non-deterministic issues that were not uncovered in our testing. Please re-deploy with leap -v2 deploy to get more verbose logs and capture the complete output to provide to us for debugging.

+ +

If when deploying your debian mirror fails for some reason, network anomoly or the mirror itself is out of date, then platform deployment will not succeed properly. Check the mirror is up and try to deploy again when it is resolved (see: https://leap.se/code/issues/1091)

+ +

Deployment gives ‘error: in %: too few arguments (ArgumentError)’ - this is because you attempted to do a deploy before initializing a node, please initialize the node first and then do a deploy afterwards (see: https://leap.se/code/issues/2550)

+ +

This release has no ability to custom configure apt sources or proxies (see: https://leap.se/code/issues/1971)

+ +

When running a deploy at a verbosity level of 2 and above, you will notice puppet deprecation warnings, these are known and we are working on fixing them

+ +

IPv6

+ +

As of this release, IPv6 is not supported by the VPN configuration. If IPv6 is detected on your network as a client, it is blocked and instead it should revert to IPv4. We plan on adding IPv6 support in an upcoming release.

+ +

Special Environments

+ +

When deploying to OpenStack release “nova” or newer, you will need to do an initial deploy, then when it has finished run leap facts update and then deploy again (see: https://leap.se/code/issues/3020)

+ +

It is not possible to actually use the EIP openvpn server on vagrant nodes (see: https://leap.se/code/issues/2401)

+ +
+
+ + diff --git a/docs/en/troubleshooting/known-issues/index.html b/docs/en/troubleshooting/known-issues/index.html new file mode 100644 index 00000000..eee3b120 --- /dev/null +++ b/docs/en/troubleshooting/known-issues/index.html @@ -0,0 +1,238 @@ + + + + +Known issues - LEAP Platform Documentation + + + + + + + + +
+
+

Leap Platform Release Notes

+ +
Known issues in the Leap Platform.
+
+
+
    +
  1. + 0.6.0 +
      +
    1. + Upgrading +
      2. +
    2. + OpenVPN +
      3. +
    3. + CouchDB +
      4. +
    4. + User setup and ssh +
      5. +
    5. + Deploying +
      6. +
    6. + IPv6 +
      7. +
    7. + Special Environments +
      8. +
    +
    2. +
+ +

Here you can find documentation about known issues and potential work-arounds in the current Leap Platform release.

+ +

0.6.0

+ +

Upgrading

+ +

Upgrade your leap_platform to 0.6 and make sure you have the latest leap_cli.

+ +

Update leap_platform:

+ +
cd leap_platform
+git pull
+git checkout -b 0.6.0 0.6.0 
+
+ +

Update leap_cli:

+ +

If it is installed as a gem from rubygems:

+ +
sudo gem update leap_cli
+
+ +

If it is installed as a gem from source:

+ +
cd leap_cli
+git pull
+git checkout master
+rake build
+sudo rake install
+
+ +

If it is run directly from source:

+ +
cd leap_cli
+git pull
+git checkout master
+
+ +

To upgrade:

+ +
leap --version  # must be at least 1.6.2
+leap cert update
+leap deploy
+leap test
+
+ +

If the tests fail, try deploying again. If a test fails because there are two tapicero daemons running, you need to ssh into the server, kill all the tapicero daemons manually, and then try deploying again (sometimes the daemon from platform 0.5 would put its PID file in an odd place).

+ +

OpenVPN

+ +

On deployment to a openvpn node, if the following happens:

+ +
- err: /Stage[main]/Site_openvpn/Service[openvpn]/ensure: change from stopped to running failed: Could not start Service[openvpn]: Execution of '/etc/init.d/openvpn start' returned 1:  at /srv/leap/puppet/modules/site_openvpn/manifests/init.pp:189
+
+ +

this is likely the result of a kernel upgrade that happened during the deployment, requiring that the machine be restarted before this service can start. To confirm this, login to the node (leap ssh ) and look at the end of the /var/log/daemon.log:

+ +
# tail /var/log/daemon.log
+Nov 22 19:04:15 snail ovpn-udp_config[16173]: ERROR: Cannot open TUN/TAP dev /dev/net/tun: No such device (errno=19)
+Nov 22 19:04:15 snail ovpn-udp_config[16173]: Exiting due to fatal error
+
+ +

if you see this error, simply restart the node.

+ +

CouchDB

+ +

At the moment, we only support one couchdb server for stability purposes.

+ +

User setup and ssh

+ +

At the moment, it is only possible to add an admin who will have access to all LEAP servers (see: https://leap.se/code/issues/2280)

+ +

The command leap add-user --self allows only one SSH key. If you want to specify more than one key for a user, you can do it manually:

+ +
users/userx/userx_ssh.pub
+users/userx/otherkey_ssh.pub
+
+ +

All keys matching ‘userx/*_ssh.pub’ will be used for that user.

+ +

Deploying

+ +

If you have any errors during a run, please try to deploy again as this often solves non-deterministic issues that were not uncovered in our testing. Please re-deploy with leap -v2 deploy to get more verbose logs and capture the complete output to provide to us for debugging.

+ +

If when deploying your debian mirror fails for some reason, network anomoly or the mirror itself is out of date, then platform deployment will not succeed properly. Check the mirror is up and try to deploy again when it is resolved (see: https://leap.se/code/issues/1091)

+ +

Deployment gives ‘error: in %: too few arguments (ArgumentError)’ - this is because you attempted to do a deploy before initializing a node, please initialize the node first and then do a deploy afterwards (see: https://leap.se/code/issues/2550)

+ +

This release has no ability to custom configure apt sources or proxies (see: https://leap.se/code/issues/1971)

+ +

When running a deploy at a verbosity level of 2 and above, you will notice puppet deprecation warnings, these are known and we are working on fixing them

+ +

IPv6

+ +

As of this release, IPv6 is not supported by the VPN configuration. If IPv6 is detected on your network as a client, it is blocked and instead it should revert to IPv4. We plan on adding IPv6 support in an upcoming release.

+ +

Special Environments

+ +

When deploying to OpenStack release “nova” or newer, you will need to do an initial deploy, then when it has finished run leap facts update and then deploy again (see: https://leap.se/code/issues/3020)

+ +

It is not possible to actually use the EIP openvpn server on vagrant nodes (see: https://leap.se/code/issues/2401)

+ +
+
+ + diff --git a/docs/en/troubleshooting/tests.html b/docs/en/troubleshooting/tests.html new file mode 100644 index 00000000..e4c2fdc2 --- /dev/null +++ b/docs/en/troubleshooting/tests.html @@ -0,0 +1,201 @@ + + + + +Tests and Monitoring - LEAP Platform Documentation + + + + + + + + +
+
+

Tests and Monitoring

+ +
Testing and monitoring your infrastructure.
+
+
+ + +

Troubleshooting Tests

+ +

At any time, you can run troubleshooting tests on the nodes of your provider infrastructure to check to see if things seem to be working correctly. If there is a problem, these tests should help you narrow down precisely where the problem is.

+ +

To run tests on FILTER node list:

+ +
workstation$ leap test run FILTER
+
+ +

For example, you can also test a single node (leap test elephant); test a specific environment (leap test development), or any tag (leap test soledad).

+ +

Alternately, you can run test on all nodes (probably only useful if you have pinned the environment):

+ +
workstation$ leap test
+
+ +

The tests that are performed are located in the platform under the tests directory.

+ +

Testing with the bitmask client

+ +

Download the provider ca:

+ +
wget --no-check-certificate https://example.org/ca.crt -O /tmp/ca.crt
+
+ +

Start bitmask:

+ +
bitmask --ca-cert-file /tmp/ca.crt
+
+ +

Testing Recieving Mail

+ +

Use i.e. swaks to send a testmail

+ +
swaks -f noone@example.org -t testuser@example.org -s example.org
+
+ +

and use your favorite mail client to examine your inbox.

+ +

You can also use offlineimap to fetch mails:

+ +
 offlineimap -c vagrant/.offlineimaprc.example.org
+
+ +

WARNING: Use offlineimap only for testing/debugging, +because it will save the mails decrypted locally to +your disk !

+ +

Monitoring

+ +

In order to set up a monitoring node, you simply add a monitor service tag to the node configuration file. It could be combined with any other service, but we propose that you add it to the webapp node, as this already is public accessible via HTTPS.

+ +

After deploying, this node will regularly poll every node to ask for the status of various health checks. These health checks include the checks run with leap test, plus many others.

+ +

We use Nagios together with Check MK agent for running checks on remote hosts.

+ +

One nagios installation will monitor all nodes in all your environments. You can log into the monitoring web interface via https://DOMAIN/nagios3/. The username is nagiosadmin and the password is found in the secrets.json file in your provider directory. +Nagios will send out mails to the contacts address provided in provider.json.

+ +

Nagios Frontends

+ +

There are other ways to check and get notified by Nagios besides regularly checking the Nagios webinterface or reading email notifications. Check out the Frontends (GUIs and CLIs) on the Nagios project website. +A recommended status tray application is Nagstamon, which is available for Linux, MacOS X and Windows. It can not only notify you of hosts/services failures, you can also acknowledge or recheck them.

+ +

Log Monitoring

+ +

At the moment, we use check-mk-agent-logwatch for searching logs for irregularities. +Logs are parsed for patterns using a blacklist, and are stored in /var/lib/check_mk/logwatch/<Nodename>.

+ +

In order to “acknowledge” a log warning, you need to log in to the monitoring server, and delete the corresponding file in /var/lib/check_mk/logwatch/<Nodename>. This should be done via the nagios webinterface in the future.

+ +
+
+ + diff --git a/docs/en/troubleshooting/tests/index.html b/docs/en/troubleshooting/tests/index.html new file mode 100644 index 00000000..f46eddc7 --- /dev/null +++ b/docs/en/troubleshooting/tests/index.html @@ -0,0 +1,201 @@ + + + + +Tests and Monitoring - LEAP Platform Documentation + + + + + + + + +
+
+

Tests and Monitoring

+ +
Testing and monitoring your infrastructure.
+
+
+ + +

Troubleshooting Tests

+ +

At any time, you can run troubleshooting tests on the nodes of your provider infrastructure to check to see if things seem to be working correctly. If there is a problem, these tests should help you narrow down precisely where the problem is.

+ +

To run tests on FILTER node list:

+ +
workstation$ leap test run FILTER
+
+ +

For example, you can also test a single node (leap test elephant); test a specific environment (leap test development), or any tag (leap test soledad).

+ +

Alternately, you can run test on all nodes (probably only useful if you have pinned the environment):

+ +
workstation$ leap test
+
+ +

The tests that are performed are located in the platform under the tests directory.

+ +

Testing with the bitmask client

+ +

Download the provider ca:

+ +
wget --no-check-certificate https://example.org/ca.crt -O /tmp/ca.crt
+
+ +

Start bitmask:

+ +
bitmask --ca-cert-file /tmp/ca.crt
+
+ +

Testing Recieving Mail

+ +

Use i.e. swaks to send a testmail

+ +
swaks -f noone@example.org -t testuser@example.org -s example.org
+
+ +

and use your favorite mail client to examine your inbox.

+ +

You can also use offlineimap to fetch mails:

+ +
 offlineimap -c vagrant/.offlineimaprc.example.org
+
+ +

WARNING: Use offlineimap only for testing/debugging, +because it will save the mails decrypted locally to +your disk !

+ +

Monitoring

+ +

In order to set up a monitoring node, you simply add a monitor service tag to the node configuration file. It could be combined with any other service, but we propose that you add it to the webapp node, as this already is public accessible via HTTPS.

+ +

After deploying, this node will regularly poll every node to ask for the status of various health checks. These health checks include the checks run with leap test, plus many others.

+ +

We use Nagios together with Check MK agent for running checks on remote hosts.

+ +

One nagios installation will monitor all nodes in all your environments. You can log into the monitoring web interface via https://DOMAIN/nagios3/. The username is nagiosadmin and the password is found in the secrets.json file in your provider directory. +Nagios will send out mails to the contacts address provided in provider.json.

+ +

Nagios Frontends

+ +

There are other ways to check and get notified by Nagios besides regularly checking the Nagios webinterface or reading email notifications. Check out the Frontends (GUIs and CLIs) on the Nagios project website. +A recommended status tray application is Nagstamon, which is available for Linux, MacOS X and Windows. It can not only notify you of hosts/services failures, you can also acknowledge or recheck them.

+ +

Log Monitoring

+ +

At the moment, we use check-mk-agent-logwatch for searching logs for irregularities. +Logs are parsed for patterns using a blacklist, and are stored in /var/lib/check_mk/logwatch/<Nodename>.

+ +

In order to “acknowledge” a log warning, you need to log in to the monitoring server, and delete the corresponding file in /var/lib/check_mk/logwatch/<Nodename>. This should be done via the nagios webinterface in the future.

+ +
+
+ + diff --git a/docs/en/troubleshooting/where-to-look.html b/docs/en/troubleshooting/where-to-look.html new file mode 100644 index 00000000..a1207aca --- /dev/null +++ b/docs/en/troubleshooting/where-to-look.html @@ -0,0 +1,451 @@ + + + + +Where to look - LEAP Platform Documentation + + + + + + + + +
+
+

Where to look for errors

+ +
The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment.
+
+
+ + +

General

+ +
    +
  • Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. -v 5 after the leap cmd, i.e. leap -v 2 deploy.
    • +
  • We use the example.org domain for documentation purposes here, please replace it with the you domain.
    • +
+ + +

Firewall

+ +

Every node in your provider has its own restrictive firewall, but you might have a network firewall in place as well that is not managed by LEAP platform. To see what ports and addresses must be open, run this command:

+ +
workstation$ leap compile firewall
+
+ +

If any of those are blocked, then your provider will not work.

+ +

Webapp

+ +

Places to look for errors

+ +
    +
  • /var/log/apache2/error.log
    • +
  • /srv/leap/webapp/log/production.log
    • +
  • /var/log/syslog (watch out for stunnel issues)
    • +
  • /var/log/leap/*
    • +
+ + +

Is haproxy ok ?

+ +
curl -s -X  GET "http://127.0.0.1:4096"
+
+ +

Is couchdb accessible through stunnel ?

+ +
    +

  • Depending on how many couch nodes you have, increase the port for every test +(see /etc/haproxy/haproxy.cfg for the server/port mapping):

    + +

    curl -s -X GET “http://127.0.0.1:4000” + curl -s -X GET “http://127.0.0.1:4001” + …

    • +
+ + +

Check couchdb acl as admin

+ +
mkdir /etc/couchdb
+cat /srv/leap/webapp/config/couchdb.yml.admin  # see username and password
+echo "machine 127.0.0.1 login admin password <PASSWORD>" > /etc/couchdb/couchdb-admin.netrc
+chmod 600 /etc/couchdb/couchdb-admin.netrc
+
+curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096"
+curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
+
+ +

Check couchdb acl as unpriviledged user

+ +
cat /srv/leap/webapp/config/couchdb.yml  # see username and password
+echo "machine 127.0.0.1 login webapp password <PASSWORD>" > /etc/couchdb/couchdb-webapp.netrc
+chmod 600 /etc/couchdb/couchdb-webapp.netrc
+
+curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096"
+curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
+
+ +

All URLs accessible ?

+ + + + +

Check client config files

+ + + + +

Soledad

+ +
/var/log/soledad.log
+
+ +

Couchdb

+ +

Places to look for errors

+ +
    +
  • /var/log/couchdb/couch.log
    • +
  • /var/log/syslog (watch out for stunnel issues)
    • +
+ + +

Databases

+ +
    +
  • Following output shows all neccessary DBs that should be present. Note that the user-0123456.... DBs are the data stores for a particular user.
    • +
+ + +
+    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_all_dbs'
+    ["customers","identities","sessions","shared","tickets","tokens","user-0","user-9d34680b01074c75c2ec58c7321f540c","user-9d34680b01074c75c2ec58c7325fb7ff","users"]
+
+ + +

Design Documents

+ +
    +
  • Is User _design doc available ?
    • +
+ + +
+    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X  GET "http://127.0.0.1:5984/users/_design/User"
+
+ + +

Is couchdb cluster backend accessible through stunnel ?

+ +
    +
  • Find out how many connections are set up for the couchdb cluster backend:
    • +
+ + +
+    grep "accept = 127.0.0.1" /etc/stunnel/*
+
+ + +
    +
  • Now connect to all of those local endpoints to see if they up. All these tests should return “localhost [127.0.0.1] 4000 (?) open”
    • +
+ + +
+    nc -v 127.0.0.1 4000
+    nc -v 127.0.0.1 4001
+    ...
+
+ + +

MX

+ +

Places to look for errors

+ +
    +
  • /var/log/mail.log
    • +
  • /var/log/leap_mx.log
    • +
  • /var/log/syslog (watch out for stunnel issues)
    • +
+ + +

Is couchdb accessible through stunnel ?

+ +
    +

  • Depending on how many couch nodes you have, increase the port for every test +(see /etc/haproxy/haproxy.cfg for the server/port mapping):

    + +

    curl -s -X GET “http://127.0.0.1:4000” + curl -s -X GET “http://127.0.0.1:4001” + …

    • +
+ + +

Query leap-mx

+ +
    +
  • for useraccount
    • +
+ + +
+    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:2244
+    ...
+    postmap: dict_tcp_lookup: send: get jow@dev.bitmask.net
+    postmap: dict_tcp_lookup: recv: 200
+    ...
+
+ + +
    +
  • for mailalias
    • +
+ + +
+    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:4242
+    ...
+    postmap: dict_tcp_lookup: send: get joe@dev.bitmask.net
+    postmap: dict_tcp_lookup: recv: 200 f01bc1c70de7d7d80bc1ad77d987e73a
+    postmap: dict_tcp_lookup: found: f01bc1c70de7d7d80bc1ad77d987e73a
+    f01bc1c70de7d7d80bc1ad77d987e73a
+    ...
+
+ + +

Check couchdb acl as unpriviledged user

+ +
cat /etc/leap/mx.conf  # see username and password
+echo "machine 127.0.0.1 login leap_mx password <PASSWORD>" > /etc/couchdb/couchdb-leap_mx.netrc
+chmod 600 /etc/couchdb/couchdb-leap_mx.netrc
+
+curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/_all_dbs"   # pick one "user-<hash>" db
+curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/user-de9c77a3d7efbc779c6c20da88e8fb9c"
+
+ +
    +
  • you may check multiple times, cause 127.0.0.1:4096 is haproxy load-balancing the different couchdb nodes
    • +
+ + +

Mailspool

+ +
    +
  • Any file in the leap_mx mailspool longer for a few seconds ?
    • +
+ + +
+    ls -la /var/mail/vmail/Maildir/cur/
+
+ + +
    +
  • Any mails in postfix mailspool longer than a few seconds ?
    • +
+ + +
+    mailq
+
+ + +

Testing mail delivery

+ +
swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 25
+swaks -f varac@cdev.bitmask.net -t varac@cdev.bitmask.net -s chipmonk.cdev.bitmask.net --port 465 --tlsc
+swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 587 --tls
+
+ +

VPN

+ +

Places to look for errors

+ +
    +
  • /var/log/syslog (watch out for openvpn issues)
    • +
+ + +
+
+ + diff --git a/docs/en/troubleshooting/where-to-look/index.html b/docs/en/troubleshooting/where-to-look/index.html new file mode 100644 index 00000000..ab3115af --- /dev/null +++ b/docs/en/troubleshooting/where-to-look/index.html @@ -0,0 +1,451 @@ + + + + +Where to look - LEAP Platform Documentation + + + + + + + + +
+
+

Where to look for errors

+ +
The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment.
+
+
+ + +

General

+ +
    +
  • Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. -v 5 after the leap cmd, i.e. leap -v 2 deploy.
    • +
  • We use the example.org domain for documentation purposes here, please replace it with the you domain.
    • +
+ + +

Firewall

+ +

Every node in your provider has its own restrictive firewall, but you might have a network firewall in place as well that is not managed by LEAP platform. To see what ports and addresses must be open, run this command:

+ +
workstation$ leap compile firewall
+
+ +

If any of those are blocked, then your provider will not work.

+ +

Webapp

+ +

Places to look for errors

+ +
    +
  • /var/log/apache2/error.log
    • +
  • /srv/leap/webapp/log/production.log
    • +
  • /var/log/syslog (watch out for stunnel issues)
    • +
  • /var/log/leap/*
    • +
+ + +

Is haproxy ok ?

+ +
curl -s -X  GET "http://127.0.0.1:4096"
+
+ +

Is couchdb accessible through stunnel ?

+ +
    +

  • Depending on how many couch nodes you have, increase the port for every test +(see /etc/haproxy/haproxy.cfg for the server/port mapping):

    + +

    curl -s -X GET “http://127.0.0.1:4000” + curl -s -X GET “http://127.0.0.1:4001” + …

    • +
+ + +

Check couchdb acl as admin

+ +
mkdir /etc/couchdb
+cat /srv/leap/webapp/config/couchdb.yml.admin  # see username and password
+echo "machine 127.0.0.1 login admin password <PASSWORD>" > /etc/couchdb/couchdb-admin.netrc
+chmod 600 /etc/couchdb/couchdb-admin.netrc
+
+curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096"
+curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
+
+ +

Check couchdb acl as unpriviledged user

+ +
cat /srv/leap/webapp/config/couchdb.yml  # see username and password
+echo "machine 127.0.0.1 login webapp password <PASSWORD>" > /etc/couchdb/couchdb-webapp.netrc
+chmod 600 /etc/couchdb/couchdb-webapp.netrc
+
+curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096"
+curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
+
+ +

All URLs accessible ?

+ + + + +

Check client config files

+ + + + +

Soledad

+ +
/var/log/soledad.log
+
+ +

Couchdb

+ +

Places to look for errors

+ +
    +
  • /var/log/couchdb/couch.log
    • +
  • /var/log/syslog (watch out for stunnel issues)
    • +
+ + +

Databases

+ +
    +
  • Following output shows all neccessary DBs that should be present. Note that the user-0123456.... DBs are the data stores for a particular user.
    • +
+ + +
+    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_all_dbs'
+    ["customers","identities","sessions","shared","tickets","tokens","user-0","user-9d34680b01074c75c2ec58c7321f540c","user-9d34680b01074c75c2ec58c7325fb7ff","users"]
+
+ + +

Design Documents

+ +
    +
  • Is User _design doc available ?
    • +
+ + +
+    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X  GET "http://127.0.0.1:5984/users/_design/User"
+
+ + +

Is couchdb cluster backend accessible through stunnel ?

+ +
    +
  • Find out how many connections are set up for the couchdb cluster backend:
    • +
+ + +
+    grep "accept = 127.0.0.1" /etc/stunnel/*
+
+ + +
    +
  • Now connect to all of those local endpoints to see if they up. All these tests should return “localhost [127.0.0.1] 4000 (?) open”
    • +
+ + +
+    nc -v 127.0.0.1 4000
+    nc -v 127.0.0.1 4001
+    ...
+
+ + +

MX

+ +

Places to look for errors

+ +
    +
  • /var/log/mail.log
    • +
  • /var/log/leap_mx.log
    • +
  • /var/log/syslog (watch out for stunnel issues)
    • +
+ + +

Is couchdb accessible through stunnel ?

+ +
    +

  • Depending on how many couch nodes you have, increase the port for every test +(see /etc/haproxy/haproxy.cfg for the server/port mapping):

    + +

    curl -s -X GET “http://127.0.0.1:4000” + curl -s -X GET “http://127.0.0.1:4001” + …

    • +
+ + +

Query leap-mx

+ +
    +
  • for useraccount
    • +
+ + +
+    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:2244
+    ...
+    postmap: dict_tcp_lookup: send: get jow@dev.bitmask.net
+    postmap: dict_tcp_lookup: recv: 200
+    ...
+
+ + +
    +
  • for mailalias
    • +
+ + +
+    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:4242
+    ...
+    postmap: dict_tcp_lookup: send: get joe@dev.bitmask.net
+    postmap: dict_tcp_lookup: recv: 200 f01bc1c70de7d7d80bc1ad77d987e73a
+    postmap: dict_tcp_lookup: found: f01bc1c70de7d7d80bc1ad77d987e73a
+    f01bc1c70de7d7d80bc1ad77d987e73a
+    ...
+
+ + +

Check couchdb acl as unpriviledged user

+ +
cat /etc/leap/mx.conf  # see username and password
+echo "machine 127.0.0.1 login leap_mx password <PASSWORD>" > /etc/couchdb/couchdb-leap_mx.netrc
+chmod 600 /etc/couchdb/couchdb-leap_mx.netrc
+
+curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/_all_dbs"   # pick one "user-<hash>" db
+curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/user-de9c77a3d7efbc779c6c20da88e8fb9c"
+
+ +
    +
  • you may check multiple times, cause 127.0.0.1:4096 is haproxy load-balancing the different couchdb nodes
    • +
+ + +

Mailspool

+ +
    +
  • Any file in the leap_mx mailspool longer for a few seconds ?
    • +
+ + +
+    ls -la /var/mail/vmail/Maildir/cur/
+
+ + +
    +
  • Any mails in postfix mailspool longer than a few seconds ?
    • +
+ + +
+    mailq
+
+ + +

Testing mail delivery

+ +
swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 25
+swaks -f varac@cdev.bitmask.net -t varac@cdev.bitmask.net -s chipmonk.cdev.bitmask.net --port 465 --tlsc
+swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 587 --tls
+
+ +

VPN

+ +

Places to look for errors

+ +
    +
  • /var/log/syslog (watch out for openvpn issues)
    • +
+ + +
+
+ + diff --git a/docs/en/tutorials.html b/docs/en/tutorials.html new file mode 100644 index 00000000..3eeac685 --- /dev/null +++ b/docs/en/tutorials.html @@ -0,0 +1,138 @@ + + + + +Tutorials - LEAP Platform Documentation + + + + + + + + +
+
+

Platform Tutorials

+ +
The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment.
+
+
+
    +
+ +
+

+ Quick Start Tutorial +

+
This tutorial walks you through the initial process of creating and deploying a minimal service provider running the LEAP Platform.
+
+
+

+ Quick VPN +

+
Tutorial for setting up a simple VPN provider.
+
+
+

+ Quick email +

+
Tutorial for setting up a simple email provider.
+
+
+

+ Vagrant +

+
Running a local provider with Vagrant
+
+ +
+
+ + diff --git a/docs/en/tutorials/quick-start.html b/docs/en/tutorials/quick-start.html new file mode 100644 index 00000000..d2670b30 --- /dev/null +++ b/docs/en/tutorials/quick-start.html @@ -0,0 +1,446 @@ + + + + +Quick Start Tutorial - LEAP Platform Documentation + + + + + + + + +
+
+

Quick Start Tutorial

+ +
This tutorial walks you through the initial process of creating and deploying a minimal service provider running the LEAP Platform.
+
+
+ + +

Introduction

+ +

Our goal

+ +

We are going to create a minimal LEAP provider, but one that does not offer any actual services. Check out the other tutorials for adding VPN or email services.

+ +

Our goal is something like this:

+ +
$ leap list
+        NODES   SERVICES          TAGS
+   wildebeest   couchdb, webapp
+
+ +

NOTE: You won’t be able to run that leap list command yet, not until we actually create the node configurations.

+ +

Requirements

+ +
    +
  1. A workstation: This is your local machine that you will run commands on.
    2. +
  2. A server: This is the machine that you will deploy to. The server can be either: + +
      +
    1. A local Vagrant virtual machine: a Vagrant machine can only be useful for testing.
      2. +
    2. A real or paravirtualized server: The server must have Debian Jessie installed, and you must be able to SSH into the machine as root. Paravirtualization includes KVM, Xen, OpenStack, Amazon, but not VirtualBox or OpenVZ.
      3. +
    +
    3. +
+ + +

Other things to keep in mind:

+ +
    +
  • The ability to create/modify DNS entries for your domain is preferable, but not needed. If you don’t have access to DNS, you can workaround this by modifying your local resolver, i.e. editing /etc/hosts.
    • +
  • You need to be aware that this process will make changes to your servers, so please be sure that these machines are a basic install with nothing configured or running for other purposes.
    • +
  • Your servers will need to be connected to the internet, and not behind a restrictive firewall.
    • +
+ + +

Prepare your workstation

+ +

In order to be able to manage your servers, you need to install the leap command on your workstation:

+ +

Install pre-requisites

+ +

Install core prerequisites on your workstation.

+ +

Debian & Ubuntu

+ +
workstation$ sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make bzip2
+
+ +

Mac OS

+ +
workstation$ brew install ruby-install
+workstation$ ruby-install ruby
+
+ +

Install the LEAP command-line utility

+ +

Install the leap command system-wide:

+ +
workstation$ sudo gem install leap_cli
+
+ +

Alternately, you can install leap locally without root privileges:

+ +
workstation$ gem install --user-install leap_cli
+workstation$ PATH="$PATH:$(ruby -e 'puts Gem.user_dir')/bin"
+
+ +

If you choose a local install, you probably want to permanently add the –user-install directory to your PATH by adding this to your ~/.profile file (requires logout):

+ +
[ $(which ruby) ] && PATH="$PATH:$(ruby -e 'puts Gem.user_dir')/bin"
+
+ +

To confirm that you installed leap correctly, try running leap --version.

+ +

Create a provider instance

+ +

A provider instance is a directory tree, residing on your workstation, that contains everything you need to manage an infrastructure for a service provider.

+ +

In this case, we create one for example.org and call the instance directory ‘example’.

+ +
workstation$ leap new ~/example
+
+ +

The leap new command will ask you for several required values:

+ +
    +
  • domain: The primary domain name of your service provider. In this tutorial, we will be using “example.org”.
    • +
  • name: The name of your service provider (we use “Example”).
    • +
  • contact emails: A comma separated list of email addresses that should be used for important service provider contacts (for things like postmaster aliases, Tor contact emails, etc).
    • +
  • platform: The directory where you have a copy of the leap_platform git repository checked out. If the platform directory does not yet exist, the leap_platform will be downloaded and placed in that directory.
    • +
+ + +

You could also have passed these configuration options on the command-line, like so:

+ +
workstation$ leap new --contacts your@email.here --domain example.org --name Example --platform=~/leap/leap_platform .
+
+ +

You should now have the following files:

+ +
workstation$ tree example
+example
+├── common.json
+├── Leapfile
+├── nodes/
+├── provider.json
+├── services/
+└── tags/
+
+ +

Now add yourself as a privileged sysadmin who will have access to deploy to servers:

+ +
workstation$ cd example
+workstation$ leap add-user louise --self
+
+ +

Replace “louise” with whatever you want your sysadmin username to be.

+ +

NOTE: Make sure you change directories so that the leap command is run from within the provider instance directory. Most leap commands only work when run from a provider instance.

+ +

Now create the necessary keys and certificates:

+ +
workstation$ leap cert ca
+workstation$ leap cert csr
+
+ +

What do these commands do? The first command will create two Certificate Authorities, one that clients will use to authenticate with the servers and one for backend servers to authenticate with each other. The second command creates a Certificate Signing Request suitable for submission to a commercial CA. It also creates two “dummy” files for you to use temporarily:

+ +
    +
  • files/cert/example.org.crt – This is a “dummy” certificate for your domain that can be used temporarily for testing. Once you get a real certificate from a CA, you should replace this file.
    • +
  • files/cert/commercial_ca.crt – This is “dummy” CA cert the corresponds to the dummy domain certificate. Once you replace the domain certificate, also replace this file with the CA cert from the real Certificate Authority.
    • +
+ + +

If you plan to run a real service provider, see important information on managing keys and certificates.

+ +

Add a node to the provider

+ +

A “node” is a server that is part of your infrastructure. Every node can have one or more services associated with it. We will now add a single node with two services, “webapp” and “couchdb”.

+ +

You have two choices for node type: a real node or a local node.

+ +
    +
  • Real Node: A real node is any physical or paravirtualized server, including KVM, Xen, OpenStack Compute, Amazon EC2, but not VirtualBox or OpenVZ (VirtualBox and OpenVZ use a more limited form of virtualization). The server must be running Debian Jessie.
    • +
  • Local Node: A local node is a virtual machine created by Vagrant, useful for local testing on your workstation.
    • +
+ + +

Getting Vagrant working can be a pain and is covered in other tutorials. If you have a real server available, we suggest you try this tutorial with a real node first.

+ +

Option A: Add a real node

+ +

Note: Installing LEAP Platform on this server will potentially destroy anything you have previously installed on this machine.

+ +

Create a node, with the services “webapp” and “couchdb”:

+ +
workstation$ leap node add wildebeest ip_address:x.x.x.w services:webapp,couchdb
+
+ +

NOTE: replace x.x.x.x with the actual IP address of this server.

+ +

Option B: Add a local node

+ +

Create a node, with the services “webapp” and “couchdb”, and then start the local virtual machine:

+ +
workstation$ leap node add --local wildebeest services:webapp,couchdb
+workstation$ leap local start wildebeest
+
+ +

It will take a while to download the Virtualbox base box and create the virtual machine.

+ +

Deploy your provider

+ +

Initialize the node

+ +

Node initialization only needs to be done once, but there is no harm in doing it multiple times:

+ +
workstation$ leap node init wildebeest
+
+ +

This will initialize the node wildebeest.

+ +

For non-local nodes, when leap node init is run, you will be prompted to verify the fingerprint of the SSH host key and to provide the root password of the server(s). You should only need to do this once.

+ +

Deploy to the node

+ +

The next step is to deploy the LEAP platform to your node. Deployment can take a while to run, especially on the first run, as it needs to update the packages on the new machine.

+ +
workstation$ leap deploy wildebeest
+
+ +

Watch the output for any errors (in red), if everything worked fine, you should now have your first running node. If you do have errors, try doing the deploy again.

+ +

Setup DNS

+ +

The next step is to configure the DNS for your provider. For testing purposes, you can just modify your /etc/hosts file. Please don’t forget about these entries, they will override DNS queries if you setup your DNS later. For a list of what entries to add to /etc/hosts, run this command:

+ +
workstation$ leap compile hosts
+
+ +

Alternately, if you have access to modify the DNS zone entries for your domain:

+ +
workstation$ leap compile zone
+
+ +

NOTE: The resulting zone file is incomplete because it is missing a serial number. Use the output of leap compile zone as a guide, but do not just copy and paste the output. Also, the compile zone output will always exclude mention of local nodes.

+ +

The DNS method will not work for local nodes created with Vagrant.

+ +

Test that things worked correctly

+ +

To run troubleshooting tests:

+ +
workstation$ leap test
+
+ +

Alternately, you can run these same tests from the server itself:

+ +
workstation$ leap ssh wildebeest
+wildebeest# run_tests
+
+ +

Create an administrator

+ +

Assuming that you set up your DNS or /etc/hosts file, you should be able to load https://example.org in your web browser (where example.org is whatever domain name you actually used).

+ +

Your browser will complain about an untrusted cert, but for now just bypass this. From there, you should be able to register a new user and login.

+ +

Once you have created a user, you can now make this user an administrator. For example, if you created a user kangaroo, you would create the file services/webapp.json with the following content:

+ +
{
+    "webapp": {
+        "admins": ["kangaroo"]
+    }
+}
+
+ +

Save that file and run leap deploy again. When you next log on to the web application, the user kangaroo will now be an admin.

+ +

If you want to restrict who can register a new user, see webapp for configuration options.

+ +

What is next?

+ +

Add an end-user service

+ +

You should now have a minimal service provider with a single node. This service provider is pointless at the moment, because it does not include any end-user services like VPN or email. To add one of these services, continue with one of the following tutorials:

+ + + + +

Learn more

+ +

We have only just scratched the surface of the possible ways to configure and deploy your service provider. Your next step should be:

+ + + + +
+
+ + diff --git a/docs/en/tutorials/quick-start/index.html b/docs/en/tutorials/quick-start/index.html new file mode 100644 index 00000000..27b21238 --- /dev/null +++ b/docs/en/tutorials/quick-start/index.html @@ -0,0 +1,446 @@ + + + + +Quick Start Tutorial - LEAP Platform Documentation + + + + + + + + +
+
+

Quick Start Tutorial

+ +
This tutorial walks you through the initial process of creating and deploying a minimal service provider running the LEAP Platform.
+
+
+ + +

Introduction

+ +

Our goal

+ +

We are going to create a minimal LEAP provider, but one that does not offer any actual services. Check out the other tutorials for adding VPN or email services.

+ +

Our goal is something like this:

+ +
$ leap list
+        NODES   SERVICES          TAGS
+   wildebeest   couchdb, webapp
+
+ +

NOTE: You won’t be able to run that leap list command yet, not until we actually create the node configurations.

+ +

Requirements

+ +
    +
  1. A workstation: This is your local machine that you will run commands on.
    2. +
  2. A server: This is the machine that you will deploy to. The server can be either: + +
      +
    1. A local Vagrant virtual machine: a Vagrant machine can only be useful for testing.
      2. +
    2. A real or paravirtualized server: The server must have Debian Jessie installed, and you must be able to SSH into the machine as root. Paravirtualization includes KVM, Xen, OpenStack, Amazon, but not VirtualBox or OpenVZ.
      3. +
    +
    3. +
+ + +

Other things to keep in mind:

+ +
    +
  • The ability to create/modify DNS entries for your domain is preferable, but not needed. If you don’t have access to DNS, you can workaround this by modifying your local resolver, i.e. editing /etc/hosts.
    • +
  • You need to be aware that this process will make changes to your servers, so please be sure that these machines are a basic install with nothing configured or running for other purposes.
    • +
  • Your servers will need to be connected to the internet, and not behind a restrictive firewall.
    • +
+ + +

Prepare your workstation

+ +

In order to be able to manage your servers, you need to install the leap command on your workstation:

+ +

Install pre-requisites

+ +

Install core prerequisites on your workstation.

+ +

Debian & Ubuntu

+ +
workstation$ sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make bzip2
+
+ +

Mac OS

+ +
workstation$ brew install ruby-install
+workstation$ ruby-install ruby
+
+ +

Install the LEAP command-line utility

+ +

Install the leap command system-wide:

+ +
workstation$ sudo gem install leap_cli
+
+ +

Alternately, you can install leap locally without root privileges:

+ +
workstation$ gem install --user-install leap_cli
+workstation$ PATH="$PATH:$(ruby -e 'puts Gem.user_dir')/bin"
+
+ +

If you choose a local install, you probably want to permanently add the –user-install directory to your PATH by adding this to your ~/.profile file (requires logout):

+ +
[ $(which ruby) ] && PATH="$PATH:$(ruby -e 'puts Gem.user_dir')/bin"
+
+ +

To confirm that you installed leap correctly, try running leap --version.

+ +

Create a provider instance

+ +

A provider instance is a directory tree, residing on your workstation, that contains everything you need to manage an infrastructure for a service provider.

+ +

In this case, we create one for example.org and call the instance directory ‘example’.

+ +
workstation$ leap new ~/example
+
+ +

The leap new command will ask you for several required values:

+ +
    +
  • domain: The primary domain name of your service provider. In this tutorial, we will be using “example.org”.
    • +
  • name: The name of your service provider (we use “Example”).
    • +
  • contact emails: A comma separated list of email addresses that should be used for important service provider contacts (for things like postmaster aliases, Tor contact emails, etc).
    • +
  • platform: The directory where you have a copy of the leap_platform git repository checked out. If the platform directory does not yet exist, the leap_platform will be downloaded and placed in that directory.
    • +
+ + +

You could also have passed these configuration options on the command-line, like so:

+ +
workstation$ leap new --contacts your@email.here --domain example.org --name Example --platform=~/leap/leap_platform .
+
+ +

You should now have the following files:

+ +
workstation$ tree example
+example
+├── common.json
+├── Leapfile
+├── nodes/
+├── provider.json
+├── services/
+└── tags/
+
+ +

Now add yourself as a privileged sysadmin who will have access to deploy to servers:

+ +
workstation$ cd example
+workstation$ leap add-user louise --self
+
+ +

Replace “louise” with whatever you want your sysadmin username to be.

+ +

NOTE: Make sure you change directories so that the leap command is run from within the provider instance directory. Most leap commands only work when run from a provider instance.

+ +

Now create the necessary keys and certificates:

+ +
workstation$ leap cert ca
+workstation$ leap cert csr
+
+ +

What do these commands do? The first command will create two Certificate Authorities, one that clients will use to authenticate with the servers and one for backend servers to authenticate with each other. The second command creates a Certificate Signing Request suitable for submission to a commercial CA. It also creates two “dummy” files for you to use temporarily:

+ +
    +
  • files/cert/example.org.crt – This is a “dummy” certificate for your domain that can be used temporarily for testing. Once you get a real certificate from a CA, you should replace this file.
    • +
  • files/cert/commercial_ca.crt – This is “dummy” CA cert the corresponds to the dummy domain certificate. Once you replace the domain certificate, also replace this file with the CA cert from the real Certificate Authority.
    • +
+ + +

If you plan to run a real service provider, see important information on managing keys and certificates.

+ +

Add a node to the provider

+ +

A “node” is a server that is part of your infrastructure. Every node can have one or more services associated with it. We will now add a single node with two services, “webapp” and “couchdb”.

+ +

You have two choices for node type: a real node or a local node.

+ +
    +
  • Real Node: A real node is any physical or paravirtualized server, including KVM, Xen, OpenStack Compute, Amazon EC2, but not VirtualBox or OpenVZ (VirtualBox and OpenVZ use a more limited form of virtualization). The server must be running Debian Jessie.
    • +
  • Local Node: A local node is a virtual machine created by Vagrant, useful for local testing on your workstation.
    • +
+ + +

Getting Vagrant working can be a pain and is covered in other tutorials. If you have a real server available, we suggest you try this tutorial with a real node first.

+ +

Option A: Add a real node

+ +

Note: Installing LEAP Platform on this server will potentially destroy anything you have previously installed on this machine.

+ +

Create a node, with the services “webapp” and “couchdb”:

+ +
workstation$ leap node add wildebeest ip_address:x.x.x.w services:webapp,couchdb
+
+ +

NOTE: replace x.x.x.x with the actual IP address of this server.

+ +

Option B: Add a local node

+ +

Create a node, with the services “webapp” and “couchdb”, and then start the local virtual machine:

+ +
workstation$ leap node add --local wildebeest services:webapp,couchdb
+workstation$ leap local start wildebeest
+
+ +

It will take a while to download the Virtualbox base box and create the virtual machine.

+ +

Deploy your provider

+ +

Initialize the node

+ +

Node initialization only needs to be done once, but there is no harm in doing it multiple times:

+ +
workstation$ leap node init wildebeest
+
+ +

This will initialize the node wildebeest.

+ +

For non-local nodes, when leap node init is run, you will be prompted to verify the fingerprint of the SSH host key and to provide the root password of the server(s). You should only need to do this once.

+ +

Deploy to the node

+ +

The next step is to deploy the LEAP platform to your node. Deployment can take a while to run, especially on the first run, as it needs to update the packages on the new machine.

+ +
workstation$ leap deploy wildebeest
+
+ +

Watch the output for any errors (in red), if everything worked fine, you should now have your first running node. If you do have errors, try doing the deploy again.

+ +

Setup DNS

+ +

The next step is to configure the DNS for your provider. For testing purposes, you can just modify your /etc/hosts file. Please don’t forget about these entries, they will override DNS queries if you setup your DNS later. For a list of what entries to add to /etc/hosts, run this command:

+ +
workstation$ leap compile hosts
+
+ +

Alternately, if you have access to modify the DNS zone entries for your domain:

+ +
workstation$ leap compile zone
+
+ +

NOTE: The resulting zone file is incomplete because it is missing a serial number. Use the output of leap compile zone as a guide, but do not just copy and paste the output. Also, the compile zone output will always exclude mention of local nodes.

+ +

The DNS method will not work for local nodes created with Vagrant.

+ +

Test that things worked correctly

+ +

To run troubleshooting tests:

+ +
workstation$ leap test
+
+ +

Alternately, you can run these same tests from the server itself:

+ +
workstation$ leap ssh wildebeest
+wildebeest# run_tests
+
+ +

Create an administrator

+ +

Assuming that you set up your DNS or /etc/hosts file, you should be able to load https://example.org in your web browser (where example.org is whatever domain name you actually used).

+ +

Your browser will complain about an untrusted cert, but for now just bypass this. From there, you should be able to register a new user and login.

+ +

Once you have created a user, you can now make this user an administrator. For example, if you created a user kangaroo, you would create the file services/webapp.json with the following content:

+ +
{
+    "webapp": {
+        "admins": ["kangaroo"]
+    }
+}
+
+ +

Save that file and run leap deploy again. When you next log on to the web application, the user kangaroo will now be an admin.

+ +

If you want to restrict who can register a new user, see webapp for configuration options.

+ +

What is next?

+ +

Add an end-user service

+ +

You should now have a minimal service provider with a single node. This service provider is pointless at the moment, because it does not include any end-user services like VPN or email. To add one of these services, continue with one of the following tutorials:

+ + + + +

Learn more

+ +

We have only just scratched the surface of the possible ways to configure and deploy your service provider. Your next step should be:

+ + + + +
+
+ + diff --git a/docs/en/tutorials/single-node-email.html b/docs/en/tutorials/single-node-email.html new file mode 100644 index 00000000..6678fec3 --- /dev/null +++ b/docs/en/tutorials/single-node-email.html @@ -0,0 +1,200 @@ + + + + +Quick email - LEAP Platform Documentation + + + + + + + + +
+
+

Single node email tutorial

+ +
Tutorial for setting up a simple email provider.
+
+
+ + +

This tutorial walks you through the initial process of creating and deploying a minimal email service provider. Please first complete the Quick Start Tutorial. This tutorial will pick up where that one left off.

+ +

Our goal

+ +

We are going to create a minimal LEAP provider offering email service.

+ +

Our goal is something like this:

+ +
$ leap list
+    NODES       SERVICES                       TAGS
+    wildebeest  couchdb, mx, soledad, webapp
+
+ +

Where ‘wildebeest’ is whatever name you chose for your node in the Quick Start Tutorial.

+ +

Add email services to the node

+ +

In order to add services to a node, edit the node’s JSON configuration file.

+ +

In our example, we would edit nodes/wildebeest.json:

+ +
{
+  "ip_address": "1.1.1.1",
+  "services": ["couchdb", "webapp", "mx", "soledad"]
+}
+
+ +

Here, we added mx and soledad to the node’s services list. Briefly:

+ +
    +
  • mx: nodes with the mx service will run postfix mail transfer agent, and are able to receive and relay email on behalf of your domain. You can have as many as you want, spread out over as many nodes as you want.
    • +
  • soledad: nodes with soledad service run the server-side daemon that allows the client to synchronize a user’s personal data store among their devices. Currently, soledad only runs on nodes that are also couchdb nodes.
    • +
+ + +

For more details, see the Services overview, or the individual pages for the mx and soledad services.

+ +

Deploy to the node

+ +

Now you should deploy to your node.

+ +
workstation$ leap deploy
+
+ +

Setup DNS

+ +

There are several important DNS entries that all email providers should have:

+ +
    +
  • SPF (Sender Policy Framework): With SPF, an email provider advertises in their DNS which servers should be allowed to relay email on behalf of your domain.
    • +
  • DKIM (DomainKey Identified Mail): With DKIM, an email provider is able to vouch for the validity of certain headers in outgoing mail, allowing the receiving provider to have more confidence in these values when processing the message for spam or abuse.
    • +
+ + +

In order to take advantage of SPF and DKIM, run this command:

+ +
workstation$ leap compile zone
+
+ +

Then take the output of that command and merge it with the DNS zone file for your domain.

+ +

CAUTION: the output of leap compile zone is not a complete zone file since it is missing a serial number. You will need to manually merge it with your existing zone file.

+ +

Test it out

+ +

First, run:

+ +
workstation# leap test
+
+ +

Then fire up the bitmask client, register a new user with your provider, and try sending and receiving email.

+ +
+
+ + diff --git a/docs/en/tutorials/single-node-email/index.html b/docs/en/tutorials/single-node-email/index.html new file mode 100644 index 00000000..45a1264f --- /dev/null +++ b/docs/en/tutorials/single-node-email/index.html @@ -0,0 +1,200 @@ + + + + +Quick email - LEAP Platform Documentation + + + + + + + + +
+
+

Single node email tutorial

+ +
Tutorial for setting up a simple email provider.
+
+
+ + +

This tutorial walks you through the initial process of creating and deploying a minimal email service provider. Please first complete the Quick Start Tutorial. This tutorial will pick up where that one left off.

+ +

Our goal

+ +

We are going to create a minimal LEAP provider offering email service.

+ +

Our goal is something like this:

+ +
$ leap list
+    NODES       SERVICES                       TAGS
+    wildebeest  couchdb, mx, soledad, webapp
+
+ +

Where ‘wildebeest’ is whatever name you chose for your node in the Quick Start Tutorial.

+ +

Add email services to the node

+ +

In order to add services to a node, edit the node’s JSON configuration file.

+ +

In our example, we would edit nodes/wildebeest.json:

+ +
{
+  "ip_address": "1.1.1.1",
+  "services": ["couchdb", "webapp", "mx", "soledad"]
+}
+
+ +

Here, we added mx and soledad to the node’s services list. Briefly:

+ +
    +
  • mx: nodes with the mx service will run postfix mail transfer agent, and are able to receive and relay email on behalf of your domain. You can have as many as you want, spread out over as many nodes as you want.
    • +
  • soledad: nodes with soledad service run the server-side daemon that allows the client to synchronize a user’s personal data store among their devices. Currently, soledad only runs on nodes that are also couchdb nodes.
    • +
+ + +

For more details, see the Services overview, or the individual pages for the mx and soledad services.

+ +

Deploy to the node

+ +

Now you should deploy to your node.

+ +
workstation$ leap deploy
+
+ +

Setup DNS

+ +

There are several important DNS entries that all email providers should have:

+ +
    +
  • SPF (Sender Policy Framework): With SPF, an email provider advertises in their DNS which servers should be allowed to relay email on behalf of your domain.
    • +
  • DKIM (DomainKey Identified Mail): With DKIM, an email provider is able to vouch for the validity of certain headers in outgoing mail, allowing the receiving provider to have more confidence in these values when processing the message for spam or abuse.
    • +
+ + +

In order to take advantage of SPF and DKIM, run this command:

+ +
workstation$ leap compile zone
+
+ +

Then take the output of that command and merge it with the DNS zone file for your domain.

+ +

CAUTION: the output of leap compile zone is not a complete zone file since it is missing a serial number. You will need to manually merge it with your existing zone file.

+ +

Test it out

+ +

First, run:

+ +
workstation# leap test
+
+ +

Then fire up the bitmask client, register a new user with your provider, and try sending and receiving email.

+ +
+
+ + diff --git a/docs/en/tutorials/single-node-vpn.html b/docs/en/tutorials/single-node-vpn.html new file mode 100644 index 00000000..1bfeb937 --- /dev/null +++ b/docs/en/tutorials/single-node-vpn.html @@ -0,0 +1,250 @@ + + + + +Quick VPN - LEAP Platform Documentation + + + + + + + + +
+
+

Single node VPN tutorial

+ +
Tutorial for setting up a simple VPN provider.
+
+
+ + +

This tutorial walks you through the initial process of creating and deploying a minimal VPN service provider. Please first complete the Quick Start Tutorial. This tutorial will pick up where that one left off.

+ +

NOTE: For the VPN to work, you must use a real or paravirtualized node, not a local Vagrant node.

+ +

Our goal

+ +

We are going to create a minimal LEAP provider offering VPN service.

+ +

Our goal is something like this:

+ +
$ leap list
+    NODES       SERVICES                       TAGS
+    wildebeest  couchdb, webapp, openvpn, tor
+
+ +

Where ‘wildebeest’ is whatever name you chose for your node in the Quick Start Tutorial.

+ +

Add VPN service to the node

+ +

In order to add services to a node, edit the node’s JSON configuration file.

+ +

In our example, we would edit nodes/wildebeest.json:

+ +
{
+  "ip_address": "1.1.1.1",
+  "services": ["couchdb", "webapp", "openvpn", "tor"]
+}
+
+ +

Here, we added openvpn and tor to the node’s services list. Briefly:

+ +
    +
  • openvpn: nodes with the openvpn service will become OpenVPN gateways that clients connect to in order to proxy their internet connection. You can have as many as you want, spread out over as many nodes as you want.
    • +
  • tor: nodes with tor service become Tor exit nodes. This is entirely optional, and will add additional bandwidth to your node. If you don’t have many VPN users, the added traffic will help create cover traffic for your users. On the down side, this VPN gateway will get flagged as an anonymous proxy and some sites may block traffic from it.
    • +
+ + +

For more details, see the Services overview, or the individual pages for the openvpn and tor services.

+ +

Add gateway_address to the node

+ +

VPN gateways require two different IP addresses:

+ +
    +
  • ip_address: This property is used for VPN traffic egress. In other words, all VPN traffic appears to come from this IP address. This is also the main IP of the server.
    • +
  • openvpn.gateway_address: This property is used for VPN traffic ingress. In other words, clients will connect to this IP address.
    • +
+ + +

The node configuration file should now look like this:

+ +
{
+  "ip_address": "1.1.1.1",
+  "services": ["couchdb", "webapp", "openvpn", "tor"],
+  "openvpn": {
+    "gateway_address": "2.2.2.2"
+  }
+}
+
+ +

Why two different addresses? Without this, the traffic from one VPN user to another would not be encrypted. This is because the routing table of VPN clients must ensure that packets with a destination of the VPN gateway are sent unmodified and don’t get passed through the VPN’s encryption.

+ +

Generate a Diffie-Hellman file

+ +

Next we need to create a Diffie-Hellman parameter file, used for forward secret OpenVPN ciphers. You only need to do this once.

+ +
workstation$ leap cert dh
+
+ +

Feel free to erase the resulting DH file and regenerate it as you please.

+ +

Deploy to the node

+ +

Now you should deploy to your node. This may take a while.

+ +
workstation$ leap deploy
+
+ +

If the deploy was not successful, try to run it again.

+ +

Test it out

+ +

First, run:

+ +
workstation$ leap test
+
+ +

Then fire up the Bitmask client, register a new user with your provider, and turn on the VPN connection.

+ +

Alternately, you can also manually connect to your VPN gateway using OpenVPN on the command line:

+ +
workstation$ sudo apt install openvpn
+workstation$ leap test init
+workstation$ sudo openvpn --config test/openvpn/default_unlimited.ovpn
+
+ +

Make sure that Bitmask is not connected to the VPN when you run that command.

+ +

The name of the test configuration might differ depending on your setup. The test configuration created by leap test init includes a client certificate that will expire, so you may need to re-run leap test init if it has been a while since you last generated the test configuration.

+ +

What do do next

+ +

A VPN provider with a single gateway is kind of limited. You can add as many nodes with service openvpn as you like. There is no communication among the VPN gateways or with the webapp or couchdb nodes, so there is no issue with scaling out the number of gateways.

+ +

For example, add some more nodes:

+ +
workstation$ leap node add giraffe ip_address:1.1.1.2 services:openvpn openvpn.gateway_address:2.2.2.3
+workstation$ leap node add rhino ip_address:1.1.1.3 services:openvpn openvpn.gateway_address:2.2.2.4
+workstation$ leap node init giraffe rhino
+workstation$ leap deploy
+
+ +

Now you have three VPN gateways.

+ +

One consideration is that you should tag each VPN gateway with a location. This helps the client determine which VPN gateway it should connect to by default and will allow the user to choose among gateways based on location.

+ +
+
+ + diff --git a/docs/en/tutorials/single-node-vpn/index.html b/docs/en/tutorials/single-node-vpn/index.html new file mode 100644 index 00000000..adceb66f --- /dev/null +++ b/docs/en/tutorials/single-node-vpn/index.html @@ -0,0 +1,250 @@ + + + + +Quick VPN - LEAP Platform Documentation + + + + + + + + +
+
+

Single node VPN tutorial

+ +
Tutorial for setting up a simple VPN provider.
+
+
+ + +

This tutorial walks you through the initial process of creating and deploying a minimal VPN service provider. Please first complete the Quick Start Tutorial. This tutorial will pick up where that one left off.

+ +

NOTE: For the VPN to work, you must use a real or paravirtualized node, not a local Vagrant node.

+ +

Our goal

+ +

We are going to create a minimal LEAP provider offering VPN service.

+ +

Our goal is something like this:

+ +
$ leap list
+    NODES       SERVICES                       TAGS
+    wildebeest  couchdb, webapp, openvpn, tor
+
+ +

Where ‘wildebeest’ is whatever name you chose for your node in the Quick Start Tutorial.

+ +

Add VPN service to the node

+ +

In order to add services to a node, edit the node’s JSON configuration file.

+ +

In our example, we would edit nodes/wildebeest.json:

+ +
{
+  "ip_address": "1.1.1.1",
+  "services": ["couchdb", "webapp", "openvpn", "tor"]
+}
+
+ +

Here, we added openvpn and tor to the node’s services list. Briefly:

+ +
    +
  • openvpn: nodes with the openvpn service will become OpenVPN gateways that clients connect to in order to proxy their internet connection. You can have as many as you want, spread out over as many nodes as you want.
    • +
  • tor: nodes with tor service become Tor exit nodes. This is entirely optional, and will add additional bandwidth to your node. If you don’t have many VPN users, the added traffic will help create cover traffic for your users. On the down side, this VPN gateway will get flagged as an anonymous proxy and some sites may block traffic from it.
    • +
+ + +

For more details, see the Services overview, or the individual pages for the openvpn and tor services.

+ +

Add gateway_address to the node

+ +

VPN gateways require two different IP addresses:

+ +
    +
  • ip_address: This property is used for VPN traffic egress. In other words, all VPN traffic appears to come from this IP address. This is also the main IP of the server.
    • +
  • openvpn.gateway_address: This property is used for VPN traffic ingress. In other words, clients will connect to this IP address.
    • +
+ + +

The node configuration file should now look like this:

+ +
{
+  "ip_address": "1.1.1.1",
+  "services": ["couchdb", "webapp", "openvpn", "tor"],
+  "openvpn": {
+    "gateway_address": "2.2.2.2"
+  }
+}
+
+ +

Why two different addresses? Without this, the traffic from one VPN user to another would not be encrypted. This is because the routing table of VPN clients must ensure that packets with a destination of the VPN gateway are sent unmodified and don’t get passed through the VPN’s encryption.

+ +

Generate a Diffie-Hellman file

+ +

Next we need to create a Diffie-Hellman parameter file, used for forward secret OpenVPN ciphers. You only need to do this once.

+ +
workstation$ leap cert dh
+
+ +

Feel free to erase the resulting DH file and regenerate it as you please.

+ +

Deploy to the node

+ +

Now you should deploy to your node. This may take a while.

+ +
workstation$ leap deploy
+
+ +

If the deploy was not successful, try to run it again.

+ +

Test it out

+ +

First, run:

+ +
workstation$ leap test
+
+ +

Then fire up the Bitmask client, register a new user with your provider, and turn on the VPN connection.

+ +

Alternately, you can also manually connect to your VPN gateway using OpenVPN on the command line:

+ +
workstation$ sudo apt install openvpn
+workstation$ leap test init
+workstation$ sudo openvpn --config test/openvpn/default_unlimited.ovpn
+
+ +

Make sure that Bitmask is not connected to the VPN when you run that command.

+ +

The name of the test configuration might differ depending on your setup. The test configuration created by leap test init includes a client certificate that will expire, so you may need to re-run leap test init if it has been a while since you last generated the test configuration.

+ +

What do do next

+ +

A VPN provider with a single gateway is kind of limited. You can add as many nodes with service openvpn as you like. There is no communication among the VPN gateways or with the webapp or couchdb nodes, so there is no issue with scaling out the number of gateways.

+ +

For example, add some more nodes:

+ +
workstation$ leap node add giraffe ip_address:1.1.1.2 services:openvpn openvpn.gateway_address:2.2.2.3
+workstation$ leap node add rhino ip_address:1.1.1.3 services:openvpn openvpn.gateway_address:2.2.2.4
+workstation$ leap node init giraffe rhino
+workstation$ leap deploy
+
+ +

Now you have three VPN gateways.

+ +

One consideration is that you should tag each VPN gateway with a location. This helps the client determine which VPN gateway it should connect to by default and will allow the user to choose among gateways based on location.

+ +
+
+ + diff --git a/docs/en/tutorials/vagrant.html b/docs/en/tutorials/vagrant.html new file mode 100644 index 00000000..3d4f0520 --- /dev/null +++ b/docs/en/tutorials/vagrant.html @@ -0,0 +1,724 @@ + + + + +Vagrant - LEAP Platform Documentation + + + + + + + + +
+
+

Vagrant and the LEAP Platform

+ +
Running a local provider with Vagrant
+
+
+ + +

What is Vagrant?

+ +

Vagrant is a tool to make it easier to manage virtual machines running on your desktop computer (typically for testing or development purposes). You can use Vagrant to create virtual machines and deploy the LEAP platform locally.

+ +

Vagrant can be a pain to get working initially, but this page should help you get through the process. Please make sure you have at least Vagrant v1.5 installed.

+ +

There are two ways you can setup LEAP platform using Vagrant.

+ +
    +
  1. use the leap command: this will allow you to create multiple virtual machines.
    2. +
  2. use static Vagrantfile: there is a static Vagrantfile that is distributed with the leap_platform.git. This only supports a single, pre-configured virtual machine, but can get you started more quickly.
    3. +
+ + +

Install Vagrant

+ +

Requirements:

+ +
    +
  • A real machine with virtualization support in the CPU (VT-x or AMD-V). In other words, not a virtual machine.
    • +
  • Have at least 4gb of RAM.
    • +
  • Have a fast internet connection (because you will be downloading a lot of big files, like virtual machine images).
    • +
  • You should do everything described below as an unprivileged user, and only run those commands as root that are noted with sudo in front of them. Other than those commands, there is no need for privileged access to your machine, and in fact things may not work correctly.
    • +
+ + +

Debian & Ubuntu

+ +

Install core prerequisites:

+ +
sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make
+
+ +

Install Vagrant:

+ +
sudo apt-get install vagrant virtualbox
+
+ +

If you want to use libvirt instead of virtualbox, you don’t need to install virtualbox. See support for libvirt.

+ +

Mac OS X 10.9 (Mavericks)

+ +

Install Homebrew package manager from http://brew.sh/ and enable the System Duplicates Repository (needed to update old software versions delivered by Apple) with

+ +
brew tap homebrew/dupes
+
+ +

Update OpenSSH to support ECDSA keys. Follow this guide to let your system use the Homebrew binary.

+ +
brew install openssh --with-brewed-openssl --with-keychain-support
+
+ +

The certtool provided by Apple it’s really old, install the one provided by GnuTLS and shadow the system’s default.

+ +
sudo brew install gnutls
+ln -sf /usr/local/bin/gnutls-certtool /usr/local/bin/certool
+
+ +

Install the Vagrant and VirtualBox packages for OS X from their respective Download pages.

+ + + + +

Vagrant with leap command

+ +

If you have not done so, install leap command line tool:

+ +
gem install leap_cli
+
+ +

Creating local nodes

+ +

When you create a service provider, your servers are called “nodes”. When a node is virtual and exists only locally using vagrant, this type of node is called a “local node”.

+ +

If you do not have a provider already, you will need to create one and configure it before continuing (see the Quick Start guide).

+ +

These commands, for example, will create an initial provider directory “myprovider”:

+ +
$ leap new --domain example.org --name Example myprovider
+$ cd myprovider
+$ leap add-user --self
+$ leap cert ca
+$ leap cert csr
+
+ +

To create local nodes, add the flag --local to the leap node add command. For example:

+ +
$ leap node add --local web1 services:webapp
+ = created nodes/web1.json
+ = created files/nodes/web1/
+ = created files/nodes/web1/web1.key
+ = created files/nodes/web1/web1.crt
+
+ +

This command creates a node configuration file in nodes/web1.json with the webapp service.

+ +

Starting local nodes

+ +

In order to test the node “web1” we need to start it. Starting a node for the first time will spin up a virtual machine. The first time you do this will take some time because it will need to download a VM image (about 700mb). After you’ve downloaded the base image, you will not need to download it again, and instead you will re-use the downloaded image (until you need to update the image).

+ +

NOTE: Many people have difficulties getting Vagrant working. If the following commands do not work, please see the troubleshooting section below.

+ +
$ leap local start web1
+ = created test/
+ = created test/Vagrantfile
+ = installing vagrant plugin 'sahara'
+Bringing machine 'web1' up with 'virtualbox' provider...
+[web1] Box 'leap-jessie' was not found. Fetching box from specified URL for
+the provider 'virtualbox'. Note that if the URL does not have
+a box for this provider, you should interrupt Vagrant now and add
+the box yourself. Otherwise Vagrant will attempt to download the
+full box prior to discovering this error.
+Downloading or copying the box...
+Progress: 3% (Rate: 560k/s, Estimated time remaining: 0:13:36)
+...
+Bringing machine 'web1' up with 'virtualbox' provider...
+[web1] Importing base box 'leap-jessie'...
+0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
+
+ +

Now the virtual machine ‘web1’ is running. You can add another local node using the same process. For example, the webapp node needs a databasse to run, so let’s add a “couchdb” node:

+ +
$ leap node add --local db1 services:couchdb
+$ leap local start
+ = updated test/Vagrantfile
+Bringing machine 'db1' up with 'virtualbox' provider...
+[db1] Importing base box 'leap-jessie'...
+[db1] Matching MAC address for NAT networking...
+[db1] Setting the name of the VM...
+[db1] Clearing any previously set forwarded ports...
+[db1] Fixed port collision for 22 => 2222. Now on port 2202.
+[db1] Creating shared folders metadata...
+[db1] Clearing any previously set network interfaces...
+[db1] Preparing network interfaces based on configuration...
+[db1] Forwarding ports...
+[db1] -- 22 => 2202 (adapter 1)
+[db1] Running any VM customizations...
+[db1] Booting VM...
+[db1] Waiting for VM to boot. This can take a few minutes.
+[db1] VM booted and ready for use!
+[db1] Configuring and enabling network interfaces...
+[db1] Mounting shared folders...
+[db1] -- /vagrant
+
+ +

You now can follow the normal LEAP process and initialize it and then deploy your recipes to it:

+ +
$ leap node init web1
+$ leap deploy web1
+$ leap node init db1
+$ leap deploy db1
+
+ +

Useful local commands

+ +

There are many useful things you can do with a virtualized development environment.

+ +

Listing what machines are running

+ +

Now you have the two virtual machines “web1” and “db1” running, you can see the running machines as follows:

+ +
$ leap local status
+Current machine states:
+
+db1                      running (virtualbox)
+web1                     running (virtualbox)
+
+This environment represents multiple VMs. The VMs are all listed
+above with their current state. For more information about a specific
+VM, run `vagrant status NAME`.
+
+ +

Stopping machines

+ +

It is not recommended that you leave your virtual machines running when you are not using them. They consume memory and other resources! To stop your machines, simply do the following:

+ +
$ leap local stop web1 db1
+
+ +

Connecting to machines

+ +

You can connect to your local nodes just like you do with normal LEAP nodes, by running ‘leap ssh node’.

+ +

However, if you cannot connect to your local node, because the networking is not setup properly, or you have deployed a firewall that locks you out, you may need to access the graphical console.

+ +

In order to do that, you will need to configure Vagrant to launch a graphical console and then you can login as root there to diagnose the networking problem. To do this, add the following to your $HOME/.leaprc:

+ +
@custom_vagrant_vm_line = 'config.vm.provider "virtualbox" do |v|
+  v.gui = true
+end'
+
+ +

and then start, or restart, your local Vagrant node. You should get a VirtualBox graphical interface presented to you showing you the bootup and eventually the login.

+ +

Snapshotting machines

+ +

A very useful feature of local Vagrant development nodes is the ability to snapshot the current state and then revert to that when you need.

+ +

For example, perhaps the base image is a little bit out of date and you want to get the packages updated to the latest before continuing. You can do that simply by starting the node, connecting to it and updating the packages and then snapshotting the node:

+ +
$ leap local start web1
+$ leap ssh web1
+web1# apt-get -u dist-upgrade
+web1# exit
+$ leap local save web1
+
+ +

Now you can deploy to web1 and if you decide you want to revert to the state before deployment, you simply have to reset the node to your previous save:

+ +
$ leap local reset web1
+
+ +

More information

+ +

See leap help local for a complete list of local-only commands and how they can be used.

+ +

2. Vagrant with static Vagrantfile

+ +

You can use the static Vagrantfile if you want to get up a running with a pre-canned test provider.

+ +

It will install a single node mail server in the default configuration with one single command.

+ +

Clone the platform with

+ +
git clone --recursive -b develop https://github.com/leapcode/leap_platform.git
+
+ +

Start the vagrant box with

+ +
cd leap_platform
+vagrant up
+
+ +

Follow the instructions how to configure your /etc/hosts +in order to use the provider!

+ +

You can login via ssh with the systemuser vagrant and the same password.

+ +
vagrant ssh
+
+ +

On the host, run the tests to check if everything is working as expected:

+ +
cd /home/vagrant/leap/configuration/
+leap test
+
+ +

Use the bitmask client to do an initial soledad sync

+ +

Copy the self-signed CA certificate from the host. +The easiest way is to use the vagrant-scp plugin:

+ +
vagrant scp :/home/vagrant/leap/configuration/files/ca/ca.crt /tmp/example.org.ca.crt
+
+vagrant@node1:~/leap/configuration$ cat files/ca/ca.crt
+
+ +

and write it into a file, needed by the bitmask client:

+ +
bitmask --ca-cert-file /tmp/example.org.ca.crt
+
+ +

On the first run, bitmask is creating a gpg keypair. This is +needed for delivering and encrypting incoming mails.

+ +

Testing email

+ +
sudo apt install swaks
+swaks -f test22@leap.se -t test22@example.org -s example.org
+
+ +

check the logs:

+ +
sudo less /var/log/mail.log
+sudo less /var/log/leap/mx.log
+
+ +

if an error occurs, see if the mail is still laying in the mailspool dir:

+ +
sudo ls /var/mail/leap-mx/Maildir/new
+
+ +

Re-run bitmask client to sync your mail

+ +
bitmask --ca-cert-file /tmp/example.org.ca.crt
+
+ +

Now, connect your favorite mail client to the imap and smtp proxy +started by the bitmask client:

+ +
https://bitmask.net/en/help/email
+
+ +

Happy testing !

+ +

Using the Webapp

+ +

There are 2 users preconfigured:

+ +

. testuser with pw hallo123 +. testadmin with pw hallo123

+ +

login as testadmin to access the webapp with admin priviledges.

+ +

Support for libvirt

+ +

Install libvirt plugin

+ +

By default, Vagrant will use VirtualBox to create the virtual machines, but this is how you can use libvirt. Using libvirt is more efficient, but VirtualBox is more stable and easier to set up.

+ +

For debian/ubuntu:

+ +
sudo apt-get install libvirt-bin libvirt-dev
+
+# to build the vagrant-libvirt plugin you need the following packages:
+sudo apt-get install ruby-dev libxslt-dev libxml2-dev libvirt-dev
+
+# install the required plugins
+vagrant plugin install vagrant-libvirt fog fog-libvirt sahara
+
+ +

Log out and then log back in.

+ +

Note: if running ubuntu 15.10 as the host OS, you will probably need to run the following commands before “vagrant plugin install vagrant-libvirt” will work:

+ +
ln -sf /usr/lib/liblzma.so.5 /opt/vagrant/embedded/lib
+ln -sf /usr/lib/liblzma.so.5.0.0 /opt/vagrant/embedded/lib
+
+ +

Create libvirt pool

+ +

Next, you must create the libvirt image pool. The “default” pool uses /var/lib/libvirt/images, but Vagrant will not download base boxes there. Instead, create a libvirt pool called “vagrant”, like so:

+ +
virsh pool-define-as vagrant dir - - - - /home/$USER/.vagrant.d/boxes
+virsh pool-start vagrant
+virsh pool-autostart vagrant
+
+ +

If you want to use a name different than “vagrant” for the pool, you can change the name in Leapfile by setting the @vagrant_libvirt_pool variable:

+ +
@vagrant_libvirt_pool = "vagrant"
+
+ +

Force use of libvirt

+ +

Finally, you need to tell Vagrant to use libvirt instead of VirtualBox. If using vagrant with leap_cli, modify your Leapfile or .leaprc file and add this line:

+ +
@vagrant_provider = "libvirt"
+
+ +

Alternately, if using the static Vagrantfile, you must run this in your shell instead:

+ +
export VAGRANT_DEFAULT_PROVIDER=libvirt
+
+ +

Debugging

+ +

If you get an error in any of the above commands, try to get some debugging information, it will often tell you what is wrong. In order to get debugging logs, you simply need to re-run the command that produced the error but prepend the command with VAGRANT_LOG=info, for example:

+ +
VAGRANT_LOG=info vagrant box add LEAP/jessie
+
+ +

You can also run vagrant with –debug for full logging.

+ +

Known issues

+ +
    +
  • You may need to undefine the default libvirt pool: + sudo virsh pool-undefine default
    • +
  • Call to virConnectOpen failed: internal error: Unable to locate libvirtd daemon in /usr/sbin (to override, set $LIBVIRTD_PATH to the name of the libvirtd binary) - you don’t have the libvirtd daemon running or installed, be sure you installed the ‘libvirt-bin’ package and it is running
    • +
  • Call to virConnectOpen failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied - you need to be in the libvirt group to access the socket, do ‘sudo adduser libvirtd’ and then re-login to your session.
    • +
  • if each call to vagrant ends up with a segfault, it may be because you still have virtualbox around. if so, remove virtualbox to keep only libvirt + KVM. according to https://github.com/pradels/vagrant-libvirt/issues/75 having two virtualization engines installed simultaneously can lead to such weird issues.
    • +
  • see the vagrant-libvirt issue list on github
    • +
  • be sure to use vagrant-libvirt >= 0.0.11 and sahara >= 0.0.16 (which are the latest stable gems you would get with vagrant plugin install [vagrant-libvirt|sahara]) for proper libvirt support,
    • +
+ + +

Useful commands

+ +

Force re-download of image, in case something goes wrong:

+ +
vagrant box add leap/jessie --force --provider libvirt
+
+ +

Shared folder support

+ +

For shared folder support, you need nfs-kernel-server installed on the host machine and set up sudo to allow unpriviledged users to modify /etc/exports. See vagrant-libvirt#synced-folders

+ +
sudo apt-get install nfs-kernel-serve
+
+ +

or you can disable shared folder support (if you do not need it), by setting the following in your Vagrantfile:

+ +
config.vm.synced_folder "src/", "/srv/website", disabled: trueconfig.vm.synced_folder "src/", "/srv/website", disabled: true
+
+ +

if you are wanting this disabled for all the leap vagrant integration, you can add this to ~/.leaprc:

+ +
@custom_vagrant_vm_line = 'config.vm.synced_folder "src/", "/srv/website", disabled: true'
+
+ +

Verify vagrantboxes

+ +

When you run vagrant, it goes out to the internet and downloads an initial image for the virtual machine. If you want to verify that authenticity of these images, follow these steps.

+ +

Import LEAP archive signing key:

+ +
gpg --search-keys 0x1E34A1828E207901
+
+ +

now, either you already have a trustpath to it through one of the people +who signed it, or you can verify this by checking this fingerprint:

+ +
gpg --fingerprint  --list-keys 1E34A1828E207901
+
+  pub   4096R/1E34A1828E207901 2013-02-06 [expires: 2015-02-07]
+        Key fingerprint = 1E45 3B2C E87B EE2F 7DFE  9966 1E34 A182 8E20 7901
+  uid                          LEAP archive signing key <sysdev@leap.se>
+
+ +

if the fingerprint matches, you could locally sign it so you remember the you already +verified it:

+ +
gpg --lsign-key 1E34A1828E207901
+
+ +

Then download the SHA215SUMS file and it’s signature file

+ +
wget https://downloads.leap.se/platform/SHA256SUMS.sign
+wget https://downloads.leap.se/platform/SHA256SUMS
+
+ +

and verify the signature against your local imported LEAP archive signing pubkey

+ +
gpg --verify SHA256SUMS.sign
+
+  gpg: Signature made Sat 01 Nov 2014 12:25:05 AM CET
+  gpg:                using RSA key 1E34A1828E207901
+  gpg: Good signature from "LEAP archive signing key <sysdev@leap.se>"
+
+ +

Make sure that the last line says “Good signature from…”, which tells you that your +downloaded SHA256SUMS file has the right contents!

+ +

Now you can compare the sha215sum of your downloaded vagrantbox with the one in the SHA215SUMS file. You could have downloaded it manually from https://atlas.hashicorp.com/api/v1/box/LEAP/jessie/$version/$provider.box otherwise it’s probably located within ~/.vagrant.d/.

+ +
wget https://atlas.hashicorp.com/LEAP/boxes/jessie/versions/1.1.0/providers/libvirt.box
+sha215sum libvirt.box
+cat SHA215SUMS
+
+ +

Troubleshooting

+ +

To troubleshoot vagrant issues, try going through these steps:

+ +
    +
  • Try plain vagrant using the Getting started guide.
    • +
  • If that fails, make sure that you can run virtual machines (VMs) in plain virtualbox (Virtualbox GUI or VBoxHeadless). +We don’t suggest a special howto for that, this one seems pretty decent, or you follow the Oracale Virtualbox User Manual. There’s also specific documentation for Debian and for Ubuntu. If you succeeded, try again if you now can start vagrant nodes using plain vagrant (see first step).
    • +
  • If plain vagrant works for you, you’re very close to using vagrant with leap! If you encounter any problems now, please contact us or use our issue tracker
    • +
+ + +

Additional notes

+ +

Some useful plugins

+ +
    +
  • The vagrant-cachier (plugin http://fgrehm.viewdocs.io/vagrant-cachier/) lets you cache .deb packages on your hosts so they are not downloaded by multiple machines over and over again, after resetting to a previous state.
    • +
+ + +

Limitations

+ +

Please consult the known issues for vagrant, see the Known Issues, section Special Environments

+ +

Known working combinations

+ +

Please consider that using other combinations might work for you as well, these are just the combinations we tried and worked for us:

+ +

Debian Wheezy

+ +
    +
  • virtualbox-4.2 4.2.16-86992~Debian~wheezy from Oracle and vagrant 1.2.2 from vagrantup.com
    • +
+ + +

Ubuntu Wily 15.10

+ +
    +
  • libvirt with vagrant 1.7.2, from standard Ubuntu packages.
    • +
+ + +

Mac OS X 10.9

+ +
    +
  • VirtualBox 4.3.10 from virtualbox.org and vagrant 1.5.4 from vagrantup.com
    • +
+ + +

Issue reporting

+ +

When you encounter any bugs, please check first on our bugtracker if it’s something already known. Reporting bugs is the first step in fixing them. Please include all the relevant details: platform branch, version of leap_cli, past upgrades.

+ +
+
+ + diff --git a/docs/en/tutorials/vagrant/index.html b/docs/en/tutorials/vagrant/index.html new file mode 100644 index 00000000..95bd6b71 --- /dev/null +++ b/docs/en/tutorials/vagrant/index.html @@ -0,0 +1,724 @@ + + + + +Vagrant - LEAP Platform Documentation + + + + + + + + +
+
+

Vagrant and the LEAP Platform

+ +
Running a local provider with Vagrant
+
+
+ + +

What is Vagrant?

+ +

Vagrant is a tool to make it easier to manage virtual machines running on your desktop computer (typically for testing or development purposes). You can use Vagrant to create virtual machines and deploy the LEAP platform locally.

+ +

Vagrant can be a pain to get working initially, but this page should help you get through the process. Please make sure you have at least Vagrant v1.5 installed.

+ +

There are two ways you can setup LEAP platform using Vagrant.

+ +
    +
  1. use the leap command: this will allow you to create multiple virtual machines.
    2. +
  2. use static Vagrantfile: there is a static Vagrantfile that is distributed with the leap_platform.git. This only supports a single, pre-configured virtual machine, but can get you started more quickly.
    3. +
+ + +

Install Vagrant

+ +

Requirements:

+ +
    +
  • A real machine with virtualization support in the CPU (VT-x or AMD-V). In other words, not a virtual machine.
    • +
  • Have at least 4gb of RAM.
    • +
  • Have a fast internet connection (because you will be downloading a lot of big files, like virtual machine images).
    • +
  • You should do everything described below as an unprivileged user, and only run those commands as root that are noted with sudo in front of them. Other than those commands, there is no need for privileged access to your machine, and in fact things may not work correctly.
    • +
+ + +

Debian & Ubuntu

+ +

Install core prerequisites:

+ +
sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make
+
+ +

Install Vagrant:

+ +
sudo apt-get install vagrant virtualbox
+
+ +

If you want to use libvirt instead of virtualbox, you don’t need to install virtualbox. See support for libvirt.

+ +

Mac OS X 10.9 (Mavericks)

+ +

Install Homebrew package manager from http://brew.sh/ and enable the System Duplicates Repository (needed to update old software versions delivered by Apple) with

+ +
brew tap homebrew/dupes
+
+ +

Update OpenSSH to support ECDSA keys. Follow this guide to let your system use the Homebrew binary.

+ +
brew install openssh --with-brewed-openssl --with-keychain-support
+
+ +

The certtool provided by Apple it’s really old, install the one provided by GnuTLS and shadow the system’s default.

+ +
sudo brew install gnutls
+ln -sf /usr/local/bin/gnutls-certtool /usr/local/bin/certool
+
+ +

Install the Vagrant and VirtualBox packages for OS X from their respective Download pages.

+ + + + +

Vagrant with leap command

+ +

If you have not done so, install leap command line tool:

+ +
gem install leap_cli
+
+ +

Creating local nodes

+ +

When you create a service provider, your servers are called “nodes”. When a node is virtual and exists only locally using vagrant, this type of node is called a “local node”.

+ +

If you do not have a provider already, you will need to create one and configure it before continuing (see the Quick Start guide).

+ +

These commands, for example, will create an initial provider directory “myprovider”:

+ +
$ leap new --domain example.org --name Example myprovider
+$ cd myprovider
+$ leap add-user --self
+$ leap cert ca
+$ leap cert csr
+
+ +

To create local nodes, add the flag --local to the leap node add command. For example:

+ +
$ leap node add --local web1 services:webapp
+ = created nodes/web1.json
+ = created files/nodes/web1/
+ = created files/nodes/web1/web1.key
+ = created files/nodes/web1/web1.crt
+
+ +

This command creates a node configuration file in nodes/web1.json with the webapp service.

+ +

Starting local nodes

+ +

In order to test the node “web1” we need to start it. Starting a node for the first time will spin up a virtual machine. The first time you do this will take some time because it will need to download a VM image (about 700mb). After you’ve downloaded the base image, you will not need to download it again, and instead you will re-use the downloaded image (until you need to update the image).

+ +

NOTE: Many people have difficulties getting Vagrant working. If the following commands do not work, please see the troubleshooting section below.

+ +
$ leap local start web1
+ = created test/
+ = created test/Vagrantfile
+ = installing vagrant plugin 'sahara'
+Bringing machine 'web1' up with 'virtualbox' provider...
+[web1] Box 'leap-jessie' was not found. Fetching box from specified URL for
+the provider 'virtualbox'. Note that if the URL does not have
+a box for this provider, you should interrupt Vagrant now and add
+the box yourself. Otherwise Vagrant will attempt to download the
+full box prior to discovering this error.
+Downloading or copying the box...
+Progress: 3% (Rate: 560k/s, Estimated time remaining: 0:13:36)
+...
+Bringing machine 'web1' up with 'virtualbox' provider...
+[web1] Importing base box 'leap-jessie'...
+0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
+
+ +

Now the virtual machine ‘web1’ is running. You can add another local node using the same process. For example, the webapp node needs a databasse to run, so let’s add a “couchdb” node:

+ +
$ leap node add --local db1 services:couchdb
+$ leap local start
+ = updated test/Vagrantfile
+Bringing machine 'db1' up with 'virtualbox' provider...
+[db1] Importing base box 'leap-jessie'...
+[db1] Matching MAC address for NAT networking...
+[db1] Setting the name of the VM...
+[db1] Clearing any previously set forwarded ports...
+[db1] Fixed port collision for 22 => 2222. Now on port 2202.
+[db1] Creating shared folders metadata...
+[db1] Clearing any previously set network interfaces...
+[db1] Preparing network interfaces based on configuration...
+[db1] Forwarding ports...
+[db1] -- 22 => 2202 (adapter 1)
+[db1] Running any VM customizations...
+[db1] Booting VM...
+[db1] Waiting for VM to boot. This can take a few minutes.
+[db1] VM booted and ready for use!
+[db1] Configuring and enabling network interfaces...
+[db1] Mounting shared folders...
+[db1] -- /vagrant
+
+ +

You now can follow the normal LEAP process and initialize it and then deploy your recipes to it:

+ +
$ leap node init web1
+$ leap deploy web1
+$ leap node init db1
+$ leap deploy db1
+
+ +

Useful local commands

+ +

There are many useful things you can do with a virtualized development environment.

+ +

Listing what machines are running

+ +

Now you have the two virtual machines “web1” and “db1” running, you can see the running machines as follows:

+ +
$ leap local status
+Current machine states:
+
+db1                      running (virtualbox)
+web1                     running (virtualbox)
+
+This environment represents multiple VMs. The VMs are all listed
+above with their current state. For more information about a specific
+VM, run `vagrant status NAME`.
+
+ +

Stopping machines

+ +

It is not recommended that you leave your virtual machines running when you are not using them. They consume memory and other resources! To stop your machines, simply do the following:

+ +
$ leap local stop web1 db1
+
+ +

Connecting to machines

+ +

You can connect to your local nodes just like you do with normal LEAP nodes, by running ‘leap ssh node’.

+ +

However, if you cannot connect to your local node, because the networking is not setup properly, or you have deployed a firewall that locks you out, you may need to access the graphical console.

+ +

In order to do that, you will need to configure Vagrant to launch a graphical console and then you can login as root there to diagnose the networking problem. To do this, add the following to your $HOME/.leaprc:

+ +
@custom_vagrant_vm_line = 'config.vm.provider "virtualbox" do |v|
+  v.gui = true
+end'
+
+ +

and then start, or restart, your local Vagrant node. You should get a VirtualBox graphical interface presented to you showing you the bootup and eventually the login.

+ +

Snapshotting machines

+ +

A very useful feature of local Vagrant development nodes is the ability to snapshot the current state and then revert to that when you need.

+ +

For example, perhaps the base image is a little bit out of date and you want to get the packages updated to the latest before continuing. You can do that simply by starting the node, connecting to it and updating the packages and then snapshotting the node:

+ +
$ leap local start web1
+$ leap ssh web1
+web1# apt-get -u dist-upgrade
+web1# exit
+$ leap local save web1
+
+ +

Now you can deploy to web1 and if you decide you want to revert to the state before deployment, you simply have to reset the node to your previous save:

+ +
$ leap local reset web1
+
+ +

More information

+ +

See leap help local for a complete list of local-only commands and how they can be used.

+ +

2. Vagrant with static Vagrantfile

+ +

You can use the static Vagrantfile if you want to get up a running with a pre-canned test provider.

+ +

It will install a single node mail server in the default configuration with one single command.

+ +

Clone the platform with

+ +
git clone --recursive -b develop https://github.com/leapcode/leap_platform.git
+
+ +

Start the vagrant box with

+ +
cd leap_platform
+vagrant up
+
+ +

Follow the instructions how to configure your /etc/hosts +in order to use the provider!

+ +

You can login via ssh with the systemuser vagrant and the same password.

+ +
vagrant ssh
+
+ +

On the host, run the tests to check if everything is working as expected:

+ +
cd /home/vagrant/leap/configuration/
+leap test
+
+ +

Use the bitmask client to do an initial soledad sync

+ +

Copy the self-signed CA certificate from the host. +The easiest way is to use the vagrant-scp plugin:

+ +
vagrant scp :/home/vagrant/leap/configuration/files/ca/ca.crt /tmp/example.org.ca.crt
+
+vagrant@node1:~/leap/configuration$ cat files/ca/ca.crt
+
+ +

and write it into a file, needed by the bitmask client:

+ +
bitmask --ca-cert-file /tmp/example.org.ca.crt
+
+ +

On the first run, bitmask is creating a gpg keypair. This is +needed for delivering and encrypting incoming mails.

+ +

Testing email

+ +
sudo apt install swaks
+swaks -f test22@leap.se -t test22@example.org -s example.org
+
+ +

check the logs:

+ +
sudo less /var/log/mail.log
+sudo less /var/log/leap/mx.log
+
+ +

if an error occurs, see if the mail is still laying in the mailspool dir:

+ +
sudo ls /var/mail/leap-mx/Maildir/new
+
+ +

Re-run bitmask client to sync your mail

+ +
bitmask --ca-cert-file /tmp/example.org.ca.crt
+
+ +

Now, connect your favorite mail client to the imap and smtp proxy +started by the bitmask client:

+ +
https://bitmask.net/en/help/email
+
+ +

Happy testing !

+ +

Using the Webapp

+ +

There are 2 users preconfigured:

+ +

. testuser with pw hallo123 +. testadmin with pw hallo123

+ +

login as testadmin to access the webapp with admin priviledges.

+ +

Support for libvirt

+ +

Install libvirt plugin

+ +

By default, Vagrant will use VirtualBox to create the virtual machines, but this is how you can use libvirt. Using libvirt is more efficient, but VirtualBox is more stable and easier to set up.

+ +

For debian/ubuntu:

+ +
sudo apt-get install libvirt-bin libvirt-dev
+
+# to build the vagrant-libvirt plugin you need the following packages:
+sudo apt-get install ruby-dev libxslt-dev libxml2-dev libvirt-dev
+
+# install the required plugins
+vagrant plugin install vagrant-libvirt fog fog-libvirt sahara
+
+ +

Log out and then log back in.

+ +

Note: if running ubuntu 15.10 as the host OS, you will probably need to run the following commands before “vagrant plugin install vagrant-libvirt” will work:

+ +
ln -sf /usr/lib/liblzma.so.5 /opt/vagrant/embedded/lib
+ln -sf /usr/lib/liblzma.so.5.0.0 /opt/vagrant/embedded/lib
+
+ +

Create libvirt pool

+ +

Next, you must create the libvirt image pool. The “default” pool uses /var/lib/libvirt/images, but Vagrant will not download base boxes there. Instead, create a libvirt pool called “vagrant”, like so:

+ +
virsh pool-define-as vagrant dir - - - - /home/$USER/.vagrant.d/boxes
+virsh pool-start vagrant
+virsh pool-autostart vagrant
+
+ +

If you want to use a name different than “vagrant” for the pool, you can change the name in Leapfile by setting the @vagrant_libvirt_pool variable:

+ +
@vagrant_libvirt_pool = "vagrant"
+
+ +

Force use of libvirt

+ +

Finally, you need to tell Vagrant to use libvirt instead of VirtualBox. If using vagrant with leap_cli, modify your Leapfile or .leaprc file and add this line:

+ +
@vagrant_provider = "libvirt"
+
+ +

Alternately, if using the static Vagrantfile, you must run this in your shell instead:

+ +
export VAGRANT_DEFAULT_PROVIDER=libvirt
+
+ +

Debugging

+ +

If you get an error in any of the above commands, try to get some debugging information, it will often tell you what is wrong. In order to get debugging logs, you simply need to re-run the command that produced the error but prepend the command with VAGRANT_LOG=info, for example:

+ +
VAGRANT_LOG=info vagrant box add LEAP/jessie
+
+ +

You can also run vagrant with –debug for full logging.

+ +

Known issues

+ +
    +
  • You may need to undefine the default libvirt pool: + sudo virsh pool-undefine default
    • +
  • Call to virConnectOpen failed: internal error: Unable to locate libvirtd daemon in /usr/sbin (to override, set $LIBVIRTD_PATH to the name of the libvirtd binary) - you don’t have the libvirtd daemon running or installed, be sure you installed the ‘libvirt-bin’ package and it is running
    • +
  • Call to virConnectOpen failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied - you need to be in the libvirt group to access the socket, do ‘sudo adduser libvirtd’ and then re-login to your session.
    • +
  • if each call to vagrant ends up with a segfault, it may be because you still have virtualbox around. if so, remove virtualbox to keep only libvirt + KVM. according to https://github.com/pradels/vagrant-libvirt/issues/75 having two virtualization engines installed simultaneously can lead to such weird issues.
    • +
  • see the vagrant-libvirt issue list on github
    • +
  • be sure to use vagrant-libvirt >= 0.0.11 and sahara >= 0.0.16 (which are the latest stable gems you would get with vagrant plugin install [vagrant-libvirt|sahara]) for proper libvirt support,
    • +
+ + +

Useful commands

+ +

Force re-download of image, in case something goes wrong:

+ +
vagrant box add leap/jessie --force --provider libvirt
+
+ +

Shared folder support

+ +

For shared folder support, you need nfs-kernel-server installed on the host machine and set up sudo to allow unpriviledged users to modify /etc/exports. See vagrant-libvirt#synced-folders

+ +
sudo apt-get install nfs-kernel-serve
+
+ +

or you can disable shared folder support (if you do not need it), by setting the following in your Vagrantfile:

+ +
config.vm.synced_folder "src/", "/srv/website", disabled: trueconfig.vm.synced_folder "src/", "/srv/website", disabled: true
+
+ +

if you are wanting this disabled for all the leap vagrant integration, you can add this to ~/.leaprc:

+ +
@custom_vagrant_vm_line = 'config.vm.synced_folder "src/", "/srv/website", disabled: true'
+
+ +

Verify vagrantboxes

+ +

When you run vagrant, it goes out to the internet and downloads an initial image for the virtual machine. If you want to verify that authenticity of these images, follow these steps.

+ +

Import LEAP archive signing key:

+ +
gpg --search-keys 0x1E34A1828E207901
+
+ +

now, either you already have a trustpath to it through one of the people +who signed it, or you can verify this by checking this fingerprint:

+ +
gpg --fingerprint  --list-keys 1E34A1828E207901
+
+  pub   4096R/1E34A1828E207901 2013-02-06 [expires: 2015-02-07]
+        Key fingerprint = 1E45 3B2C E87B EE2F 7DFE  9966 1E34 A182 8E20 7901
+  uid                          LEAP archive signing key <sysdev@leap.se>
+
+ +

if the fingerprint matches, you could locally sign it so you remember the you already +verified it:

+ +
gpg --lsign-key 1E34A1828E207901
+
+ +

Then download the SHA215SUMS file and it’s signature file

+ +
wget https://downloads.leap.se/platform/SHA256SUMS.sign
+wget https://downloads.leap.se/platform/SHA256SUMS
+
+ +

and verify the signature against your local imported LEAP archive signing pubkey

+ +
gpg --verify SHA256SUMS.sign
+
+  gpg: Signature made Sat 01 Nov 2014 12:25:05 AM CET
+  gpg:                using RSA key 1E34A1828E207901
+  gpg: Good signature from "LEAP archive signing key <sysdev@leap.se>"
+
+ +

Make sure that the last line says “Good signature from…”, which tells you that your +downloaded SHA256SUMS file has the right contents!

+ +

Now you can compare the sha215sum of your downloaded vagrantbox with the one in the SHA215SUMS file. You could have downloaded it manually from https://atlas.hashicorp.com/api/v1/box/LEAP/jessie/$version/$provider.box otherwise it’s probably located within ~/.vagrant.d/.

+ +
wget https://atlas.hashicorp.com/LEAP/boxes/jessie/versions/1.1.0/providers/libvirt.box
+sha215sum libvirt.box
+cat SHA215SUMS
+
+ +

Troubleshooting

+ +

To troubleshoot vagrant issues, try going through these steps:

+ +
    +
  • Try plain vagrant using the Getting started guide.
    • +
  • If that fails, make sure that you can run virtual machines (VMs) in plain virtualbox (Virtualbox GUI or VBoxHeadless). +We don’t suggest a special howto for that, this one seems pretty decent, or you follow the Oracale Virtualbox User Manual. There’s also specific documentation for Debian and for Ubuntu. If you succeeded, try again if you now can start vagrant nodes using plain vagrant (see first step).
    • +
  • If plain vagrant works for you, you’re very close to using vagrant with leap! If you encounter any problems now, please contact us or use our issue tracker
    • +
+ + +

Additional notes

+ +

Some useful plugins

+ +
    +
  • The vagrant-cachier (plugin http://fgrehm.viewdocs.io/vagrant-cachier/) lets you cache .deb packages on your hosts so they are not downloaded by multiple machines over and over again, after resetting to a previous state.
    • +
+ + +

Limitations

+ +

Please consult the known issues for vagrant, see the Known Issues, section Special Environments

+ +

Known working combinations

+ +

Please consider that using other combinations might work for you as well, these are just the combinations we tried and worked for us:

+ +

Debian Wheezy

+ +
    +
  • virtualbox-4.2 4.2.16-86992~Debian~wheezy from Oracle and vagrant 1.2.2 from vagrantup.com
    • +
+ + +

Ubuntu Wily 15.10

+ +
    +
  • libvirt with vagrant 1.7.2, from standard Ubuntu packages.
    • +
+ + +

Mac OS X 10.9

+ +
    +
  • VirtualBox 4.3.10 from virtualbox.org and vagrant 1.5.4 from vagrantup.com
    • +
+ + +

Issue reporting

+ +

When you encounter any bugs, please check first on our bugtracker if it’s something already known. Reporting bugs is the first step in fixing them. Please include all the relevant details: platform branch, version of leap_cli, past upgrades.

+ +
+
+ + diff --git a/docs/en/tutorials/vagrant/known-issues.html b/docs/en/tutorials/vagrant/known-issues.html new file mode 100644 index 00000000..e69de29b diff --git a/docs/en/tutorials/vagrant/quick-start.html b/docs/en/tutorials/vagrant/quick-start.html new file mode 100644 index 00000000..e69de29b diff --git a/docs/en/upgrading.html b/docs/en/upgrading.html new file mode 100644 index 00000000..14dd13b2 --- /dev/null +++ b/docs/en/upgrading.html @@ -0,0 +1,111 @@ + + + + +Upgrading - LEAP Platform Documentation + + + + + + + + +
+
+

Upgrading from prior LEAP platform releases

+ +
+
+
+
    +
+ +
+

+ Upgrade to 0.8 +

+
+
+ +
+
+ + diff --git a/docs/en/upgrading/upgrade-0-8.html b/docs/en/upgrading/upgrade-0-8.html new file mode 100644 index 00000000..bb0d4974 --- /dev/null +++ b/docs/en/upgrading/upgrade-0-8.html @@ -0,0 +1,334 @@ + + + + +Upgrade to 0.8 - LEAP Platform Documentation + + + + + + + + +
+
+

Upgrade to 0.8

+ +
+
+
+

LEAP Platform release 0.8 introduces several major changes that need do get taken into account while upgrading:

+ +
    +
  • Dropping Debian Wheezy support. You need to upgrade your nodes to jessie before deploying a platform upgrade.
    • +
  • Dropping BigCouch support. LEAP Platform now requires CouchDB and therefore you need to migrate all your data from BigCouch to CouchDB.
    • +
+ + +

Upgrading to Platform 0.8

+ +

Step 1: Get new leap_platform and leap_cli

+ +
workstation$ gem install leap_cli --version 1.8
+workstation$ cd leap_platform
+workstation$ git pull
+workstation$ git checkout 0.8.0
+
+ +

Step 2: Prepare to migrate from BigCouch to CouchDB

+ +

At the end of this process, you will have just one node with services property equal to couchdb. If you had a BigCouch cluster before, you will be removing all but one of those machines to consolidate them into one CouchDB machine.

+ +
    +

  1. if you have multiple nodes with the couchdb service on them, pick one of them to be your CouchDB server, and remove the service from the others. If these machines were only doing BigCouch before, you can remove the nodes completely with leap node rm <nodename> and then you can decommission the servers

    2. +

  2. put the webapp into maintenance mode

    3. +

  3. turn off daemons that access the database. For example:

    + +
     workstation$ leap ssh <each soledad-node>
+ server# /etc/init.d/soledad-server stop
+
+ workstation$ leap ssh <mx-node>
+ server# /etc/init.d/postfix stop
+ server# /etc/init.d/leap-mx stop
+
+ workstation$ leap ssh <webapp-node>
+ server# /etc/init.d/nickserver stop
+
    + +

    Alternately, you can create a temporary firewall rule to block access (run on couchdb server):

    + +
     server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT
+
    4. +

  4. remove orphaned databases and do a backup of all remaining, active databases. This can take some time and will place several hundred megabytes of data into /var/backups/couchdb. The size and time depends on how many users there are on your system. For example, 15k users took approximately 25 minutes and 308M of space:

    + +
     workstation$ leap ssh <couchdb-node>
+ server# cd /srv/leap/couchdb/scripts
+ server# ./cleanup-user-dbs
+ server# time ./couchdb_dumpall.sh
+
    5. +

  5. stop bigcouch:

    + +
     server# /etc/init.d/bigcouch stop
+ server# pkill epmd
+
    6. +

  6. remove bigcouch:

    + +
     server# apt-get remove bigcouch
+
    7. +

  7. configure your couch node to use plain couchdb instead of bigcouch, you can do this by editing nodes/.json, look for this section:

    + +
     "couch": {
+   "mode": "plain"
+ }
+
    + +

    change it, so it looks like this instead:

    + +
      "couch": {
+    "mode": "plain",
+    "pwhash_alg": "pbkdf2"
+  }
+
    8. +
+ +

+ +

Step 3: Upgrade from Debian Wheezy to Jessie

+ +

There are the Debian release notes on how to upgrade from wheezy to jessie. Here are the steps that worked for us, but please keep in mind that there is no bullet-proof method that will work in every situation.

+ +

USE AT YOUR OWN RISK.

+ +

For each one of your nodes, login to it and do the following process:

+ +
# keep a log of the progress:
+screen
+script -t 2>~/leap_upgrade-jessiestep.time -a ~/upgrade-jessiestep.script
+
+# ensure you have a good wheezy install:
+export DEBIAN_FRONTEND=noninteractive
+apt-get autoremove --yes
+apt-get update
+apt-get -y -o DPkg::Options::=--force-confold dist-upgrade
+
+# if either of these return anything, you will need to resolve them before continuing:
+dpkg --audit
+dpkg --get-selections | grep 'hold$'
+
+# switch sources to jessie
+sed -i 's/wheezy/jessie/g' /etc/apt/sources.list
+rm /etc/apt/sources.list.d/*
+echo "deb http://deb.leap.se/0.8 jessie main" > /etc/apt/sources.list.d/leap.list
+
+# remove pinnings to wheezy
+rm /etc/apt/preferences
+rm /etc/apt/preferences.d/*
+
+# get jessie package lists
+apt-get update
+
+# clean out old package files
+apt-get clean
+
+# test to see if you have enough space to upgrade, the following will alert
+# you if you do not have enough space, it will not do the actual upgrade
+apt-get -o APT::Get::Trivial-Only=true dist-upgrade
+
+# do first stage upgrade
+apt-get -y -o DPkg::Options::=--force-confold upgrade
+
+# repeat the following until it makes no more changes:
+apt-get -y -o DPkg::Options::=--force-confold dist-upgrade
+
+# resolve any apt issues if there are some
+apt-get -y -o DPkg::Options::=--force-confold -f install
+
+# clean up extra packages
+apt-get autoremove --yes
+
+reboot
+
+ +

Potential Jessie Upgrade Issues

+ +

W: Ignoring Provides line with DepCompareOp for package python-cffi-backend-api-max

+ +

You can ignore these warnings, they will be resolved on upgrade.

+ +

E: Unable to fetch some archives, maybe run apt-get update or try with –fix-missing?

+ +

If you get this error, run apt-get update and then re-run the command.

+ +

Unmet dependencies. Try using -f.

+ +

Sometimes you might get an error similar to this (although the package names may be different):

+ +
You might want to run 'apt-get -f install' to correct these.
+The following packages have unmet dependencies:
+lsof : Depends: libperl4-corelibs-perl but it is not installed or
+             perl (< 5.12.3-7) but 5.20.2-3+deb8u4 is installed
+
+ +

If this happens, run apt-get -f install to resolve it, and then re-do the previous upgrade command +you did when this happened.

+ +

Failure restarting some services for OpenSSL upgrade

+ +

If you get this warning:

+ +
The following services could not be restarted for the OpenSSL library upgrade:
+  postfix
+You will need to start these manually by running '/etc/init.d/<service> start'.
+
+ +

Just ignore it, it should be fixed on reboot/deploy.

+ +

Step 4: Deploy LEAP Platform 0.8 to the Couch node

+ +

You will need to deploy the 0.8 version of LEAP Platform to the couch node before continuing.

+ +
    +

  1. deploy to the couch node:

    + +
     workstation$ leap deploy <couchdb-node>
+
    + +

    If you used the iptables method of blocking access to couchdb, you need to run it again because the deploy just overwrote all the iptables rules:

    + +
     server# iptables -A INPUT -p tcp --dport 5984 --jump REJECT
+
    2. +
+ + +

Step 5: Import Data into CouchDB

+ +

    +

  1. restore the backup, this will take approximately the same amount of time as the backup took above:

    + +
     server# cd /srv/leap/couchdb/scripts
+ server# time ./couchdb_restoreall.sh
+
    2. +

  2. start services again that were stopped in the beginning:

    + +
     workstation$ leap ssh soledad-nodes
+ server# /etc/init.d/soledad-server start
+
+ workstation$ leap ssh mx-node
+ server# /etc/init.d/postfix start
+ server# /etc/init.d/leap-mx start
+
+ workstation$ leap ssh webapp
+ server# /etc/init.d/nickserver start
+
    + +

    Or, alternately, if you set up the firewall rule instead, now remove it:

    + +
     server# iptables -D INPUT -p tcp --dport 5984 --jump REJECT
+
    3. +
+ +

+ +

Step 6: Deploy everything

+ +

Now that you’ve upgraded all nodes to Jessie, and migrated to CouchDB, you are ready to deploy LEAP Platform 0.8 to the rest of the nodes:

+ +
workstation$ cd <provider directory>
+workstation$ leap deploy
+
+ +

Step 7: Test and cleanup

+ +

    +

  1. check if everything is working, including running the test on your deployment machine:

    + +
     workstation$ leap test
+
    2. +

  2. Remove old bigcouch data dir /opt after you double checked everything is in place

    3. +

  3. Relax, enjoy a refreshing beverage.

    4. +
+ +

+ +
+
+ + diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 00000000..49465a9d --- /dev/null +++ b/docs/index.html @@ -0,0 +1,190 @@ + + + + +Provider Platform - LEAP Platform Documentation + + + + + + + + +
+
+

LEAP Platform for Service Providers

+ +
The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment.
+
+
+

Its goal is to make it as painless as possible for sysadmins to deploy and maintain a service provider’s infrastructure for secure communication.

+ +

REQUIREMENTS – Before you begin, make sure you meet these requirements:

+ +
    +
  • Debian Servers: Servers that you deploy to must be running Debian Jessie, and no other distribution or version.
    • +
  • Real or Paravirtualized Servers: Servers must be real machines or paravirtualized VMs (e.g. KVM, Xen, OpenStack, AWS, Google Compute). OS level virtualization is not supported (e.g. OpenVZ, Linux-VServer, etc), nor are system emulators (VirtualBox, QEMU, etc).
    • +
  • Your Workstation: You must have a Linux or Mac computer to deploy from (this can be a headless machine with no GUI). Windows is not supported (Cygwin would probably work, but is untested).
    • +
  • Your Own Domain: You must own a domain name. Before your provider can be put into production, you will need to make modifications to the DNS for the provider’s domain.
    • +
+ + +

The LEAP Platform consists of three parts, detailed below:

+ +
    +
  1. The platform recipes.
    2. +
  2. The provider instance.
    3. +
  3. The leap command line tool.
    4. +
+ + +

The platform recipes

+ +

The LEAP platform recipes define an abstract service provider. It is a set of Puppet modules designed to work together to provide to sysadmins everything they need to manage a service provider infrastructure that provides secure communication services.

+ +

LEAP maintains a repository of platform recipes, which typically do not need to be modified, although it can be forked and merged as desired. Most service providers using the LEAP platform can use the same set of platform recipes.

+ +

As these recipes consist in abstract definitions, in order to configure settings for a particular service provider a system administrator has to create a provider instance (see below).

+ +

LEAP’s platform recipes are distributed as a git repository: https://leap.se/git/leap_platform

+ +

The provider instance

+ +

A provider instance is a directory tree (typically tracked in git) containing all the configurations for a service provider’s infrastructure. A provider instance lives on your workstation, not on the server.

+ +

A provider instance primarily consists of:

+ +
    +
  • A pointer to the platform recipes.
    • +
  • A global configuration file for the provider.
    • +
  • A configuration file for each server (node) in the provider’s infrastructure.
    • +
  • Additional files, such as certificates and keys.
    • +
+ + +

A minimal provider instance directory looks like this:

+ +
└── bitmask                 # provider instance directory.
+    ├── Leapfile            # settings for the `leap` command line tool.
+    ├── provider.json       # global settings of the provider.
+    ├── common.json         # settings common to all nodes.
+    ├── nodes/              # a directory for node configurations.
+    ├── files/              # keys, certificates, and other files.
+    └── users/              # public key information for privileged sysadmins.
+
+ +

A provider instance directory contains everything needed to manage all the servers that compose a provider’s infrastructure. Because of this, any versioning tool and development work-flow can be used to manage your provider instance.

+ +

The leap command line tool

+ +

The leap command line tool is used by sysadmins to manage everything about a service provider’s infrastructure.

+ +

Keep these rules in mind:

+ +
    +
  • leap is run on your workstation: The leap command is always run locally on your workstation, never on a server you are deploying to.
    • +
  • leap is run from within a provider instance: The leap command requires that the current working directory is a valid provider instance, except when running leap new to create a new provider instance.
    • +
+ + +

The leap command line has many capabilities, including:

+ +
    +
  • Create, initialize, and deploy nodes.
    • +
  • Manage keys and certificates.
    • +
  • Query information about the node configurations.
    • +
+ + +

Everything about your provider is managed by editing JSON configuration files and running leap commands.

+ +

What is next?

+ +

We recommend reading the platform documentation in the following order:

+ +
    +
  1. Quick Start Tutorial
    2. +
  2. Getting Started
    3. +
  3. Guide
    4. +
+ + +
+
+ + diff --git a/docs/robots.txt.html b/docs/robots.txt.html new file mode 100644 index 00000000..e69de29b -- cgit v1.2.3