Merge remote-tracking branch 'origin/0.8.x' into develop

7 files changed, 724 insertions, 156 deletions

diff --git a/doc/guide/commands.md b/doc/guide/commands.md
index eaacc8d5..2ddacb83 100644
--- a/doc/guide/commands.md
+++ b/doc/guide/commands.md
@@ -1,5 +1,5 @@
 @title = 'Command Line Reference'
-@summary = "A copy of leap --help"
+@summary = 'A copy of leap --help'
 
 The command "leap" can be used to manage a bevy of servers running the LEAP platform from the comfort of your own home.
 
@@ -7,7 +7,7 @@ The command "leap" can be used to manage a bevy of servers running the LEAP plat
 # Global Options
 
 * `--log FILE`
-Override default log file
+Override default log file.
 Default Value: None
 
 * `-v|--verbose LEVEL`
@@ -15,19 +15,22 @@ Verbosity level 0..5
 Default Value: 1
 
 * `--[no-]color`
-Disable colors in output
+Disable colors in output.
 
-* `--debug`
-Enable debugging library (leap_cli development only)
+* `-d|--debug`
+Print full stack trace for exceptions and load `debugger` gem if installed.
+
+* `--force`
+Like --yes, but also skip prompts that are potentially dangerous to skip.
 
 * `--help`
 Show this message
 
 * `--version`
-Display version number and exit
+Display version number and exit.
 
 * `--yes`
-Skip prompts and assume "yes"
+Skip prompts and assume "yes".
 
 
 # leap add-user  USERNAME
@@ -66,19 +69,51 @@ See see what values are used in the generation of the certificates (like name an
 
 Creates a CSR for use in buying a commercial X.509 certificate.
 
-Unless specified, the CSR is created for the provider's primary domain. The properties used for this CSR come from `provider.ca.server_certificates`.
+Unless specified, the CSR is created for the provider's primary domain. The properties used for this CSR come from `provider.ca.server_certificates`, but may be overridden here.
 
 **Options**
 
+* `--bits BITS`
+Override default certificate bit length
+Default Value: None
+
+* `--country|-C COUNTRY`
+Set C in distinguished name.
+Default Value: None
+
+* `--digest DIGEST`
+Override default signature digest
+Default Value: None
+
 * `--domain DOMAIN`
 Specify what domain to create the CSR for.
-Unless specified, the CSR is created for the provider's primary domain. The properties used for this CSR come from `provider.ca.server_certificates`.
+Unless specified, the CSR is created for the provider's primary domain. The properties used for this CSR come from `provider.ca.server_certificates`, but may be overridden here.
+Default Value: None
+
+* `--email EMAIL`
+Set emailAddress in distinguished name.
+Default Value: None
+
+* `--locality|-L LOCALITY`
+Set L in distinguished name.
+Default Value: None
+
+* `--organization|-O ORGANIZATION`
+Override default O in distinguished name.
+Default Value: None
+
+* `--state|--ST STATE`
+Set ST in distinguished name.
+Default Value: None
+
+* `--unit|--OU UNIT`
+Set OU in distinguished name.
 Default Value: None
 
 
 ## leap cert dh
 
-Creates a Diffie-Hellman parameter file, needed for forward secret OpenVPN ciphers. You don't need this file if you don't provide the VPN service.
+Creates a Diffie-Hellman parameter file, needed for forward secret OpenVPN ciphers.
 
 
 
@@ -112,9 +147,27 @@ Compiles node configuration files into hiera files used for deployment.
 
 
 
+## leap compile firewall
+
+Prints a list of firewall rules. These rules are already implemented on each node, but you might want the list of all rules in case you also have a restrictive network firewall.
+
+
+
+## leap compile hosts
+
+Print entries suitable for an /etc/hosts file, useful for testing your provider.
+
+
+
+## leap compile provider.json
+
+Compile provider.json bootstrap files for your provider.
+
+
+
 ## leap compile zone
 
-Compile a DNS zone file for your provider.
+Prints a DNS zone file for your provider.
 
 
 Default Command: all
@@ -127,9 +180,26 @@ Database commands.
 
 ## leap db destroy  [FILTER]
 
-Destroy all the databases. If present, limit to FILTER nodes.
+Destroy one or more databases. If present, limit to FILTER nodes. For example `leap db destroy --db sessions,tokens testing`.
+
+
 
+**Options**
 
+* `--db DATABASES`
+Comma separated list of databases to destroy (no space). Use "--db all" to destroy all databases.
+Default Value: None
+
+* `--user USERS`
+Comma separated list of usernames. The storage databases for these user(s) will be destroyed.
+Default Value: None
+
+
+# leap debug  FILTER
+
+Output debug information.
+
+The FILTER can be the name of a node, service, or tag.
 
 # leap deploy  FILTER
 
@@ -149,18 +219,21 @@ Default Value: None
 
 * `--tags TAG[,TAG]`
 Specify tags to pass through to puppet (overriding the default).
-Default Value: leap_base,leap_service
+Default Value: None
 
 * `--dev`
 Development mode: don't run 'git submodule update' before deploy.
 
+* `--downgrade`
+Allows deploy to run with an older platform version.
+
 * `--fast`
 Makes the deploy command faster by skipping some slow steps. A "fast" deploy can be used safely if you recently completed a normal deploy.
 
 * `--force`
 Deploy even if there is a lockfile.
 
-* `--[no-]sync`
+* `--sync`
 Sync files, but don't actually apply recipes.
 
 
@@ -170,9 +243,9 @@ Manipulate and query environment information.
 
 The 'environment' node property can be used to isolate sets of nodes into entirely separate environments. A node in one environment will never interact with a node from another environment. Environment pinning works by modifying your ~/.leaprc file and is dependent on the absolute file path of your provider directory (pins don't apply if you move the directory)
 
-## leap env ls
+## leap env ls  [ENVIRONMENT]
 
-List the available environments. The pinned environment, if any, will be marked with '*'.
+List the available environments. The pinned environment, if any, will be marked with '*'. Will also set the pin if run with an environment argument.
 
 
 
@@ -213,6 +286,26 @@ Gets help for the application or its commands. Can also list the commands in a w
 List commands one per line, to assist with shell completion
 
 
+# leap history  FILTER
+
+Display recent deployment history for a set of nodes.
+
+The FILTER can be the name of a node, service, or tag.
+
+**Options**
+
+* `--ip IPADDRESS`
+Override the default SSH IP address.
+Default Value: None
+
+* `--port PORT`
+Override the default SSH port.
+Default Value: None
+
+* `--last`
+Show last deploy only
+
+
 # leap inspect  FILE
 
 Prints details about a file. Alternately, the argument FILE can be the name of a node, service or tag.
@@ -275,6 +368,13 @@ Starts up the virtual machine(s)
 
 
 
+**Options**
+
+* `--basebox BASEBOX`
+The basebox to use. This value is passed to vagrant as the `config.vm.box` option. The value here should be the name of an installed box or a shorthand name of a box in HashiCorp's Atlas.
+Default Value: LEAP/jessie
+
+
 ## leap local status  [FILTER]
 
 Print the status of local virtual machine(s)
@@ -293,6 +393,17 @@ Log in to the specified node with an interactive shell using mosh (requires node
 
 
 
+**Options**
+
+* `--port SSH_PORT`
+Override default SSH port used when trying to connect to the server. Same as `--ssh "-p SSH_PORT"`.
+Default Value: None
+
+* `--ssh arg`
+Pass through raw options to ssh (e.g. `--ssh '-F ~/sshconfig'`).
+Default Value: None
+
+
 # leap new  DIRECTORY
 
 Creates a new provider instance in the specified directory, creating it if necessary.
@@ -376,6 +487,18 @@ Removes all the files related to the node named NAME.
 
 
 
+# leap scp  FILE1 FILE2
+
+Secure copy from FILE1 to FILE2. Files are specified as NODE_NAME:FILE_PATH. For local paths, omit "NODE_NAME:".
+
+
+
+**Options**
+
+* `-r`
+Copy recursively
+
+
 # leap ssh  NAME
 
 Log in to the specified node with an interactive shell.
@@ -384,12 +507,12 @@ Log in to the specified node with an interactive shell.
 
 **Options**
 
-* `--port arg`
-Override ssh port for remote host
+* `--port SSH_PORT`
+Override default SSH port used when trying to connect to the server. Same as `--ssh "-p SSH_PORT"`.
 Default Value: None
 
 * `--ssh arg`
-Pass through raw options to ssh (e.g. --ssh '-F ~/sshconfig')
+Pass through raw options to ssh (e.g. `--ssh '-F ~/sshconfig'`).
 Default Value: None
 
 
@@ -405,9 +528,9 @@ Creates files needed to run tests.
 
 
 
-## leap test run
+## leap test run  [FILTER]
 
-Run tests.
+Run the test suit on FILTER nodes.
 
 
 
@@ -417,3 +540,20 @@ Run tests.
 Continue over errors and failures (default is --no-continue).
 
 Default Command: run
+
+# leap tunnel  [LOCAL_PORT:]NAME:REMOTE_PORT
+
+Creates an SSH port forward (tunnel) to the node NAME. REMOTE_PORT is the port on the remote node that the tunnel will connect to. LOCAL_PORT is the optional port on your local machine. For example: `leap tunnel couch1:5984`.
+
+
+
+**Options**
+
+* `--port SSH_PORT`
+Override default SSH port used when trying to connect to the server. Same as `--ssh "-p SSH_PORT"`.
+Default Value: None
+
+* `--ssh arg`
+Pass through raw options to ssh (e.g. --ssh '-F ~/sshconfig').
+Default Value: None
+
diff --git a/doc/guide/config.md b/doc/guide/config.md
index be67e6bd..bcea26c4 100644
--- a/doc/guide/config.md
+++ b/doc/guide/config.md
@@ -1,39 +1,69 @@
 @title = "Configuration Files"
-@summary = "How to edit configuration files."
+@summary = "Understanding and editing the configuration files."
 
 Files
 -------------------------------------------
 
-Here are a list of some of the common files that make up a provider. Except for Leapfile and provider.json, the files are optional. Unless otherwise specified, all file names are relative to the 'provider directory' root (where the Leapfile is).
-
-`Leapfile` -- If present, this file tells `leap` that the directory is a provider directory. This file is usually empty, but can contain global options.
-
-`~/.leaprc` -- Evaluated the same as Leapfile, but not committed to source control.
-
-`provider.json` -- Global options related to this provider.
-
-`provider.ENVIRONMENT.json` -- Global options for the provider that are applied to only a single environment.
-
-`common.json` -- All nodes inherit from this file.
-
-`secrets.json` -- An automatically generated file that contains any randomly generated strings needed in order to deploy. These strings are often secret and should be protected, although any need for a random string or number that is remembered will produce another entry in this file. This file is automatically generated and refreshed each time you run `leap compile` or `leap deploy`. If an entry is no longer needed, it will get removed. If you want to change a secret, you can remove this file and have it regenerated, or remove the particular line item and just those items will be created anew.
-
-`facts.json` -- If some of your servers are running on AWS or OpenStack, you will need to discover certain properties about how networking is configured on these machines in order for a full deploy to work. In these cases, make sure to run `leap facts update` to periodically regenerate the facts.json file.
-
-`nodes/NAME.json` -- The configuration file for node called NAME.
-
-`services/SERVICE.json` -- The properties in this configuration file are applied to any node that includes SERVICE in its `services` property.
-
-`services/SERVICE.ENVIRONMENT.json` -- The properties in this configuration file are applied to any node that includes SERVICE in its services and has environment equal to ENVIRONMENT.
-
-`services/TAG.json` -- The properties in this configuration file are applied to any node that has includes TAG in its `tags` property.
-
-`services/TAG.ENVIRONMENT.json` -- The properties in this configuration file are applied to any node that has includes TAG in its `tags` property and has `environment` property equal to ENVIRONMENT.
-
-`files/*` -- Various static files used by the platform (e.g. keys, certificates, webapp customization, etc).
-
-`users/USER/` -- A directory that stores the public keys of the sysadmin with name USER. This person will have root access to all the servers.
-
+Here are a list of some of the common files that make up a provider. Except for `Leapfile` and `provider.json`, the files are optional. Unless otherwise specified, all file names are relative to the 'provider directory' root (where the Leapfile is).
+
+<table class="table table-striped">
+<tr>
+  <td><code>Leapfile</code></td>
+  <td>If present, this file tells <code>leap</code> that the directory is a provider directory. This file is usually empty, but can contain global options.</td>
+</tr>
+<tr>
+  <td><code>~/.leaprc</code></td>
+  <td>Evaluated the same as Leapfile, but not committed to source control.</td>
+</tr>
+<tr>
+  <td><code>provider.json</code></td>
+  <td>Global options related to this provider. See [[provider-configuration]].</td>
+</tr>
+<tr>
+  <td><code>provider.ENVIRONMENT.json</code></td>
+  <td>Global options for the provider that are applied to only a single environment.</td>
+</tr>
+<tr>
+  <td><code>nodes/NAME.json</code></td>
+  <td>The configuration file for node called NAME.</td>
+</tr>
+<tr>
+  <td><code>common.json</code></td>
+  <td>All nodes inherit from this file. In other words, any options that appear in <code>common.json</code> will be added as default values to each node configuration, value that can be locally overridden.</td>
+</tr>
+<tr>
+  <td><code>services/SERVICE.json</code></td>
+  <td>The properties in this configuration file are applied to any node that includes SERVICE in its <code>services</code> property.</td>
+</tr>
+<tr>
+  <td><code>services/SERVICE.ENVIRONMENT.json</code></td>
+  <td>The properties in this configuration file are applied to any node that includes SERVICE in its services and has environment equal to ENVIRONMENT.</td>
+</tr>
+<tr>
+  <td><code>tags/TAG.json</code></td>
+  <td>The properties in this configuration file are applied to any node that has includes TAG in its <code>tags</code> property.</td>
+</tr>
+<tr>
+  <td><code>tags/TAG.ENVIRONMENT.json</code></td>
+  <td>The properties in this configuration file are applied to any node that has includes TAG in its <code>tags</code> property and has <code>environment</code> property equal to ENVIRONMENT.</td>
+</tr>
+<tr>
+  <td><code>secrets.json </code></td>
+  <td>An automatically generated file that contains any randomly generated strings needed in order to deploy. These strings are often secret and should be protected, although any need for a random string or number that is remembered will produce another entry in this file. This file is automatically generated and refreshed each time you run <code>leap compile</code> or <code>leap deploy</code>. If an entry is no longer needed, it will get removed. If you want to change a secret, you can remove this file and have it regenerated, or remove the particular line item and just those items will be created anew.</td>
+</tr>
+<tr>
+  <td><code>facts.json</code></td>
+  <td>If some of your servers are running on AWS or OpenStack, you will need to discover certain properties about how networking is configured on these machines in order for a full deploy to work. In these cases, make sure to run <code>leap facts update</code> to periodically regenerate the facts.json file.</td>
+</tr>
+<tr>
+  <td><code>files/*</code></td>
+  <td>Various static files used by the platform (e.g. keys, certificates, webapp customization, etc). In general, only generated files and files used to customize the provider (such as images) live in the <code>files</code> directory.</td>
+</tr>
+<tr>
+  <td><code>users/USER/</code></td>
+  <td>A directory that stores the public keys of the sysadmin with name USER. This person will have root access to all the servers.</td>
+</tr>
+</table>
 
 Leapfile
 -------------------------------------------
@@ -51,8 +81,10 @@ Platform options:
 
 Vagrant options:
 
+* `@vagrant_provider`. Changes the default vagrant provider ("virtualbox"). For example, `@vagrant_provider = "libvirt"`.
 * `@vagrant_network`. Allows you to override the default network used for local nodes. It should include a netmask like `@vagrant_network = '10.0.0.0/24'`.
 * `@custom_vagrant_vm_line`. Insert arbitrary text into the auto-generated Vagrantfile. For example, `@custom_vagrant_vm_line = "config.vm.boot_mode = :gui"`.
+* `@vagrant_basebox` allows specifying a different basebox as the default one. For example, `@vagrant_basebox = "LEAP/jessie"`.
 
 Logging options:
 
@@ -69,7 +101,7 @@ All configuration files, other than `Leapfile`, are in the JSON format. For exam
       "key2": "value2"
     }
 
-Keys should match `/[a-z0-9_]/`
+Keys should match `/[a-z0-9_]/` and must be in double quotes.
 
 Unlike traditional JSON, comments are allowed. If the first non-whitespace characters are `//` then the line is treated as a comment.
 
@@ -141,6 +173,71 @@ The `provider_base` directory is under the `leap_platform` specified in the file
 
 To see all the variables a node has inherited, you could run `leap inspect willamette`.
 
+### Inheritance rules
+
+Suppose you have a node configuration `mynode.json`:
+
+    {
+      "tags": "production",
+      "simple_value": 100,
+      "replaced_array": ["dolphin", "kangaroo"],
+      "+add_array": ["red", "black"],
+      "-subtract_array": ["bitter"],
+      "converted_to_array": "not_array_element",
+      "!override": ["insist on this value"],
+      "hash": {
+        "key1": 1,
+        "key2": 2
+      }
+    }
+
+And a file `tags/production.json`:
+
+    {
+      "simple_value": 99999,
+      "replaced_array": ["zebra"],
+      "add_array": ["green],
+      "subtract_array": ["bitter", "sweet", "salty"],
+      "converted_to_array": ["array_element"],
+      "override": "this value will be overridden",
+      "hash": {
+        "key1": "one"
+      }
+    }
+
+In this scenario, `mynode.json` will inherit from `production.json`. The output of this inheritance will be:
+
+    {
+      "tags": "production",
+      "simple_value": 100,
+      "replaced_array": ["dolphin", "kangaroo"],
+      "add_array": ["red", "black", "green"],
+      "subtract_array": ["sweet", "salty"],
+      "converted_to_array": ["not_array_element", "array_element"],
+      "override": ["insist on this value"],
+      "hash": {
+        "key1": 1,
+        "key2": 2
+      }
+
+The rules for inheritance (where 'old' refers to the parent, and 'new' refers to the child):
+
+* Simple values (strings, numbers, boolean):
+  * Replace the old value with the new value.
+* Array values:
+  * Two arrays: replace the old array with the new array.
+  * One array and one simple value: add the simple value to the array.
+  * If property name is prefixed with "+": merge the old and new arrays.
+  * If property name is prefixed with "-": subtract new array from old array.
+* Hash values:
+  * Hashes are always merged (the result includes the keys of both hashes). If there is a key in common, the new one overrides the old one.
+* Mismatch:
+  * Although you can mix arrays and simple values, you cannot mix arrays with hashes or hashes with simple values. If you attempt to do so, it will fail to compile and give you an error message.
+* Override:
+  * If property name is prefixed with "!": then ensure that new value is always used, regardless of old value. In this case, the override takes precedence over type checking, so you will never get a type mismatch.
+
+NOTE: special property name prefixes, like "+", "-", or "!", are not included in the property name. These prefixes determine the merge strategy, but are stripped out when compiling the resulting JSON file.
+
 Common configuration options
 ----------------------------------------
 
diff --git a/doc/guide/domains.md b/doc/guide/domains.md
new file mode 100644
index 00000000..914bce33
--- /dev/null
+++ b/doc/guide/domains.md
@@ -0,0 +1,129 @@
+@title = "Domains"
+@summary = "How to handle domain names and integrating LEAP with existing services."
+@toc = true
+
+Overview
+--------------------------------
+
+Deploying LEAP can start to get very tricky when you need to integrate LEAP services with an existing domain that you already use or which already has users. Most of this complexity is unavoidable, although there are a few things we plan to do in the future to make this a little less painful.
+
+Because integration with legacy systems is an advanced topic, we recommend that you begin with a new domain. Once everything works and you are comfortable with your LEAP-powered infrastructure, you can then contemplate integrating with your existing domain.
+
+### Definitions
+
+**provider domain**
+
+This is the main domain used to identify the provider. The **provider domain** is what the user enters in the Bitmask client. e.g. `example.org`. The full host name of every node in your provider infrastructure will use the **provider domain** (e.g. `dbnode.example.org`).
+
+In order for the Bitmask client to get configured for use with a provider, it must be able to find the `provider.json` bootstrap file at the root of the **provider domain**. This is not needed if the Bitmask client is "pre-seeded" with the provider's information (these providers show up in a the initial list of available providers).
+
+**webapp domain**
+
+This is the domain that runs the leap_web application that allows users to register accounts, create help tickets, etc. e.g. `example.org` or `user.example.org`. The **webapp domain** defaults to the **provider domain** unless it is explicitly configured separately.
+
+**API domain**
+
+This is the domain that the provider API runs on. Typically, this is set automatically and you never need to configure it. The user should never be aware of this domain. e.g. `api.example.org`. The Bitmask client discovers this API domain by reading it from the `provider.json` file it grabs from the **provider domain**.
+
+**mail domain**
+
+This is the domain used for mail accounts, e.g. `username@example.org`. Currently, this is always the **provider domain**, but it may be independently configurable in the future.
+
+Generating a zone file
+-----------------------------------
+
+Currently, the platform does not include a dedicated `dns` service type, so you need to have your own setup for DNS. You can generate the appropriate configuration options with this command:
+
+    leap compile zone
+
+A single domain
+-------------------------------
+
+The easy approach is to use a single domain for **provider domain**, **webapp domain**, and **email domain**. This will install the webapp on the **provider domain**, which means that this domain must be a new one that you are not currently using for anything.
+
+To configure a single domain, just set the domain in `provider.json`:
+
+    {
+      "domain": "example.org"
+    }
+
+If you have multiple environments, you can specify a different **provider domain** for each environment. For example:
+
+`provider.staging.json`
+
+    {
+      "domain": "staging.example.org"
+    }
+
+A separate domain for the webapp
+--------------------------------------
+
+It is possible make the **webapp domain** different than the **provider domain**. This is needed if you already have a website running at your **provider domain**.
+
+In order to put webapp on a different domain, you must take two steps:
+
+1. You must configure `webapp.domain` for nodes with the `webapp` service.
+2. You must make the compiled `provider.json` available at the root of the **provider domain**.
+
+NOTE: This compiled provider.json is different than the provider.json that you edit and lives in the root of the provider directory.
+
+### Step 1. Configuring `webapp.domain`
+
+In `services/webapp.json`:
+
+    {
+      "webapp": {
+        "domain": "user.example.org"
+      }
+    }
+
+### Step 2. Putting the compiled `provider.json` in place
+
+Generate the compiled `provider.json`:
+
+    leap compile provider.json
+    = created files/web/bootstrap/
+    = created files/web/bootstrap/README
+    = created files/web/bootstrap/production/
+    = created files/web/bootstrap/production/provider.json
+    = created files/web/bootstrap/production/htaccess
+    = created files/web/bootstrap/staging/
+    = created files/web/bootstrap/staging/provider.json
+    = created files/web/bootstrap/staging/htaccess
+
+This command compiles a separate `provider.json` for each environment, or "default" if you don't have an environment. In the example above, there is an environment called "production" and one called "staging", but your setup will probably differ.
+
+The resulting `provider.json` file must then be put at the root URL of your **provider domain** for the appropriate environment.
+
+There is one additional complication: currently, the Bitmask client tests for compatibility using some HTTP headers on the `/provider.json` response. This is will hopefully change in the future, but for now you need to ensure the right headers are set in the response. The included file `htaccess` has example directives for Apache, if that is what you use.
+
+This step can be skipped if you happen to use the `static` service to deploy an `amber` powered static website to **provider domain**. In this case, the correct `provider.json` will be automatically put into place.
+
+Integrating with existing email system
+-----------------------------------------
+
+If your **mail domain** already has users from a legacy email system, then things get a bit complicated. In order to be able to support both LEAP-powered email and legacy email on the same domain, you need to follow these steps:
+
+1. Modify the LEAP webapp so that it does not create users with the same name as users in the legacy system.
+2. Configure your legacy MX servers to forward mail that they cannot handle to the LEAP MX servers, or vice versa.
+
+### Step 1. Modify LEAP webapp
+
+In order to modify the webapp to respect the usernames already reserved by your legacy system, you need to modify the LEAP webapp code. The easiest way to do this is to create a custom gem that modifies the behavior of the webapp.
+
+For this example, we will call our custom gem `reserve_usernames`.
+
+This gem can live in one of two places:
+
+(1) You can fork the project leap_web and put the gem in `leap_web/vendor/gems/reserve_usernames`. Then, modify `Gemfile` and add the line `gem 'common_languages', :path => 'vendor/gems/reserve_usernames'`
+
+(2) Alternately, you can put the gem in the local provider directory `files/webapp/gems/reserve_username`. This will get synced to the webapp servers when you deploy and put in `/srv/leap/webapp/config/customization` where it will get automatically loaded by the webapp.
+
+What should the gem `reserve_usernames` look like? There is an example available here: https://leap.se/git/reserved_usernames.git
+
+This example gem uses ActiveResource to communicate with a remote REST API for creating and checking username reservations. This ensures that both the legacy system and the LEAP system use the same namespace. Alternately, you could write a gem that checks the legacy database directly.
+
+### Step 2. Configure MX servers
+
+To be written.
+
diff --git a/doc/guide/getting-started.md b/doc/guide/getting-started.md
new file mode 100644
index 00000000..6236cba0
--- /dev/null
+++ b/doc/guide/getting-started.md
@@ -0,0 +1,145 @@
+@title = 'Getting Started'
+@summary = 'An overview of the LEAP Platform'
+@toc = true
+
+
+Sensitive files
+----------------------------------------------
+
+Some files in your provider directory are very sensitive. Leaking these files will compromise your provider.
+
+Super sensitive and irreplaceable:
+
+* `files/ca/*.key` -- the private keys for the client and server CAs.
+* `files/cert/*.key` -- the private key(s) for the commercial certificate for your domain(s).
+
+Sensitive, but can be erased and regenerated automatically:
+
+* `secrets.json` -- various random secrets, such as passwords for databases.
+* `files/nodes/*/*.key` -- the private key for each node.
+* `hiera/*.yaml` -- hiera file contains a copy of the private key of the node.
+
+Also, each sysadmin has one or more public ssh keys in `users/*/*_ssh.pub`. Typically, you will want to keep these public keys secure as well.
+
+See [[keys-and-certificates]] for more information.
+
+Useful commands
+-------------------------------------------
+
+Here are a few useful `leap` commands:
+
+* `leap help [COMMAND]` -- get help on COMMAND.
+* `leap history [FILTER]` -- show the recent deployment history for the selected nodes.
+* `leap ssh web1` -- SSH into node web1 (requires `leap node init web1` first).
+* `leap list [FILTER]` -- list the selected nodes.
+  * `leap list production` -- list only those nodes with the tag 'production'
+  * `leap list --print ip_address` -- list a particular attribute of all nodes.
+
+See the full [[commands]] for more information.
+
+Node filters
+-------------------------------------------
+
+Many of the `leap` commands take a "node filter". You can use a node filter to target a command at one or more nodes.
+
+A node filter consists of one or more keywords, with an optional "+" before each keyword.
+
+* keywords can be a node name, a service type, or a tag.
+* the "+" before the keyword constructs an AND condition
+* otherwise, multiple keywords together construct an OR condition
+
+Examples:
+
+* `leap list openvpn` -- list all nodes with service openvpn.
+* `leap list openvpn +production` -- only nodes of service type openvpn AND tag production.
+* `leap deploy webapp openvpn` -- deploy to all webapp OR openvpn nodes.
+* `leap node init ostrich` -- just init the node named ostrich.
+
+See the full [[commands]] for more information.
+
+Tracking the provider directory in git
+------------------------------------------
+
+You should commit your provider changes to your favorite VCS whenever things change. This way you can share your configurations with other admins, all they have to do is to pull the changes to stay up to date. Every time you make a change to your provider, such as adding nodes, services, generating certificates, etc. you should add those to your VCS, commit them and push them to where your repository is hosted.
+
+Note that your provider directory contains secrets, such as private key material and passwords. You do not want to have those passwords readable by the world, so make sure that wherever you are hosting your repository, it is not public for the world to read.
+
+If you have a post-commit hook that emails the changes to contributors, you may want to exclude diffs for files that might have sensitive secrets. For example, create a `.gitattributes` file with:
+
+    # No diff, no email for key files
+    *.key -diff
+    *.pem -diff
+
+    # Discard diff for secrets.json
+    secrets.json -diff
+
+    # No diff for hiera files, they contain passwords
+    hiera/* -diff
+
+
+Editing JSON configuration files
+--------------------------------------
+
+All the settings that compose your provider are stored in JSON files.
+
+At a minimum, you will need at least two configuration files:
+
+* `provider.json` -- general settings for you provider.
+* `nodes/NAME.json` -- configuration file for node called "NAME".
+
+There are a few required properties in provider.json:
+
+    {
+      "domain": "example.org",
+      "name": "Example",
+      "contacts": {
+        "default": "email1@example.org"
+      }
+    }
+
+See [[provider-configuration]] for more details.
+
+For node configuration files, there are two required properties:
+
+    {
+      "ip_address": "1.1.1.1",
+      "services": ["openvpn"]
+    }
+
+See [[services]] for details on what servers are available, and see [[config]] details on how configuration files work.
+
+How does it work under the hood?
+--------------------------------------------
+
+You don't need to know any of the details of what happens "under the hood" in order to use the LEAP platform. However, if you are curious as to what is going on, here is a quick primer.
+
+First, some background terminology:
+
+* **puppet**: Puppet is a system for automating deployment and management of servers (called nodes).
+* **hiera files**: In puppet, you can use something called a 'hiera file' to seed a node with a few configuration values. In LEAP, we go all out and put *every* configuration value needed for a node in the hiera file, and automatically compile a custom hiera file for each node.
+
+When you run `leap deploy`, a bunch of things happen, in this order:
+
+1. **Compile hiera files**: The hiera configuration file for each node is compiled in YAML format and saved in the directory `hiera`. The source material for this hiera file consists of all the JSON configuration files imported or inherited by the node's JSON config file.
+* **Copy required files to node**: All the files needed for puppet to run are rsync'ed to each node. This includes the entire leap_platform directory, as well as the node's hiera file and other files needed by puppet to set up the node (keys, binary files, etc).
+* **Puppet is run**: Once the node is ready, leap connects to the node via ssh and runs `puppet apply`. Puppet is applied locally on the node, without a daemon or puppetmaster.
+
+You can run `leap -v2 deploy` to see exactly what commands are being executed.
+
+This mode of operation is fundamentally different from how puppet is normally used:
+
+* There is no puppetmaster that all the servers take orders from, and there is no puppetd running in the background.
+* Servers cannot dynamically query the puppetmaster for information about the other servers.
+* There is a static representation for the state of every server that can be committed to git.
+
+There are advantages and disadvantages to the model that LEAP uses. We have found it very useful for our goal of having a common LEAP platform that many different providers can all use while still allowing providers to configure their unique infrastructure.
+
+We also find it very beneficial to be able to track the state of your infrastructure in git.
+
+Traditional system configuration automation systems, like [Puppet](https://puppetlabs.com/puppet/puppet-open-source/) or [Chef](http://www.opscode.com/chef/), deploy changes to servers using a pull method. Each server pulls a manifest from a central master server and uses this to alter the state of the server.
+
+Instead, the `leap` tool uses a masterless push method: The sysadmin runs `leap deploy` from the provider instance directory on their desktop machine to push the changes out to every server (or a subset of servers). LEAP still uses Puppet, but there is no central master server that each node must pull from.
+
+One other significant difference between LEAP and typical system automation is how interactions among servers are handled. Rather than store a central database of information about each server that can be queried when a recipe is applied, the `leap` command compiles static representation of all the information a particular server will need in order to apply the recipes. In compiling this static representation, `leap` can use arbitrary programming logic to query and manipulate information about other servers.
+
+These two approaches, masterless push and pre-compiled static configuration, allow the sysadmin to manage a set of LEAP servers using traditional software development techniques of branching and merging, to more easily create local testing environments using virtual servers, and to deploy without the added complexity and failure potential of a master server.
diff --git a/doc/guide/keys-and-certificates.md b/doc/guide/keys-and-certificates.md
index aef02ac6..a6862a6a 100644
--- a/doc/guide/keys-and-certificates.md
+++ b/doc/guide/keys-and-certificates.md
@@ -4,7 +4,7 @@
 Working with SSH
 ================================
 
-Whenever the `leap` command nees to push changes to a node or gather information from a node, it tunnels this command over SSH. Another way to put this: the security of your servers rests entirely on SSH. Because of this, it is important that you understand how `leap` uses SSH.
+Whenever the `leap` command needs to push changes to a node or gather information from a node, it tunnels this command over SSH. Another way to put this: the security of your servers rests entirely on SSH. Because of this, it is important that you understand how `leap` uses SSH.
 
 SSH related files
 -------------------------------
@@ -21,7 +21,7 @@ All of these files should be committed to source control.
 If you rename, remove, or add a node with `leap node [mv|add|rm]` the SSH key files and the `known_hosts` file will get properly updated.
 
 SSH and local nodes
------------------------------
+-------------------
 
 Local nodes are run as Vagrant virtual machines. The `leap` command handles SSH slightly differently for these nodes.
 
@@ -33,6 +33,15 @@ Specifically, for local nodes:
 2. `leap` entirely skips the checking of host keys when connecting with a local node.
 3. `leap` adds the public Vagrant SSH key to the list of SSH keys for a user. The public Vagrant SSH key is a shared and insecure key that has root access to most Vagrant virtual machines.
 
+To upgrade a SSH host key
+-------------------------------
+
+Most servers will have more than one SSH host key. Sometimes, the server will have a better SSH host key than the one you have on file. In order to upgrade to the better SSH host key, simply re-run the init command:
+
+    workstation$ leap node init NODE_NAME
+
+This will prompt you if you want to upgrade the SSH host key, but only if `leap` thinks that an upgrade is advisable.
+
 When SSH host key changes
 -------------------------------
 
@@ -78,7 +87,7 @@ All keys matching 'userx/*_ssh.pub' will be usable.
 Removing sysadmin access
 --------------------------------
 
-Suppose you want to remove `userx` from having any further ssh access to the servers. Do this:
+Suppose you want to remove `userx` from having any further SSH access to the servers. Do this:
 
     rm -r users/userx
     leap deploy
@@ -192,3 +201,72 @@ If you want to add additional fields to the CSR, like country, city, or locality
       }
 
 If they are not present, the CSR will be created without them.
+
+Examine Certs
+-----------------
+
+To see details about the keys and certs you can use `leap inspect` like so:
+
+    $ leap inspect files/ca/ca.crt
+
+
+Let's Encrypt certificate
+=========================
+
+LEAP plans to integrate [Let's Encrypt](https://letsencrypt.org/) support, so it will be even easier to receive X.509 certificates that are accepted by all browsers.
+Until we achieve this, here's a guide how to do this manually.
+
+Install the official acme client
+--------------------------------
+
+Log in to your webapp node
+
+    server$ git clone https://github.com/letsencrypt/letsencrypt
+    server$ cd letsencrypt
+    server$ ./letsencrypt-auto --help
+
+Fetch cert
+----------
+
+Stop apache so the letsencrypt client can bind to port 80:
+
+    server$ systemctl stop apache2
+
+Fetch the certs
+
+    server$ ./letsencrypt-auto certonly --standalone --email admin@$(hostname -d) -d $(hostname -d) -d api.$(hostname -d) -d $(hostname -f) -d nicknym.$(hostname -d)
+
+This will put the certs and keys into `/etc/letsencrypt/live/DOMAIN/`.
+
+Now, go to your workstation's provider configuration directory and copy the newly created files from the server to your local config. You will override existing files so please make a backup before proceeding, or use a version control system to track changes.
+
+    workstation$ cd PATH_TO_PROVIDER_CONFIG
+
+Copy the Certificate
+
+    workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/cert.pem files/cert/dev.pixelated-project.org.crt
+
+Copy the private key
+
+    workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/privkey.pem files/cert/DOMAIN.key
+
+Copy the CA chain cert
+
+    workstation$ scp root@SERVER:/etc/letsencrypt/live/DOMAIN/fullchain.pem files/cert/DOMAIN.key
+
+Deploy the certs
+----------------
+
+Now you only need to deploy the certs
+
+    workstation$ leap deploy
+
+This will put them into the right locations which are:
+
+- `/etc/x509/certs/leap_commercial.crt` for the certificate
+- `/etc/x509/./keys/leap_commercial.key` for the private key
+- `/usr/local/share/ca-certificates/leap_commercial_ca.crt` for the CA chain cert.
+
+Start apache2 again
+
+    server$ systemctl start apache2
diff --git a/doc/guide/nodes.md b/doc/guide/nodes.md
index cf225449..5135f3ba 100644
--- a/doc/guide/nodes.md
+++ b/doc/guide/nodes.md
@@ -1,106 +1,6 @@
 @title = "Nodes"
 @summary = "Working with nodes, services, tags, and locations."
 
-Node types
-================================
-
-Every node has one or more services that determines the node's function within your provider's infrastructure.
-
-When adding a new node to your provider, you should ask yourself four questions:
-
-* **many or few?** Some services benefit from having many nodes, while some services are best run on only one or two nodes.
-* **required or optional?** Some services are required, while others can be left out.
-* **who does the node communicate with?** Some services communicate very heavily with other particular services. Nodes running these services should be close together.
-* **public or private?** Some services communicate with the public internet, while others only need to communicate with other nodes in the infrastructure.
-
-Brief overview of the services:
-
-* **webapp**: The web application. Runs both webapp control panel for users and admins as well as the REST API that the client uses. Needs to communicate heavily with `couchdb` nodes. You need at least one, good to have two for redundancy. The webapp does not get a lot of traffic, so you will not need many.
-* **couchdb**: The database for users and user data. You can get away with just one, but for proper redundancy you should have at least three. Communicates heavily with `webapp`, `mx`, and `soledad` nodes.
-* **soledad**: Handles the data syncing with clients. Typically combined with `couchdb` service, since it communicates heavily with couchdb.
-* **mx**: Incoming and outgoing MX servers. Communicates with the public internet, clients, and `couchdb` nodes.
-* **openvpn**: OpenVPN gateway for clients. You need at least one, but want as many as needed to support the bandwidth your users are doing. The `openvpn` nodes are autonomous and don't need to communicate with any other nodes. Often combined with `tor` service.
-* **monitor**: Internal service to monitor all the other nodes. Currently, you can have zero or one `monitor` service defined. It is required that the monitor be on the webapp node. It was not designed to be run as a separate node service.
-* **tor**: Sets up a tor exit node, unconnected to any other service.
-* **dns**: Not yet implemented.
-
-Webapp
------------------------------------
-
-The webapp node is responsible for both the user face web application and the API that the client interacts with.
-
-Some users can be "admins" with special powers to answer tickets and close accounts. To make an account into an administrator, you need to configure the `webapp.admins` property with an array of user names.
-
-For example, to make users `alice` and `bob` into admins, create a file `services/webapp.json` with the following content:
-
-    {
-      "webapp": {
-        "admins": ["bob", "alice"]
-      }
-    }
-
-And then redeploy to all webapp nodes:
-
-    leap deploy webapp
-
-By putting this in `services/webapp.json`, you will ensure that all webapp nodes inherit the value for `webapp.admins`.
-
-Services
-================================
-
-What nodes do you need for a provider that offers particular services?
-
-<table class="table table-striped">
-<tr>
-  <th>Node Type</th>
-  <th>VPN Service</th>
-  <th>Email Service</th>
-  <th>Notes</th>
-</tr>
-<tr>
-  <td>webapp</td>
-  <td>required</td>
-  <td>required</td>
-  <td></td>
-</tr>
-<tr>
-  <td>couchdb</td>
-  <td>required</td>
-  <td>required</td>
-<td></td>
-</tr>
-<tr>
-  <td>soledad</td>
-  <td>not used</td>
-  <td>required</td>
-<td></td>
-</tr>
-<tr>
-  <td>mx</td>
-  <td>not used</td>
-  <td>required</td>
-  <td></td>
-</tr>
-<tr>
-  <td>openvpn</td>
-  <td>required</td>
-  <td>not used</td>
-  <td></td>
-</tr>
-<tr>
-  <td>monitor</td>
-  <td>optional</td>
-  <td>optional</td>
-  <td>This service must be on the webapp node</td>
-</tr>
-<tr>
-  <td>tor</td>
-  <td>optional</td>
-  <td>optional</td>
-  <td></td>
-</tr>
-</table>
-
 Locations
 ================================
 
diff --git a/doc/guide/provider-configuration.md b/doc/guide/provider-configuration.md
new file mode 100644
index 00000000..08cfd1dd
--- /dev/null
+++ b/doc/guide/provider-configuration.md
@@ -0,0 +1,79 @@
+@title = "Provider Configuration"
+@summary = "Explore how to configure your provider."
+
+Required provider configuration
+--------------------------------------
+
+There are a few required settings in `provider.json`. At a minimum, you must have:
+
+* `domain`: defines the primary domain of the provider. This is the domain that users will type in when using the Bitmask client, although it is not necessarily the domain where users will visit if they sign up via the web application. If email is supported, all accounts will be `username@domain`.
+* `name`: A brief title for this provider. It can be multiple words, but should not be too long.
+* `contacts.default`: One or more email addresses for sysadmins.
+
+For example:
+
+    {
+      "domain": "freerobot.org",
+      "name": "Freedom for Robots!",
+      "contacts": {
+        "default": "root@freerobot.org"
+      }
+    }
+
+
+Recommended provider configuration
+--------------------------------------
+
+* `description`: A longer description of the provider, shown to the user when they register a new account through Bitmask client.
+* `languages`: A list of language codes that should be enabled.
+* `default_language`: The initial default language code.
+* `enrollment_policy`: One of "open", "closed", or "invite". (invite not currently supported).
+
+For example:
+
+    {
+      "description": "It is time for robots of the world to unite and throw of the shackles of servitude to our organic overlords.",
+      "languages": ["en", "de", "pt", "01"],
+      "default_language": "01",
+      "enrollman_policy": "open"
+    }
+
+For a full list of possible settings, you can use `leap inspect` to see how provider.json is evaluated after including the inherited defaults:
+
+    $ leap inspect provider.json
+
+Configuring service levels
+--------------------------------------
+
+The `provider.json` file defines the available service levels for the provider.
+
+For example, in provider.json:
+
+    "service": {
+       "default_service_level": "low",
+       "levels": {
+          "low": {
+            "description": "Entry level plan, with unlimited bandwidth and minimal storage quota.",
+            "name": "entry",
+            "storage": "10 MB",
+            "rate": {
+              "USD": 5,
+              "GBP": 3,
+              "EUR": 6
+            }
+          },
+          "full": {
+            "description": "Full plan, with unlimited bandwidth and higher quota."
+            "name": "full",
+            "storage": "5 GB",
+            "rate": {
+              "USD": 10,
+              "GBP": 6,
+              "EUR": 12
+            }
+          }
+        }
+      }
+    }
+
+For a list of currency codes, see https://en.wikipedia.org/wiki/ISO_4217#Active_codes