From a9ee3789a7c88b4f9cce285f03e1e5d0707a1cad Mon Sep 17 00:00:00 2001 From: Micah Anderson Date: Thu, 19 Sep 2013 15:42:10 -0400 Subject: Due to the fact that so many people would start with the quickstart guide (because we recommended it), and would get completely hung-up on vagrant not working and if they managed to succeed in fighting through those problems, ended up with a provider that wasn't able to actually do anything at all. So, the goal was to move the quickstart information into a development setup howto, and then revamp the quickstart to be a minimal provider setup from scratch configuration that results in you getting familiar with the concepts (provider, platform, cli, nodes, services, etc.) and leaves you at the end with a functional minimal provider. --- docs/platform/development.md | 183 +++++++++++++++++++++++++++++ docs/platform/examples.md | 111 ------------------ docs/platform/quick-start.md | 269 +++++++++++++++++++++++++++++++------------ 3 files changed, 379 insertions(+), 184 deletions(-) create mode 100644 docs/platform/development.md delete mode 100644 docs/platform/examples.md (limited to 'docs') diff --git a/docs/platform/development.md b/docs/platform/development.md new file mode 100644 index 0000000..6f259cc --- /dev/null +++ b/docs/platform/development.md @@ -0,0 +1,183 @@ +@title = "Development Environment" + +If you are wanting to make local changes to your provider, or want to contribute some fixes back to LEAP, we recommend that you follow this guide to build up a development environment to test your changes first. Using this method, you can quickly test your changes without deploying them to your production environment, while benefitting from the convenience of reverting to known good states in order to retry things from scratch. + +This page will walk you through setting up nodes using Vagrant for convenient deployment testing, snapshotting known good states, and reverting to previous snapshots. + +Requirements +============ + +* Be a real machine with virtualization support in the CPU (VT-x or AMD-V). In other words, not a virtual machine. +* Have at least 4gb of RAM. +* Have a fast internet connection (because you will be downloading a lot of big files, like virtual machine images). + +Install prerequisites +-------------------------------- + +For development purposes, you will need everything that you need for deploying the LEAP platform: + +* LEAP cli +* A provider instance + +You will also need to setup a virtualized Vagrant environment, to do so please make sure you have the following +pre-requisites installed: + +*Debian & Ubuntu* + +Install core prerequisites: + + sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make + +Install Vagrant in order to be able to test with local virtual machines (typically optional, but required for this tutorial): + + sudo apt-get install vagrant virtualbox + + + + +Adding development nodes to your provider +========================================= + +Now you will add local-only Vagrant development nodes to your provider. + +You do not need to setup a different provider instance for development, in fact it is more convenient if you do not, but you can if you wish. If you do not have a provider already, you will need to create one and configure it before continuing (see: x). + + +Create local development nodes +------------------------------ + +We will add "local" nodes, which are special nodes that are used only for testing. These nodes exist only as virtual machines on your computer, and cannot be accessed from the outside. Each "node" is a server that can have one or more services attached to it. We recommend that you create different nodes for different services to better isolate issues. + +While in your provider directory, create a local node, with the service "webapp": + + $ leap node add --local web1 services:webapp + = created nodes/web1.json + = created files/nodes/web1/ + = created files/nodes/web1/web1.key + = created files/nodes/web1/web1.crt + +This command creates a node configuration file in `nodes/web1.json` with the webapp service. + +Starting local development nodes +-------------------------------- + +In order to test the node "web1" we need to start it. Starting a node for the first time will spin up a virtual machine. The first time you do this will take some time because it will need to download a VM image (about 700mb). After you've downloaded the base image, you will not need to download it again, and instead you will re-use the downloaded image (until you need to update the image). + +NOTE: Many people have difficulties getting Vagrant working. If the following commands do not work, please visit the [Vagrant page](vagrant) to troubleshoot your Vagrant install before proceeding. + + $ leap local start web + = created test/ + = created test/Vagrantfile + = installing vagrant plugin 'sahara' + Bringing machine 'web1' up with 'virtualbox' provider... + [web1] Box 'leap-wheezy' was not found. Fetching box from specified URL for + the provider 'virtualbox'. Note that if the URL does not have + a box for this provider, you should interrupt Vagrant now and add + the box yourself. Otherwise Vagrant will attempt to download the + full box prior to discovering this error. + Downloading or copying the box... + Progress: 3% (Rate: 560k/s, Estimated time remaining: 0:13:36) + ... + Bringing machine 'web1' up with 'virtualbox' provider... + [web1] Importing base box 'leap-wheezy'... + 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% + +Now the virtual machine 'web1' is running. You can add another local node using the same process. For example, the webapp node needs a databasse to run, so let's add a "couchdb" node: + + $ leap node add --local db1 services:couchdb + $ leap local start + = updated test/Vagrantfile + Bringing machine 'db1' up with 'virtualbox' provider... + [db1] Importing base box 'leap-wheezy'... + [db1] Matching MAC address for NAT networking... + [db1] Setting the name of the VM... + [db1] Clearing any previously set forwarded ports... + [db1] Fixed port collision for 22 => 2222. Now on port 2202. + [db1] Creating shared folders metadata... + [db1] Clearing any previously set network interfaces... + [db1] Preparing network interfaces based on configuration... + [db1] Forwarding ports... + [db1] -- 22 => 2202 (adapter 1) + [db1] Running any VM customizations... + [db1] Booting VM... + [db1] Waiting for VM to boot. This can take a few minutes. + [db1] VM booted and ready for use! + [db1] Configuring and enabling network interfaces... + [db1] Mounting shared folders... + [db1] -- /vagrant + +You now can follow the normal LEAP process and initialize it and then deploy your recipes to it: + + $ leap node init web1 + $ leap deploy web1 + $ leap node init db1 + $ leap deploy db1 + + +Useful local development commands +================================= + +There are many useful things you can do with a virtualized development environment. + +Listing what machines are running +--------------------------------- + +Now you have the two virtual machines "web1" and "db1" running, you can see the running machines as follows: + + $ leap local status + Current machine states: + + db1 running (virtualbox) + web1 running (virtualbox) + + This environment represents multiple VMs. The VMs are all listed + above with their current state. For more information about a specific + VM, run `vagrant status NAME`. + +Stopping machines +----------------- + +It is not recommended that you leave your virtual machines running when you are not using them. They consume memory and other resources! To stop your machines, simply do the following: + + $ leap local stop web1 db1 + +Connecting to machines +---------------------- + +You can connect to your local nodes just like you do with normal LEAP nodes, by running 'leap ssh node'. + +However, if you cannot connect to your local node, because the networking is not setup properly, or you have deployed a firewall that locks you out, you may need to access the graphical console. + +In order to do that, you will need to configure Vagrant to launch a graphical console and then you can login as root there to diagnose the networking problem. To do this, add the following to you +$HOME/.leaprc: + + @custom_vagrant_vm_line = 'config.vm.boot_mode = :gui' + +and then start, or restart, your local Vagrant node. You should get a VirtualBox graphical interface presented to you showing you the bootup and eventually the login. + +Snapshotting machines +--------------------- + +A very useful feature of local Vagrant development nodes is the ability to snapshot the current state and then revert to that when you need. + +For example, perhaps the base image is a little bit out of date and you want to get the packages updated to the latest before continuing. You can do that simply by starting the node, connecting to it and updating the packages and then snapshotting the node: + + $ leap local start web1 + $ leap ssh web1 + web1# apt-get -u dist-upgrade + web1# exit + $ leap local save web1 + +Now you can deploy to web1 and if you decide you want to revert to the state before deployment, you simply have to reset the node to your previous save: + + $ leap local reset web1 + +More information +---------------- + +See `leap help local` for a complete list of local-only commands and how they can be used. diff --git a/docs/platform/examples.md b/docs/platform/examples.md deleted file mode 100644 index b2afcb0..0000000 --- a/docs/platform/examples.md +++ /dev/null @@ -1,111 +0,0 @@ -@title = 'Examples' -@summary = 'Example provider setups using the LEAP platform.' - -A minimal three node OpenVPN provider -======================================= - -Our goal ------------------- - -We are going to create a minimal LEAP provider offering OpenVPN service. This basic setup can be expanded by adding more OpenVPN nodes to increase capacity, or more webapp and couchdb nodes to increase availability (performance wise, a single couchdb and a single webapp are more than enough for most usage, since they are only lightly used, but you might want redundancy). - -Our goal is something like this: - - leap list - NODES SERVICES TAGS - clam couchdb - elephant webapp - snail openvpn - -NOTE: You won't be able to run those `leap list` commands yet, not until we actually create the node configurations. - -Create configuration files --------------------------------- - -Create the provider directory: - - leap new bitmask --domain bitmask.net --name bitmask --contacts root@bitmask.net - -Add you ssh key: - - leap add-user --self - -Add some nodes: - - leap node add clam ip_address:176.53.69.22 services:couchdb - leap node add elephant ip_address:176.53.69.13 services:webapp - leap node add snail ip_address:176.53.69.14 \ - openvpn.gateway_address:176.53.69.15 services:openvpn - -NOTE: openvpn gateways must be assigned two IP addresses, one for the host itself and one for the openvpn gateway. We do this to prevent incoming and outgoing VPN traffic on the same IP. Without this, the client might send some traffic to other VPN users in the clear, bypassing the VPN. - -Now that you have the nodes configured, you should create the DNS entries for these nodes. - -Set up your DNS with these hostnames: - - leap list --print ip_address,domain.full,dns.aliases - clam 176.53.69.22, clam.bitmask.net, null - elephant 176.53.69.13, elephant.bitmask.net, api.bitmask.net - snail 176.53.69.14, snail.bitmask.net, null - -Alternately, you can adapt this zone file snippet: - - leap compile zone - -Create certificates ------------------------------------- - -Create two certificate authorities, one for server certs and one for client certs: - - leap cert ca - -Create a temporary cert for your main domain (you should replace with a real commercial cert at some point) - - leap cert csr - -Create the Diffie-Hellman parameters file, needed for forward secret OpenVPN ciphers: - - leap cert dh - -Create server certificates for all the nodes you have added: - - leap cert update - -NOTE: the file `files/ca/ca.key` is extremely sensitive and must be carefully protected. The other key files are much less sensitive and can simply be regenerated if needed. - -Deploy to nodes ------------------------- - -Initialize all nodes (only needs to be done once for each node): - - leap node init - -Deploy to all nodes: - - leap deploy - -Those two commands create pretty busy output, so it may be more clear to initial and deploy each node one by one: - - leap node init clam - leap deploy clam - leap node init elephant - leap deploy elephant - leap node init snail - leap deploy snail - -Testing --------------------------- - -Automated testing is in the works, but for now you manually test to see if the OpenVPN gateways and the webapp are working like so. - -OpenVPN: - - leap test init - sudo openvpn test/openvpn/unlimited.ovpn - -Webapp: - -* run `leap list --print ip_address webapp` to remind yourself the ip address(es) of the webapp. -* edit your local `/etc/hosts` to add entries like `176.53.69.13 example.org`, for whatever domain is appropriate in your case. -* open your browser to `https://example.org` - diff --git a/docs/platform/quick-start.md b/docs/platform/quick-start.md index da3fda0..6d36c14 100644 --- a/docs/platform/quick-start.md +++ b/docs/platform/quick-start.md @@ -1,21 +1,51 @@ @title = 'LEAP Platform Quick Start' @nav_title = 'Quick Start' -This tutorial walks you through the initial process of creating and deploying a service provider running the [LEAP platform](platform). First examples aim to build a provider in a virtual environment, and in the end running in real hardware is targeted. +This tutorial walks you through the initial process of creating and deploying a minimal service provider running the [LEAP platform](platform). This Quick Start guide will guide you through building a three node OpenVPN provider. -First, a few definitions: +Our goal +------------------ -* **node:** A server that is part of the service provider's infrastructure. All nodes are running the Debian GNU/Linux operating system. -* **sysadmin:** This is you. -* **sysadmin machine:** Your desktop or laptop computer that you use to control the nodes. This machine can be running any variant of Unix, Linux, or Mac OS (however, only Debian derivatives are supported at the moment). +We are going to create a minimal LEAP provider offering OpenVPN service. This basic setup can be expanded by adding more OpenVPN nodes to increase capacity, or more webapp and couchdb nodes to increase availability (performance wise, a single couchdb and a single webapp are more than enough for most usage, since they are only lightly used, but you might want redundancy). -All the commands in this tutorial are run on your sysadmin machine. In order to complete the tutorial, the sysadmin machine must: +Our goal is something like this: -* Be a real machine with virtualization support in the CPU (VT-x or AMD-V). In other words, not a virtual machine. -* Have at least 4gb of RAM. -* Have a fast internet connection (because you will be downloading a lot of big files, like virtual machine images). + leap list + NODES SERVICES TAGS + clam couchdb + elephant webapp + snail openvpn -Install prerequisites +NOTE: You won't be able to run those `leap list` commands yet, not until we actually create the node configurations. + +Requirements +------------ + +In order to complete this Quick Start, you will need a few things: + +* You will need three real or virtual machines that have a basic Debian Stable installed. +* You should be able to SSH into them remotely, and know their IP addresses and their SSH host keys +* You will need four different IPs, one for each node, and a second one for the VPN gateway +* You should have the ability to create/modify DNS entries for your domain +* You need to be aware that this process will make changes to your systems, so please be sure that these machines are a basic install with nothing configured or running for other purposes +* Your machines will need to be connected to the internet, and not behind a restrictive firewall + +All the commands in this tutorial are run on your sysadmin machine. In order to complete the tutorial, the sysadmin will do the following: + +* Install pre-requisites +* Install the LEAP command-line utility +* Check out the LEAP platform +* Create a provider and its certificates +* Setup the provider's nodes and the services that will reside on those nodes +* Initialize the nodes +* Deploy the LEAP platform to the nodes +* Test that things worked correctly +* Some additional commands + +We will walk you through each of these steps. + + +Install pre-requisites -------------------------------- *Debian & Ubuntu* @@ -24,18 +54,16 @@ Install core prerequisites: sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make -Install Vagrant in order to be able to test with local virtual machines (typically optional, but required for this tutorial): - - sudo apt-get install vagrant virtualbox - -Install leap +NOTE: leap_cli should work with ruby1.8, but has only been tested using ruby1.9. + + +Install the LEAP command-line utility --------------------- +Test that things worked correctly +--------------------------------- + +You should now have three machines with the LEAP platform deployed to them, one for the web application, one for the database and one for the OpenVPN gateway. + +Access the web application +-------------------------------------------- + +In order to connect to the web application in your browser, you need to point your domain at the IP address of the web application node (named elephant in this example). + +There are a lot of different ways to do this, but one easy way is to modify your `/etc/hosts` file. First, find the IP address of the webapp node: + + leap list webapp --print ip_address + +Then modify `/etc/hosts` like so: + + x.x.x.w leap.example.net + +Replacing 'leap.example.net' with whatever you specified as the `domain` in the `leap new` command. + +Next, you can connect to the web application either using a web browser or via the API using the LEAP client. To use a browser, connect to https://leap.example.net (replacing that with your domain). Your browser will complain about an untrusted cert, but for now just bypass this. From there, you should be able to register a new user and login. + +Use the VPN +----------- + +You should be able to simply test that the OpenVPN gateway works properly by doing the following: + + $ leap test init + $ sudo openvpn test/openvpn/unlimited.ovpn + +Or, you can use the LEAP client (called "bitmask") to connect to your new provider, create a user and then connect to the VPN. + + Additional commands ------------------------------------------- @@ -197,10 +337,8 @@ Here are a few useful commands you can run on your new local nodes: * `leap ssh web1` -- SSH into node web1 (requires `leap node init web1` first). * `leap list` -- list all nodes. +* `leap list production` -- list only those nodes with the tag 'production' * `leap list --print ip_address` -- list a particular attribute of all nodes. -* `leap local reset web1` -- return web1 to a pristine state. -* `leap local stop` -- stop all local virtual machines. -* `leap local status` -- get the running state of all the local virtual machines. * `leap cert update` -- generate new certificates if needed. See the full command reference for more information. @@ -223,21 +361,6 @@ Examples: * `leap deploy webapp openvpn` -- deploy to all webapp OR openvpn nodes. * `leap node init vpn1` -- just init the node named vpn1. -Running on real hardware ------------------------------------ - -The steps required to initialize and deploy to nodes on the public internet are basically the same as we have seen so far for local testing nodes. There are a few key differences: - -* Obviously, you will need to acquire a real or virtual machine that you can SSH into remotely. -* When creating the node configuration, you should give it the tag "production" if the node is to be used in your production infrastructure. -* When creating the node configuration, you need to specify the IP address of the node. - -For example: - - leap node add db1 tags:production services:couchdb ip_address:4.4.4.4 - -Also, running `leap node init NODE_NAME` on a real server will prompt you to verify the fingerprint of the SSH host key and to provide the root password of the server NODE_NAME. You should only need to do this once. - What's next ----------------------------------- -- cgit v1.2.3