diff options
110 files changed, 1692 insertions, 439 deletions
@@ -1,2 +1,2 @@ -.reviewboardrc +/.vagrant /puppet/modules/site_custom diff --git a/CHANGES.md b/CHANGES.md new file mode 100644 index 00000000..6540bd0c --- /dev/null +++ b/CHANGES.md @@ -0,0 +1,67 @@ +Platform 0.7.0 +----------------------- + +Compatibility: + +* Requires leap_cli version 1.7.2 +* Requires bitmask client version >= 0.7 +* Previous releases supported cookies when using the provider API. Now, only + tokens are supported. +* Includes: + * leap_mx 0.7.0 + * tapicero 0.7 + * webapp 0.7 + * soledad 0.7 + +Commits: https://leap.se/git/leap_platform.git/shortlog/refs/tags/0.7.0 +Issues fixed: https://leap.se/code/versions/168 + +Upgrading: + +* `gem install leap_cli --version 1.7.2`. +* `cd leap_platform; git pull; git checkout 0.7.0`. +* `leap deploy` +* `leap db destroy --db sessions,tokens` You can ignore message about needing to redeploy (since, in this case, we just want to permanently delete those databases). +* `leap test` to make sure everything is working + +New features: + +* rotating couchdb databases: CouchDB is not designed to handle ephemeral data, like sessions, because documents are never really deleted (a tombstone document is always kept to record the deletion). To overcome this limitation, we now rotate the `sessions` and `tokens` databases monthly. The new database names are `tokens_XXX` and `sessions_XXX` where XXX is a counter since the epoch that increments every month (not a calendar month, but a month's worth of seconds). Additionally, nagios checks and `leap test run` now will create and destroy test users in the `tmp_users` database, which will get periodically deleted and recreated. +* deployment logging: information on every deploy is logged to `/var/log/leap` on the node, including the user, leap_cli version, and platform version. +* you must now run `leap deploy --downgrade` if you want to deploy an older version over a newer platform version. +* the install source for each custom daemons (e.g. tapicero, etc) can now configured in `common.json`. +* you can configure apt sources in common.json +* improved nagios graphs integration (with pnp4nagios) +* default MTU was reduced to 1400 for better overall compatibility +* install haveged for some minimal entropy on depleted systems +* switch to release branches for webapp, tapicero +* implement weakdh recommendations, and update minimal cipher lists for web TLS connections +* many bug bugfixes, security improvements, and tests + + +Platform 0.6 +------------------------------------- + +Compatibility: + +* Requires leap_cli version 1.6 +* Requires bitmask client version >= 0.5 + +Commits: https://leap.se/git/leap_platform.git/shortlog/refs/tags/0.6.0 + +New features: + +* single node deployment +* include custom puppet modules and manifests +* couch flexibility +* stunnel rework +* new debian repository structure +* dependency pinning +* leap_cli modularization +* improved cert generation +* monitoring improvements such as per-environment tooling and notifications +* tor hidden service support +* switch away from NIST curve and ensure TLSv1 is used +* tests made significantly more robust +* add support for webapp deployment to a subdomain +* many, many bugfixes and stability improvements @@ -6,7 +6,7 @@ The LEAP Platform is set of complementary packages and server recipes to automat Getting started ============================= -It is highly recommended that you start by reading the overview of the [LEAP Platform](https://leap.se/docs/platform) and then begin with the [Quick Start guide](https://leap.se/docs/platform/quick-start) to walk through a test environment setup to get familiar with how things work before deploying to live servers. +It is highly recommended that you start by reading the overview of the [LEAP Platform](https://leap.se/docs/platform) and then begin with the [Quick Start tutorial](https://leap.se/en/docs/platform/tutorials/quick-start) to walk through a test environment setup to get familiar with how things work before deploying to live servers. An offline copy of this documentation is contained in the `doc` subdirectory. For more current updates to the documentation, visit the website. @@ -42,7 +42,7 @@ You can't deploy new couchdb nodes after one or more have been deployed. Make *s User setup and ssh ------------------ -. if you aren't using a single ssh key, but have different ones, you will need to define the following at the top of your ~/.ssh/config: +. if you aren't using a single ssh key, but have different ones, you will need to define the following at the top of your ~/.ssh/config: HostName <ip address> IdentityFile <path to identity file> @@ -52,7 +52,7 @@ User setup and ssh . At the moment, only ECDSA ssh host keys are supported. If you get the following error: `= FAILED ssh-keyscan: no hostkey alg (must be missing an ecdsa public host key)` then you should confirm that you have the following line defined in your server's **/etc/ssh/sshd_config**: `HostKey /etc/ssh/ssh_host_ecdsa_key`. If that file doesn't exist, run `ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ""` in order to create it. If you made a change to your sshd_config, then you need to run `/etc/init.d/ssh restart` (see: https://leap.se/code/issues/2373) -. To remove an admin's access to your servers, please remove the directory for that user under the `users/` subdirectory in your provider directory and then remove that user's ssh keys from files/ssh/authorized_keys. When finished you *must* run a `leap deploy` to update that information on the servers. +. To remove an admin's access to your servers, please remove the directory for that user under the `users/` subdirectory in your provider directory and then remove that user's ssh keys from files/ssh/authorized_keys. When finished you *must* run a `leap deploy` to update that information on the servers. . At the moment, it is only possible to add an admin who will have access to all LEAP servers (see: https://leap.se/code/issues/2280) @@ -77,13 +77,23 @@ Special Environments . When deploying to OpenStack release "nova" or newer, you will need to do an initial deploy, then when it has finished run `leap facts update` and then deploy again (see: https://leap.se/code/issues/3020) +leap-mx +------- -Changelog -========= +. see https://github.com/leapcode/leap_mx#070 for issues regarding leap_mx + + +Contributing +============ -For a changelog of the current branch: +In order to validate the syntax and style guide compliance +before you commit, see https://github.com/pixelated-project/puppet-git-hooks#installation + + +Changes +========= - git log +Read CHANGES.md or run `git log`. Authors and Credits =================== diff --git a/Vagrantfile b/Vagrantfile new file mode 100644 index 00000000..4a91c459 --- /dev/null +++ b/Vagrantfile @@ -0,0 +1,27 @@ +Vagrant.configure("2") do |config| + config.vm.define :node1 do |config| + + # Please verify the sha512 sum of the downloaded box before importing it into vagrant ! + # see https://leap.se/en/docs/platform/details/development#Verify.vagrantbox.download + # for details + + config.vm.box = "leap-wheezy" + config.vm.box_url = "https://downloads.leap.se/platform/vagrant/virtualbox/leap-wheezy.box" + #config.vm.network :private_network, ip: "10.5.5.102" + config.vm.provider "virtualbox" do |v| + v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"] + v.name = "node1" + end + + config.vm.provision "puppet" do |puppet| + puppet.manifests_path = "./vagrant" + puppet.module_path = "./puppet/modules" + puppet.manifest_file = "install-platform.pp" + puppet.options = "--verbose" + end + config.vm.provision "shell", path: "vagrant/configure-leap.sh" + + config.ssh.username = "vagrant" + + end +end diff --git a/bin/puppet_command b/bin/puppet_command index cdb0b027..1e74522a 100755 --- a/bin/puppet_command +++ b/bin/puppet_command @@ -4,11 +4,14 @@ # This is a wrapper script around the puppet command used by the LEAP platform. # # We do this in order to make it faster and easier to control puppet remotely -# (exit codes, lockfile, multiple manifests, etc) +# (exit codes, logging, lockfile, version check, etc) # require 'pty' require 'yaml' +require 'logger' +require 'socket' +require 'fileutils' PUPPET_BIN = '/usr/bin/puppet' PUPPET_DIRECTORY = '/srv/leap' @@ -18,6 +21,12 @@ SITE_MODULES = 'puppet/modules' CUSTOM_MODULES = ':files/puppet/modules' DEFAULT_TAGS = 'leap_base,leap_service' HIERA_FILE = '/etc/leap/hiera.yaml' +LOG_DIR = '/var/log/leap' +DEPLOY_LOG = '/var/log/leap/deploy.log' +SUMMARY_LOG = '/var/log/leap/deploy-summary.log' +SUMMARY_LOG_1 = '/var/log/leap/deploy-summary.log.1' +APPLY_START_STR = "STARTING APPLY" +APPLY_FINISH_STR = "APPLY COMPLETE" def main process_command_line_arguments @@ -28,15 +37,39 @@ def main end end -def puts(str) +def open_log_files + FileUtils.mkdir_p(LOG_DIR) + $logger = Logger.new(DEPLOY_LOG) + $summary_logger = Logger.new(SUMMARY_LOG) + [$logger, $summary_logger].each do |logger| + logger.level = Logger::INFO + logger.formatter = proc do |severity, datetime, progname, msg| + "%s %s: %s\n" % [datetime.strftime("%b %d %H:%M:%S"), Socket.gethostname, msg] + end + end +end + +def close_log_files + $logger.close + $summary_logger.close +end + +def log(str, *args) + str = str.strip $stdout.puts str $stdout.flush + $logger.info(str) + if args.include? :summary + $summary_logger.info(str) + end end def process_command_line_arguments @commands = [] @verbosity = 1 @tags = DEFAULT_TAGS + @info = {} + @downgrade = false loop do case ARGV[0] when 'apply' then ARGV.shift; @commands << 'apply' @@ -44,6 +77,8 @@ def process_command_line_arguments when '--verbosity' then ARGV.shift; @verbosity = ARGV.shift.to_i when '--force' then ARGV.shift; remove_lockfile when '--tags' then ARGV.shift; @tags = ARGV.shift + when '--info' then ARGV.shift; @info = parse_info(ARGV.shift) + when '--downgrade' then ARGV.shift; @downgrade = true when /^-/ then usage("Unknown option: #{ARGV[0].inspect}") else break end @@ -52,16 +87,18 @@ def process_command_line_arguments end def apply + platform_version_check! unless @downgrade + log "#{APPLY_START_STR} {#{format_info(@info)}}", :summary exit_code = puppet_apply do |line| - puts line + log line end - puts "Puppet apply complete (#{exitcode_description(exit_code)})." + log "#{APPLY_FINISH_STR} (#{exitcode_description(exit_code)}) {#{format_info(@info)}}", :summary end def set_hostname hostname = hiera_file['name'] if hostname.nil? || hostname.empty? - puts('ERROR: "name" missing from hiera file') + log('ERROR: "name" missing from hiera file') exit(1) end current_hostname_file = File.read('/etc/hostname') rescue nil @@ -73,18 +110,18 @@ def set_hostname f.write hostname end if File.read('/etc/hostname') == hostname - puts "Changed /etc/hostname to #{hostname}" + log "Changed /etc/hostname to #{hostname}" else - puts "ERROR: failed to update /etc/hostname" + log "ERROR: failed to update /etc/hostname" end end # call /bin/hostname if current_hostname != hostname if run("/bin/hostname #{hostname}") == 0 - puts "Changed hostname to #{hostname}" + log "Changed hostname to #{hostname}" else - puts "ERROR: call to `/bin/hostname #{hostname}` returned an error." + log "ERROR: call to `/bin/hostname #{hostname}` returned an error." end end end @@ -96,7 +133,7 @@ def puppet_apply(options={}, &block) options = {:verbosity => @verbosity, :tags => @tags}.merge(options) manifest = options[:manifest] || SITE_MANIFEST modulepath = options[:module_path] || SITE_MODULES + CUSTOM_MODULES - fqdn = hiera_file['domain']['name'] + fqdn = hiera_file['domain']['full'] domain = hiera_file['domain']['full_suffix'] Dir.chdir(PUPPET_DIRECTORY) do return run("FACTER_fqdn='#{fqdn}' FACTER_domain='#{domain}' #{PUPPET_BIN} apply #{custom_parameters(options)} --modulepath='#{modulepath}' #{PUPPET_PARAMETERS} #{manifest}", &block) @@ -104,17 +141,60 @@ def puppet_apply(options={}, &block) end # +# parse the --info flag. example str: "key1: value1, key2: value2, ..." +# +def parse_info(str) + str.split(', '). + map {|i| i.split(': ')}. + inject({}) {|h,i| h[i[0]] = i[1]; h} +rescue Exception => exc + {"platform" => "INVALID_FORMAT"} +end + +def format_info(info) + info.to_a.map{|i|i.join(': ')}.join(', ') +end + +# +# exits with a warning message if the last successful deployed +# platform was newer than the one we are currently attempting to +# deploy. +# +PLATFORM_RE = /\{.*platform: ([0-9\.]+)[ ,\}].*[\}$]/ +def platform_version_check! + return unless @info["platform"] + new_version = @info["platform"].split(' ').first + return unless new_version + if File.exists?(SUMMARY_LOG) && File.size(SUMMARY_LOG) != 0 + file = SUMMARY_LOG + elsif File.exists?(SUMMARY_LOG_1) && File.size(SUMMARY_LOG_1) != 0 + file = SUMMARY_LOG_1 + else + return + end + most_recent_line = `tail '#{file}'`.split("\n").grep(PLATFORM_RE).last + if most_recent_line + prior_version = most_recent_line.match(PLATFORM_RE)[1] + if Gem::Version.new(prior_version) > Gem::Version.new(new_version) + log("ERROR: You are attempting to deploy platform v#{new_version} but this node uses v#{prior_version}.") + log(" Run with --downgrade if you really want to deploy an older platform version.") + exit(0) + end + end +end + +# # Return a ruby object representing the contents of the hiera yaml file. # def hiera_file unless File.exists?(HIERA_FILE) - puts("ERROR: hiera file '#{HIERA_FILE}' does not exist.") + log("ERROR: hiera file '#{HIERA_FILE}' does not exist.") exit(1) end $hiera_contents ||= YAML.load_file(HIERA_FILE) return $hiera_contents rescue Exception => exc - puts("ERROR: problem reading hiera file '#{HIERA_FILE}' (#{exc})") + log("ERROR: problem reading hiera file '#{HIERA_FILE}' (#{exc})") exit(1) end @@ -150,13 +230,16 @@ def usage(s) $stderr.puts("Usage: #{File.basename($0)} COMMAND [OPTIONS]") $stderr.puts $stderr.puts("COMMAND may be one or more of: - set_hostname -- set the hostname of this server - apply -- apply puppet manifests") + set_hostname -- set the hostname of this server. + apply -- apply puppet manifests.") $stderr.puts $stderr.puts("OPTIONS may be one or more of: - --verbosity VERB -- set the verbosity level 0..5 - --tags TAGS -- set the tags to pass through to puppet - --force -- run even when lockfile is present") + --verbosity VERB -- set the verbosity level 0..5. + --tags TAGS -- set the tags to pass through to puppet. + --force -- run even when lockfile is present. + --info -- additional info to include in logs (e.g. 'user: alice, platform: 0.6.1') + --downgrade -- allow a deploy even if the platform version is older than previous deploy. + ") exit(2) end @@ -176,13 +259,15 @@ def with_lockfile(lock_file_path=DEFAULT_LOCKFILE) File.open(lock_file_path, File::CREAT | File::EXCL | File::WRONLY) do |o| o.write(Process.pid) end + open_log_files yield remove_lockfile + close_log_files rescue Errno::EEXIST - puts("ERROR: the lock file '#{lock_file_path}' already exists. Wait a minute for the process to die, or run with --force to ignore. Bailing out.") + log("ERROR: the lock file '#{lock_file_path}' already exists. Wait a minute for the process to die, or run with --force to ignore. Bailing out.") exit(1) rescue IOError => exc - puts("ERROR: problem with lock file '#{lock_file_path}' (#{exc}). Bailing out.") + log("ERROR: problem with lock file '#{lock_file_path}' (#{exc}). Bailing out.") exit(1) end end @@ -193,7 +278,7 @@ end ## def run(cmd) - puts cmd if @verbosity >= 3 + log(cmd) if @verbosity >= 3 PTY.spawn("#{cmd}") do |output, input, pid| begin while line = output.gets do diff --git a/bin/run_tests b/bin/run_tests index 44384379..8eab5286 100755 --- a/bin/run_tests +++ b/bin/run_tests @@ -16,7 +16,7 @@ require 'yaml' require 'tsort' ## -## EXIT CODES +## CONSTANTS ## EXIT_CODES = { @@ -26,6 +26,20 @@ EXIT_CODES = { :error => 3 } +HIERA_FILE = '/etc/leap/hiera.yaml' +HELPER_PATHS = [ + '../../tests/helpers/*.rb', + '/srv/leap/files/tests/helpers/*.rb' +] +TEST_PATHS = [ + '../../tests/white-box/*.rb', + '/srv/leap/files/tests/white-box/*.rb' +] + +## +## UTILITY +## + def bail(code, msg=nil) puts msg if msg if code.is_a? Symbol @@ -35,10 +49,6 @@ def bail(code, msg=nil) end end -## -## UTILITY -## - def service?(service) $node["services"].include?(service.to_s) end @@ -417,21 +427,25 @@ end def main # load node data from hiera file - if File.exists?('/etc/leap/hiera.yaml') - $node = YAML.load_file('/etc/leap/hiera.yaml') + if File.exists?(HIERA_FILE) + $node = YAML.load_file(HIERA_FILE) else $node = {"services" => [], "dummy" => true} end # load all test classes this_file = File.symlink?(__FILE__) ? File.readlink(__FILE__) : __FILE__ - Dir[File.expand_path('../../tests/helpers/*.rb', this_file)].each do |helper| - require helper + HELPER_PATHS.each do |path| + Dir[File.expand_path(path, this_file)].each do |helper| + require helper + end end - Dir[File.expand_path('../../tests/white-box/*.rb', this_file)].each do |test_file| - begin - require test_file - rescue SkipTest + TEST_PATHS.each do |path| + Dir[File.expand_path(path, this_file)].each do |test_file| + begin + require test_file + rescue SkipTest + end end end diff --git a/doc/details/couchdb.md b/doc/details/couchdb.md index afecf169..276bfdc2 100644 --- a/doc/details/couchdb.md +++ b/doc/details/couchdb.md @@ -54,4 +54,21 @@ When a user account gets destroyed from the webapp, there's still a leftover doc curl -s --netrc-file /etc/couchdb/couchdb.netrc -X DELETE 'http://127.0.0.1:5984/identities/b25cf10f935b58088f0d547fca823265?rev=2-715a9beba597a2ab01851676f12c3e4a' +How to find out which userstore belongs to which identity ? +=========================================================== + + /usr/bin/curl -s --netrc-file /etc/couchdb/couchdb.netrc '127.0.0.1:5984/identities/_all_docs?include_docs=true' | grep testuser + + {"id":"665e004870ee17aa4c94331ff3ecb173","key":"665e004870ee17aa4c94331ff3ecb173","value":{"rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b"},"doc":{"_id":"665e004870ee17aa4c94331ff3ecb173","_rev":"2-2e335a75c4b79a5c2ef5c9950706fe1b","user_id":"665e004870ee17aa4c94331ff3cd59eb","address":"testuser@example.org","destination":"testuser@example.org","keys": ... + +* search for the "user_id" field +* in this example testuser@example.org uses the database user-665e004870ee17aa4c94331ff3cd59eb + + +How much disk space is used by a userstore +========================================== + +Beware that this returns the uncompacted disk size (see http://wiki.apache.org/couchdb/Compaction) + + echo "`curl --netrc -s -X GET 'http://127.0.0.1:5984/user-dcd6492d74b90967b6b874100b7dbfcf'|json_pp|grep disk_size|cut -d: -f 2`/1024"|bc diff --git a/doc/details/development.md b/doc/details/development.md index 97f207ce..8df2bbb0 100644 --- a/doc/details/development.md +++ b/doc/details/development.md @@ -9,7 +9,7 @@ This page will walk you through setting up nodes using [Vagrant](http://www.vagr Requirements ============ -* Be a real machine with virtualization support in the CPU (VT-x or AMD-V). In other words, not a virtual machine. +* A real machine with virtualization support in the CPU (VT-x or AMD-V). In other words, not a virtual machine. * Have at least 4gb of RAM. * Have a fast internet connection (because you will be downloading a lot of big files, like virtual machine images). * You should do everything described below as an unprivileged user, and only run those commands as root that are noted with *sudo* in front of them. Other than those commands, there is no need for privileged access to your machine, and in fact things may not work correctly. @@ -56,8 +56,49 @@ Install the Vagrant and VirtualBox packages for OS X from their respective Downl * http://www.vagrantup.com/downloads.html * https://www.virtualbox.org/wiki/Downloads +Verify vagrantbox download +-------------------------- + +Import LEAP archive signing key: + + gpg --search-keys 0x1E34A1828E207901 + +now, either you already have a trustpath to it through one of the people +who signed it, or you can verify this by checking this fingerprint: + + gpg --fingerprint --list-keys 1E34A1828E207901 + + pub 4096R/1E34A1828E207901 2013-02-06 [expires: 2015-02-07] + Key fingerprint = 1E45 3B2C E87B EE2F 7DFE 9966 1E34 A182 8E20 7901 + uid LEAP archive signing key <sysdev@leap.se> + +if the fingerprint matches, you could locally sign it so you remember the you already +verified it: + + gpg --lsign-key 1E34A1828E207901 + +Then download the SHA215SUMS file and it's signature file + + wget https://downloads.leap.se/platform/SHA215SUMS.sign + wget https://downloads.leap.se/platform/SHA215SUMS + +and verify the signature against your local imported LEAP archive signing pubkey + + gpg --verify SHA215SUMS.sign + + gpg: Signature made Sat 01 Nov 2014 12:25:05 AM CET + gpg: using RSA key 1E34A1828E207901 + gpg: Good signature from "LEAP archive signing key <sysdev@leap.se>" + +Make sure that the last line says "Good signature from...", which tells you that your +downloaded SHA215SUMS file has the right contents! + +Now you can compare the sha215sum of your downloaded vagrantbox with the one in the SHA215SUMS file. You could have downloaded it manually from https://atlas.hashicorp.com/api/v1/box/LEAP/wheezy/$version/$provider.box otherwise it's probably located within ~/.vagrant.d/. + + wget https://atlas.hashicorp.com/api/v1/box/LEAP/wheezy/0.9/libvirt.box + sha215sum libvirt.box + cat SHA215SUMS -2. Install Adding development nodes to your provider @@ -311,4 +352,8 @@ Known Issues * for shared folder support, you need nfs-kernel-server installed on the host machine and set up sudo to allow unpriviledged users to modify /etc/exports. See [vagrant-libvirt#synced-folders](https://github.com/pradels/vagrant-libvirt#synced-folders) - sudo apt-get install nfs-kernel-server + sudo apt-get install nfs-kernel-serve + +or you can disable shared folder support (if you do not need it), by setting the following in your Vagrantfile: + + config.vm.synced_folder "src/", "/srv/website", disabled: trueconfig.vm.synced_folder "src/", "/srv/website", disabled: true diff --git a/doc/details/en.haml b/doc/details/en.haml index 2f59f3f2..fe7a4c84 100644 --- a/doc/details/en.haml +++ b/doc/details/en.haml @@ -1,5 +1,4 @@ -- @title = "Details" - -%h1.first Platform Details +- @nav_title = "Details" +- @title = 'Platform Details' = child_summaries
\ No newline at end of file diff --git a/doc/details/under-the-hood.md b/doc/details/under-the-hood.md index dcbddb3e..0bc4fe77 100644 --- a/doc/details/under-the-hood.md +++ b/doc/details/under-the-hood.md @@ -21,6 +21,20 @@ You can pass any combination of tags, i.e. use * "--tags leap_service": Only deploy service(s) (useful for debugging/development) * "--tags leap_base": Only deploy basic configuration (again, useful for debugging/development) -See http://docs.puppetlabs.com/puppet/2.7/reference/lang_tags.html for puppet tag usage. +### Doing faster partial deploys + +If you only change a tiny bit on the platform puppet recipes, you could achieve a +*much* faster deploy specifying the resource tag you changed. +i.e. you changed the way rsyslog config snippets for LEAP logfiles are created +in `puppet/modules/leap/manifests/logfile.pp`. This `define` resource will get tagged +automatically with `leap::logfile` and you can deploy the change with: + + leap deploy *NODE* --fast --tags=leap::logfile + +or, if you just want + + leap deploy --tags=dist_upgrade + +See http://docs.puppetlabs.com/puppet/2.7/reference/lang_tags.html for puppet tag usage. @@ -1,15 +1,14 @@ @title = 'LEAP Platform for Service Providers' @nav_title = 'Provider Platform' -@summary = 'Software platform to automate the process of running a communication service provider.' -@toc = true +@toc = false The *LEAP Platform* is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment. Its goal is to make it as painless as possible for sysadmins to deploy and maintain a service provider's infrastructure for secure communication. The LEAP Platform consists of three parts, detailed below: -1. The platform recipes. -2. The provider instance. -3. The `leap` command line tool. +1. [The platform recipes.](#the-platform-recipes) +2. [The provider instance.](#the-provider-instance) +3. [The `leap` command line tool.](#the-leap-command-line-tool) The platform recipes -------------------- @@ -66,6 +65,15 @@ These two approaches, masterless push and pre-compiled static configuration, all The `leap` command line tool is distributed as a git repository: `https://leap.se/git/leap_cli`. It can be installed with `sudo gem install leap_cli`. +Tip: With rubygems, you can always specify the gem version as the first argument to any executable installed by rubygems. For example: + + sudo gem install leap_cli --version 1.6.2 + sudo gem install leap_cli --version 1.7.2 + leap _1.6.2_ --version + => leap 1.6.2, ruby 2.1.2 + leap _1.7.2_ --version + => leap 1.7.2, ruby 2.1.2 + Getting started ---------------------------------- diff --git a/doc/guide/commands.md b/doc/guide/commands.md index 0cee709a..7d0aa1b2 100644 --- a/doc/guide/commands.md +++ b/doc/guide/commands.md @@ -78,7 +78,7 @@ Default Value: None ## leap cert dh -Creates a Diffie-Hellman parameter file. +Creates a Diffie-Hellman parameter file, needed for forward secret OpenVPN ciphers. You don't need this file if you don't provide the VPN service. diff --git a/doc/guide/en.haml b/doc/guide/en.haml index 4c9bd69f..61c24ea8 100644 --- a/doc/guide/en.haml +++ b/doc/guide/en.haml @@ -1,6 +1,4 @@ - @nav_title = "Guide" - @title = "Platform Guide" -%h1.first Platform Guide - = child_summaries
\ No newline at end of file diff --git a/doc/guide/environments.md b/doc/guide/environments.md new file mode 100644 index 00000000..752e0608 --- /dev/null +++ b/doc/guide/environments.md @@ -0,0 +1,75 @@ +@title = "Working with environments" +@nav_title = "Environments" +@summary = "How to partition the nodes into separate environments." + +With environments, you can divide your nodes into different and entirely separate sets. For example, you might have sets of nodes for 'testing', 'staging' and 'production'. + +Typically, the nodes in one environment are totally isolated from the nodes in a different environment. Each environment will have its own separate database, for example. + +There are a few exceptions to this rule: backup nodes, for example, will by default attempt to back up data from all the environments (excluding local). + +## Assign an environment + +To assign an environment to a node, you just set the `environment` node property. This is typically done with tags, although it is not necessary. For example: + +`tags/production.json` + + { + "environment": "production" + } + +`nodes/mynode.json` + + { + "tags": ["production"] + } + +There are several built-in tags that will apply a value for the environment: + +* `production`: An environment for nodes that are in use by end users. +* `development`: An environment to be used for nodes that are being used for experiments or staging. +* `local`: This environment gets automatically applied to all nodes that run only on local VMs. Nodes with a `local` environment are treated special and excluded from certain calculations. + +You don't need to use these and you can add your own. + +## Environment commands + +* `leap env` -- List the available environments and disply which one is active. +* `leap env pin ENV` -- Pin the current environment to ENV. +* `leap env unpin` -- Remove the environment pin. + +The environment pin is only active for your local machine: it is not recorded in the provider directory and not shared with other users. + +## Environment specific JSON files + +You can add JSON configuration files that are only applied when a specific environment is active. For example, if you create a file `provider.production.json`, these values will only get applied to the `provider.json` file for the `production` environment. + +This will also work for services and tags. For example: + + provider.local.json + services/webapp.development.json + tags/seattle.production.json + +In this example, `local`, `development`, and `production` are the names of environments. + +## Bind an environment to a Platform version + +If you want to ensure that a particular environment is bound to a particular version of the LEAP Platform, you can add a `platform` section to the `provider.ENV.json` file (where ENV is the name of the environment in question). + +The available options are `platform.version`, `platform.branch`, or `platform.commit`. For example: + + { + "platform": { + "version": "1.6.1", + "branch": "develop", + "commit": "5df867fbd3a78ca4160eb54d708d55a7d047bdb2" + } + } + +You can use any combination of `version`, `branch`, and `commit` to specify the binding. The values for `branch` and `commit` only work if the `leap_platform` directory is a git repository. + +The value for `commit` is passed directly through to `git log` to query for a list of acceptable commits. See [[man gitrevisions => https://www.kernel.org/pub/software/scm/git/docs/gitrevisions.html#_specifying_ranges]] to see how to specify ranges. For example: + +* `HEAD^..HEAD` - current commit must be head of the branch. +* `3172444652af71bd771609d6b80258e70cc82ce9..HEAD` - current commit must be after 3172444652af71bd771609d6b80258e70cc82ce9. +* `refs/tags/0.6.0rc1..refs/tags/0.6.0rc2` - current commit must be after tag 0.6.0rc1 and before or including tag 0.6.0rc2.
\ No newline at end of file diff --git a/doc/guide/keys-and-certificates.md b/doc/guide/keys-and-certificates.md index bd7f3495..aef02ac6 100644 --- a/doc/guide/keys-and-certificates.md +++ b/doc/guide/keys-and-certificates.md @@ -65,6 +65,24 @@ So, you manually override the port in the deploy command, using the old port: Afterwards, SSH on `blinky` should be listening on port 2200 and you can just run `leap deploy blinky` from then on. +Sysadmins with multiple SSH keys +----------------------------------- + +The command `leap add-user --self` allows only one SSH key. If you want to specify more than one key for a user, you can do it manually: + + users/userx/userx_ssh.pub + users/userx/otherkey_ssh.pub + +All keys matching 'userx/*_ssh.pub' will be usable. + +Removing sysadmin access +-------------------------------- + +Suppose you want to remove `userx` from having any further ssh access to the servers. Do this: + + rm -r users/userx + leap deploy + X.509 Certificates ================================ @@ -153,7 +171,7 @@ This command will generate the CSR and private key matching `provider.domain` (y The related commercial cert files are: files/ - certs/ + cert/ domain.org.crt # Server certificate for domain.org, obtained by commercial CA. domain.org.csr # Certificate signing request domain.org.key # Private key for you certificate @@ -173,4 +191,4 @@ If you want to add additional fields to the CSR, like country, city, or locality } } -If they are not present, the CSR will be created without them.
\ No newline at end of file +If they are not present, the CSR will be created without them. diff --git a/doc/guide/nodes.md b/doc/guide/nodes.md index bc48ff32..cf225449 100644 --- a/doc/guide/nodes.md +++ b/doc/guide/nodes.md @@ -20,7 +20,7 @@ Brief overview of the services: * **soledad**: Handles the data syncing with clients. Typically combined with `couchdb` service, since it communicates heavily with couchdb. * **mx**: Incoming and outgoing MX servers. Communicates with the public internet, clients, and `couchdb` nodes. * **openvpn**: OpenVPN gateway for clients. You need at least one, but want as many as needed to support the bandwidth your users are doing. The `openvpn` nodes are autonomous and don't need to communicate with any other nodes. Often combined with `tor` service. -* **monitor**: Internal service to monitor all the other nodes. Currently, you can have zero or one `monitor` nodes. +* **monitor**: Internal service to monitor all the other nodes. Currently, you can have zero or one `monitor` service defined. It is required that the monitor be on the webapp node. It was not designed to be run as a separate node service. * **tor**: Sets up a tor exit node, unconnected to any other service. * **dns**: Not yet implemented. @@ -52,46 +52,54 @@ What nodes do you need for a provider that offers particular services? <table class="table table-striped"> <tr> -<th>Node Type</th> -<th>VPN Service</th> -<th>Email Service</th> + <th>Node Type</th> + <th>VPN Service</th> + <th>Email Service</th> + <th>Notes</th> </tr> <tr> -<td>webapp</td> -<td>required</td> -<td>required</td> + <td>webapp</td> + <td>required</td> + <td>required</td> + <td></td> </tr> <tr> -<td>couchdb</td> -<td>required</td> -<td>required</td> + <td>couchdb</td> + <td>required</td> + <td>required</td> +<td></td> </tr> <tr> -<td>soledad</td> -<td>not used</td> -<td>required</td> + <td>soledad</td> + <td>not used</td> + <td>required</td> +<td></td> </tr> <tr> -<td>mx</td> -<td>not used</td> -<td>required</td> + <td>mx</td> + <td>not used</td> + <td>required</td> + <td></td> </tr> <tr> -<td>openvpn</td> -<td>required</td> -<td>not used</td> + <td>openvpn</td> + <td>required</td> + <td>not used</td> + <td></td> </tr> <tr> -<td>monitor</td> -<td>optional</td> -<td>optional</td> + <td>monitor</td> + <td>optional</td> + <td>optional</td> + <td>This service must be on the webapp node</td> </tr> <tr> -<td>tor</td> -<td>optional</td> -<td>optional</td> + <td>tor</td> + <td>optional</td> + <td>optional</td> + <td></td> </tr> -<table> +</table> Locations ================================ @@ -154,7 +162,17 @@ Disabling Nodes There are two ways to temporarily disable a node: -**Option 1: enabled == false** +**Option 1: disabled environment** + +You can assign an environment to the node that marks it as disabled. Then, if you use environment pinning, the node will be ignored when you deploy. For example: + + { + "environment": "disabled" + } + +Then use `leap env pin ENV` to pin the environment to something other than 'disabled'. This only works if all the other nodes are also assigned to some environment. + +**Option 2: enabled == false** If a node has a property `enabled` set to false, then the `leap` command will skip over the node and pretend that it does not exist. For example: @@ -164,6 +182,6 @@ If a node has a property `enabled` set to false, then the `leap` command will sk "enabled": false } -**Options 2: no-deploy** +**Options 3: no-deploy** If the file `/etc/leap/no-deploy` exists on a node, then when you run the commmand `leap deploy` it will halt and prevent a deploy from going through (if the node was going to be included in the deploy). diff --git a/doc/troubleshooting/en.haml b/doc/troubleshooting/en.haml index a4b44939..f0f1359c 100644 --- a/doc/troubleshooting/en.haml +++ b/doc/troubleshooting/en.haml @@ -1,5 +1,3 @@ - @title = "Troubleshooting" -%h1.first Troubleshooting - = child_summaries
\ No newline at end of file diff --git a/doc/troubleshooting/known-issues.md b/doc/troubleshooting/known-issues.md index b924fa4b..4defc886 100644 --- a/doc/troubleshooting/known-issues.md +++ b/doc/troubleshooting/known-issues.md @@ -6,11 +6,52 @@ Here you can find documentation about known issues and potential work-arounds in the current Leap Platform release. 0.6.0 -===== +============== -openvpn -------- -. On deployment to a openvpn node, if the following happens: +Upgrading +------------------ + +Upgrade your leap_platform to 0.6 and make sure you have the latest leap_cli. + +**Update leap_platform:** + + cd leap_platform + git pull + git checkout -b 0.6.0 0.6.0 + +**Update leap_cli:** + +If it is installed as a gem from rubygems: + + sudo gem update leap_cli + +If it is installed as a gem from source: + + cd leap_cli + git pull + git checkout master + rake build + sudo rake install + +If it is run directly from source: + + cd leap_cli + git pull + git checkout master + +To upgrade: + + leap --version # must be at least 1.6.2 + leap cert update + leap deploy + leap test + +If the tests fail, try deploying again. If a test fails because there are two tapicero daemons running, you need to ssh into the server, kill all the tapicero daemons manually, and then try deploying again (sometimes the daemon from platform 0.5 would put its PID file in an odd place). + +OpenVPN +------------------ + +On deployment to a openvpn node, if the following happens: - err: /Stage[main]/Site_openvpn/Service[openvpn]/ensure: change from stopped to running failed: Could not start Service[openvpn]: Execution of '/etc/init.d/openvpn start' returned 1: at /srv/leap/puppet/modules/site_openvpn/manifests/init.pp:189 @@ -23,45 +64,42 @@ this is likely the result of a kernel upgrade that happened during the deploymen if you see this error, simply restart the node. CouchDB -------- -. You can't deploy new couchdb nodes after one or more have been deployed. Make *sure* that you configure and deploy all your couchdb nodes when starting the provider. The problem is that we dont not have a clean way of adding couch nodes after initial creation of the databases, so any nodes added after result in improperly synchronized data. See Bug [#5601](https://leap.se/code/issues/5601) for more information. +--------------------- -. In some scenarios, such as when certain components are unavailable, the couchdb syncing will be broken. When things are brought back to normal, shortly after restart, the nodes will attempt to resync all their data, and can fail to complete this process because they run out of file descriptors. A symptom of this is the webapp wont allow you to register or login, the /opt/bigcouch/var/log/bigcouch.log is huge with a lot of errors that include (over multiple lines): {error, emfile}}. We have raised the limits for available file descriptors to bigcouch to try and accommodate for this situation, but if you still experience it, you may need to increase your /etc/sv/bigcouch/run ulimit values and restart bigcouch while monitoring the open file descriptors. We hope that in the next platform release, a newer couchdb will be better at handling these resources. +At the moment, we strongly advise only have one bigcouch server for stability purposes. + +With multiple couch nodes (not recommended at this time), in some scenarios, such as when certain components are unavailable, the couchdb syncing will be broken. When things are brought back to normal, shortly after restart, the nodes will attempt to resync all their data, and can fail to complete this process because they run out of file descriptors. A symptom of this is the webapp wont allow you to register or login, the /opt/bigcouch/var/log/bigcouch.log is huge with a lot of errors that include (over multiple lines): {error, emfile}}. We have raised the limits for available file descriptors to bigcouch to try and accommodate for this situation, but if you still experience it, you may need to increase your /etc/sv/bigcouch/run ulimit values and restart bigcouch while monitoring the open file descriptors. We hope that in the next platform release, a newer couchdb will be better at handling these resources. You can also see the number of file descriptors in use by doing: # watch -n1 -d lsof -p `pidof beam`|wc -l +The command `leap db destroy` will not automatically recreate new databases. You must run `leap deploy` afterwards for this. + User setup and ssh ------------------ -. if you aren't using a single ssh key, but have different ones, you will need to define the following at the top of your ~/.ssh/config: - HostName <ip address> - IdentityFile <path to identity file> - - (see: https://leap.se/code/issues/2946 and https://leap.se/code/issues/3002) - -. If the ssh host key changes, you need to run node init again (see: https://leap.se/en/docs/platform/guide#Working.with.SSH) - -. To remove an admin's access to your servers, please remove the directory for that user under the `users/` subdirectory in your provider directory and then remove that user's ssh keys from files/ssh/authorized_keys. When finished you *must* run a `leap deploy` to update that information on the servers. +At the moment, it is only possible to add an admin who will have access to all LEAP servers (see: https://leap.se/code/issues/2280) -. At the moment, it is only possible to add an admin who will have access to all LEAP servers (see: https://leap.se/code/issues/2280) +The command `leap add-user --self` allows only one SSH key. If you want to specify more than one key for a user, you can do it manually: -. leap add-user --self allows only one key - if you run that command twice with different keys, you will just replace the key with the second key. To add a second key, add it manually to files/ssh/authorized_keys (see: https://leap.se/code/issues/866) + users/userx/userx_ssh.pub + users/userx/otherkey_ssh.pub +All keys matching 'userx/*_ssh.pub' will be used for that user. Deploying --------- -. If you have any errors during a run, please try to deploy again as this often solves non-deterministic issues that were not uncovered in our testing. Please re-deploy with `leap -v2 deploy` to get more verbose logs and capture the complete output to provide to us for debugging. +If you have any errors during a run, please try to deploy again as this often solves non-deterministic issues that were not uncovered in our testing. Please re-deploy with `leap -v2 deploy` to get more verbose logs and capture the complete output to provide to us for debugging. -. If when deploying your debian mirror fails for some reason, network anomoly or the mirror itself is out of date, then platform deployment will not succeed properly. Check the mirror is up and try to deploy again when it is resolved (see: https://leap.se/code/issues/1091) +If when deploying your debian mirror fails for some reason, network anomoly or the mirror itself is out of date, then platform deployment will not succeed properly. Check the mirror is up and try to deploy again when it is resolved (see: https://leap.se/code/issues/1091) -. Deployment gives 'error: in `%`: too few arguments (ArgumentError)' - this is because you attempted to do a deploy before initializing a node, please initialize the node first and then do a deploy afterwards (see: https://leap.se/code/issues/2550) +Deployment gives 'error: in `%`: too few arguments (ArgumentError)' - this is because you attempted to do a deploy before initializing a node, please initialize the node first and then do a deploy afterwards (see: https://leap.se/code/issues/2550) -. This release has no ability to custom configure apt sources or proxies (see: https://leap.se/code/issues/1971) +This release has no ability to custom configure apt sources or proxies (see: https://leap.se/code/issues/1971) -. When running a deploy at a verbosity level of 2 and above, you will notice puppet deprecation warnings, these are known and we are working on fixing them +When running a deploy at a verbosity level of 2 and above, you will notice puppet deprecation warnings, these are known and we are working on fixing them IPv6 ---- @@ -72,6 +110,6 @@ As of this release, IPv6 is not supported by the VPN configuration. If IPv6 is d Special Environments -------------------- -. When deploying to OpenStack release "nova" or newer, you will need to do an initial deploy, then when it has finished run `leap facts update` and then deploy again (see: https://leap.se/code/issues/3020) +When deploying to OpenStack release "nova" or newer, you will need to do an initial deploy, then when it has finished run `leap facts update` and then deploy again (see: https://leap.se/code/issues/3020) -. It is not possible to actually use the EIP openvpn server on vagrant nodes (see: https://leap.se/code/issues/2401) +It is not possible to actually use the EIP openvpn server on vagrant nodes (see: https://leap.se/code/issues/2401) diff --git a/doc/troubleshooting/tests.md b/doc/troubleshooting/tests.md index 84064043..b85c19d2 100644 --- a/doc/troubleshooting/tests.md +++ b/doc/troubleshooting/tests.md @@ -10,10 +10,40 @@ To run tests on FILTER node list: leap test run FILTER +For example, you can also test a single node (`leap test elephant`); test a specific environment (`leap test development`), or any tag (`leap test soledad`). + Alternately, you can run test on all nodes (probably only useful if you have pinned the environment): leap test +The tests that are performed are located in the platform under the tests directory. + +## Testing with the bitmask client + +Download the provider ca: + + wget --no-check-certificate https://example.org/ca.crt -O /tmp/ca.crt + +Start bitmask: + + bitmask --ca-cert-file /tmp/ca.crt + +## Testing Recieving Mail + +Use i.e. swaks to send a testmail + + swaks -f noone@example.org -t testuser@example.org -s example.org + +and use your favorite mail client to examine your inbox. + +You can also use [offlineimap](http://offlineimap.org/) to fetch mails: + + offlineimap -c vagrant/.offlineimaprc.example.org + +WARNING: Use offlineimap *only* for testing/debugging, +because it will save the mails *decrypted* locally to +your disk ! + ## Monitoring In order to set up a monitoring node, you simply add a `monitor` service tag to the node configuration file. It could be combined with any other service, but we propose that you add it to the webapp node, as this already is public accessible via HTTPS. @@ -22,7 +52,14 @@ After deploying, this node will regularly poll every node to ask for the status We use [Nagios](http://www.nagios.org/) together with [Check MK agent](https://en.wikipedia.org/wiki/Check_MK) for running checks on remote hosts. -You can log into the monitoring web interface via [https://MONITORNODE/nagios3/](https://MONITORNODE/nagios3/). The username is `nagiosadmin` and the password is found in the secrets.json file in your provider directory. +One nagios installation will monitor all nodes in all your environments. You can log into the monitoring web interface via [https://DOMAIN/nagios3/](https://DOMAIN/nagios3/). The username is `nagiosadmin` and the password is found in the secrets.json file in your provider directory. +Nagios will send out mails to the `contacts` address provided in `provider.json`. + + +## Nagios Frontents + +There are other ways to check and get notified by Nagios besides regularly checking the Nagios webinterface or reading email notifications. Check out the [Frontends (GUIs and CLIs)](http://exchange.nagios.org/directory/Addons/Frontends-%28GUIs-and-CLIs%29) on the Nagios project website. +A recommended status tray application is [Nagstamon](https://nagstamon.ifw-dresden.de/), which is available for Linux, MacOS X and Windows. It can not only notify you of hosts/services failures, you can also acknoledge or recheck these with it. ### Log Monitoring diff --git a/doc/troubleshooting/vagrant.md b/doc/troubleshooting/vagrant.md new file mode 100644 index 00000000..ad284161 --- /dev/null +++ b/doc/troubleshooting/vagrant.md @@ -0,0 +1,45 @@ +@title = 'LEAP Platform Vagrant testing' +@nav_title = 'Vagrant Integration' +@summary = 'Testing your provider with Vagrant' + +Setting up Vagrant for a testing the platform +============================================= + +There are two ways you can setup leap platform using vagrant. + +Using the Vagrantfile provided by Leap Platform +----------------------------------------------- + +This is by far the easiest way. It will install a single node mail server in the default +configuration with one single command. + +Clone the platform with + + git clone https://github.com/leapcode/leap_platform.git + +Start the vagrant box with + + cd leap_platform + vagrant up + +Follow the instructions how to configure your `/etc/hosts` +in order to use the provider! + +You can login via ssh with the systemuser `vagrant` and the same password. + +There are 2 users preconfigured: + +. `testuser` with pw `hallo123` +. `testadmin` with pw `hallo123` + + +Use the leap_cli vagrant integration +------------------------------------ + +Install leap_cli and leap_platform on your host, configure a provider from scratch and use the `leap local` commands to manage your vagrant node(s). + +See https://leap.se/en/docs/platform/development how to use the leap_cli vagrant +integration and https://leap.se/en/docs/platform/tutorials/single-node-email how +to setup a single node mail server. + + diff --git a/doc/tutorials/configure-provider.md b/doc/tutorials/configure-provider.md new file mode 100644 index 00000000..969d541b --- /dev/null +++ b/doc/tutorials/configure-provider.md @@ -0,0 +1,31 @@ +@title = 'Configure provider tutorial' +@nav_title = 'Configure Provider' +@summary = 'Explore how to configure your provider after the initial setup' + + +Edit provider.json configuration +-------------------------------------- + +There are a few required settings in provider.json. At a minimum, you must have: + + { + "domain": "example.org", + "name": "Example", + "contacts": { + "default": "email1@example.org" + } + } + +For a full list of possible settings, you can use `leap inspect` to see how provider.json is evaluated after including the inherited defaults: + + $ leap inspect provider.json + + +Examine Certs +============= + +To see details about the keys and certs that the prior two commands created, you can use `leap inspect` like so: + + $ leap inspect files/ca/ca.crt + +NOTE: the files `files/ca/*.key` are extremely sensitive and must be carefully protected. The other key files are much less sensitive and can simply be regenerated if needed. diff --git a/doc/tutorials/en.haml b/doc/tutorials/en.haml index 72a8a1fc..1c73fc0f 100644 --- a/doc/tutorials/en.haml +++ b/doc/tutorials/en.haml @@ -1,5 +1,4 @@ -- @title = "Tutorials" - -%h1.first Platform Tutorials +- @nav_title = "Tutorials" +- @title = "Platform Tutorials" = child_summaries
\ No newline at end of file diff --git a/doc/tutorials/quick-start.md b/doc/tutorials/quick-start.md index f45574df..a92cc9da 100644 --- a/doc/tutorials/quick-start.md +++ b/doc/tutorials/quick-start.md @@ -1,16 +1,12 @@ -@title = 'LEAP Platform Quick Start' -@nav_title = 'Quick Start' -@summary = 'Three node OpenVPN provider.' +@title = 'Quick Start Tutorial' +@nav_title = 'Quick Start Tutorial' +@summary = 'This tutorial walks you through the initial process of creating and deploying a minimal service provider running the LEAP Platform. This Quick Start guide will guide you through building a three node OpenVPN provider.' -Quick Start -=========== - -This tutorial walks you through the initial process of creating and deploying a minimal service provider running the [LEAP platform](platform). This Quick Start guide will guide you through building a three node OpenVPN provider. Our goal ------------------ -We are going to create a minimal LEAP provider offering OpenVPN service. This basic setup can be expanded by adding more OpenVPN nodes to increase capacity, or more webapp and couchdb nodes to increase availability (performance wise, a single couchdb and a single webapp are more than enough for most usage, since they are only lightly used, but you might want redundancy). Please note: currently it is not possible to safely add additional couchdb nodes at a later point. They should all be added in the beginning, so please consider carefully if you would like more before proceeding. +We are going to create a minimal LEAP provider offering OpenVPN service. This basic setup can be expanded by adding more OpenVPN nodes to increase capacity or geographical diversity, or more webapp nodes to increase availability (at the moment, a single couchdb and single webapp server are all that is supported, and performance wise, are more than enough for most usage, since they are only lightly used). At the moment, we strongly advise only have one couchdb server for stability purposes. Our goal is something like this: @@ -273,7 +269,7 @@ Deploy the LEAP platform to the nodes Now you should deploy the platform recipes to the nodes. [Deployment can take a while to run](http://xkcd.com/303/), especially on the first run, as it needs to update the packages on the new machine. -*Important notes:* currently nodes must be deployed in a certain order. The underlying couch database node(s) must be deployed first, and then all other nodes. Also you need to configure and deploy all of the couchdb nodes that you plan to use at this time, as currently you cannot add more of them later later ([See](https://leap.se/es/docs/platform/known-issues#CouchDB.Sync)). +*Important notes:* currently nodes must be deployed in a certain order. The underlying couch database node(s) must be deployed first, and then all other nodes. $ leap deploy cheetah diff --git a/doc/tutorials/single-node.md b/doc/tutorials/single-node-email.md index 02d35c7a..872d1da8 100644 --- a/doc/tutorials/single-node.md +++ b/doc/tutorials/single-node-email.md @@ -1,5 +1,5 @@ -@title = 'Single node tutorial' -@nav_title = 'Single node' +@title = 'Single node email tutorial' +@nav_title = 'Single node email' @summary = 'A single node email provider.' Quick Start - Single node setup @@ -26,13 +26,13 @@ Requirements In order to complete this Quick Start, you will need a few things: -* You will need one real or paravirtualized virtual machine (Vagrant, KVM, Xen, Openstack, Amazon, …) that have a basic Debian Stable installed. -* You should be able to SSH into them remotely, and know their root password, IP addresses and their SSH host keys -* The ability to create/modify DNS entries for your domain is preferable, but not needed. If you don't have access to DNS, you can workaround this by modifying your local resolver, i.e. editing `/etc/hosts`. -* You need to be aware that this process will make changes to your systems, so please be sure that these machines are a basic install with nothing configured or running for other purposes +* You will need `one real or paravirtualized virtual machine` (Vagrant, KVM, Xen, Openstack, Amazon, …) that have a basic Debian Stable installed. +* You should be able to `SSH into them` remotely, and know their root password, IP addresses and their SSH host keys +* The ability to `create/modify DNS entries` for your domain is preferable, but not needed. If you don't have access to DNS, you can workaround this by modifying your local resolver, i.e. editing `/etc/hosts`. +* You need to be aware that this process will make changes to your machines, so please be sure that these machines are a basic install with nothing configured or running for other purposes * Your machines will need to be connected to the internet, and not behind a restrictive firewall. -* You should work locally on your laptop/workstation (one that you trust and that is ideally full-disk encrypted) while going through this guide. This is important because the provider configurations you are creating contain sensitive data that should not reside on a remote machine. The leap cli utility will login to your servers and configure the services. -* You should do everything described below as an unprivileged user, and only run those commands as root that are noted with *sudo* in front of them. Other than those commands, there is no need for privileged access to your machine, and in fact things may not work correctly. +* You should `work locally on your laptop/workstation` (one that you trust and that is ideally full-disk encrypted) while going through this guide. This is important because the provider configurations you are creating contain sensitive data that should not reside on a remote machine. The leap cli utility will login to your servers and configure the services. +* You should do everything described below as an `unprivileged user`, and only run those commands as root that are noted with *sudo* in front of them. Other than those commands, there is no need for privileged access to your machine, and in fact things may not work correctly. All the commands in this tutorial are run on your sysadmin machine. In order to complete the tutorial, the sysadmin will do the following: @@ -40,9 +40,9 @@ All the commands in this tutorial are run on your sysadmin machine. In order to * Install the LEAP command-line utility * Check out the LEAP platform * Create a provider and its certificates -* Setup the provider's nodes and the services that will reside on those nodes -* Initialize the nodes -* Deploy the LEAP platform to the nodes +* Setup the provider's node and the services that will reside on it +* Initialize the node +* Deploy the LEAP platform to the node * Test that things worked correctly * Some additional commands @@ -63,11 +63,10 @@ Install core prerequisites: $ sudo apt-get install git ruby ruby-dev rsync openssh-client openssl rake make bzip2 -<!-- *Mac OS* -1. Install rubygems from https://rubygems.org/pages/download (unless the `gem` command is already installed). ---> +Install rubygems from https://rubygems.org/pages/download (unless the `gem` command is already installed). + NOTE: leap_cli should work with ruby1.8, but has only been tested using ruby1.9. @@ -75,51 +74,23 @@ NOTE: leap_cli should work with ruby1.8, but has only been tested using ruby1.9. Install the LEAP command-line utility ------------------------------------------------- -Install the `leap` command from rubygems.org: +Install the LEAP command-line utility (leap_cli) from rubygems.org: $ sudo gem install leap_cli -Alternately, you can install `leap` from source: - - $ git clone https://leap.se/git/leap_cli - $ cd leap_cli - $ rake build - $ sudo rake install - -You can also install from source as an unprivileged user, if you want. For example, instead of `sudo rake install` you can do something like this: - - $ rake install - # watch out for the directory leap is installed to, then i.e. - $ sudo ln -s ~/.gem/ruby/1.9.1/bin/leap /usr/local/bin/leap +Alternately, you can install `leap_cli` from source, please refer to https://leap.se/git/leap_cli/README.md. -With either `rake install` or `sudo rake install`, you can use now /usr/local/bin/leap, which in most cases will be in your $PATH. - -If you have successfully installed the `leap` command, then you should be able to do the following: +If you have successfully installed `leap_cli`, then you should be able to do the following: $ leap --help This will list the command-line help options. If you receive an error when doing this, please read through the README.md in the `leap_cli` source to try and resolve any problems before going forwards. -Check out the platform --------------------------- - -The LEAP Platform is a series of puppet recipes and modules that will be used to configure your provider. You will need a local copy of the platform that will be used to setup your nodes and manage your services. To begin with, you will not need to modify the LEAP Platform. -Until we have a up to date stable release we recommend using the `develop` branch of the platform and leap_cli for all features of LEAP. - -First we'll create a directory for LEAP things, and then we'll check out the platform code and initalize the modules: - - $ mkdir ~/leap - $ cd ~/leap - $ git clone https://leap.se/git/leap_platform.git - $ cd leap_platform - $ git checkout develop - $ git submodule sync; git submodule update --init - Provider Setup ============== -A provider instance is a directory tree, usually stored in git, that contains everything you need to manage an infrastructure for a service provider. In this case, we create one for example.org and call the instance directory 'example'. +A provider instance is a directory tree that contains everything you need to manage an infrastructure for a service provider. In this case, we create one for example.org and call the instance directory 'example'. $ mkdir -p ~/leap/example @@ -139,21 +110,13 @@ The `leap new` command will ask you for several required values: * domain: The primary domain name of your service provider. In this tutorial, we will be using "example.org". * name: The name of your service provider (we use "Example"). * contact emails: A comma separated list of email addresses that should be used for important service provider contacts (for things like postmaster aliases, Tor contact emails, etc). -* platform: The directory where you have a copy of the `leap_platform` git repository checked out. - -You could also have passed these configuration options on the command-line, like so: - - $ leap new --contacts your@email.here --domain leap.example.org --name Example --platform=~/leap/leap_platform . - -You may want to poke around and see what is in the files we just created. For example: - - $ cat provider.json +* platform: The directory where you either have a copy of the `leap_platform` git repository already checked out, or where `leap_cli` should download it too. You could just accept the suggested path for this example. + The LEAP Platform is a series of puppet recipes and modules that will be used to configure your provider. You will need a local copy of the platform that will be used to setup your nodes and manage your services. To begin with, you will not need to modify the LEAP Platform. -Optionally, commit your provider directory using the version control software you fancy. For example: +These steps should be sufficient for this example. If you want to configure your provider further or like to examine the files, please refer to the [Configure Provider](configure-provider) section. - $ git init - $ git add . - $ git commit -m "initial provider commit" +Add Users who will have administrative access +--------------------------------------------- Now add yourself as a privileged sysadmin who will have access to deploy to servers: @@ -161,6 +124,7 @@ Now add yourself as a privileged sysadmin who will have access to deploy to serv NOTE: in most cases, `leap` must be run from within a provider instance directory tree (e.g. ~/leap/example). + Create provider certificates ---------------------------- @@ -173,43 +137,39 @@ Create a temporary cert for your main domain (you should replace with a real com $ leap cert csr -To see details about the keys and certs that the prior two commands created, you can use `leap inspect` like so: - $ leap inspect files/ca/ca.crt +Setup the provider's node and services +-------------------------------------- -NOTE: the files `files/ca/*.key` are extremely sensitive and must be carefully protected. The other key files are much less sensitive and can simply be regenerated if needed. +A "node" is a server that is part of your infrastructure. Every node can have one or more services associated with it. Some nodes are "local" and used only for testing, see [Development](development) for more information. +Create a node, with `all the services needed for Email: "couchdb", "mx", "soledad" and "webapp"` -Edit provider.json configuration --------------------------------------- + $ leap node add node1 ip_address:x.x.x.w services:couchdb,mx,soledad,webapp -There are a few required settings in provider.json. At a minimum, you must have: +NOTE: replace x.x.x.w with the actual IP address of this node - { - "domain": "example.org", - "name": "Example", - "contacts": { - "default": "email1@example.org" - } - } +This created a node configuration file in `nodes/node1.json`, but it did not do anything else. It also added the 'tag' called 'production' to this node. Tags allow us to conveniently group nodes together. When creating nodes, you should give them the tag 'production' if the node is to be used in your production infrastructure. -For a full list of possible settings, you can use `leap inspect` to see how provider.json is evaluated after including the inherited defaults: +Initialize the nodes +-------------------- - $ leap inspect provider.json +Node initialization only needs to be done once, but there is no harm in doing it multiple times: + $ leap node init node1 -Setup the provider's node and services --------------------------------------- +This will initialize the node "node1". When `leap node init` is run, you will be prompted to verify the fingerprint of the SSH host key and to provide the root password of the server. You should only need to do this once. -A "node" is a server that is part of your infrastructure. Every node can have one or more services associated with it. Some nodes are "local" and used only for testing, see [Development](development) for more information. -Create a node, with all the services needed for Email - "couchdb", "mx", "soledad" and "webapp": +Deploy the LEAP platform to the nodes +-------------------- - $ leap node add node1 ip_address:x.x.x.w services:couchdb,mx,soledad,webapp tags:production +Now you should deploy the platform recipes to the node. [Deployment can take a while to run](http://xkcd.com/303/), especially on the first run, as it needs to update the packages on the new machine. -NOTE: replace x.x.x.w with the actual IP address of this node + $ leap deploy + +Watch the output for any errors (in red), if everything worked fine, you should now have your first running node. If you do have errors, try doing the deploy again. -This created a node configuration file in `nodes/node1.json`, but it did not do anything else. It also added the 'tag' called 'production' to this node. Tags allow us to conveniently group nodes together. When creating nodes, you should give them the tag 'production' if the node is to be used in your production infrastructure. Setup DNS --------- @@ -232,27 +192,6 @@ If you cannot edit your DNS zone file, you can still test your provider by addin Please don't forget about these entries, they will override DNS queries if you setup your DNS later. -Initialize the nodes --------------------- - -Node initialization only needs to be done once, but there is no harm in doing it multiple times: - - $ leap node init production - -This will initialize the node with the tag "production". When `leap node init` is run, you will be prompted to verify the fingerprint of the SSH host key and to provide the root password of the server. You should only need to do this once. - - -Deploy the LEAP platform to the nodes --------------------- - -Now you should deploy the platform recipes to the nodes. [Deployment can take a while to run](http://xkcd.com/303/), especially on the first run, as it needs to update the packages on the new machine. - - $ leap deploy - -Watch the output for any errors (in red), if everything worked fine, you should now have your first running node. If you do have errors, try doing the deploy again. - -NOTE: the output from deploying can be quite busy, so we often do them each node one by one. - What is going on here? -------------------------------------------- @@ -283,7 +222,7 @@ Access the web application In order to connect to the web application in your browser, you need to point your domain at the IP address of your new node. -Next, you can connect to the web application either using a web browser or via the API using the LEAP client. To use a browser, connect to https://leap.example.org (replacing that with your domain). Your browser will complain about an untrusted cert, but for now just bypass this. From there, you should be able to register a new user and login. +Next, you can connect to the web application either using a web browser or via the API using the LEAP client. To use a browser, connect to https://example.org (replacing that with your domain). Your browser will complain about an untrusted cert, but for now just bypass this. From there, you should be able to register a new user and login. Testing with leap_cli --------------------- diff --git a/platform.rb b/platform.rb index c37b6d29..bb77b0d9 100644 --- a/platform.rb +++ b/platform.rb @@ -4,8 +4,8 @@ # Leap::Platform.define do - self.version = "0.6" - self.compatible_cli = "1.6.1".."1.99" + self.version = "0.7" + self.compatible_cli = "1.7.0".."1.7.99" # # the facter facts that should be gathered @@ -52,10 +52,12 @@ Leap::Platform.define do :soledad_service_json_template => 'files/service-definitions/#{arg}/soledad-service.json.erb', :smtp_service_json_template => 'files/service-definitions/#{arg}/smtp-service.json.erb', - # custom puppet + # custom files :custom_puppet_dir => 'files/puppet', :custom_puppet_modules_dir => 'files/puppet/modules', :custom_puppet_manifests_dir => 'files/puppet/manifests', + :custom_tests => 'files/tests', + :custom_bin => 'files/bin', # output files :facts => 'facts.json', diff --git a/provider_base/common.json b/provider_base/common.json index 649db0d9..c7be5cf4 100644 --- a/provider_base/common.json +++ b/provider_base/common.json @@ -29,8 +29,8 @@ "cert": "= x509.use ? file(:node_x509_cert, :missing => 'x509 certificate for node $node. Run `leap cert update`') : nil", "key": "= x509.use ? file(:node_x509_key, :missing => 'x509 key for node $node. Run `leap cert update`') : nil", "ca_cert": "= try_file :ca_cert", - "commercial_cert": "= x509.use_commercial ? file([:commercial_cert, try{webapp.domain}||domain.full_suffix], :missing => 'commercial x509 certificate for node $node. Add file $file, or run `leap cert csr` to generate a temporary self-signed cert and CSR you can use to purchase a real cert.') : nil", - "commercial_key": "= x509.use_commercial ? file([:commercial_key, try{webapp.domain}||domain.full_suffix], :missing => 'commercial x509 certificate for node $node. Add file $file, or run `leap cert csr` to generate a temporary self-signed cert and CSR you can use to purchase a real cert.') : nil", + "commercial_cert": "= x509.use_commercial ? file([:commercial_cert, try{webapp.domain}||domain.full_suffix], :missing => 'commercial x509 certificate for node $node. Add file $file, or run `leap cert csr --domain %s` to generate a temporary self-signed cert and CSR you can use to purchase a real cert.' % (try{webapp.domain}||domain.full_suffix)) : nil", + "commercial_key": "= x509.use_commercial ? file([:commercial_key, try{webapp.domain}||domain.full_suffix], :missing => 'commercial x509 certificate for node $node. Add file $file, or run `leap cert csr --domain %s` to generate a temporary self-signed cert and CSR you can use to purchase a real cert.' % (try{webapp.domain}||domain.full_suffix)) : nil", "commercial_ca_cert": "= x509.use_commercial ? try_file(:commercial_ca_cert) : nil" }, "service_type": "internal_service", @@ -50,5 +50,37 @@ "platform": { "version": "= Leap::Platform.version.to_s", "major_version": "= Leap::Platform.major_version" + }, + "sources": { + "apt": { + "basic": "http://httpredir.debian.org/debian/", + "security": "http://security.debian.org/", + "backports": "http://httpredir.debian.org/debian/" + }, + "leap-mx": { + "type": "apt", + "package": "leap-mx", + "revision": "latest" + }, + "nickserver": { + "type": "git", + "source": "https://leap.se/git/nickserver", + "revision": "origin/master" + }, + "soledad": { + "type": "apt", + "package": "soledad-server", + "revision": "latest" + }, + "tapicero": { + "type": "git", + "source": "https://leap.se/git/tapicero", + "revision": "origin/version/0.7" + }, + "webapp": { + "type": "git", + "source": "https://leap.se/git/leap_web", + "revision": "origin/version/0.7" + } } } diff --git a/provider_base/files/service-definitions/provider.json.erb b/provider_base/files/service-definitions/provider.json.erb index 2d0a5886..be8ae484 100644 --- a/provider_base/files/service-definitions/provider.json.erb +++ b/provider_base/files/service-definitions/provider.json.erb @@ -7,7 +7,7 @@ hsh['domain'] = domain.full_suffix # advertise services that are 'user services' and for which there are actually nodes - hsh['services'] ||= global.env(environment).services[:service_type => :user_service].field(:name).select do |service| + hsh['services'] ||= global.services[:service_type => :user_service].field(:name).select do |service| nodes_like_me[:services => service].any? end diff --git a/provider_base/lib/macros/nodes.rb b/provider_base/lib/macros/nodes.rb index 0c6668a0..8b961cbc 100644 --- a/provider_base/lib/macros/nodes.rb +++ b/provider_base/lib/macros/nodes.rb @@ -15,10 +15,10 @@ module LeapCli end # - # grab an environment appropriate provider + # simple alias for global.provider # def provider - global.env(@node.environment).provider + global.provider end # diff --git a/provider_base/lib/macros/secrets.rb b/provider_base/lib/macros/secrets.rb index 51bf3971..8d1feb55 100644 --- a/provider_base/lib/macros/secrets.rb +++ b/provider_base/lib/macros/secrets.rb @@ -13,17 +13,17 @@ module LeapCli # +length+ is the character length of the generated password. # def secret(name, length=32) - @manager.secrets.set(name, Util::Secret.generate(length), @node[:environment]) + manager.secrets.set(name, @node.environment) { Util::Secret.generate(length) } end # inserts a base32 encoded secret def base32_secret(name, length=20) - @manager.secrets.set(name, Base32.encode(Util::Secret.generate(length)), @node[:environment]) + manager.secrets.set(name, @node.environment) { Base32.encode(Util::Secret.generate(length)) } end # Picks a random obfsproxy port from given range def rand_range(name, range) - @manager.secrets.set(name, rand(range), @node[:environment]) + manager.secrets.set(name, @node.environment) { rand(range) } end # @@ -32,7 +32,7 @@ module LeapCli # +bit_length+ is the bits in the secret, (ie length of resulting hex string will be bit_length/4) # def hex_secret(name, bit_length=128) - @manager.secrets.set(name, Util::Secret.generate_hex(bit_length), @node[:environment]) + manager.secrets.set(name, @node.environment) { Util::Secret.generate_hex(bit_length) } end end diff --git a/provider_base/provider.json b/provider_base/provider.json index 77437935..60ad2a9e 100644 --- a/provider_base/provider.json +++ b/provider_base/provider.json @@ -42,22 +42,22 @@ "organizational_unit": "= 'https://' + provider.domain", "bit_size": 4096, "digest": "SHA256", - "life_span": "10y", + "life_span": "10 years", "server_certificates": { "bit_size": 4096, "digest": "SHA256", - "life_span": "1y" + "life_span": "1 years" }, "client_certificates": { "bit_size": 2048, "digest": "SHA256", - "life_span": "2m", + "life_span": "2 months", "limited_prefix": "LIMITED", "unlimited_prefix": "UNLIMITED" } }, "client_version": { - "min": "0.5", + "min": "0.7", "max": null } } diff --git a/provider_base/services/webapp.json b/provider_base/services/webapp.json index 67744f99..941f4f61 100644 --- a/provider_base/services/webapp.json +++ b/provider_base/services/webapp.json @@ -1,14 +1,18 @@ { "webapp": { "admins": [], - "forbidden_usernames": ["admin", "administrator", "arin-admin", "certmaster", "contact", "info", "maildrop", "postmaster", "ssladmin", "www-data"], + "forbidden_usernames": [ + "admin", "admins", "administrator", "administrators", "arin-admin", + "certmaster", "contact", "email", "help", "help-desk", "help-ticket", + "help-tickets", "help_desk", "help_ticket", "help_tickets", "helpdesk", + "helpticket", "helptickets", "info", "mail", "maildrop", "noreply", + "owner", "owners", "postmaster", "reply", "robot", "ssladmin", "staff", + "support", "tech-support", "tech_support", "techsupport", "ticket", + "tickets", "vmail", "www-data"], "domain": "= domain.full_suffix", "modules": ["user", "billing", "help"], - "couchdb_webapp_user": { - "username": "= global.services[:couchdb].couch.users[:webapp].username", - "password": "= secret :couch_webapp_password", - "salt": "= hex_secret :couch_webapp_password_salt, 128" - }, + "couchdb_webapp_user": "= global.services[:couchdb].couch.users[:webapp]", + "couchdb_admin_user": "= global.services[:couchdb].couch.users[:admin]", "customization_dir": "= file_path 'webapp'", "client_certificates": "= provider.ca.client_certificates", "allow_limited_certs": "= provider.service.allow_limited_bandwidth", @@ -20,10 +24,6 @@ "secret_token": "= secret :webapp_secret_token", "api_version": 1, "secure": false, - "git": { - "source": "https://leap.se/git/leap_web", - "revision": "origin/version/0.6" - }, "client_version": "= provider.client_version", "nagios_test_user": { "username": "nagios_test", diff --git a/puppet/lib/puppet/parser/functions/sorted_json.rb b/puppet/lib/puppet/parser/functions/sorted_json.rb new file mode 100644 index 00000000..605da00e --- /dev/null +++ b/puppet/lib/puppet/parser/functions/sorted_json.rb @@ -0,0 +1,47 @@ +# +# Written by Gavin Mogan, from https://gist.github.com/halkeye/2287885 +# Put in the public domain by the author. +# + +require 'json' + +def sorted_json(obj) + case obj + when String, Fixnum, Float, TrueClass, FalseClass, NilClass + return obj.to_json + when Array + arrayRet = [] + obj.each do |a| + arrayRet.push(sorted_json(a)) + end + return "[" << arrayRet.join(',') << "]"; + when Hash + ret = [] + obj.keys.sort.each do |k| + ret.push(k.to_json << ":" << sorted_json(obj[k])) + end + return "{" << ret.join(",") << "}"; + else + raise Exception("Unable to handle object of type <%s>" % obj.class.to_s) + end +end + +module Puppet::Parser::Functions + newfunction(:sorted_json, :type => :rvalue, :doc => <<-EOS +This function takes data, outputs making sure the hash keys are sorted + +*Examples:* + + sorted_json({'key'=>'value'}) + +Would return: {'key':'value'} + EOS + ) do |arguments| + raise(Puppet::ParseError, "sorted_json(): Wrong number of arguments " + + "given (#{arguments.size} for 1)") if arguments.size != 1 + + json = arguments[0] + return sorted_json(json) + end +end + diff --git a/puppet/manifests/setup.pp b/puppet/manifests/setup.pp deleted file mode 100644 index 4dd03203..00000000 --- a/puppet/manifests/setup.pp +++ /dev/null @@ -1,5 +0,0 @@ -# -# this is applied before each run of site.pp -# - -include ::site_config::setup diff --git a/puppet/manifests/site.pp b/puppet/manifests/site.pp index 57942d99..912234ac 100644 --- a/puppet/manifests/site.pp +++ b/puppet/manifests/site.pp @@ -1,5 +1,10 @@ # set a default exec path -Exec { path => '/usr/bin:/usr/sbin/:/bin:/sbin:/usr/local/bin:/usr/local/sbin' } +# the logoutput exec parameter defaults to "on_error" in puppet 3, +# but to "false" in puppet 2.7, so we need to set this globally here +Exec { + logoutput => on_failure, + path => '/usr/bin:/usr/sbin/:/bin:/sbin:/usr/local/bin:/usr/local/sbin' +} include site_config::setup include site_config::default diff --git a/puppet/modules/apt b/puppet/modules/apt -Subproject 64fb988c0e37d64fb3e241dc95f156072e43bf2 +Subproject fca103484ddc1f647a54135b6a902edabf45955 diff --git a/puppet/modules/augeas b/puppet/modules/augeas -Subproject 4d8c8ba362cc57c12451e581f27feea97797e8c +Subproject 58ab2b90c52a5d951fa41596827bc3b6f52310e diff --git a/puppet/modules/common b/puppet/modules/common -Subproject 0961ad453b8befb4ea61bbd19f6ecea32b9619c +Subproject ae149624f9bc551865b93b9b7155af2de8deeb7 diff --git a/puppet/modules/couchdb b/puppet/modules/couchdb -Subproject 4c0d5673df02fe42e1bbadfee7d4ea1ca1f88e9 +Subproject 23b557c6fb07929a9b04e5fb75375a85a473437 diff --git a/puppet/modules/haveged/manifests/init.pp b/puppet/modules/haveged/manifests/init.pp new file mode 100644 index 00000000..8f901937 --- /dev/null +++ b/puppet/modules/haveged/manifests/init.pp @@ -0,0 +1,16 @@ +class haveged { + + package { 'haveged': + ensure => present, + } + + service { 'haveged': + ensure => running, + hasrestart => true, + hasstatus => true, + enable => true, + require => Package['haveged']; + } + + include site_check_mk::agent::haveged +} diff --git a/puppet/modules/leap/manifests/init.pp b/puppet/modules/leap/manifests/init.pp new file mode 100644 index 00000000..bbae3781 --- /dev/null +++ b/puppet/modules/leap/manifests/init.pp @@ -0,0 +1,3 @@ +class leap { + +}
\ No newline at end of file diff --git a/puppet/modules/leap/manifests/logfile.pp b/puppet/modules/leap/manifests/logfile.pp new file mode 100644 index 00000000..63dbd16b --- /dev/null +++ b/puppet/modules/leap/manifests/logfile.pp @@ -0,0 +1,25 @@ +# +# make syslog log to a particular file for a particular process. +# + +define leap::logfile($process=$name) { + $logfile = "/var/log/leap/${name}.log" + + rsyslog::snippet { "50-${name}": + content => template('leap/rsyslog.erb') + } + + augeas { + "logrotate_${name}": + context => "/files/etc/logrotate.d/${name}/rule", + changes => [ + "set file ${logfile}", + 'set rotate 5', + 'set schedule daily', + 'set compress compress', + 'set missingok missingok', + 'set ifempty notifempty', + 'set copytruncate copytruncate' + ] + } +} diff --git a/puppet/modules/leap/templates/rsyslog.erb b/puppet/modules/leap/templates/rsyslog.erb new file mode 100644 index 00000000..7bb5316f --- /dev/null +++ b/puppet/modules/leap/templates/rsyslog.erb @@ -0,0 +1,5 @@ +if $programname startswith '<%= @process %>' then { + action(type="omfile" file="<%= @logfile %>" template="RSYSLOG_TraditionalFileFormat") + stop +} + diff --git a/puppet/modules/leap_mx/manifests/init.pp b/puppet/modules/leap_mx/manifests/init.pp index c90fc231..6bcdd19a 100644 --- a/puppet/modules/leap_mx/manifests/init.pp +++ b/puppet/modules/leap_mx/manifests/init.pp @@ -7,6 +7,8 @@ class leap_mx { $couchdb_host = 'localhost' $couchdb_port = '4096' + $sources = hiera('sources') + include soledad::common include site_apt::preferences::twisted @@ -39,16 +41,26 @@ class leap_mx { notify => Service['leap-mx']; } + file { '/etc/default/leap_mx': + content => 'LOGFILE=/var/log/leap/mx.log', + owner => 'root', + group => 'root', + mode => '0644', + notify => Service['leap-mx']; + } + # # LEAP-MX CODE AND DEPENDENCIES # package { - 'leap-mx': - ensure => latest, - require => Class['site_apt::preferences::twisted']; + $sources['leap-mx']['package']: + ensure => $sources['leap-mx']['revision'], + require => [ + Class['site_apt::preferences::twisted'], + Class['site_apt::leap_repo'] ]; - [ 'leap-keymanager' ]: + 'leap-keymanager': ensure => latest; } @@ -63,4 +75,18 @@ class leap_mx { hasrestart => true, require => [ Package['leap-mx'] ]; } + + augeas { + "logrotate_mx": + context => "/files/etc/logrotate.d/leap-mx/rule", + changes => [ + "set file /var/log/leap/mx.log", + 'set rotate 5', + 'set schedule daily', + 'set compress compress', + 'set missingok missingok', + 'set ifempty notifempty', + 'set copytruncate copytruncate' + ] + } } diff --git a/puppet/modules/nagios b/puppet/modules/nagios -Subproject 57a1140b437a8cfb9cfd5d94a5759b1e3ed86d4 +Subproject b55f23d4d90c97cec08251544aa9700df86ad0b diff --git a/puppet/modules/postfix b/puppet/modules/postfix -Subproject 1103a73ab4253712c6446bba7a443619fe51671 +Subproject f09cd0eff2bcab7e12c09ec67be3c918bc83fac diff --git a/puppet/modules/site_apache/files/include.d/ssl_common.inc b/puppet/modules/site_apache/files/include.d/ssl_common.inc new file mode 100644 index 00000000..2d282c84 --- /dev/null +++ b/puppet/modules/site_apache/files/include.d/ssl_common.inc @@ -0,0 +1,7 @@ +SSLEngine on +SSLProtocol all -SSLv2 -SSLv3 +SSLHonorCipherOrder on +SSLCompression off +SSLCipherSuite "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!RC4:!MD5:!PSK!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA" + +RequestHeader set X_FORWARDED_PROTO 'https'
\ No newline at end of file diff --git a/puppet/modules/site_apache/manifests/common.pp b/puppet/modules/site_apache/manifests/common.pp index 72f24838..2b83ffa5 100644 --- a/puppet/modules/site_apache/manifests/common.pp +++ b/puppet/modules/site_apache/manifests/common.pp @@ -23,4 +23,5 @@ class site_apache::common { content => template('site_apache/vhosts.d/common.conf.erb') } + apache::config::include{ 'ssl_common.inc': } } diff --git a/puppet/modules/site_apache/templates/vhosts.d/api.conf.erb b/puppet/modules/site_apache/templates/vhosts.d/api.conf.erb index e4732289..0396f54b 100644 --- a/puppet/modules/site_apache/templates/vhosts.d/api.conf.erb +++ b/puppet/modules/site_apache/templates/vhosts.d/api.conf.erb @@ -11,18 +11,12 @@ Listen 0.0.0.0:<%= api_port %> ServerName <%= api_domain %> CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log common - SSLEngine on - SSLProtocol all -SSLv2 -SSLv3 - SSLHonorCipherOrder on - SSLCompression off - SSLCipherSuite "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK" - SSLCACertificatePath /etc/ssl/certs SSLCertificateChainFile <%= scope.lookupvar('x509::variables::local_CAs') %>/<%= scope.lookupvar('site_config::params::ca_name') %>.crt SSLCertificateKeyFile <%= scope.lookupvar('x509::variables::keys') %>/<%= scope.lookupvar('site_config::params::cert_name') %>.key SSLCertificateFile <%= scope.lookupvar('x509::variables::certs') %>/<%= scope.lookupvar('site_config::params::cert_name') %>.crt - RequestHeader set X_FORWARDED_PROTO 'https' + Include include.d/ssl_common.inc <IfModule mod_headers.c> <% if @webapp['secure'] -%> diff --git a/puppet/modules/site_apache/templates/vhosts.d/common.conf.erb b/puppet/modules/site_apache/templates/vhosts.d/common.conf.erb index a9733a97..ee5cd707 100644 --- a/puppet/modules/site_apache/templates/vhosts.d/common.conf.erb +++ b/puppet/modules/site_apache/templates/vhosts.d/common.conf.erb @@ -1,5 +1,7 @@ <VirtualHost *:80> - ServerName <%= domain %> + ServerName <%= webapp_domain %> + ServerAlias <%= domain_name %> + ServerAlias <%= domain %> ServerAlias www.<%= domain %> RewriteEngine On RewriteRule ^.*$ https://<%= domain -%>%{REQUEST_URI} [R=permanent,L] @@ -7,23 +9,18 @@ </VirtualHost> <VirtualHost *:443> - ServerName <%= domain_name %> + ServerName <%= webapp_domain %> + ServerAlias <%= domain_name %> ServerAlias <%= domain %> ServerAlias www.<%= domain %> CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log common - SSLEngine on - SSLProtocol all -SSLv2 -SSLv3 - SSLHonorCipherOrder on - SSLCompression off - SSLCipherSuite "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK" - SSLCACertificatePath /etc/ssl/certs SSLCertificateChainFile <%= scope.lookupvar('x509::variables::local_CAs') %>/<%= scope.lookupvar('site_config::params::commercial_ca_name') %>.crt SSLCertificateKeyFile <%= scope.lookupvar('x509::variables::keys') %>/<%= scope.lookupvar('site_config::params::commercial_cert_name') %>.key SSLCertificateFile <%= scope.lookupvar('x509::variables::certs') %>/<%= scope.lookupvar('site_config::params::commercial_cert_name') %>.crt - RequestHeader set X_FORWARDED_PROTO 'https' + Include include.d/ssl_common.inc <IfModule mod_headers.c> <% if (defined? @services) and (@services.include? 'webapp') and (@webapp['secure']) -%> @@ -60,7 +57,7 @@ <% if (defined? @services) and (@services.include? 'monitor') -%> - <DirectoryMatch (/usr/share/nagios3/htdocs|/usr/lib/cgi-bin/nagios3|/etc/nagios3/stylesheets)> + <DirectoryMatch (/usr/share/nagios3/htdocs|/usr/lib/cgi-bin/nagios3|/etc/nagios3/stylesheets|/usr/share/pnp4nagios)> <% if (defined? @services) and (@services.include? 'webapp') -%> PassengerEnabled off <% end -%> diff --git a/puppet/modules/site_apt/manifests/init.pp b/puppet/modules/site_apt/manifests/init.pp index 633ccf1e..cf49f870 100644 --- a/puppet/modules/site_apt/manifests/init.pp +++ b/puppet/modules/site_apt/manifests/init.pp @@ -1,7 +1,17 @@ +# setup apt on all nodes class site_apt { + $sources = hiera('sources') + $apt_config = $sources['apt'] + $apt_url_basic = $apt_config['basic'] + $apt_url_security = $apt_config['security'] + $apt_url_backports = $apt_config['backports'] + class { 'apt': - custom_key_dir => 'puppet:///modules/site_apt/keys' + custom_key_dir => 'puppet:///modules/site_apt/keys', + debian_url => $apt_url_basic, + security_url => $apt_url_security, + backports_url => $apt_url_backports } # enable http://deb.leap.se debian package repository @@ -22,12 +32,19 @@ class site_apt { priority => 999 } + apt::preferences_snippet { 'leap': + priority => 999, + package => '*', + pin => 'origin "deb.leap.se"' + } + # All packages should be installed _after_ refresh_apt is called, # which does an apt-get update. # There is one exception: # The creation of sources.list depends on the lsb package File['/etc/apt/preferences'] -> + Apt::Preferences_snippet <| |> -> Exec['refresh_apt'] -> - Package <| ( title != 'lsb' ) |> + Package <| ( title != 'lsb' ) |> } diff --git a/puppet/modules/site_apt/manifests/unattended_upgrades.pp b/puppet/modules/site_apt/manifests/unattended_upgrades.pp index daebffab..40111deb 100644 --- a/puppet/modules/site_apt/manifests/unattended_upgrades.pp +++ b/puppet/modules/site_apt/manifests/unattended_upgrades.pp @@ -1,10 +1,9 @@ -class site_apt::unattended_upgrades inherits apt::unattended_upgrades { +class site_apt::unattended_upgrades { # override unattended-upgrades package resource to make sure # that it is upgraded on every deploy (#6245) - include ::apt::unattended_upgrades - - Package['unattended-upgrades'] { - ensure => latest + class { 'apt::unattended_upgrades': + config_content => template('site_apt/50unattended-upgrades'), + ensure_version => latest } } diff --git a/puppet/modules/site_apt/files/Debian/50unattended-upgrades b/puppet/modules/site_apt/templates/50unattended-upgrades index f2f574fc..9ae3ab84 100644 --- a/puppet/modules/site_apt/files/Debian/50unattended-upgrades +++ b/puppet/modules/site_apt/templates/50unattended-upgrades @@ -1,7 +1,7 @@ // this file is managed by puppet ! Unattended-Upgrade::Allowed-Origins { - "${distro_id}:stable"; + "${distro_id}:oldstable"; "${distro_id}:${distro_codename}-security"; "${distro_id}:${distro_codename}-updates"; "${distro_id} Backports:${distro_codename}-backports"; diff --git a/puppet/modules/site_check_mk/files/agent/local_checks/couchdb/leap_couch_stats.sh b/puppet/modules/site_check_mk/files/agent/local_checks/couchdb/leap_couch_stats.sh new file mode 100755 index 00000000..95474ccb --- /dev/null +++ b/puppet/modules/site_check_mk/files/agent/local_checks/couchdb/leap_couch_stats.sh @@ -0,0 +1,119 @@ +#!/bin/bash +# +# todo: +# - thresholds +# - couch response time +# - make CURL/URL/DBLIST_EXCLUDE vars configurable +# - move load_nagios_utils() to helper library so we can use it from multiple scripts + +start_time=$(date +%s.%N) + +CURL='curl -s --netrc-file /etc/couchdb/couchdb.netrc' +URL='http://127.0.0.1:5984' +TMPFILE=$(mktemp) +DBLIST_EXCLUDE='(user-|sessions_|tokens_)' +PREFIX='Couchdb_' + + +load_nagios_utils () { + # load the nagios utils + # in debian, the package nagios-plugins-common installs utils.sh to /usr/lib/nagios/plugins/utils.sh + utilsfn= + for d in $PROGPATH /usr/lib/nagios/plugins /usr/lib64/nagios/plugins /usr/local/nagios/libexec /opt/nagios-plugins/libexec . ; do + if [ -f "$d/utils.sh" ]; then + utilsfn=$d/utils.sh; + fi + done + if [ "$utilsfn" = "" ]; then + echo "UNKNOWN - cannot find utils.sh (part of nagios plugins)"; + exit 3; + fi + . "$utilsfn"; + STATE[$STATE_OK]='OK' + STATE[$STATE_WARNING]='Warning' + STATE[$STATE_CRITICAL]='Critical' + STATE[$STATE_UNKNOWN]='Unknown' + STATE[$STATE_DEPENDENT]='Dependend' +} + +get_global_stats_perf () { + trap "localexit=3" ERR + local localexit db_count + localexit=0 + + # get a list of all dbs + $CURL -X GET $URL/_all_dbs | json_pp | egrep -v '(\[|\])' > $TMPFILE + + db_count=$( wc -l < $TMPFILE) + excluded_db_count=$( egrep -c "$DBLIST_EXCLUDE" $TMPFILE ) + + echo "db_count=$db_count|excluded_db_count=$excluded_db_count" + return ${localexit} +} + +db_stats () { + trap "localexit=3" ERR + local db db_stats doc_count del_doc_count localexit + localexit=0 + + db="$1" + name="$2" + + if [ -z "$name" ] + then + name="$db" + fi + + perf="$perf|${db}_docs=$( $CURL -s -X GET ${URL}/$db | json_pp |grep 'doc_count' | sed 's/[^0-9]//g' )" + db_stats=$( $CURL -s -X GET ${URL}/$db | json_pp ) + + doc_count=$( echo "$db_stats" | grep 'doc_count' | grep -v 'deleted_doc_count' | sed 's/[^0-9]//g' ) + del_doc_count=$( echo "$db_stats" | grep 'doc_del_count' | sed 's/[^0-9]//g' ) + + # don't divide by zero + if [ $del_doc_count -eq 0 ] + then + del_doc_perc=0 + else + del_doc_perc=$(( del_doc_count * 100 / doc_count )) + fi + + bytes=$( echo "$db_stats" | grep disk_size | sed 's/[^0-9]//g' ) + disk_size=$( echo "scale = 2; $bytes / 1024 / 1024" | bc -l ) + + echo -n "${localexit} ${PREFIX}${name}_database ${name}_docs=$doc_count|${name}_deleted_docs=$del_doc_count|${name}_deleted_docs_percentage=${del_doc_perc}%" + printf "|${name}_disksize_mb=%02.2fmb ${STATE[localexit]}: database $name\n" "$disk_size" + + return ${localexit} +} + +# main + +load_nagios_utils + +# per-db stats +# get a list of all dbs +$CURL -X GET $URL/_all_dbs | json_pp | egrep -v '(\[|\])' > $TMPFILE + +# get list of dbs to check +dbs=$( egrep -v "${DBLIST_EXCLUDE}" $TMPFILE | tr -d '\n"' | sed 's/,/ /g' ) + +for db in $dbs +do + db_stats "$db" +done + +# special handling for rotated dbs +suffix=$(($(date +'%s') / (60*60*24*30) + 1)) +db_stats "sessions_${suffix}" "sessions" +db_stats "tokens_${suffix}" "tokens" + + +# show global couchdb stats +global_stats_perf=$(get_global_stats_perf) +exitcode=$? + +end_time=$(date +%s.%N) +duration=$( echo "scale = 2; $end_time - $start_time" | bc -l ) + +printf "${exitcode} ${PREFIX}global_stats ${global_stats_perf}|script_duration=%02.2fs ${STATE[exitcode]}: global couchdb status\n" "$duration" diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/leap_mx.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/leap_mx.cfg index c71c5392..166d0230 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/leap_mx.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/leap_mx.cfg @@ -1,4 +1,4 @@ -/var/log/leap_mx.log +/var/log/leap/mx.log W Don't know how to deliver mail W No public key, stopping the processing chain diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/openvpn.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/openvpn.cfg index ac17c0ca..ed50f420 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/openvpn.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/openvpn.cfg @@ -1,3 +1,4 @@ +/var/log/leap/openvpn.log # ignore openvpn TLS initialization errors when clients # suddenly hangup before properly establishing # a tls connection diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/stunnel.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/stunnel.cfg index eb3131f2..b1e6cf2f 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/stunnel.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/stunnel.cfg @@ -1,3 +1,4 @@ +/var/log/leap/stunnel.log # check for stunnel failures # # these are temporary failures and happen very often, so we diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/tapicero.cfg index e5721eea..d98f5094 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/tapicero.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/tapicero.cfg @@ -1,5 +1,6 @@ +/var/log/leap/tapicero.log # Ignore transient Tapicero errors when creating a db (#6511) - I tapicero.*(Creating database|Checking security of|Writing security to|Uploading design doc to) user-.* failed (\(trying again soon\)|(twice )?due to): (RestClient::Resource Not Found|RestClient::InternalServerError): (404 Resource Not Found|500 Internal Server Error) + I tapicero.*(Creating database|Checking security of|Writing security to|Uploading design doc to) user-.* failed (\(trying again soon\)|(twice )?due to): (RestClient::ResourceNotFound|RestClient::InternalServerError): (404 Resource Not Found|500 Internal Server Error) C tapicero.*RestClient::InternalServerError: # possible race condition between multiple tapicero # instances, so we ignore it diff --git a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/webapp.cfg b/puppet/modules/site_check_mk/files/agent/logwatch/webapp.cfg index 00f9c7fd..008e9e09 100644 --- a/puppet/modules/site_check_mk/files/agent/logwatch/syslog/webapp.cfg +++ b/puppet/modules/site_check_mk/files/agent/logwatch/webapp.cfg @@ -1,3 +1,4 @@ +/var/log/leap/webapp.log # check for webapp errors C webapp.*Could not connect to couch database messages due to 401 Unauthorized: {"error":"unauthorized","reason":"You are not a server admin."} # ignore RoutingErrors that rails throw when it can't handle a url diff --git a/puppet/modules/site_check_mk/files/extra_service_conf.mk b/puppet/modules/site_check_mk/files/extra_service_conf.mk index 03d1ea76..c7120a96 100644 --- a/puppet/modules/site_check_mk/files/extra_service_conf.mk +++ b/puppet/modules/site_check_mk/files/extra_service_conf.mk @@ -1,13 +1,14 @@ # retry 3 times before setting a service into a hard state # and send out notification -extra_service_conf["max_check_attempts"] = [ - ("4", ALL_HOSTS , ALL_SERVICES ) +extra_service_conf["max_check_attempts"] = [ + ("4", ALL_HOSTS , ALL_SERVICES ) ] -# run check_mk_agent every 2 minutes if it terminates -# successfully. +# +# run check_mk_agent every 4 minutes if it terminates successfully. # see https://leap.se/code/issues/6539 for the rationale +# extra_service_conf["normal_check_interval"] = [ - ("2", ALL_HOSTS , "Check_MK" ) + ("4", ALL_HOSTS , "Check_MK" ) ] diff --git a/puppet/modules/site_check_mk/manifests/agent/couchdb.pp b/puppet/modules/site_check_mk/manifests/agent/couchdb.pp index ee0268a3..abfc7ad0 100644 --- a/puppet/modules/site_check_mk/manifests/agent/couchdb.pp +++ b/puppet/modules/site_check_mk/manifests/agent/couchdb.pp @@ -12,13 +12,21 @@ class site_check_mk::agent::couchdb { # check bigcouch processes - file_line { + augeas { 'Bigcouch_epmd_procs': - line => 'Bigcouch_epmd_procs /usr/lib/nagios/plugins/check_procs -w 1:1 -c 1:1 -a /opt/bigcouch/erts-5.9.1/bin/epmd', - path => '/etc/check_mk/mrpe.cfg'; + incl => '/etc/check_mk/mrpe.cfg', + lens => 'Spacevars.lns', + changes => [ + 'rm /files/etc/check_mk/mrpe.cfg/Bigcouch_epmd_procs', + 'set Bigcouch_epmd_procs \'/usr/lib/nagios/plugins/check_procs -w 1:1 -c 1:1 -a /opt/bigcouch/erts-5.9.1/bin/epmd\'' ], + require => File['/etc/check_mk/mrpe.cfg']; 'Bigcouch_beam_procs': - line => 'Bigcouch_beam_procs /usr/lib/nagios/plugins/check_procs -w 1:1 -c 1:1 -a /opt/bigcouch/erts-5.9.1/bin/beam', - path => '/etc/check_mk/mrpe.cfg'; + incl => '/etc/check_mk/mrpe.cfg', + lens => 'Spacevars.lns', + changes => [ + 'rm /files/etc/check_mk/mrpe.cfg/Bigcouch_beam_procs', + 'set Bigcouch_beam_procs \'/usr/lib/nagios/plugins/check_procs -w 1:1 -c 1:1 -a /opt/bigcouch/erts-5.9.1/bin/beam\'' ], + require => File['/etc/check_mk/mrpe.cfg']; } # check open files for bigcouch proc @@ -27,10 +35,21 @@ class site_check_mk::agent::couchdb { source => 'puppet:///modules/site_check_mk/agent/nagios_plugins/check_unix_open_fds.pl', mode => '0755' } - file_line { + augeas { 'Bigcouch_open_files': - line => 'Bigcouch_open_files /srv/leap/nagios/plugins/check_unix_open_fds.pl -a beam -w 28672,28672 -c 30720,30720', - path => '/etc/check_mk/mrpe.cfg'; + incl => '/etc/check_mk/mrpe.cfg', + lens => 'Spacevars.lns', + changes => [ + 'rm /files/etc/check_mk/mrpe.cfg/Bigcouch_open_files', + 'set Bigcouch_open_files \'/srv/leap/nagios/plugins/check_unix_open_fds.pl -a beam -w 28672,28672 -c 30720,30720\'' ], + require => File['/etc/check_mk/mrpe.cfg']; } + + # check different couchdb stats + file { '/usr/lib/check_mk_agent/local/leap_couch_stats.sh': + source => 'puppet:///modules/site_check_mk/agent/local_checks/couchdb/leap_couch_stats.sh', + mode => '0755', + require => Package['check_mk-agent'] + } } diff --git a/puppet/modules/site_check_mk/manifests/agent/haproxy.pp b/puppet/modules/site_check_mk/manifests/agent/haproxy.pp index e7986db1..6d52efba 100644 --- a/puppet/modules/site_check_mk/manifests/agent/haproxy.pp +++ b/puppet/modules/site_check_mk/manifests/agent/haproxy.pp @@ -3,10 +3,13 @@ class site_check_mk::agent::haproxy { include site_check_mk::agent::package::nagios_plugins_contrib # local nagios plugin checks via mrpe - file_line { - 'haproxy': - line => 'Haproxy /usr/lib/nagios/plugins/check_haproxy -u "http://localhost:8000/haproxy;csv"', - path => '/etc/check_mk/mrpe.cfg'; + augeas { 'haproxy': + incl => '/etc/check_mk/mrpe.cfg', + lens => 'Spacevars.lns', + changes => [ + 'rm /files/etc/check_mk/mrpe.cfg/Haproxy', + 'set Haproxy \'/usr/lib/nagios/plugins/check_haproxy -u "http://localhost:8000/haproxy;csv"\'' ], + require => File['/etc/check_mk/mrpe.cfg']; } } diff --git a/puppet/modules/site_check_mk/manifests/agent/haveged.pp b/puppet/modules/site_check_mk/manifests/agent/haveged.pp new file mode 100644 index 00000000..cacbea8c --- /dev/null +++ b/puppet/modules/site_check_mk/manifests/agent/haveged.pp @@ -0,0 +1,15 @@ +class site_check_mk::agent::haveged { + +# check haveged process + augeas { + 'haveged_proc': + incl => '/etc/check_mk/mrpe.cfg', + lens => 'Spacevars.lns', + changes => [ + 'rm /files/etc/check_mk/mrpe.cfg/haveged_proc', + 'set haveged_proc \'/usr/lib/nagios/plugins/check_procs -w 1:1 -c 1:1 -a /usr/sbin/haveged\'' ], + require => File['/etc/check_mk/mrpe.cfg']; + + } + +} diff --git a/puppet/modules/site_check_mk/manifests/agent/mrpe.pp b/puppet/modules/site_check_mk/manifests/agent/mrpe.pp index 6921574f..5e1f087a 100644 --- a/puppet/modules/site_check_mk/manifests/agent/mrpe.pp +++ b/puppet/modules/site_check_mk/manifests/agent/mrpe.pp @@ -11,8 +11,14 @@ class site_check_mk::agent::mrpe { ensure => present, require => Package['check-mk-agent'] } -> - file_line { 'Apt': - line => 'APT /usr/lib/nagios/plugins/check_apt', - path => '/etc/check_mk/mrpe.cfg', + + augeas { + 'Apt': + incl => '/etc/check_mk/mrpe.cfg', + lens => 'Spacevars.lns', + changes => [ + 'rm /files/etc/check_mk/mrpe.cfg/APT', + 'set APT \'/usr/lib/nagios/plugins/check_apt\'' ]; } + } diff --git a/puppet/modules/site_check_mk/manifests/agent/mx.pp b/puppet/modules/site_check_mk/manifests/agent/mx.pp index 1e370125..98757b59 100644 --- a/puppet/modules/site_check_mk/manifests/agent/mx.pp +++ b/puppet/modules/site_check_mk/manifests/agent/mx.pp @@ -6,13 +6,16 @@ class site_check_mk::agent::mx { } # local nagios plugin checks via mrpe - file_line { + augeas { 'Leap_MX_Procs': - line => 'Leap_MX_Procs /usr/lib/nagios/plugins/check_procs -w 1:1 -c 1:1 -a \'/usr/bin/python /usr/bin/twistd --pidfile=/var/run/leap_mx.pid --rundir=/var/lib/leap_mx/ --python=/usr/share/app/leap_mx.tac --logfile=/var/log/leap_mx.log\'', - path => '/etc/check_mk/mrpe.cfg'; + incl => '/etc/check_mk/mrpe.cfg', + lens => 'Spacevars.lns', + changes => [ + 'rm /files/etc/check_mk/mrpe.cfg/Leap_MX_Procs', + 'set Leap_MX_Procs \'/usr/lib/nagios/plugins/check_procs -w 1:1 -c 1:1 -a "/usr/bin/python /usr/bin/twistd --pidfile=/var/run/leap_mx.pid --rundir=/var/lib/leap_mx/ --python=/usr/share/app/leap_mx.tac --logfile=/var/log/leap/mx.log"\'' ], + require => File['/etc/check_mk/mrpe.cfg']; } - # check stale files in queue dir file { '/usr/lib/check_mk_agent/local/check_leap_mx.sh': source => 'puppet:///modules/site_check_mk/agent/local_checks/mx/check_leap_mx.sh', diff --git a/puppet/modules/site_check_mk/manifests/agent/openvpn.pp b/puppet/modules/site_check_mk/manifests/agent/openvpn.pp index 919a408d..0596a497 100644 --- a/puppet/modules/site_check_mk/manifests/agent/openvpn.pp +++ b/puppet/modules/site_check_mk/manifests/agent/openvpn.pp @@ -2,7 +2,7 @@ class site_check_mk::agent::openvpn { # check syslog concat::fragment { 'syslog_openpvn': - source => 'puppet:///modules/site_check_mk/agent/logwatch/syslog/openvpn.cfg', + source => 'puppet:///modules/site_check_mk/agent/logwatch/openvpn.cfg', target => '/etc/check_mk/logwatch.d/syslog.cfg', order => '02'; } diff --git a/puppet/modules/site_check_mk/manifests/agent/soledad.pp b/puppet/modules/site_check_mk/manifests/agent/soledad.pp index 512d1a3d..f4a3f3a6 100644 --- a/puppet/modules/site_check_mk/manifests/agent/soledad.pp +++ b/puppet/modules/site_check_mk/manifests/agent/soledad.pp @@ -5,10 +5,13 @@ class site_check_mk::agent::soledad { } # local nagios plugin checks via mrpe - file_line { - 'Soledad_Procs': - line => 'Soledad_Procs /usr/lib/nagios/plugins/check_procs -w 1:1 -c 1:1 -a \'/usr/bin/python /usr/bin/twistd --pidfile=/var/run/soledad.pid --logfile=/var/log/soledad.log web --wsgi=leap.soledad.server.application\'', - path => '/etc/check_mk/mrpe.cfg'; - } + augeas { 'Soledad_Procs': + incl => '/etc/check_mk/mrpe.cfg', + lens => 'Spacevars.lns', + changes => [ + 'rm /files/etc/check_mk/mrpe.cfg/Soledad_Procs', + 'set Soledad_Procs \'/usr/lib/nagios/plugins/check_procs -w 1:1 -c 1:1 -a "/usr/bin/python /usr/bin/twistd --uid=soledad --gid=soledad --pidfile=/var/run/soledad.pid --logfile=/var/log/soledad.log web --wsgi=leap.soledad.server.application --port=ssl:2323:privateKey=/etc/x509/keys/leap.key:certKey=/etc/x509/certs/leap.crt:sslmethod=SSLv23_METHOD"\'' ], + require => File['/etc/check_mk/mrpe.cfg']; + } } diff --git a/puppet/modules/site_check_mk/manifests/agent/stunnel.pp b/puppet/modules/site_check_mk/manifests/agent/stunnel.pp index 64022824..7f765771 100644 --- a/puppet/modules/site_check_mk/manifests/agent/stunnel.pp +++ b/puppet/modules/site_check_mk/manifests/agent/stunnel.pp @@ -1,7 +1,7 @@ class site_check_mk::agent::stunnel { concat::fragment { 'syslog_stunnel': - source => 'puppet:///modules/site_check_mk/agent/logwatch/syslog/stunnel.cfg', + source => 'puppet:///modules/site_check_mk/agent/logwatch/stunnel.cfg', target => '/etc/check_mk/logwatch.d/syslog.cfg', order => '02'; } diff --git a/puppet/modules/site_check_mk/manifests/agent/tapicero.pp b/puppet/modules/site_check_mk/manifests/agent/tapicero.pp index ffd11100..4a5ec68e 100644 --- a/puppet/modules/site_check_mk/manifests/agent/tapicero.pp +++ b/puppet/modules/site_check_mk/manifests/agent/tapicero.pp @@ -2,21 +2,24 @@ class site_check_mk::agent::tapicero { include ::site_nagios::plugins - concat::fragment { 'syslog_tapicero': - source => 'puppet:///modules/site_check_mk/agent/logwatch/syslog/tapicero.cfg', - target => '/etc/check_mk/logwatch.d/syslog.cfg', - order => '02'; + # watch logs + file { '/etc/check_mk/logwatch.d/tapicero.cfg': + source => 'puppet:///modules/site_check_mk/agent/logwatch/tapicero.cfg', } # local nagios plugin checks via mrpe - file_line { + augeas { 'Tapicero_Procs': - line => 'Tapicero_Procs /usr/lib/nagios/plugins/check_procs -w 1:1 -c 1:1 -a tapicero', - path => '/etc/check_mk/mrpe.cfg'; - + incl => '/etc/check_mk/mrpe.cfg', + lens => 'Spacevars.lns', + changes => [ + 'rm /files/etc/check_mk/mrpe.cfg/Tapicero_Procs', + 'set Tapicero_Procs "/usr/lib/nagios/plugins/check_procs -w 1:1 -c 1:1 -a tapicero"' ], + require => File['/etc/check_mk/mrpe.cfg']; 'Tapicero_Heartbeat': - line => 'Tapicero_Heartbeat /usr/local/lib/nagios/plugins/check_last_regex_in_log -f /var/log/syslog -r "tapicero" -w 300 -c 600', - path => '/etc/check_mk/mrpe.cfg'; + incl => '/etc/check_mk/mrpe.cfg', + lens => 'Spacevars.lns', + changes => 'set Tapicero_Heartbeat \'/usr/local/lib/nagios/plugins/check_last_regex_in_log -f /var/log/leap/tapicero.log -r "tapicero" -w 300 -c 600\'', + require => File['/etc/check_mk/mrpe.cfg']; } - } diff --git a/puppet/modules/site_check_mk/manifests/agent/webapp.pp b/puppet/modules/site_check_mk/manifests/agent/webapp.pp index 88c3da30..9bf3b197 100644 --- a/puppet/modules/site_check_mk/manifests/agent/webapp.pp +++ b/puppet/modules/site_check_mk/manifests/agent/webapp.pp @@ -7,11 +7,9 @@ class site_check_mk::agent::webapp { ensure => absent } - # check syslog - concat::fragment { 'syslog_webapp': - source => 'puppet:///modules/site_check_mk/agent/logwatch/syslog/webapp.cfg', - target => '/etc/check_mk/logwatch.d/syslog.cfg', - order => '02'; + # watch logs + file { '/etc/check_mk/logwatch.d/webapp.cfg': + source => 'puppet:///modules/site_check_mk/agent/logwatch/webapp.cfg', } } diff --git a/puppet/modules/site_config/lib/facter/dhcp_enabled.rb b/puppet/modules/site_config/lib/facter/dhcp_enabled.rb new file mode 100644 index 00000000..33220da3 --- /dev/null +++ b/puppet/modules/site_config/lib/facter/dhcp_enabled.rb @@ -0,0 +1,22 @@ +require 'facter' +def dhcp_enabled?(ifs, recurse=true) + dhcp = false + included_ifs = [] + if FileTest.exists?(ifs) + File.open(ifs) do |file| + dhcp = file.enum_for(:each_line).any? do |line| + if recurse && line =~ /^\s*source\s+([^\s]+)/ + included_ifs += Dir.glob($1) + end + line =~ /inet\s+dhcp/ + end + end + end + dhcp || included_ifs.any? { |ifs| dhcp_enabled?(ifs, false) } +end +Facter.add(:dhcp_enabled) do + confine :osfamily => 'Debian' + setcode do + dhcp_enabled?('/etc/network/interfaces') + end +end diff --git a/puppet/modules/site_config/manifests/default.pp b/puppet/modules/site_config/manifests/default.pp index 790b5a16..e69e4b7b 100644 --- a/puppet/modules/site_config/manifests/default.pp +++ b/puppet/modules/site_config/manifests/default.pp @@ -1,6 +1,10 @@ class site_config::default { tag 'leap_base' + # the logoutput exec parameter defaults to "on_error" in puppet 3, + # but to "false" in puppet 2.7, so we need to set this globally here + Exec<||> { logoutput => on_failure } + $services = hiera('services', []) $domain_hash = hiera('domain') include site_config::params @@ -25,10 +29,7 @@ class site_config::default { # i.e. openstack/aws nodes, vagrant nodes # fix dhclient from changing resolver information - if $::ec2_instance_id { - include site_config::dhclient - } - if $::virtual == 'virtualbox' { + if $::dhcp_enabled == 'true' { include site_config::dhclient } @@ -38,22 +39,26 @@ class site_config::default { # configure caching, local resolver include site_config::caching_resolver - # install/configure syslog + # install/configure syslog and core log rotations include site_config::syslog + # provide a basic level of quality entropy + include haveged + # install/remove base packages include site_config::packages::base # include basic shorewall config include site_shorewall::defaults - Class['git'] -> Vcsrepo<||> + Package['git'] -> Vcsrepo<||> # include basic shell config include site_config::shell # set up core leap files and directories include site_config::files + include site_config::remove_files if ! member($services, 'mx') { include site_postfix::satellite diff --git a/puppet/modules/site_config/manifests/dhclient.pp b/puppet/modules/site_config/manifests/dhclient.pp index 7ac0caf3..dbe2ef1c 100644 --- a/puppet/modules/site_config/manifests/dhclient.pp +++ b/puppet/modules/site_config/manifests/dhclient.pp @@ -17,7 +17,9 @@ class site_config::dhclient { exec { 'reload_dhclient': refreshonly => true, - command => '/usr/local/sbin/reload_dhclient'; + command => '/usr/local/sbin/reload_dhclient', + before => Class['site_config::resolvconf'], + require => File['/usr/local/sbin/reload_dhclient'], } file { '/etc/dhcp/dhclient-enter-hooks.d/disable_resolvconf': diff --git a/puppet/modules/site_config/manifests/packages/base.pp b/puppet/modules/site_config/manifests/packages/base.pp index ae47963c..f20d04a4 100644 --- a/puppet/modules/site_config/manifests/packages/base.pp +++ b/puppet/modules/site_config/manifests/packages/base.pp @@ -7,7 +7,7 @@ class site_config::packages::base { } # base set of packages that we want to remove everywhere - package { [ 'acpi', 'acpid', 'acpi-support-base', 'eject', 'ftp', 'fontconfig-config', + package { [ 'acpi', 'acpid', 'acpi-support-base', 'eject', 'ftp', 'laptop-detect', 'lpr', 'nfs-common', 'nfs-kernel-server', 'portmap', 'pppconfig', 'pppoe', 'pump', 'qstat', 'rpcbind', 'samba-common', 'samba-common-bin', 'smbclient', 'tcl8.5', diff --git a/puppet/modules/site_config/manifests/remove_files.pp b/puppet/modules/site_config/manifests/remove_files.pp new file mode 100644 index 00000000..3f46659c --- /dev/null +++ b/puppet/modules/site_config/manifests/remove_files.pp @@ -0,0 +1,46 @@ +# +# Sometimes when we upgrade the platform, we need to ensure that files that +# the platform previously created will get removed. +# +# These file removals don't need to be kept forever: we only need to remove +# files that are present in the prior platform release. +# +# We can assume that the every node is upgraded from the previous platform +# release. +# + +class site_config::remove_files { + + # + # Platform 0.7 removals + # + + tidy { + '/etc/rsyslog.d/99-tapicero.conf':; + '/etc/rsyslog.d/99-leap-mx.conf':; + '/etc/rsyslog.d/01-webapp.conf':; + '/etc/rsyslog.d/50-stunnel.conf':; + '/etc/logrotate.d/mx':; + '/etc/logrotate.d/stunnel':; + '/var/log/stunnel4/stunnel.log':; + 'leap_mx': + path => '/var/log/', + recurse => true, + matches => 'leap_mx*'; + '/srv/leap/webapp/public/provider.json':; + '/srv/leap/couchdb/designs/tmp_users': + recurse => true, + rmdirs => true; + } + + # leax-mx logged to /var/log/leap_mx.log in the past + # we need to use a dumb exec here because file_line doesn't + # allow removing lines that match a regex in the current version + # of stdlib, see https://tickets.puppetlabs.com/browse/MODULES-1903 + exec { 'rm_old_leap_mx_log_destination': + command => "/bin/sed -i '/leap_mx.log/d' /etc/check_mk/logwatch.state", + onlyif => "/bin/grep -qe 'leap_mx.log' /etc/check_mk/logwatch.state" + } + + +} diff --git a/puppet/modules/site_config/manifests/syslog.pp b/puppet/modules/site_config/manifests/syslog.pp index 26c65f02..83b49c8e 100644 --- a/puppet/modules/site_config/manifests/syslog.pp +++ b/puppet/modules/site_config/manifests/syslog.pp @@ -11,4 +11,29 @@ class site_config::syslog { content => '$ModLoad mmanon action(type="mmanon" ipv4.bits="32" mode="rewrite")' } + + augeas { + 'logrotate_leap_deploy': + context => '/files/etc/logrotate.d/leap_deploy/rule', + changes => [ 'set file /var/log/leap/deploy.log', + 'set rotate 5', + 'set size 1M', + 'set compress compress', + 'set missingok missingok', + 'set copytruncate copytruncate' ]; + + # NOTE: + # the puppet_command script requires the option delaycompress + # be set on the summary log file. + + 'logrotate_leap_deploy_summary': + context => '/files/etc/logrotate.d/leap_deploy_summary/rule', + changes => [ 'set file /var/log/leap/deploy-summary.log', + 'set rotate 5', + 'set size 100k', + 'set delaycompress delaycompress', + 'set compress compress', + 'set missingok missingok', + 'set copytruncate copytruncate' ] + } } diff --git a/puppet/modules/site_couchdb/lib/puppet/parser/functions/rotated_db_name.rb b/puppet/modules/site_couchdb/lib/puppet/parser/functions/rotated_db_name.rb new file mode 100644 index 00000000..6458ae81 --- /dev/null +++ b/puppet/modules/site_couchdb/lib/puppet/parser/functions/rotated_db_name.rb @@ -0,0 +1,24 @@ +module Puppet::Parser::Functions + newfunction(:rotated_db_name, :type => :rvalue, :doc => <<-EOS +This function takes a database name string and returns a database name with the current rotation stamp appended. +The first argument is the base name of the database. Subsequent arguments may contain these options: + * 'next' -- return the db name for the next rotation, not the current one. + * 'monthly' -- rotate monthly (default) + * 'weekly' -- rotate weekly +*Examples:* + rotated_db_name('tokens') => 'tokens_551' + EOS + ) do |arguments| + if arguments.include?('weekly') + rotation_period = 604800 # 1 week + else + rotation_period = 2592000 # 1 month + end + suffix = Time.now.utc.to_i / rotation_period + if arguments.include?('next') + suffix += 1 + end + "#{arguments.first}_#{suffix}" + end +end + diff --git a/puppet/modules/site_couchdb/manifests/bigcouch.pp b/puppet/modules/site_couchdb/manifests/bigcouch.pp index 16593ec7..82c85b52 100644 --- a/puppet/modules/site_couchdb/manifests/bigcouch.pp +++ b/puppet/modules/site_couchdb/manifests/bigcouch.pp @@ -17,6 +17,7 @@ class site_couchdb::bigcouch { # stunnel must running correctly before bigcouch dbs can be set up. # Class['site_config::default'] + -> Class['site_config::resolvconf'] -> Class['couchdb::bigcouch::package::cloudant'] -> Service['shorewall'] -> Exec['refresh_stunnel'] diff --git a/puppet/modules/site_couchdb/manifests/create_dbs.pp b/puppet/modules/site_couchdb/manifests/create_dbs.pp index 4322f773..b743127a 100644 --- a/puppet/modules/site_couchdb/manifests/create_dbs.pp +++ b/puppet/modules/site_couchdb/manifests/create_dbs.pp @@ -1,10 +1,9 @@ class site_couchdb::create_dbs { Class['site_couchdb::setup'] + -> Class['site_couchdb::bigcouch::settle_cluster'] -> Class['site_couchdb::create_dbs'] - # Couchdb databases - ### customer database ### r/w: webapp, couchdb::create_db { 'customers': @@ -29,7 +28,14 @@ class site_couchdb::create_dbs { ## sessions database ## r/w: webapp - couchdb::create_db { 'sessions': + $sessions_db = rotated_db_name('sessions', 'monthly') + couchdb::create_db { $sessions_db: + members => "{ \"names\": [\"$site_couchdb::couchdb_webapp_user\"], \"roles\": [\"replication\"] }", + require => Couchdb::Query::Setup['localhost'] + } + + $sessions_next_db = rotated_db_name('sessions', 'monthly', 'next') + couchdb::create_db { $sessions_next_db: members => "{ \"names\": [\"$site_couchdb::couchdb_webapp_user\"], \"roles\": [\"replication\"] }", require => Couchdb::Query::Setup['localhost'] } @@ -51,7 +57,14 @@ class site_couchdb::create_dbs { ## tokens database ## r: soledad - needs to be restricted with a design document ## r/w: webapp - couchdb::create_db { 'tokens': + $tokens_db = rotated_db_name('tokens', 'monthly') + couchdb::create_db { $tokens_db: + members => "{ \"names\": [], \"roles\": [\"replication\", \"tokens\"] }", + require => Couchdb::Query::Setup['localhost'] + } + + $tokens_next_db = rotated_db_name('tokens', 'monthly', 'next') + couchdb::create_db { $tokens_next_db: members => "{ \"names\": [], \"roles\": [\"replication\", \"tokens\"] }", require => Couchdb::Query::Setup['localhost'] } @@ -63,6 +76,13 @@ class site_couchdb::create_dbs { require => Couchdb::Query::Setup['localhost'] } + ## tmp_users database + ## r/w: webapp + couchdb::create_db { 'tmp_users': + members => "{ \"names\": [], \"roles\": [\"replication\", \"users\"] }", + require => Couchdb::Query::Setup['localhost'] + } + ## messages db ## store messages to the clients such as payment reminders ## r/w: webapp diff --git a/puppet/modules/site_couchdb/manifests/designs.pp b/puppet/modules/site_couchdb/manifests/designs.pp index 9e88de64..1ab1c6a1 100644 --- a/puppet/modules/site_couchdb/manifests/designs.pp +++ b/puppet/modules/site_couchdb/manifests/designs.pp @@ -11,10 +11,35 @@ class site_couchdb::designs { mode => '0755' } - exec { '/srv/leap/couchdb/scripts/load_design_documents.sh': - require => Vcsrepo['/srv/leap/couchdb/scripts'], - refreshonly => false + site_couchdb::upload_design { + 'customers': design => 'customers/Customer.json'; + 'identities': design => 'identities/Identity.json'; + 'tickets': design => 'tickets/Ticket.json'; + 'messages': design => 'messages/Message.json'; + 'users': design => 'users/User.json'; + 'tmp_users': design => 'users/User.json'; + 'shared_docs': + db => 'shared', + design => 'shared/docs.json'; + 'shared_syncs': + db => 'shared', + design => 'shared/syncs.json'; + 'shared_transactions': + db => 'shared', + design => 'shared/transactions.json'; } -} + $sessions_db = rotated_db_name('sessions', 'monthly') + $sessions_next_db = rotated_db_name('sessions', 'monthly', 'next') + site_couchdb::upload_design { + $sessions_db: design => 'sessions/Session.json'; + $sessions_next_db: design => 'sessions/Session.json'; + } + $tokens_db = rotated_db_name('tokens', 'monthly') + $tokens_next_db = rotated_db_name('tokens', 'monthly', 'next') + site_couchdb::upload_design { + $tokens_db: design => 'tokens/Token.json'; + $tokens_next_db: design => 'tokens/Token.json'; + } +} diff --git a/puppet/modules/site_couchdb/manifests/upload_design.pp b/puppet/modules/site_couchdb/manifests/upload_design.pp new file mode 100644 index 00000000..7b0cabd7 --- /dev/null +++ b/puppet/modules/site_couchdb/manifests/upload_design.pp @@ -0,0 +1,13 @@ +define site_couchdb::upload_design($db = $title, $design) { + $design_name = regsubst($design, '^.*\/(.*)\.json$', '\1') + $id = "_design/${design_name}" + $file = "/srv/leap/couchdb/designs/${design}" + exec { + "upload_design_${name}": + command => "/usr/local/bin/couch-doc-update --host 127.0.0.1:5984 --db '${db}' --id '${id}' --data '{}' --file '${file}'", + refreshonly => false, + loglevel => debug, + logoutput => on_failure, + require => File['/srv/leap/couchdb/designs']; + } +} diff --git a/puppet/modules/site_nagios/files/configs/Debian/nagios.cfg b/puppet/modules/site_nagios/files/configs/Debian/nagios.cfg index e46ebf62..0d729b8c 100644 --- a/puppet/modules/site_nagios/files/configs/Debian/nagios.cfg +++ b/puppet/modules/site_nagios/files/configs/Debian/nagios.cfg @@ -1273,4 +1273,15 @@ debug_file=/var/lib/nagios3/nagios.debug max_debug_file_size=1000000 +process_performance_data=1 +service_perfdata_file=/var/lib/nagios3/service-perfdata +service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$ +service_perfdata_file_mode=a +service_perfdata_file_processing_interval=15 +service_perfdata_file_processing_command=process-service-perfdata-file-pnp4nagios-bulk-npcd +host_perfdata_file=/var/lib/nagios3/host-perfdata +host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$ +host_perfdata_file_mode=a +host_perfdata_file_processing_interval=15 +host_perfdata_file_processing_command=process-host-perfdata-file-pnp4nagios-bulk-npcd diff --git a/puppet/modules/site_nagios/files/plugins/check_last_regex_in_log b/puppet/modules/site_nagios/files/plugins/check_last_regex_in_log index cf7c03e5..47569388 100755 --- a/puppet/modules/site_nagios/files/plugins/check_last_regex_in_log +++ b/puppet/modules/site_nagios/files/plugins/check_last_regex_in_log @@ -50,7 +50,7 @@ done [ $warn -eq 0 -o $crit -eq 0 -o -z "$regex" -o -z "$log" ] && ( usage; exit $STATE_UNKNOWN) [ -f "$log" ] || (echo "$log doesn't exist"; exit $STATE_UNKNOWN) -lastmsg=$(tac $log | grep -i $regex | head -1 | cut -d' ' -f 1-3) +lastmsg=$(tac $log | grep -i $regex | head -1 | sed 's/ / /g' | cut -d' ' -f 1-3) if [ -z "$lastmsg" ] then diff --git a/puppet/modules/site_nagios/manifests/server.pp b/puppet/modules/site_nagios/manifests/server.pp index 092ca503..cb6c8d95 100644 --- a/puppet/modules/site_nagios/manifests/server.pp +++ b/puppet/modules/site_nagios/manifests/server.pp @@ -1,3 +1,4 @@ +# configures nagios on monitoring node class site_nagios::server inherits nagios::base { # First, purge old nagios config (see #1467) @@ -13,7 +14,8 @@ class site_nagios::server inherits nagios::base { include nagios::defaults::commands include nagios::defaults::templates include nagios::defaults::timeperiods - include nagios::defaults::plugins + include nagios::pnp4nagios + include nagios::pnp4nagios::popup class { 'nagios': # don't manage apache class from nagios, cause we already include @@ -41,10 +43,11 @@ class site_nagios::server inherits nagios::base { # deploy serverside plugins file { '/usr/lib/nagios/plugins/check_openvpn_server.pl': - source => 'puppet:///modules/nagios/plugins/check_openvpn_server.pl', - mode => '0755', - owner => 'nagios', - group => 'nagios', + source => 'puppet:///modules/nagios/plugins/check_openvpn_server.pl', + mode => '0755', + owner => 'nagios', + group => 'nagios', + require => Package['nagios-plugins']; } create_resources ( site_nagios::add_host_services, $nagios_hosts ) diff --git a/puppet/modules/site_nickserver/manifests/init.pp b/puppet/modules/site_nickserver/manifests/init.pp index eaf90d55..c2deab0f 100644 --- a/puppet/modules/site_nickserver/manifests/init.pp +++ b/puppet/modules/site_nickserver/manifests/init.pp @@ -34,11 +34,12 @@ class site_nickserver { # See site_webapp/templates/haproxy_couchdb.cfg.erg $couchdb_port = '4096' + $sources = hiera('sources') + # temporarily for now: $domain = hiera('domain') $address_domain = $domain['full_suffix'] - include site_config::x509::cert include site_config::x509::key include site_config::x509::ca @@ -69,9 +70,9 @@ class site_nickserver { vcsrepo { '/srv/leap/nickserver': ensure => present, - revision => 'origin/master', - provider => git, - source => 'https://leap.se/git/nickserver', + revision => $sources['nickserver']['revision'], + provider => $sources['nickserver']['type'], + source => $sources['nickserver']['source'], owner => 'nickserver', group => 'nickserver', require => [ User['nickserver'], Group['nickserver'] ], diff --git a/puppet/modules/site_nickserver/templates/nickserver-proxy.conf.erb b/puppet/modules/site_nickserver/templates/nickserver-proxy.conf.erb index 56a8d9f6..d4e734c3 100644 --- a/puppet/modules/site_nickserver/templates/nickserver-proxy.conf.erb +++ b/puppet/modules/site_nickserver/templates/nickserver-proxy.conf.erb @@ -8,17 +8,13 @@ Listen 0.0.0.0:<%= @nickserver_port -%> ServerName <%= @nickserver_domain %> ServerAlias <%= @address_domain %> - SSLEngine on - SSLProtocol all -SSLv2 -SSLv3 - SSLHonorCipherOrder on - SSLCompression off - SSLCipherSuite "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK" - SSLCACertificatePath /etc/ssl/certs SSLCertificateChainFile <%= scope.lookupvar('x509::variables::local_CAs') %>/<%= scope.lookupvar('site_config::params::ca_name') %>.crt SSLCertificateKeyFile <%= scope.lookupvar('x509::variables::keys') %>/<%= scope.lookupvar('site_config::params::cert_name') %>.key SSLCertificateFile <%= scope.lookupvar('x509::variables::certs') %>/<%= scope.lookupvar('site_config::params::cert_name') %>.crt + Include include.d/ssl_common.inc + ProxyPass / http://localhost:<%= @nickserver_local_port %>/ ProxyPreserveHost On # preserve Host header in HTTP request </VirtualHost> diff --git a/puppet/modules/site_openvpn/manifests/init.pp b/puppet/modules/site_openvpn/manifests/init.pp index d6f9150b..e2a3124e 100644 --- a/puppet/modules/site_openvpn/manifests/init.pp +++ b/puppet/modules/site_openvpn/manifests/init.pp @@ -228,6 +228,7 @@ class site_openvpn { order => 10; } + leap::logfile { 'openvpn': } include site_check_mk::agent::openvpn } diff --git a/puppet/modules/site_openvpn/manifests/server_config.pp b/puppet/modules/site_openvpn/manifests/server_config.pp index 466f6d00..221c79a7 100644 --- a/puppet/modules/site_openvpn/manifests/server_config.pp +++ b/puppet/modules/site_openvpn/manifests/server_config.pp @@ -57,6 +57,8 @@ define site_openvpn::server_config( $management, $config, $tls_remote = undef) { $openvpn_configname = $name + $shortname = regsubst(regsubst($name, '_config', ''), '_', '-') + $openvpn_status_filename = "/var/run/openvpn-status-${shortname}" concat { "/etc/openvpn/${openvpn_configname}.conf": @@ -187,7 +189,7 @@ define site_openvpn::server_config( server => $openvpn_configname; "status ${openvpn_configname}": key => 'status', - value => '/var/run/openvpn-status 10', + value => "${openvpn_status_filename} 10", server => $openvpn_configname; "status-version ${openvpn_configname}": key => 'status-version', diff --git a/puppet/modules/site_postfix/manifests/mx.pp b/puppet/modules/site_postfix/manifests/mx.pp index 81f10b77..49692d24 100644 --- a/puppet/modules/site_postfix/manifests/mx.pp +++ b/puppet/modules/site_postfix/manifests/mx.pp @@ -1,3 +1,6 @@ +# +# configure mx node +# class site_postfix::mx { $domain_hash = hiera('domain') @@ -35,6 +38,12 @@ class site_postfix::mx { # because the satellites need to have a different value 'smtp_tls_security_level': value => 'may'; + # reject inbound mail to system users + # see https://leap.se/code/issues/6829 + # this blocks *only* mails to system users, that don't appear in the + # alias map + 'local_recipient_maps': + value => '$alias_maps'; } include site_postfix::mx::smtpd_checks diff --git a/puppet/modules/site_static/manifests/domain.pp b/puppet/modules/site_static/manifests/domain.pp index 6941b1a3..b9177f25 100644 --- a/puppet/modules/site_static/manifests/domain.pp +++ b/puppet/modules/site_static/manifests/domain.pp @@ -12,9 +12,18 @@ define site_static::domain ( create_resources(site_static::location, $locations) - x509::cert { $domain: content => $cert } - x509::key { $domain: content => $key } - x509::ca { "${domain}_ca": content => $ca_cert } + x509::cert { $domain: + content => $cert, + notify => Service[apache] + } + x509::key { $domain: + content => $key, + notify => Service[apache] + } + x509::ca { "${domain}_ca": + content => $ca_cert, + notify => Service[apache] + } apache::vhost::file { $domain: content => template('site_static/apache.conf.erb') diff --git a/puppet/modules/site_static/manifests/init.pp b/puppet/modules/site_static/manifests/init.pp index aed9775e..ce79c00f 100644 --- a/puppet/modules/site_static/manifests/init.pp +++ b/puppet/modules/site_static/manifests/init.pp @@ -44,7 +44,7 @@ class site_static { if (member($formats, 'amber')) { include site_config::ruby::dev - rubygems::gem{'amber-0.3.0': } + rubygems::gem{'amber-0.3.4': } } create_resources(site_static::domain, $domains) diff --git a/puppet/modules/site_static/templates/apache.conf.erb b/puppet/modules/site_static/templates/apache.conf.erb index 9b516a10..4d61cc08 100644 --- a/puppet/modules/site_static/templates/apache.conf.erb +++ b/puppet/modules/site_static/templates/apache.conf.erb @@ -45,12 +45,8 @@ #RewriteLog "/var/log/apache2/rewrite.log" #RewriteLogLevel 3 - SSLEngine on - SSLProtocol all -SSLv2 -SSLv3 - SSLHonorCipherOrder on - SSLCompression off - SSLCipherSuite "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK" - + Include include.d/ssl_common.inc + <%- if @tls_only -%> Header add Strict-Transport-Security: "max-age=15768000;includeSubdomains" <%- end -%> diff --git a/puppet/modules/site_stunnel/manifests/client.pp b/puppet/modules/site_stunnel/manifests/client.pp index 3b10ecb8..c9e034f1 100644 --- a/puppet/modules/site_stunnel/manifests/client.pp +++ b/puppet/modules/site_stunnel/manifests/client.pp @@ -14,7 +14,9 @@ define site_stunnel::client ( $verify = '2', $pid = $name, $rndfile = '/var/lib/stunnel4/.rnd', - $debuglevel = '4' ) { + $debuglevel = 'warning' ) { + + $logfile = "/var/log/stunnel4/${name}.log" include site_config::x509::cert include site_config::x509::key @@ -35,7 +37,20 @@ define site_stunnel::client ( pid => "/var/run/stunnel4/${pid}.pid", rndfile => $rndfile, debuglevel => $debuglevel, - sslversion => 'TLSv1'; + sslversion => 'TLSv1', + syslog => 'no', + output => $logfile; + } + + # define the log files so that we can purge the + # files from /var/log/stunnel4 that are not defined. + file { + $logfile:; + "${logfile}.1.gz":; + "${logfile}.2.gz":; + "${logfile}.3.gz":; + "${logfile}.4.gz":; + "${logfile}.5.gz":; } site_shorewall::stunnel::client { $name: diff --git a/puppet/modules/site_stunnel/manifests/init.pp b/puppet/modules/site_stunnel/manifests/init.pp index 2e0cf5b8..d919a072 100644 --- a/puppet/modules/site_stunnel/manifests/init.pp +++ b/puppet/modules/site_stunnel/manifests/init.pp @@ -29,6 +29,20 @@ class site_stunnel { $client_sections = keys($clients) site_stunnel::clients { $client_sections: } + # remove any old stunnel logs that are not + # defined by this puppet run + file {'/var/log/stunnel4': purge => true;} + + # the default is to keep 356 log files for each stunnel. + # here we set a more reasonable number. + augeas { + "logrotate_stunnel": + context => "/files/etc/logrotate.d/stunnel4/rule", + changes => [ + 'set rotate 5', + ] + } + include site_stunnel::override_service } diff --git a/puppet/modules/site_webapp/manifests/apache.pp b/puppet/modules/site_webapp/manifests/apache.pp index 21243d34..93e172a0 100644 --- a/puppet/modules/site_webapp/manifests/apache.pp +++ b/puppet/modules/site_webapp/manifests/apache.pp @@ -7,6 +7,9 @@ class site_webapp::apache { $web_domain = hiera('domain') $domain_name = $web_domain['name'] + $webapp = hiera('webapp') + $webapp_domain = $webapp['domain'] + include site_apache::common include site_apache::module::headers include site_apache::module::alias diff --git a/puppet/modules/site_webapp/manifests/couchdb.pp b/puppet/modules/site_webapp/manifests/couchdb.pp index 3ae4d266..1dbc745d 100644 --- a/puppet/modules/site_webapp/manifests/couchdb.pp +++ b/puppet/modules/site_webapp/manifests/couchdb.pp @@ -6,6 +6,8 @@ class site_webapp::couchdb { $couchdb_port = '4096' $couchdb_webapp_user = $webapp['couchdb_webapp_user']['username'] $couchdb_webapp_password = $webapp['couchdb_webapp_user']['password'] + $couchdb_admin_user = $webapp['couchdb_admin_user']['username'] + $couchdb_admin_password = $webapp['couchdb_admin_user']['password'] include x509::variables @@ -17,6 +19,13 @@ class site_webapp::couchdb { mode => '0600', require => Vcsrepo['/srv/leap/webapp']; + '/srv/leap/webapp/config/couchdb.admin.yml': + content => template('site_webapp/couchdb.admin.yml.erb'), + owner => leap-webapp, + group => leap-webapp, + mode => '0600', + require => Vcsrepo['/srv/leap/webapp']; + '/srv/leap/webapp/log': ensure => directory, owner => leap-webapp, diff --git a/puppet/modules/site_webapp/manifests/cron.pp b/puppet/modules/site_webapp/manifests/cron.pp index 811ad11d..d26ee312 100644 --- a/puppet/modules/site_webapp/manifests/cron.pp +++ b/puppet/modules/site_webapp/manifests/cron.pp @@ -2,11 +2,26 @@ class site_webapp::cron { # cron tasks that need to be performed to cleanup the database cron { + 'rotate_databases': + command => 'cd /srv/leap/webapp && bundle exec rake db:rotate', + environment => 'RAILS_ENV=production', + hour => [0,6,12,18], + minute => 0; + + 'delete_tmp_databases': + command => 'cd /srv/leap/webapp && bundle exec rake db:deletetmp', + environment => 'RAILS_ENV=production', + hour => 1, + minute => 1; + + # there is no longer a need to remove expired sessions, since the database + # will get destroyed. 'remove_expired_sessions': command => 'cd /srv/leap/webapp && bundle exec rake cleanup:sessions', environment => 'RAILS_ENV=production', hour => 2, - minute => 30; + minute => 30, + ensure => absent; 'remove_expired_tokens': command => 'cd /srv/leap/webapp && bundle exec rake cleanup:tokens', diff --git a/puppet/modules/site_webapp/manifests/init.pp b/puppet/modules/site_webapp/manifests/init.pp index 9f97d2c5..ec94c090 100644 --- a/puppet/modules/site_webapp/manifests/init.pp +++ b/puppet/modules/site_webapp/manifests/init.pp @@ -11,13 +11,13 @@ class site_webapp { $api_version = $webapp['api_version'] $secret_token = $webapp['secret_token'] $tor = hiera('tor', false) + $sources = hiera('sources') Class['site_config::default'] -> Class['site_webapp'] include site_config::ruby::dev include site_webapp::apache include site_webapp::couchdb - include site_webapp::logging include site_haproxy include site_webapp::cron include site_config::x509::cert @@ -43,9 +43,9 @@ class site_webapp { vcsrepo { '/srv/leap/webapp': ensure => present, force => true, - revision => $webapp['git']['revision'], - provider => git, - source => $webapp['git']['source'], + revision => $sources['webapp']['revision'], + provider => $sources['webapp']['type'], + source => $sources['webapp']['source'], owner => 'leap-webapp', group => 'leap-webapp', require => [ User['leap-webapp'], Group['leap-webapp'] ], @@ -92,10 +92,6 @@ class site_webapp { require => Vcsrepo['/srv/leap/webapp'], owner => leap-webapp, group => leap-webapp, mode => '0644'; - # old provider.json location. this can be removed after everyone upgrades. - '/srv/leap/webapp/public/provider.json': - ensure => absent; - '/srv/leap/webapp/public/ca.crt': ensure => link, require => Vcsrepo['/srv/leap/webapp'], @@ -172,6 +168,8 @@ class site_webapp { ensure => latest, } + leap::logfile { 'webapp': } + include site_shorewall::webapp include site_check_mk::agent::webapp } diff --git a/puppet/modules/site_webapp/manifests/logging.pp b/puppet/modules/site_webapp/manifests/logging.pp deleted file mode 100644 index b414b82c..00000000 --- a/puppet/modules/site_webapp/manifests/logging.pp +++ /dev/null @@ -1,16 +0,0 @@ -class site_webapp::logging { - - rsyslog::snippet { '01-webapp': - content => 'if $programname == "webapp" then /var/log/leap/webapp.log -&~' - } - - augeas { - 'logrotate_webapp': - context => '/files/etc/logrotate.d/webapp/rule', - changes => [ 'set file /var/log/leap/webapp.log', 'set rotate 7', - 'set schedule daily', 'set compress compress', - 'set missingok missingok', 'set ifempty notifempty', - 'set copytruncate copytruncate' ] - } -} diff --git a/puppet/modules/site_webapp/templates/config.yml.erb b/puppet/modules/site_webapp/templates/config.yml.erb index 0c75f3ca..ccde2d2e 100644 --- a/puppet/modules/site_webapp/templates/config.yml.erb +++ b/puppet/modules/site_webapp/templates/config.yml.erb @@ -7,7 +7,7 @@ production: client_ca_key: <%= scope.lookupvar('x509::variables::keys') %>/<%= scope.lookupvar('site_config::params::client_ca_name') %>.key client_ca_cert: <%= scope.lookupvar('x509::variables::local_CAs') %>/<%= scope.lookupvar('site_config::params::client_ca_name') %>.crt secret_token: "<%= @secret_token %>" - client_cert_lifespan: <%= cert_options['life_span'].to_i %> + client_cert_lifespan: <%= cert_options['life_span'] %> client_cert_bit_size: <%= cert_options['bit_size'].to_i %> client_cert_hash: <%= cert_options['digest'] %> allow_limited_certs: <%= @webapp['allow_limited_certs'].inspect %> @@ -17,7 +17,7 @@ production: unlimited_cert_prefix: "<%= cert_options['unlimited_prefix'] %>" minimum_client_version: "<%= @webapp['client_version']['min'] %>" default_service_level: "<%= @webapp['default_service_level'] %>" - service_levels: <%= @webapp['service_levels'].to_json %> + service_levels: <%= scope.function_sorted_json([@webapp['service_levels']]) %> allow_registration: <%= @webapp['allow_registration'].inspect %> handle_blacklist: <%= @webapp['forbidden_usernames'].inspect %> <%- if @webapp['engines'] && @webapp['engines'].any? -%> diff --git a/puppet/modules/site_webapp/templates/couchdb.admin.yml.erb b/puppet/modules/site_webapp/templates/couchdb.admin.yml.erb new file mode 100644 index 00000000..a0921add --- /dev/null +++ b/puppet/modules/site_webapp/templates/couchdb.admin.yml.erb @@ -0,0 +1,9 @@ +production: + prefix: "" + protocol: 'http' + host: <%= @couchdb_host %> + port: <%= @couchdb_port %> + auto_update_design_doc: false + username: <%= @couchdb_admin_user %> + password: <%= @couchdb_admin_password %> + diff --git a/puppet/modules/soledad/manifests/server.pp b/puppet/modules/soledad/manifests/server.pp index 394e6032..b71fab69 100644 --- a/puppet/modules/soledad/manifests/server.pp +++ b/puppet/modules/soledad/manifests/server.pp @@ -12,6 +12,8 @@ class soledad::server { $soledad_port = $soledad['port'] + $sources = hiera('sources') + include site_config::x509::cert include site_config::x509::key include site_config::x509::ca @@ -29,8 +31,8 @@ class soledad::server { require => Class['soledad']; } - package { 'soledad-server': - ensure => latest, + package { $sources['soledad']['package']: + ensure => $sources['soledad']['revision'], require => [ Class['site_apt::preferences::twisted'], Class['site_apt::leap_repo'] ]; diff --git a/puppet/modules/stdlib b/puppet/modules/stdlib -Subproject 71cb0f4c2c3bf95f62c9f189f5cef155b09a968 +Subproject 71123634744b9fe2ec7d6a3e38e9789fd84801e diff --git a/puppet/modules/tapicero/manifests/init.pp b/puppet/modules/tapicero/manifests/init.pp index 28711b94..ca8488c8 100644 --- a/puppet/modules/tapicero/manifests/init.pp +++ b/puppet/modules/tapicero/manifests/init.pp @@ -15,6 +15,8 @@ class tapicero { $couchdb_mode = $couchdb['mode'] $couchdb_replication = $couchdb['replication'] + $sources = hiera('sources') + Class['site_config::default'] -> Class['tapicero'] include site_config::ruby::dev @@ -42,9 +44,9 @@ class tapicero { file { - ## - ## TAPICERO DIRECTORIES - ## + # + # TAPICERO DIRECTORIES + # '/srv/leap/tapicero': ensure => directory, @@ -65,9 +67,9 @@ class tapicero { group => 'tapicero', require => User['tapicero']; - ## - ## TAPICERO CONFIG - ## + # + # TAPICERO CONFIG + # '/etc/leap/tapicero.yaml': content => template('tapicero/tapicero.yaml.erb'), @@ -76,9 +78,9 @@ class tapicero { mode => '0600', notify => Service['tapicero']; - ## - ## TAPICERO INIT - ## + # + # TAPICERO INIT + # '/etc/init.d/tapicero': source => 'puppet:///modules/tapicero/tapicero.init', @@ -95,9 +97,9 @@ class tapicero { vcsrepo { '/srv/leap/tapicero': ensure => present, force => true, - revision => 'origin/version/0.6', - provider => git, - source => 'https://leap.se/git/tapicero', + revision => $sources['tapicero']['revision'], + provider => $sources['tapicero']['type'], + source => $sources['tapicero']['source'], owner => 'tapicero', group => 'tapicero', require => [ User['tapicero'], Group['tapicero'] ], @@ -131,4 +133,5 @@ class tapicero { Couchdb::Add_user[$::site_couchdb::couchdb_tapicero_user] ]; } + leap::logfile { 'tapicero': } } diff --git a/puppet/modules/tapicero/templates/tapicero.yaml.erb b/puppet/modules/tapicero/templates/tapicero.yaml.erb index 510450ad..8b08b49c 100644 --- a/puppet/modules/tapicero/templates/tapicero.yaml.erb +++ b/puppet/modules/tapicero/templates/tapicero.yaml.erb @@ -13,13 +13,15 @@ connection: password: <%= @couchdb_admin_password %> prefix : "" suffix : "" + netrc: "/etc/couchdb/couchdb.netrc" # file to store the last processed user record in so we can resume after # a restart: -seq_file: "/var/lib/leap/tapicero/tapicero.seq" +seq_dir: "/var/lib/leap/tapicero/" # Configure log_file like this if you want to log to a file instead of syslog: -# log_file: "/var/leap/log/tapicero.log" +#log_file: "/var/log/leap/tapicero.log" +#log_level: debug log_level: info # tapicero specific options diff --git a/tests/white-box/couchdb.rb b/tests/white-box/couchdb.rb index 450c4201..5ee12ff3 100644 --- a/tests/white-box/couchdb.rb +++ b/tests/white-box/couchdb.rb @@ -82,7 +82,9 @@ class CouchDB < LeapTest end def test_05_Do_required_databases_exist? - dbs_that_should_exist = ["customers","identities","keycache","sessions","shared","tickets","tokens","users"] + dbs_that_should_exist = ["customers","identities","keycache","shared","tickets","users", "tmp_users"] + dbs_that_should_exist << "tokens_#{rotation_suffix}" + dbs_that_should_exist << "sessions_#{rotation_suffix}" dbs_that_should_exist.each do |db_name| url = couchdb_url("/"+db_name, :username => 'admin') assert_get(url) do |body| @@ -113,9 +115,9 @@ class CouchDB < LeapTest #end def test_07_Can_records_be_created? - token = Token.new - url = couchdb_url("/tokens", :username => 'admin') - assert_post(url, token, :format => :json) do |body| + record = DummyRecord.new + url = couchdb_url("/tokens_#{rotation_suffix}", :username => 'admin') + assert_post(url, record, :format => :json) do |body| assert response = JSON.parse(body), "POST response should be JSON" assert response["ok"], "POST response should be OK" assert_delete(File.join(url, response["id"]), :rev => response["rev"]) do |body| @@ -142,13 +144,16 @@ class CouchDB < LeapTest couchdb_url(path, options) end + def rotation_suffix + rotation_suffix = Time.now.utc.to_i / 2592000 # monthly + end + require 'securerandom' require 'digest/sha2' - class Token < Hash + class DummyRecord < Hash def initialize - self['token'] = SecureRandom.urlsafe_base64(32).gsub(/^_*/, '') - self['_id'] = Digest::SHA512.hexdigest(self['token']) - self['last_seen_at'] = Time.now + self['data'] = SecureRandom.urlsafe_base64(32).gsub(/^_*/, '') + self['_id'] = Digest::SHA512.hexdigest(self['data']) end end diff --git a/tests/white-box/webapp.rb b/tests/white-box/webapp.rb index 9f104899..1e78c8a5 100644 --- a/tests/white-box/webapp.rb +++ b/tests/white-box/webapp.rb @@ -42,11 +42,16 @@ class Webapp < LeapTest end def test_05_Can_create_and_authenticate_and_delete_user_via_API? - assert_tmp_user - pass + if property('webapp.allow_registration') + assert_tmp_user + pass + else + skip "New user registrations are disabled." + end end def test_06_Can_sync_Soledad? + return unless property('webapp.allow_registration') soledad_config = property('definition_files.soledad_service') if soledad_config && !soledad_config.empty? soledad_server = pick_soledad_server(soledad_config) @@ -96,14 +101,14 @@ class Webapp < LeapTest def assert_user_db_exists(user) last_body, last_response, last_error = nil 3.times do - sleep 0.1 + sleep 0.2 get(couchdb_url("/user-#{user.id}/_design/docs")) do |body, response, error| last_body, last_response, last_error = body, response, error if response.code.to_i == 200 return end end - sleep 0.2 + sleep 0.5 end assert false, "Could not find user db for test user #{user.username}\nuuid=#{user.id}\nHTTP #{last_response.code} #{last_error} #{last_body}" end diff --git a/vagrant/configure-leap.sh b/vagrant/configure-leap.sh new file mode 100755 index 00000000..9541e194 --- /dev/null +++ b/vagrant/configure-leap.sh @@ -0,0 +1,83 @@ +#!/bin/bash + + +. /vagrant/vagrant/vagrant.config + +#OPTS='--no-color' +OPTS='' +PROVIDERDIR='/srv/leap/configuration' +NODE='node1' +LEAP='/usr/local/bin/leap' + +echo '===============================================' +echo 'configuring leap' +echo '===============================================' + +# purge $PROVIDERDIR so this script can be run multiple times +[ -e $PROVIDERDIR ] && rm -rf $PROVIDERDIR +mkdir $PROVIDERDIR +cd $PROVIDERDIR + +$LEAP $OPTS new --contacts "$contacts" --domain "$provider_domain" --name "$provider_name" --platform=/vagrant . +echo -e '\n@log = "/var/log/leap/deploy.log"' >> Leapfile + +if [ ! -e /root/.ssh/id_rsa ]; then + ssh-keygen -f /root/.ssh/id_rsa -P '' + cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys +fi + +mkdir -p $PROVIDERDIR/files/nodes/$NODE +sh -c "cat /etc/ssh/ssh_host_rsa_key.pub | cut -d' ' -f1,2 >> $PROVIDERDIR/files/nodes/$NODE/${NODE}_ssh.pub" + +$LEAP $OPTS add-user --self +$LEAP $OPTS cert ca +$LEAP $OPTS cert csr +$LEAP $OPTS node add $NODE ip_address:"$(facter ipaddress)" services:"$services" tags:production +echo '{ "webapp": { "admins": ["testadmin"] } }' > services/webapp.json + +$LEAP $OPTS compile + +git init +git add . +git commit -m'configured provider' + +$LEAP $OPTS node init $NODE +if [ $? -eq 1 ]; then + echo 'node init failed' + exit 1 +fi + +$LEAP $OPTS -v 2 deploy +if [ $? -eq 1 ]; then + echo 'deploy failed' + exit 1 +fi + +set +e +git add . +git commit -m'initialized and deployed provider' +set -e + +echo '===============================================' +echo 'testing the platform' +echo '===============================================' + +$LEAP $OPTS -v 2 test --continue + +echo '===============================================' +echo 'setting node to demo-mode' +echo '===============================================' +postconf -e default_transport='error: in demo mode' + +sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/g' /etc/ssh/sshd_config +/etc/init.d/ssh reload + +# add users: testadmin and testuser with passwords "hallo123" +curl -s -k https://localhost/1/users.json -d "user%5Blogin%5D=testuser&user%5Bpassword_salt%5D=7d4880237a038e0e&user%5Bpassword_verifier%5D=b98dc393afcd16e5a40fb57ce9cddfa6a978b84be326196627c111d426cada898cdaf3a6427e98b27daf4b0ed61d278bc856515aeceb2312e50c8f816659fcaa4460d839a1e2d7ffb867d32ac869962061368141c7571a53443d58dc84ca1fca34776894414c1090a93e296db6cef12c2cc3f7a991b05d49728ed358fd868286" +curl -s -k https://localhost/1/users.json -d "user%5Blogin%5D=testadmin&user%5Bpassword_salt%5D=ece1c457014d8282&user%5Bpassword_verifier%5D=9654d93ab409edf4ff1543d07e08f321107c3fd00de05c646c637866a94f28b3eb263ea9129dacebb7291b3374cc6f0bf88eb3d231eb3a76eed330a0e8fd2a5c477ed2693694efc1cc23ae83c2ae351a21139701983dd595b6c3225a1bebd2a4e6122f83df87606f1a41152d9890e5a11ac3749b3bfcf4407fc83ef60b4ced68" + +echo -e '\n\n\n' +echo 'You are now ready to use your provider. Please update your /etc/hosts with following dns overrides:' + +$LEAP list --print ip_address,domain.full,dns.aliases | sed 's/,//g' | cut -d' ' -f 2- + diff --git a/vagrant/install-platform.pp b/vagrant/install-platform.pp new file mode 100755 index 00000000..465ca78a --- /dev/null +++ b/vagrant/install-platform.pp @@ -0,0 +1,36 @@ +class {'apt': } +File['/etc/apt/preferences'] -> + Exec['refresh_apt'] -> + Package <| ( title != 'lsb' ) |> + +package { [ 'rsync', 'ruby-hiera-puppet', 'git', 'ruby1.9.1-dev', 'rake', 'jq' ]: + ensure => installed +} + +file { '/etc/gemrc': + content => "---\n:sources:\n - https://rubygems.org/" +} + +vcsrepo { '/srv/leap/leap_cli': + ensure => present, + force => true, + revision => 'develop', + provider => 'git', + source => 'https://leap.se/git/leap_cli.git', + owner => 'root', + group => 'root', + notify => Exec['install_leap_cli'], + require => Package['git'] +} + +exec { 'install_leap_cli': + command => '/usr/bin/rake build && /usr/bin/rake install', + cwd => '/srv/leap/leap_cli', + refreshonly => true, + require => [ Package['ruby1.9.1-dev'], File['/etc/gemrc'], Package['rake'] ] +} + +file { [ '/srv/leap', '/srv/leap/configuration', '/var/log/leap' ]: + ensure => directory +} + diff --git a/vagrant/offlineimaprc.example.org b/vagrant/offlineimaprc.example.org new file mode 100644 index 00000000..3d119634 --- /dev/null +++ b/vagrant/offlineimaprc.example.org @@ -0,0 +1,24 @@ +# WARNING: Use offlineimap *only* for testing/debugging, +# because it will save the mails *decrypted* locally to +# your disk ! + +[general] +accounts = testuser@example.org + +[Account testuser@example.org] +localrepository = testuser@example.org_local +remoterepository = testuser@example.org_remote + +[Repository testuser@example.org_local] +type = Maildir +localfolders = /tmp/offlineimap.testuser@example.org + +[Repository testuser@example.org_remote] +type = IMAP +remotehost = localhost +remoteuser = testuser@example.org +remoteport = 1984 +ssl = no +remotepass = every_pw_works_here + + diff --git a/vagrant/vagrant.config b/vagrant/vagrant.config new file mode 100644 index 00000000..ae124246 --- /dev/null +++ b/vagrant/vagrant.config @@ -0,0 +1,12 @@ +# config values used by configure-leap.sh +provider_domain='example.org' +provider_name='Leap Example Provider' +contacts="no-reply@$provider_domain" + +# serivces that get configured +# note that the "openvpn" service does currently *not* work +# in a vagrant setup, +# see https://leap.se/en/docs/platform/troubleshooting/known-issues#Special.Environments +services='webapp,mx,couchdb,soledad,monitor' + + |