diff options
| author | Micah Anderson <micah@riseup.net> | 2016-11-04 10:54:28 -0400 |
|---|---|---|
| committer | Micah Anderson <micah@riseup.net> | 2016-11-04 10:54:28 -0400 |
| commit | 34a381efa8f6295080c843f86bfa07d4e41056af (patch) | |
| tree | 9282cf5d4c876688602705a7fa0002bc4a810bde /docs/en/troubleshooting | |
| parent | 0a72bc6fd292bf9367b314fcb0347c4d35042f16 (diff) | |
| parent | 5821964ff7e16ca7aa9141bd09a77d355db492a9 (diff) | |
Merge branch 'develop'
Diffstat (limited to 'docs/en/troubleshooting')
| -rw-r--r-- | docs/en/troubleshooting/known-issues.html | 238 | ||||
| -rw-r--r-- | docs/en/troubleshooting/known-issues/index.html | 238 | ||||
| -rw-r--r-- | docs/en/troubleshooting/tests.html | 201 | ||||
| -rw-r--r-- | docs/en/troubleshooting/tests/index.html | 201 | ||||
| -rw-r--r-- | docs/en/troubleshooting/where-to-look.html | 451 | ||||
| -rw-r--r-- | docs/en/troubleshooting/where-to-look/index.html | 451 |
6 files changed, 1780 insertions, 0 deletions
diff --git a/docs/en/troubleshooting/known-issues.html b/docs/en/troubleshooting/known-issues.html new file mode 100644 index 00000000..607970b1 --- /dev/null +++ b/docs/en/troubleshooting/known-issues.html @@ -0,0 +1,238 @@ +<!DOCTYPE html> +<html lang='en'> +<head> +<title> +Known issues - LEAP Platform Documentation +</title> +<meta content='width=device-width, initial-scale=1.0' name='viewport'> +<meta charset='UTF-8'> +<base href="" /> +<style> + body { + background: #444; + display: flex; + flex-direction: row; + padding: 10px; + margin: 0px; + } + #sidebar { + flex: 0 0 250px; + background: white; + margin-right: 10px; + padding: 20px; + } + #sidebar ul { + list-style-type: none; + padding-left: 0px; + margin: 0; + } + #sidebar li { padding: 4px } + #sidebar li a { text-decoration: none } + #sidebar li.active { background: #444 } + #sidebar li.active a { color: white } + #sidebar li.level1 { padding-left: 20px } + #sidebar li.level2 { padding-left: 40px } + #main { + flex: 1 1 auto; + background: white; + padding: 20px; + } + #title-box { + padding-bottom: 20px; + border-bottom: 5px solid #eee; + } + #title-box h1 { + margin-top: 0px; + } + pre { + padding: 10px; + background: #eef; + } + code { + background: #eef; + } + table {border-collapse: collapse} + table td { + border: 1px solid #ccc; + padding: 4px; + vertical-align: top; + } +</style> +</head> +<body> +<div id='sidebar'> +<ul> +<li class=''> +<a href='../../index.html'>Home</a> +</li> +<li class=' level0'> +<a class='' href='../guide.html'>Guide</a> +</li> +<li class=' level0'> +<a class='' href='../tutorials.html'>Tutorials</a> +</li> +<li class=' level0'> +<a class='' href='../services.html'>Services</a> +</li> +<li class=' level0'> +<a class='' href='../upgrading.html'>Upgrading</a> +</li> +<li class='semi-active level0'> +<a class='' href='../troubleshooting.html'>Troubleshooting</a> +</li> +<li class=' level1'> +<a class='' href='tests.html'>Tests and Monitoring</a> +</li> +<li class='active level1'> +<a class='' href='known-issues.html'>Known issues</a> +</li> +<li class=' level1'> +<a class='' href='where-to-look.html'>Where to look</a> +</li> +<li class=' level0'> +<a class='' href='../details.html'>Details</a> +</li> +</ul> +</div> +<div id='main'> +<div id='title-box'> +<h1>Leap Platform Release Notes</h1> + +<div id='summary'>Known issues in the Leap Platform.</div> +</div> +<div id='content-box'> +<div id="TOC"><ol> + <li> + <a href="known-issues/index.html#060">0.6.0</a> + <ol> + <li> + <a href="known-issues/index.html#upgrading">Upgrading</a> + </li> + <li> + <a href="known-issues/index.html#openvpn">OpenVPN</a> + </li> + <li> + <a href="known-issues/index.html#couchdb">CouchDB</a> + </li> + <li> + <a href="known-issues/index.html#user-setup-and-ssh">User setup and ssh</a> + </li> + <li> + <a href="known-issues/index.html#deploying">Deploying</a> + </li> + <li> + <a href="known-issues/index.html#ipv6">IPv6</a> + </li> + <li> + <a href="known-issues/index.html#special-environments">Special Environments</a> + </li> + </ol> + </li> +</ol></div> + +<p>Here you can find documentation about known issues and potential work-arounds in the current Leap Platform release.</p> + +<h1><a name="060"></a>0.6.0</h1> + +<h2><a name="upgrading"></a>Upgrading</h2> + +<p>Upgrade your leap_platform to 0.6 and make sure you have the latest leap_cli.</p> + +<p><strong>Update leap_platform:</strong></p> + +<pre><code>cd leap_platform +git pull +git checkout -b 0.6.0 0.6.0 +</code></pre> + +<p><strong>Update leap_cli:</strong></p> + +<p>If it is installed as a gem from rubygems:</p> + +<pre><code>sudo gem update leap_cli +</code></pre> + +<p>If it is installed as a gem from source:</p> + +<pre><code>cd leap_cli +git pull +git checkout master +rake build +sudo rake install +</code></pre> + +<p>If it is run directly from source:</p> + +<pre><code>cd leap_cli +git pull +git checkout master +</code></pre> + +<p>To upgrade:</p> + +<pre><code>leap --version # must be at least 1.6.2 +leap cert update +leap deploy +leap test +</code></pre> + +<p>If the tests fail, try deploying again. If a test fails because there are two tapicero daemons running, you need to ssh into the server, kill all the tapicero daemons manually, and then try deploying again (sometimes the daemon from platform 0.5 would put its PID file in an odd place).</p> + +<h2><a name="openvpn"></a>OpenVPN</h2> + +<p>On deployment to a openvpn node, if the following happens:</p> + +<pre><code>- err: /Stage[main]/Site_openvpn/Service[openvpn]/ensure: change from stopped to running failed: Could not start Service[openvpn]: Execution of '/etc/init.d/openvpn start' returned 1: at /srv/leap/puppet/modules/site_openvpn/manifests/init.pp:189 +</code></pre> + +<p>this is likely the result of a kernel upgrade that happened during the deployment, requiring that the machine be restarted before this service can start. To confirm this, login to the node (leap ssh <nodename>) and look at the end of the /var/log/daemon.log:</p> + +<pre><code># tail /var/log/daemon.log +Nov 22 19:04:15 snail ovpn-udp_config[16173]: ERROR: Cannot open TUN/TAP dev /dev/net/tun: No such device (errno=19) +Nov 22 19:04:15 snail ovpn-udp_config[16173]: Exiting due to fatal error +</code></pre> + +<p>if you see this error, simply restart the node.</p> + +<h2><a name="couchdb"></a>CouchDB</h2> + +<p>At the moment, we only support one couchdb server for stability purposes.</p> + +<h2><a name="user-setup-and-ssh"></a>User setup and ssh</h2> + +<p>At the moment, it is only possible to add an admin who will have access to all LEAP servers (see: <a href="https://leap.se/code/issues/2280">https://leap.se/code/issues/2280</a>)</p> + +<p>The command <code>leap add-user --self</code> allows only one SSH key. If you want to specify more than one key for a user, you can do it manually:</p> + +<pre><code>users/userx/userx_ssh.pub +users/userx/otherkey_ssh.pub +</code></pre> + +<p>All keys matching ‘userx/*_ssh.pub’ will be used for that user.</p> + +<h2><a name="deploying"></a>Deploying</h2> + +<p>If you have any errors during a run, please try to deploy again as this often solves non-deterministic issues that were not uncovered in our testing. Please re-deploy with <code>leap -v2 deploy</code> to get more verbose logs and capture the complete output to provide to us for debugging.</p> + +<p>If when deploying your debian mirror fails for some reason, network anomoly or the mirror itself is out of date, then platform deployment will not succeed properly. Check the mirror is up and try to deploy again when it is resolved (see: <a href="https://leap.se/code/issues/1091">https://leap.se/code/issues/1091</a>)</p> + +<p>Deployment gives ‘error: in <code>%</code>: too few arguments (ArgumentError)’ - this is because you attempted to do a deploy before initializing a node, please initialize the node first and then do a deploy afterwards (see: <a href="https://leap.se/code/issues/2550">https://leap.se/code/issues/2550</a>)</p> + +<p>This release has no ability to custom configure apt sources or proxies (see: <a href="https://leap.se/code/issues/1971">https://leap.se/code/issues/1971</a>)</p> + +<p>When running a deploy at a verbosity level of 2 and above, you will notice puppet deprecation warnings, these are known and we are working on fixing them</p> + +<h2><a name="ipv6"></a>IPv6</h2> + +<p>As of this release, IPv6 is not supported by the VPN configuration. If IPv6 is detected on your network as a client, it is blocked and instead it should revert to IPv4. We plan on adding IPv6 support in an upcoming release.</p> + +<h2><a name="special-environments"></a>Special Environments</h2> + +<p>When deploying to OpenStack release “nova” or newer, you will need to do an initial deploy, then when it has finished run <code>leap facts update</code> and then deploy again (see: <a href="https://leap.se/code/issues/3020">https://leap.se/code/issues/3020</a>)</p> + +<p>It is not possible to actually use the EIP openvpn server on vagrant nodes (see: <a href="https://leap.se/code/issues/2401">https://leap.se/code/issues/2401</a>)</p> + +</div> +</div> +</body> +</html> diff --git a/docs/en/troubleshooting/known-issues/index.html b/docs/en/troubleshooting/known-issues/index.html new file mode 100644 index 00000000..eee3b120 --- /dev/null +++ b/docs/en/troubleshooting/known-issues/index.html @@ -0,0 +1,238 @@ +<!DOCTYPE html> +<html lang='en'> +<head> +<title> +Known issues - LEAP Platform Documentation +</title> +<meta content='width=device-width, initial-scale=1.0' name='viewport'> +<meta charset='UTF-8'> +<base href="" /> +<style> + body { + background: #444; + display: flex; + flex-direction: row; + padding: 10px; + margin: 0px; + } + #sidebar { + flex: 0 0 250px; + background: white; + margin-right: 10px; + padding: 20px; + } + #sidebar ul { + list-style-type: none; + padding-left: 0px; + margin: 0; + } + #sidebar li { padding: 4px } + #sidebar li a { text-decoration: none } + #sidebar li.active { background: #444 } + #sidebar li.active a { color: white } + #sidebar li.level1 { padding-left: 20px } + #sidebar li.level2 { padding-left: 40px } + #main { + flex: 1 1 auto; + background: white; + padding: 20px; + } + #title-box { + padding-bottom: 20px; + border-bottom: 5px solid #eee; + } + #title-box h1 { + margin-top: 0px; + } + pre { + padding: 10px; + background: #eef; + } + code { + background: #eef; + } + table {border-collapse: collapse} + table td { + border: 1px solid #ccc; + padding: 4px; + vertical-align: top; + } +</style> +</head> +<body> +<div id='sidebar'> +<ul> +<li class=''> +<a href='../../../index.html'>Home</a> +</li> +<li class=' level0'> +<a class='' href='../../guide.html'>Guide</a> +</li> +<li class=' level0'> +<a class='' href='../../tutorials.html'>Tutorials</a> +</li> +<li class=' level0'> +<a class='' href='../../services.html'>Services</a> +</li> +<li class=' level0'> +<a class='' href='../../upgrading.html'>Upgrading</a> +</li> +<li class='semi-active level0'> +<a class='' href='../../troubleshooting.html'>Troubleshooting</a> +</li> +<li class=' level1'> +<a class='' href='../tests.html'>Tests and Monitoring</a> +</li> +<li class='active level1'> +<a class='' href='../known-issues.html'>Known issues</a> +</li> +<li class=' level1'> +<a class='' href='../where-to-look.html'>Where to look</a> +</li> +<li class=' level0'> +<a class='' href='../../details.html'>Details</a> +</li> +</ul> +</div> +<div id='main'> +<div id='title-box'> +<h1>Leap Platform Release Notes</h1> + +<div id='summary'>Known issues in the Leap Platform.</div> +</div> +<div id='content-box'> +<div id="TOC"><ol> + <li> + <a href="index.html#060">0.6.0</a> + <ol> + <li> + <a href="index.html#upgrading">Upgrading</a> + </li> + <li> + <a href="index.html#openvpn">OpenVPN</a> + </li> + <li> + <a href="index.html#couchdb">CouchDB</a> + </li> + <li> + <a href="index.html#user-setup-and-ssh">User setup and ssh</a> + </li> + <li> + <a href="index.html#deploying">Deploying</a> + </li> + <li> + <a href="index.html#ipv6">IPv6</a> + </li> + <li> + <a href="index.html#special-environments">Special Environments</a> + </li> + </ol> + </li> +</ol></div> + +<p>Here you can find documentation about known issues and potential work-arounds in the current Leap Platform release.</p> + +<h1><a name="060"></a>0.6.0</h1> + +<h2><a name="upgrading"></a>Upgrading</h2> + +<p>Upgrade your leap_platform to 0.6 and make sure you have the latest leap_cli.</p> + +<p><strong>Update leap_platform:</strong></p> + +<pre><code>cd leap_platform +git pull +git checkout -b 0.6.0 0.6.0 +</code></pre> + +<p><strong>Update leap_cli:</strong></p> + +<p>If it is installed as a gem from rubygems:</p> + +<pre><code>sudo gem update leap_cli +</code></pre> + +<p>If it is installed as a gem from source:</p> + +<pre><code>cd leap_cli +git pull +git checkout master +rake build +sudo rake install +</code></pre> + +<p>If it is run directly from source:</p> + +<pre><code>cd leap_cli +git pull +git checkout master +</code></pre> + +<p>To upgrade:</p> + +<pre><code>leap --version # must be at least 1.6.2 +leap cert update +leap deploy +leap test +</code></pre> + +<p>If the tests fail, try deploying again. If a test fails because there are two tapicero daemons running, you need to ssh into the server, kill all the tapicero daemons manually, and then try deploying again (sometimes the daemon from platform 0.5 would put its PID file in an odd place).</p> + +<h2><a name="openvpn"></a>OpenVPN</h2> + +<p>On deployment to a openvpn node, if the following happens:</p> + +<pre><code>- err: /Stage[main]/Site_openvpn/Service[openvpn]/ensure: change from stopped to running failed: Could not start Service[openvpn]: Execution of '/etc/init.d/openvpn start' returned 1: at /srv/leap/puppet/modules/site_openvpn/manifests/init.pp:189 +</code></pre> + +<p>this is likely the result of a kernel upgrade that happened during the deployment, requiring that the machine be restarted before this service can start. To confirm this, login to the node (leap ssh <nodename>) and look at the end of the /var/log/daemon.log:</p> + +<pre><code># tail /var/log/daemon.log +Nov 22 19:04:15 snail ovpn-udp_config[16173]: ERROR: Cannot open TUN/TAP dev /dev/net/tun: No such device (errno=19) +Nov 22 19:04:15 snail ovpn-udp_config[16173]: Exiting due to fatal error +</code></pre> + +<p>if you see this error, simply restart the node.</p> + +<h2><a name="couchdb"></a>CouchDB</h2> + +<p>At the moment, we only support one couchdb server for stability purposes.</p> + +<h2><a name="user-setup-and-ssh"></a>User setup and ssh</h2> + +<p>At the moment, it is only possible to add an admin who will have access to all LEAP servers (see: <a href="https://leap.se/code/issues/2280">https://leap.se/code/issues/2280</a>)</p> + +<p>The command <code>leap add-user --self</code> allows only one SSH key. If you want to specify more than one key for a user, you can do it manually:</p> + +<pre><code>users/userx/userx_ssh.pub +users/userx/otherkey_ssh.pub +</code></pre> + +<p>All keys matching ‘userx/*_ssh.pub’ will be used for that user.</p> + +<h2><a name="deploying"></a>Deploying</h2> + +<p>If you have any errors during a run, please try to deploy again as this often solves non-deterministic issues that were not uncovered in our testing. Please re-deploy with <code>leap -v2 deploy</code> to get more verbose logs and capture the complete output to provide to us for debugging.</p> + +<p>If when deploying your debian mirror fails for some reason, network anomoly or the mirror itself is out of date, then platform deployment will not succeed properly. Check the mirror is up and try to deploy again when it is resolved (see: <a href="https://leap.se/code/issues/1091">https://leap.se/code/issues/1091</a>)</p> + +<p>Deployment gives ‘error: in <code>%</code>: too few arguments (ArgumentError)’ - this is because you attempted to do a deploy before initializing a node, please initialize the node first and then do a deploy afterwards (see: <a href="https://leap.se/code/issues/2550">https://leap.se/code/issues/2550</a>)</p> + +<p>This release has no ability to custom configure apt sources or proxies (see: <a href="https://leap.se/code/issues/1971">https://leap.se/code/issues/1971</a>)</p> + +<p>When running a deploy at a verbosity level of 2 and above, you will notice puppet deprecation warnings, these are known and we are working on fixing them</p> + +<h2><a name="ipv6"></a>IPv6</h2> + +<p>As of this release, IPv6 is not supported by the VPN configuration. If IPv6 is detected on your network as a client, it is blocked and instead it should revert to IPv4. We plan on adding IPv6 support in an upcoming release.</p> + +<h2><a name="special-environments"></a>Special Environments</h2> + +<p>When deploying to OpenStack release “nova” or newer, you will need to do an initial deploy, then when it has finished run <code>leap facts update</code> and then deploy again (see: <a href="https://leap.se/code/issues/3020">https://leap.se/code/issues/3020</a>)</p> + +<p>It is not possible to actually use the EIP openvpn server on vagrant nodes (see: <a href="https://leap.se/code/issues/2401">https://leap.se/code/issues/2401</a>)</p> + +</div> +</div> +</body> +</html> diff --git a/docs/en/troubleshooting/tests.html b/docs/en/troubleshooting/tests.html new file mode 100644 index 00000000..e4c2fdc2 --- /dev/null +++ b/docs/en/troubleshooting/tests.html @@ -0,0 +1,201 @@ +<!DOCTYPE html> +<html lang='en'> +<head> +<title> +Tests and Monitoring - LEAP Platform Documentation +</title> +<meta content='width=device-width, initial-scale=1.0' name='viewport'> +<meta charset='UTF-8'> +<base href="" /> +<style> + body { + background: #444; + display: flex; + flex-direction: row; + padding: 10px; + margin: 0px; + } + #sidebar { + flex: 0 0 250px; + background: white; + margin-right: 10px; + padding: 20px; + } + #sidebar ul { + list-style-type: none; + padding-left: 0px; + margin: 0; + } + #sidebar li { padding: 4px } + #sidebar li a { text-decoration: none } + #sidebar li.active { background: #444 } + #sidebar li.active a { color: white } + #sidebar li.level1 { padding-left: 20px } + #sidebar li.level2 { padding-left: 40px } + #main { + flex: 1 1 auto; + background: white; + padding: 20px; + } + #title-box { + padding-bottom: 20px; + border-bottom: 5px solid #eee; + } + #title-box h1 { + margin-top: 0px; + } + pre { + padding: 10px; + background: #eef; + } + code { + background: #eef; + } + table {border-collapse: collapse} + table td { + border: 1px solid #ccc; + padding: 4px; + vertical-align: top; + } +</style> +</head> +<body> +<div id='sidebar'> +<ul> +<li class=''> +<a href='../../index.html'>Home</a> +</li> +<li class=' level0'> +<a class='' href='../guide.html'>Guide</a> +</li> +<li class=' level0'> +<a class='' href='../tutorials.html'>Tutorials</a> +</li> +<li class=' level0'> +<a class='' href='../services.html'>Services</a> +</li> +<li class=' level0'> +<a class='' href='../upgrading.html'>Upgrading</a> +</li> +<li class='semi-active level0'> +<a class='' href='../troubleshooting.html'>Troubleshooting</a> +</li> +<li class='active level1'> +<a class='' href='tests.html'>Tests and Monitoring</a> +</li> +<li class=' level1'> +<a class='' href='known-issues.html'>Known issues</a> +</li> +<li class=' level1'> +<a class='' href='where-to-look.html'>Where to look</a> +</li> +<li class=' level0'> +<a class='' href='../details.html'>Details</a> +</li> +</ul> +</div> +<div id='main'> +<div id='title-box'> +<h1>Tests and Monitoring</h1> + +<div id='summary'>Testing and monitoring your infrastructure.</div> +</div> +<div id='content-box'> +<div id="TOC"><ol> + <li> + <a href="tests/index.html#troubleshooting-tests">Troubleshooting Tests</a> + </li> + <li> + <a href="tests/index.html#testing-with-the-bitmask-client">Testing with the bitmask client</a> + </li> + <li> + <a href="tests/index.html#testing-recieving-mail">Testing Recieving Mail</a> + </li> + <li> + <a href="tests/index.html#monitoring">Monitoring</a> + </li> + <li> + <a href="tests/index.html#nagios-frontends">Nagios Frontends</a> + <ol> + <li> + <a href="tests/index.html#log-monitoring">Log Monitoring</a> + </li> + </ol> + </li> +</ol></div> + +<h2><a name="troubleshooting-tests"></a>Troubleshooting Tests</h2> + +<p>At any time, you can run troubleshooting tests on the nodes of your provider infrastructure to check to see if things seem to be working correctly. If there is a problem, these tests should help you narrow down precisely where the problem is.</p> + +<p>To run tests on FILTER node list:</p> + +<pre><code>workstation$ leap test run FILTER +</code></pre> + +<p>For example, you can also test a single node (<code>leap test elephant</code>); test a specific environment (<code>leap test development</code>), or any tag (<code>leap test soledad</code>).</p> + +<p>Alternately, you can run test on all nodes (probably only useful if you have pinned the environment):</p> + +<pre><code>workstation$ leap test +</code></pre> + +<p>The tests that are performed are located in the platform under the tests directory.</p> + +<h2><a name="testing-with-the-bitmask-client"></a>Testing with the bitmask client</h2> + +<p>Download the provider ca:</p> + +<pre><code>wget --no-check-certificate https://example.org/ca.crt -O /tmp/ca.crt +</code></pre> + +<p>Start bitmask:</p> + +<pre><code>bitmask --ca-cert-file /tmp/ca.crt +</code></pre> + +<h2><a name="testing-recieving-mail"></a>Testing Recieving Mail</h2> + +<p>Use i.e. swaks to send a testmail</p> + +<pre><code>swaks -f noone@example.org -t testuser@example.org -s example.org +</code></pre> + +<p>and use your favorite mail client to examine your inbox.</p> + +<p>You can also use <a href="http://offlineimap.org/">offlineimap</a> to fetch mails:</p> + +<pre><code> offlineimap -c vagrant/.offlineimaprc.example.org +</code></pre> + +<p>WARNING: Use offlineimap <em>only</em> for testing/debugging, +because it will save the mails <em>decrypted</em> locally to +your disk !</p> + +<h2><a name="monitoring"></a>Monitoring</h2> + +<p>In order to set up a monitoring node, you simply add a <code>monitor</code> service tag to the node configuration file. It could be combined with any other service, but we propose that you add it to the webapp node, as this already is public accessible via HTTPS.</p> + +<p>After deploying, this node will regularly poll every node to ask for the status of various health checks. These health checks include the checks run with <code>leap test</code>, plus many others.</p> + +<p>We use <a href="https://www.nagios.org/">Nagios</a> together with <a href="https://en.wikipedia.org/wiki/Check_MK">Check MK agent</a> for running checks on remote hosts.</p> + +<p>One nagios installation will monitor all nodes in all your environments. You can log into the monitoring web interface via <a href="https://DOMAIN/nagios3/">https://DOMAIN/nagios3/</a>. The username is <code>nagiosadmin</code> and the password is found in the secrets.json file in your provider directory. +Nagios will send out mails to the <code>contacts</code> address provided in <code>provider.json</code>.</p> + +<h2><a name="nagios-frontends"></a>Nagios Frontends</h2> + +<p>There are other ways to check and get notified by Nagios besides regularly checking the Nagios webinterface or reading email notifications. Check out the <a href="http://exchange.nagios.org/directory/Addons/Frontends-%28GUIs-and-CLIs%29">Frontends (GUIs and CLIs)</a> on the Nagios project website. +A recommended status tray application is <a href="https://nagstamon.ifw-dresden.de/">Nagstamon</a>, which is available for Linux, MacOS X and Windows. It can not only notify you of hosts/services failures, you can also acknowledge or recheck them.</p> + +<h3><a name="log-monitoring"></a>Log Monitoring</h3> + +<p>At the moment, we use <a href="https://mathias-kettner.de/checkmk_check_logwatch.html">check-mk-agent-logwatch</a> for searching logs for irregularities. +Logs are parsed for patterns using a blacklist, and are stored in <code>/var/lib/check_mk/logwatch/<Nodename></code>.</p> + +<p>In order to “acknowledge” a log warning, you need to log in to the monitoring server, and delete the corresponding file in <code>/var/lib/check_mk/logwatch/<Nodename></code>. This should be done via the nagios webinterface in the future.</p> + +</div> +</div> +</body> +</html> diff --git a/docs/en/troubleshooting/tests/index.html b/docs/en/troubleshooting/tests/index.html new file mode 100644 index 00000000..f46eddc7 --- /dev/null +++ b/docs/en/troubleshooting/tests/index.html @@ -0,0 +1,201 @@ +<!DOCTYPE html> +<html lang='en'> +<head> +<title> +Tests and Monitoring - LEAP Platform Documentation +</title> +<meta content='width=device-width, initial-scale=1.0' name='viewport'> +<meta charset='UTF-8'> +<base href="" /> +<style> + body { + background: #444; + display: flex; + flex-direction: row; + padding: 10px; + margin: 0px; + } + #sidebar { + flex: 0 0 250px; + background: white; + margin-right: 10px; + padding: 20px; + } + #sidebar ul { + list-style-type: none; + padding-left: 0px; + margin: 0; + } + #sidebar li { padding: 4px } + #sidebar li a { text-decoration: none } + #sidebar li.active { background: #444 } + #sidebar li.active a { color: white } + #sidebar li.level1 { padding-left: 20px } + #sidebar li.level2 { padding-left: 40px } + #main { + flex: 1 1 auto; + background: white; + padding: 20px; + } + #title-box { + padding-bottom: 20px; + border-bottom: 5px solid #eee; + } + #title-box h1 { + margin-top: 0px; + } + pre { + padding: 10px; + background: #eef; + } + code { + background: #eef; + } + table {border-collapse: collapse} + table td { + border: 1px solid #ccc; + padding: 4px; + vertical-align: top; + } +</style> +</head> +<body> +<div id='sidebar'> +<ul> +<li class=''> +<a href='../../../index.html'>Home</a> +</li> +<li class=' level0'> +<a class='' href='../../guide.html'>Guide</a> +</li> +<li class=' level0'> +<a class='' href='../../tutorials.html'>Tutorials</a> +</li> +<li class=' level0'> +<a class='' href='../../services.html'>Services</a> +</li> +<li class=' level0'> +<a class='' href='../../upgrading.html'>Upgrading</a> +</li> +<li class='semi-active level0'> +<a class='' href='../../troubleshooting.html'>Troubleshooting</a> +</li> +<li class='active level1'> +<a class='' href='../tests.html'>Tests and Monitoring</a> +</li> +<li class=' level1'> +<a class='' href='../known-issues.html'>Known issues</a> +</li> +<li class=' level1'> +<a class='' href='../where-to-look.html'>Where to look</a> +</li> +<li class=' level0'> +<a class='' href='../../details.html'>Details</a> +</li> +</ul> +</div> +<div id='main'> +<div id='title-box'> +<h1>Tests and Monitoring</h1> + +<div id='summary'>Testing and monitoring your infrastructure.</div> +</div> +<div id='content-box'> +<div id="TOC"><ol> + <li> + <a href="index.html#troubleshooting-tests">Troubleshooting Tests</a> + </li> + <li> + <a href="index.html#testing-with-the-bitmask-client">Testing with the bitmask client</a> + </li> + <li> + <a href="index.html#testing-recieving-mail">Testing Recieving Mail</a> + </li> + <li> + <a href="index.html#monitoring">Monitoring</a> + </li> + <li> + <a href="index.html#nagios-frontends">Nagios Frontends</a> + <ol> + <li> + <a href="index.html#log-monitoring">Log Monitoring</a> + </li> + </ol> + </li> +</ol></div> + +<h2><a name="troubleshooting-tests"></a>Troubleshooting Tests</h2> + +<p>At any time, you can run troubleshooting tests on the nodes of your provider infrastructure to check to see if things seem to be working correctly. If there is a problem, these tests should help you narrow down precisely where the problem is.</p> + +<p>To run tests on FILTER node list:</p> + +<pre><code>workstation$ leap test run FILTER +</code></pre> + +<p>For example, you can also test a single node (<code>leap test elephant</code>); test a specific environment (<code>leap test development</code>), or any tag (<code>leap test soledad</code>).</p> + +<p>Alternately, you can run test on all nodes (probably only useful if you have pinned the environment):</p> + +<pre><code>workstation$ leap test +</code></pre> + +<p>The tests that are performed are located in the platform under the tests directory.</p> + +<h2><a name="testing-with-the-bitmask-client"></a>Testing with the bitmask client</h2> + +<p>Download the provider ca:</p> + +<pre><code>wget --no-check-certificate https://example.org/ca.crt -O /tmp/ca.crt +</code></pre> + +<p>Start bitmask:</p> + +<pre><code>bitmask --ca-cert-file /tmp/ca.crt +</code></pre> + +<h2><a name="testing-recieving-mail"></a>Testing Recieving Mail</h2> + +<p>Use i.e. swaks to send a testmail</p> + +<pre><code>swaks -f noone@example.org -t testuser@example.org -s example.org +</code></pre> + +<p>and use your favorite mail client to examine your inbox.</p> + +<p>You can also use <a href="http://offlineimap.org/">offlineimap</a> to fetch mails:</p> + +<pre><code> offlineimap -c vagrant/.offlineimaprc.example.org +</code></pre> + +<p>WARNING: Use offlineimap <em>only</em> for testing/debugging, +because it will save the mails <em>decrypted</em> locally to +your disk !</p> + +<h2><a name="monitoring"></a>Monitoring</h2> + +<p>In order to set up a monitoring node, you simply add a <code>monitor</code> service tag to the node configuration file. It could be combined with any other service, but we propose that you add it to the webapp node, as this already is public accessible via HTTPS.</p> + +<p>After deploying, this node will regularly poll every node to ask for the status of various health checks. These health checks include the checks run with <code>leap test</code>, plus many others.</p> + +<p>We use <a href="https://www.nagios.org/">Nagios</a> together with <a href="https://en.wikipedia.org/wiki/Check_MK">Check MK agent</a> for running checks on remote hosts.</p> + +<p>One nagios installation will monitor all nodes in all your environments. You can log into the monitoring web interface via <a href="https://DOMAIN/nagios3/">https://DOMAIN/nagios3/</a>. The username is <code>nagiosadmin</code> and the password is found in the secrets.json file in your provider directory. +Nagios will send out mails to the <code>contacts</code> address provided in <code>provider.json</code>.</p> + +<h2><a name="nagios-frontends"></a>Nagios Frontends</h2> + +<p>There are other ways to check and get notified by Nagios besides regularly checking the Nagios webinterface or reading email notifications. Check out the <a href="http://exchange.nagios.org/directory/Addons/Frontends-%28GUIs-and-CLIs%29">Frontends (GUIs and CLIs)</a> on the Nagios project website. +A recommended status tray application is <a href="https://nagstamon.ifw-dresden.de/">Nagstamon</a>, which is available for Linux, MacOS X and Windows. It can not only notify you of hosts/services failures, you can also acknowledge or recheck them.</p> + +<h3><a name="log-monitoring"></a>Log Monitoring</h3> + +<p>At the moment, we use <a href="https://mathias-kettner.de/checkmk_check_logwatch.html">check-mk-agent-logwatch</a> for searching logs for irregularities. +Logs are parsed for patterns using a blacklist, and are stored in <code>/var/lib/check_mk/logwatch/<Nodename></code>.</p> + +<p>In order to “acknowledge” a log warning, you need to log in to the monitoring server, and delete the corresponding file in <code>/var/lib/check_mk/logwatch/<Nodename></code>. This should be done via the nagios webinterface in the future.</p> + +</div> +</div> +</body> +</html> diff --git a/docs/en/troubleshooting/where-to-look.html b/docs/en/troubleshooting/where-to-look.html new file mode 100644 index 00000000..a1207aca --- /dev/null +++ b/docs/en/troubleshooting/where-to-look.html @@ -0,0 +1,451 @@ +<!DOCTYPE html> +<html lang='en'> +<head> +<title> +Where to look - LEAP Platform Documentation +</title> +<meta content='width=device-width, initial-scale=1.0' name='viewport'> +<meta charset='UTF-8'> +<base href="" /> +<style> + body { + background: #444; + display: flex; + flex-direction: row; + padding: 10px; + margin: 0px; + } + #sidebar { + flex: 0 0 250px; + background: white; + margin-right: 10px; + padding: 20px; + } + #sidebar ul { + list-style-type: none; + padding-left: 0px; + margin: 0; + } + #sidebar li { padding: 4px } + #sidebar li a { text-decoration: none } + #sidebar li.active { background: #444 } + #sidebar li.active a { color: white } + #sidebar li.level1 { padding-left: 20px } + #sidebar li.level2 { padding-left: 40px } + #main { + flex: 1 1 auto; + background: white; + padding: 20px; + } + #title-box { + padding-bottom: 20px; + border-bottom: 5px solid #eee; + } + #title-box h1 { + margin-top: 0px; + } + pre { + padding: 10px; + background: #eef; + } + code { + background: #eef; + } + table {border-collapse: collapse} + table td { + border: 1px solid #ccc; + padding: 4px; + vertical-align: top; + } +</style> +</head> +<body> +<div id='sidebar'> +<ul> +<li class=''> +<a href='../../index.html'>Home</a> +</li> +<li class=' level0'> +<a class='' href='../guide.html'>Guide</a> +</li> +<li class=' level0'> +<a class='' href='../tutorials.html'>Tutorials</a> +</li> +<li class=' level0'> +<a class='' href='../services.html'>Services</a> +</li> +<li class=' level0'> +<a class='' href='../upgrading.html'>Upgrading</a> +</li> +<li class='semi-active level0'> +<a class='' href='../troubleshooting.html'>Troubleshooting</a> +</li> +<li class=' level1'> +<a class='' href='tests.html'>Tests and Monitoring</a> +</li> +<li class=' level1'> +<a class='' href='known-issues.html'>Known issues</a> +</li> +<li class='active level1'> +<a class='' href='where-to-look.html'>Where to look</a> +</li> +<li class=' level0'> +<a class='' href='../details.html'>Details</a> +</li> +</ul> +</div> +<div id='main'> +<div id='title-box'> +<h1>Where to look for errors</h1> + +<div id='summary'>The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment.</div> +</div> +<div id='content-box'> +<div id="TOC"><ol> + <li> + <a href="where-to-look/index.html#general">General</a> + </li> + <li> + <a href="where-to-look/index.html#firewall">Firewall</a> + </li> + <li> + <a href="where-to-look/index.html#webapp">Webapp</a> + <ol> + <li> + <a href="where-to-look/index.html#places-to-look-for-errors">Places to look for errors</a> + </li> + <li> + <a href="where-to-look/index.html#is-haproxy-ok">Is haproxy ok ?</a> + </li> + <li> + <a href="where-to-look/index.html#is-couchdb-accessible-through-stunnel">Is couchdb accessible through stunnel ?</a> + </li> + <li> + <a href="where-to-look/index.html#check-couchdb-acl-as-admin">Check couchdb acl as admin</a> + </li> + <li> + <a href="where-to-look/index.html#check-couchdb-acl-as-unpriviledged-user">Check couchdb acl as unpriviledged user</a> + </li> + <li> + <a href="where-to-look/index.html#all-urls-accessible">All URLs accessible ?</a> + </li> + <li> + <a href="where-to-look/index.html#check-client-config-files">Check client config files</a> + </li> + </ol> + </li> + <li> + <a href="where-to-look/index.html#soledad">Soledad</a> + </li> + <li> + <a href="where-to-look/index.html#couchdb">Couchdb</a> + <ol> + <li> + <a href="where-to-look/index.html#places-to-look-for-errors-2">Places to look for errors</a> + </li> + <li> + <a href="where-to-look/index.html#databases">Databases</a> + </li> + <li> + <a href="where-to-look/index.html#design-documents">Design Documents</a> + </li> + <li> + <a href="where-to-look/index.html#is-couchdb-cluster-backend-accessible-through-stunnel">Is couchdb cluster backend accessible through stunnel ?</a> + </li> + </ol> + </li> + <li> + <a href="where-to-look/index.html#mx">MX</a> + <ol> + <li> + <a href="where-to-look/index.html#places-to-look-for-errors-3">Places to look for errors</a> + </li> + <li> + <a href="where-to-look/index.html#is-couchdb-accessible-through-stunnel-2">Is couchdb accessible through stunnel ?</a> + </li> + <li> + <a href="where-to-look/index.html#query-leap-mx">Query leap-mx</a> + </li> + <li> + <a href="where-to-look/index.html#check-couchdb-acl-as-unpriviledged-user-2">Check couchdb acl as unpriviledged user</a> + </li> + <li> + <a href="where-to-look/index.html#mailspool">Mailspool</a> + </li> + <li> + <a href="where-to-look/index.html#testing-mail-delivery">Testing mail delivery</a> + </li> + </ol> + </li> + <li> + <a href="where-to-look/index.html#vpn">VPN</a> + <ol> + <li> + <a href="where-to-look/index.html#places-to-look-for-errors-4">Places to look for errors</a> + </li> + </ol> + </li> +</ol></div> + +<h1><a name="general"></a>General</h1> + +<ul> +<li>Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. <code>-v 5</code> after the <code>leap</code> cmd, i.e. <code>leap -v 2 deploy</code>.</li> +<li>We use the <code>example.org</code> domain for documentation purposes here, please replace it with the you domain.</li> +</ul> + + +<h1><a name="firewall"></a>Firewall</h1> + +<p>Every node in your provider has its own restrictive firewall, but you might have a network firewall in place as well that is not managed by LEAP platform. To see what ports and addresses must be open, run this command:</p> + +<pre><code>workstation$ leap compile firewall +</code></pre> + +<p>If any of those are blocked, then your provider will not work.</p> + +<h1><a name="webapp"></a>Webapp</h1> + +<h2><a name="places-to-look-for-errors"></a>Places to look for errors</h2> + +<ul> +<li><code>/var/log/apache2/error.log</code></li> +<li><code>/srv/leap/webapp/log/production.log</code></li> +<li><code>/var/log/syslog</code> (watch out for stunnel issues)</li> +<li><code>/var/log/leap/*</code></li> +</ul> + + +<h2><a name="is-haproxy-ok"></a>Is haproxy ok ?</h2> + +<pre><code>curl -s -X GET "http://127.0.0.1:4096" +</code></pre> + +<h2><a name="is-couchdb-accessible-through-stunnel"></a>Is couchdb accessible through stunnel ?</h2> + +<ul> +<li><p>Depending on how many couch nodes you have, increase the port for every test +(see /etc/haproxy/haproxy.cfg for the server/port mapping):</p> + +<p> curl -s -X GET “<a href="http://127.0.0.1:4000">http://127.0.0.1:4000</a>” + curl -s -X GET “<a href="http://127.0.0.1:4001">http://127.0.0.1:4001</a>” + …</p></li> +</ul> + + +<h2><a name="check-couchdb-acl-as-admin"></a>Check couchdb acl as admin</h2> + +<pre><code>mkdir /etc/couchdb +cat /srv/leap/webapp/config/couchdb.yml.admin # see username and password +echo "machine 127.0.0.1 login admin password <PASSWORD>" > /etc/couchdb/couchdb-admin.netrc +chmod 600 /etc/couchdb/couchdb-admin.netrc + +curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096" +curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096/_all_dbs" +</code></pre> + +<h2><a name="check-couchdb-acl-as-unpriviledged-user"></a>Check couchdb acl as unpriviledged user</h2> + +<pre><code>cat /srv/leap/webapp/config/couchdb.yml # see username and password +echo "machine 127.0.0.1 login webapp password <PASSWORD>" > /etc/couchdb/couchdb-webapp.netrc +chmod 600 /etc/couchdb/couchdb-webapp.netrc + +curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096" +curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096/_all_dbs" +</code></pre> + +<h2><a name="all-urls-accessible"></a>All URLs accessible ?</h2> + +<ul> +<li><a href="https://example.org">https://example.org</a></li> +<li><a href="https://api.example.org:4430/provider.json">https://api.example.org:4430/provider.json</a></li> +<li><a href="https://example.org/ca.crt">https://example.org/ca.crt</a></li> +</ul> + + +<h2><a name="check-client-config-files"></a>Check client config files</h2> + +<ul> +<li><a href="https://example.net/provider.json">https://example.net/provider.json</a></li> +<li><a href="https://example.net/1/config/smtp-service.json">https://example.net/1/config/smtp-service.json</a></li> +<li><a href="https://example.net/1/config/soledad-service.json">https://example.net/1/config/soledad-service.json</a></li> +<li><a href="https://example.net/1/config/eip-service.json">https://example.net/1/config/eip-service.json</a></li> +</ul> + + +<h1><a name="soledad"></a>Soledad</h1> + +<pre><code>/var/log/soledad.log +</code></pre> + +<h1><a name="couchdb"></a>Couchdb</h1> + +<h2><a name="places-to-look-for-errors-2"></a>Places to look for errors</h2> + +<ul> +<li><code>/var/log/couchdb/couch.log</code></li> +<li><code>/var/log/syslog</code> (watch out for stunnel issues)</li> +</ul> + + +<h2><a name="databases"></a>Databases</h2> + +<ul> +<li>Following output shows all neccessary DBs that should be present. Note that the <code>user-0123456....</code> DBs are the data stores for a particular user.</li> +</ul> + + +<pre> + curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_all_dbs' + ["customers","identities","sessions","shared","tickets","tokens","user-0","user-9d34680b01074c75c2ec58c7321f540c","user-9d34680b01074c75c2ec58c7325fb7ff","users"] +</pre> + + +<h2><a name="design-documents"></a>Design Documents</h2> + +<ul> +<li>Is User <code>_design doc</code> available ?</li> +</ul> + + +<pre> + curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET "http://127.0.0.1:5984/users/_design/User" +</pre> + + +<h2><a name="is-couchdb-cluster-backend-accessible-through-stunnel"></a>Is couchdb cluster backend accessible through stunnel ?</h2> + +<ul> +<li>Find out how many connections are set up for the couchdb cluster backend:</li> +</ul> + + +<pre> + grep "accept = 127.0.0.1" /etc/stunnel/* +</pre> + + +<ul> +<li>Now connect to all of those local endpoints to see if they up. All these tests should return “localhost [127.0.0.1] 4000 (?) open”</li> +</ul> + + +<pre> + nc -v 127.0.0.1 4000 + nc -v 127.0.0.1 4001 + ... +</pre> + + +<h1><a name="mx"></a>MX</h1> + +<h2><a name="places-to-look-for-errors-3"></a>Places to look for errors</h2> + +<ul> +<li><code>/var/log/mail.log</code></li> +<li><code>/var/log/leap_mx.log</code></li> +<li><code>/var/log/syslog</code> (watch out for stunnel issues)</li> +</ul> + + +<h2><a name="is-couchdb-accessible-through-stunnel-2"></a>Is couchdb accessible through stunnel ?</h2> + +<ul> +<li><p>Depending on how many couch nodes you have, increase the port for every test +(see /etc/haproxy/haproxy.cfg for the server/port mapping):</p> + +<p> curl -s -X GET “<a href="http://127.0.0.1:4000">http://127.0.0.1:4000</a>” + curl -s -X GET “<a href="http://127.0.0.1:4001">http://127.0.0.1:4001</a>” + …</p></li> +</ul> + + +<h2><a name="query-leap-mx"></a>Query leap-mx</h2> + +<ul> +<li>for useraccount</li> +</ul> + + +<pre> + postmap -v -q "joe@dev.bitmask.net" tcp:localhost:2244 + ... + postmap: dict_tcp_lookup: send: get jow@dev.bitmask.net + postmap: dict_tcp_lookup: recv: 200 + ... +</pre> + + +<ul> +<li>for mailalias</li> +</ul> + + +<pre> + postmap -v -q "joe@dev.bitmask.net" tcp:localhost:4242 + ... + postmap: dict_tcp_lookup: send: get joe@dev.bitmask.net + postmap: dict_tcp_lookup: recv: 200 f01bc1c70de7d7d80bc1ad77d987e73a + postmap: dict_tcp_lookup: found: f01bc1c70de7d7d80bc1ad77d987e73a + f01bc1c70de7d7d80bc1ad77d987e73a + ... +</pre> + + +<h2><a name="check-couchdb-acl-as-unpriviledged-user-2"></a>Check couchdb acl as unpriviledged user</h2> + +<pre><code>cat /etc/leap/mx.conf # see username and password +echo "machine 127.0.0.1 login leap_mx password <PASSWORD>" > /etc/couchdb/couchdb-leap_mx.netrc +chmod 600 /etc/couchdb/couchdb-leap_mx.netrc + +curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/_all_dbs" # pick one "user-<hash>" db +curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/user-de9c77a3d7efbc779c6c20da88e8fb9c" +</code></pre> + +<ul> +<li>you may check multiple times, cause 127.0.0.1:4096 is haproxy load-balancing the different couchdb nodes</li> +</ul> + + +<h2><a name="mailspool"></a>Mailspool</h2> + +<ul> +<li>Any file in the leap_mx mailspool longer for a few seconds ?</li> +</ul> + + +<pre> + ls -la /var/mail/vmail/Maildir/cur/ +</pre> + + +<ul> +<li>Any mails in postfix mailspool longer than a few seconds ?</li> +</ul> + + +<pre> + mailq +</pre> + + +<h2><a name="testing-mail-delivery"></a>Testing mail delivery</h2> + +<pre><code>swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 25 +swaks -f varac@cdev.bitmask.net -t varac@cdev.bitmask.net -s chipmonk.cdev.bitmask.net --port 465 --tlsc +swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 587 --tls +</code></pre> + +<h1><a name="vpn"></a>VPN</h1> + +<h2><a name="places-to-look-for-errors-4"></a>Places to look for errors</h2> + +<ul> +<li><code>/var/log/syslog</code> (watch out for openvpn issues)</li> +</ul> + + +</div> +</div> +</body> +</html> diff --git a/docs/en/troubleshooting/where-to-look/index.html b/docs/en/troubleshooting/where-to-look/index.html new file mode 100644 index 00000000..ab3115af --- /dev/null +++ b/docs/en/troubleshooting/where-to-look/index.html @@ -0,0 +1,451 @@ +<!DOCTYPE html> +<html lang='en'> +<head> +<title> +Where to look - LEAP Platform Documentation +</title> +<meta content='width=device-width, initial-scale=1.0' name='viewport'> +<meta charset='UTF-8'> +<base href="" /> +<style> + body { + background: #444; + display: flex; + flex-direction: row; + padding: 10px; + margin: 0px; + } + #sidebar { + flex: 0 0 250px; + background: white; + margin-right: 10px; + padding: 20px; + } + #sidebar ul { + list-style-type: none; + padding-left: 0px; + margin: 0; + } + #sidebar li { padding: 4px } + #sidebar li a { text-decoration: none } + #sidebar li.active { background: #444 } + #sidebar li.active a { color: white } + #sidebar li.level1 { padding-left: 20px } + #sidebar li.level2 { padding-left: 40px } + #main { + flex: 1 1 auto; + background: white; + padding: 20px; + } + #title-box { + padding-bottom: 20px; + border-bottom: 5px solid #eee; + } + #title-box h1 { + margin-top: 0px; + } + pre { + padding: 10px; + background: #eef; + } + code { + background: #eef; + } + table {border-collapse: collapse} + table td { + border: 1px solid #ccc; + padding: 4px; + vertical-align: top; + } +</style> +</head> +<body> +<div id='sidebar'> +<ul> +<li class=''> +<a href='../../../index.html'>Home</a> +</li> +<li class=' level0'> +<a class='' href='../../guide.html'>Guide</a> +</li> +<li class=' level0'> +<a class='' href='../../tutorials.html'>Tutorials</a> +</li> +<li class=' level0'> +<a class='' href='../../services.html'>Services</a> +</li> +<li class=' level0'> +<a class='' href='../../upgrading.html'>Upgrading</a> +</li> +<li class='semi-active level0'> +<a class='' href='../../troubleshooting.html'>Troubleshooting</a> +</li> +<li class=' level1'> +<a class='' href='../tests.html'>Tests and Monitoring</a> +</li> +<li class=' level1'> +<a class='' href='../known-issues.html'>Known issues</a> +</li> +<li class='active level1'> +<a class='' href='../where-to-look.html'>Where to look</a> +</li> +<li class=' level0'> +<a class='' href='../../details.html'>Details</a> +</li> +</ul> +</div> +<div id='main'> +<div id='title-box'> +<h1>Where to look for errors</h1> + +<div id='summary'>The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment.</div> +</div> +<div id='content-box'> +<div id="TOC"><ol> + <li> + <a href="index.html#general">General</a> + </li> + <li> + <a href="index.html#firewall">Firewall</a> + </li> + <li> + <a href="index.html#webapp">Webapp</a> + <ol> + <li> + <a href="index.html#places-to-look-for-errors">Places to look for errors</a> + </li> + <li> + <a href="index.html#is-haproxy-ok">Is haproxy ok ?</a> + </li> + <li> + <a href="index.html#is-couchdb-accessible-through-stunnel">Is couchdb accessible through stunnel ?</a> + </li> + <li> + <a href="index.html#check-couchdb-acl-as-admin">Check couchdb acl as admin</a> + </li> + <li> + <a href="index.html#check-couchdb-acl-as-unpriviledged-user">Check couchdb acl as unpriviledged user</a> + </li> + <li> + <a href="index.html#all-urls-accessible">All URLs accessible ?</a> + </li> + <li> + <a href="index.html#check-client-config-files">Check client config files</a> + </li> + </ol> + </li> + <li> + <a href="index.html#soledad">Soledad</a> + </li> + <li> + <a href="index.html#couchdb">Couchdb</a> + <ol> + <li> + <a href="index.html#places-to-look-for-errors-2">Places to look for errors</a> + </li> + <li> + <a href="index.html#databases">Databases</a> + </li> + <li> + <a href="index.html#design-documents">Design Documents</a> + </li> + <li> + <a href="index.html#is-couchdb-cluster-backend-accessible-through-stunnel">Is couchdb cluster backend accessible through stunnel ?</a> + </li> + </ol> + </li> + <li> + <a href="index.html#mx">MX</a> + <ol> + <li> + <a href="index.html#places-to-look-for-errors-3">Places to look for errors</a> + </li> + <li> + <a href="index.html#is-couchdb-accessible-through-stunnel-2">Is couchdb accessible through stunnel ?</a> + </li> + <li> + <a href="index.html#query-leap-mx">Query leap-mx</a> + </li> + <li> + <a href="index.html#check-couchdb-acl-as-unpriviledged-user-2">Check couchdb acl as unpriviledged user</a> + </li> + <li> + <a href="index.html#mailspool">Mailspool</a> + </li> + <li> + <a href="index.html#testing-mail-delivery">Testing mail delivery</a> + </li> + </ol> + </li> + <li> + <a href="index.html#vpn">VPN</a> + <ol> + <li> + <a href="index.html#places-to-look-for-errors-4">Places to look for errors</a> + </li> + </ol> + </li> +</ol></div> + +<h1><a name="general"></a>General</h1> + +<ul> +<li>Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. <code>-v 5</code> after the <code>leap</code> cmd, i.e. <code>leap -v 2 deploy</code>.</li> +<li>We use the <code>example.org</code> domain for documentation purposes here, please replace it with the you domain.</li> +</ul> + + +<h1><a name="firewall"></a>Firewall</h1> + +<p>Every node in your provider has its own restrictive firewall, but you might have a network firewall in place as well that is not managed by LEAP platform. To see what ports and addresses must be open, run this command:</p> + +<pre><code>workstation$ leap compile firewall +</code></pre> + +<p>If any of those are blocked, then your provider will not work.</p> + +<h1><a name="webapp"></a>Webapp</h1> + +<h2><a name="places-to-look-for-errors"></a>Places to look for errors</h2> + +<ul> +<li><code>/var/log/apache2/error.log</code></li> +<li><code>/srv/leap/webapp/log/production.log</code></li> +<li><code>/var/log/syslog</code> (watch out for stunnel issues)</li> +<li><code>/var/log/leap/*</code></li> +</ul> + + +<h2><a name="is-haproxy-ok"></a>Is haproxy ok ?</h2> + +<pre><code>curl -s -X GET "http://127.0.0.1:4096" +</code></pre> + +<h2><a name="is-couchdb-accessible-through-stunnel"></a>Is couchdb accessible through stunnel ?</h2> + +<ul> +<li><p>Depending on how many couch nodes you have, increase the port for every test +(see /etc/haproxy/haproxy.cfg for the server/port mapping):</p> + +<p> curl -s -X GET “<a href="http://127.0.0.1:4000">http://127.0.0.1:4000</a>” + curl -s -X GET “<a href="http://127.0.0.1:4001">http://127.0.0.1:4001</a>” + …</p></li> +</ul> + + +<h2><a name="check-couchdb-acl-as-admin"></a>Check couchdb acl as admin</h2> + +<pre><code>mkdir /etc/couchdb +cat /srv/leap/webapp/config/couchdb.yml.admin # see username and password +echo "machine 127.0.0.1 login admin password <PASSWORD>" > /etc/couchdb/couchdb-admin.netrc +chmod 600 /etc/couchdb/couchdb-admin.netrc + +curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096" +curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096/_all_dbs" +</code></pre> + +<h2><a name="check-couchdb-acl-as-unpriviledged-user"></a>Check couchdb acl as unpriviledged user</h2> + +<pre><code>cat /srv/leap/webapp/config/couchdb.yml # see username and password +echo "machine 127.0.0.1 login webapp password <PASSWORD>" > /etc/couchdb/couchdb-webapp.netrc +chmod 600 /etc/couchdb/couchdb-webapp.netrc + +curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096" +curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096/_all_dbs" +</code></pre> + +<h2><a name="all-urls-accessible"></a>All URLs accessible ?</h2> + +<ul> +<li><a href="https://example.org">https://example.org</a></li> +<li><a href="https://api.example.org:4430/provider.json">https://api.example.org:4430/provider.json</a></li> +<li><a href="https://example.org/ca.crt">https://example.org/ca.crt</a></li> +</ul> + + +<h2><a name="check-client-config-files"></a>Check client config files</h2> + +<ul> +<li><a href="https://example.net/provider.json">https://example.net/provider.json</a></li> +<li><a href="https://example.net/1/config/smtp-service.json">https://example.net/1/config/smtp-service.json</a></li> +<li><a href="https://example.net/1/config/soledad-service.json">https://example.net/1/config/soledad-service.json</a></li> +<li><a href="https://example.net/1/config/eip-service.json">https://example.net/1/config/eip-service.json</a></li> +</ul> + + +<h1><a name="soledad"></a>Soledad</h1> + +<pre><code>/var/log/soledad.log +</code></pre> + +<h1><a name="couchdb"></a>Couchdb</h1> + +<h2><a name="places-to-look-for-errors-2"></a>Places to look for errors</h2> + +<ul> +<li><code>/var/log/couchdb/couch.log</code></li> +<li><code>/var/log/syslog</code> (watch out for stunnel issues)</li> +</ul> + + +<h2><a name="databases"></a>Databases</h2> + +<ul> +<li>Following output shows all neccessary DBs that should be present. Note that the <code>user-0123456....</code> DBs are the data stores for a particular user.</li> +</ul> + + +<pre> + curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_all_dbs' + ["customers","identities","sessions","shared","tickets","tokens","user-0","user-9d34680b01074c75c2ec58c7321f540c","user-9d34680b01074c75c2ec58c7325fb7ff","users"] +</pre> + + +<h2><a name="design-documents"></a>Design Documents</h2> + +<ul> +<li>Is User <code>_design doc</code> available ?</li> +</ul> + + +<pre> + curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET "http://127.0.0.1:5984/users/_design/User" +</pre> + + +<h2><a name="is-couchdb-cluster-backend-accessible-through-stunnel"></a>Is couchdb cluster backend accessible through stunnel ?</h2> + +<ul> +<li>Find out how many connections are set up for the couchdb cluster backend:</li> +</ul> + + +<pre> + grep "accept = 127.0.0.1" /etc/stunnel/* +</pre> + + +<ul> +<li>Now connect to all of those local endpoints to see if they up. All these tests should return “localhost [127.0.0.1] 4000 (?) open”</li> +</ul> + + +<pre> + nc -v 127.0.0.1 4000 + nc -v 127.0.0.1 4001 + ... +</pre> + + +<h1><a name="mx"></a>MX</h1> + +<h2><a name="places-to-look-for-errors-3"></a>Places to look for errors</h2> + +<ul> +<li><code>/var/log/mail.log</code></li> +<li><code>/var/log/leap_mx.log</code></li> +<li><code>/var/log/syslog</code> (watch out for stunnel issues)</li> +</ul> + + +<h2><a name="is-couchdb-accessible-through-stunnel-2"></a>Is couchdb accessible through stunnel ?</h2> + +<ul> +<li><p>Depending on how many couch nodes you have, increase the port for every test +(see /etc/haproxy/haproxy.cfg for the server/port mapping):</p> + +<p> curl -s -X GET “<a href="http://127.0.0.1:4000">http://127.0.0.1:4000</a>” + curl -s -X GET “<a href="http://127.0.0.1:4001">http://127.0.0.1:4001</a>” + …</p></li> +</ul> + + +<h2><a name="query-leap-mx"></a>Query leap-mx</h2> + +<ul> +<li>for useraccount</li> +</ul> + + +<pre> + postmap -v -q "joe@dev.bitmask.net" tcp:localhost:2244 + ... + postmap: dict_tcp_lookup: send: get jow@dev.bitmask.net + postmap: dict_tcp_lookup: recv: 200 + ... +</pre> + + +<ul> +<li>for mailalias</li> +</ul> + + +<pre> + postmap -v -q "joe@dev.bitmask.net" tcp:localhost:4242 + ... + postmap: dict_tcp_lookup: send: get joe@dev.bitmask.net + postmap: dict_tcp_lookup: recv: 200 f01bc1c70de7d7d80bc1ad77d987e73a + postmap: dict_tcp_lookup: found: f01bc1c70de7d7d80bc1ad77d987e73a + f01bc1c70de7d7d80bc1ad77d987e73a + ... +</pre> + + +<h2><a name="check-couchdb-acl-as-unpriviledged-user-2"></a>Check couchdb acl as unpriviledged user</h2> + +<pre><code>cat /etc/leap/mx.conf # see username and password +echo "machine 127.0.0.1 login leap_mx password <PASSWORD>" > /etc/couchdb/couchdb-leap_mx.netrc +chmod 600 /etc/couchdb/couchdb-leap_mx.netrc + +curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/_all_dbs" # pick one "user-<hash>" db +curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/user-de9c77a3d7efbc779c6c20da88e8fb9c" +</code></pre> + +<ul> +<li>you may check multiple times, cause 127.0.0.1:4096 is haproxy load-balancing the different couchdb nodes</li> +</ul> + + +<h2><a name="mailspool"></a>Mailspool</h2> + +<ul> +<li>Any file in the leap_mx mailspool longer for a few seconds ?</li> +</ul> + + +<pre> + ls -la /var/mail/vmail/Maildir/cur/ +</pre> + + +<ul> +<li>Any mails in postfix mailspool longer than a few seconds ?</li> +</ul> + + +<pre> + mailq +</pre> + + +<h2><a name="testing-mail-delivery"></a>Testing mail delivery</h2> + +<pre><code>swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 25 +swaks -f varac@cdev.bitmask.net -t varac@cdev.bitmask.net -s chipmonk.cdev.bitmask.net --port 465 --tlsc +swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 587 --tls +</code></pre> + +<h1><a name="vpn"></a>VPN</h1> + +<h2><a name="places-to-look-for-errors-4"></a>Places to look for errors</h2> + +<ul> +<li><code>/var/log/syslog</code> (watch out for openvpn issues)</li> +</ul> + + +</div> +</div> +</body> +</html> |