From 0a09a6e6f247729457d15480f8d2b9bb0b89ae5e Mon Sep 17 00:00:00 2001 From: elijah Date: Mon, 29 Aug 2016 22:55:41 -0700 Subject: Updated (very out of date) docs and README.md --- docs/en/troubleshooting/known-issues.html | 238 ++++++++++++ docs/en/troubleshooting/known-issues/index.html | 238 ++++++++++++ docs/en/troubleshooting/tests.html | 201 ++++++++++ docs/en/troubleshooting/tests/index.html | 201 ++++++++++ docs/en/troubleshooting/where-to-look.html | 451 +++++++++++++++++++++++ docs/en/troubleshooting/where-to-look/index.html | 451 +++++++++++++++++++++++ 6 files changed, 1780 insertions(+) create mode 100644 docs/en/troubleshooting/known-issues.html create mode 100644 docs/en/troubleshooting/known-issues/index.html create mode 100644 docs/en/troubleshooting/tests.html create mode 100644 docs/en/troubleshooting/tests/index.html create mode 100644 docs/en/troubleshooting/where-to-look.html create mode 100644 docs/en/troubleshooting/where-to-look/index.html (limited to 'docs/en/troubleshooting') diff --git a/docs/en/troubleshooting/known-issues.html b/docs/en/troubleshooting/known-issues.html new file mode 100644 index 00000000..607970b1 --- /dev/null +++ b/docs/en/troubleshooting/known-issues.html @@ -0,0 +1,238 @@ + + + + +Known issues - LEAP Platform Documentation + + + + + + + + +
+
+

Leap Platform Release Notes

+ +
Known issues in the Leap Platform.
+
+
+
    +
  1. + 0.6.0 +
      +
    1. + Upgrading +
      2. +
    2. + OpenVPN +
      3. +
    3. + CouchDB +
      4. +
    4. + User setup and ssh +
      5. +
    5. + Deploying +
      6. +
    6. + IPv6 +
      7. +
    7. + Special Environments +
      8. +
    +
    2. +
+ +

Here you can find documentation about known issues and potential work-arounds in the current Leap Platform release.

+ +

0.6.0

+ +

Upgrading

+ +

Upgrade your leap_platform to 0.6 and make sure you have the latest leap_cli.

+ +

Update leap_platform:

+ +
cd leap_platform
+git pull
+git checkout -b 0.6.0 0.6.0 
+
+ +

Update leap_cli:

+ +

If it is installed as a gem from rubygems:

+ +
sudo gem update leap_cli
+
+ +

If it is installed as a gem from source:

+ +
cd leap_cli
+git pull
+git checkout master
+rake build
+sudo rake install
+
+ +

If it is run directly from source:

+ +
cd leap_cli
+git pull
+git checkout master
+
+ +

To upgrade:

+ +
leap --version  # must be at least 1.6.2
+leap cert update
+leap deploy
+leap test
+
+ +

If the tests fail, try deploying again. If a test fails because there are two tapicero daemons running, you need to ssh into the server, kill all the tapicero daemons manually, and then try deploying again (sometimes the daemon from platform 0.5 would put its PID file in an odd place).

+ +

OpenVPN

+ +

On deployment to a openvpn node, if the following happens:

+ +
- err: /Stage[main]/Site_openvpn/Service[openvpn]/ensure: change from stopped to running failed: Could not start Service[openvpn]: Execution of '/etc/init.d/openvpn start' returned 1:  at /srv/leap/puppet/modules/site_openvpn/manifests/init.pp:189
+
+ +

this is likely the result of a kernel upgrade that happened during the deployment, requiring that the machine be restarted before this service can start. To confirm this, login to the node (leap ssh ) and look at the end of the /var/log/daemon.log:

+ +
# tail /var/log/daemon.log
+Nov 22 19:04:15 snail ovpn-udp_config[16173]: ERROR: Cannot open TUN/TAP dev /dev/net/tun: No such device (errno=19)
+Nov 22 19:04:15 snail ovpn-udp_config[16173]: Exiting due to fatal error
+
+ +

if you see this error, simply restart the node.

+ +

CouchDB

+ +

At the moment, we only support one couchdb server for stability purposes.

+ +

User setup and ssh

+ +

At the moment, it is only possible to add an admin who will have access to all LEAP servers (see: https://leap.se/code/issues/2280)

+ +

The command leap add-user --self allows only one SSH key. If you want to specify more than one key for a user, you can do it manually:

+ +
users/userx/userx_ssh.pub
+users/userx/otherkey_ssh.pub
+
+ +

All keys matching ‘userx/*_ssh.pub’ will be used for that user.

+ +

Deploying

+ +

If you have any errors during a run, please try to deploy again as this often solves non-deterministic issues that were not uncovered in our testing. Please re-deploy with leap -v2 deploy to get more verbose logs and capture the complete output to provide to us for debugging.

+ +

If when deploying your debian mirror fails for some reason, network anomoly or the mirror itself is out of date, then platform deployment will not succeed properly. Check the mirror is up and try to deploy again when it is resolved (see: https://leap.se/code/issues/1091)

+ +

Deployment gives ‘error: in %: too few arguments (ArgumentError)’ - this is because you attempted to do a deploy before initializing a node, please initialize the node first and then do a deploy afterwards (see: https://leap.se/code/issues/2550)

+ +

This release has no ability to custom configure apt sources or proxies (see: https://leap.se/code/issues/1971)

+ +

When running a deploy at a verbosity level of 2 and above, you will notice puppet deprecation warnings, these are known and we are working on fixing them

+ +

IPv6

+ +

As of this release, IPv6 is not supported by the VPN configuration. If IPv6 is detected on your network as a client, it is blocked and instead it should revert to IPv4. We plan on adding IPv6 support in an upcoming release.

+ +

Special Environments

+ +

When deploying to OpenStack release “nova” or newer, you will need to do an initial deploy, then when it has finished run leap facts update and then deploy again (see: https://leap.se/code/issues/3020)

+ +

It is not possible to actually use the EIP openvpn server on vagrant nodes (see: https://leap.se/code/issues/2401)

+ +
+
+ + diff --git a/docs/en/troubleshooting/known-issues/index.html b/docs/en/troubleshooting/known-issues/index.html new file mode 100644 index 00000000..eee3b120 --- /dev/null +++ b/docs/en/troubleshooting/known-issues/index.html @@ -0,0 +1,238 @@ + + + + +Known issues - LEAP Platform Documentation + + + + + + + + +
+
+

Leap Platform Release Notes

+ +
Known issues in the Leap Platform.
+
+
+
    +
  1. + 0.6.0 +
      +
    1. + Upgrading +
      2. +
    2. + OpenVPN +
      3. +
    3. + CouchDB +
      4. +
    4. + User setup and ssh +
      5. +
    5. + Deploying +
      6. +
    6. + IPv6 +
      7. +
    7. + Special Environments +
      8. +
    +
    2. +
+ +

Here you can find documentation about known issues and potential work-arounds in the current Leap Platform release.

+ +

0.6.0

+ +

Upgrading

+ +

Upgrade your leap_platform to 0.6 and make sure you have the latest leap_cli.

+ +

Update leap_platform:

+ +
cd leap_platform
+git pull
+git checkout -b 0.6.0 0.6.0 
+
+ +

Update leap_cli:

+ +

If it is installed as a gem from rubygems:

+ +
sudo gem update leap_cli
+
+ +

If it is installed as a gem from source:

+ +
cd leap_cli
+git pull
+git checkout master
+rake build
+sudo rake install
+
+ +

If it is run directly from source:

+ +
cd leap_cli
+git pull
+git checkout master
+
+ +

To upgrade:

+ +
leap --version  # must be at least 1.6.2
+leap cert update
+leap deploy
+leap test
+
+ +

If the tests fail, try deploying again. If a test fails because there are two tapicero daemons running, you need to ssh into the server, kill all the tapicero daemons manually, and then try deploying again (sometimes the daemon from platform 0.5 would put its PID file in an odd place).

+ +

OpenVPN

+ +

On deployment to a openvpn node, if the following happens:

+ +
- err: /Stage[main]/Site_openvpn/Service[openvpn]/ensure: change from stopped to running failed: Could not start Service[openvpn]: Execution of '/etc/init.d/openvpn start' returned 1:  at /srv/leap/puppet/modules/site_openvpn/manifests/init.pp:189
+
+ +

this is likely the result of a kernel upgrade that happened during the deployment, requiring that the machine be restarted before this service can start. To confirm this, login to the node (leap ssh ) and look at the end of the /var/log/daemon.log:

+ +
# tail /var/log/daemon.log
+Nov 22 19:04:15 snail ovpn-udp_config[16173]: ERROR: Cannot open TUN/TAP dev /dev/net/tun: No such device (errno=19)
+Nov 22 19:04:15 snail ovpn-udp_config[16173]: Exiting due to fatal error
+
+ +

if you see this error, simply restart the node.

+ +

CouchDB

+ +

At the moment, we only support one couchdb server for stability purposes.

+ +

User setup and ssh

+ +

At the moment, it is only possible to add an admin who will have access to all LEAP servers (see: https://leap.se/code/issues/2280)

+ +

The command leap add-user --self allows only one SSH key. If you want to specify more than one key for a user, you can do it manually:

+ +
users/userx/userx_ssh.pub
+users/userx/otherkey_ssh.pub
+
+ +

All keys matching ‘userx/*_ssh.pub’ will be used for that user.

+ +

Deploying

+ +

If you have any errors during a run, please try to deploy again as this often solves non-deterministic issues that were not uncovered in our testing. Please re-deploy with leap -v2 deploy to get more verbose logs and capture the complete output to provide to us for debugging.

+ +

If when deploying your debian mirror fails for some reason, network anomoly or the mirror itself is out of date, then platform deployment will not succeed properly. Check the mirror is up and try to deploy again when it is resolved (see: https://leap.se/code/issues/1091)

+ +

Deployment gives ‘error: in %: too few arguments (ArgumentError)’ - this is because you attempted to do a deploy before initializing a node, please initialize the node first and then do a deploy afterwards (see: https://leap.se/code/issues/2550)

+ +

This release has no ability to custom configure apt sources or proxies (see: https://leap.se/code/issues/1971)

+ +

When running a deploy at a verbosity level of 2 and above, you will notice puppet deprecation warnings, these are known and we are working on fixing them

+ +

IPv6

+ +

As of this release, IPv6 is not supported by the VPN configuration. If IPv6 is detected on your network as a client, it is blocked and instead it should revert to IPv4. We plan on adding IPv6 support in an upcoming release.

+ +

Special Environments

+ +

When deploying to OpenStack release “nova” or newer, you will need to do an initial deploy, then when it has finished run leap facts update and then deploy again (see: https://leap.se/code/issues/3020)

+ +

It is not possible to actually use the EIP openvpn server on vagrant nodes (see: https://leap.se/code/issues/2401)

+ +
+
+ + diff --git a/docs/en/troubleshooting/tests.html b/docs/en/troubleshooting/tests.html new file mode 100644 index 00000000..e4c2fdc2 --- /dev/null +++ b/docs/en/troubleshooting/tests.html @@ -0,0 +1,201 @@ + + + + +Tests and Monitoring - LEAP Platform Documentation + + + + + + + + +
+
+

Tests and Monitoring

+ +
Testing and monitoring your infrastructure.
+
+
+ + +

Troubleshooting Tests

+ +

At any time, you can run troubleshooting tests on the nodes of your provider infrastructure to check to see if things seem to be working correctly. If there is a problem, these tests should help you narrow down precisely where the problem is.

+ +

To run tests on FILTER node list:

+ +
workstation$ leap test run FILTER
+
+ +

For example, you can also test a single node (leap test elephant); test a specific environment (leap test development), or any tag (leap test soledad).

+ +

Alternately, you can run test on all nodes (probably only useful if you have pinned the environment):

+ +
workstation$ leap test
+
+ +

The tests that are performed are located in the platform under the tests directory.

+ +

Testing with the bitmask client

+ +

Download the provider ca:

+ +
wget --no-check-certificate https://example.org/ca.crt -O /tmp/ca.crt
+
+ +

Start bitmask:

+ +
bitmask --ca-cert-file /tmp/ca.crt
+
+ +

Testing Recieving Mail

+ +

Use i.e. swaks to send a testmail

+ +
swaks -f noone@example.org -t testuser@example.org -s example.org
+
+ +

and use your favorite mail client to examine your inbox.

+ +

You can also use offlineimap to fetch mails:

+ +
 offlineimap -c vagrant/.offlineimaprc.example.org
+
+ +

WARNING: Use offlineimap only for testing/debugging, +because it will save the mails decrypted locally to +your disk !

+ +

Monitoring

+ +

In order to set up a monitoring node, you simply add a monitor service tag to the node configuration file. It could be combined with any other service, but we propose that you add it to the webapp node, as this already is public accessible via HTTPS.

+ +

After deploying, this node will regularly poll every node to ask for the status of various health checks. These health checks include the checks run with leap test, plus many others.

+ +

We use Nagios together with Check MK agent for running checks on remote hosts.

+ +

One nagios installation will monitor all nodes in all your environments. You can log into the monitoring web interface via https://DOMAIN/nagios3/. The username is nagiosadmin and the password is found in the secrets.json file in your provider directory. +Nagios will send out mails to the contacts address provided in provider.json.

+ +

Nagios Frontends

+ +

There are other ways to check and get notified by Nagios besides regularly checking the Nagios webinterface or reading email notifications. Check out the Frontends (GUIs and CLIs) on the Nagios project website. +A recommended status tray application is Nagstamon, which is available for Linux, MacOS X and Windows. It can not only notify you of hosts/services failures, you can also acknowledge or recheck them.

+ +

Log Monitoring

+ +

At the moment, we use check-mk-agent-logwatch for searching logs for irregularities. +Logs are parsed for patterns using a blacklist, and are stored in /var/lib/check_mk/logwatch/<Nodename>.

+ +

In order to “acknowledge” a log warning, you need to log in to the monitoring server, and delete the corresponding file in /var/lib/check_mk/logwatch/<Nodename>. This should be done via the nagios webinterface in the future.

+ +
+
+ + diff --git a/docs/en/troubleshooting/tests/index.html b/docs/en/troubleshooting/tests/index.html new file mode 100644 index 00000000..f46eddc7 --- /dev/null +++ b/docs/en/troubleshooting/tests/index.html @@ -0,0 +1,201 @@ + + + + +Tests and Monitoring - LEAP Platform Documentation + + + + + + + + +
+
+

Tests and Monitoring

+ +
Testing and monitoring your infrastructure.
+
+
+ + +

Troubleshooting Tests

+ +

At any time, you can run troubleshooting tests on the nodes of your provider infrastructure to check to see if things seem to be working correctly. If there is a problem, these tests should help you narrow down precisely where the problem is.

+ +

To run tests on FILTER node list:

+ +
workstation$ leap test run FILTER
+
+ +

For example, you can also test a single node (leap test elephant); test a specific environment (leap test development), or any tag (leap test soledad).

+ +

Alternately, you can run test on all nodes (probably only useful if you have pinned the environment):

+ +
workstation$ leap test
+
+ +

The tests that are performed are located in the platform under the tests directory.

+ +

Testing with the bitmask client

+ +

Download the provider ca:

+ +
wget --no-check-certificate https://example.org/ca.crt -O /tmp/ca.crt
+
+ +

Start bitmask:

+ +
bitmask --ca-cert-file /tmp/ca.crt
+
+ +

Testing Recieving Mail

+ +

Use i.e. swaks to send a testmail

+ +
swaks -f noone@example.org -t testuser@example.org -s example.org
+
+ +

and use your favorite mail client to examine your inbox.

+ +

You can also use offlineimap to fetch mails:

+ +
 offlineimap -c vagrant/.offlineimaprc.example.org
+
+ +

WARNING: Use offlineimap only for testing/debugging, +because it will save the mails decrypted locally to +your disk !

+ +

Monitoring

+ +

In order to set up a monitoring node, you simply add a monitor service tag to the node configuration file. It could be combined with any other service, but we propose that you add it to the webapp node, as this already is public accessible via HTTPS.

+ +

After deploying, this node will regularly poll every node to ask for the status of various health checks. These health checks include the checks run with leap test, plus many others.

+ +

We use Nagios together with Check MK agent for running checks on remote hosts.

+ +

One nagios installation will monitor all nodes in all your environments. You can log into the monitoring web interface via https://DOMAIN/nagios3/. The username is nagiosadmin and the password is found in the secrets.json file in your provider directory. +Nagios will send out mails to the contacts address provided in provider.json.

+ +

Nagios Frontends

+ +

There are other ways to check and get notified by Nagios besides regularly checking the Nagios webinterface or reading email notifications. Check out the Frontends (GUIs and CLIs) on the Nagios project website. +A recommended status tray application is Nagstamon, which is available for Linux, MacOS X and Windows. It can not only notify you of hosts/services failures, you can also acknowledge or recheck them.

+ +

Log Monitoring

+ +

At the moment, we use check-mk-agent-logwatch for searching logs for irregularities. +Logs are parsed for patterns using a blacklist, and are stored in /var/lib/check_mk/logwatch/<Nodename>.

+ +

In order to “acknowledge” a log warning, you need to log in to the monitoring server, and delete the corresponding file in /var/lib/check_mk/logwatch/<Nodename>. This should be done via the nagios webinterface in the future.

+ +
+
+ + diff --git a/docs/en/troubleshooting/where-to-look.html b/docs/en/troubleshooting/where-to-look.html new file mode 100644 index 00000000..a1207aca --- /dev/null +++ b/docs/en/troubleshooting/where-to-look.html @@ -0,0 +1,451 @@ + + + + +Where to look - LEAP Platform Documentation + + + + + + + + +
+
+

Where to look for errors

+ +
The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment.
+
+
+ + +

General

+ +
    +
  • Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. -v 5 after the leap cmd, i.e. leap -v 2 deploy.
    • +
  • We use the example.org domain for documentation purposes here, please replace it with the you domain.
    • +
+ + +

Firewall

+ +

Every node in your provider has its own restrictive firewall, but you might have a network firewall in place as well that is not managed by LEAP platform. To see what ports and addresses must be open, run this command:

+ +
workstation$ leap compile firewall
+
+ +

If any of those are blocked, then your provider will not work.

+ +

Webapp

+ +

Places to look for errors

+ +
    +
  • /var/log/apache2/error.log
    • +
  • /srv/leap/webapp/log/production.log
    • +
  • /var/log/syslog (watch out for stunnel issues)
    • +
  • /var/log/leap/*
    • +
+ + +

Is haproxy ok ?

+ +
curl -s -X  GET "http://127.0.0.1:4096"
+
+ +

Is couchdb accessible through stunnel ?

+ +
    +

  • Depending on how many couch nodes you have, increase the port for every test +(see /etc/haproxy/haproxy.cfg for the server/port mapping):

    + +

    curl -s -X GET “http://127.0.0.1:4000” + curl -s -X GET “http://127.0.0.1:4001” + …

    • +
+ + +

Check couchdb acl as admin

+ +
mkdir /etc/couchdb
+cat /srv/leap/webapp/config/couchdb.yml.admin  # see username and password
+echo "machine 127.0.0.1 login admin password <PASSWORD>" > /etc/couchdb/couchdb-admin.netrc
+chmod 600 /etc/couchdb/couchdb-admin.netrc
+
+curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096"
+curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
+
+ +

Check couchdb acl as unpriviledged user

+ +
cat /srv/leap/webapp/config/couchdb.yml  # see username and password
+echo "machine 127.0.0.1 login webapp password <PASSWORD>" > /etc/couchdb/couchdb-webapp.netrc
+chmod 600 /etc/couchdb/couchdb-webapp.netrc
+
+curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096"
+curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
+
+ +

All URLs accessible ?

+ + + + +

Check client config files

+ + + + +

Soledad

+ +
/var/log/soledad.log
+
+ +

Couchdb

+ +

Places to look for errors

+ +
    +
  • /var/log/couchdb/couch.log
    • +
  • /var/log/syslog (watch out for stunnel issues)
    • +
+ + +

Databases

+ +
    +
  • Following output shows all neccessary DBs that should be present. Note that the user-0123456.... DBs are the data stores for a particular user.
    • +
+ + +
+    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_all_dbs'
+    ["customers","identities","sessions","shared","tickets","tokens","user-0","user-9d34680b01074c75c2ec58c7321f540c","user-9d34680b01074c75c2ec58c7325fb7ff","users"]
+
+ + +

Design Documents

+ +
    +
  • Is User _design doc available ?
    • +
+ + +
+    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X  GET "http://127.0.0.1:5984/users/_design/User"
+
+ + +

Is couchdb cluster backend accessible through stunnel ?

+ +
    +
  • Find out how many connections are set up for the couchdb cluster backend:
    • +
+ + +
+    grep "accept = 127.0.0.1" /etc/stunnel/*
+
+ + +
    +
  • Now connect to all of those local endpoints to see if they up. All these tests should return “localhost [127.0.0.1] 4000 (?) open”
    • +
+ + +
+    nc -v 127.0.0.1 4000
+    nc -v 127.0.0.1 4001
+    ...
+
+ + +

MX

+ +

Places to look for errors

+ +
    +
  • /var/log/mail.log
    • +
  • /var/log/leap_mx.log
    • +
  • /var/log/syslog (watch out for stunnel issues)
    • +
+ + +

Is couchdb accessible through stunnel ?

+ +
    +

  • Depending on how many couch nodes you have, increase the port for every test +(see /etc/haproxy/haproxy.cfg for the server/port mapping):

    + +

    curl -s -X GET “http://127.0.0.1:4000” + curl -s -X GET “http://127.0.0.1:4001” + …

    • +
+ + +

Query leap-mx

+ +
    +
  • for useraccount
    • +
+ + +
+    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:2244
+    ...
+    postmap: dict_tcp_lookup: send: get jow@dev.bitmask.net
+    postmap: dict_tcp_lookup: recv: 200
+    ...
+
+ + +
    +
  • for mailalias
    • +
+ + +
+    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:4242
+    ...
+    postmap: dict_tcp_lookup: send: get joe@dev.bitmask.net
+    postmap: dict_tcp_lookup: recv: 200 f01bc1c70de7d7d80bc1ad77d987e73a
+    postmap: dict_tcp_lookup: found: f01bc1c70de7d7d80bc1ad77d987e73a
+    f01bc1c70de7d7d80bc1ad77d987e73a
+    ...
+
+ + +

Check couchdb acl as unpriviledged user

+ +
cat /etc/leap/mx.conf  # see username and password
+echo "machine 127.0.0.1 login leap_mx password <PASSWORD>" > /etc/couchdb/couchdb-leap_mx.netrc
+chmod 600 /etc/couchdb/couchdb-leap_mx.netrc
+
+curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/_all_dbs"   # pick one "user-<hash>" db
+curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/user-de9c77a3d7efbc779c6c20da88e8fb9c"
+
+ +
    +
  • you may check multiple times, cause 127.0.0.1:4096 is haproxy load-balancing the different couchdb nodes
    • +
+ + +

Mailspool

+ +
    +
  • Any file in the leap_mx mailspool longer for a few seconds ?
    • +
+ + +
+    ls -la /var/mail/vmail/Maildir/cur/
+
+ + +
    +
  • Any mails in postfix mailspool longer than a few seconds ?
    • +
+ + +
+    mailq
+
+ + +

Testing mail delivery

+ +
swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 25
+swaks -f varac@cdev.bitmask.net -t varac@cdev.bitmask.net -s chipmonk.cdev.bitmask.net --port 465 --tlsc
+swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 587 --tls
+
+ +

VPN

+ +

Places to look for errors

+ +
    +
  • /var/log/syslog (watch out for openvpn issues)
    • +
+ + +
+
+ + diff --git a/docs/en/troubleshooting/where-to-look/index.html b/docs/en/troubleshooting/where-to-look/index.html new file mode 100644 index 00000000..ab3115af --- /dev/null +++ b/docs/en/troubleshooting/where-to-look/index.html @@ -0,0 +1,451 @@ + + + + +Where to look - LEAP Platform Documentation + + + + + + + + +
+
+

Where to look for errors

+ +
The LEAP Platform is set of complementary packages and server recipes to automate the maintenance of LEAP services in a hardened Debian environment.
+
+
+ + +

General

+ +
    +
  • Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. -v 5 after the leap cmd, i.e. leap -v 2 deploy.
    • +
  • We use the example.org domain for documentation purposes here, please replace it with the you domain.
    • +
+ + +

Firewall

+ +

Every node in your provider has its own restrictive firewall, but you might have a network firewall in place as well that is not managed by LEAP platform. To see what ports and addresses must be open, run this command:

+ +
workstation$ leap compile firewall
+
+ +

If any of those are blocked, then your provider will not work.

+ +

Webapp

+ +

Places to look for errors

+ +
    +
  • /var/log/apache2/error.log
    • +
  • /srv/leap/webapp/log/production.log
    • +
  • /var/log/syslog (watch out for stunnel issues)
    • +
  • /var/log/leap/*
    • +
+ + +

Is haproxy ok ?

+ +
curl -s -X  GET "http://127.0.0.1:4096"
+
+ +

Is couchdb accessible through stunnel ?

+ +
    +

  • Depending on how many couch nodes you have, increase the port for every test +(see /etc/haproxy/haproxy.cfg for the server/port mapping):

    + +

    curl -s -X GET “http://127.0.0.1:4000” + curl -s -X GET “http://127.0.0.1:4001” + …

    • +
+ + +

Check couchdb acl as admin

+ +
mkdir /etc/couchdb
+cat /srv/leap/webapp/config/couchdb.yml.admin  # see username and password
+echo "machine 127.0.0.1 login admin password <PASSWORD>" > /etc/couchdb/couchdb-admin.netrc
+chmod 600 /etc/couchdb/couchdb-admin.netrc
+
+curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096"
+curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
+
+ +

Check couchdb acl as unpriviledged user

+ +
cat /srv/leap/webapp/config/couchdb.yml  # see username and password
+echo "machine 127.0.0.1 login webapp password <PASSWORD>" > /etc/couchdb/couchdb-webapp.netrc
+chmod 600 /etc/couchdb/couchdb-webapp.netrc
+
+curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096"
+curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
+
+ +

All URLs accessible ?

+ + + + +

Check client config files

+ + + + +

Soledad

+ +
/var/log/soledad.log
+
+ +

Couchdb

+ +

Places to look for errors

+ +
    +
  • /var/log/couchdb/couch.log
    • +
  • /var/log/syslog (watch out for stunnel issues)
    • +
+ + +

Databases

+ +
    +
  • Following output shows all neccessary DBs that should be present. Note that the user-0123456.... DBs are the data stores for a particular user.
    • +
+ + +
+    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_all_dbs'
+    ["customers","identities","sessions","shared","tickets","tokens","user-0","user-9d34680b01074c75c2ec58c7321f540c","user-9d34680b01074c75c2ec58c7325fb7ff","users"]
+
+ + +

Design Documents

+ +
    +
  • Is User _design doc available ?
    • +
+ + +
+    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X  GET "http://127.0.0.1:5984/users/_design/User"
+
+ + +

Is couchdb cluster backend accessible through stunnel ?

+ +
    +
  • Find out how many connections are set up for the couchdb cluster backend:
    • +
+ + +
+    grep "accept = 127.0.0.1" /etc/stunnel/*
+
+ + +
    +
  • Now connect to all of those local endpoints to see if they up. All these tests should return “localhost [127.0.0.1] 4000 (?) open”
    • +
+ + +
+    nc -v 127.0.0.1 4000
+    nc -v 127.0.0.1 4001
+    ...
+
+ + +

MX

+ +

Places to look for errors

+ +
    +
  • /var/log/mail.log
    • +
  • /var/log/leap_mx.log
    • +
  • /var/log/syslog (watch out for stunnel issues)
    • +
+ + +

Is couchdb accessible through stunnel ?

+ +
    +

  • Depending on how many couch nodes you have, increase the port for every test +(see /etc/haproxy/haproxy.cfg for the server/port mapping):

    + +

    curl -s -X GET “http://127.0.0.1:4000” + curl -s -X GET “http://127.0.0.1:4001” + …

    • +
+ + +

Query leap-mx

+ +
    +
  • for useraccount
    • +
+ + +
+    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:2244
+    ...
+    postmap: dict_tcp_lookup: send: get jow@dev.bitmask.net
+    postmap: dict_tcp_lookup: recv: 200
+    ...
+
+ + +
    +
  • for mailalias
    • +
+ + +
+    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:4242
+    ...
+    postmap: dict_tcp_lookup: send: get joe@dev.bitmask.net
+    postmap: dict_tcp_lookup: recv: 200 f01bc1c70de7d7d80bc1ad77d987e73a
+    postmap: dict_tcp_lookup: found: f01bc1c70de7d7d80bc1ad77d987e73a
+    f01bc1c70de7d7d80bc1ad77d987e73a
+    ...
+
+ + +

Check couchdb acl as unpriviledged user

+ +
cat /etc/leap/mx.conf  # see username and password
+echo "machine 127.0.0.1 login leap_mx password <PASSWORD>" > /etc/couchdb/couchdb-leap_mx.netrc
+chmod 600 /etc/couchdb/couchdb-leap_mx.netrc
+
+curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/_all_dbs"   # pick one "user-<hash>" db
+curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/user-de9c77a3d7efbc779c6c20da88e8fb9c"
+
+ +
    +
  • you may check multiple times, cause 127.0.0.1:4096 is haproxy load-balancing the different couchdb nodes
    • +
+ + +

Mailspool

+ +
    +
  • Any file in the leap_mx mailspool longer for a few seconds ?
    • +
+ + +
+    ls -la /var/mail/vmail/Maildir/cur/
+
+ + +
    +
  • Any mails in postfix mailspool longer than a few seconds ?
    • +
+ + +
+    mailq
+
+ + +

Testing mail delivery

+ +
swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 25
+swaks -f varac@cdev.bitmask.net -t varac@cdev.bitmask.net -s chipmonk.cdev.bitmask.net --port 465 --tlsc
+swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 587 --tls
+
+ +

VPN

+ +

Places to look for errors

+ +
    +
  • /var/log/syslog (watch out for openvpn issues)
    • +
+ + +
+
+ + -- cgit v1.2.3