diff options
Diffstat (limited to 'doc/troubleshooting/where-to-look.md')
| -rw-r--r-- | doc/troubleshooting/where-to-look.md | 267 |
1 files changed, 0 insertions, 267 deletions
diff --git a/doc/troubleshooting/where-to-look.md b/doc/troubleshooting/where-to-look.md deleted file mode 100644 index c92fba8f..00000000 --- a/doc/troubleshooting/where-to-look.md +++ /dev/null @@ -1,267 +0,0 @@ -@title = 'Where to look for errors' -@nav_title = 'Where to look' -@toc = true - - -General -======= - -* Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. `-v 5` after the `leap` cmd, i.e. `leap -v 2 deploy`. -* We use the `example.org` domain for documentation purposes here, please replace it with the you domain. - -Firewall -======================= - -Every node in your provider has its own restrictive firewall, but you might have a network firewall in place as well that is not managed by LEAP platform. To see what ports and addresses must be open, run this command: - - workstation$ leap compile firewall - -If any of those are blocked, then your provider will not work. - -Webapp -====== - -Places to look for errors -------------------------- - -* `/var/log/apache2/error.log` -* `/srv/leap/webapp/log/production.log` -* `/var/log/syslog` (watch out for stunnel issues) -* `/var/log/leap/*` - - -Is haproxy ok ? ---------------- - - curl -s -X GET "http://127.0.0.1:4096" - -Is couchdb accessible through stunnel ? ---------------------------------------- - -* Depending on how many couch nodes you have, increase the port for every test - (see /etc/haproxy/haproxy.cfg for the server/port mapping): - - - curl -s -X GET "http://127.0.0.1:4000" - curl -s -X GET "http://127.0.0.1:4001" - ... - - -Check couchdb acl as admin --------------------------- - - mkdir /etc/couchdb - cat /srv/leap/webapp/config/couchdb.yml.admin # see username and password - echo "machine 127.0.0.1 login admin password <PASSWORD>" > /etc/couchdb/couchdb-admin.netrc - chmod 600 /etc/couchdb/couchdb-admin.netrc - - curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096" - curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096/_all_dbs" - -Check couchdb acl as unpriviledged user ---------------------------------------- - - cat /srv/leap/webapp/config/couchdb.yml # see username and password - echo "machine 127.0.0.1 login webapp password <PASSWORD>" > /etc/couchdb/couchdb-webapp.netrc - chmod 600 /etc/couchdb/couchdb-webapp.netrc - - curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096" - curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096/_all_dbs" - - -All URLs accessible ? ---------------------- - -* https://example.org -* https://api.example.org:4430/provider.json -* https://example.org/ca.crt - - -Check client config files -------------------------- - -* https://example.net/provider.json -* https://example.net/1/config/smtp-service.json -* https://example.net/1/config/soledad-service.json -* https://example.net/1/config/eip-service.json - - -Soledad -======= - - /var/log/soledad.log - - -Couchdb -======= - -Places to look for errors -------------------------- - -* `/opt/bigcouch/var/log/bigcouch.log` -* `/var/log/syslog` (watch out for stunnel issues) - - - -Bigcouch membership -------------------- - -* All nodes configured for the provider should appear here: - -<pre> - curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5986/nodes/_all_docs' -</pre> - -* All configured nodes should show up under "cluster_nodes", and the ones online and communicating with each other should appear under "all_nodes". This example output shows the configured cluster nodes `couch1.bitmask.net` and `couch2.bitmask.net`, but `couch2.bitmask.net` is currently not accessible from `couch1.bitmask.net` - - -<pre> - curl -s --netrc-file /etc/couchdb/couchdb.netrc 'http://127.0.0.1:5984/_membership' - {"all_nodes":["bigcouch@couch1.bitmask.net"],"cluster_nodes":["bigcouch@couch1.bitmask.net","bigcouch@couch2.bitmask.net"]} -</pre> - -* Sometimes a `/etc/init.d/bigcouch restart` on all nodes is needed, to register new nodes - -Databases ---------- - -* Following output shows all neccessary DBs that should be present. Note that the `user-0123456....` DBs are the data stores for a particular user. - -<pre> - curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_all_dbs' - ["customers","identities","sessions","shared","tickets","tokens","user-0","user-9d34680b01074c75c2ec58c7321f540c","user-9d34680b01074c75c2ec58c7325fb7ff","users"] -</pre> - - - - -Design Documents ----------------- - -* Is User `_design doc` available ? - - -<pre> - curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET "http://127.0.0.1:5984/users/_design/User" -</pre> - -Is couchdb cluster backend accessible through stunnel ? -------------------------------------------------------- - -* Find out how many connections are set up for the couchdb cluster backend: - -<pre> - grep "accept = 127.0.0.1" /etc/stunnel/* -</pre> - - -* Now connect to all of those local endpoints to see if they up. All these tests should return "localhost [127.0.0.1] 4000 (?) open" - -<pre> - nc -v 127.0.0.1 4000 - nc -v 127.0.0.1 4001 - ... -</pre> - - -MX -== - -Places to look for errors -------------------------- - -* `/var/log/mail.log` -* `/var/log/leap_mx.log` -* `/var/log/syslog` (watch out for stunnel issues) - -Is couchdb accessible through stunnel ? ---------------------------------------- - -* Depending on how many couch nodes you have, increase the port for every test - (see /etc/haproxy/haproxy.cfg for the server/port mapping): - - - curl -s -X GET "http://127.0.0.1:4000" - curl -s -X GET "http://127.0.0.1:4001" - ... - -Query leap-mx -------------- - -* for useraccount - - -<pre> - postmap -v -q "joe@dev.bitmask.net" tcp:localhost:2244 - ... - postmap: dict_tcp_lookup: send: get jow@dev.bitmask.net - postmap: dict_tcp_lookup: recv: 200 - ... -</pre> - -* for mailalias - - -<pre> - postmap -v -q "joe@dev.bitmask.net" tcp:localhost:4242 - ... - postmap: dict_tcp_lookup: send: get joe@dev.bitmask.net - postmap: dict_tcp_lookup: recv: 200 f01bc1c70de7d7d80bc1ad77d987e73a - postmap: dict_tcp_lookup: found: f01bc1c70de7d7d80bc1ad77d987e73a - f01bc1c70de7d7d80bc1ad77d987e73a - ... -</pre> - - -Check couchdb acl as unpriviledged user ---------------------------------------- - - - - cat /etc/leap/mx.conf # see username and password - echo "machine 127.0.0.1 login leap_mx password <PASSWORD>" > /etc/couchdb/couchdb-leap_mx.netrc - chmod 600 /etc/couchdb/couchdb-leap_mx.netrc - - curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/_all_dbs" # pick one "user-<hash>" db - curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/user-de9c77a3d7efbc779c6c20da88e8fb9c" - - -* you may check multiple times, cause 127.0.0.1:4096 is haproxy load-balancing the different couchdb nodes - - -Mailspool ---------- - -* Any file in the leap_mx mailspool longer for a few seconds ? - - - -<pre> - ls -la /var/mail/vmail/Maildir/cur/ -</pre> - -* Any mails in postfix mailspool longer than a few seconds ? - -<pre> - mailq -</pre> - - - -Testing mail delivery ---------------------- - - swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 25 - swaks -f varac@cdev.bitmask.net -t varac@cdev.bitmask.net -s chipmonk.cdev.bitmask.net --port 465 --tlsc - swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 587 --tls - - -VPN -=== - -Places to look for errors -------------------------- - -* `/var/log/syslog` (watch out for openvpn issues) - - |