@title = 'Troubleshooting Guide'
@nav_title = 'Troubleshooting'
@toc = true


General
=======

* Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. `-v 5` after the `leap` cmd, i.e. `leap -v 2 deploy`.

Webapp node
===========

Places to look for errors
-------------------------

* `/var/log/apache2/error.log`
* `/srv/leap/webapp/log/production.log`
* `/var/log/syslog` (watch out for stunnel issues)

Is haproxy ok ? 
---------------


    curl -s -X  GET "http://127.0.0.1:4096"

Is couchdb accessible through stunnel ?
---------------------------------------


    curl -s -X  GET "http://127.0.0.1:4000"


Check couchdb acl
-----------------


    mkdir /etc/couchdb
    cat /srv/leap/webapp/config/couchdb.yml.admin  # see username and password
    echo "machine 127.0.0.1 login admin password <PASSWORD>" > /etc/couchdb/couchdb-admin.netrc
    chmod 600 /etc/couchdb/couchdb-admin.netrc

    curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096"
    curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
    

Couchdb node
============

Places to look for errors
-------------------------

* `/opt/bigcouch/var/log/bigcouch.log`
* `/var/log/syslog` (watch out for stunnel issues)


Bigcouch membership
-------------------

* All nodes configured for the provider should appear here:


    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5986/nodes/_all_docs'

* All configured nodes should show up under "cluster_nodes", and the ones online and communicating with each other should appear under "all_nodes". This example output shows the configured cluster nodes `couch1.bitmask.net` and `couch2.bitmask.net`, but `couch2.bitmask.net` is currently not accessible from `couch1.bitmask.net`


    curl -s --netrc-file /etc/couchdb/couchdb.netrc 'http://127.0.0.1:5984/_membership'
    {"all_nodes":["bigcouch@couch1.bitmask.net"],"cluster_nodes":["bigcouch@couch1.bitmask.net","bigcouch@couch2.bitmask.net"]}



Databases
---------

* Following output shows all neccessary DBs that should be present. Note that the `user-0123456....` DBs are the data stores for a particular user. 


    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_all_dbs' 
    ["customers","identities","sessions","shared","tickets","tokens","user-0","user-9d34680b01074c75c2ec58c7321f540c","user-9d34680b01074c75c2ec58c7325fb7ff","users"]



Design Documents
----------------

* Is User `_design doc` available ?


    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X  GET "http://127.0.0.1:5984/users/_design/User"



MX node
=======

Places to look for errors
-------------------------

* `/var/log/mail.log`
* `/var/log/leap_mx.log`
* `/var/log/syslog` (watch out for stunnel issues)


Query leap-mx
-------------

* for useraccount 


    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:2244
    ...
    postmap: dict_tcp_lookup: send: get jow@dev.bitmask.net
    postmap: dict_tcp_lookup: recv: 200 
    ...

* for mailalias


    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:4242
    ...
    postmap: dict_tcp_lookup: send: get joe@dev.bitmask.net
    postmap: dict_tcp_lookup: recv: 200 f01bc1c70de7d7d80bc1ad77d987e73a
    postmap: dict_tcp_lookup: found: f01bc1c70de7d7d80bc1ad77d987e73a
    f01bc1c70de7d7d80bc1ad77d987e73a
    ...



Mailspool
---------

* Any file in the mailspool longer for a few seconds ?


    ls -la /var/mail/vmail/Maildir/cur/


VPN node
========

Places to look for errors
-------------------------

* `/var/log/syslog` (watch out for openvpn issues)