summaryrefslogtreecommitdiff
path: root/docs/platform/troubleshooting.md
blob: 61149a05cce37e0642ae34fbedf2e920c425e7df (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
@title = 'Troubleshooting Guide'
@nav_title = 'Troubleshooting'
@toc = true


General
=======

* Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. `-v 5` after the `leap` cmd, i.e. `leap -v 2 deploy`.

Webapp node
===========

Places to look for errors
-------------------------

* `/var/log/apache2/error.log`
* `/srv/leap/webapp/log/production.log`
* `/var/log/syslog` (watch out for stunnel issues)

Is haproxy ok ? 
---------------


    curl -s -X  GET "http://127.0.0.1:4096"

Is couchdb accessible through stunnel ?
---------------------------------------


    curl -s -X  GET "http://127.0.0.1:4000"


Check couchdb acl
-----------------

    mkdir /etc/couchdb
    cat /srv/leap/webapp/config/couchdb.yml.admin  # see username and password
    echo "machine 127.0.0.1 login admin password <PASSWORD>" > /etc/couchdb/couchdb-admin.netrc
    chmod 600 /etc/couchdb/couchdb-admin.netrc

    curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096"
    curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
    
Check client config files
-------------------------

    https://example.net/provider.json
    https://example.net/1/config/smtp-service.json
    https://example.net/1/config/soledad-service.json
    https://example.net/1/config/eip-service.json


Couchdb node
============

Places to look for errors
-------------------------

* `/opt/bigcouch/var/log/bigcouch.log`
* `/var/log/syslog` (watch out for stunnel issues)



Bigcouch membership
-------------------

* All nodes configured for the provider should appear here:


    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5986/nodes/_all_docs'

* All configured nodes should show up under "cluster_nodes", and the ones online and communicating with each other should appear under "all_nodes". This example output shows the configured cluster nodes `couch1.bitmask.net` and `couch2.bitmask.net`, but `couch2.bitmask.net` is currently not accessible from `couch1.bitmask.net`


    curl -s --netrc-file /etc/couchdb/couchdb.netrc 'http://127.0.0.1:5984/_membership'
    {"all_nodes":["bigcouch@couch1.bitmask.net"],"cluster_nodes":["bigcouch@couch1.bitmask.net","bigcouch@couch2.bitmask.net"]}

* Sometimes a `/etc/init.d/bigcouch restart` on all nodes is needed, to register new nodes

Databases
---------

* Following output shows all neccessary DBs that should be present. Note that the `user-0123456....` DBs are the data stores for a particular user. 


    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_all_dbs' 
    ["customers","identities","sessions","shared","tickets","tokens","user-0","user-9d34680b01074c75c2ec58c7321f540c","user-9d34680b01074c75c2ec58c7325fb7ff","users"]



Design Documents
----------------

* Is User `_design doc` available ?


    curl -s --netrc-file /etc/couchdb/couchdb.netrc -X  GET "http://127.0.0.1:5984/users/_design/User"



MX node
=======

Places to look for errors
-------------------------

* `/var/log/mail.log`
* `/var/log/leap_mx.log`
* `/var/log/syslog` (watch out for stunnel issues)


Query leap-mx
-------------

* for useraccount 


    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:2244
    ...
    postmap: dict_tcp_lookup: send: get jow@dev.bitmask.net
    postmap: dict_tcp_lookup: recv: 200 
    ...

* for mailalias


    postmap -v -q  "joe@dev.bitmask.net" tcp:localhost:4242
    ...
    postmap: dict_tcp_lookup: send: get joe@dev.bitmask.net
    postmap: dict_tcp_lookup: recv: 200 f01bc1c70de7d7d80bc1ad77d987e73a
    postmap: dict_tcp_lookup: found: f01bc1c70de7d7d80bc1ad77d987e73a
    f01bc1c70de7d7d80bc1ad77d987e73a
    ...



Mailspool
---------

* Any file in the leap_mx mailspool longer for a few seconds ?



    ls -la /var/mail/vmail/Maildir/cur/

* Any mails in postfix mailspool longer than a few seconds ?



    mailq


Testing mail delivery
---------------------

    swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 25 
    swaks -f varac@cdev.bitmask.net -t varac@cdev.bitmask.net -s chipmonk.cdev.bitmask.net --port 465 --tlsc
    swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 587 --tls


VPN node
========

Places to look for errors
-------------------------

* `/var/log/syslog` (watch out for openvpn issues)