1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
|
@title = 'Troubleshooting Guide'
@nav_title = 'Troubleshooting'
@toc = true
General
=======
* Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. `-v 5` after the `leap` cmd, i.e. `leap -v 2 deploy`.
Webapp node
===========
Places to look for errors
-------------------------
* `/var/log/apache2/error.log`
* `/srv/leap/webapp/log/production.log`
* `/var/log/syslog` (watch out for stunnel issues)
Is haproxy ok ?
---------------
curl -s -X GET "http://127.0.0.1:4096"
Is couchdb accessible through stunnel ?
---------------------------------------
curl -s -X GET "http://127.0.0.1:4000"
Check couchdb acl
-----------------
mkdir /etc/couchdb
cat /srv/leap/webapp/config/couchdb.yml.admin # see username and password
echo "machine 127.0.0.1 login admin password <PASSWORD>" > /etc/couchdb/couchdb-admin.netrc
chmod 600 /etc/couchdb/couchdb-admin.netrc
curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096"
curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
Check client config files
-------------------------
https://example.net/provider.json
https://example.net/1/config/smtp-service.json
https://example.net/1/config/soledad-service.json
https://example.net/1/config/eip-service.json
Couchdb node
============
Places to look for errors
-------------------------
* `/opt/bigcouch/var/log/bigcouch.log`
* `/var/log/syslog` (watch out for stunnel issues)
Bigcouch membership
-------------------
* All nodes configured for the provider should appear here:
<pre>
curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5986/nodes/_all_docs'
</pre>
* All configured nodes should show up under "cluster_nodes", and the ones online and communicating with each other should appear under "all_nodes". This example output shows the configured cluster nodes `couch1.bitmask.net` and `couch2.bitmask.net`, but `couch2.bitmask.net` is currently not accessible from `couch1.bitmask.net`
<pre>
curl -s --netrc-file /etc/couchdb/couchdb.netrc 'http://127.0.0.1:5984/_membership'
{"all_nodes":["bigcouch@couch1.bitmask.net"],"cluster_nodes":["bigcouch@couch1.bitmask.net","bigcouch@couch2.bitmask.net"]}
</pre>
* Sometimes a `/etc/init.d/bigcouch restart` on all nodes is needed, to register new nodes
Databases
---------
* Following output shows all neccessary DBs that should be present. Note that the `user-0123456....` DBs are the data stores for a particular user.
<pre>
curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_all_dbs'
["customers","identities","sessions","shared","tickets","tokens","user-0","user-9d34680b01074c75c2ec58c7321f540c","user-9d34680b01074c75c2ec58c7325fb7ff","users"]
</pre>
Design Documents
----------------
* Is User `_design doc` available ?
<pre>
curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET "http://127.0.0.1:5984/users/_design/User"
</pre>
MX node
=======
Places to look for errors
-------------------------
* `/var/log/mail.log`
* `/var/log/leap_mx.log`
* `/var/log/syslog` (watch out for stunnel issues)
Query leap-mx
-------------
* for useraccount
<pre>
postmap -v -q "joe@dev.bitmask.net" tcp:localhost:2244
...
postmap: dict_tcp_lookup: send: get jow@dev.bitmask.net
postmap: dict_tcp_lookup: recv: 200
...
</pre>
* for mailalias
<pre>
postmap -v -q "joe@dev.bitmask.net" tcp:localhost:4242
...
postmap: dict_tcp_lookup: send: get joe@dev.bitmask.net
postmap: dict_tcp_lookup: recv: 200 f01bc1c70de7d7d80bc1ad77d987e73a
postmap: dict_tcp_lookup: found: f01bc1c70de7d7d80bc1ad77d987e73a
f01bc1c70de7d7d80bc1ad77d987e73a
...
</pre>
Mailspool
---------
* Any file in the leap_mx mailspool longer for a few seconds ?
<pre>
ls -la /var/mail/vmail/Maildir/cur/
</pre>
* Any mails in postfix mailspool longer than a few seconds ?
<pre>
mailq
</pre>
Testing mail delivery
---------------------
swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 25
swaks -f varac@cdev.bitmask.net -t varac@cdev.bitmask.net -s chipmonk.cdev.bitmask.net --port 465 --tlsc
swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 587 --tls
VPN node
========
Places to look for errors
-------------------------
* `/var/log/syslog` (watch out for openvpn issues)
|
