1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
|
@title = 'Troubleshooting Guide'
@nav_title = 'Troubleshooting'
@toc = true
General
=======
* Please increase verbosity when debugging / filing issues in our issue tracker. You can do this with adding i.e. `-v 5` after the `leap` cmd, i.e. `leap -v 2 deploy`.
Webapp
======
Places to look for errors
-------------------------
* `/var/log/apache2/error.log`
* `/srv/leap/webapp/log/production.log`
* `/var/log/syslog` (watch out for stunnel issues)
* `/var/log/leap/*`
Is haproxy ok ?
---------------
curl -s -X GET "http://127.0.0.1:4096"
Is couchdb accessible through stunnel ?
---------------------------------------
* Depending on how many couch nodes you have, increase the port for every test
(see /etc/haproxy/haproxy.cfg for the server/port mapping):
curl -s -X GET "http://127.0.0.1:4000"
curl -s -X GET "http://127.0.0.1:4001"
...
Check couchdb acl as admin
--------------------------
mkdir /etc/couchdb
cat /srv/leap/webapp/config/couchdb.yml.admin # see username and password
echo "machine 127.0.0.1 login admin password <PASSWORD>" > /etc/couchdb/couchdb-admin.netrc
chmod 600 /etc/couchdb/couchdb-admin.netrc
curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096"
curl -s --netrc-file /etc/couchdb/couchdb-admin.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
Check couchdb acl as unpriviledged user
---------------------------------------
cat /srv/leap/webapp/config/couchdb.yml # see username and password
echo "machine 127.0.0.1 login webapp password <PASSWORD>" > /etc/couchdb/couchdb-webapp.netrc
chmod 600 /etc/couchdb/couchdb-webapp.netrc
curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096"
curl -s --netrc-file /etc/couchdb/couchdb-webapp.netrc -X GET "http://127.0.0.1:4096/_all_dbs"
Check client config files
-------------------------
https://example.net/provider.json
https://example.net/1/config/smtp-service.json
https://example.net/1/config/soledad-service.json
https://example.net/1/config/eip-service.json
Soledad
=======
/var/log/soledad.log
Couchdb
=======
Places to look for errors
-------------------------
* `/opt/bigcouch/var/log/bigcouch.log`
* `/var/log/syslog` (watch out for stunnel issues)
Bigcouch membership
-------------------
* All nodes configured for the provider should appear here:
<pre>
curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5986/nodes/_all_docs'
</pre>
* All configured nodes should show up under "cluster_nodes", and the ones online and communicating with each other should appear under "all_nodes". This example output shows the configured cluster nodes `couch1.bitmask.net` and `couch2.bitmask.net`, but `couch2.bitmask.net` is currently not accessible from `couch1.bitmask.net`
<pre>
curl -s --netrc-file /etc/couchdb/couchdb.netrc 'http://127.0.0.1:5984/_membership'
{"all_nodes":["bigcouch@couch1.bitmask.net"],"cluster_nodes":["bigcouch@couch1.bitmask.net","bigcouch@couch2.bitmask.net"]}
</pre>
* Sometimes a `/etc/init.d/bigcouch restart` on all nodes is needed, to register new nodes
Databases
---------
* Following output shows all neccessary DBs that should be present. Note that the `user-0123456....` DBs are the data stores for a particular user.
<pre>
curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_all_dbs'
["customers","identities","sessions","shared","tickets","tokens","user-0","user-9d34680b01074c75c2ec58c7321f540c","user-9d34680b01074c75c2ec58c7325fb7ff","users"]
</pre>
Design Documents
----------------
* Is User `_design doc` available ?
<pre>
curl -s --netrc-file /etc/couchdb/couchdb.netrc -X GET "http://127.0.0.1:5984/users/_design/User"
</pre>
Is couchdb cluster backend accessible through stunnel ?
-------------------------------------------------------
* Find out how many connections are set up for the couchdb cluster backend:
<pre>
grep "accept = 127.0.0.1" /etc/stunnel/*
</pre>
* Now connect to all of those local endpoints to see if they up. All these tests should return "localhost [127.0.0.1] 4000 (?) open"
<pre>
nc -v 127.0.0.1 4000
nc -v 127.0.0.1 4001
...
</pre>
MX
==
Places to look for errors
-------------------------
* `/var/log/mail.log`
* `/var/log/leap_mx.log`
* `/var/log/syslog` (watch out for stunnel issues)
Is couchdb accessible through stunnel ?
---------------------------------------
* Depending on how many couch nodes you have, increase the port for every test
(see /etc/haproxy/haproxy.cfg for the server/port mapping):
curl -s -X GET "http://127.0.0.1:4000"
curl -s -X GET "http://127.0.0.1:4001"
...
Query leap-mx
-------------
* for useraccount
<pre>
postmap -v -q "joe@dev.bitmask.net" tcp:localhost:2244
...
postmap: dict_tcp_lookup: send: get jow@dev.bitmask.net
postmap: dict_tcp_lookup: recv: 200
...
</pre>
* for mailalias
<pre>
postmap -v -q "joe@dev.bitmask.net" tcp:localhost:4242
...
postmap: dict_tcp_lookup: send: get joe@dev.bitmask.net
postmap: dict_tcp_lookup: recv: 200 f01bc1c70de7d7d80bc1ad77d987e73a
postmap: dict_tcp_lookup: found: f01bc1c70de7d7d80bc1ad77d987e73a
f01bc1c70de7d7d80bc1ad77d987e73a
...
</pre>
Check couchdb acl as unpriviledged user
---------------------------------------
cat /etc/leap/mx.conf # see username and password
echo "machine 127.0.0.1 login leap_mx password <PASSWORD>" > /etc/couchdb/couchdb-leap_mx.netrc
chmod 600 /etc/couchdb/couchdb-leap_mx.netrc
curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/_all_dbs" # pick one "user-<hash>" db
curl -s --netrc-file /etc/couchdb/couchdb-leap_mx.netrc -X GET "http://127.0.0.1:4096/user-de9c77a3d7efbc779c6c20da88e8fb9c"
* you may check multiple times, cause 127.0.0.1:4096 is haproxy load-balancing the different couchdb nodes
Mailspool
---------
* Any file in the leap_mx mailspool longer for a few seconds ?
<pre>
ls -la /var/mail/vmail/Maildir/cur/
</pre>
* Any mails in postfix mailspool longer than a few seconds ?
<pre>
mailq
</pre>
Testing mail delivery
---------------------
swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 25
swaks -f varac@cdev.bitmask.net -t varac@cdev.bitmask.net -s chipmonk.cdev.bitmask.net --port 465 --tlsc
swaks -f alice@example.org -t bob@example.net -s mx1.example.net --port 587 --tls
VPN
===
Places to look for errors
-------------------------
* `/var/log/syslog` (watch out for openvpn issues)
|
