1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
|
<!DOCTYPE html>
<html lang='en'>
<head>
<title>
Keys and Certificates - LEAP Platform Documentation
</title>
<meta content='width=device-width, initial-scale=1.0' name='viewport'>
<meta charset='UTF-8'>
<base href="" />
<style>
body {
background: #444;
display: flex;
flex-direction: row;
padding: 10px;
margin: 0px;
}
#sidebar {
flex: 0 0 250px;
background: white;
margin-right: 10px;
padding: 20px;
}
#sidebar ul {
list-style-type: none;
padding-left: 0px;
margin: 0;
}
#sidebar li { padding: 4px }
#sidebar li a { text-decoration: none }
#sidebar li.active { background: #444 }
#sidebar li.active a { color: white }
#sidebar li.level1 { padding-left: 20px }
#sidebar li.level2 { padding-left: 40px }
#main {
flex: 1 1 auto;
background: white;
padding: 20px;
}
#title-box {
padding-bottom: 20px;
border-bottom: 5px solid #eee;
}
#title-box h1 {
margin-top: 0px;
}
pre {
padding: 10px;
background: #eef;
}
code {
background: #eef;
}
table {border-collapse: collapse}
table td {
border: 1px solid #ccc;
padding: 4px;
vertical-align: top;
}
</style>
</head>
<body>
<div id='sidebar'>
<ul>
<li class=''>
<a href='../../../index.html'>Home</a>
</li>
<li class='semi-active level0'>
<a class='' href='../../guide.html'>Guide</a>
</li>
<li class=' level1'>
<a class='' href='../getting-started.html'>Getting Started</a>
</li>
<li class=' level1'>
<a class='' href='../config.html'>Configuration Files</a>
</li>
<li class=' level1'>
<a class='' href='../nodes.html'>Nodes</a>
</li>
<li class='active level1'>
<a class='' href='../keys-and-certificates.html'>Keys and Certificates</a>
</li>
<li class=' level1'>
<a class='' href='../domains.html'>Domains</a>
</li>
<li class=' level1'>
<a class='' href='../provider-configuration.html'>Provider Configuration</a>
</li>
<li class=' level1'>
<a class='' href='../environments.html'>Environments</a>
</li>
<li class=' level1'>
<a class='' href='../virtual-machines.html'>Virtual Machines</a>
</li>
<li class=' level1'>
<a class='' href='../miscellaneous.html'>Miscellaneous</a>
</li>
<li class=' level1'>
<a class='' href='../commands.html'>Command Line Reference</a>
</li>
<li class=' level0'>
<a class='' href='../../tutorials.html'>Tutorials</a>
</li>
<li class=' level0'>
<a class='' href='../../services.html'>Services</a>
</li>
<li class=' level0'>
<a class='' href='../../upgrading.html'>Upgrading</a>
</li>
<li class=' level0'>
<a class='' href='../../troubleshooting.html'>Troubleshooting</a>
</li>
<li class=' level0'>
<a class='' href='../../details.html'>Details</a>
</li>
</ul>
</div>
<div id='main'>
<div id='title-box'>
<h1>Keys and Certificates</h1>
<div id='summary'>Working with SSH keys, secrets, and X.509 certificates.</div>
</div>
<div id='content-box'>
<div id="TOC"><ol>
<li>
<a href="index.html#working-with-ssh">Working with SSH</a>
<ol>
<li>
<a href="index.html#ssh-related-files">SSH related files</a>
</li>
<li>
<a href="index.html#ssh-and-local-nodes">SSH and local nodes</a>
</li>
<li>
<a href="index.html#to-upgrade-a-ssh-host-key">To upgrade a SSH host key</a>
</li>
<li>
<a href="index.html#when-ssh-host-key-changes">When SSH host key changes</a>
</li>
<li>
<a href="index.html#changing-the-ssh-port">Changing the SSH port</a>
</li>
<li>
<a href="index.html#sysadmins-with-multiple-ssh-keys">Sysadmins with multiple SSH keys</a>
</li>
<li>
<a href="index.html#removing-sysadmin-access">Removing sysadmin access</a>
</li>
</ol>
</li>
<li>
<a href="index.html#x509-certificates">X.509 Certificates</a>
<ol>
<li>
<a href="index.html#configuration-options">Configuration options</a>
</li>
<li>
<a href="index.html#certificate-authorities">Certificate Authorities</a>
</li>
<li>
<a href="index.html#server-certificates">Server certificates</a>
</li>
<li>
<a href="index.html#client-certificates">Client certificates</a>
</li>
<li>
<a href="index.html#signed-certificates">Signed certificates</a>
</li>
<li>
<a href="index.html#examine-certs">Examine Certs</a>
</li>
</ol>
</li>
<li>
<a href="index.html#lets-encrypt">Let’s Encrypt</a>
<ol>
<li>
<a href="index.html#creating-a-certificate">Creating a certificate</a>
</li>
<li>
<a href="index.html#renewing-a-certificate">Renewing a certificate</a>
</li>
<li>
<a href="index.html#issues">Issues</a>
<ol>
<li>
<a href="index.html#certs-already-expired">Certs already expired</a>
<ol>
<li>
<a href="index.html#install-the-official-acme-client">Install the official acme client</a>
</li>
<li>
<a href="index.html#fetch-cert">Fetch cert</a>
</li>
<li>
<a href="index.html#deploy-the-certs">Deploy the certs</a>
</li>
</ol>
</li>
</ol>
</li>
</ol>
</li>
</ol></div>
<h1><a name="working-with-ssh"></a>Working with SSH</h1>
<p>Whenever the <code>leap</code> command needs to push changes to a node or gather information from a node, it tunnels this command over SSH. Another way to put this: the security of your servers rests entirely on SSH. Because of this, it is important that you understand how <code>leap</code> uses SSH.</p>
<h2><a name="ssh-related-files"></a>SSH related files</h2>
<p>Assuming your provider directory is called ‘provider’:</p>
<ul>
<li><code>provider/nodes/crow/crow_ssh.pub</code> – The public SSH host key for node ‘crow’.</li>
<li><code>provider/users/alice/alice_ssh.pub</code> – The public SSH user key for user ‘alice’. Anyone with the private key that corresponds to this public key will have root access to all nodes.</li>
<li><code>provider/files/ssh/known_hosts</code> – An autogenerated known_hosts, built from combining <code>provider/nodes/*/*_ssh.pub</code>. You must not edit this file directly. If you need to change it, remove or change one of the files that is used to generate <code>known_hosts</code> and then run <code>leap compile</code>.</li>
<li><code>provider/files/ssh/authorized_keys</code> – An autogenerated list of all the user SSH keys with root access to the notes. It is created from <code>provider/users/*/*_ssh.pub</code>. You must not edit this file directly. If you need to change it, remove or change one of the files that is used to generate <code>authorized_keys</code> and then run <code>leap compile</code>.</li>
</ul>
<p>All of these files should be committed to source control.</p>
<p>If you rename, remove, or add a node with <code>leap node [mv|add|rm]</code> the SSH key files and the <code>known_hosts</code> file will get properly updated.</p>
<h2><a name="ssh-and-local-nodes"></a>SSH and local nodes</h2>
<p>Local nodes are run as Vagrant virtual machines. The <code>leap</code> command handles SSH slightly differently for these nodes.</p>
<p>Basically, all the SSH security is turned off for local nodes. Since local nodes only exist for a short time on your computer and can’t be reached from the internet, this is not a problem.</p>
<p>Specifically, for local nodes:</p>
<ol>
<li><code>known_hosts</code> is never updated with local node keys, since the SSH public key of a local node is different for each user.</li>
<li><code>leap</code> entirely skips the checking of host keys when connecting with a local node.</li>
<li><code>leap</code> adds the public Vagrant SSH key to the list of SSH keys for a user. The public Vagrant SSH key is a shared and insecure key that has root access to most Vagrant virtual machines.</li>
</ol>
<h2><a name="to-upgrade-a-ssh-host-key"></a>To upgrade a SSH host key</h2>
<p>Most servers will have more than one SSH host key. Sometimes, the server will have a better SSH host key than the one you have on file. In order to upgrade to the better SSH host key, simply re-run the init command:</p>
<pre><code>workstation$ leap node init NODE_NAME
</code></pre>
<p>This will prompt you if you want to upgrade the SSH host key, but only if <code>leap</code> thinks that an upgrade is advisable.</p>
<h2><a name="when-ssh-host-key-changes"></a>When SSH host key changes</h2>
<p>If the host key for a node has changed, you will get an error “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED”.</p>
<p>To fix this, you need to remove the file <code>files/nodes/stompy/stompy_ssh.pub</code> and run <code>leap node init stompy</code>, where the node’s name is ‘stompy’. <strong>Only do this if you are ABSOLUTELY CERTAIN that the node’s SSH host key has changed</strong>.</p>
<h2><a name="changing-the-ssh-port"></a>Changing the SSH port</h2>
<p>Suppose you have a node <code>blinky</code> that has SSH listening on port 22 and you want to make it port 2200.</p>
<p>First, modify the configuration for <code>blinky</code> to specify the variable <code>ssh.port</code> as 2200. Usually, this is done in <code>common.json</code> or in a tag file.</p>
<p>For example, you could put this in <code>tags/production.json</code>:</p>
<pre><code>{
"ssh": {
"port": 2200
}
}
</code></pre>
<p>Run <code>leap compile</code> and open <code>hiera/blinky.yaml</code> to confirm that <code>ssh.port</code> is set to 2200. The port number must be specified as a number, not a string (no quotes).</p>
<p>Then, you need to deploy this change so that SSH will bind to 2200. You cannot simply run <code>leap deploy blinky</code> because this command will default to using the variable <code>ssh.port</code> which is now <code>2200</code> but SSH on the node is still bound to 22.</p>
<p>So, you manually override the port in the deploy command, using the old port:</p>
<pre><code>leap deploy --port 22 blinky
</code></pre>
<p>Afterwards, SSH on <code>blinky</code> should be listening on port 2200 and you can just run <code>leap deploy blinky</code> from then on.</p>
<h2><a name="sysadmins-with-multiple-ssh-keys"></a>Sysadmins with multiple SSH keys</h2>
<p>The command <code>leap add-user --self</code> allows only one SSH key. If you want to specify more than one key for a user, you can do it manually:</p>
<pre><code>users/userx/userx_ssh.pub
users/userx/otherkey_ssh.pub
</code></pre>
<p>All keys matching ‘userx/*_ssh.pub’ will be usable.</p>
<h2><a name="removing-sysadmin-access"></a>Removing sysadmin access</h2>
<p>Suppose you want to remove <code>userx</code> from having any further SSH access to the servers. Do this:</p>
<pre><code>rm -r users/userx
leap deploy
</code></pre>
<h1><a name="x509-certificates"></a>X.509 Certificates</h1>
<h2><a name="configuration-options"></a>Configuration options</h2>
<p>The <code>ca</code> option in provider.json provides settings used when generating CAs and certificates. The defaults are as follows:</p>
<pre><code>{
"ca": {
"name": "= global.provider.ca.organization + ' Root CA'",
"organization": "= global.provider.name[global.provider.default_language]",
"organizational_unit": "= 'https://' + global.provider.domain",
"bit_size": 4096,
"digest": "SHA256",
"life_span": "10y",
"server_certificates": {
"bit_size": 2048,
"digest": "SHA256",
"life_span": "1y"
},
"client_certificates": {
"bit_size": 2048,
"digest": "SHA256",
"life_span": "2m",
"limited_prefix": "LIMITED",
"unlimited_prefix": "UNLIMITED"
}
}
}
</code></pre>
<p>You should not need to override these defaults in your own provider.json, but you can if you want to. To see what values are used for your provider, run <code>leap inspect provider.json</code>.</p>
<p>NOTE: A certificate <code>bit_size</code> greater than 2048 will probably not be recognized by most commercial CAs.</p>
<h2><a name="certificate-authorities"></a>Certificate Authorities</h2>
<p>There are three x.509 certificate authorities (CA) associated with your provider:</p>
<ol>
<li><strong>Commercial CA:</strong> It is strongly recommended that you purchase a commercial cert for your primary domain. The goal of platform is to not depend on the commercial CA system, but it does increase security and usability if you purchase a certificate. The cert for the commercial CA must live at <code>files/cert/commercial_ca.crt</code>.</li>
<li><strong>Server CA:</strong> This is a self-signed CA responsible for signing all the <strong>server</strong> certificates. The private key lives at <code>files/ca/ca.key</code> and the public cert lives at <code>files/ca/ca.crt</code>. The key is very sensitive information and must be kept private. The public cert is distributed publicly.</li>
<li><strong>Client CA:</strong> This is a self-signed CA responsible for signing all the <strong>client</strong> certificates. The private key lives at <code>files/ca/client_ca.key</code> and the public cert lives at <code>files/ca/client_ca.crt</code>. Neither file is distribute publicly. It is not a big deal if the private key for the client CA is compromised, you can just generate a new one and re-deploy.</li>
</ol>
<p>To generate both the Server CA and the Client CA, run the command:</p>
<pre><code>leap cert ca
</code></pre>
<h2><a name="server-certificates"></a>Server certificates</h2>
<p>Most every server in your service provider will have a x.509 certificate, generated by the <code>leap</code> command using the Server CA. Whenever you modify any settings of a node that might affect it’s certificate (like changing the IP address, hostname, or settings in provider.json), you can magically regenerate all the certs that need to be regenerated with this command:</p>
<pre><code>leap cert update
</code></pre>
<p>Run <code>leap help cert update</code> for notes on usage options.</p>
<p>Because the server certificates are generated locally on your personal machine, the private key for the Server CA need never be put on any server. It is up to you to keep this file secure.</p>
<h2><a name="client-certificates"></a>Client certificates</h2>
<p>Every leap client gets its own time-limited client certificate. This cert is use to connect to the OpenVPN gateway (and probably other things in the future). It is generated on the fly by the webapp using the Client CA.</p>
<p>To make this work, the private key of the Client CA is made available to the webapp. This might seem bad, but compromise of the Client CA simply allows the attacker to use the OpenVPN gateways without paying. In the future, we plan to add a command to automatically regenerate the Client CA periodically.</p>
<p>There are two types of client certificates: limited and unlimited. A client using a limited cert will have its bandwidth limited to the rate specified by <code>provider.service.bandwidth_limit</code> (in Bytes per second). An unlimited cert is given to the user if they authenticate and the user’s service level matches one configured in <code>provider.service.levels</code> without bandwidth limits. Otherwise, the user is given a limited client cert.</p>
<h2><a name="signed-certificates"></a>Signed certificates</h2>
<p>We strongly recommend that the primary domain for your provider has a certificate signed by a “trusted CA” (e.g. A Certificate Authority that is trusted by the web browsers and in the Debian <code>ca-certificates</code> package). This provides several benefits:</p>
<ol>
<li>When users visit your website, they don’t get a scary notice that something is wrong.</li>
<li>When a user runs the LEAP client, selecting your service provider will not cause a warning message.</li>
<li>When other providers first discover your provider, they are more likely to trust your provider key if it is fetched over a commercially verified link.</li>
</ol>
<p>The LEAP platform is designed so that it assumes you are using a certificate signed by a “trusted CA” for the primary domain of your provider, but all other servers are assumed to use certs signed by the Server CA you create.</p>
<p>To generate a CSR, run:</p>
<pre><code>leap cert csr [DOMAIN]
</code></pre>
<p>This command will generate the CSR and private key matching <code>provider.domain</code> or use DOMAIN. It also generates a server certificate signed with the Server CA. You should delete this certificate and replace it with a real one you get back from a “trusted CA”.</p>
<p>The related commercial cert files are:</p>
<pre><code>files/
cert/
domain.org.crt # Server certificate for domain.org, obtained from
# the trusted CA (this file is initially signed with
# the Server CA, but you should replace it).
domain.org.csr # Certificate signing request (PEM format)
domain.org.key # Private key for you certificate (PEM format)
commercial_ca.crt # DEPRECATED: The certificate chain obtained from
# the trusted CA (PEM format)
</code></pre>
<p>The private key file is extremely sensitive and care should be taken with its provenance.</p>
<p>A few notes on the certificate chain:</p>
<ul>
<li>A certificate is basically just a key signed by another key. In x.509, the signing key might be signed by yet another key, and so on, all the way to a ‘root’ key. It is the root key that a browser trusts or is in the Debian <code>ca-certificates</code> package. The chain is the set of all the keys from the root to the end certificate.</li>
<li>For TLS, both the server and the client need the full chain from the certificate to the CA’s root.</li>
<li>The full chain should be appended in the file <code>domain.org.crt</code> after the server certificate. The chain can also live in <code>commercial_ca.crt</code>, but this is deprecated.</li>
</ul>
<p>If you want to add additional fields to the CSR, like country, city, or locality, you can configure these values in provider.json like so:</p>
<pre><code> "ca": {
"server_certificates": {
"country": "US",
"state": "Washington",
"locality": "Seattle"
}
}
</code></pre>
<p>If they are not present, the CSR will be created without them.</p>
<h2><a name="examine-certs"></a>Examine Certs</h2>
<p>To see details about the keys and certs you can use <code>leap inspect</code> like so:</p>
<pre><code>$ leap inspect files/ca/ca.crt
</code></pre>
<h1><a name="lets-encrypt"></a>Let’s Encrypt</h1>
<p>Let’s Encrypt is a free “trusted CA”. You can obtain signed certificates from Let’s Encrypt very easily using the LEAP command line, so long as you have first set up DNS correctly.</p>
<h2><a name="creating-a-certificate"></a>Creating a certificate</h2>
<p>For example:</p>
<pre><code>workstation$ leap cert register
workstation$ leap cert csr demo.bitmask.net
workstation$ leap cert renew demo.bitmask.net
workstation$ leap deploy
</code></pre>
<p>Some notes:</p>
<ol>
<li>You only need to run <code>leap cert register</code> once. Registering will save the Let’s Encrypt account key to <code>files/ca/lets-encrypt-account.key</code>. If you delete this file, just run <code>leap cert register</code> again.</li>
<li>Let’s Encrypt support requires that you have already platform 0.9 or later.</li>
<li>This requires that the DNS records are correct for the domain.</li>
</ol>
<h2><a name="renewing-a-certificate"></a>Renewing a certificate</h2>
<p>Let’s Encrypt validations are short lived. You will need to renew the certificate at least once every three months. There is no harm in doing it more regularly, however. You can renew your cert every day if you wanted.</p>
<pre><code>workstation$ leap cert renew demo.bitmask.net
workstation$ leap deploy
</code></pre>
<p>There is no need to create a new CSR: renewing will reuse the old private key and the old CSR. It is especially important to not create a new CSR if you have advertised public key pins using HPKP.</p>
<h2><a name="issues"></a>Issues</h2>
<h3><a name="certs-already-expired"></a>Certs already expired</h3>
<p>When a cert is already expired, you can get into a possible deadlock situation on your servers which you can only resolve manually at the moment.</p>
<h4><a name="install-the-official-acme-client"></a>Install the official acme client</h4>
<p>Log in to your webapp node and install the <code>certbot</code> package:</p>
<pre><code>server$ apt install -t jessie-backports certbot
</code></pre>
<h4><a name="fetch-cert"></a>Fetch cert</h4>
<p>Stop apache so the letsencrypt client can bind to port 80:</p>
<pre><code>server$ systemctl stop apache2
</code></pre>
<p>Fetch the certs</p>
<pre><code>server$ certbot certonly --standalone --email admin@$(hostname -d) -d $(hostname -d) -d api.$(hostname -d) -d $(hostname -f) -d nicknym.$(hostname -d)
</code></pre>
<p>This will put the certs and keys into <code>/etc/letsencrypt/live/DOMAIN/</code>.</p>
<p>Now, go to your workstation’s provider configuration directory and copy the newly created files from the server to your local config. You will override existing files so please make a backup before proceeding, or use a version control system to track changes.</p>
<pre><code>workstation$ cd PATH_TO_PROVIDER_CONFIG
</code></pre>
<p>Copy the Certificate</p>
<pre><code>workstation$ scp 'root@SERVER:/etc/letsencrypt/live/$(hostname -d)/cert.pem' files/cert/DOMAIN.crt
</code></pre>
<p>Copy the private key</p>
<pre><code>workstation$ scp 'root@SERVER:/etc/letsencrypt/live/$(hostname -d)/privkey.pem' files/cert/DOMAIN.key
</code></pre>
<p>Copy the CA chain cert</p>
<pre><code>workstation$ scp 'root@SERVER:/etc/letsencrypt/live/$(hostname -d)/fullchain.pem' files/cert/commercial_ca.crt
</code></pre>
<h4><a name="deploy-the-certs"></a>Deploy the certs</h4>
<p>Now you only need to deploy the certs</p>
<pre><code>workstation$ leap deploy
</code></pre>
<p>This will put them into the right locations which are:</p>
<ul>
<li><code>/etc/x509/certs/leap_commercial.crt</code> for the certificate</li>
<li><code>/etc/x509/./keys/leap_commercial.key</code> for the private key</li>
<li><code>/usr/local/share/ca-certificates/leap_commercial_ca.crt</code> for the CA chain cert.</li>
</ul>
<p>Start apache2 again</p>
<pre><code>server$ systemctl start apache2
</code></pre>
<p>Done! In the future please make sure to always renew letsencrypt certificates before they expire ;).</p>
</div>
</div>
</body>
</html>
|
