[feat] add fallback on trust sources for ssl verification

With the merge of platformTrust in twisted, the situation for cert chain
verification in linux improved a lot.

This patch implements fallbacks to do the following:

- Try to use whatever trust sources are found in the system. This means
that if ca-certificates is installed, pyopenssl will have a valid set of
root certificates and verification will likely work (twisted uses
platformTrust for this).

- If that fails, try to use certifi. We could/should depend on that from
now on, *but* it's not packaged before stretch.

- So, I'm not deprecating its usage right now, but this one should be
the last cacert.pem bundle that we ship with leap.common.

- If the cacert.pem from leap.common fails to be found, well, there's
nothing you can do. Your TOFU attempt with a cert coming from the
CArtel will fail.

Most of this MR should be sent as a patch upstream, see https://twistedmatrix.com/trac/ticket/6934
Also related: https://twistedmatrix.com/trac/ticket/9209

I think proper testing will depend on merging https://github.com/pyca/pyopenssl/pull/473

- Resolves: #8958
- Release: 0.6.0

4 files changed, 251 insertions, 48 deletions

diff --git a/src/leap/common/EFFchain.pem b/src/leap/common/EFFchain.pem
new file mode 100644
index 0000000..15a79d8
--- /dev/null
+++ b/src/leap/common/EFFchain.pem
@@ -0,0 +1,116 @@
+CONNECTED(00000003)
+---
+Certificate chain
+ 0 s:/CN=eff.org
+   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
+-----BEGIN CERTIFICATE-----
+MIIGVTCCBT2gAwIBAgISAx9kTOWisGpqooJ4k5cVH4SGMA0GCSqGSIb3DQEBCwUA
+MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
+ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNzA3MDYxNzMzMDBaFw0x
+NzEwMDQxNzMzMDBaMBIxEDAOBgNVBAMTB2VmZi5vcmcwggIiMA0GCSqGSIb3DQEB
+AQUAA4ICDwAwggIKAoICAQCzZ72xy22icrzIE+m0NeoMNgB4qTB39bOPx+2Lod3J
+ZwYpUHN2QJDdvOC51Tanwnuutnyjhahwi7JLOFzLY9Tz14FGICrAosILnHuBQBML
+QtgpeybCs+IAukZuu2n0Pt0u290JUwzmiWxjx18mxkVFB5NyeHwhg6JG59uOYoJ/
+JMIGDz4kTQuIAOTZFgV7bMOHUJNrYuN/tUB00zriy7cAJ5aovq30gIhpePbeDAja
+BE0pj5UdV5V9EsYFz7kZMe/VIgPY/O3KxD7k+40Dv2W6XOQPxiDXB/oAQkzT/KO2
+yIgdRWJfD2ohmGWi5cJdP99rvtUshmhYcynenqRP2bKtZIEi7DvGj2r6MNiTHqtC
+HTZJmlrfJkqkjLMbPuQC0skOhfYImLVZtMGz6nzU4Uh6ZVM+2YM1S5oxJ9d74S7k
+Rvh58AIaz1yf8drMc+PvrQqeZhiQA9od4i4ldtAjA2b7fX4YCN7eG4Q/YQi90rRE
+xnmeHJaJYC/sYr6igaV63HXGyJ63JNoSDc3u9tDAB0goLh1kRXrmfsGX4B+ADYxt
+cEzA1LvVeLcWf06edO+mJFZSUYYhVb5JUloaSdGRAa3MlEbZ/2xBeQZUY4kjwv/o
+2axeRRC1BuQxeyizs40NF2t+ziwFlEoERqFVNlAKn7R+7QvVu0TGvLW/CmYXuIyP
+NwIDAQABo4ICazCCAmcwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUF
+BwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBTC4NDcy8n+xv0N
+1GNig3ieBGnUkTAfBgNVHSMEGDAWgBSoSmpjBH3duubRObemRWXv86jsoTBvBggr
+BgEFBQcBAQRjMGEwLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwLmludC14My5sZXRz
+ZW5jcnlwdC5vcmcwLwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRz
+ZW5jcnlwdC5vcmcvMHYGA1UdEQRvMG2CDWF0bGFzLmVmZi5vcmeCB2VmZi5vcmeC
+Hmh0dHBzLWV2ZXJ5d2hlcmUtYXRsYXMuZWZmLm9yZ4IUaHR0cHNlLWF0bGFzLmVm
+Zi5vcmeCD2tpdHRlbnMuZWZmLm9yZ4IMbWFwcy5lZmYub3JnMIH+BgNVHSAEgfYw
+gfMwCAYGZ4EMAQIBMIHmBgsrBgEEAYLfEwEBATCB1jAmBggrBgEFBQcCARYaaHR0
+cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwgasGCCsGAQUFBwICMIGeDIGbVGhpcyBD
+ZXJ0aWZpY2F0ZSBtYXkgb25seSBiZSByZWxpZWQgdXBvbiBieSBSZWx5aW5nIFBh
+cnRpZXMgYW5kIG9ubHkgaW4gYWNjb3JkYW5jZSB3aXRoIHRoZSBDZXJ0aWZpY2F0
+ZSBQb2xpY3kgZm91bmQgYXQgaHR0cHM6Ly9sZXRzZW5jcnlwdC5vcmcvcmVwb3Np
+dG9yeS8wDQYJKoZIhvcNAQELBQADggEBAGWQ738VV+3dK3fB2gJBHE/MEaHg000P
+koHe1NKc5eoCLWbjUqP0QzcxKha1LwqFz8EaDglO23R9ZkXkI6IlhXsj6n3MTT+j
+FkF5ccbuYd1sY69ghcEHBiZss4b/qepSxcu82LUU2UiuIc6zfxUfglEzzMsV72sb
+Z1qjhA7E5iTyZHJ+0kwj+2XbtxqUbBrzZjN6ku0dyul3d43hnaEoJkeEDlADeWZM
+gfDzoQ4FtIyVYq1FZVODBEr3kjccAlwWMO59YmJeFgjFRHASDw1akT+95h9puS/F
+z+9Sior3hfcLNdGZUZWpd7GQMKKoEbDPm8GubFTvcPfivu8I9Lc5428=
+-----END CERTIFICATE-----
+ 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
+   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
+-----BEGIN CERTIFICATE-----
+MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
+MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
+DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
+SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
+GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
+AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
+q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
+SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
+Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
+a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
+/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
+AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
+CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
+bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
+c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
+VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
+ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
+MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
+Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
+AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
+uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
+wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
+X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
+PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
+KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
+-----END CERTIFICATE-----
+---
+Server certificate
+subject=/CN=eff.org
+issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
+---
+No client certificate CA names sent
+Peer signing digest: SHA512
+Server Temp Key: ECDH, P-256, 256 bits
+---
+SSL handshake has read 3728 bytes and written 302 bytes
+Verification: OK
+---
+New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
+Server public key is 4096 bit
+Secure Renegotiation IS supported
+Compression: NONE
+Expansion: NONE
+No ALPN negotiated
+SSL-Session:
+    Protocol  : TLSv1.2
+    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
+    Session-ID: D448EDCC480EF1C06FE70E6D8B4B868431F613E1D4308FC9A40A0DAA3460C06B
+    Session-ID-ctx: 
+    Master-Key: 59C06531A77DCF1603ABC42B700311866C7D85A3154D9F733F1E671BC59C0287C81CBD9FD39516871CD434BD939379A0
+    PSK identity: None
+    PSK identity hint: None
+    SRP username: None
+    TLS session ticket lifetime hint: 600 (seconds)
+    TLS session ticket:
+    0000 - f8 47 e4 94 45 bd d3 df-61 76 8f b5 98 a1 b8 b5   .G..E...av......
+    0010 - 56 6d 1f 59 43 5e 5e c5-2a 90 66 66 3e 6e b4 45   Vm.YC^^.*.ff>n.E
+    0020 - 7d 15 76 d9 cc 6a d4 d5-db 26 bf d5 e5 4a 5e 9a   }.v..j...&...J^.
+    0030 - 96 ce 88 00 23 64 36 6e-1a 26 7b 94 6b da c4 95   ....#d6n.&{.k...
+    0040 - 96 49 1e 96 5e 34 35 3c-38 e1 0c 3e 41 57 64 1f   .I..^45<8..>AWd.
+    0050 - b0 fe 09 0b 3b f6 bf 8a-f5 6c 54 6e bc 63 35 4e   ....;....lTn.c5N
+    0060 - 3a 37 27 64 8f a4 0c 5b-4f 6b f3 17 a1 9a 12 be   :7'd...[Ok......
+    0070 - 4f 75 8d aa 40 01 58 83-be db 07 77 38 8c 04 ff   Ou..@.X....w8...
+    0080 - f2 58 f0 36 ba dc 74 39-1f 14 16 57 8a ac d2 e9   .X.6..t9...W....
+    0090 - 98 4d 30 2f ce 5e 5b d8-1e 66 96 bd f8 a9 eb 04   .M0/.^[..f......
+    00a0 - 35 14 c0 ad 84 7e 93 28-10 22 8c 7e 50 d2 ca ca   5....~.(.".~P...
+
+    Start Time: 1499775511
+    Timeout   : 7200 (sec)
+    Verify return code: 0 (ok)
+    Extended master secret: no
+---
diff --git a/src/leap/common/ca_bundle.py b/src/leap/common/ca_bundle.py
index e2a624d..66fc778 100644
--- a/src/leap/common/ca_bundle.py
+++ b/src/leap/common/ca_bundle.py
@@ -30,7 +30,7 @@ _system = platform.system()
 IS_MAC = _system == "Darwin"
 
 
-def where():
+def where(name='cacert.pem'):
     """
     Return the preferred certificate bundle.
     :rtype: str
@@ -39,7 +39,7 @@ def where():
         # we are running in a |PyInstaller| bundle
         path = sys._MEIPASS
         return os.path.join(path, 'cacert.pem')
-    return os.path.join(os.path.dirname(__file__), 'cacert.pem')
+    return os.path.join(os.path.dirname(__file__), name)
 
 if __name__ == '__main__':
     print(where())
diff --git a/src/leap/common/certs.py b/src/leap/common/certs.py
index 95704a6..db513f6 100644
--- a/src/leap/common/certs.py
+++ b/src/leap/common/certs.py
@@ -30,8 +30,6 @@ from leap.common.check import leap_assert
 
 logger = logging.getLogger(__name__)
 
-SKIP_SSL_CHECK = os.environ.get('SKIP_TWISTED_SSL_CHECK', False)
-
 
 def get_cert_from_string(string):
     """
@@ -180,36 +178,3 @@ def should_redownload(certfile, now=time.gmtime):
         return True
 
     return False
-
-
-def get_compatible_ssl_context_factory(cert_path=None):
-    import twisted
-    from twisted.internet import ssl
-    cert = None
-
-    if SKIP_SSL_CHECK:
-        # This should be used *only* for testing purposes.
-
-        class WebClientContextFactory(ssl.ClientContextFactory):
-            """
-            A web context factory which ignores the hostname and port and does
-            no certificate verification.
-            """
-            def getContext(self, hostname, port):
-                return ssl.ClientContextFactory.getContext(self)
-
-        contextFactory = WebClientContextFactory()
-        return contextFactory
-
-    if twisted.version.base() > '14.0.1':
-        from twisted.web.client import BrowserLikePolicyForHTTPS
-        if cert_path:
-            cert = ssl.Certificate.loadPEM(open(cert_path).read())
-        policy = BrowserLikePolicyForHTTPS(cert)
-        return policy
-    else:
-        raise Exception(("""
-            Twisted 14.0.2 is needed in order to have secure
-            Client Web SSL Contexts, not %s
-            See: http://twistedmatrix.com/trac/ticket/7647
-            """) % (twisted.version.base()))
diff --git a/src/leap/common/http.py b/src/leap/common/http.py
index 0dee3a2..f6a7f7e 100644
--- a/src/leap/common/http.py
+++ b/src/leap/common/http.py
@@ -16,8 +16,13 @@
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 """
 Twisted HTTP/HTTPS client.
+This module will be deprecated and slowly migrated to use treq instead.
 """
 
+import os
+import re
+
+
 try:
     import twisted
     assert twisted
@@ -26,21 +31,28 @@ except ImportError:
     print "Twisted is needed to use leap.common.http module"
     print ""
     print "Install the extra requirement of the package:"
-    print "$ pip install leap.common[Twisted]"
+    print "$ pip install leap.common[http]"
     import sys
     sys.exit(1)
 
+from leap.common import ca_bundle
 
-from leap.common.certs import get_compatible_ssl_context_factory
-from leap.common.check import leap_assert
 
-from zope.interface import implements
+from OpenSSL.crypto import X509StoreContext
+from OpenSSL.crypto import X509StoreContextError
+from OpenSSL.SSL import Context
+from OpenSSL.SSL import TLSv1_METHOD
 
 from twisted.internet import reactor
 from twisted.internet import defer
+from twisted.internet.ssl import Certificate, trustRootFromCertificates
+from twisted.internet.ssl import ClientContextFactory
+from twisted.logger import Logger
 from twisted.python import failure
+from twisted.python.filepath import FilePath
 
 from twisted.web.client import Agent
+from twisted.web.client import BrowserLikePolicyForHTTPS
 from twisted.web.client import HTTPConnectionPool
 from twisted.web.client import _HTTP11ClientFactory as HTTP11ClientFactory
 from twisted.web.client import readBody
@@ -48,15 +60,82 @@ from twisted.web.http_headers import Headers
 from twisted.web.iweb import IBodyProducer
 from twisted.web._newclient import HTTP11ClientProtocol
 
+from zope.interface import implements
 
 __all__ = ["HTTPClient"]
 
 
+log = Logger()
+
+
 # A default HTTP timeout is used for 2 distinct purposes:
 #   1. as HTTP connection timeout, prior to connection estabilshment.
 #   2. as data reception timeout, after the connection has been established.
+
 DEFAULT_HTTP_TIMEOUT = 30  # seconds
 
+SKIP_SSL_CHECK = os.environ.get('SKIP_TWISTED_SSL_CHECK', False)
+
+
+def certsFromBundle(path, x509=False):
+    PEM_RE = re.compile(
+        "-----BEGIN CERTIFICATE-----\r?.+?\r?"
+        "-----END CERTIFICATE-----\r?\n?""",
+        re.DOTALL)
+    if not os.path.isfile(path):
+        log.warn("Attempted to load non-existent certificate bundle path %s"
+                 % path)
+        return []
+
+    pems = FilePath(path).getContent()
+    cstr = [match.group(0) for match in PEM_RE.finditer(pems)]
+    certs = [Certificate.loadPEM(cert) for cert in cstr]
+    if x509:
+        certs = [cert.original for cert in certs]
+    return certs
+
+
+def hasUsablePlatformTrust():
+
+    _knownchain = certsFromBundle(ca_bundle.where('EFFchain.pem'), x509=True)
+    _knowncert = _knownchain[0]
+    _knowninterm = _knownchain[1:]
+
+    def _verify_test_cert(store, cert):
+        store_ctx = X509StoreContext(store, cert)
+        try:
+            assert store_ctx.verify_certificate() is None
+        except (X509StoreContextError, AssertionError):
+            return False
+        else:
+            return True
+
+    def _add_intermediates(store, intermediates):
+        for _cert in intermediates:
+            store.add_cert(_cert)
+
+    ctx = Context(TLSv1_METHOD)
+    ctx.set_default_verify_paths()
+    store = ctx.get_cert_store()
+    _add_intermediates(store, _knowninterm)
+
+    return _verify_test_cert(store, _knowncert)
+
+
+def getCertifiTrustRoot():
+    try:
+        import certifi
+        bundle = certifi.where()
+    except ImportError:
+        log.warn("certifi was not found. Using leap.common bundle")
+        bundle = ca_bundle.where()
+    if bundle is None:
+        log.error("Cannot find an usable cacert bundle. "
+                  "Certificate verification will fail")
+        return None
+    cacerts = certsFromBundle(bundle)
+    return trustRootFromCertificates(cacerts)
+
 
 class _HTTP11ClientFactory(HTTP11ClientFactory):
     """
@@ -102,6 +181,39 @@ class _HTTPConnectionPool(HTTPConnectionPool):
         return endpoint.connect(factory)
 
 
+# TODO deprecate this in favor of treq.
+# We need treq to have support for:
+
+# [ ] timeout
+# [ ] retries
+# [ ] download/upload pool.
+
+
+def getPolicyForHTTPS(trustRoot=None):
+
+    if SKIP_SSL_CHECK:
+        log.info("---------------------------------------")
+        log.info("SKIPPING SSL CERT VERIFICATION!!!")
+        log.info("I assume you know WHAT YOU ARE DOING...")
+        log.info("---------------------------------------")
+
+        class WebClientContextFactory(ClientContextFactory):
+            """
+            A web context factory which ignores the hostname and port and does
+            no certificate verification.
+            """
+            def getContext(self, hostname, port):
+                return ClientContextFactory.getContext(self)
+
+        contextFactory = WebClientContextFactory()
+        return contextFactory
+
+    if isinstance(trustRoot, str):
+        trustRoot = Certificate.loadPEM(FilePath(trustRoot).getContent())
+
+    return BrowserLikePolicyForHTTPS(trustRoot)
+
+
 class HTTPClient(object):
     """
     HTTP client done the twisted way, with a main focus on pinning the SSL
@@ -122,13 +234,14 @@ class HTTPClient(object):
         maxPersistentPerHost=10
     )
 
-    def __init__(self, cert_file=None,
+    def __init__(self, cert_path=None,
                  timeout=DEFAULT_HTTP_TIMEOUT, pool=None):
         """
         Init the HTTP client
 
-        :param cert_file: The path to the certificate file, if None given the
-                          system's CAs will be used.
+        :param cert_file: The path to the ca certificate file to verify
+                          certificates, if None given the system's CAs will be
+                          used.
         :type cert_file: str
         :param timeout: The amount of time that this Agent will wait for the
                         peer to accept a connection and for each request to be
@@ -139,9 +252,19 @@ class HTTPClient(object):
 
         self._timeout = timeout
         self._pool = pool if pool is not None else self._pool
+
+        if cert_path is None:
+            if hasUsablePlatformTrust():
+                # Twisted Knows What To Do
+                trustRoot = None
+            else:
+                trustRoot = getCertifiTrustRoot()
+        else:
+            trustRoot = cert_path
+
         self._agent = Agent(
             reactor,
-            get_compatible_ssl_context_factory(cert_file),
+            contextFactory=getPolicyForHTTPS(trustRoot),
             pool=self._pool,
             connectTimeout=self._timeout)
         self._semaphore = defer.DeferredSemaphore(
@@ -205,9 +328,7 @@ class HTTPClient(object):
         :return: A deferred that fires with the body of the request.
         :rtype: twisted.internet.defer.Deferred
         """
-        leap_assert(
-            callable(callback),
-            message="The callback parameter should be a callable!")
+        assert callable(callback), "The callback parameter should be a callable!"
         return self._semaphore.run(self._request, url, method, body, headers,
                                    callback)
 
@@ -217,6 +338,7 @@ class HTTPClient(object):
         """
         self._pool.closeCachedConnections()
 
+
 #
 # An IBodyProducer to write the body of an HTTP request as a string.
 #