summaryrefslogtreecommitdiff
path: root/README.md
blob: dd59a16479d196538a3c90051e93e1d84465bc5a (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
GETTING STARTED
============================================

Install necessary gems:

    $ bundle

Create a config file with the necessary secret:

    $ sed -e s/CHANGEME/$(pwgen -s 30)/ config/config.yml.example > config/config.yml

USAGE
============================================

    rake reset
    cat postfix.log.1 | bin/parse-email-logs

NOTES
============================================

encoded list sender
--------------------------------------------

The envelope "from" for mailing lists often encodes the recipient. For example:

   bounce-debian-backports=micah=debian.org@lists.debian.org

This is an entry for the mailing list debian-backports@lists.debian.org
delivering mail to micah@debian.org.

So, the data will appear to have many more unique envelope from addresses than
there really are.

quota
--------------------------------------------

The way we have postfix configured, we reject messages for users who are over
quota very early on in the pipeline. By doing this, we radically reduce the
overhead that the mail servers have for dealing with users who are over quota.

One consequence of this is that incoming messages to users who are over quota
never get a queue ID and will never show up in the dataset.


TODO
============================================

handle over quota errors?

    NOQUEUE: reject: RCPT from hotmail.com[0.0.0.0]: 450 4.7.1 <bob@riseup.net>: Recipient address rejected: Sorry, your message cannot be delivered to that person because their mailbox is full. If you can contact them another way, you may wish to tell them of this problem; from=<alice@hotmail.com> to=<bob@riseup.net> proto=ESMTP helo=<mx100.hotmail.com>

what is "resent-message-id"?

    May 20 22:16:47 mx1 postfix/smtpd[23894]: 106FC1A1FCB: client=bendel.debian.org[0.0.0.0]
    May 20 22:16:47 mx1 postfix/cleanup[21313]: 106FC1A1FCB: message-id=<20160520221607.GA5201@riseup.net>
    May 20 22:16:47 mx1 postfix/cleanup[21313]: 106FC1A1FCB: resent-message-id=<36m1DyUlwO.A.vVC.Nz4PXB@bendel>
    May 20 22:16:47 mx1 postfix/qmgr[5938]: 106FC1A1FCB: from=<bounce-debian-glibc=xxxx=debian.org@lists.debian.org>, size=32505, nrcpt=1 (queue active)
    May 20 22:16:47 mx1 postfix/smtp[21920]: 106FC1A1FCB: to=<xxxx@riseup.net>, relay=0.0.0.0[0.0.0.0]:25, delay=1.1, delays=1.1/0/0.04/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 87B333F0)
    May 20 22:16:47 mx1 postfix/qmgr[5938]: 106FC1A1FCB: removed