[Docs] [txt|pdf] [Tracker] [WG] [Email] [Diff1] [Diff2] [Nits]

Versions: (draft-kucherawy-greylisting-bcp) 00 01 02 03 04 05 06 07 08 09 RFC 6647

Individual submission                                       M. Kucherawy
Internet-Draft                                           Cloudmark, Inc.
Intended status: Standards Track                        January 30, 2012
Expires: August 2, 2012

         Email Greylisting: An Applicability Statement for SMTP


   This memo describes the art of email greylisting, the practice of
   providing temporarily degraded service to unknown email clients as an
   anti-abuse mechanism.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on August 2, 2012.

Copyright Notice

   Copyright (c) 2012 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Kucherawy                Expires August 2, 2012                 [Page 1]

Internet-Draft                 Greylisting                  January 2012

Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
   2.  Definitions  . . . . . . . . . . . . . . . . . . . . . . . . .  3
     2.1.  Keywords . . . . . . . . . . . . . . . . . . . . . . . . .  3
     2.2.  E-Mail Architecture Terminology  . . . . . . . . . . . . .  3
   3.  Background . . . . . . . . . . . . . . . . . . . . . . . . . .  3
   4.  Connection-Level Greylisting . . . . . . . . . . . . . . . . .  4
   5.  SMTP HELO/EHLO Greylisting . . . . . . . . . . . . . . . . . .  4
   6.  SMTP MAIL Greylisting  . . . . . . . . . . . . . . . . . . . .  4
   7.  SMTP RCPT Greylisting  . . . . . . . . . . . . . . . . . . . .  5
   8.  SMTP DATA Greylisting  . . . . . . . . . . . . . . . . . . . .  5
   9.  Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . .  6
   10. Benefits and Costs . . . . . . . . . . . . . . . . . . . . . .  6
   11. Unintended Effects On Deliveries . . . . . . . . . . . . . . .  7
   12. Unintended Effects On Clients  . . . . . . . . . . . . . . . .  8
   13. Address Space Saturation . . . . . . . . . . . . . . . . . . .  9
   14. Recommendations  . . . . . . . . . . . . . . . . . . . . . . .  9
   15. Measuring Effectiveness  . . . . . . . . . . . . . . . . . . . 10
   16. IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 10
   17. Security Considerations  . . . . . . . . . . . . . . . . . . . 10
     17.1. Exceptions . . . . . . . . . . . . . . . . . . . . . . . . 10
     17.2. Database . . . . . . . . . . . . . . . . . . . . . . . . . 11
   18. References . . . . . . . . . . . . . . . . . . . . . . . . . . 11
     18.1. Normative References . . . . . . . . . . . . . . . . . . . 11
     18.2. Informative References . . . . . . . . . . . . . . . . . . 11
   Appendix A.  Acknowledgments . . . . . . . . . . . . . . . . . . . 11

Kucherawy                Expires August 2, 2012                 [Page 2]

Internet-Draft                 Greylisting                  January 2012

1.  Introduction

   There are many techniques in use for dealing with email abuse.  One
   is a set of techniques known as "greylisting".  Broadly, this refers
   to any degradation of service for an unknown or suspect source, over
   a period of time.  The narrow use of the term refers to generation of
   an SMTP temporary failure reply code for traffic from such sources.

   There are diverse implementations of this general technique, and,
   predictably therefore, some blurred terminology.

   This memo documents common greylisting techniques and discusses their
   benefits and costs.  It also defines terminology to enable clear
   distinction and discussion of these techniques.

2.  Definitions

2.1.  Keywords

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   document are to be interpreted as described in [KEYWORDS].

2.2.  E-Mail Architecture Terminology

   Readers should be familiar with the material and terminology
   discussed in [MAIL] and [EMAIL-ARCH].

3.  Background

   For many years, large amounts of spam have been sent through purpose-
   built software, or "spamware", rather than conventional Mail Transfer
   Agents (MTAs).  If recipient hosts can identify distinctive
   characteristics of spamware that differ from legitimate MTAs, the
   recipient hosts can reject mail from spamware during the SMTP
   session, avoiding the need to receive the spam.

   Spamware consistently does little or no error recovery.  If it can't
   deliver a message, it just goes on since, in spamming, volume counts
   for far more than reliability.  Greylisting tries to detect spamware
   by rejecting mail from unfamiliar sources with a "soft fail" (4xx)
   [SMTP] error code, on the theory that standard MTAs will retry, and
   spamware won't.  Another, less well-developed, application of
   greylisting is to delay mail from newly seen IP addresses on the
   theory that, if it's a spam source, then by the time it retries, it
   will appear in a list of sources to be filtered, and the mail will
   not be accepted.

Kucherawy                Expires August 2, 2012                 [Page 3]

Internet-Draft                 Greylisting                  January 2012

4.  Connection-Level Greylisting

   Connection-level greylisting involves deciding whether to accept the
   connection from a new [SMTP] client.  At this point in the
   communication between the client and the server, the only information
   known is the incoming IP address.  This, of course, is often (but not
   always) translatable into a host name.

   The typical application of greylisting here is to keep a record of
   SMTP client IP addresses and/or host names (collectively, "sources")
   that have been seen.  Such a database may or may not expire records
   after some period.  If the source is not in the database, or the
   record of the source has not reached some required minimum age, the
   server does one of the following, inviting a later retry:

   o  returns a 421 SMTP reply, and closes the connection;

   o  returns a different 4yz SMTP reply to all further commands in this
      SMTP session

5.  SMTP HELO/EHLO Greylisting

   HELO/EHLO greylisting refers to the first [SMTP] command verb in an
   SMTP session.  It includes a single, required parameter that is
   supposed to contain the client's fully-qualified host name or its
   literal IP address.

   Greylisting implemented at this phase involves retaining a record of
   sources coupled with HELO/EHLO parameters, and returning 4yz SMTP
   replies to all commands until the end of the SMTP session if that
   tuple has not previously been recorded or if the record exists but
   has not reached some configured minimum age.

6.  SMTP MAIL Greylisting

   MAIL greylisting refers to the [SMTP] command verb in an SMTP session
   that initiates a new transaction.  It includes at least one required
   parameter that indicates the return email address of the message
   being relayed from the client to the server.

   Greylisting implemented at this phase involves retaining a record of
   sources coupled with return email addresses, and returning 4yz SMTP
   replies to all commands for the remainder of the SMTP session if that
   tuple has not previously been recorded or if the record exists but
   has not met some configured minimum age.

Kucherawy                Expires August 2, 2012                 [Page 4]

Internet-Draft                 Greylisting                  January 2012

7.  SMTP RCPT Greylisting

   RCPT greylisting refers to the [SMTP] command verb in an SMTP session
   that specifies intended recipients of an email transaction.  It
   includes at least one required parameter that indicates the email
   address of an intended recipient of the message being relayed from
   the client to the server.

   Greylisting implemented at this phase involves retaining a record of
   tuples that combine the provided recipient address with any
   combination of the following:

   o  the source, as described above;

   o  the return email address;

   o  other recipients of the message

   If the selected tuple is not found in the database, or if the record
   is present but has not reached some configured minimum age, the
   greylisting MTA returns 4yz SMTP replies to all commands for the
   remainder of the SMTP session.

   Note that often a match on a tuple involving the first valid RCPT is
   sufficient to identify a retry correctly, and further checks can be

8.  SMTP DATA Greylisting

   DATA greylisting refers to the [SMTP] command verb in an SMTP session
   that contains the actual message content as opposed to its envelope
   details (see [MAIL]).

   There are two places in the SMTP sequence that this type of
   greylisting can be done:

   1.  on receipt of the DATA command, because at that point the entire
       envelope has been received (i.e., all MAIL and RCPT commands have
       been issued);

   2.  on completion of the DATA command, i.e., after the "." that
       terminates transmission of the message body, since at that point
       a digest of the message could be computed.

   Some implementations do filtering here because there are clients that
   don't bother checking SMTP reply codes to commands other than DATA.

   Numerous implementations of greylisting are possible at this point.

Kucherawy                Expires August 2, 2012                 [Page 5]

Internet-Draft                 Greylisting                  January 2012

   All of them involve retaining a record of tuples that combine the
   various parts of the SMTP transaction in some combination, including:

   o  the source, as described above;

   o  the return email address;

   o  the recipients of the message, as a set or individually;

   o  identifiers in the message, such as the contents of the
      RFC5322.From or RFC5322.To fields;

   o  other prominent parts of the content, such as the RFC5322.Subject

   o  a digest of some or all of the message content, as a test for

   (The last three items in that list are only possible at the end of
   DATA, not on receipt of the DATA command.)

   If the selected tuple is not found in the database, or if the record
   exists but has not reached some configured minimum age, the
   greylisting MTA returns 4yz SMTP replies to all commands for the
   remainder of the SMTP session.

9.  Exceptions

   Most greylisting systems provide for an exception mechanism, allowing
   one to specify IP addresses, CIDR blocks, hostnames or domain names
   that are exempt from greylisting checks and thus whose SMTP client
   sessions are not subject to such interference.

10.  Benefits and Costs

   The most obvious benefit to any of the above techniques is that
   spamware deliveries, since they don't retry, are less likely to
   succeed as they are unlikely to make an attempt when a record of a
   previous delivery attempt has been made.

   The most obvious detriment to implementing greylisting is the
   imposition of delay on legitimate mail.  Some popular MTAs don't
   retry failed delivery attempts for an hour or more, which can cause
   expensive delays when delivery of mail is timely.  The
   counterargument to this is that email has always been a "best-effort"
   mechanism, and thus this cost is ultimately low in comparison to the
   cost of dealing with high volumes of unwanted mail.

Kucherawy                Expires August 2, 2012                 [Page 6]

Internet-Draft                 Greylisting                  January 2012

   Another obvious cost is the database needed.  It has to be large
   enough to keep enough history to do an effective job, and fast enough
   to avoid introducing delay into the delivery.  The primary
   consideration is the maximum age of records in the database.  If
   records age out too soon, then hosts that do retry per [SMTP] will be
   periodically subjected to greylisting even though they are well-
   behaved; if records age out after too long a period, then eventually
   spamware that launches a new campaign will not be identified as
   "unknown" in this manner, and will not be required to retry.

11.  Unintended Effects On Deliveries

   There are a few failure modes worth considering.  For example,
   consider an email message intended for user@example.com.  The
   example.com domain is served by two mail servers, one called
   mail1.example.com and one called mail2.example.com.  On the first
   delivery attempt, mail1.example.com greylists the client, and thus
   the client places the message in its outgoing queue for later retry.
   Later, when a retry is attempted, mail2.example.com is selected for
   the delivery, either because mail1.example.com is unavailable or
   because a round-robin [DNS] evaluation produced that result.
   However, the two example.com hosts do not share greylisting
   databases, so the second host again denies the attempt.  Thus,
   although example.com has sought to improve its email throughput, it
   has in fact amplified the problem of legitimate mail delay introduced
   by greylisting.

   Similarly, consider a site with multiple outbound MTAs that share a
   common queue.  On a first outbound delivery attempt to example.com,
   the attempt is greylisted.  On a later retry, a different outbound
   MTA is selected, which means example.com sees a different source, and
   once again greylisting occurs on the same message.

   For systems that do DATA-level greylisting, if any part of the
   message has changed since the first attempt, the tuple constructed
   might be different than the one for the first attempt, and the
   delivery is again greylisted.

   A host that sends mail without sufficient frequency to stay in the
   server's database of known sources will be greylisted for a high
   percentage of mail despite possibly being a legitimate sender.

   All of these and other similar cases can cause greylisting to be
   applied inproperly to legitimate MTAs multiple times, leading to long
   delays in delivery or ultimately the return of the message to its
   sender.  Other side effects include out-of-order delivery of related,
   sequenced messages.

Kucherawy                Expires August 2, 2012                 [Page 7]

Internet-Draft                 Greylisting                  January 2012

12.  Unintended Effects On Clients

   Atypical client behaviours also need to be considered when deploying

   Some clients do not retry messages for very long periods.  Popular
   open source MTAs implement increasing backoff times when messages
   receive temporary failure messages, and/or degrade queue priority for
   very large messages.  This means greylisting introduces even more
   delay for MTAs implementing such schemes, and the delay can become
   large enough to become a nuisance to users.

   Some clients do not retry messages at all.  This means greylisting
   will cause outright delivery failure right away for sources,
   envelopes, or messages that it has not seen before, regardless of the
   client attempting the delivery, essentially treating legitimate mail
   and spam the same.

   If a greylisting scheme requires a database record to have reached a
   certain age rather than merely testing for the presence of the record
   in the database, and the client has a retry schedule that is too
   aggressive, the client could be subjected to rate limiting by the MTA
   independent of the restrictions imposed by greylisting.

   There exist SMTP implementations that make the error of treating all
   error codes as fatal; that is, a 4yz [SMTP] response is treated as if
   it was a 5yz response, and the message is returned to the sender as
   undeliverable.  This can result in such things as inadvertent removal
   from mailing lists in response to the percevied rejections.

   Some clients encode message-specific details in the address parameter
   to the [SMTP] MAIL command.  If doing so causes the parameter to
   change between retry attempts, a greylisting implementation could see
   it as a new delivery rather than a retry, and disallow the delivery.
   In such cases, the mail will never be delivered, and will be returned
   to the sender after the retry timeout expires.

   A client subjected to greylisting might improperly move to the host
   found in the [DNS] MX record set for the destination domain and re-
   attempt delivery.  This can generate unwanted traffic to those
   servers.  Moreover, it is likely that those servers will be able to
   bypass greylisting, either because the servers are in a bypass list
   at the destination or due to the traffic their special relationship
   with the intended recipient implies.

Kucherawy                Expires August 2, 2012                 [Page 8]

Internet-Draft                 Greylisting                  January 2012

13.  Address Space Saturation

   Greylisting is obviously not a fool-proof solution to avoiding
   abusive traffic.  Bad actors that send mail with just enough
   frequency to avoid having their records expire will never be caught
   by this mechanism after the first instance.

   Where this is a concern, combining greylisting with some form of
   reputation service that estimates the likely behaviour for IP
   addresses that are not intercepted by the greylisting function would
   be a good choice.

14.  Recommendations

   The following practices are RECOMMENDED based on collected

   1.  Implement greylisting based on each tuple (IP address,
       RFC5321.MailFrom, RFC5321.RcptTo).  This means a single message
       may produce multiple tuples for evaluation as a message always
       has one RFC5321.MailFrom but can have multiple RFC5321.RcptTo
       values.  After a successful retry, allow all further [SMTP]
       traffic from the IP address in that tuple.

   2.  Include a time window within which a retry is considered, and
       ignored otherwise.  The default window SHOULD be from one minute
       to 24 hours.  Retries before this window is reached are not
       counted as they may be immediate retries by the spamware; retries
       after the window don't benefit the source in case it happens to
       retry a week later as part of a different mailing.  Some sites
       use a longer minimum time to match common legitimate MTA retry
       timeouts, but additional benefit from doing so appears unlikely.

   3.  Include another timeout after which hosts that do retry are
       forgotten by the greylisting function.  The default SHOULD be one

   4.  Multiple border MTAs listed in the [DNS] as having the same MX
       precedence SHOULD share a common greylisting database.  This
       handles the case where a retry goes to a different server after
       the first 4yz reply, and lets all servers share the list of hosts
       that did retry successfully.

   5.  To accommodate those senders that have clusters of outgoing mail
       servers, greylisting servers MAY track CIDR blocks of a size of
       its own choosing, such as /24, rather than the full IP address.

Kucherawy                Expires August 2, 2012                 [Page 9]

Internet-Draft                 Greylisting                  January 2012

   6.  Include a manual override capability for adding specific IP
       addresses or network blocks that always bypass checks.  There are
       legitimate senders that simply don't respond well to greylisting
       for a variety of reasons, most of which do not conflict with

   There is no specific recommendation as to the choice of which 4yz
   code to be returned when greylisting.  It is possible that some
   clients treat different 4yz codes differently, but no data are
   available on whether or not using 421 versus some other 4yz code is
   particularly advantageous.

15.  Measuring Effectiveness

   A few techniques are common when measuring the effectiveness of
   greylisting in a particular installation:

   o  arrange to log the spam vs. legitimate determinations of messages
      and what the greylisting decision would have been if enabled, and
      determine whether or not there is a correlation (and, of course,
      whether or not too much legitimate email would also be affected)

   o  continuing from the previous point, query the set of IP addresses
      subjected to greylisting in any popular [DNSBL] to see if there is
      a strong correlation

16.  IANA Considerations

   No actions are requested of IANA in this memo.

17.  Security Considerations

   This section discusses potential security issues related to

17.1.  Exceptions

   The discussion above highlights the fact that, although greylisting
   provides some obvious and valuable defenses, it can introduce
   unintentional but detrimental consequences to email delivery.  Where
   timely delivery of email is essential, especially for security-
   related applications, the possible consequences of such systems need
   to be carefully considered.

   Specific sources can be excepted from greylisting, but of course that
   means they have elevated privilege in terms of access to the
   mailboxes on the greylisting system, and malefactors may seek to
   exploit this.

Kucherawy                Expires August 2, 2012                [Page 10]

Internet-Draft                 Greylisting                  January 2012

17.2.  Database

   The database that has to be maintained as part of any greylisting
   system will grow as the diversity of its SMTP clients grows, and of
   course is larger in general depending on the nature of the tuple
   stored about each delivery attempt.  Even with a record aging policy
   in place, such a database can grow large enough to interfere with the
   system hosting it, or at least to a point where greylisting service
   is degraded.  Moreover, an attacker knowing which greylisting scheme
   is in use could rotate parameters of SMTP clients under its control
   in an attempt to inflate the database to the point of denial-of-

   Implementers could consider configuring an appropriate failure policy
   so that something locally acceptable happens when the database is
   attacked or otherwise unavailable.

18.  References

18.1.  Normative References

   [KEYWORDS]    Bradner, S., "Key words for use in RFCs to Indicate
                 Requirement Levels", BCP 14, RFC 2119, March 1997.

   [SMTP]        Klensin, J., "Simple Mail Transfer Protocol", RFC 5321,
                 October 2008.

18.2.  Informative References

   [DNS]         Mockapetris, P., "Domain names - implementation and
                 specification", STD 13, RFC 1035, November 1987.

   [DNSBL]       Levine, J., "DNS Blacklists and Whitelists", RFC 5782,
                 February 2010.

   [EMAIL-ARCH]  Crocker, D., "Internet Mail Architecture", RFC 5598,
                 October 2008.

   [MAIL]        Resnick, P., Ed., "Internet Message Format", RFC 5322,
                 October 2008.

Appendix A.  Acknowledgments

   The author wishes to acknowledge Mike Adkins, Steve Atkins, Dave
   Crocker, Peter J. Holzer, John Levine, Jose-Marcio Martins da Cruz,
   S. Moonesamy, Mark Risher, Jordan Rosenwald, Gregory Shapiro, and Joe
   Sniderman for their contributions to this memo.

Kucherawy                Expires August 2, 2012                [Page 11]

Internet-Draft                 Greylisting                  January 2012

Author's Address

   Murray S. Kucherawy
   Cloudmark, Inc.
   128 King St., 2nd Floor
   San Francisco, CA  94107

   Phone: +1 415 946 3800
   EMail: msk@cloudmark.com

Kucherawy                Expires August 2, 2012                [Page 12]

Html markup produced by rfcmarkup 1.129c, available from https://tools.ietf.org/tools/rfcmarkup/