Network Working Group B. Davie Internet-Draft Cisco Systems, Inc. Intended status: Standards Track B. Briscoe Expires:
December 21, 2007April 6, 2008 J. Tay BT Research June 19,October 4, 2007 Explicit Congestion Marking in MPLS draft-ietf-tsvwg-ecn-mpls-01.txtdraft-ietf-tsvwg-ecn-mpls-02.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on December 21, 2007.April 6, 2008. Copyright Notice Copyright (C) The IETF Trust (2007). Abstract RFC 3270 defines how to support the Diffserv architecture in MPLS networks, including how to encode Diffserv Code Points (DSCPs) in an MPLS header. DSCPs may be encoded in the EXP field, while other uses of that field are not precluded. RFC3270 makes no statement about how Explicit Congestion Notification (ECN) marking might be encoded in the MPLS header. This draft defines how an operator might define some of the EXP codepoints for explicit congestion notification, without precluding other uses. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Change History [Note to RFC Editor: This section to be removed before publication] Changes in this version (draft-ietf-tsvwg-ecn-mpls-01.txt)(draft-ietf-tsvwg-ecn-mpls-02.txt) relative to the last (draft-ietf-tsvwg-ecn-mpls-00.txt):(draft-ietf-tsvwg-ecn-mpls-01.txt): o Added new text about misordering considerations in Section 6. o Swapped order of Section 8 and Section 9. o Explained more fully the example of congestion-based traffic engineering in Section 9.3. o Trimmed the example of PCN in Section 9.4 and updated to latest preferred PCN terminology in PCN appendix. Changes in draft-ietf-tsvwg-ecn-mpls-01.txt relative to draft-ietf-tsvwg-ecn-mpls-00.txt: o Moved the detailed discussion of marking procedures for Pre- Congestion Notification (PCN) to an appendix. o Removed PCN as a motivation for the efficient code-point usage in Section 2. o Clarified the rationale for preferring the ECT-checking approach over the approach of [Floyd] in Section 184.108.40.206. o Updated discussion of relationship to RFC3168 in Section 7 o Removed discussion of re-ECN from Security Considerations. o Fixed typos and nits. Changes in draft-ietf-tsvwg-ecn-mpls-00.txt relative to draft-davie-ecn-mpls-00: o Corrected the description of ECN-MPLS marking proposed in [Shayman], which closely corresponds to that proposed in this document. o Pre-congestion notification (PCN) marking is now described in a way that does not require normative references to PCN specifications. PCN discussion now serves only to illustrate how the ECN marking concepts can be extended to cover more complex scenarios, with PCN being an example. o Added specification of behavior when MPLS encapsulated packets cross from an ECN-enabled domain to a domain that is not ECN- enabled. o Clarified that copying MPLS ECN or PCN marking into exposed IP header on egress is not mandatory o Fixed typos and nits Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 45 1.1. Background . . . . . . . . . . . . . . . . . . . . . . . . 45 1.2. Intent . . . . . . . . . . . . . . . . . . . . . . . . . . 45 1.3. Terminology . . . . . . . . . . . . . . . . . . . . . . . 56 2. Use of MPLS EXP Field for ECN . . . . . . . . . . . . . . . . 67 3. Per-domain ECT checking . . . . . . . . . . . . . . . . . . . 89 4. ECN-enabled MPLS domain . . . . . . . . . . . . . . . . . . . 89 4.1. Pushing (adding) one or more labels to an IP packet . . . 910 4.2. Pushing one or more labels onto an MPLS labelled packet . 910 4.3. Congestion experienced in an interior MPLS node . . . . . 910 4.4. Crossing a Diffserv Domain Boundary . . . . . . . . . . . 910 4.5. Popping an MPLS label (not the end of the stack) . . . . . 1011 4.6. Popping the last MPLS label in the stack . . . . . . . . . 1011 4.7. Diffserv Tunneling Models . . . . . . . . . . . . . . . . 1011 5. ECN-disabled MPLS domain . . . . . . . . . . . . . . . . . . . 1112 6. The use of more codepoints with E-LSPs and L-LSPs . . . . . . 1112 7. Relationship to tunnel behavior in RFC 3168 . . . . . . . . . 1112 8. Example UsesDeployment Considerations . . . . . . . . . . . . . . . . . . 13 8.1. Marking non-ECN Capable Packets . . . . . . . 12 8.1. RFC3168-style ECN. . . . . . 13 8.2. Non-ECN capable routers in an MPLS Domain . . . . . . . . 14 9. Example Uses . . . . . . 12 8.2. ECN Co-existence with Diffserv E-LSPs. . . . . . . . . . 12 8.3. Congestion-feedback-based Traffic Engineering. . . . . . 13 8.4. PCN flow admission control and flow pre-emption. . . 14 9.1. RFC3168-style ECN . . 13 9. Deployment Considerations. . . . . . . . . . . . . . . . . . 14 9.1. Marking non-ECN Capable Packets9.2. ECN Co-existence with Diffserv E-LSPs . . . . . . . . . . 15 9.3. Congestion-feedback-based Traffic Engineering . . . 14 9.2. Non-ECN capable routers in an MPLS Domain. . . 15 9.4. PCN flow admission control and flow termination . . . . . 1516 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 1516 11. Security Considerations . . . . . . . . . . . . . . . . . . . 1516 12. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 1617 Appendix A. Extension to Pre-Congestion Notification . . . . . . 1617 Appendix A.1. Label Push onto IP packet . . . . . . . . . . . . . 1718 Appendix A.2. Pushing Additional MPLS Labels . . . . . . . . . . . 1718 Appendix A.3. Admission Control or Pre-emptionFlow Termination Marking inside MPLS domain . . . . . . . . . . . . . . . . . . . . 1718 Appendix A.4. Popping an MPLS Label (not end of stack) . . . . . . 1718 Appendix A.5. Popping the last MPLS Label to expose IP header . . 1819 13. References . . . . . . . . . . . . . . . . . . . . . . . . . . 1819 13.1. Normative References . . . . . . . . . . . . . . . . . . . 1819 13.2. Informative References . . . . . . . . . . . . . . . . . . 1920 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 2021 Intellectual Property and Copyright Statements . . . . . . . . . . 2223 1. Introduction 1.1. Background [RFC3168] defines Explicit Congestion Notification for IP. The primary purpose of ECN is to allow congestion to be signalled without dropping packets. [RFC3270] defines how to support the Diffserv architecture in MPLS networks, including how to encode Diffserv Code Points (DSCPs) in an MPLS header. DSCPs may be encoded in the EXP field, while other uses of that field are not precluded. RFC3270 makes no statement about how Explicit Congestion Notification (ECN) marking might be encoded in the MPLS header. This draft defines how an operator might define some of the EXP codepoints for explicit congestion notification, without precluding other uses. In parallel to the activity defining the addition of ECN to IP [RFC3168], two proposals were made to add ECN to MPLS [Floyd][Shayman]. These proposals, however, fell by the wayside. With ECN for IP now being a proposed standard, and developing interest in using pre-congestion notification (PCN) for admission control and flow pre-emption [I-D.briscoe-tsvwg-cl-architecture],termination [I-D.ietf-pcn-architecture], there is consequent interest in being able to support ECN across IP networks consisting of MPLS-enabled domains. Therefore it is necessary to specify the protocol for including ECN in the MPLS shim header, and the protocol behavior of edge MPLS nodes. We note that in [RFC3168] there are four codepoints used for ECN marking, which are encoded using two bits of the IP header. The MPLS EXP field is the logical place to encode ECN codepoints, but with only 3 bits (8 codepoints) available, and with the same field being used to convey DSCP information as well, there is a clear incentive to conserve the number of codepoints consumed for ECN purposes. Efficient use of the EXP field has been a focus of prior drafts [Floyd] [Shayman] and we draw on those efforts in this draft as well. We also note that [RFC3168] defines default usage of the ECN field but allows for the possibility that some Diffserv PHBs might include different specifications on how the ECN field is to be used. This draft seeks to preserve that capability. 1.2. Intent Our intent is to specify how the MPLS shim header[RFC3032]header [RFC3032] should denote ECN marking and how MPLS nodes should understand whether the transport for a packet will be ECN capable. We offer this as a building block, from which to build different congestion notification systems. We do not intend to specify how the resulting congestion notification is fed back to an upstream node that can mitigate congestion. For instance, unlike [Shayman], we do not specify edge- to-edge MPLS domain feedback, but we also do not preclude it. Nonetheless, we do specify how the egress node of an MPLS domain should copy congestion notification from the MPLS shim into the encapsulated IP header if the ECN is to be carried onward towards the IP receiver. But we do NOT mandate that MPLS congestion notification must be copied into the IP header for onward transmission. This draft aims to be generic for any use of congestion notification in MPLS. Support of [RFC3168] is our primary motivation; some additional potential applications to illustrate the flexibility of our approach are described in Section 8.9. In particular, we aim to support possible future schemes that may use more than one level of congestion marking. 1.3. Terminology This document draws freely on the terminology of ECN [RFC3168] and MPLS [RFC3031]. For ease of reference, we have included some definitions here, but refer the reader to the references above for complete specifications of the relevant technologies: o CE: Congestion Experienced. One of the states with which a packet may be marked in a network supporting ECN. A packet is marked in this state by an ECN-capable router, to indicate that this router was experiencing congestion at the time the packet arrived. o ECT: ECN-capable Transport. One of the ECN states which a packet may be in when it is sent by an end system. An end system marks a packet with an ECT codepoint to indicate that the end-points of the transport protocol are ECN-capable. A router may not mark a packet as CE unless the packet was marked ECT when it arrived. o Not-ECT: Not ECN capable transport. An end system marks a packet with this codepoint to indicate that the end-points of the transport protocol are not ECN-capable. A congested router cannot mark such packets as CE, and thus can only drop them to indicate congestion. o EXP field. A 3 bit field in the MPLS label header [RFC3032] which may be used to convey Diffserv information (and is also used in this draft to carry ECN information). o PHP. Penultimate Hop Popping. An MPLS operation in which the penultimate Label Switching Router (LSR) on a Label Switched Path (LSP) removes the top label from the packet before forwarding the packet to the final LSR on the LSP. 2. Use of MPLS EXP Field for ECN We propose that LSRs configured for explicit congestion notification should use the EXP field in the MPLS shim header. However, [RFC3270] already defines use of codepoints in the EXP field for differentiated services. Although it does not preclude other compatible uses of the EXP field, this clearly seems to limit the space available for ECN, given the field is only 3 bits (8 codepoints). [RFC3270] defines two possible approaches for requesting differentiated service treatment from an LSR. o In the E-LSP approach, different codepoints of the EXP field in the MPLS shim header are used to indicate the packet's per hop behavior (PHB). o In the L-LSP approach, an MPLS label is assigned for each PHB scheduling class (PSC, as defined in [RFC3260], so that an LSR determines both its forwarding and its scheduling behavior from the label. If an MPLS domain uses the L-LSP approach, there is likely to be space in the EXP field for ECN codepoint(s). Where the E-LSP approach is used, then codepoint space in the EXP field is likely to be scarce. This draft focuses on interworking ECN marking with the E-LSP approach as it is the tougher problem. Consequently the same approach can also be applied with L-LSPs. We recommend that explicit congestion notification in MPLS should use codepoints instead of bits in the EXP field. Since not every PHB will necessarily require an associated ECN codepoint it would be wasteful to assign a dedicated bit for ECN. (There may also be cases where a given PHB might need more than one ECN-like codepoint; see Section 8.49.4 for an example.) For each PHB that uses ECN marking, we assume one EXP codepoint will be defined meaning not congestion marked (Not-CM), and at least one other codepoint will be defined meaning congestion marked (CM). Therefore, each PHB that uses ECN marking will consume at least two EXP codepoints. But PHBs that do not use ECN marking will only consume one. Further, we wish to use minimal space in the MPLS shim header to tell interior LSRs whether each packet will be received by an ECN-capable transport (ECT). Nonetheless, we must ensure that an end-point that would not understand an ECN mark will not receive one, otherwise it will not be able to respond to congestion as it should. In the past, three solutions to this problem have been proposed: o One possible approach is for congested LSRs to mark the ECN field in the underlying IP header at the bottom of the label stack. Although many commercial LSRs routinely access the IP header for other reasons (ECMP), there are numerous drawbacks to attempting to find an IP header beneath an MPLS label stack. Notably, there is the challenge of detecting the absence of an IP header when non-IP packets are carried on an LSP. Therefore we will not consider this approach further. o In the scheme suggested by [Floyd] ECT and CE are overloaded into one bit, so that a 0 means ECT while a 1 might either mean Not-ECT or it might mean CE. A packet that has been marked as having experienced congestion upstream, and then is picked out for marking at a second congested LSR, will be dropped by the second LSR since it cannot determine whether the packet has previously experienced congestion or if ECN is not supported by the transport. While such an approach seemed potentially palatable, we do not recommend it here for the following reasons. In some cases we wish to be able to use ECN marking long before actual congestion (e.g. pre-congestion notification). In these circumstances, marking rates at each LSR might be non-negligible most of the time, so the chances of a previously marked packet encountering an LSR that wants to mark it again will also be non-negligible. In the case where CE and not-ECT are indistinguishable to core routers, such a scenario could lead to unacceptable drop rates. If the typical marking rate at every router or LSR is p, and the typical diameter of the network of LSRs is d, then the probability that a marked packet will be chosen for marking more than once is 1-[Pr(never marked) + Pr(marked at exactly one hop)] = 1- [(1-p)^d + dp(1-p)^(d-1)]. For instance, with 6 LSRs in a row, each marking ECN with 1% probability, the chances of a packet that is already marked being chosen for marking a second time is 0.15%. The bit overloading scheme would therefore introduce a drop rate of 0.15% unnecessarily. Given that most modern core networks are sized to introduce near-zero packet drop, it may be unacceptable to drop over one in a thousand packets unnecessarily. o A third possible approach was suggested by [Shayman]. In this scheme, interior LSRs assume that the endpoints are ECN-capable, but this assumption is checked when the final label is popped. If an interior LSR has marked ECN in the EXP field of the shim header, but the IP header says the endpoints are not ECN capable, the edge router (or penultimate router, if using penultimate hop popping) drops the packet. We recommend this scheme, which we call `per-domain ECT checking', and define it more precisely in the following section. Its chief drawback is that it can cause packets to be forwarded after encountering congestion only to be dropped at the egress of the MPLS domain. The rationale for this decision is given in Section 220.127.116.11. 3. Per-domain ECT checking For the purposes of this discussion, we define the egress nodes of an MPLS domain as the nodes that pop the last MPLS label from the label stack, exposing the IP (or, potentially non-IP) header. Note that such a node may be the ultimate or penultimate hop of an LSP, depending on whether penultimate hop popping (PHP) is employed. In the per-domain ECT checking approach, the egress nodes take responsibility for checking whether the transport is ECN capable. This draft does not specify how these nodes should pass on congestion notification, because different approaches are likely in different scenarios. However, if congestion notification in the MPLS header is copied into the IP header, the procedure MUST conform to the specification given here. If congestion notification is passed to the transport without first passing it onward in the IP header, the approach used must take similar care to check that the transport is ECN capable before passing it ECN markings. Specifically, if the transport for a particular congestion marked MPLS packet is found not to be ECN- capable, the packet MUST be dropped at this egress node. In the per-domain ECT checking approach, only the egress nodes check whether an IP packet is destined for an ECN-capable transport. Therefore, any single LSR within an MPLS domain MUST NOT be configured to enable ECN marking unless all the egress LSRs surrounding it are already configured to handle ECN marking. We call a domain surrounded by ECN-capable egress LSRs an ECN-enabled MPLS domain. This term only implies that all the egress LSRs are ECN-enabled; some interior LSRs may not be ECN-enabled. For instance, it would be possible to use some legacy LSRs incapable of supporting ECN in the interior of an MPLS domain as long as all the egress LSRs were ECN-capable. Note that if PHP is used, the "penultimate hop" routers which perform the pop operation do need to be ECN-enabled, since they are acting in this context as egress LSRs. 4. ECN-enabled MPLS domain In the following subsections we describe various operations affecting the ECN marking of a packet that may be performed at MPLS edge and core LSRs. 4.1. Pushing (adding) one or more labels to an IP packet On encapsulating an IP packet with an MPLS label stack, the ECN field must be translated from the IP packet into the MPLS EXP field. The Not-CM (not congestion marked) state is set in the MPLS EXP field if the ECN status of the IP packet is "Not ECT" or ECT(1) or ECT(0). The CM state is set if the ECN status of the IP packet is "CE". If more than one label is pushed at one time, the same value should be placed in the EXP value of all label stack entries. 4.2. Pushing one or more labels onto an MPLS labelled packet The EXP field is copied directly from the topmost label before the push to the newly added outer label. If more than one label is being pushed, the same EXP value is copied to all label stack entries. 4.3. Congestion experienced in an interior MPLS node If the EXP codepoint of the packet maps to a PHB that uses ECN marking and the marking algorithm requires the packet to be marked, the CM state is set (irrespective of whether it is already in the CM state). If the buffer is full, a packet is dropped. 4.4. Crossing a Diffserv Domain Boundary If an MPLS-encapsulated packet crosses a Diffserv domain boundary, it may be the case that the two domains use different encodings of the same PHB in the EXP field. In such cases, the EXP field must be rewritten at the domain boundary. If the PHB is one that supports ECN, then the appropriate ECN marking should also be preserved when the EXP field is mapped at the boundary. If an MPLS-encapsulated packet that is in the CM state crosses from a domain that is ECN-enabled (as defined in Section 3) to a domain that is not ECN-enabled, then it is necessary to perform the egress checking procedures at the egress LSR of the ECN-enabled domain. This means that if the encapsulated packet is not ECN capable, the packet MUST be dropped. Note that this implies the egress LSR must be able to look beneath the MPLS header without popping the label stack. The related issue of Diffserv tunnel models is discussed in Section 4.7. 4.5. Popping an MPLS label (not the end of the stack) When a packet has more than one MPLS label in the stack and the top label is popped, another MPLS label is exposed. In this case the ECN information should be transferred from the outer EXP field to the inner MPLS label in the following manner. If the inner EXP field is Not-CM, the inner EXP field is set to the same CM or Not-CM state as the outer EXP field. If the inner EXP field is CM, it remains unchanged whatever the outer EXP field. Note that an inner value of CM and an outer value of not-CM should be considered anomalous, and SHOULD be logged in some way by the LSR. 4.6. Popping the last MPLS label in the stack When the last MPLS label is popped from the packet, its payload is exposed. If that packet is not IP, and does not have any capability equivalent to ECT, it is assumed Not-ECT and treated as such. That means that if the EXP value of the MPLS header was CM, the packet MUST be dropped. Assuming an IP packet was exposed, we have to examine whether that packet is ECT or not. A Not-ECT packet MUST be dropped if the EXP field is CM. For the remainder of this section, we describe the behavior that is required if the ECN information is to be transferred from the MPLS header into the exposed IP header for onward transmission. As noted in Section 1.2, such behavior is not mandated by this document, but may be selected by an operator. If the inner IP packet is Not-ECT, its ECN field remains unchanged if the EXP field is Not-CM. If the ECN field of the inner packet is set to ECT(0), ECT(1) or CE, the ECN field remains unchanged if the EXP field is set to Not-CM. The ECN field is set to CE if the EXP field is CM. Note that an inner value of CE and an outer value of not-CM should be considered anomalous, and SHOULD be logged in some way by the LSR. 4.7. Diffserv Tunneling Models [RFC3270] describes three tunneling models for Diffserv support across MPLS Domains, referred to as the uniform, short pipe, and pipe models. The differences between these models lie in whether the Diffserv treatment that applies to a packet while it travels along a particular LSP is carried to the last hop of the LSP and beyond the last hop. Depending on which mode is preferred by an operator, the EXP value or DSCP value of an exposed header following a label pop may or may not be dependent on the EXP value of the label that is removed by the pop operation. We believe that in the case of ECN marking, the use of these models should only apply to the encoding of the Diffserv PHB in the EXP value, and that the choice of codepoint for ECN should always be made based on the procedures described above, independent of the tunneling model. 5. ECN-disabled MPLS domain If ECN is not enabled on all the egress LSRs of a domain, ECN MUST NOT be enabled on any LSRs throughout the domain. If congestion is experienced on any LSR in an ECN-disabled MPLS domain, packets MUST be dropped, NOT marked. The exact algorithm for deciding when to drop packets during congestion (e.g. tail-drop, RED, etc.) is a local matter for the operator of the domain. 6. The use of more codepoints with E-LSPs and L-LSPs [RFC3270] gives different options with E-LSPs and L-LSPs and some of those could potentially provide ample EXP codepoints for ECN. However, deploying L-LSPs vs E-LSPs has many implications such as platform support and operational complexity. The above ECN MPLS solution should provide some flexibility. If the operator has deployed one L-LSP per PHB scheduling class, then EXP space will be a non-issue and it could be used to achieve more sophisticated ECN behavior if required. If the operator wants to stick to E-LSPs and uses a handful of EXP codepoints for Diffserv, it may be desirable to operate with a minimum number of extra ECN codepoints, even if this comes with some compromise on ECN optimality. See Section 89 for discussion of some possible deployment scenarios. 7. Relationship to tunnel behaviorWe note that in RFC 3168 [RFC3168] defines two modes of encapsulating ECN-marked IP packets inside additional IP headers when tunnels are used. The two modesa network where L-LSPs are used, ECN marking SHOULD NOT cause packets from the "full functionality"same microflow but with different ECN markings to be sent on different LSPs. As discussed in [RFC3270], packets of a single microflow should always travel on the same LSP to avoid possible misordering. Thus, ECN marking of packets on L-LSPs SHOULD only affect the EXP value of the packets. 7. Relationship to tunnel behavior in RFC 3168 [RFC3168] defines two modes of encapsulating ECN-marked IP packets inside additional IP headers when tunnels are used. The two modes are the "full functionality" and "limited functionality" modes. In the full functionality mode, the ECT information from the inner header is copied to the outer header at the tunnel ingress, but the CE information is not. In the limited functionality mode, neither ECT nor CE information is copied to the outer header, and thus ECN cannot be applied to the encapsulated packet. The behavior that is specified in Section 4 of this document resembles the "full functionality" mode in the sense that it conveys some information from inner to outer header, and in the sense that it enables full ECN support along the MPLS LSP (which is analogous to an IP tunnel in this context). However it differs in one respect, which is that the CE information is conveyed from the inner header to the outer header. Our original reason for this different design choice was to give interior routers and LSRs more information about upstream marking in multi-bottleneck cases. For instance, the flow pre- emptiontermination marking mechanism proposed for PCN works by only considering packets for marking that have not already been marked upstream. Unless existing pre-emptionflow termination marking is copied from the inner to the outer header at tunnel ingress, the mechanism doesn't pre-emptterminate enough traffic in cases where anomalous events hit multiple domains at once. [RFC3168] does not give any reasons against conveying CE information from the inner header to the outer in the "full functionality" mode. Furthermore, [RFC4301] specifies that the ECN marking should be copied from inner header to outer header in IPSEC tunnels, consistent with the approach defined here. [Briscoe][I-D.briscoe-tsvwg-ecn-tunnel] discusses this issue in more detail. In summary, the approach described in Section 4 appears to be both a sound technical choice and consistent with the current state of thinking in the IETF. 8. Example UsesDeployment Considerations 8.1. RFC3168-style ECN [RFC3168] proposes the use of ECN in TCP and introducesMarking non-ECN Capable Packets What are the useconsequences of ECN-Echo and CWR flags in the TCP header for initialization. The TCP sender responds accordingly (such asmarking a packet that is not increasingECN- capable? Even if it will be dropped before leaving the domain, doesn't this consume resources unnecessarily? The problem only arises if there is congestion window) when it receives an ECN-Echo (ECE) ACK packet (that is,downstream of an ACK packet with ECN-Echo flag setearlier congested queue in the TCP header), then the sender knows that congestion was encountered insame MPLS domain. Downstream congested LSRs might forward packets already marked, even though they will be dropped later when the networkinner IP header is found to be Not-ECT on decapsulation. Such packets might cause the path from the senderdownstream LSRs to the receiver. Itmark (or drop) other packets that they would otherwise not have had to. We expect congestion will typically be possible to enable ECNrare in anMPLS domain for Diffserv PHBs like AF and best efforts that are expected to be used by TCP and similar transports (e.g. DCCP [RFC4340]). Then end-to-end congestion controlnetworks, but it might not be. The extra unnecessary load at downstream LSRs will not be more than the fraction of marked packets from upstream LSRs, even in the worst case where no transports capable of understandingare ECN wouldcapable. Therefore the amount of unnecessary marking (or drop) on an LSR will not be able to respondmore than the product of its local marking rate and the marking rate due to approaching congestion onupstream LSRs without having to rely on packet discard to signal congestion. 8.2. ECN Co-existence with Diffserv E-LSPs Many operators today have deployed Diffserv usingwithin the E-LSP approach of [RFC3270]. In many casessame domain - typically the numberproduct of PHBs usedtwo small (often zero) probabilities. This is less than 8, and hence there remain available codepoints in the EXP space. If an operator wishedwhy we decided to support ECN for single PHB, this canuse the per-domain ECT checking approach - because the most likely effect would be accomplished by simply allocateda second codepoint to the PHBvery slightly increased marking rate, which would result in very slightly higher drop only for non-ECN-capable transports. We chose not to use the "CM" state[Floyd] alternative which introduced a low but persistent level of unnecessary packet drop for all time, even for ECN-capable transports. Although that PHB and retainingscheme did not carry traffic to the old codepoint foredge of the "not-CM" state. An operator withMPLS domain only four deployed PHBs could of course enable ECN markingto be dropped on all those PHBs. It is easydecapsulation, we felt our minor inefficiency was a small price to imagine cases where some PHBs might benefit more from ECN than others - for example, an operator might usepay. And it would get smaller still if ECN on a premium data service but not ondeployment widened. A partial solution would be to preferentially drop packets arriving at a PHB used for best effort internet traffic. As an illustrative example of howcongested router that were already marked. There is no solution to the EXP field might be used in this case, considerproblem of marking a packet when congestion is caused by another packet that should have been dropped. However, the examplechance of such an operator whooccurrence is usingvery low and the aggregated service classes proposedconsequences are not significant. It merely causes an application to very occasionally slow down its rate when it did not have to. 8.2. Non-ECN capable routers in [I-D.ietf-tsvwg-diffserv-class-aggr]. He may choosean MPLS Domain What if an MPLS domain wants to use ECN, but not all legacy routers are able to support ECN only forit? If the Assured Elastic Treatment Aggregate, usinglegacy router(s) are used in the EXP codepoint 010 forinterior, this is not a problem. They will simply have to drop the not-CM state and 011packets if they are congested, rather than mark them, which is the standard behavior for IP routers that are not ECN-enabled. If the CM state. All other codepoints couldlegacy router were used as an egress router, it would not be able to check the same as in [I-D.ietf-tsvwg-diffserv-class-aggr]. Of course any other combinationECN capability of EXP values canthe transport correctly. An operator in this position would not be used accordingable to use this solution and therefore MUST NOT enable ECN unless all egress routers are ECN- capable. 9. Example Uses 9.1. RFC3168-style ECN [RFC3168] proposes the specific setuse of PHBsECN in TCP and marking conventions used within that operator's network. 8.3. Congestion-feedback-based Traffic Engineering Shayman's traffic engineering [Shayman] proposedintroduces the use of ECN by an egress LSR feeding backECN-Echo and CWR flags in the TCP header for initialization. The TCP sender responds accordingly (such as not increasing the congestion towindow) when it receives an ingress LSR to mitigate congestion by employing dynamic traffic engineering techniques such as shifting flows toECN-Echo (ECE) ACK packet (that is, an alternate path. It proposed a new RSVP TUNNEL CONGESTION message whichACK packet with ECN-Echo flag set in the TCP header), then the sender knows that congestion was sent toencountered in the ingress LSR and ignored by transit LSRs. 8.4. PCN flow admission control and flow pre-emption [I-D.briscoe-tsvwg-cl-architecture] proposes using pre-congestion notification (PCN)network on routers withinthe path from the sender to the receiver. It would be possible to enable ECN in an edge-to-edgeMPLS domain for Diffserv regionPHBs like AF and best efforts that are expected to be used by TCP and similar transports (e.g. DCCP [RFC4340]). Then end-to-end congestion control admissionin transports capable of new flowsunderstanding ECN would be able to the region and, if necessary,respond to pre-empt existing flows in responseapproaching congestion on LSRs without having to disasters and other anomalous routing events. In this approach, the current level of PCN marking is picked up by the signalling used to initiate each flow in orderrely on packet discard to inform the admission control decision forsignal congestion. 9.2. ECN Co-existence with Diffserv E-LSPs Many operators today have deployed Diffserv using the whole region at once. As an example, a minor extension to RSVP signalling has been proposed [I-D.lefaucheur-rsvp-ecn] to carry this message, but a similarE-LSP approach has also been proposed that uses NSIS signalling [I-D.ietf-nsis-rmd]. If it is possible for LSRs to signify congestion in MPLS, PCN marking could be used for admission control and flow pre-emption across a Diffserv region, irrespectiveof whether it contained pure IP routers, MPLS LSRs, or both. Indeed, the solution could be somewhat more efficient to implement if aggregates could identify themselves by their MPLS label. Appendix A describes the mechanisms by which the necessary markings for PCN could be carried in[RFC3270]. In many cases the MPLS header. As an illustrative examplenumber of how the EXP field might bePHBs used in this case, consider the example of an operator whois using the aggregated service classes proposed in [I-D.ietf-tsvwg-diffserv-class-aggr]. He may choose to support PCN only for the Real Time Treatment Aggregate, using the EXP codepoint 100 for the not-marked (NM) state, 101 for the Admission Marked (AM) state,less than 8, and 111 for the Pre-emption Marked (PM) state. All otherhence there remain available codepoints could be the same as in [I-D.ietf-tsvwg-diffserv-class-aggr]. Of course any other combination of EXP values can be used according to the specific set of PHBs and marking conventions used within that operator's network. It might also be possible to deploy a similar solution using PCN marking over MPLS for just admission control alone, or just flow pre- emption alone, particularly if codepoint space was at a premiumin the MPLSEXP field. However, the feasibility of deploying one without the other would require further study. We also note thatspace. If an approach to deploying PCN using only a single marking codepointoperator wished to support both pre-emption and admission control has been proposed[I-D.charny-pcn-single-marking]. 9. Deployment Considerations 9.1. Marking non-ECN Capable Packets What are the consequences of marking a packet that is not ECN- capable? Even if it will be dropped before leaving the domain, doesn'tECN for single PHB, this consume resources unnecessarily? The problem only arises if there is congestion downstream of an earlier congested queue in the same MPLS domain. Downstream congested LSRs might forward packets already marked, even though they will be dropped later when the inner IP header is found to be Not-ECT on decapsulation. Such packets might cause the downstream LSRs to mark (or drop) other packets that they would otherwise not have had to. We expect congestion will typically be rare in MPLS networks, but it might not be. The extra unnecessary load at downstream LSRs will notcan be more thanaccomplished by simply allocated a second codepoint to the fractionPHB for the "CM" state of marked packets from upstream LSRs, even inthat PHB and retaining the worst case where no transports are ECN capable. Thereforeold codepoint for the amount"not-CM" state. An operator with only four deployed PHBs could of unnecessarycourse enable ECN marking (or drop)on all those PHBs. It is easy to imagine cases where some PHBs might benefit more from ECN than others - for example, an LSR willoperator might use ECN on a premium data service but not on a PHB used for best effort internet traffic. As an illustrative example of how the EXP field might be more thanused in this case, consider the productexample of its local marking rate andan operator who is using the marking rate dueaggregated service classes proposed in [I-D.ietf-tsvwg-diffserv-class-aggr]. He may choose to upstream LSRs withinsupport ECN only for the same domain - typicallyAssured Elastic Treatment Aggregate, using the product of two small (often zero) probabilities. This is why we decided to useEXP codepoint 010 for the per-domain ECT checking approach - becausenot-CM state and 011 for the most likely effect wouldCM state. All other codepoints could be a very slightly increased marking rate, which would resultthe same as in very slightly higher drop only for non-ECN-capable transports. We chose not[I-D.ietf-tsvwg-diffserv-class-aggr]. Of course any other combination of EXP values can be used according to usethe [Floyd] alternative which introduced a low but persistent levelspecific set of unnecessary packet drop for all time, even for ECN-capable transports. Although that scheme did not carryPHBs and marking conventions used within that operator's network. 9.3. Congestion-feedback-based Traffic Engineering Shayman's traffic to the edgeengineering [Shayman] presents another example application of theECN feedback in an MPLS domain onlydomain. Shayman proposed the use of ECN by an egress LSR feeding back congestion to be dropped on decapsulation, we felt our minor inefficiency was a small pricean ingress LSR to pay. And it would get smaller still if ECN deployment widened. A partial solution would bemitigate congestion by employing dynamic traffic engineering techniques such as shifting flows to preferentially drop packets arriving atan alternate path. It proposed a congested router that were already marked. There is no solutionnew RSVP message which was sent by the egress LSR to the problem of marking a packet when congestion is causedingress LSR (and ignored by another packet that should have been dropped. However,transit LSRs) to indicate congestion along the chance of such an occurrence is very low andpath. Thus, rather than providing the consequences are not significant. It merely causes an applicationsame style of congestion notification to very occasionally slow down its rate when it did not have to. 9.2. Non-ECN capable routersendpoints as defined in an[RFC3168], [Shayman] limits its scope to the MPLS Domain What ifdomain only. This application of ECN in an MPLS domain wants tocould make use ECN, but not all legacyof the ECN encoding in the MPLS header that is defined in this document. 9.4. PCN flow admission control and flow termination [I-D.ietf-pcn-architecture] proposes using pre-congestion notification (PCN) on routers are ablewithin an edge-to-edge Diffserv region to control admission of new flows to the region and, if necessary, to terminate existing flows in response to support it? Ifdisasters and other anomalous routing events. In this approach, the legacy router(s) arecurrent level of PCN marking is picked up by the signalling used to initiate each flow in the interior, this is not a problem. They will simply haveorder to dropinform the packets if theyadmission control decision for the whole region at once. For example, extensions to RSVP [I-D.lefaucheur-rsvp-ecn] and NSIS [I-D.ietf-nsis-rmd], [I-D.arumaithurai-nsis-pcn] have been proposed. If LSRs are congested, rather thanable to mark them, which is the standard behaviorpackets to signify congestion in MPLS, PCN marking could be used for admission control and flow termination across a Diffserv region, irrespective of whether it contained pure IP routers that are not ECN-enabled. Ifrouters, MPLS LSRs, or both. Indeed, the legacy router were used as an egress router, it would notsolution could be ablesomewhat more efficient to checkimplement if aggregates could identify themselves by their MPLS label. Appendix A describes the ECN capability ofmechanisms by which the transport correctly. An operator in this position would notnecessary markings for PCN could be able to use this solution and therefore MUST NOT enable ECN unless all egress routers are ECN- capable.carried in the MPLS header. 10. IANA Considerations This document makes no request of IANA. Note to RFC Editor: this section may be removed on publication as an RFC. 11. Security Considerations We believe no new vulnerabilities are introduced by this draft. We have considered whether malicious sources might be able to exploit the fact that interior LSRs will mark packets that are Not-ECT, relying on their egress LSR to drop them. Although this might allow sources to engineer a situation where more traffic is carried across an MPLS domain than should be, we figured that even if we hadn't introduced this feature, these sources would have been able to prevent these LSRs dropping this traffic anyway, simply by setting ECT in the first place. An ECN sender can use the ECN nonce [RFC3540] to detect a misbehaving receiver. The ECN nonce works correctly across an MPLS domain without requiring any specific support from the proposal in this draft. The nonce does not need to be present in the MPLS shim header. As long as the nonce is present in the IP header when the ECN information is copied from the last MPLS shim header, it will be overwritten if congestion has been experienced by an LSR. This is all that is necessary for the sender to detect a misbehaving receiver. 12. Acknowledgments Thanks to K.K. Ramakrishnan and Sally Floyd for getting us thinking about this in the first place and for providing advice on tunneling of ECN packets, and to Sally Floyd, Joe Babiarz, Ben Niven-Jenkins, Phil Eardley, Ruediger Geib, and Magnus Westerlund for their comments on the draft. Appendix A. Extension to Pre-Congestion Notification This appendix describes how the mechanisms decribed in the body of the document can be extended to support PCN [I-D.briscoe-tsvwg-cl-architecture].[I-D.ietf-pcn-architecture]. Our intent here is to show that the mechanisms are readily extended to more complex scenarios than ECN, particulary in the case where more codepoints are needed, but this appendix may be safely ignored if one is interested only in supporting ECN. Note that the PCN standards are still very much under development at the time of writing, hence the precise details contained in this appendix may be subject to change, and we stress that this appendix is for illustrative purposes only. The relevant aspects of PCN for the purposes of this discussion are: o PCN uses 3 states rather than 2 for ECN - these are referred to as admission marked (AM), pre-emptiontermination marked (PM)(TM) and not marked (NM) states. (See Section 8.49.4 for further discussion of PCN and the possibility of using fewer codepoints.) o A packet can go from NM to AM, from NM to PM,TM, or from AM to PM,TM, but no other transition is possible. o The determination of whether a packet is subject to PCN is based on the PHB of the packet. Thus, to support PCN fully in an MPLS domain for a particular PHB, a total of 3 codepoints need to be allocated for that PHB. These 3 codepoints represent the admission marked (AM), pre-emptiontermination marked (PM)(TM) and not marked (NM) states. The procedures described in Section 4 above need to be slightly modified to support this scenario. The following procedures are invoked when the topmost DSCP or EXP value indicates a PHB that supports PCN. Appendix A.1. Label Push onto IP packet If the IP packet header indicates AM, set the EXP value of all entries in the label stack to AM. If the IP packet header indicates PM,TM, set the EXP value of all entries in the label stack to PM.TM. For any other marking of the IP header, set the EXP value of all entries in the label stack to NM. Appendix A.2. Pushing Additional MPLS Labels The procedures of Section 4.2 apply. Appendix A.3. Admission Control or Pre-emptionFlow Termination Marking inside MPLS domain The EXP value can be set to AM or PMTM according to the same procedures as described in [I-D.briscoe-tsvwg-cl-phb]. For the purposes of this document, it does not matter exactly what algorithms are used to decide when to set AM or PM;TM; all that matters is that if a router would have marked AM (or PM)TM) in the IP header, it should set the EXP value in the MPLS header to the AM (or PM)TM) codepoint. Appendix A.4. Popping an MPLS Label (not end of stack) When popping an MPLS Label exposes another MPLS label, the AM or PMTM marking should be transferred to the exposed EXP field in the following manner: o If the inner EXP value is NM, then it should be set to the same marking state as the EXP value of the popped label stack entry. o If the inner EXP value is AM, it should be unchanged if the popped EXP value was AM, and it should be set to PMTM if the popped EXP value was PM.TM. If the popped EXP value was NM, this should be logged in some way and the inner EXP value should be unchanged. o If the inner EXP value is PM,TM, it should be unchanged whatever the popped EXP value was, but any EXP value other than PMTM should be logged. Appendix A.5. Popping the last MPLS Label to expose IP header When popping the last MPLS Label exposes the IP header, there are two cases to consider: o the popping LSR is NOT the egress router of the PCN region, in which case AM or PMTM marking should be transferred to the exposed IP header field; or o the popping LSR IS the egress router of the PCN region. In the latter case, the behavior of the egress LSR is defined in [I-D.briscoe-tsvwg-cl-architecture][I-D.ietf-pcn-architecture] and is beyond the scope of this document. In the former case, the marking should be transferred from the popped MPLS header to the exposed IP header as follows: o If the inner IP header value is neither AM nor PM,TM, and the EXP value was NM, then the IP header should be unchanged. For any other EXP value, the IP header should be set to the same marking state as the EXP value of the popped label stack entry. o If the inner IP header value is AM, it should be unchanged if the popped EXP value was AM, and it should be set to PMTM if the popped EXP value was PM.TM. If the popped EXP value was NM, this should be logged in some way and the inner IP header value should be unchanged. o If the IP header value is PM,TM, it should be unchanged whatever the popped EXP value was, but any EXP value other than PMTM should be logged. 13. References 13.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol Label Switching Architecture", RFC 3031, January 2001. [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack Encoding", RFC 3032, January 2001. [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, September 2001. [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- Protocol Label Switching (MPLS) Support of Differentiated Services", RFC 3270, May 2002. [RFC4301] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, December 2005. 13.2. Informative References [Briscoe] "Layered Encapsulation of Congestion Notification", June 2007. Work in progress.[Floyd] "A Proposal to Incorporate ECN in MPLS", 1999. Work in progress. http://www.icir.org/floyd/papers/ draft-ietf-mpls-ecn-00.txt [I-D.briscoe-tsvwg-cl-architecture] Briscoe, B., "An edge-to-edge Deployment[I-D.arumaithurai-nsis-pcn] Arumaithurai, M., "NSIS PCN-QoSM: A Quality of Service Model for Pre- Congestion Notification: Admission Control over a DiffServ Region", draft-briscoe-tsvwg-cl-architecture-04Pre-Congestion Notification (PCN)", draft-arumaithurai-nsis-pcn-00 (work in progress), October 2006.September 2007. [I-D.briscoe-tsvwg-cl-phb] Briscoe, B., "Pre-Congestion Notification marking", draft-briscoe-tsvwg-cl-phb-03 (work in progress), October 2006. [I-D.charny-pcn-single-marking] Charny, A., "Pre-Congestion Notification Using Single Marking for Admission and Pre-emption", draft-charny-pcn-single-marking-01[I-D.briscoe-tsvwg-ecn-tunnel] Briscoe, B., "Layered Encapsulation of Congestion Notification", draft-briscoe-tsvwg-ecn-tunnel-00 (work in progress), MarchJuly 2007. [I-D.ietf-nsis-rmd] Bader, A., "RMD-QOSM - The Resource Management in Diffserv QOS Model", draft-ietf-nsis-rmd-09draft-ietf-nsis-rmd-11 (work in progress), MarchAugust 2007. [I-D.ietf-pcn-architecture] Eardley, P., "Pre-Congestion Notification Architecture", draft-ietf-pcn-architecture-00 (work in progress), August 2007. [I-D.ietf-tsvwg-diffserv-class-aggr] Chan, K., "Aggregation of DiffServ Service Classes", draft-ietf-tsvwg-diffserv-class-aggr-02draft-ietf-tsvwg-diffserv-class-aggr-04 (work in progress), MarchAugust 2007. [I-D.lefaucheur-rsvp-ecn] Faucheur, F., "RSVP Extensions for Admission Control over Diffserv using Pre-congestion Notification (PCN)", draft-lefaucheur-rsvp-ecn-01 (work in progress), June 2006. [RFC3260] Grossman, D., "New Terminology and Clarifications for Diffserv", RFC 3260, April 2002. [RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit Congestion Notification (ECN) Signaling with Nonces", RFC 3540, June 2003. [RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram Congestion Control Protocol (DCCP)", RFC 4340, March 2006. [Shayman] "Using ECN to Signal Congestion Within an MPLS Domain", 2000. Work in progress. http://www.ee.umd.edu/~shayman/papers.d/ draft-shayman-mpls-ecn-00.txt Authors' Addresses Bruce Davie Cisco Systems, Inc. 1414 Mass. Ave. Boxborough, MA 01719 USA Email: email@example.com Bob Briscoe BT Research B54/77, Sirius House Adastral Park Martlesham Heath Ipswich Suffolk IP5 3RE United Kingdom Email: firstname.lastname@example.org June Tay BT Research B54/77, Sirius House Adastral Park Martlesham Heath Ipswich Suffolk IP5 3RE United Kingdom Email: email@example.com Full Copyright Statement Copyright (C) The IETF Trust (2007). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at firstname.lastname@example.org. AcknowledgmentAcknowledgments Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA). This document was produced using xml2rfc v1.32 (of http://xml.resource.org/) from a source in RFC-2629 XML format.