[Docs] [txt|pdf] [Tracker] [WG] [Email] [Diff1] [Diff2] [Nits]

Versions: (draft-narten-armd-problem-statement) 00 01 02 03 04 RFC 6820

Internet Engineering Task Force                                T. Narten
Internet-Draft                                                       IBM
Intended status: Informational                                  M. Karir
Expires: December 24, 2012                            Merit Network Inc.
                                                                  I. Foo
                                                     Huawei Technologies
                                                           June 22, 2012


                       Problem Statement for ARMD
                  draft-ietf-armd-problem-statement-03

Abstract

   This document examines address resolution issues related to the
   massive scaling of data centers.  Our initial scope is relatively
   narrow.  Specifically, it focuses on address resolution (ARP and ND)
   within the data center.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on December 24, 2012.

Copyright Notice

   Copyright (c) 2012 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of



Narten, et al.          Expires December 24, 2012               [Page 1]

Internet-Draft                armd-problem                     June 2012


   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.


Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
   2.  Terminology  . . . . . . . . . . . . . . . . . . . . . . . . .  3
   3.  Background . . . . . . . . . . . . . . . . . . . . . . . . . .  4
   4.  Address Resolution in IPv4 . . . . . . . . . . . . . . . . . .  6
   5.  Address Resolution in IPv6 . . . . . . . . . . . . . . . . . .  7
   6.  Generalized Data Center Design . . . . . . . . . . . . . . . .  7
     6.1.  Access Layer . . . . . . . . . . . . . . . . . . . . . . .  8
     6.2.  Aggregation Layer  . . . . . . . . . . . . . . . . . . . .  8
     6.3.  Core . . . . . . . . . . . . . . . . . . . . . . . . . . .  9
     6.4.  L3 / L2 Topological Variations . . . . . . . . . . . . . .  9
       6.4.1.  L3 to Access Switches  . . . . . . . . . . . . . . . .  9
       6.4.2.  L3 to Aggregation Switches . . . . . . . . . . . . . .  9
       6.4.3.  L3 in the Core only  . . . . . . . . . . . . . . . . . 10
       6.4.4.  Overlays . . . . . . . . . . . . . . . . . . . . . . . 10
     6.5.  Factors that Affect Data Center Design . . . . . . . . . . 10
       6.5.1.  Traffic Patterns . . . . . . . . . . . . . . . . . . . 10
       6.5.2.  Virtualization . . . . . . . . . . . . . . . . . . . . 11
       6.5.3.  Summary  . . . . . . . . . . . . . . . . . . . . . . . 11
   7.  Problem Itemization  . . . . . . . . . . . . . . . . . . . . . 12
     7.1.  ARP Processing on Routers  . . . . . . . . . . . . . . . . 12
     7.2.  IPv6 Neighbor Discovery  . . . . . . . . . . . . . . . . . 13
     7.3.  MAC Address Table Size Limitations in Switches . . . . . . 14
   8.  Summary  . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
   9.  Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . . 15
   10. IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 15
   11. Security Considerations  . . . . . . . . . . . . . . . . . . . 15
   12. Change Log . . . . . . . . . . . . . . . . . . . . . . . . . . 15
     12.1. Changes from -02 . . . . . . . . . . . . . . . . . . . . . 15
     12.2. Changes from -01 . . . . . . . . . . . . . . . . . . . . . 16
     12.3. Changes from -00 . . . . . . . . . . . . . . . . . . . . . 16
   13. Informative References . . . . . . . . . . . . . . . . . . . . 16
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 16













Narten, et al.          Expires December 24, 2012               [Page 2]

Internet-Draft                armd-problem                     June 2012


1.  Introduction

   This document examines issues related to the massive scaling of data
   centers.  Specifically, we focus on address resolution (ARP in IPv4
   and Neighbor Discovery in IPv6) within the data center.  Although
   strictly speaking the scope of address resolution is confined to a
   single L2 broadcast domain (i.e., ARP runs at the L2 layer below IP),
   the issue is complicated by routers having many interfaces on which
   address resolution must be performed or with the presence of IEEE
   802.1Q domains, where individual VLANs effectively form their own L2
   broadcast domains.  Thus, the scope of address resolution spans both
   the L2 link and the devices attached to those links.

   This document identifies potential issues associated with address
   resolution in data centers with massive number of hosts.  The scope
   of this document is intentionally relatively narrow as it mirrors the
   ARMD WG charter.  This document aims to list "pain points" that are
   being experienced in current data centers.  The goal of this document
   is to focus on address resolution issues and not other broader issues
   that might arise in data centers.


2.  Terminology

   Application:  software that runs on either a physical or virtual
      machine, providing a service (e.g., web server, database server,
      etc.)

   L2 Broadcast Domain:  The set of all links, repeaters, and switches
      that are traversed to reach all nodes that are members of a given
      L2 broadcast domain.  In IEEE 802.1Q networks, a broadcast domain
      corresponds to a single VLAN.

   Host (or server):  A computer system on the network.

   Hypervisor:  Software running on a host that allows multiple VMs to
      run on the same host.

   Virtual machine (VM):  A software implementation of a physical
      machine that runs programs as if they were executing on a
      physical, non-virtualized machine.  Applications (generally) do
      not know they are running on a VM as opposed to running on a
      "bare" host or server, though some systems provide a
      paravirtualization environment that allows an operating systems or
      application to be aware of the presences of virtualization for
      optimization purposes.





Narten, et al.          Expires December 24, 2012               [Page 3]

Internet-Draft                armd-problem                     June 2012


   ToR:  Top of Rack Switch.  A switch placed in a single rack to
      aggregate network connectivity to and from hosts in that rack.

   EoR:  End of Row Switch.  A switch used to aggregate network
      connectivity from multiple racks.  EoR switches are the next level
      of switching above ToR switches.


3.  Background

   Large, flat L2 networks have long been known to have scaling
   problems.  As the size of an L2 broadcast domain increases, the level
   of broadcast traffic from protocols like ARP increases.  Large
   amounts of broadcast traffic pose a particular burden because every
   device (switch, host and router) must process and possibly act on
   such traffic.  In extreme cases, "broadcast storms" can occur where
   the quantity of broadcast traffic reaches a level that effectively
   brings down part or all of a network.  For example, poor
   implementations of loop detection and prevention or misconfiguration
   errors can create conditions that lead to broadcast storms as network
   conditions change.  The conventional wisdom for addressing such
   problems has been to say "don't do that".  That is, split large L2
   networks into multiple smaller L2 networks, each operating as its own
   L3/IP subnet.  Numerous data center networks have been designed with
   this principle, e.g., with each rack placed within its own L3 IP
   subnet.  By doing so, the broadcast domain (and address resolution)
   is confined to one Top of Rack switch, which works well from a
   scaling perspective.  Unfortunately, this conflicts in some ways with
   the current trend towards dynamic work load shifting in data centers
   and increased virtualization as discussed below.

   Workload placement has become a challenging task within data centers.
   Ideally, it is desirable to be able to dynamically reassign workloads
   within a data center in order to optimize server utilization, add
   additional servers in response to increased demand, etc.  However,
   servers are often pre-configured to run with a given set of IP
   addresses.  Placement of such servers is then subject to constraints
   of the IP addressing restrictions of the data center.  For example,
   servers configured with addresses from a particular subnet could only
   be placed where they connect to the IP subnet corresponding to their
   IP addresses.  If each top of rack switch is acting as a gateway for
   its own subnet, a server can only be connected to the one top of rack
   switch.  This gateway switch represents the L2/L3 boundary.  A
   similar constraint occurs in virtualized environments, as discussed
   next.

   Server virtualization is fast becoming the norm in data centers.
   With server virtualization, each physical server supports multiple



Narten, et al.          Expires December 24, 2012               [Page 4]

Internet-Draft                armd-problem                     June 2012


   virtual servers, each running its own operating system, middleware
   and applications.  Virtualization is a key enabler of workload
   agility, i.e., allowing any server to host any application and
   providing the flexibility of adding, shrinking, or moving
   applications within the physical infrastructure.  Server
   virtualization provides numerous benefits, including higher
   utilization, increased data security, reduced user downtime, and even
   significant power conservation, along with the promise of a more
   flexible and dynamic computing environment.

   The discussion below focuses on VM placement and migration.  Keep in
   mind, however, that even in a non-virtualized environment, many of
   the same issues apply to individual workloads running on standalone
   machines.  For example, when increasing the number of servers running
   a particular workload to meet demand, placement of those workloads
   may be constrained by IP subnet numbering considerations, as
   discussed earlier.

   The greatest flexibility in VM and workload management occurs when it
   is possible to place a VM (or workload) anywhere in the data center
   regardless of what IP addresses the VM uses and how the physical
   network is laid out.  In practice, movement of VMs within a data
   center is easiest when VM placement and movement does not conflict
   with the IP subnet boundaries of the data center's network, so that
   the VM's IP address need not be changed to reflect its actual point
   of attachment on the network from an L3/IP perspective.  In contrast,
   if a VM moves to a new IP subnet, its address must change, and
   clients will need to be made aware of that change.  From a VM
   management perspective, management is simplified if all servers are
   on a single large L2 network.

   With virtualization, it is not uncommon to have a single physical
   server host ten (or more) VMs, each having its own IP (and MAC)
   addresses.  Consequently, the number of addresses per machine (and
   hence per subnet) is increasing, even when the number of physical
   machines stays constant.  In a few years, the numbers will likely be
   even higher.

   In the past, applications were static in the sense that they tended
   to stay in one physical place.  An application installed on a machine
   would stay on that machine because the cost of moving an application
   elsewhere was generally high.  Moreover, applications would tend to
   be placed in such a way as to facilitate communication locality.
   That is, servers would be physically located near the applications
   they accessed most heavily.  The network traffic patterns in such
   environments could thus be optimized, in some cases keeping
   significant traffic local to one network segment.  In these more
   static and carefully managed environments, it was possible to build



Narten, et al.          Expires December 24, 2012               [Page 5]

Internet-Draft                armd-problem                     June 2012


   networks that approached scaling limitations, but did not actually
   cross the threshold.

   Today, with the proliferation of VMs, traffic patterns are becoming
   more diverse and less predictable.  In particular, there can easily
   be less locality of network traffic as applications are moved for
   such reasons as reducing overall power usage (by consolidating VMs
   and powering off idle machine) or to move a an application to a
   physical server with more capacity or a lower load.  In today's
   changing environments, it is becoming more difficult to engineer
   networks as traffic patterns continually shift as VMs move around.

   In summary, both the size and density of L2 networks is increasing.
   In addition, increasingly dynamic workloads and the increased usage
   of VMs is creating pressure for ever larger L2 networks.  Today,
   there are already data centers with over 100,000 physical machines
   and many times that number of VMs.  This number will only increase
   going forward.  In addition, traffic patterns within a data center
   are also constantly changing.  Ultimately, the issues described in
   this document might be observed at any scale depending on the
   particular design of the data center.


4.  Address Resolution in IPv4

   In IPv4 over Ethernet, ARP provides the function of address
   resolution.  To determine the link-layer address of a given IP
   address, a node broadcasts an ARP Request.  The request is delivered
   to all portions of the L2 network, and the node with the requested IP
   address replies with an ARP Reply.  ARP is an old protocol, and by
   current standards, is sparsely documented.  For example, there are no
   clear requirement for retransmitting ARP Requests in the absence of
   replies.  Consequently, implementations vary in the details of what
   they actually implement [RFC0826][RFC1122].

   From a scaling perspective, there are a number of problems with ARP.
   First, it uses broadcast, and any network with a large number of
   attached hosts will see a correspondingly large amount of broadcast
   ARP traffic.  The second problem is that it is not feasible to change
   host implementations of ARP - current implementations are too widely
   entrenched, and any changes to host implementations of ARP would take
   years to become sufficiently deployed to matter.  That said, it may
   be possible to change ARP implementations in hypervisors, L2/L3
   boundary routers, and/or ToR access switches, to leverage such
   techniques as Proxy ARP [RFC1027] and/or OpenFlow [OpenFlow] infused
   directory assistance approaches.  Finally, ARP implementations need
   to take steps to flush out stale or otherwise invalid entries.
   Unfortunately, existing standards do not provide clear implementation



Narten, et al.          Expires December 24, 2012               [Page 6]

Internet-Draft                armd-problem                     June 2012


   guidelines for how to do this.  Consequently, implementations vary
   significantly, and some implementations are "chatty" in that they
   just periodically flush caches every few minutes and send new ARP
   queries.


5.  Address Resolution in IPv6

   Broadly speaking, from the perspective of address resolution, IPv6's
   Neighbor Discovery (ND) behaves much like ARP, with a few notable
   differences.  First, ARP uses broadcast, whereas ND uses multicast.
   Specifically, when querying for a target IP address, ND maps the
   target address into an IPv6 Solicited Node multicast address.  From
   an L2 perspective, sending to a multicast vs. broadcast address may
   result in the packet being delivered to all nodes, but most (if not
   all) nodes will filter out the (unwanted) query via filters installed
   in the NIC -- hosts will never see such packets.  Thus, whereas all
   nodes must process every ARP query, ND queries are processed only by
   the nodes to which they are intended.  In cases where multicast
   filtering can't effectively be implemented in the NIC (e.g., as on
   hypervisors supporting virtualization), filtering would need to be
   done in software (e.g., in the hypervisor's vSwitch).


6.  Generalized Data Center Design

   There are many different ways in which data center networks might be
   designed.  The designs are usually engineered to suit the particular
   application that is being deployed in the data center.  For example,
   a massive web server farm might be engineered in a very different way
   than a general-purpose multi-tenant cloud hosting service.  However
   in most cases the designs can be abstracted into a typical three-
   layer model consisting of an access layer, an aggregation layer and
   the Core.  The access layer generally refers to the switches that are
   closest to the physical or virtual severs, the aggregation layer
   serves to interconnect multiple access layer devices.  The Core
   switches connect the aggregation switches to the larger network core.
   Figure 1 shows a generalized data center design, which captures the
   essential elements of various alternatives.












Narten, et al.          Expires December 24, 2012               [Page 7]

Internet-Draft                armd-problem                     June 2012


                  +-----+-----+     +-----+-----+
                  |   Core0   |     |    Core1  |      Core
                  +-----+-----+     +-----+-----+
                        /    \        /       /
                       /      \----------\   /
                      /    /---------/    \ /
                    +-------+           +------+
                  +/------+ |         +/-----+ |
                  | Aggr11| + --------|AggrN1| +      Aggregation Layer
                  +---+---+/          +------+/
                    /     \            /      \
                   /       \          /        \
                 +---+    +---+      +---+     +---+
                 |T11|... |T1x|      |TN1|     |TNy|  Access Layer
                 +---+    +---+      +---+     +---+
                 |   |    |   |      |   |     |   |
                 +---+    +---+      +---+     +---+
                 |   |... |   |      |   |     |   |
                 +---+    +---+      +---+     +---+  Server racks
                 |   |... |   |      |   |     |   |
                 +---+    +---+      +---+     +---+
                 |   |... |   |      |   |     |   |
                 +---+    +---+      +---+     +---+

   Typical Layered Architecture in DC

                                 Figure 1

6.1.  Access Layer

   The access switches provide connectivity directly to/from physical
   and virtual servers.  The access layer may be implemented by wiring
   the servers within a rack to a top-of-rack (ToR) switch or, less
   commonly, the servers could be wired directly to an end-of-row (EoR)
   switch.  A server rack may have a single uplink to one access switch,
   or may have dual uplinks to two different access switches.

6.2.  Aggregation Layer

   In a typical data center, aggregation switches interconnect many ToR
   switches.  Usually there are multiple parallel aggregation switches,
   serving the same group of ToRs to achieve load sharing.  It is no
   longer uncommon to see aggregation switches interconnecting hundreds
   of ToR switches in large data centers.







Narten, et al.          Expires December 24, 2012               [Page 8]

Internet-Draft                armd-problem                     June 2012


6.3.  Core

   Core switches connect multiple aggregation switches and are the data
   center gateway(s) to external networks or interconnect to different
   sets of racks within one data center.

6.4.  L3 / L2 Topological Variations

6.4.1.  L3 to Access Switches

   In this scenario the L3 domain is extended all the way to the access
   switches.  Each rack enclosure consists of a single L2 domain, which
   is confined to the rack.  In general, there are no significant ARP/ND
   scaling issues in this scenario as the L2 domain cannot grow very
   large.  This topology has benefits in scenarios where servers
   attached to a particular access switch generally run applications
   that are confined to using a single subnet.  These applications
   aren't moved (migrated) to other racks which might be attached to
   different access switches (and different IP subnets).  A small server
   farm or very static compute cluster might be best served via this
   design.

6.4.2.  L3 to Aggregation Switches

   When the L3 domain only extends to aggregation switches, hosts in any
   of the IP subnets configured on the aggregation switches can be
   reachable via L2 through any access switches if access switches
   enable all the VLANs.  This topology allows a greater level of
   flexibility as servers attached to any access switch can be reloaded
   with applications that have been provisioned with IP addresses from
   multiple prefixes as needed.  Further, in such an environment, VMs
   can migrate between racks without IP address changes.  The drawback
   of this design however is that multiple VLANs have to be enabled on
   all access switches and all access-facing ports on aggregation
   switches.  Even though L2 traffic is still partitioned by VLANs, the
   fact that all VLANs are enabled on all ports can lead to broadcast
   traffic on all VLANs to traverse all links and ports, which is same
   effect as one big L2 domain on the access-facing side of the
   aggregation switch.  In addition, internal traffic itself might have
   to cross different L2 boundaries resulting in significant ARP/ND load
   at the aggregation switches.  This design provides a good tradeoff
   between flexibility and L2 domain size.  A moderate sized data center
   might utilize this approach to provide high availability services at
   a single location.







Narten, et al.          Expires December 24, 2012               [Page 9]

Internet-Draft                armd-problem                     June 2012


6.4.3.  L3 in the Core only

   In some cases where a wider range of VM mobility is desired (i.e.
   greater number of racks among which VMs can move without IP address
   change), the L3 routed domain might be terminated at the core routers
   themselves.  In this case VLANs can span across multiple groups of
   aggregation switches, which allow hosts to be moved among more number
   of server racks without IP address change.  This scenario results in
   the largest ARP/ND performance impact as explained later.  A data
   center with very rapid workload shifting may consider this kind of
   design.

6.4.4.  Overlays

   There are several approaches where overlay networks can be used to
   build very large L2 networks to enable VM mobility.  Overlay networks
   using various L2 or L3 mechanisms allow interior switches/routers to
   mask host addresses.  In addition, L3 overlays can help the data
   center designer control the size of the L2 domain and also enhance
   the ability to provide multi tenancy in data center networks.
   However, the use of overlays does not eliminate traffic associated
   with address resolution, it simply moves it to regular data traffic.
   That is, address resolution is implemented in the overlay, and is not
   directly visible to the switches of the DC network.

   A potential problem that arises in a large data center is when a
   large number of hosts communicate with their peers in different
   subnets, all these hosts send (and receive) data packets to their
   respective L2/L3 boundary nodes as the traffic flows are generally
   bi-directional.  This has the potential to further highlight any
   scaling problems.  These L2/L3 boundary nodes have to process ARP/ND
   requests sent from originating subnets and resolve physical (MAC)
   addresses in the target subnets for what are generally bi-directional
   flows.  Therefore, for maximum flexibility in managing the data
   center workload, it is often desirable to use overlays to place
   related groups of hosts in the same topological subnet to avoid the
   L2/L3 boundary translation.  The use of overlays in the data center
   network can be a useful design mechanism to help manage a potential
   bottleneck at the L2 / L3 boundary by redefining where that boundary
   exists.

6.5.  Factors that Affect Data Center Design

6.5.1.  Traffic Patterns

   Expected traffic patterns play an important role in designing the
   appropriately sized access, aggregation and core networks.  Traffic
   patterns also vary based on the expected use of the data center.



Narten, et al.          Expires December 24, 2012              [Page 10]

Internet-Draft                armd-problem                     June 2012


   Broadly speaking it is desirable to keep as much traffic as possible
   on the access layer in order to minimize the bandwidth usage at the
   aggregation layer.  If the expected use of the data center is to
   serve as a large web server farm, where thousands of nodes are doing
   similar things and the traffic pattern is largely in and out a large
   data center, an access layer with EoR switches might be used as it
   minimizes complexity, allows for servers and databases to be located
   in the same L2 domain and provides for maximum density.

   A data center that is expected to host a multi-tenant cloud hosting
   service might have some completely unique requirements.  In order to
   isolate inter-customer traffic smaller L2 domains might be preferred
   and though the size of the overall data center might be comparable to
   the previous example, the multi-tenant nature of the cloud hosting
   application requires a smaller more compartmentalized access layer.
   A multi-tenant environment might also require the use of L3 all the
   way to the access layer ToR switch.

   Yet another example of an application with a unique traffic pattern
   is a high performance compute cluster where most of the traffic is
   expected to stay within the cluster but at the same time there is a
   high degree of crosstalk between the nodes.  This would once again
   call for a large access layer in order to minimize the requirements
   at the aggregation layer.

6.5.2.  Virtualization

   Using virtualization in the data center further serves to increase
   the possible densities that can be achieved.  Virtualization also
   further complicates the requirements on the access layer as that
   determines the scope of server migrations or failover of servers on
   physical hardware failures.

   Virtualization also can place additional requirements on the m
   aggregation switches in terms of address resolution table size and
   the scalability of any address learning protocols that might be used
   on those switches.  The use of virtualization often also requires the
   use of additional VLANs for High Availability beaconing which would
   need to span across the entire virtualized infrastructure.  This
   would require the access layer to span as wide as the virtualized
   infrastructure.

6.5.3.  Summary

   The designs described in this section have a number of tradeoffs.
   The L3 to access switches design described in Section 6.4.1 is the
   only design that constrains L2 domain size in a fashion that avoids
   ARP/ND scaling problems.  However, that design has limitations and



Narten, et al.          Expires December 24, 2012              [Page 11]

Internet-Draft                armd-problem                     June 2012


   does not address some of the other requirements that lead to
   configurations that make use of larger L2 domains.  Consequently,
   ARP/ND scaling issues are a real problem in practice.


7.  Problem Itemization

   This section articulates some specific problems or "pain points" that
   are related to large data centers.  It is a future activity to
   determine which of these areas can or will be addressed by ARMD or
   some other IETF WG.

7.1.  ARP Processing on Routers

   One pain point with large L2 broadcast domains is that the routers
   connected to the L2 domain need to process "a lot of" ARP traffic.
   Even though the vast majority of ARP traffic may well not be aimed at
   that router, the router still has to process enough of the ARP
   Request to determine whether it can safely be ignored.  The ARP
   algorithm specifies that a recipient must update its ARP cache if it
   receives an ARP query from a source for which it has an entry
   [RFC0826].

   ARP processing in routers is commonly handled in a "slow path"
   software processor rather than directly by a hardware ASIC as is the
   case when forwarding packets.  Such a design significantly limits the
   rate at which ARP traffic can be processed.  Current implementations
   today can support ARP processing in the low thousands of ARP packets
   per second, which is several orders of magnitude lower than the rate
   at which packets can be forwarded by ASICs.

   To further reduce the ARP load, some routers have implemented
   additional optimizations in their forwarding ASIC paths.  For
   example, some routers can be configured to discard ARP Requests for
   target addresses other than those assigned to the router.  That way,
   the router's software processor only receives ARP Requests for
   addresses it owns and must respond to.  This can significantly reduce
   the number of ARP Requests that must be processed by the router.

   Another optimization concerns reducing the number of ARP queries
   targeted at routers, whether for address resolution or to validate
   existing cache entries.  Some routers can be configured to broadcast
   periodic gratuitous ARPs.  Upon receipt of a gratuitous ARP,
   implementations mark the associated entry as "fresh", resetting the
   aging timer to its maximum setting.  Consequently, sending out
   periodic gratuitous ARPs can effectively prevent nodes from needing
   to send ARP Requests intended to revalidate stale entries for a
   router.  The net result is an overall reduction in the number of ARP



Narten, et al.          Expires December 24, 2012              [Page 12]

Internet-Draft                armd-problem                     June 2012


   queries routers receive.  Gratuitous ARPs, broadcast to all nodes in
   the L2 broadcast domain, can also pre-populate ARP caches on
   neighboring devices, further reducing ARP traffic.

   Finally, another area concerns how routers process IP packets for
   which no ARP entry exists.  Such packets must be held in a queue
   while address resolution is performed.  Once an ARP query has been
   resolved, the packets are forwarded on.  Again, the processing of
   such packets is handled in the "slow path".  This effectively limits
   the rate at which a router can process ARP "cache misses" and is
   viewed as a problem in some deployments today.  Additionally, if no
   response is received, the router has to send the ARP/ND query
   multiple times.  If no response is received after a number of ARP/ND
   requests, the router needs to drop all those data packets.  This
   entire process can be CPU intensive.

   Although address-resolution traffic remains local to one L2 network,
   some data center designs terminate L2 domains at individual
   aggregation switches/routers (e.g., see Section 4.4.2).  Such routers
   can be connected to a large number of interfaces (e.g., 100 or more).
   While the address resolution traffic on any one interface may be
   manageable, the aggregate address resolution traffic across all
   interfaces can become problematic.

   Another variant of the above issue has individual routers servicing a
   relatively small number of interfaces, with the individual interfaces
   themselves serving very large subnets.  Once again, it is the
   aggregate quantity of ARP traffic seen across all of the router's
   interfaces that can be problematic.  This "pain point" is essentially
   the same as the one discussed above, the only difference being
   whether a given number of hosts are spread across a few large IP
   subnets or many smaller ones.

   When hosts in two different subnets under the same L2/L3 boundary
   router need to communicate with each other, the L2/L3 router not only
   has to initiate ARP/ND requests to the target's subnet, it also has
   to process the ARP/ND requests from the originating subnet.  This
   process further adds to the overall ARP processing load.

7.2.  IPv6 Neighbor Discovery

   Though IPv6's Neighbor Discovery behaves much like ARP there are
   several notable differences which result in a different set of
   potential issues.  From an L2 perspective there is the simple
   difference between sending to a multicast versus broadcast address
   which results in ND queries only being processed by the nodes for
   which they are intended.




Narten, et al.          Expires December 24, 2012              [Page 13]

Internet-Draft                armd-problem                     June 2012


   Another key difference concerns revalidating stale ND entries.  ND
   requires that nodes periodically re-validate any entries they are
   using, to ensure that bad entries are timed out quickly enough that
   TCP does not terminate a connection.  Consequently, some
   implementations will send out "probe" ND queries to validate in-use
   ND entries as frequently as every 35 seconds [RFC4861].  Such probes
   are sent via unicast (unlike in the case of ARP).  However, on larger
   networks, such probes can result in routers receiving many such
   queries.  Unfortunately, the IPv4 mitigation technique of sending
   gratuitous ARPs does not work in IPv6.  The ND specification
   specifically states that gratuitous ND "updates" cannot cause an ND
   entry to be marked "valid".  Rather, such entries are marked "probe",
   which causes the receiving node to (eventually) generate a probe back
   to the sender, which in this case is precisely the behavior that the
   router is trying to prevent!

   It should be noted that ND does not require the sending of probes in
   all cases.  Section 7.3.1 of [RFC4861] describes a technique whereby
   hints from TCP can be used to verify that an existing ND entry is
   working fine and does not need to be revalidated.

7.3.  MAC Address Table Size Limitations in Switches

   L2 switches maintain L2 MAC address forwarding tables for all sources
   and destinations traversing through the switch.  These tables are
   populated through learning and are used to forward L2 frames to their
   correct destination.  The larger the L2 domain, the larger the tables
   have to be.  While in theory a switch only needs to keep track of
   addresses it is actively using (sometimes called "conversational
   learning"), switches flood broadcast frames (e.g., from ARP),
   multicast frames (e.g., from Neighbor Discovery) and unicast frames
   to unknown destinations.  Switches add entries for the source
   addresses of such flooded frames to their forwarding tables.
   Consequently, MAC address table size can become a problem as the size
   of the L2 domain increases.  The table size problem is made worse
   with VMs, where a single physical machine now hosts many VMs (in the
   10's today, but growing rapidly as the number of cores per CPU
   increases), since each VM has its own MAC address that is visible to
   switches.

   When L3 extends all the way to access switches (see Section 4.4.1),
   the size of MAC address tables in switches is not generally a
   problem.  When L3 extends only to aggregation switches (see Section
   4.4.2), however, MAC table size limitations can be a real issue.

   Finally, IPv6 and IPv4 are often run simultaneously and in parallel
   on the same network, i.e., in dual-stack mode.  In such environments,
   the IPv4 and IPv6 issues enumerated above compound each other.



Narten, et al.          Expires December 24, 2012              [Page 14]

Internet-Draft                armd-problem                     June 2012


8.  Summary

   This document has outlined a number of issues related to address
   resolution in large data centers.  In particular we have described
   different scenarios where such issues might arise, what these
   potential issues are, and along with outlining fundamental factors
   that cause them.  It is hoped that describing specific pain points
   will facilitate a discussion as to whether and how to best address
   them.


9.  Acknowledgments

   This document has been significantly improved by comments from David
   Black, Linda Dunbar, Donald Eastlake, Anoop Ghanwani, Sue Hares,
   Benson Schliesser, T. Sridhar and Lucy Yong.  Igor Gashinsky deserves
   additional credit for highlighting some of the ARP-related pain
   points and for clarifying the difference between what the standards
   require and what some router vendors have actually implemented in
   response to operator requests.


10.  IANA Considerations

   This document makes no request of IANA.  [Note: this section should
   be removed upon final RFC publication.]


11.  Security Considerations

   This documents lists existing problems or pain points with address
   resolution in data centers.  This document does not create any
   security implications nor does it have any security implications.
   The security vulnerabilities in ARP are well known and this document
   does not change or mitigate them in any way.


12.  Change Log

12.1.  Changes from -02

   1.  Wordsmithing and editorial improvements in response to comments
       from David Black, Donald Eastlake, Anoop Ghanwani, Benson
       Schliesser, T. Sridhar and Lucy Yong







Narten, et al.          Expires December 24, 2012              [Page 15]

Internet-Draft                armd-problem                     June 2012


12.2.  Changes from -01

   1.  Wordsmithing and editorial improvements.

12.3.  Changes from -00

   1.  Merged draft-karir-armd-datacenter-reference-arch-00.txt into
       this document.

   2.  Added section explaining how ND differs from ARP and the
       implication on address resolution "pain".


13.  Informative References

   [OpenFlow]
              McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G.,
              Peterson, L., Rexford, J., Shenker, S., and J. Turner,
              "OpenFlow: Enabling Innovation in Campus Networks",
              March 2008.

   [RFC0826]  Plummer, D., "Ethernet Address Resolution Protocol: Or
              converting network protocol addresses to 48.bit Ethernet
              address for transmission on Ethernet hardware", STD 37,
              RFC 826, November 1982.

   [RFC1027]  Carl-Mitchell, S. and J. Quarterman, "Using ARP to
              implement transparent subnet gateways", RFC 1027,
              October 1987.

   [RFC1122]  Braden, R., "Requirements for Internet Hosts -
              Communication Layers", STD 3, RFC 1122, October 1989.

   [RFC4861]  Narten, T., Nordmark, E., Simpson, W., and H. Soliman,
              "Neighbor Discovery for IP version 6 (IPv6)", RFC 4861,
              September 2007.


Authors' Addresses

   Thomas Narten
   IBM

   Email: narten@us.ibm.com







Narten, et al.          Expires December 24, 2012              [Page 16]

Internet-Draft                armd-problem                     June 2012


   Manish Karir
   Merit Network Inc.

   Email: mkarir@merit.edu


   Ian Foo
   Huawei Technologies

   Email: Ian.Foo@huawei.com









































Narten, et al.          Expires December 24, 2012              [Page 17]


Html markup produced by rfcmarkup 1.108, available from http://tools.ietf.org/tools/rfcmarkup/