[Docs] [txt|pdf] [Tracker] [Email] [Diff1] [Diff2] [Nits]

Versions: 00 01

Individual                                                      R. White
Internet-Draft                                                  R. Adams
Expires: April 4, 2005                                     Cisco Systems
                                                               V. Manral
                                                            SiNett Corp.
                                                         October 4, 2004



  Considerations in Benchmarking Routing Protocol Network Convergence
                    draft-white-network-benchmark-01


Status of this Memo


   This document is an Internet-Draft and is subject to all provisions
   of section 3 of RFC 3667.  By submitting this Internet-Draft, each
   author represents that any applicable patent or other IPR claims of
   which he or she is aware have been or will be disclosed, and any of
   which he or she become aware will be disclosed, in accordance with
   RFC 3668.


   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.


   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."


   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.


   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.


   This Internet-Draft will expire on April 4, 2005.


Copyright Notice


   Copyright (C) The Internet Society (2004).


Abstract


   This document attempts to discuss some of the definitions required to
   undertake the specifications of benchmarks measuring routing
   protocols convergence within a network, and also to discuss some of
   the possible ways to benchmark a routing protocol performance within
   a network, and some of the implications of those benchmarks.  The




White, et al.            Expires April 4, 2005                  [Page 1]


Internet-Draft     Considerations in Benchmarking RPs       October 2004



   definition of convergence is discussed first, then polling network
   devices.  Several tests which are commonly used to measure network
   convergence are examined.


   This draft does not attempt to define what techniques should be used
   to benchmark network convergence, but only to provide considerations
   testers should consider when attempting to measure network
   convergence using various methods.


1.  Motivation


   As the ability to benchmark components within a network appears to be
   coming under greater scrutiny, and specifications are being written
   to standardize ways to measure the performance of individual
   components within given frameworks, the next level of benchmarking
   has not been approached, that of measuring the performance of
   networks.  But what is meant when we say the performance of a
   network, from the perspective of routing protocols? Various tests
   have been used in the past to measure the convergence of a network,
   some of which actually measure completely different things than
   others.


   It's important to attempt to examine the measurement of network
   convergence in a way that exposes these differences, and helps
   vendors, end users, and those in the research community have some
   common ground when discussing network convergence.


2.  A Problem of Definitions


   As we examine the issues and concepts surrounding the measurement of
   network performance in terms of convergence, we find that most of the
   basic problems we face surround defining the terms in use.  For
   instance, what is convergence, exactly? What is a network? In the
   following sections, we discuss each of these concepts, and attempt to
   address each one.


2.1  Networks


   In its most nominal form, a network is composed of a group of devices
   interconnected in some way, which send data over these
   interconnections for various purposes.  But, when we discuss the
   concept of routing protocol convergence within a network, the
   definition needs to be more precise.  For instance, since hosts do
   not, generally, participate in routing, should they be considered a
   part of the network when benchmarking the performance of a routing
   protocol?  The obvious answer appears to be a resounding no, but, in
   some possible tests types, hosts, acting as testing devices, play a
   large part in the test itself.




White, et al.            Expires April 4, 2005                  [Page 2]


Internet-Draft     Considerations in Benchmarking RPs       October 2004



   When considering tests in which hosts participate as traffic or route
   generators, or other testing devices, we must consider the impact
   these hosts have on the test results, although we may not consider
   them a part of the network.


2.2  Convergence


   Convergence is probably one of the hardest words in networking to
   define.  Just about everyone who has worked on networks for a period
   of time knows what it means, but no-one can explain it sufficiently
   to someone who doesn't understand how a network works for it to be
   understood.  In fact, this is because there are several different
   meanings attributed to convergence, and which meaning is intended
   depends on the context in which the word is set.  Convergence can
   mean:
   o  The time at which all the routing protocol processes running on
      devices which participate in routing in the network agree on the
      best path to each reachable destination in the network.
   o  The time at which the best path to each reachable destination in
      the network has been loaded into some local table which may then
      be used to forward packets (the routing information base, or RIB).
   o  The time at which each router in the network has built the tables
      necessary to actually forward packets through the net- work, so
      that a packet transmitted from one part of the network would
      actually reach any given reachable destination within the network.


   For instance, on a Cisco router, show ip ospf stats would allow the
   tester to see the time of the last completed SPF, show ip route would
   allow the tester to see what routes are installed in the RIB, and
   show ip cef would allow the tester to see the forwarding information
   which has been built from the RIB.  Each test designed to measure the
   performance of routing protocols within a network must determine
   which type of convergence is being measured, if that measurement is
   acceptable to the information being gathered, and which test will
   actually measure the desired type of convergence.


2.3  Polling Devices in a Network


   One common way to measure network convergence is to poll the devices
   in the network, using some command supplied within the routing
   software, to determine when particular events have occurred, or par-
   ticular pieces of information have reached all the routers in the
   network.  Polling eliminates the need for the clock of each device
   within the network to be synchronized for the test to have meaningful
   results.  However, there are some issues with the rate of polling
   dev- ices within the network which need to be addressed in any test
   which polls devices for this information; the first is the rate at
   which polling takes place.




White, et al.            Expires April 4, 2005                  [Page 3]


Internet-Draft     Considerations in Benchmarking RPs       October 2004



   If, in a test, you are attempting to measure some parameter to within
   one second of its occurrence, then you would need to poll at a rate
   much higher than once per second.


          test starts here
           |
           |                   event occurs here
           |                   |
           v                   v
         -+----------+----------+----------+---
          ^          ^          ^          ^
          |          |          |          |
          0 seconds  1 second   2 seconds  3 seconds


   For instance, in this time line, suppose a polling event is set up to
   take place every second.  An event is started just after some polling
   event takes place, but the polling process doesn't recognize the test
   as starting until the 1 second poll.  An event occurs just before the
   2 second poll, and the polling process detects this at the 2 second
   poll.  The polling process would indicate that from the time the
   event started until the time the event has finished, one second has
   elapsed.  In reality, closer to two seconds has elapsed.


   The interval of the polling process can be reduced until the
   measurement is felt to be accurate, but it should be at least half of
   the desired accuracy.  Common practice actually shows that it should
   be about one tenth of the desired accuracy.


   A second consideration when polling for network events is the
   performance of the device running the polling process.  If the
   process cannot poll each device at the scheduled interval, or the
   polling is "jittered," the time between each actual poll varies by
   some amount, the accuracy of the tests will be called into question.
   The amount of jitter introduced by the polling device, and the rate
   at which the device can effectively poll, should be measured in some
   way, and this measurement should be taken into account when designing
   tests which rely on polling.


   Finally, when polling devices to determine when a network event
   occurs, issues with serialization must be considered.  Most devices
   used for polling will not be able to poll several devices within the
   network at once, and will thus serialize the polling of devices.










White, et al.            Expires April 4, 2005                  [Page 4]


Internet-Draft     Considerations in Benchmarking RPs       October 2004



         p1   p3  p5  p7  p9
          | p2| p4| p6| p8| p10
          | | | | | | | | | |
          v v v v v v v v v v
        -+----------+----------+----------+---
         ^          ^          ^          ^
         |          |          |          |
         0 seconds  1 second   2 seconds  3 seconds


   Suppose, for instance, a single device is polling ten devices in the
   network.  If it can poll five devices per second, it will take a full
   two seconds for it to detect any event on all ten devices, giving an
   effective accuracy of about four seconds.  The amount of time
   required for a polling device to serialize through all the devices it
   is polling needs to be considered when polling a very large number of
   devices.


2.4  Passing Traffic Through the Network to Determine Convergence


   One of the most widely used tests for determining network convergence
   is starting some traffic stream at one end of a network, disrupting
   or completing the network, and determining how long the traffic
   stream is either not delivered, or takes to be delivered.  For
   instance:


   Source----R1----R2----Sink


   A traffic stream is generated on Source, and the link between R1 and
   R2 is connected in some way.  The time between the connection of this
   link and the arrival of the traffic at the Sink is measured as
   network convergence.  This type of test is extremely useful in
   testing real the response of a network to changing conditions.  There
   are some considerations which should be examined when using this sort
   of test, or examining the results of this sort of test


2.4.1  The Various Elements of Performance Cannot Be Separated


   Using this sort of testing, there is no way to separate the
   performance of a routing protocol from the performance the
   interaction between the routing protocol and the forwarding engine,
   nor from the performance of the forwarding engine itself.  In many
   tests, this is acceptable, since these are all elements of the
   network in total, but if specific elements of routing protocol
   performance are being measured, such tests can be problematic when
   attempting to analyze the results.







White, et al.            Expires April 4, 2005                  [Page 5]


Internet-Draft     Considerations in Benchmarking RPs       October 2004



2.4.2  The Total Convergence of the Network May Not Be Measured


   If you have the following topology:


           Source-----R1----R2-----Sink
                      |      |
                      R3    R4
                      |      |
                      R5----R6


   Suppose a traffic stream is sourced from Source, and then all the
   devices in the network are brought up (R1 through R6).  The time from
   the device startup to the traffic stream reaching the sink is
   measured as network convergence.


   As soon as the path Source, R1, R2, Sink converges, the Sink will
   begin receiving traffic, and the network will be considered to be
   converged by the test.  However, without polling the remaining
   routers, R3 through R6, there is no way to know if those routers have
   also converged on the best path to the Sink and the Source.  While
   this example may be considered extreme, there are many complex
   topologies where:
   o  The path chosen by the traffic stream may not be the path
      expected.
   o  The path chosen by the traffic stream may switch during network
      convergence, with the stream taking some secondary path at first,
      and the successively better paths converging over the life of the
      test.
   o  The path chosen by the traffic stream switches so quickly that no
      traffic is lost, while the routing protocols still take some time
      to converge.


   Tests which rely on traffic passing through the network to determine
   network convergence times should thoroughly examine the way in which
   the test topology converges, and examine the consistency of that
   convergence, with enough test runs to get a good feel for the range
   of possible results.  Examining the same test sequence with slight
   changes in the network topology may help to provide an understanding
   of how the network under test converges, and also may help to provide
   more insight into the factors impacting convergence in the test
   network.


   It's also possible that if the test network does not converge
   completely for some time after the test traffic successfully passes
   through the topology, the continuing convergence could impact the
   results of a second test run, if the test runs are placed too closely
   together.  If a first test is run, and a second test is started
   immediately on traffic making it through the test topology, the




White, et al.            Expires April 4, 2005                  [Page 6]


Internet-Draft     Considerations in Benchmarking RPs       October 2004



   results of the second test may be skewed by convergence which is
   still taking place from the first test run.


   These are important considerations which should be noted when
   examining or performing tests which rely on the presence of a data
   stream within a routing system to measure convergence.


3  Informative References


   [OSPF-BENCH]
              Manral, V., White, R. and A. Shaikh, "Benchmarking Basic
              OSPF Single Router Control Plane Convergence",
              draft-ietf-bmwg-ospfconv-intraarea-10 (work in progress),
              July 2004.



Authors' Addresses


   Russ White
   Cisco Systems
   riw@cisco.com


   Robert Adams
   Cisco Systems
   robeadam@cisco.com


   Vishwas Manral
   SiNett Corp.
   vishwas@sinett.com























White, et al.            Expires April 4, 2005                  [Page 7]


Internet-Draft     Considerations in Benchmarking RPs       October 2004



Intellectual Property Statement


   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.


   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.


   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at
   ietf-ipr@ietf.org.



Disclaimer of Validity


   This document and the information contained herein are provided on an
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
   ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
   INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
   INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.



Copyright Statement


   Copyright (C) The Internet Society (2004).  This document is subject
   to the rights, licenses and restrictions contained in BCP 78, and
   except as set forth therein, the authors retain all their rights.



Acknowledgment


   Funding for the RFC Editor function is currently provided by the
   Internet Society.





White, et al.            Expires April 4, 2005                  [Page 8]

Html markup produced by rfcmarkup 1.129d, available from https://tools.ietf.org/tools/rfcmarkup/