[Docs] [txt|pdf] [Tracker] [WG] [Email] [Nits]

Versions: 00

   Network Working Group
   INTERNET-DRAFT
   Expires in: April 2004
                                                Scott Poretsky
                                                Quarry Technologies

                                                                Shankar Rao
                                                                Qwest Communications

                                                                Ray Piatt
                                                                Cable and Wireless

                                                October 2003


                        Framework for Accelerated Stress Benchmarking
                <draft-ietf-bmwg-acc-bench-framework-00.txt>


   Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force  (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other
   documents at any time.  It is inappropriate to use Internet-Drafts
   as reference material or to cite them other than as "work in
   progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.


   Abstract
   This document provides a framework for executing the Accelerated
   Stress Benchmarking.  It is intended that this framework be applied
   with the Terminology document when using the Methodology document.
   Discussion to specify and apply Startup Conditions, Configuration
   Sets, and Instability Conditions is provided with examples.  The
   motivation and benefits of stress testing are also discussed.






Poretsky, Rao, Piatt                                                            [Page 1]

INTERNET-DRAFT Framework for Accelerated Stress Benchmarking  October 2003

Table of Contents
     1. Introduction ................................................ 2
     2. Existing definitions ........................................ 2
     3. Motivation for Accelerated Stress Benchmarking............... 2
     4. Application of Configuration Sets............................ 3
     5. Application of Startup Conditions............................ 5
     6. Application of Instability Conditions........................ 6
     7. Service Provider Application of Accelerated Stress Testing... 6
     8. Security Considerations...................................... 6
     9. References................................................... 6
     10. Author's Address............................................ 6
     11. Full Copyright Statement.................................... 7


   1. Introduction
   This document provides the motivation and framework to perform
   Accelerated Stress Benchmarking.  The terminology to be used
   for Accelerated Stress Benchmarking is defined in [1] and the
   methodology is provided in [2].  This document discusses how to
   apply the terminology to the benchmarking for producing effective
   reproducible tests.  Configuration Sets, Startup Conditions, and
   Instability Conditions are defined [1] and examples are provided
   in this document.

   2.  Existing definitions
   RFC 1242 "Benchmarking Terminology for Network Interconnect
   Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching
   Devices" should be consulted before attempting to make use of this
   document.

   For the sake of clarity and continuity this RFC adopts the template
   for definitions set out in Section 2 of RFC 1242.  Definitions are
   indexed and grouped together in sections for ease of reference.
   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in
   this document are to be interpreted as described in RFC 2119.

   3. Motivation for Accelerated Stress Benchmarking

   Router testing benchmarks have consistently been made in a
   monolithic fashion in which a single protocol or behavior is
   measured in an isolated environment.  It is important to know the
   limits for a router/switch's (hereby referred to as Router) behavior
   for each protocol, however this does not produce a reliable benchmark
   of the router's behavior in a deployed network.  Routers in an
   operational network are simultaneously configured with multiple
   protocols and security policies while forwarding traffic and being
   managed.

   To accurately benchmark a router for deployment it is necessary to
   test that router in operational conditions by simultaneously
   configuring the network protocols and security  policies, sourcing
   traffic, and managing the router.  The benchmarks are externally

Poretsky, Rao, Piatt                                                            [Page 2]

INTERNET-DRAFT Framework for Accelerated Stress Benchmarking  October 2003

   observable as control plane or data plane errors at the DUT.  It is
   helpful to accelerate these network operational conditions so that
   the DUT can be benchmarked with faster test duration.   Accelerated
   Stress Testing of routers provides the following benefits:

        1. Evaluation of multiple protocols enabled simultaneously as
        configured in deployed networks
        2. Evaluation of System and Software Stability
        3. Evaluation of Manageability under stressful conditions
        4. Identification of Software Coding bugs such as:
                a. Memory Leaks
                b. Suboptimal CPU Utilization
                c. Coding Logic

   These benefits produce three advantages for netowrk operations:
        1.  Increased stability of routers and protocols
        2.  Hardened routers to DoS attacks
        3.  Verified manageability under stress

   4. Application of Configuration Sets

   Configuration Sets are defined in [1] for the Control Plane, Data
   Plane, Management Plane, and Security Plane.  It is intended that
   the user of these documents specify the specific parameters of the
   Configuration Set based upon applicability to the device and
   network.  Example Configuration Sets are provided below.

   4.1 Control Plane Configuration Sets

   Key protocols for the Control Plane are Routing Protocols, MPLS
   Signaling Protocols, and Multicast Protocols.  Examples for these
   are as follow:

   Example Routing Protocol Configuration Set-

           PARAMETER                            UNITS
           BGP                                  Enabled/Disabled
           Number of EBGP Peers                 Peers
           Number of IBGP Peers                 Peers
           Number of BGP Route Instances        Routes
           Number of BGP Installed Routes       Routes

           MBGP                                 Enabled/Disabled
           Number of MBGP Route Instances       Routes
           Number of MBGP Installed Routes      Routes

           ISIS                                 Enabled/Disabled
           ISIS-TE                                      Enabled/Disabled
           Number of ISIS Adjacencies           Adjacencies
           Number of ISIS Routes                Routes
           Number of Nodes per Area             Nodes


Poretsky, Rao, Piatt                                                            [Page 3]

INTERNET-DRAFT Framework for Accelerated Stress Benchmarking  October 2003

           OSPF                                 Enabled/Disabled
           OSPF-TE                                      Enabled/Disabled
           Number of OSPF Adjacencies           Adjacencies
           Number of OSPF Routes                Routes
           Number of Nodes per Area             Nodes

   Example MPLS Protocol Configuration Set-

           PARAMETER                            UNITS
           MPLS-TE
           Number of Ingress Tunnels            Tunnels
           Number of Mid-Point Tunnels  Tunnels
           Number of Egress Tunnels             Tunnels

           LDP
           Number of Sessions                   Sessions
           Number of FECs                               FECs




 Example Multicast Protocol Configuration Set-

           PARAMETER                            UNITS
           PIM-SM                                       Enabled/Disabled
           RP                                           Enabled/Disabled
           Number of Multicast Groups           Groups
           MSDP                                 Enabled/Disabled

   4.2 Data Plane Configuration Set

   The Data Plane Configuration Set includes the Traffic Profile
   as defined in [1].  The example configuration set is as follows:

   Example Data Plane Configuration Set-

        PARAMETER                               UNITS
        Traffic Forwarding              Enabled/Disabled
        Aggregate Offered Load          bps (or pps)
        Number of Ingress Interfaces    number
        Number of Ingress Interfaces    number

        TRAFFIC PROFILE
        Packet Size(s)                  bytes
        Packet Rate(interface)          array of packets per second
        Number of Flows                 number
        Encapsulation(flow)             array of encapsulation type






Poretsky, Rao, Piatt                                                            [Page 4]

INTERNET-DRAFT Framework for Accelerated Stress Benchmarking  October 2003

   4.3 Management Configuration Set

   The Management Configuration Set can include SNMP, Logging, Debug,
   Telnet, FTP, SSH, and RADIUS parameters.  An example is as follows:

   Example Management Configuration Set-

        PARAMETER                                       UNITS
        SNMP GET Rate                           SNMP Gets/minute
        Logging                                 Enabled/Disabled
        Protocol Debug                          Enabled/Disabled
        Telnet Rate                                     Sessions/Hour
        FTP Rate                                        Sessions/Hour
        Concurrent Telnet Sessions              Sessions
        Concurrent FTP Session                  Sessions
        Packet Statistics Collector             Enabled/Disabled
        Statistics Sampling Rate                X:1 packets

   4.4 Security Configuration Set

   The Security Configuration Set can include Packet Filters and
   Access session restrictions.  An example is as follows:

   Example Security Configuration Set -

        PARAMETER                                               UNITS
        Packet Filters                                  Enabled/Disabled
        Number of Filters For-Me                        number
        Number of Filter Rules For-Me                   number
        Number of Traffic Filters                       number
        Number of Traffic Filter Rules          number
        SSH                                                     Enabled/Disabled
        Number of simultaneous SSH sessions             number
        RADIUS                                          Enabled/Disabled
        TACACS                                          Enabled/Disabled

   5. Application of Startup Conditions

        Startup conditions are the conditions that must be met in order
        for Accelerated Stress benchmarking to begin. Startup Conditions
        specify how a particular Configuration Set should be obtained.
        Example Startup Conditions include:

        PARAMETER                                               UNITS
        Routing Session Establishment Rate              sessions per minute
        User Config Session Establishment Rate  number per minute
        Security Session Establishment Rate             number per minute
        Routes Learned Rate                             routes per minute
        MPLS LSPs Establishment Rate                    number per minute




Poretsky, Rao, Piatt                                                            [Page 5]

INTERNET-DRAFT Framework for Accelerated Stress Benchmarking  October 2003

   6. Application of Instability Conditions

        Test conditions that occur during the Accelerated Stress Test
        should simulate instability in an operational network.
        Repeating these conditions should stress the SUT.  Example
        Instability Conditions are provided below:

        PARAMETER                                               UNITS
        Interface Shutdown Cycling Rate         interfaces per minute
        BGP Session Loss Rate                           sessions per minute
        BGP Route Flap Rate                             routes per minutes
        IGP Route Flap Rate                             routes per minutes
        Route Convergence from Better Next-Hop          routes per minutes
        LSP Reroute Rate                                        LSP per minute
        Overloaded Links                                        number
        Amount Links Overloaded                         % of bandwidth
        FTP Rate                                                Mb/minute
        IPsec Session Loss                              sessions per minute
        Filter Policy Changes                           policies per minute
        SSH Session     Re-Start                                SSH sessions per minute

   7.  Accelerated Stress Benchmarking Application
        The Accelerated Stress Benchmarking test can be applied in
        service provider test environments to benchmark DUTs under
        stress in an environment that is reflective of an operational
        network.  A particular Configuration Set is defined and the
        DUT is benchmarked using this and the Instability Conditions.
        Varying ConfigurationSets and/or Instability Conditions for
        repeated iterations can provide a characterization of the DUT
        to help determine future network deployments.

   8. Security Considerations
        Documents of this type do not directly effect the security of
        the Internet or of corporate networks as long as benchmarking
        is not performed on devices or systems connected to operating
        networks.

   9. References

   [1]  Poretsky, Scott, Rao, Shankar, and Piatt, Ray, "Terminology for
        Accelerated Stress Benchmarking, draft-ietf-bmwg-acc-bench-term-
        01, work in progress, October 2003.

   10. Author's Address

        Scott Poretsky
        Quarry Technologies
        8 New England Executive Park
        Burlington, MA 01803
        USA
        Phone: + 1 781 395 5090
        EMail: sporetsky@quarrytech.com

Poretsky, Rao, Piatt                                                            [Page 6]

INTERNET-DRAFT Framework for Accelerated Stress Benchmarking  October 2003

        Shankar Rao
        950 17th Street
        Suite 1900
        Qwest Communications
        Denver, CO 80210
        USA
        Phone: + 1 303 437 6643
        Email: srao@qwest.net

        Ray Piatt
        Cable and Wireless
        11700 Plaza America Drive
        Reston, VA 20190
        USA
        Phone: + 1 703 292 2113
        Email: rpiatt@cw.net

   11.  Full Copyright Statement

        Copyright (C) The Internet Society (1998).  All Rights
        Reserved.

        This document and translations of it may be copied and
        furnished to others, and derivative works that comment on or
        otherwise explain it or assist in its implementation may be
        prepared, copied, published and distributed, in whole or in
        part, without restriction of any kind, provided that the above
        copyright notice and this paragraph are included on all such
        copies and derivative works.  However, this document itself may
        not be modified in any way, such as by removing the copyright
        notice or references to the Internet Society or other Internet
        organizations, except as needed for the purpose of developing
        Internet standards in which case the procedures for copyrights
        defined in the Internet Standards process must be followed, or
        as required to translate it into languages other than English.

        The limited permissions granted above are perpetual and will
        not be revoked by the Internet Society or its successors or
        assigns.  This document and the information contained herein is
        provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE
        INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES,
        EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY
        THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY
        RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
        FOR A PARTICULAR PURPOSE.








Poretsky, Rao, Piatt                                                    [Page 7]


Html markup produced by rfcmarkup 1.107, available from http://tools.ietf.org/tools/rfcmarkup/