[Docs] [txt|pdf] [Tracker] [Email] [Diff1] [Diff2] [Nits]

Versions: 00 01 RFC 2438

Network Working Group                                        Mike O'Dell
Internet-Draft                                        UUNET Technologies
                                                    Harald T. Alvestrand
                                                                 Maxware
                                                             Bert Wijnen
                                               IBM T. J. Watson Research
                                                           Scott Bradner
                                                      Harvard University
                                                             August 1998

     Advancement of MIB specifications on the IETF Standards Track

                   <draft-iesg-odell-mibtest-01.txt>

1. Status of this Memo

   This document is an Internet-Draft.  Internet-Drafts are working
   documents of the Internet Engineering Task Force (IETF), its areas,
   and its working groups.  Note that other groups may also distribute
   working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet- Drafts as reference
   material or to cite them other than as "work in progress."

   To view the entire list of current Internet-Drafts, please check the
   "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow
   Directories on ftp.is.co.za (Africa), ftp.nordu.net (Northern
   Europe), ftp.nis.garr.it (Southern Europe), munnari.oz.au (Pacific
   Rim), ftp.ietf.org (US East Coast), or ftp.isi.edu (US West Coast).

   A revised version of this draft document will be submitted to the RFC
   editor as a BCP (Best Current Practice) documenting an IESG procedure
   for the Internet Community.  Discussion and suggestions for
   improvement are requested.  This document will expire before January,
   1999. Distribution of this draft is unlimited.


2. Abstract

   The Internet Standards Process [1] requires that all IETF Standards
   Track specifications must have "multiple, independent, and
   interoperable implementations" before they can be advanced beyond
   Proposed Standard status. This document specifies the process which
   the IESG will use to determine if a MIB document meets these
   requirements.  It also discusses the rationale for this process.




O'Dell/Alvestrand/Wijnen/Bradner                                [Page 1]


Internet-Draft               MIB Advancement                 August 1998


3. The Nature of the Problem

   The Internet Standards Process [1] requires that for a IETF
   specification to advance beyond the Proposed Standard level, at least
   two genetically unrelated implementations must be shown to
   interoperate correctly with all features and options. There are two
   distinct reasons for this requirement.

   The first reason is to verify that the text of the specification is
   adequately clear and accurate.  This is demonstrated by showing that
   multiple implementation efforts have used the specification to
   achieved interoperable implementations.

   The second reason is to discourage excessive options and "feature
   creep". This is accomplished by requiring interoperable
   implementation of all features, including options.  If an option is
   not included in at least two different interoperable implementations,
   it is safe to assume that it has not been deemed useful and must be
   removed before the specification can advance.

   In the case of a protocol specification which specifies the "bits on
   the wire" exchanged by executing state machines, the notion of
   "interoperability" is reasonably intuitive - the implementations must
   successfully "talk to each other", exchanging "bits on the wire",
   while exercising all features and options.

   In the case of an SNMP Management Information Base (MIB)
   specification, exactly what constitutes "interoperation" is less
   obvious.  This document specifies how the IESG has decided to judge
   "MIB interoperability" in the context of the IETF Standards Process.

   There are a number of plausible interpretations of MIB
   interoperability, many of which have merit but which have very
   different costs and difficulties in realization.

   The aim is to ensure that the dual goals of specification clarity and
   feature evaluation have been met using an interpretation of the
   concept of MIB interoperability that strikes a balance between
   testing complexity and practicality.

4. On The Nature of MIBs

   Compared to "state machine" protocols which focus on procedural
   specifications, a MIB is much more data oriented. To over-
   generalize, in a typical MIB the collection of data type and instance
   specifications outnumbers inter-object procedural or causal semantics
   by a significant amount.




O'Dell/Alvestrand/Wijnen/Bradner                                [Page 2]


Internet-Draft               MIB Advancement                 August 1998


   A central issue is that a MIB does not stand alone; it forms the
   access interface to the instrumentation underneath it. Without the
   instrumentation, a MIB has form but no values. Coupled with the large
   number of objects even in a simple MIB, a MIB tends to have more of
   the look and feel of an API or a dictionary than a state machine
   protocol.

   It is important to distinguish between assessing the interoperability
   of applications which may use or interact with MIBs, and the MIBs
   themselves.  It is fairly obvious that "black-box testing" can be
   applied to such applications and that the approach enjoys a certain
   maturity in the software engineering arts.  A MIB, on the other hand
   is not readily amenable to black box test plans.

5. Discussion and Recommended Process

   In order to meet their obligations under the IETF Standards Process
   the Operations and Management Area Directors and the IESG must be
   convinced that each MIB specification advanced to Draft Standard or
   Internet Standard status is clearly written, that there are the
   required multiple interoperable implementations, and that all options
   have been implemented.  There are multiple ways to achieve this goal.
   Appendix A lists some testing approaches that could be used when
   attempting to document multiple implementations.

   The Full Coverage or Stimulus-Response approaches are very through,
   and would increase confidence that the requirement has been met, if
   applied.  However, experience in real-world software engineering
   makes it clear that such confidence comes at an extremely high price;
   even with the most exhaustive testing, it is often not clear what
   precisely has been demonstrated by such testing.  We believe that
   both of those standards of evidence are materially beyond what can be
   reasonably accomplished in an operational sense, and achieving the
   requisite semantic specifications are even more unlikely.

   Therefore, the Operations and Management Area and the IESG have
   adopted a more pragmatic approach to determining the suitability of a
   MIB specification for advancement on the standards track beyond
   Proposed Standard status.  Each MIB specification suggested for
   advancement must have one or more advocates who can make a convincing
   argument that the MIB specification meets the multiple implementation
   and feature support requirements of the IETF Standards Process.  The
   specific way to make the argument is left to the advocate, but will
   normally include reports that basic object comparison testing has
   been done.

   Thus any recommendation for the advancement of a MIB specification
   must be accompanied by an implementation report, as is the case with



O'Dell/Alvestrand/Wijnen/Bradner                                [Page 3]


Internet-Draft               MIB Advancement                 August 1998


   all requests for the advancement of IETF specifications.  The
   implementation report must include the reasons why the IESG should
   believe that there are multiple implementations of the MIB in
   question and that the all of the MIB objects in the specification to
   be advanced are supported in more than one implementation.  But note
   that the prime concern of the IESG will be that the underlying
   reasons for the interoperable implementations are met, i.e. that the
   text of the specification is clear and unambiguous, and that features
   of the specification which have not garnered support have been
   removed.

   The implementation report will be placed on the IETF web page along
   with the other pre-advancement implementation reports and will be
   specifically referred to in the IETF Last-Call.  As with all such
   implementation reports, the determination of adequacy is made by the
   Area Director(s) of the relevant IETF Area. This determination of
   adequacy can be challenged during the Last-Call period.

6. Security Considerations

   Some may view this policy as possibly leading to a reduction in the
   level of confidence people can have in MIBs but the O&M Area
   Directors and the IESG feel that it will adequately ensure a
   reasonable evaluation of the level of clarity of MIB specifications
   and to ensure that unused options can be identified and removed
   before the advancement of a specification.

   Good, clearly written MIBs can be of great assistance in the
   management of the Internet and other networks and thus assist in the
   reduction of some types of security threats.


8. References

   [RFC2026] "The Internet Standards Process -- Revision 3", Bradner,
   October 1996

9. Author's Addresses

   Michael D. O'Dell
   UUNET Technologies, Inc.
   3060 Williams Drive
   Fairfax, VA 22031

   Email: mo@uu.net
   Phone: +1-703-206-5890





O'Dell/Alvestrand/Wijnen/Bradner                                [Page 4]


Internet-Draft               MIB Advancement                 August 1998


   Harald T. Alvestrand
   Maxware
   Pirsenteret
   N-7005 Trondheim, Norway

   EMail: Harald.Alvestrand@maxware.no
   Phone: +47-73-54-57-94


   Bert Wijnen
   IBM T. J. Watson Research
   Schagen 33
   3461 GL Linschoten
   Netherlands

   EMail: wijnen@vnet.ibm.com
   phone: +31-348-432-794


   Scott Bradner
   Harvard University
   1350 Mass. Ave.
   Cambridge MA 02138

   Email: sob@harvard.edu
   Phone: +1-617-495-3864

Appendix A

A. Some Testing Alternatives

   The IESG debated a number of interoperability and testing
   models in formulating this specification.  The following
   list is not an exhaustive enumeration of the alternatives,
   but it does capture the major plausible models which were
   examined in the course of the discussion.

A.1 Basic Object Comparison

   Assume the requisite two genetically unrelated implementations of
   the MIB in an SNMP agent and an SNMP management station which can
   do a "MIB Dump" (extract the complete set of MIB object types and
   values from the agent implementation).  Extract a MIB Dump from
   each implementation and compare the two dumps to verify that both
   provide the complete set of mandatory and optional objects and
   that the individual objects are of the correct types.

A.2 Stimulus/Response Testing



O'Dell/Alvestrand/Wijnen/Bradner                                [Page 5]


Internet-Draft               MIB Advancement                 August 1998


   Proceed as in A.1, but in addition, comprehensively exercise the
   two (network) elements containing the agent implementations to
   verify that all the MIB objects reflect plausible values in
   operational conditions.  An even stricter interpretation would
   require that the MIB objects in the two network elements track
   identically given the identical stimulus. While this would
   test "read-only" or "monitoring" information obtained from the
   underlying instrumentation, it is important to observe that
   such instrumentation is actually an *application* which uses
   the MIB and is not part of the MIB itself.

A.3 Full Coverage Testing

   This model extends the notion of Stimulus/Response Testing to its
   logical extreme. The MIB is viewed as an API and the
   software engineering notion of full coverage testing is
   applied to a MIB.  This involves exercising all paths through the
   causal semantics and verifying that all objects change state
   correctly in all cases.  Again, note that much more than the
   MIB definition is being exercised and evaluated.































O'Dell/Alvestrand/Wijnen/Bradner                                [Page 6]


Html markup produced by rfcmarkup 1.129d, available from https://tools.ietf.org/tools/rfcmarkup/