Network Working Group                                             P. Lei
Internet-Draft                                       Cisco Systems, Inc.
Intended status: Informational                                    L. Ong
Expires: October 30, 2007 January 7, 2008                               Ciena Corporation
                                                               M. Tuexen
                                      Muenster Univ. of Applied Sciences
                                                          April 28,
                                                            T. Dreibholz
                                            University of Duisburg-Essen
                                                            July 6, 2007

            An Overview of Reliable Server Pooling Protocols

Status of this Memo

   By submitting this Internet-Draft, each author represents that any
   applicable patent or other IPR claims of which he or she is aware
   have been or will be disclosed, and any of which he or she becomes
   aware will be disclosed, in accordance with Section 6 of BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at

   The list of Internet-Draft Shadow Directories can be accessed at

   This Internet-Draft will expire on October 30, 2007. January 7, 2008.

Copyright Notice

   Copyright (C) The IETF Trust (2007).


   The Reliable Server Pooling effort (abbreviated "RSerPool"), provides
   an application-independent set of services and protocols for building
   fault tolerant and highly available client/server applications.  This
   document provides an overview of the protocols and mechanisms in the
   reliable server pooling suite.

Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
   2.  Aggregate Server Access Protocol (ASAP) Overview . . . . . . .  5
     2.1.  Pool Initialization  . . . . . . . . . . . . . . . . . . .  5  6
     2.2.  Pool Entity Registration . . . . . . . . . . . . . . . . .  5  6
     2.3.  Pool Entity Selection  . . . . . . . . . . . . . . . . . .  6
     2.4.  Endpoint Keepalive . Keep-Alive  . . . . . . . . . . . . . . . . . . .  6
     2.5.  Failover Services  . . . . . . . . . . . . . . . . . . . .  6  7
       2.5.1.  Cookie Mechanism . . . . . . . . . . . . . . . . . . .  6  7
       2.5.2.  Business Card Mechanism  . . . . . . . . . . . . . . .  7
   3.  Endpoint Nameserver Handlespace Redundancy Protocol (ENRP) Overview . . .  7
     3.1.  Initialization . . . . . . . . . . . . . . . . . . . . . .  7
     3.2.  Server Discovery and Home Server Selection . . . . . . . .  7  8
     3.3.  Server Pool Maintenance  Failure Detection, Handlespace Audit and
           Synchronization  . . . . . . . . . . . . . . . . .  8
   4.  Example Scenarios  . . . . .  8
     3.4.  Server Takeover  . . . . . . . . . . . . . . . . .  8
     4.1.  Example Scenario Using RSerPool Resolution Service . . . .  8
       4.1.1.  Standalone Mode  . . . . . . . . . . . . . .
   4.  Example Scenarios  . . . . .  8
       4.1.2.  Pool Mode  . . . . . . . . . . . . . . . . . .  9
     4.1.  Example Scenario using RSerPool Resolution Service . . . .  8  9
     4.2.  Example Scenario Using using RSerPool Session Services . . . . .  9 10
   5.  Security Considerations  . . . . . . . . . . . . . . . . . . . 11
   6.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 11
   7.  Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 11
   8.  Normative References . . . . . . . . . . . . . . . . . . . . . 11 12
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 12
   Intellectual Property and Copyright Statements . . . . . . . . . . 13 14

1.  Introduction

   The Reliable Server Pooling (RSerPool) protocol suite is designed to
   provide client applications ("pool users") with the ability to select
   a server (a "pool element") from among a group of servers providing
   equivalent service (a "pool").

   The RSerPool architecture supports high-availability and load
   balancing by enabling a pool user to identify the most appropriate
   server from the server pool at a given time.  The architecture is
   defined to support a set of basic goals:

   o  application-independent protocol mechanisms

   o  separation of server naming from IP addressing

   o  use of the end-to-end principle to avoid dependancies on
      intermediate equipment

   o  separation of session availability/failover functionality from
      application itself

   o  facilitate different server selection policies

   o  facilitate a set of application-independent failover capabilities

   o  peer-to-peer structure

   The basic components of the Rserpool RSerPool architecture are shown in
   Figure 1 below: 1below:

                       ------        .      +-------+      .
                      / ENRP \       .      |       |      .
               /---->| Server |      .      |  PE 1 |      .
               | /--- \______/       .      |       |      .
               | |                   .      +-------+      .
               | |                   .                     .
               | |                   .     server pool     Server Pool     .
               | V                   .                     .
           +-------+                 .      +-------+      .
           |       |                 .      |       |      .
           |  PU 1 |-----------------.------|  PE 2 |      .
           |       |                 .      |       |      .
           +-------+                 .      +-------+      .
                                     .                     .
                                     .      +-------+      .
                                     .      |       |      .
                                     .      |  PE 3 |      .
                                     .      |       |      .
                                     .      +-------+      .

                                 Figure 1

   A server pool is defined as a set of one or more servers providing
   the same application functionality.  The servers are called Pool
   Elements (PEs).  Multiple PEs in a server pool can be used to provide
   fault tolerance or load sharing, for example.  The PEs register into
   and deregisters out of the pool using the Aggregate Server Access
   Protocol ASAP [2].

   Each server pool is identified by a unique byte string called the
   pool handle. handle (PH).  The pool handle allows a mapping from the pool to
   a specific Pool Element located by its IP address and port.  The pool
   handle is what is specified by the Pool User (PU) when it attempts to
   access a server in the pool, again using ASAP.  Both IPv4 and IPv6 PE
   addresses are supported.

   To resolve the pool handle to the address necessary to access a Pool
   Element, the PU consults an entity called the Endpoint haNdlespace
   Redundancy Protocol (ENRP) server.  This server may be a standalone
   server supporting many PUs or a part of the PU itself, however it is
   envisioned that ENRP servers provide a fully distributed and fault-
   tolerant registry service using ENRP [3] to maintain synchronization
   of data concerning the pool handle mapping space.


   RSerPool provides a number of tools to aid client migration between
   servers on server failure: it allows the client to identify
   alternative servers, either on initial discovery or in real time; it
   also allows the original server to provide a state cookie to the
   client that can be forwarded to an alternative server to provide
   application-specific state information.

   The requirements for the Reliable Server Pooling effort framework are
   defined in RFC3237 [1].  It is worth noting that the requirements on
   RSerPool in the area of load balancing partially overlap with GRID
   computing/high performance computing.  However, the scope of both
   areas is completely different: GRID and high performance computing
   also cover topics like managing different administrative domains,
   data locking and synchronization, inter-session communication and
   resource accounting for powerful computation services, but the
   intention of RSerPool is simply a lightweight realization of load
   distribution and session management.  In particular, these
   functionalities are intended to be used on systems with small memory
   and CPU resources only.  Any further functionality is not in the
   scope of RSerPool and can -- if necessary -- provided by the
   application itself.

   This document provides an overview of the RSerPool protocol suite,
   specifically the Aggregate Server Access Protocol ASAP [2] and the
   Endpoint Nameserver Handlespace Redundancy Protocol ENRP [3].  In addition to
   the protocol specifications, there is a common parameter format
   specification COMMON [4] for both protocols, a definition of server
   selection rules (pool policies) POLICIES [5], as well as a security
   threat analysis SEC [5]. THREATS [6].

2.  Aggregate Server Access Protocol (ASAP) Overview

   ASAP is a straight-forward implementation of defines a straigt-forward set of mechanisms
   identified as necessary for to support of
   the creation and maintenance of pools of redundant servers.  These
   mechanisms include:

   o  registration of a new server for the into a server pool

   o  deregistration of an existing server from the a pool

   o  resolution of a pool 'handle' handle to a server or list of servers

   o  liveness detection for servers in the a pool

   o  failover mechanisms for handling a server failure

2.1.  Pool Initialization

   Pools come into existence when a PE registers the first instance of
   the pool name with an ENRP server.  They disappear when the last PE
   deregisters.  In other words, the starting of the first PE on some
   machine causes the creation of the pool when the registration reaches
   the ENRP server.

2.2.  Pool Entity Registration

   A new server joins an existing pool by sending a Registration message
   via ASAP to an ENRP server, indicating the 'handle' pool handle of the pool
   that it wishes to join, a pool PE identifier for itself (chosen randomly) randomly),
   information about it's its lifetime in the pool, and what transport
   protocols and selection policies policy it supports.  The ENRP server that it
   first contacts is called its Home ENRP server, and maintains a list
   of subscriptions by the PE as well as performing performs periodic audits to
   confirm that the PE is still responsive.

   Similar procedures are applied to de-register itself from the server
   pool (or alternatively the server may simply let the lifetime that it
   previously registered with expire, after which it is gracefully
   removed from the pool.

2.3.  Pool Entity Selection

   When an endpoint wishes to be connected to a server in the pool, it
   genereates a
   generates an ASAP Handle Resolution message in ASAP and sends this to its
   home ENRP server.  The ENRP server resolves the handle based on its
   knowledge of pool servers and returns a Handle Resolution Response in
   via ASAP.  The Resolution Response response contains a list of the IP addresses of one or
   more servers in the pool that can be contacted.  The process by which
   the list of servers is created may involve a number of policies for
   server selection.  The RSerPool protocol suite supports defines a few simply defined basic
   policies and allows the use of external server selection input for
   more complex policies.

2.4.  Endpoint Keepalive Keep-Alive

   ENRP servers monitor the status of pool elements using the ASAP Keep
   Endpoint Keep-Alive message.  A Pool Element PE responds to the ASAP Keep Alive Keep-Alive
   message with an Endpoint Keep-Alive Ack response.

   In addition, an endpoint a PU can notify its home ENRP server that the PE
   the endpoint was using it used
   has become unresponsive by sending the ENRP
   server an ASAP Endpoint Unreachable message.
   message to the ENRP server.

2.5.  Failover Services

   While maintaining application-independence, the RSerPool protocol
   suite provides some simple hooks for supporting failover of an
   individual session with a pool element.  Generally, mechanisms for
   failover that rely on application state or transaction status cannot
   be defined without more specific knowledge of the application being
   supported.  However, some simple mechanisms supported by RSerPool
   allow some level of failover that any application can use.

2.5.1.  Cookie Mechanism

   Cookies may optionally be generated by the ASAP layer and
   periodically sent from the PE to the PU.  The PU only stores the last
   received cookie.  In case of fail over failover the PU sends this last received
   cookie to the new PE.  This method provides a simple way of state
   sharing between the PEs.  Please note that the old PE should sign the
   cookie and the receiving PE should verify the that signature.  For the
   PU, the cookie has no structure and is only stored and transmitted to
   the new PE.

2.5.2.  Business Card Mechanism

   A PE can send a business card to its peer (PE or PU) PU), containing its
   pool handle and guidance concerning which other PEs the peer should
   use for failover.  This gives a PE a means of telling a PU what it
   identifies as the "next best" PE to use in case of failure, which may
   be based on pool considerations, such as load balancing, or user
   considerations, such as PEs that have the most up-to-date state

3.  Endpoint Nameserver Handlespace Redundancy Protocol (ENRP) Overview

   A set of server pool can be supported pools, which is denoted as a handlespace, is managed
   by one or more ENRP servers.  If
   multiple ENRP servers  Pools are used to support a single pool then not valid in the ENRP
   protocol is used between whole Internet but only
   in smaller domains, called the operational scope.  The ENRP servers
   use the ENRP protocol in order to maintain a distributed, fault-tolerant fault-
   tolerant, real-time registry service.  ENRP servers communicate with
   each other in order to exchange for information exchange, such as pool membership changes,
   handlespace data synchronization, etc. etc..

3.1.  Initialization

   Each ENRP server initially generates a 32-bit server ID that it uses
   in subsequent messaging and remains unchanged over the lifetime of
   the server.  It then attempts to learn all of the other ENRP servers
   within the scope of the server pool, either by using a pre-defined
   Mentor server or by sending out Presence messages on a well-known
   multicast channel in order to determine other ENRP servers from the
   responses and select one as Mentor.  A Mentor can be any peer ENRP server that
   it selects to provide current data about the pool.

   It then requests the
   server.  The most current data about the pool handlespace data is requested by Handle
   Table Requests from its Mentor server and unpacks the Mentor.  The received answer in form of
   Handle Table Response messages is unpacked into its the local database.

   After that, the ENRP server is then ready to provide ENRP services.

3.2.  Server Discovery and Home Server Selection

   PEs can now register their presence with the newly functioning ENRP
   server by using ASAP messages.  They discover the new ENRP server
   after the server sends out an ASAP Server Announce message on the
   well-known ASAP multicast channel.  PEs need only have to register with
   one ENRP server, as other ENRP servers supporting the pool will
   synchronize their knowledge about pool elements using the ENRP

   The PE may have a configured list of ENRP servers to talk to, in the
   form of a list of IP addresses, in which case it will start to setup
   associations with some number of them and assign the first one that
   responds to it as its Home ENRP Server.

   Alternatively it can listen on the multicast channel for a set period
   and when it hears an ENRP server, start an association.  The first
   server it gets up can then become its Home ENRP Server.

3.3.  Server Pool Maintenance  Failure Detection, Handlespace Audit and Synchronization

   ENRP servers send ENRP Presence messages to all of their peers in
   order to show their liveness.  These Presence messages also include a
   checksum computed over all PE failure detection, keepalive, etc.  TBD identities for which the ENRP server is
   in the role of a Home ENRP server.  Each ENRP server maintains an up-
   to-date list of its peers and can also compute the checksum expected
   from a certain peer, according to its local handlespace database.  By
   comparing the expected sum and the sum reported by a peer (denoted as
   handlespace audit), an inconsistency can be detected.  In such a
   case, the handlespace -- restricted to the PEs owned by that peer --
   can be requested for synchronization, analogously to Section 3.2.

3.4.  Server Takeover

   If the unresponsiveness of an ENRP server is detected, the remaining
   ENRP servers negotiate which other server takes over the Home ENRP
   role for the PEs of the failed peer.  After reaching a consensus on
   the takeover, the ENRP server taking over these PEs sends a
   notification to its peers (via ENRP) as well as to the PEs taken over
   (via ASAP).

4.  Example Scenarios

4.1.  Example Scenario Using using RSerPool Resolution Service

4.1.1.  Standalone Mode

   RSerPool can be used in a 'standalone' manner, where the application
   uses RSerPool to determine the address of a primary server in the
   pool, and then interacts directly with that server without further
   use of RSerPool services.  If the initial server fails, the
   application uses RSerPool again to find the next server in the pool.

4.1.2.  Pool Mode

   For pool user ("client") applications, if an ASAP implementation is
   available on the client system, there are typically only three
   modifications required to the application source code:

   1.  Instead of specifying the hostnames of primary, secondary,
       tertiary servers, etc., the application user specifies a pool

   2.  Instead of using a DNS based service (e.g. the Unix library
       function gethostbyname()) getaddrinfo()) to translate from a hostname to an IP
       address, the application will invoke an RSerPool service
       primitive GETPRIMARYSERVER that takes as input a pool handle, and
       returns the IP address of the primary server.  The application
       then uses that IP address just as it would have used the IP
       address returned by the DNS in the previous scenario.

   3.  Without the use of additional RSerPool services, failure
       detection and failover procedures must be designed into each
       application.  However, when failure is detected on the primary
       server, instead of invoking DNS translation again on the hostname
       of a secondary server, the application invokes the service
       primitive GETNEXTSERVER, which performs two functions in a single

       1.  First it indicates to the RSerPool layer the failure of the
           server returned by a previous GETPRIMARYSERVER or
           GETNEXTSERVER call.

       2.  Second, it provides the IP address of the next server that
           should be contacted, according to the best information
           available to the RSerPool layer at the present time (e.g. set
           of available pool elements, pool element policy in effect for
           the pool, etc.).

   For pool element ("server") applications where an ASAP implementation
   is available, two changes are required to the application source

   1.  The server should invoke the REGISTER service primitive upon
       startup to add itself into the server pool using an appropriate
       pool handle.  This also includes the address(es) protocol or
       mapping id, port (if required by the mapping), and pooling

   2.  The server should invoke the DEREGISTER service primitive to
       remove itself from the server pool when shutting down.

   When using these RSerPool services, RSerPool provides benefits that
   are limited (as compared to utilizing all services), but nevertheless
   quite useful as compared to not using RSerPool at all.  First, the
   client user need only supply a single string, i.e. the pool handle,
   rather than a list of servers.  Second, the decision as to which
   server is to be used can be determined dynamically by the server
   selection mechanism (i.e. a "pool policy" performed by ASAP; see ASAP
   [2]).  Finally, when failures occur, these are reported to the pool
   via signaling present in ASAP [2] and ENRP [3], other clients will
   eventually know (once this failure is confirmed by other elements of
   the RSerPool architecture) that this server has failed.

4.2.  Example Scenario Using using RSerPool Session Services

   When the full suite of RSerPool services are is used, all communication
   between the pool user and the pool element is mediated by the
   RSerPool framework, including session establishment and teardown, and
   the sending and receiving of data.  Accordingly, it is necessary to
   modify the application to use the service primitives (i.e. the API)
   provided by RSerPool, rather than the transport layer primitives
   provided by TCP, SCTP, or whatever transport protocol is being used.

   As in the previous case, sessions (rather than connections or
   associations) are established, and the destination endpoint is
   specified as a pool handle rather than as a list of IP addresses with
   a port number.  However, failover from one pool element to another is
   fully automatic, and can be transparent to the application (so long
   as the application has saved enough state in a state cookie):

      The RSerPool framework control channel provides maintenance
      functions to keep pool element lists, policies, etc. current.

      Since the application data (e.g. data channel) is managed by the
      RSerPool framework, unsent data (data not yet submitted by
      RSerPool to the underlying transport protocol) is automatically
      redirected to the newly selected pool element upon failover.  If
      the underlying transport layer supports retrieval of unsent data
      (as in SCTP), retrieved unsent data can also be automatically re-
      sent to the newly selected pool element.

      An application server (pool element) can provide a state cookie
      (described in Section 2.5.1) that is automatically passed on to
      another pool element (by the ASAP layer at the pool user) in the
      event of a failover.  This state cookie can be used to assist the
      application at the new pool element in recreating whatever state
      is needed to continue a session or transaction that was
      interrupted by a failure in the communication between a pool user
      and the original pool element.

      The application client (pool user) can provide a callback function
      (described in Section 2.5.2)
      that is invoked on the pool user side in the case of a failover.
      This callback function can execute any application specific
      failover code, such as generating a special message (or sequence
      of messages) that helps the new pool element construct any state
      needed to continue an in-process session.

      Suppose in a particular peer-to-peer application, PU A is
      communicating with PE B, and it so happens that PU A is also a PE
      in pool X. PU A can pass a "business card" to PE B identifying it
      as a member of pool X. In the event of a failure at A, or a
      failure in the communication link between A and B, PE B can use
      the information in the business card to contact an equivalent PE
      to PU A from pool X.

      Additionally, if the application at PU A is aware of some
      particular PEs of pool X that would be preferred for B to contact
      in the event that A becomes unreachable from B, PU A can provide
      that list to the ASAP layer, and it will be included in A's
      business card.  (See card (see Section 2.5.2).

5.  Security Considerations

   This document does not identify security requirements beyond those
   already documented in the ENRP and ASAP protocol specifications.  A
   security threat analysis of RSerPool is provided in THREATS [6].

6.  IANA Considerations

   This document does not require additional IANA actions beyond those
   already identified in the ENRP and ASAP protocol specifications.

7.  Acknowledgements

   The authors wish to thank Maureen Stillman, Qiaobing Xie, Randall
   Stewart, Scott Bradner, and many others for their invaluable

8.  Normative References

   [1]  Tuexen, M., Xie, Q., Stewart, R., Shore, M., Ong, L., Loughney,
        J., and M. Stillman, "Requirements for Reliable Server Pooling",
        RFC 3237, January 2002.

   [2]  Stewart, R., "Aggregate Server Access Protocol (ASAP)",
        draft-ietf-rserpool-asap-15 (work in progress), January 2007.

   [3]  Stewart, R., "Endpoint Handlespace Redundancy Protocol (ENRP)",
        draft-ietf-rserpool-enrp-15 (work in progress), January 2007.

   [4]  Stewart, R., "Aggregate Server Access Protocol (ASAP) and
        Endpoint Handlespace Redundancy  Protocol (ENRP) Parameters",
        draft-ietf-rserpool-common-param-11 (work in progress),
        October 2006.

   [5]  Tuexen, M. and T. Dreibholz, "Reliable Server Pooling Policies",
        draft-ietf-rserpool-policies-04 (work in progress), March 2007.

   [6]  Stillman, M., "Threats Introduced by Rserpool and Requirements
        for Security in response to  Threats",
        draft-ietf-rserpool-threats-06 (work in progress),
        November 2006.

Authors' Addresses

   Peter Lei
   Cisco Systems, Inc.
   955 Happfield Dr.
   Arlington Heights, IL  60004

   Phone: +1 773 695-8201
   Lyndon Ong
   Ciena Corporation
   PO Box 308
   Cupertino, CA  95015


   Michael Tuexen
   Muenster Univ. of Applied Sciences
   Stegerwaldstr. 39
   48565 Steinfurt


   Thomas Dreibholz
   University of Duisburg-Essen, Institute for Experimental Mathematics
   Ellernstrasse 29
   45326 Essen, Nordrhein-Westfalen

   Phone: +49 201 183-7637
   Fax:   +49 201 183-7673

Full Copyright Statement

   Copyright (C) The IETF Trust (2007).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided on an

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at


   Funding for the RFC Editor function is provided by the IETF
   Administrative Support Activity (IASA).