PCE Working Group                                          X. Zhang, Ed.
Internet-Draft                                       Huawei Technologies
Intended status: Informational                             I. Minei, Ed.
Expires: April 1, May 4, 2017                                        Google, Inc.
                                                      September 28,
                                                        October 31, 2016

       Applicability of a Stateful Path Computation Element (PCE)
                   draft-ietf-pce-stateful-pce-app-07
                   draft-ietf-pce-stateful-pce-app-08

Abstract

   A stateful Path Computation Element (PCE) maintains information about
   Label Switched Path (LSP) characteristics and resource usage within a
   network in order to provide traffic engineering calculations for its
   associated Path Computation Clients (PCCs).  This document describes
   general considerations for a stateful PCE deployment and examines its
   applicability and benefits, as well as its challenges and limitations
   through a number of use cases.  PCE Communication Protocol (PCEP)
   extensions required for stateful PCE usage are covered in separate
   documents.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on April 1, May 4, 2017.

Copyright Notice

   Copyright (c) 2016 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Overview of the Stateful PCE Protocol Extensions  .  Application Scenarios . . . . .   4
   4.  Deployment Considerations . . . . . . . . . . . . . . .   4
     3.1.  Optimization of LSP Placement . . .   5
     4.1.  Multi-PCE Deployments . . . . . . . . . . .   4
       3.1.1.  Throughput Maximization and Bin Packing . . . . . . .   5
     4.2.  LSP State Synchronization . . . .
       3.1.2.  Deadlock  . . . . . . . . . . . .   5
     4.3.  PCE Survivability . . . . . . . . . .   7
       3.1.3.  Minimum Perturbation  . . . . . . . . . .   6
   5.  Application Scenarios . . . . . .   8
       3.1.4.  Predictability  . . . . . . . . . . . . . .   6
     5.1.  Optimization of LSP Placement . . . . .   9
     3.2.  Auto-bandwidth Adjustment . . . . . . . . .   6
       5.1.1.  Throughput Maximization and Bin Packing . . . . . . .   7
       5.1.2.  Deadlock  11
     3.3.  Bandwidth Scheduling  . . . . . . . . . . . . . . . . . .  11
     3.4.  Recovery  . . . .   9
       5.1.3.  Minimum Perturbation . . . . . . . . . . . . . . . .  11
       5.1.4.  Predictability . . . .  12
       3.4.1.  Protection  . . . . . . . . . . . . . . .  12
     5.2.  Auto-bandwidth Adjustment . . . . . .  12
       3.4.2.  Restoration . . . . . . . . . .  13
     5.3.  Bandwidth Scheduling . . . . . . . . . . .  13
       3.4.3.  SRLG Diversity  . . . . . . .  14
     5.4.  Recovery . . . . . . . . . . . .  14
     3.5.  Maintenance of Virtual Network Topology (VNT) . . . . . .  15
     3.6.  LSP Re-optimization . . . . . .  14
       5.4.1.  Protection . . . . . . . . . . . . .  15
     3.7.  Resource Defragmentation  . . . . . . . .  14
       5.4.2.  Restoration . . . . . . . .  16
     3.8.  Point-to-Multi-Point Applications . . . . . . . . . . . .  17
     3.9.  Impairment-Aware Routing and Wavelength Assignment (IA-
           RWA)  .  16
       5.4.3.  SRLG Diversity . . . . . . . . . . . . . . . . . . .  16
     5.5.  Maintenance of Virtual Network Topology (VNT) . . . . . .  17
     5.6.  LSP Re-optimization .
   4.  Deployment Considerations . . . . . . . . . . . . . . . . . .  17
     5.7.  Resource Defragmentation  18
     4.1.  Multi-PCE Deployments . . . . . . . . . . . . . . . .  18
     5.8.  Point-to-Multi-Point Applications . .  18
     4.2.  LSP State Synchronization . . . . . . . . . .  19
     5.9.  Impairment-Aware Routing and Wavelength Assignment (IA-
           RWA) . . . . . .  19
     4.3.  PCE Survivability . . . . . . . . . . . . . . . . . . . .  19
   6.
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .  20
   7.  19
   6.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  21
   8.  20
   7.  Contributing Authors  . . . . . . . . . . . . . . . . . . . .  21
   9.  20
   8.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  22
   10.  21
   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  22
     10.1.  21
     9.1.  Normative References  . . . . . . . . . . . . . . . . . .  22
     10.2.  21
     9.2.  Informative References  . . . . . . . . . . . . . . . . .  23  22
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  24  23

1.  Introduction

   [RFC4655] defines the architecture for a Path Computation Element
   (PCE)-based model for the computation of Multiprotocol Label
   Switching (MPLS) and Generalized MPLS (GMPLS) Traffic Engineering
   Label Switched Paths (TE LSPs).  To perform such a constrained
   computation, a PCE stores the network topology (i.e., TE links and
   nodes) and resource information (i.e., TE attributes) in its TE
   Database (TED).  [RFC5440] describes the Path Computation Element
   Protocol (PCEP) for interaction between a Path Computation Client
   (PCC) and a PCE, or between two PCEs, enabling computation of TE
   LSPs.  Extensions for support of GMPLS in PCEP are defined in
   [I-D.ietf-pce-gmpls-pcep-extensions].

   As per [RFC4655], a PCE can be either stateful or stateless.  A
   stateful PCE maintains two sets of information for use in path
   computation.  The first is the Traffic Engineering Database (TED)
   which includes the topology and resource state in the network.  This
   information can be obtained by a stateful PCE using the same
   mechanisms as a stateless PCE (see [RFC4655]).  The second is the LSP
   State Database (LSP-DB), in which a PCE stores attributes of all
   active LSPs in the network, such as their paths through the network,
   bandwidth/resource usage, switching types and LSP constraints.  This
   state information allows the PCE to compute constrained paths while
   considering individual LSPs and their inter-dependency.  However,
   this requires reliable state synchronization mechanisms between the
   PCE and the network, between the PCE and the PCCs, and between
   cooperating PCEs, with potentially significant control plane overhead
   and maintenance of a large amount of state data, as explained in
   [RFC4655].

   This document describes how a stateful PCE can be used to solve
   various problems for MPLS-TE and GMPLS networks, and the benefits it
   brings to such deployments.  Note that alternative solutions relying
   on stateless PCEs may also be possible for some of these use cases,
   and will be mentioned for completeness where appropriate.

2.  Terminology

   This document uses the following terms defined in [RFC5440]: PCC,
   PCE, PCEP peer.

   This document uses defines the following terms defined in
   [I-D.ietf-pce-stateful-pce]: Passive Stateful PCE, Active terms:

   Stateful
   PCE, Delegation, Revocation, Delegation Timeout Interval, LSP State
   Report, LSP Update Request, LSP State Database.

   This document defines PCE:  a PCE that has access to not only the following term:

   Minimum Cut Set: network state,
      but also to the minimum set of links active paths and their reserved resources
      for a specific source
      destination pair which, when removed from the network, results in
      a specific source being completely isolated from specific
      destination.  The summed capacity of these links is equivalent to
      the maximum capacity from the source to the destination by the
      max-flow min-cut theorem.

3.  Overview of the Stateful its computations.  A stateful PCE Protocol Extensions

   This section is included for the convenience of the reader, please
   refer might also retain
      information regarding LSPs under construction in order to reduce
      churn and resource contention.  The additional state allows the referenced documents for details of the operation.

   [I-D.ietf-pce-stateful-pce] specifies a set of extensions to PCEP
      PCE to
   enable stateful control of compute constrained paths while considering individual LSPs within
      and across PCEP sessions in
   compliance with [RFC4657].  It includes mechanisms to effect LSP their interactions.  Note that this requires reliable state
      synchronization mechanisms between PCCs and PCEs, delegation of control
   over LSPs to PCEs, and the PCE control of timing and sequence of path
   computations within and across PCEP sessions.

   [I-D.ietf-pce-stateful-pce] applies equally to MPLS-TE the network, PCE
      and GMPLS LSPs PCC, and distinguishes between an active and cooperating PCEs.

   Passive Stateful PCE:  a passive stateful PCE.  A
   passive stateful PCE that uses LSP state information learned
      from PCCs to optimize path
   computations but computations.  It does not actively
      update LSP state.  In contrast, an
   active stateful  A PCC maintains synchronization with the PCE.

    Active Stateful PCE::  a PCE that may issue recommendations to the
      network.  For example, an active stateful Active Stateful PCE may utilize the
      Delegation mechanism to update LSP parameters in those PCCs that
      delegated control over their LSPs to the PCE.

   Several new functions are added in PCEP

   Delegation:  an operation to support both active and
   passive stateful PCEs.  They are described in
   [I-D.ietf-pce-stateful-pce].  A function can be initiated either from
   a PCC towards grant a PCE (C-E) temporary rights to modify a
      subset of LSP parameters on one or more PCC's LSPs.  LSPs are
      delegated from a PCE towards a PCC (E-C). to a PCE, and are referred to as delegated
      LSPs.  The new
   functions are:

   Stateful Capability negotiation (E-C,C-E):  both the PCC and the PCE
      must announce during PCEP session establishment that they support
      stateful owns the PCE PCEP extensions.

   LSP state synchronization (C-E):  after for the session between a PCC and
      a stateful PCE is initialized, LSP has the PCE can perform path
      computation and update attributes in a PCC.  However, if the goal
      of the PCE is right
      to provide accurate path information based on the
      most up-to-date state of the network, the PCE delegate it.  An LSP is owned by a single PCC at any given
      point in time.  For intra-domain LSPs, this PCC should wait until it
      learns the state of be the PCC's LSP states before doing so.
      head end.

   LSP update request (E-C):  A State Database:  information about all LSPs and their attributes.

   PCE requests the modification of one or
      more attributes (e.g., route) on a PCC's LSP.

   LSP state report (C-E):  a PCC sends an LSP state report to Initiation:  a PCE
      whenever the state of an LSP changes. PCE, assuming LSP control delegation (C-E,E-C):  a PCC grants granted by default,
      can issue recommendations to a PCE the right to
      update LSP attributes on one or more LSPs; network.

   Minimum Cut Set:  the PCE becomes minimum set of links for a specific source
      destination pair which, when removed from the
      authoritative network, results in
      a specific source being completely isolated from specific
      destination.  The summed capacity of the LSP's attributes as long as the
      delegation these links is in effect; the PCC may withdraw the delegation or equivalent to
      the PCE may give up maximum capacity from the delegation.

   [I-D.ietf-pce-pce-initiated-lsp] provides extensions source to PCEP which
   enable the setup, maintenance and teardown of PCE-initiated LSPs
   under the stateful PCE model, without destination by the need for local
   configuration on
      max-flow min-cut theorem.

3.  Application Scenarios

   In the PCC, thus allowing for a dynamic network following sections, several use cases are described,
   showcasing scenarios that is
   centrally controlled and deployed.

   [I-D.ietf-pce-stateful-pce] defines benefit from the extensions needed to support
   auto-discovery deployment of a stateful PCEs when using IGP
   PCE.

3.1.  Optimization of LSP Placement

   The following use cases demonstrate a need for visibility into global
   LSP states in PCE discovery.

4.  Deployment Considerations

   This section discusses generic issues with stateful PCE deployments,
   and how specific protocol mechanisms can be used to address them.

4.1.  Multi-PCE Deployments

   Stateless path computations, and stateful PCEs can co-exist in the same network for a PCE control of
   sequence and be timing in charge of altering LSP path computation of different types.  To solve the
   problem of distinguishing between the two types of PCEs, either
   discovery or configuration may be used.  The capability negotiation
   in [I-D.ietf-pce-stateful-pce] ensures correct operation when the PCE
   address is configured on the PCC.

   Multiple stateful PCEs can co-exist in the same network.  These PCEs
   may provide redundancy for load sharing, resilience, or partitioning
   of computation features.  Regardless of the reason characteristics within and
   across PCEP sessions.  Reference topologies for multiple PCEs,
   an LSP is only delegated to one of the PCEs at any given point use cases
   described later in
   time.  [I-D.ietf-pce-stateful-pce] describes how LSPs can be re-
   delegated between PCEs, this section are shown in Figures 1 and 2.

   Some of the procedures on a PCE failure.
   [RFC7399] discusses various approaches for synchronizing state among
   the PCEs when multiple PCEs use cases below are used for load sharing or backup and
   compute focused on MPLS-TE deployments, but
   may also apply to GMPLS.  Unless otherwise cited, use cases assume
   that all LSPs for listed exist at the same network.

4.2. LSP State Synchronization priority.

   The population of main benefit in the LSP-DB using information received cases below comes from PCCs is
   supported by moving away from an
   asynchronous PCC-driven mode of operation to a model that allows for
   central control over LSP computations and maintenance, and focuses
   specifically on the stateful PCE extensions defined in
   [I-D.ietf-pce-stateful-pce] , i.e., via LSP state report messages.
   Population of the LSP database via other means is not precluded.

   Because the accuracy of the computations depends on the accuracy of
   the databases used, it is worth noting that the PCE view lags behind
   the true state of the network, because the updates must reach the PCE
   from the network.  Thus, the use of stateful PCE reduces but cannot
   eliminate the possibility of crankbacks, nor can it guarantee optimal
   computations all the time.  [RFC7399] discusses these limitations and
   potential ways to alleviate them.

   In case of multiple PCEs with different capabilities, co-existing in
   the same network, such as a passive stateful PCE and an active
   stateful PCE, it is useful to refer to a LSP, be it delegated or not,
   by a unique identifier instead of providing detailed information
   (e.g., route, bandwidth etc.) associated with it, when these PCEs
   cooperate on path computation, such as for load sharing.

4.3.  PCE Survivability

   For a stateful PCE, an important issue is to get the LSP state
   information resynchronized after a restart.
   [I-D.ietf-pce-stateful-pce] defines a synchronization function and
   procedure, allowing a PCC to synchronize its LSP state with the PCE
   and [I-D.ietf-pce-stateful-sync-optimizations] specifies
   optimizations to the synchronization procedure.  LSP state
   synchronization procedures can be applied equally to a network node
   or another PCE, allowing multiple ways of re-acquiring the LSP
   database on a restart.  Because synchronization may also be skipped,
   if a PCE implementation has the means to retrieve its database in a
   different way (for example from a backup copy stored locally), the
   state can be restored without further overhead in the network.  A
   hybrid approach where the bulk of the state is recovered locally, and
   a small amount of state is reacquired from the network, is also
   possible.  Note that locally recovering the state would still require
   some degree of resynchronization to ensure that the recovered state
   is indeed up-to-date.  Depending on the resynchronization mechanism
   used, there may be an additional load on the PCE, and there may be a
   delay in reaching the synchronized state, which may negatively affect
   survivability.  Different resynchronization methods are suited for
   different deployments and objectives.

5.  Application Scenarios

   In the following sections, several use cases are described,
   showcasing scenarios that benefit from the deployment of a stateful
   PCE.

5.1.  Optimization of LSP Placement

   The following use cases demonstrate a need for visibility into global
   LSP states in PCE path computations, and for a PCE control of
   sequence and timing in altering LSP path characteristics within and
   across PCEP sessions.  Reference topologies for the use cases
   described later in this section are shown in Figures 1 and 2.

   Some of the use cases below are focused on MPLS-TE deployments, but
   may also apply to GMPLS.  Unless otherwise cited, use cases assume
   that all LSPs listed exist at the same LSP priority.

   The main benefit in the cases below comes from moving away from an
   asynchronous PCC-driven mode of operation to a model that allows for
   central control over LSP computations and maintenance, and focuses
   specifically on the active active stateful PCE model of operation.

          +-----+
          |  A  |
          +-----+
                 \
                  +-----+                      +-----+
                  |  C  |----------------------|  E  |
                  +-----+                      +-----+
                 /        \      +-----+      /
          +-----+          +-----|  D  |-----+
          |  B  |                +-----+
          +-----+

                      Figure 1: Reference topology 1

               +-----+        +-----+        +-----+
               |  A  |        |  B  |        |  C  |
               +--+--+        +--+--+        +--+--+
                  |              |              |
                  |              |              |
               +--+--+        +--+--+        +--+--+
               |  E  +--------+  F  +--------+  G  |
               +-----+        +-----+        +-----+

                      Figure 2: Reference topology 2

5.1.1.

3.1.1.  Throughput Maximization and Bin Packing

   Because LSP attribute changes in [RFC5440] are driven by Path
   Computation Request (PCReq) messages under control of a PCC's local
   timers, the sequence of resource reservation arrivals occurring in
   the network will be randomized.  This, coupled with a lack of global
   LSP state visibility on the part of a stateless PCE may result in
   suboptimal throughput in a given network topology, as will be shown
   in the example below.

   Reference topology 2 in Figure 2 and Tables 1 and 2 show an example
   in which throughput is at 50% of optimal as a result of lack of
   visibility and synchronized control across PCC's.  In this scenario,
   the decision must be made as to whether to route any portion of the
   E-G demand, as any demand routed for this source and destination will
   decrease system throughput.

                       +------+--------+----------+
                       | Link | Metric | Capacity |
                       +------+--------+----------+
                       | A-E  |   1    |    10    |
                       | B-F  |   1    |    10    |
                       | C-G  |   1    |    10    |
                       | E-F  |   1    |    10    |
                       | F-G  |   1    |    10    |
                       +------+--------+----------+

             Table 1: Link parameters for Throughput use case

          +------+-----+-----+-----+--------+----------+-------+
          | Time | LSP | Src | Dst | Demand | Routable |  Path |
          +------+-----+-----+-----+--------+----------+-------+
          |  1   |  1  |  E  |  G  |   10   |   Yes    | E-F-G |
          |  2   |  2  |  A  |  B  |   10   |    No    |  ---  |
          |  3   |  1  |  F  |  C  |   10   |    No    |  ---  |
          +------+-----+-----+-----+--------+----------+-------+

              Table 2: Throughput use case demand time series

   In many cases throughput maximization becomes a bin packing problem.
   While bin packing itself is an NP-hard problem, a number of common
   heuristics which run in polynomial time can provide significant
   improvements in throughput over random reservation event
   distribution, especially when traversing links which are members of
   the minimum cut set for a large subset of source destination pairs.

   Tables 3 and 4 show a simple use case using Reference Topology 1 in
   Figure 1, where LSP state visibility and control of reservation order
   across PCCs would result in significant improvement in total
   throughput.

                       +------+--------+----------+
                       | Link | Metric | Capacity |
                       +------+--------+----------+
                       | A-C  |   1    |    10    |
                       | B-C  |   1    |    10    |
                       | C-E  |   10   |    5     |
                       | C-D  |   1    |    10    |
                       | D-E  |   1    |    10    |
                       +------+--------+----------+

             Table 3: Link parameters for Bin Packing use case

         +------+-----+-----+-----+--------+----------+---------+
         | Time | LSP | Src | Dst | Demand | Routable |   Path  |
         +------+-----+-----+-----+--------+----------+---------+
         |  1   |  1  |  A  |  E  |   5    |   Yes    | A-C-D-E |
         |  2   |  2  |  B  |  E  |   10   |    No    |   ---   |
         +------+-----+-----+-----+--------+----------+---------+

             Table 4: Bin Packing use case demand time series

5.1.2.

3.1.2.  Deadlock

   This section discusses a use case of cross-LSP impact under degraded
   operation.  Most existing RSVP-TE implementations will not tear down
   established LSPs in the event of the failure of the bandwidth
   increase procedure detailed in [RFC3209].  This behavior is directly
   implied to be correct in [RFC3209] and is often desirable from an
   operator's perspective, because either a) the destination prefixes
   are not reachable via any means other than MPLS or b) this would
   result in significant packet loss as demand is shifted to other LSPs
   in the overlay mesh.

   In addition, there are currently few implementations offering dynamic
   ingress admission control (policing of the traffic volume mapped onto
   an LSP) at the label edge router (LER).  Having ingress admission
   control on a per LSP basis is not necessarily desirable from an
   operational perspective, as a) one must over-provision tunnels
   significantly in order to avoid deleterious effects resulting from
   stacked transport and flow control systems (for example for tunnels
   that are dynamically resized based on current traffic) and b) there
   is currently no efficient commonly available northbound interface for
   dynamic configuration of per LSP ingress admission control.

   Lack of ingress admission control coupled with the behavior in
   [RFC3209] may result in LSPs operating out of profile for significant
   periods of time.  It is reasonable to expect that these out-of-
   profile LSPs will be operating in a degraded state and experience
   traffic loss, but because they end up sharing common network
   interfaces with other LSPs operating within their bandwidth
   reservations, thus impacting the operation of the in-profile LSPs,
   even when there is unused network capacity elsewhere in the network.
   Furthermore, this behavior will cause information loss in the TED
   with regards to the actual available bandwidth on the links used by
   the out-of-profile LSPs, as the reservations on the links no longer
   reflect the capacity used.

   Reference Topology 1 in Figure 1 and Tables 5 and 6 show a use case
   that demonstrates this behavior.  Two LSPs, LSP 1 and LSP 2 are
   signaled with demand 2 and routed along paths A-C-D-E and B-C-D-E
   respectively.  At a later time, the demand of LSP 1 increases to 20.
   Under such a demand, the LSP cannot be resignaled.  However, the
   existing LSP will not be torn down.  In the absence of ingress
   policing, traffic on LSP 1 will cause degradation for traffic of LSP
   2 (due to oversubscription on the links C-D and D-E), as well as
   information loss in the TED with regard to the actual network state.

   The problem could be easily ameliorated by global visibility of LSP
   state coupled with PCC-external demand measurements and placement of
   two LSPs on disjoint links.  Note that while the demand of 20 for LSP
   1 could never be satisfied in the given topology, what could be
   achieved would be isolation from the ill-effects of the
   (unsatisfiable) increased demand.

                       +------+--------+----------+
                       | Link | Metric | Capacity |
                       +------+--------+----------+
                       | A-C  |   1    |    10    |
                       | B-C  |   1    |    10    |
                       | C-E  |   10   |    5     |
                       | C-D  |   1    |    10    |
                       | D-E  |   1    |    10    |
                       +------+--------+----------+

       Table 5: Link parameters for the 'Degraded operation' example

         +------+-----+-----+-----+--------+----------+---------+
         | Time | LSP | Src | Dst | Demand | Routable |   Path  |
         +------+-----+-----+-----+--------+----------+---------+
         |  1   |  1  |  A  |  E  |   2    |   Yes    | A-C-D-E |
         |  2   |  2  |  B  |  E  |   2    |   Yes    | B-C-D-E |
         |  3   |  1  |  A  |  E  |   20   |    No    |   ---   |
         +------+-----+-----+-----+--------+----------+---------+

              Table 6: Degraded operation demand time series

5.1.3.

3.1.3.  Minimum Perturbation

   As a result of both the lack of visibility into global LSP state and
   the lack of control over event ordering across PCE sessions,
   unnecessary perturbations may be introduced into the network by a
   stateless PCE.  Tables 7 and 8 show an example of an unnecessary
   network perturbation using Reference Topology 1 in Figure 1.  In this
   case an unimportant (high LSP priority value) LSP (LSP1) is first set
   up along the shortest path.  At time 2, which is assumed to be
   relatively close to time 1, a second more important (lower LSP-
   priority value) LSP (LSP2) is established, preempting LSP1,
   potentially causing traffic loss.  LSP1 is then reestablished on the
   longer A-C-E path.

                       +------+--------+----------+
                       | Link | Metric | Capacity |
                       +------+--------+----------+
                       | A-C  |   1    |    10    |
                       | B-C  |   1    |    10    |
                       | C-E  |   10   |    10    |
                       | C-D  |   1    |    10    |
                       | D-E  |   1    |    10    |
                       +------+--------+----------+

      Table 7: Link parameters for the 'Minimum-Perturbation' example

    +------+-----+-----+-----+--------+----------+----------+---------+
    | Time | LSP | Src | Dst | Demand | LSP Prio | Routable |   Path  |
    +------+-----+-----+-----+--------+----------+----------+---------+
    |  1   |  1  |  A  |  E  |   7    |    7     |   Yes    | A-C-D-E |
    |  2   |  2  |  B  |  E  |   7    |    0     |   Yes    | B-C-D-E |
    |  3   |  1  |  A  |  E  |   7    |    7     |   Yes    |  A-C-E  |
    +------+-----+-----+-----+--------+----------+----------+---------+

         Table 8: Minimum-Perturbation LSP and demand time series

   A stateful PCE can help in this scenario by computing both routes at
   the same time.  The advantages of using a stateful PCE over
   exploiting a stateless PCE via Global Concurrent Optimization(GCO)
   are three folds.  First is the ability to accommodate concurrent path
   computation from different PCCs.  Second is the reduction of control
   plane overhead since the stateful PCE has the route information of
   the affected LSPs.  Thirdly, the stateful PCE can use the LSP-DB to
   further optimize the placement of LSPs.  This will ensure placement
   of the more important LSP along the shortest path, avoiding the setup
   and subsequent preemption of the lower priority LSP.  Similarly, when
   a new higher priority LSP which requires preemption of existing lower
   priority LSP(s), a stateful PCE can determine the minimum number of
   lower priority LSP(s) to reroute using the make-before-break (MBB)
   mechanism without disrupting any service and then set up the higher
   priority LSP.

5.1.4.

3.1.4.  Predictability

   Randomization of reservation events caused by lack of control over
   event ordering across PCE sessions results in poor predictability in
   LSP routing.  An offline system applying a consistent optimization
   method will produce predictable results to within either the boundary
   of forecast error (when reservations are over-provisioned by
   reasonable margins) or to the variability of the signal and the
   forecast error (when applying some hysteresis in order to minimize
   churn).  Predictable results are valuable for being able to simulate
   the network and reliably test it under various scenarios, especially
   under various failure modes and planned maintenances when predictable
   path characteristics are desired under contention for network
   resources.

   Reference Topology 1 and Tables 9, 10 and 11 show the impact of event
   ordering and predictability of LSP routing.

                       +------+--------+----------+
                       | Link | Metric | Capacity |
                       +------+--------+----------+
                       | A-C  |   1    |    10    |
                       | B-C  |   1    |    10    |
                       | C-E  |   1    |    10    |
                       | C-D  |   1    |    10    |
                       | D-E  |   1    |    10    |
                       +------+--------+----------+

         Table 9: Link parameters for the 'Predictability' example

         +------+-----+-----+-----+--------+----------+---------+
         | Time | LSP | Src | Dst | Demand | Routable |   Path  |
         +------+-----+-----+-----+--------+----------+---------+
         |  1   |  1  |  A  |  E  |   7    |   Yes    |  A-C-E  |
         |  2   |  2  |  B  |  E  |   7    |   Yes    | B-C-D-E |
         +------+-----+-----+-----+--------+----------+---------+

           Table 10: Predictability LSP and demand time series 1

         +------+-----+-----+-----+--------+----------+---------+
         | Time | LSP | Src | Dst | Demand | Routable |   Path  |
         +------+-----+-----+-----+--------+----------+---------+
         |  1   |  2  |  B  |  E  |   7    |   Yes    |  B-C-E  |
         |  2   |  1  |  A  |  E  |   7    |   Yes    | A-C-D-E |
         +------+-----+-----+-----+--------+----------+---------+

           Table 11: Predictability LSP and demand time series 2

   As can be shown in the example, both LSPs are routed in both cases,
   but along very different paths.  This would be a challenge if
   reliable simulation of the network is attempted.  An active stateful
   PCE can solve this through control over LSP ordering.  Based on
   triggers such as a failure or an optimization trigger, the PCE can
   order the computations and path setup in a deterministic way.

5.2.

3.2.  Auto-bandwidth Adjustment

   The bandwidth requirement of LSPs often change over time, requiring
   resizing the LSP.  In most implementations available today, the head-
   end node performs this function by monitoring the actual bandwidth
   usage, triggering a recomputation and resignaling when a threshold is
   reached.  This operation is referred as auto-bandwidth adjustment.
   The head-end node either recomputes the path locally, or it requests
   a recomputation from a PCE by sending a PCReq message.  In the latter
   case, the PCE computes a new path and provides the new route
   suggestion.  Upon receiving the reply from the PCE, the PCC re-
   signals the LSP in Shared-Explicit (SE) mode along the newly computed
   path.  With a stateless PCE, the head-end node needs to provide the
   current used bandwidth and the route information via path computation
   request messages.  Note that in this scenario, the head-end node is
   the one that drives the LSP resizing based on local information, and
   that the difference between using a stateless and a passive stateful
   PCE is in the level of optimization of the LSP placement as discussed
   in the previous section.

   A more interesting smart bandwidth adjustment case is one where the
   LSP resizing decision is done by an external entity, with access to
   additional information such as historical trending data, application-
   specific information about expected demands or policy information, as
   well as knowledge of the actual desired flow volumes.  In this case
   an active stateful PCE provides an advantage in both the computation
   with knowledge of all LSPs in the domain and in the ability to
   trigger bandwidth modification of the LSP.

5.3.

3.3.  Bandwidth Scheduling

   Bandwidth scheduling allows network operators to reserve resources in
   advance according to the agreements with their customers, and allow
   them to transmit data with specified starting time and duration, for
   example for a scheduled bulk data replication between data centers.

   Traditionally, this can be supported by network management system
   (NMS) operation through path pre-establishment and activation on the
   agreed starting time.  However, this does not provide efficient
   network usage since the established paths exclude the possibility of
   being used by other services even when they are not used for
   undertaking any service.  It can also be accomplished through GMPLS
   protocol extensions by carrying the related request information
   (e.g., starting time and duration) across the network.  Nevertheless,
   this method inevitably increases the complexity of signaling and
   routing process.

   A passive stateful PCE can support this application with better
   efficiency since it can alleviate the burden of processing on network
   elements.  This requires the PCE to maintain the scheduled LSPs and
   their associated resource usage, as well as the ability of head-ends
   to trigger signaling for LSP setup/deletion at the correct time.
   This approach requires coarse time synchronization between PCEs and
   PCCs.  If an active stateful  With PCE is available, the initiation capability, a PCE can trigger the setup/deletion setup
   and deletion of scheduled requests in a centralized manner, as
   specified [I-D.ietf-pce-pce-initiated-lsp], without
   modification of existing head-end behaviors, by notifying the PCCs to
   set up or tear down the paths.

5.4.

3.4.  Recovery

   The recovery use cases discussed in the following sections show how
   leveraging a stateful PCE can simplify the computation of recovery
   path(s).  In particular, two characteristics of a stateful PCE are
   used: 1) using information stored in the LSP-DB for determining
   shared protection resources and 2) performing computations with
   knowledge of all LSPs in a domain.

5.4.1.

3.4.1.  Protection

   If a PCC can specify in a request whether the computation is for a
   working path or for protection, and a PCC can report the resource as
   a working or protection path, then the following text applies.  A PCC
   can send multiple requests to the PCE, asking for two LSPs and use
   them as working and backup paths separately.  Either way, the
   resources bound to backup paths can be shared by different LSPs to
   improve the overall network efficiency, such as m:n protection or
   pre-configured shared mesh recovery techniques as specified in
   [RFC4427].  If resource sharing is supported for LSP protection, the
   information relating to existing LSPs is required to avoid allocation
   of shared protection resources to two LSPs that might fail together
   and cause protection contention issues.  A stateless PCE can
   accommodate this use case by having the PCC pass this information as
   a constraint in the path computation request.  A passive stateful PCE
   can more easily accommodate this need using the information stored in
   its LSP-DB.  Furthermore, an active stateful PCE can help with (re)-
   optimizization of protection resource sharing as well as LSP
   maintenance operation with fewer impact on protection resources.

                 +----+
                 |PCE |
                 +----+

            +------+          +------+          +------+
            |  A   +----------+  B   +----------+  C   |
            +--+---+          +---+--+          +---+--+
               |                  |                 |
               |        +---------+                 |
               |        |                           |
               |     +--+---+          +------+     |
               +-----+  E   +----------+  D   +-----+
                     +------+          +------+

                      Figure 3: Reference topology 3

   For example, in the network depicted in Figure 3, suppose there
   exists LSP1 with working path LSP1_working following A->E and with
   backup path LSP1_backup following A->B->E.  A request arrives asking
   for a working and backup path pair to be computed for LSP2 from B to
   E.  If the PCE decides LSP2_working follows B->A->E, then the backup
   path LSP2_backup should not share the same protection resource same protection resource with
   LSP1 since LSP2 shares part of its resource (specifically A->E) with
   LSP1 (i.e., these two LSPs are in the same shared risk group).  There
   is no such constraint if B->C->D->E is chosen for LSP2_working.

   If a stateless PCE is used, the head node B needs to be aware of the
   existence of LSPs which share the route of LSP2_working and of the
   details of their protection resources.  B must pass this information
   to the PCE as a constraint so as to request a path with diversity.
   Alternatively, a stateless PCE may able to compute Shared Risk Link
   Group (SRLG)-diversified paths if TED is extended so that it includes
   the SRLG information that are protected by a given backup resource,
   but at the expense of a high complexity in routing.  On the other
   hand, a stateful PCE can get the LSPs information by itself given
   that the LSP identifier(s) and can achieve the goal of finding SRLG-
   diversified protection paths for both LSPs.  This is made possible by
   comparing the LSP resource usage exploiting the LSP-DB accessible by
   the stateful PCE.

3.4.2.  Restoration

   In case of a link failure, such as a fiber cut, multiple LSPs may
   fail at the same time.  Thus, the source nodes of the affected LSPs
   will be informed of the failure by the nodes detecting the failure.
   These source nodes will send requests to a PCE for rerouting.  In
   order to reuse the resource taken by an existing LSP, the source node
   can send a PCReq message including the Exclude Route Object (XRO)
   with
   LSP1 since LSP2 shares part of its resource (specifically A->E) Fail (F) bit set, together with
   LSP1 (i.e., these two LSPs are in the same shared risk group).  There
   is no such constraint if B->C->D->E is chosen for LSP2_working. record route object (RRO)
   containing the current route information, as specified in [RFC5521].

   If a stateless PCE is used, the head node B needs it might respond to be aware of the
   existence rerouting
   requests separately if they arrive at different times.  Thus, it
   might result in sub-optimal resource usage.  Even worse, it might
   unnecessarily block some of LSPs which share the route of LSP2_working and of rerouting requests due to
   insufficient resources for later-arrived rerouting messages.  If a
   passive stateful PCE is used to fulfill this task, the
   details procedure can
   be simplified.  The PCCs reporting the failures can include LSP
   identifiers instead of their protection resources.  B must pass this detailed information
   to and the PCE as a constraint so as to request a path with diversity.
   Alternatively, a stateless can find
   relevant LSP information by inspecting the LSP-DB.  Moreover, the PCE may able to compute Shared Risk Link
   Group (SRLG)-diversified paths if TED is extended so that
   can re-compute the affected LSPs concurrently while reusing part of
   the existing LSPs resources when it includes is informed of the SRLG information that failed link
   identifier provided by the first request.  This is made possible
   since the passive stateful PCE can check what other LSPs are protected affected
   by the failed link and their route information by inspecting its LSP-
   DB.  As a result, a better performance can be achieved, such as
   better resource usage or minimal probability of blocking upcoming new
   rerouting requests sent as a given backup resource,
   but at result of the expense link failure.

   If the target is to avoid resource contention within the time-window
   of a high complexity in routing.  On the other
   hand, number of LSP rerouting requests, a stateful PCE can get retain
   the LSPs under-construction LSP resource usage information by itself for a given
   time and exclude it from being used for forthcoming LSPs request.  In
   this way, it can ensure that the LSP identifier(s) resource will not be double-booked
   and can achieve thus the goal issue of finding SRLG-
   diversified protection paths for both LSPs.  This is made possible by
   comparing the LSP resource usage exploiting the LSP-DB accessible by
   the stateful PCE.

5.4.2.  Restoration

   In case contention and computation crank-backs
   can be alleviated.

3.4.3.  SRLG Diversity

   An alternative way to achieve efficient resilience is to maintain
   SRLG disjointness between LSPs, irrespective of a link failure, such as a fiber cut, multiple whether these LSPs may
   fail at the same time.  Thus,
   share the source and destination nodes or not.  This can be achieved
   at provisioning time, if the routes of all the affected LSPs
   will are requested
   together, using a synchronized computation of the different LSPs with
   SRLG disjointness constraint.  If the LSPs need to be informed provisioned at
   different times, the PCC can specify, as constraints to the path
   computation a set of SRLGs using the failure by Exclude Route Object [RFC5521].
   However, for the nodes detecting latter to be effective, it is needed that the failure.
   These source nodes will send entity
   that requests the route to a the PCE for rerouting.  In
   order maintains updated SRLG information
   of all the LSPs to reuse which it must maintain the resource taken by disjointness.  A
   stateless PCE can compute an existing LSP, SRLG-disjoint path by inspecting the source node
   can send a PCReq message including TED
   and precluding the Exclude Route Object (XRO)
   with Fail (F) bit set, together links with the record route object (RRO)
   containing the current route information, as same SRLG values specified in [RFC5521].

   If the
   PCReq message sent by a stateless PCC.

   A passive stateful PCE is used, it might respond to maintains the rerouting
   requests separately if they arrive at different times.  Thus, it
   might result in sub-optimal resource usage.  Even worse, it might
   unnecessarily block some updated SRLG information of the rerouting requests due to
   insufficient resources for later-arrived rerouting messages.  If
   established LSPs in a
   passive stateful PCE is used to fulfill this task, centralized manner.  Therefore, the procedure PCC can
   be simplified.  The PCCs reporting
   specify as constraints to the failures can include LSP
   identifiers instead path computation the SRLG disjointness
   of detailed information and a set of already established LSPs by only providing the LSP
   identifiers.  Similarly, a passive stateful PCE can find
   relevant LSP information by inspecting also accommodate
   disjointness using other constraints, such as link, node or path
   segment etc.

3.5.  Maintenance of Virtual Network Topology (VNT)

   In Multi-Layer Networks (MLN), a Virtual Network Topology (VNT)
   [RFC5212] consists of a set of one or more TE LSPs in the LSP-DB.  Moreover, lower layer
   which provides TE links to the PCE
   can re-compute upper layer.  In [RFC5623], the affected LSPs concurrently while reusing part PCE-
   based architecture is proposed to support path computation in MLN
   networks in order to achieve inter-layer TE.

   The establishment/teardown of a TE link in VNT needs to take into
   consideration the state of existing LSPs resources when it is informed of the failed link
   identifier provided by the first request.  This is made possible
   since and/or new LSP request(s) in
   the passive stateful higher layer.  Hence, when a stateless PCE can check what other LSPs are affected
   by cannot find the failed link and their route information by inspecting its LSP-
   DB.  As a result,
   for a better performance can be achieved, such as
   better resource usage request based on the upper layer topology information, it does
   not have enough information to decide whether to set up or minimal probability of blocking upcoming new
   rerouting requests sent as remove a
   TE link or not, which then can result in non-optimal usage of
   resource.  On the link failure.

   If the target is to avoid resource contention within the time-window
   of high number of LSP rerouting requests, other hand, a passive stateful PCE can retain make a
   better decision of when and how to modify the under-construction VNT either to
   accommodate new LSP requests or to re-optimize resource usage information for a across
   layers irrespective of the PCE models as described in [RFC5623].
   Furthermore, given
   time and exclude it from being used for forthcoming LSPs request.  In
   this way, it can ensure that the resource will not be double-booked
   and thus active capability, the stateful PCE can issue of
   VNT modification suggestions in order to accommodate path setup
   requests or re-optimize resource contention and computation crank-backs
   can be alleviated.

5.4.3.  SRLG Diversity

   An alternative way usage across layers.

3.6.  LSP Re-optimization

   In order to achieve make efficient resilience usage of network resources, it is
   sometimes desirable to maintain
   SRLG disjointness between LSPs, irrespective of whether these LSPs
   share the source and destination nodes re-optimize one or not.  This can be achieved
   at provisioning time, if more LSPs dynamically.  In
   the routes case of all the LSPs are requested
   together, using a synchronized computation of stateless PCE, in order to optimize network resource
   usage dynamically through online planning, a PCC must send a request
   to the different LSPs PCE together with
   SRLG disjointness constraint.  If detailed path/bandwidth information of the
   LSPs that need to be provisioned at
   different times, concurrently optimized.  This means the PCC can specify, as constraints must
   be able to determine when and which LSPs should be optimized.  In the path
   computation a set
   case of SRLGs using the Exclude Route Object [RFC5521].
   However, for the latter to be effective, it is needed that a passive stateful PCE, given the entity
   that requests LSP state information in
   the route to LSP database, the PCE maintains updated SRLG information process of all dynamic optimization of network
   resources can be simplified without requiring the LSPs PCC to which it must maintain the disjointness.  A
   stateless supply
   detailed LSP state information.  Moreover, an active stateful PCE can compute an SRLG-disjoint path by inspecting the TED
   and precluding the links with the same SRLG values specified in
   even make the
   PCReq message sent process automated by triggering the request since a PCC.

   A passive
   stateful PCE maintains the updated SRLG can maintain information of the
   established for all LSPs that are in a centralized manner.  Therefore, the PCC can
   specify as constraints to the path computation the SRLG disjointness
   process of a being set of already established LSPs by only providing up and it may have the ability to control timing
   and sequence of LSP
   identifiers.  Similarly, a passive setup/deletion, the optimization procedures can
   be performed more intelligently and effectively.  A stateful PCE can
   also accommodate
   disjointness using other constraints, such as link, node or path
   segment etc.

5.5.  Maintenance of Virtual Network Topology (VNT)

   In Multi-Layer Networks (MLN), a Virtual Network Topology (VNT)
   [RFC5212] consists of a set of one or more TE LSPs in the lower layer determine which provides TE links to the upper layer.  In [RFC5623], the PCE- LSP should be re-optimized based architecture on network
   events.  For example, when a LSP is proposed torn down, its resources are
   freed.  This can trigger the stateful PCE to support path computation in MLN
   networks in order automatically determine
   which LSP should be reoptimized so that the recently freed resources
   may be allocated to achieve inter-layer TE.

   The establishment/teardown it.

   A special case of a TE link in VNT needs to take into
   consideration the state LSP re-optimization is GCO [RFC5557].  Global
   control of existing LSPs and/or new LSP request(s) operation sequence in
   the higher layer.  Hence, when a stateless PCE cannot find the route
   for a request based [RFC5557] is predicated on the upper layer topology information, it does
   use of what is effectively a stateful (or semi-stateful) NMS.  The
   NMS can be either not have enough information to decide whether local to set up or remove a
   TE link or not, the network nodes, in which then can result case
   another northbound interface is required for LSP attribute changes,
   or local/collocated, in non-optimal usage of
   resource.  On the other hand, a passive which case there are significant issues with
   efficiency in resource usage.  A stateful PCE can make adds a
   better decision of when and how to modify few features
   that:

   o  Roll the VNT either to
   accommodate new LSP requests or to re-optimize resource usage across
   layers irrespective of NMS visibility into the PCE models as described in [RFC5623].
   Furthermore, given and remove the active capability, requirement
      for an additional northbound interface

   o  Allow the stateful PCE can issue
   VNT modification suggestions in order to accommodate path setup
   requests or re-optimize resource usage across layers.

5.6.  LSP Re-optimization

   In order to make efficient usage of network resources, it determine when re-optimization is
   sometimes desirable to re-optimize one needed, with
      which level (GCO or a more LSPs dynamically.  In incremental optimization)

   o  Allow the case of PCE to determine which LSPs should be re-optimized

   o  Allow a stateless PCE, in order PCE to optimize network resource
   usage control the sequence of events across multiple
      PCCs, allowing for bulk (and truly global) optimization, LSP
      shuffling etc.

3.7.  Resource Defragmentation

   If LSPs are dynamically through online planning, allocated and released over time, the
   resource becomes fragmented.  In networks with link bundle, the
   overall available resource on a PCC must send (bundle) link might be sufficient for
   a new LSP request, but if the available resource is not continuous,
   the request is rejected.  In order to perform the PCE together with detailed path/bandwidth information defragmentation
   procedure, stateful PCEs can be used, since global visibility of the LSPs that need
   in the network is required to be concurrently optimized. accurately assess resources on the
   LSPs, and perform de-fragmentation while ensuring a minimal
   disruption of the network.  This means use case cannot be accommodated by a
   stateless PCE since it does not possess the PCC must
   be able to determine when and which detailed information of
   existing LSPs should be optimized.  In in the network.

   Another case of a passive stateful PCE, given particular interest is the LSP state information optical spectrum
   defragmentation in flexible grid networks.  In Flexible grid networks
   [RFC7698], LSPs with different optical spectrum sizes (such as
   12.5GHz, 25GHz etc.) can co-exist so as to accommodate the LSP database, services
   with different bandwidth requests.  Therefore, even if the process of dynamic optimization of network
   resources overall
   spectrum size can meet the service request, it may not be simplified without requiring usable if
   the PCC to supply
   detailed available spectrum resource is not contiguous, but rather
   fragmented into smaller pieces.  Thus, with the help of existing LSP
   state information.  Moreover, an active information, a stateful PCE can
   even make the process automated by triggering the request since resource grouped
   together to be usable.  Moreover, a stateful PCE can maintain information proactively
   choose routes for all LSPs that are in upcoming path requests to reduce the
   process chance of being set up and it may have
   spectrum fragmentation.

3.8.  Point-to-Multi-Point Applications

   PCE has been identified as an appropriate technology for the ability to control timing
   and sequence
   determination of LSP setup/deletion, the optimization procedures can
   be performed more intelligently paths of point-to-multipoint (P2MP) TE LSPs
   [RFC5671].  The application scenarios and effectively.  A use-cases described in
   Section 3.1, Section 3.4 and Section 3.6 are also applicable to P2MP
   TE LSPs.

   In addition to these, the stateful nature of a PCE can
   also determine which LSP should be re-optimized based on network
   events. simplifies the
   information conveyed in PCEP messages since it is possible to refer
   to the LSPs via an identifier.  For example, when a LSP P2MP, this is torn down, its resources are
   freed.  This can trigger an added advantage,
   where the stateful PCE to automatically determine
   which LSP should be reoptimized so that size of the recently freed resources
   may be allocated to it.

   A special PCEP message is much larger.  In case of LSP re-optimization is GCO [RFC5557].  Global
   control
   stateless PCEs, modification of LSP operation sequence in [RFC5557] is predicated on the
   use a P2MP tree requires encoding of what is effectively all
   leaves along with the paths in PCReq message.  But using a stateful (or semi-stateful) NMS.  The
   NMS
   PCE with P2MP capability, the PCEP message can be either not local used to convey only
   the network nodes, in which case
   another northbound interface is required for modifications (the other information can be retrieved from the
   identifier via the LSP-DB).

3.9.  Impairment-Aware Routing and Wavelength Assignment (IA-RWA)

   In Wavelength Switched Optical Networks (WSONs) [RFC6163], a
   wavelength-switched LSP attribute changes, traverses one or local/collocated, in which case there are significant issues more fiber links.  The bit
   rates of the client signals carried by the wavelength LSPs may be the
   same or different.  Hence, a fiber link may transmit a number of
   wavelength LSPs with
   efficiency in resource usage.  A stateful PCE adds equal or mixed bit rate signals.  For example, a few features
   that:

   o  Roll
   fiber link may multiplex the wavelengths with only 10Gb/s signals,
   mixed 10Gb/s and 40Gb/s signals, or mixed 40Gb/s and 100Gb/s signals.

   IA-RWA in WSONs refers to the process (i.e., lightpath computation)
   that takes into account the optical layer/transmission imperfections
   by considering as additional (i.e., physical layer) constraints.  To
   be more specific, linear and non-linear effects associated with the NMS visibility
   optical network elements should be incorporated into the PCE route and remove
   wavelength assignment procedure.  For example, the requirement
      for an additional northbound interface

   o  Allow physical
   imperfection can result in the PCE interference of two adjacent
   lightpaths.  Thus, a guard band should be reserved between them to determine when re-optimization is needed,
   alleviate these effects.  The width of the guard band between two
   adjacent wavelengths depends on their characteristics, such as
   modulation formats and bit rates.  Two adjacent wavelengths with
      which level (GCO or
   different characteristics (e.g., different bit rates) may need a more incremental optimization)

   o  Allow
   wider guard band and with same characteristics may need a narrower
   guard band.  For example, 50GHz spacing may be acceptable for two
   adjacent wavelengths with 40G signals.  But for two adjacent
   wavelengths with different bit rates (e.g., 10G and 40G), a larger
   spacing such as 300GHz spacing may be needed.  Hence, the PCE to determine which
   characteristics (states) of the existing wavelength LSPs should be re-optimized

   o  Allow
   considered for a PCE new RWA request in WSON.

   In summary, when stateful PCEs are used to control perform the sequence IA-RWA
   procedure, they need to know the characteristics of events across multiple
      PCCs, allowing for bulk (and truly global) optimization, LSP
      shuffling etc.

5.7.  Resource Defragmentation

   If the existing
   wavelength LSPs.  The impairment information relating to existing and
   to-be-established LSPs are dynamically allocated can be obtained by nodes in WSON networks via
   external configuration or other means such as monitoring or
   estimation based on a vendor-specific impair model.  However, WSON
   related routing protocols, i.e., [RFC7688] and released over time, [RFC7580], only
   advertise limited information (i.e., availability) of the
   resource becomes fragmented.  In networks with link bundle, existing
   wavelengths, without defining the
   overall available resource on a (bundle) link might be sufficient for
   a new LSP request, but supported client bit rates.  It
   will incur substantial amount of control plane overhead if routing
   protocols are extended to support dissemination of the available resource is not continuous, new
   information relevant for the request is rejected. IA-RWA process.  In order to perform the defragmentation
   procedure, this scenario,
   stateful PCEs can PCE(s) would be used, since global visibility a more appropriate mechanism to solve this
   problem.  Stateful PCE(s) can exploit impairment information of LSPs
   stored in the network is required LSP-DB to accurately assess resources on the
   LSPs, and perform de-fragmentation while ensuring a minimal
   disruption of the network. provide accurate RWA calculation.

4.  Deployment Considerations

   This use case cannot be accommodated by a
   stateless section discusses general issues with stateful PCE since it does not possess the detailed information deployments,
   and identifies areas where additional protocol extensions and
   precedures are needed to address them.  Definitions of
   existing LSPs in protocol
   mechanisms are beyond the network.

   Another case scope of particular interest is the optical spectrum
   defragmentation in flexible grid networks.  In Flexible grid networks
   [RFC7698], LSPs with different optical spectrum sizes (such as
   12.5GHz, 25GHz etc.) this document.

4.1.  Multi-PCE Deployments

   Stateless and stateful PCEs can co-exist so as to accommodate in the services
   with same network and be
   in charge of path computation of different bandwidth requests.  Therefore, even if types.  To solve the overall
   spectrum size
   problem of distinguishing between the two types of PCEs, either
   discovery or configuration may be used.

   Multiple stateful PCEs can meet co-exist in the service request, it same network.  These PCEs
   may not be usable if provide redundancy for load sharing, resilience, or partitioning
   of computation features.  Regardless of the available spectrum resource reason for multiple PCEs,
   an LSP is not contiguous, but rather
   fragmented into smaller pieces.  Thus, with the help only delegated to one of existing the PCEs at any given point in
   time.  However, an LSP
   state information, a stateful PCE can make the resource grouped
   together to be usable.  Moreover, re-delegated between PCEs, for example
   when a stateful PCE can proactively
   choose routes fails.  [RFC7399] discusses various approaches for upcoming path requests to reduce
   synchronizing state among the chance of
   spectrum fragmentation.

5.8.  Point-to-Multi-Point Applications

   PCE has been identified as an appropriate technology PCEs when multiple PCEs are used for
   load sharing or backup and compute LSPs for the
   determination of same network.

4.2.  LSP State Synchronization

   The LSP-DB is populated using information received from the paths PCC.
   Because the accuracy of point-to-multipoint (P2MP) TE LSPs
   [RFC5671].  The application scenarios and use-cases described in
   Section 5.1, Section 5.4 and Section 5.6 are also applicable to P2MP
   TE LSPs.

   In addition to these, the stateful nature computations depends on the accuracy of a PCE simplifies
   the
   information conveyed in PCEP messages since databases used, it is possible to refer
   to worth noting that the LSPs via an identifier.  For P2MP, this is an added advantage,
   where PCE view lags behind
   the size true state of the PCEP message is much larger.  In case of
   stateless PCEs, modification of a P2MP tree requires encoding of all
   leaves along with network, because the updates must reach the PCE
   from the paths in PCReq message.  But using a network.  Thus, the use of stateful PCE with P2MP capability, the PCEP message can be used to convey only reduces but cannot
   eliminate the modifications (the other information possibility of crankbacks, nor can be retrieved from the
   identifier via it guarantee optimal
   computations all the LSP-DB).

5.9.  Impairment-Aware Routing time.  [RFC7399] discusses these limitations and Wavelength Assignment (IA-RWA)
   potential ways to alleviate them.

   In Wavelength Switched Optical Networks (WSONs) [RFC6163], a
   wavelength-switched LSP traverses one or more fiber links.  The bit
   rates case of the client signals carried by the wavelength LSPs may be multiple PCEs with different capabilities, co-existing in
   the same or different.  Hence, a fiber link may transmit a number of
   wavelength LSPs with equal or mixed bit rate signals.  For example, network, such as a
   fiber link may multiplex the wavelengths with only 10Gb/s signals,
   mixed 10Gb/s and 40Gb/s signals, or mixed 40Gb/s passive stateful PCE and 100Gb/s signals.

   IA-RWA in WSONs refers an active
   stateful PCE, it is useful to the process (i.e., lightpath computation)
   that takes into account the optical layer/transmission imperfections
   by considering as additional (i.e., physical layer) constraints.  To refer to a LSP, be more specific, linear and non-linear effects it delegated or not,
   by a unique identifier instead of providing detailed information
   (e.g., route, bandwidth etc.) associated with the
   optical network elements should be incorporated into the route and
   wavelength assignment procedure. it, when these PCEs
   cooperate on path computation, such as for load sharing.

4.3.  PCE Survivability

   For example, the physical
   imperfection can result in the interference of two adjacent
   lightpaths.  Thus, a guard band should stateful PCE, an important issue is to get the LSP state
   information resynchronized after a restart.  LSP state
   synchronization procedures can be reserved between them applied equally to
   alleviate these effects.  The width a network node
   or another PCE, allowing multiple ways of re-acquiring the guard band between two
   adjacent wavelengths depends LSP
   database on their characteristics, such as
   modulation formats and bit rates.  Two adjacent wavelengths with
   different characteristics (e.g., different bit rates) may need a
   wider guard band and with same characteristics may need a narrower
   guard band.  For example, 50GHz spacing restart.  Because synchronization may also be acceptable for two
   adjacent wavelengths with 40G signals.  But for two adjacent
   wavelengths with different bit rates (e.g., 10G and 40G), skipped,
   if a larger
   spacing such as 300GHz spacing may be needed.  Hence, PCE implementation has the
   characteristics (states) of means to retrieve its database in a
   different way (for example from a backup copy stored locally), the existing wavelength LSPs should
   state can be
   considered for a new RWA request restored without further overhead in WSON.

   In summary, when stateful PCEs are used to perform the IA-RWA
   procedure, they need to know network.  A
   hybrid approach where the characteristics bulk of the existing
   wavelength LSPs.  The impairment information relating to existing state is recovered locally, and
   to-be-established LSPs can be obtained by nodes in WSON networks via
   external configuration or other means such as monitoring or
   estimation based on
   a vendor-specific impair model.  However, WSON
   related routing protocols, i.e., [RFC7688] and [RFC7580], only
   advertise limited information (i.e., availability) small amount of state is reacquired from the existing
   wavelengths, without defining network, is also
   possible.  Note that locally recovering the supported client bit rates.  It
   will incur substantial amount state would still require
   some degree of control plane overhead if routing
   protocols are extended resynchronization to support dissemination of ensure that the new
   information relevant for recovered state
   is indeed up-to-date.  Depending on the IA-RWA process.  In this scenario,
   stateful PCE(s) would resynchronization mechanism
   used, there may be an additional load on the PCE, and there may be a more appropriate mechanism to solve this
   problem.  Stateful PCE(s) can exploit impairment information of LSPs
   stored
   delay in LSP-DB to provide accurate RWA calculation.

6. reaching the synchronized state, which may negatively affect
   survivability.  Different resynchronization methods are suited for
   different deployments and objectives.

5.  Security Considerations

   This document describes general considerations for a stateful PCE
   deployment and examines its applicability and benefits, as well as
   its challenges and limitations through a number of use cases.  No new
   protocol extensions to PCEP are defined in this document.

   The PCEP extensions in support of the stateful PCE and the delegation
   of path control, control ability can result in more information and control
   being available for a hypothetical adversary and a number of
   additional attack surfaces which must be protected.  [I-D.ietf-pce-stateful-pce] discusses
   different attack vectors and defines protocol mechanisms  This includes
   but not limited to protect
   against them.  It also lays out implementation requirements for
   configuration capabilities that allow the operator authentication and encryption of PCEP
   sessions, snooping of the state of the LSPs active in the network
   etc.  Therefore, documents where the PCEP protocol extensions are
   defined need to control consider the PCC
   behavior when faced issues and risks associated with an attack.  This document does not introduce
   any new security considerations beyond those discussed in
   [I-D.ietf-pce-stateful-pce].

7. a
   stateful PCE.

6.  IANA Considerations

   This document does not require any IANA action.

8.

7.  Contributing Authors

   The following people all contributed significantly to this document
   and are listed below in alphabetical order:

   Ramon Casellas
   CTTC - Centre Tecnologic de Telecomunicacions de Catalunya
   Av.  Carl Friedrich Gauss n7
   Castelldefels, Barcelona 08860
   Spain
   Email: ramon.casellas@cttc.es

   Edward Crabbe
   Email: edward.crabbe@gmail.com

   Dhruv Dhody
   Huawei Technology
   Leela Palace
   Bangalore, Karnataka 560008
   INDIA
   EMail: dhruv.dhody@huawei.com

   Oscar Gonzalez de Dios
   Telefonica Investigacion y Desarrollo
   Emilio Vargas 6
   Madrid, 28045
   Spain
   Phone: +34 913374013
   Email: ogondio@tid.es

   Young Lee
   Huawei
   1700 Alma Drive, Suite 100
   Plano, TX 75075
   US
   Phone: +1 972 509 5599 x2240
   Fax: +1 469 229 5397
   EMail: leeyoung@huawei.com

   Jan Medved
   Cisco Systems, Inc.
   170 West Tasman Dr.
   San Jose, CA 95134
   US
   Email: jmedved@cisco.com

   Robert Varga
   Pantheon Technologies LLC
   Mlynske Nivy 56
   Bratislava 821 05
   Slovakia
   Email: robert.varga@pantheon.sk

   Fatai Zhang
   Huawei Technologies
   F3-5-B R&D Center, Huawei Base
   Bantian, Longgang District
   Shenzhen 518129 P.R.China
   Phone: +86-755-28972912
   Email: zhangfatai@huawei.com

   Xiaobing Zi
   Email: unknown

9.

8.  Acknowledgements

   We would like to thank Cyril Margaria, Adrian Farrel, JP Vasseur and
   Ravi Torvi for the useful comments and discussions.

10.

9.  References

10.1.

9.1.  Normative References

   [I-D.ietf-pce-stateful-pce]
              Crabbe, E., Minei, I., Medved, J., and R. Varga, "PCEP
              Extensions for Stateful PCE", draft-ietf-pce-stateful-
              pce-16 (work in progress), September 2016.

   [RFC4655]  Farrel, A., Vasseur, J., and J. Ash, "A Path Computation
              Element (PCE)-Based Architecture", RFC 4655,
              DOI 10.17487/RFC4655, August 2006,
              <http://www.rfc-editor.org/info/rfc4655>.

   [RFC5440]  Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation
              Element (PCE) Communication Protocol (PCEP)", RFC 5440,
              DOI 10.17487/RFC5440, March 2009,
              <http://www.rfc-editor.org/info/rfc5440>.

   [RFC7399]  Farrel, A. and D. King, "Unanswered Questions in the Path
              Computation Element Architecture", RFC 7399,
              DOI 10.17487/RFC7399, October 2014,
              <http://www.rfc-editor.org/info/rfc7399>.

10.2.

9.2.  Informative References

   [I-D.ietf-pce-gmpls-pcep-extensions]
              Margaria, C., Dios, O., and F. Zhang, "PCEP extensions for
              GMPLS", draft-ietf-pce-gmpls-pcep-extensions-11 (work in
              progress), October 2015.

   [I-D.ietf-pce-pce-initiated-lsp]
              Crabbe, E., Minei, I., Sivabalan, S., and R. Varga, "PCEP
              Extensions for PCE-initiated LSP Setup in a Stateful PCE
              Model", draft-ietf-pce-pce-initiated-lsp-07 (work in
              progress), July 2016.

   [I-D.ietf-pce-stateful-sync-optimizations]
              Crabbe, E., Minei, I., Medved, J., Varga, R., Zhang, X.,
              and D. Dhody, "Optimizations of Label Switched Path State
              Synchronization Procedures for a Stateful PCE", draft-
              ietf-pce-stateful-sync-optimizations-05 (work in
              progress), April 2016.

   [RFC3209]  Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V.,
              and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP
              Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001,
              <http://www.rfc-editor.org/info/rfc3209>.

   [RFC4427]  Mannie, E., Ed. and D. Papadimitriou, Ed., "Recovery
              (Protection and Restoration) Terminology for Generalized
              Multi-Protocol Label Switching (GMPLS)", RFC 4427,
              DOI 10.17487/RFC4427, March 2006,
              <http://www.rfc-editor.org/info/rfc4427>.

   [RFC4657]  Ash, J., Ed. and J. Le Roux, Ed., "Path Computation
              Element (PCE) Communication Protocol Generic
              Requirements", RFC 4657, DOI 10.17487/RFC4657, September
              2006, <http://www.rfc-editor.org/info/rfc4657>.

   [RFC5212]  Shiomoto, K., Papadimitriou, D., Le Roux, JL., Vigoureux,
              M., and D. Brungard, "Requirements for GMPLS-Based Multi-
              Region and Multi-Layer Networks (MRN/MLN)", RFC 5212,
              DOI 10.17487/RFC5212, July 2008,
              <http://www.rfc-editor.org/info/rfc5212>.

   [RFC5521]  Oki, E., Takeda, T., and A. Farrel, "Extensions to the
              Path Computation Element Communication Protocol (PCEP) for
              Route Exclusions", RFC 5521, DOI 10.17487/RFC5521, April
              2009, <http://www.rfc-editor.org/info/rfc5521>.

   [RFC5557]  Lee, Y., Le Roux, JL., King, D., and E. Oki, "Path
              Computation Element Communication Protocol (PCEP)
              Requirements and Protocol Extensions in Support of Global
              Concurrent Optimization", RFC 5557, DOI 10.17487/RFC5557,
              July 2009, <http://www.rfc-editor.org/info/rfc5557>.

   [RFC5623]  Oki, E., Takeda, T., Le Roux, JL., and A. Farrel,
              "Framework for PCE-Based Inter-Layer MPLS and GMPLS
              Traffic Engineering", RFC 5623, DOI 10.17487/RFC5623,
              September 2009, <http://www.rfc-editor.org/info/rfc5623>.

   [RFC5671]  Yasukawa, S. and A. Farrel, Ed., "Applicability of the
              Path Computation Element (PCE) to Point-to-Multipoint
              (P2MP) MPLS and GMPLS Traffic Engineering (TE)", RFC 5671,
              DOI 10.17487/RFC5671, October 2009,
              <http://www.rfc-editor.org/info/rfc5671>.

   [RFC6163]  Lee, Y., Ed., Bernstein, G., Ed., and W. Imajuku,
              "Framework for GMPLS and Path Computation Element (PCE)
              Control of Wavelength Switched Optical Networks (WSONs)",
              RFC 6163, DOI 10.17487/RFC6163, April 2011,
              <http://www.rfc-editor.org/info/rfc6163>.

   [RFC7580]  Zhang, F., Lee, Y., Han, J., Bernstein, G., and Y. Xu,
              "OSPF-TE Extensions for General Network Element
              Constraints", RFC 7580, DOI 10.17487/RFC7580, June 2015,
              <http://www.rfc-editor.org/info/rfc7580>.

   [RFC7688]  Lee, Y., Ed. and G. Bernstein, Ed., "GMPLS OSPF
              Enhancement for Signal and Network Element Compatibility
              for Wavelength Switched Optical Networks", RFC 7688,
              DOI 10.17487/RFC7688, November 2015,
              <http://www.rfc-editor.org/info/rfc7688>.

   [RFC7698]  Gonzalez de Dios, O., Ed., Casellas, R., Ed., Zhang, F.,
              Fu, X., Ceccarelli, D., and I. Hussain, "Framework and
              Requirements for GMPLS-Based Control of Flexi-Grid Dense
              Wavelength Division Multiplexing (DWDM) Networks",
              RFC 7698, DOI 10.17487/RFC7698, November 2015,
              <http://www.rfc-editor.org/info/rfc7698>.

Authors' Addresses

   Xian Zhang (editor)
   Huawei Technologies
   F3-5-B R&D Center, Huawei Industrial Base, Bantian, Longgang District
   Shenzhen, Guangdong  518129
   P.R.China

   Email: zhang.xian@huawei.com

   Ina Minei (editor)
   Google, Inc.
   1600 Amphitheatre Parkway
   Mountain View, CA  94043
   US

   Email: inaminei@google.com