ForCES Working Group                 Jamal Hadi Salim
Internet Draft                       Znyx Networks
                                     Hormuzd Khosravi
                                     Intel
                                     Andi Kleen
                                     Suse
                                     Alexey Kuznetsov
                                     INR/Swsoft
                                     March
                                     June 2002

                   Netlink as an IP services protocol
                     draft-ietf-forces-netlink-02.txt Services Protocol
                   draft-ietf-forces-netlink-03.txt

Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC2026.  Internet-Drafts are working
   documents of the Internet Engineering Task Force (IETF), its areas,
   and its working groups.  Note that other groups may also distribute
   working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as ``work in progress.''

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

Conventions used in this document

     The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
     "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in
     this document are to be interpreted as described in [RFC-2119].

1.  Abstract

     This document describes Linux Netlink, which is used in Linux both
     as an inter-kernel intra-kernel messaging system as well as between kernel and

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

     user-space.  The purpose of this                                draft-forces-Netlink-03.txt

     user space.  This document is intended as informa-
     tional informational in the context con-
     text of prior art for the ForCES IETF working group.  The focus of
     this
      document is to describe netlink Netlink from a
     context perspective of a protocol
     between a Forwarding Engine Component (FEC) and a Control Plane Component(CPC)
     Component (CPC), the two components that define an IP service.

     The document ignores the ability of netlink Netlink as a inter-kernel intra-kernel mes-
     saging system, as a an inter-process communication scheme (IPC) (IPC), or
     its use in configuring other non-network as well
     as network but a configuration tool for other non-networking or non-IP network
     services (such as decnet etc). decnet, etc.).

2.  Introduction

     The concept of IP Service control-forwarding separation was first
     introduced in the early 1980s by the BSD 4.4 routing sock-
     ets[stevens]. sockets
     [Stevens].  The focus at that time was a simple IP(v4) forward-
     ing forwarding
     service and how the CPC, either via a command line configura-
     tion configuration
     tool or a dynamic route daemon, can could control forwarding tables for
     that IPV4 IPv4 forwarding service.

     The IP world has evolved considerably since those days.  Linux
     netlink,
     Netlink, when observed from a service provisioning and management
     point of view view, takes routing sockets one step further by breaking
     the barrier of focus around IPV4 IPv4 forwarding.  Since the linux Linux 2.1
     kernel, netlink Netlink has been providing the IP service abstraction to a
     few services other than the classical RFC 1812 IPv4 forwarding.

     The motivation for this document is not to list every possible ser-
     vice for which Netlink is applied.  In fact, we leave out a lot of
     services (multicast routing, tunnelling, policy routing, etc.).
     Neither is this document intended to be a tutorial on Netlink.  The
     idea is to explain the overall Netlink view with a special focus on
     the mandatory building blocks within the ForCES charter (i.e., IPv4
     and QoS).  This document also serves to capture prior art to many
     mechanisms that are useful within the context of ForCES.  The text
     is limited to a subset of what is available in kernel 2.4.6, the
     newest kernel when this document was first written.  It is also
     limited to IPv4 functionality.

     We first give some concept definitions and then describe how
     netlink
     Netlink fits in.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

2.1.  Some definitions  Definitions

     A Control plane(CP) Plane (CP) is an execution environment that may have sev-
     eral components sub-components, which we refer to as CPCs.  Each CPC provides con-
     trol
     control for a different IP service being executed by a FE Forwarding
     Engine (FE) component.  This relationship means that there might be
     several CPCs on a physical CP CP, if it is controlling several IP services. ser-
     vices.  In essence, the cohesion between a CP component and a an FE
     component is the service abstrac-
     tion.

     In abstraction.

2.1.1.  Control Plane Components (CPCs)

     Control Plane Components encompass signalling protocols, with
     diversity ranging from dynamic routing protocols, such as OSPF
     [RFC2328], to tag distribution protocols, such as CR-LDP [RFC3036].
     Classical management protocols and activities also fall under this
     category.  These include SNMP [RFC1157], COPS [RFC2748], and pro-
     prietary CLI/GUI configuration mechanisms.

     The purpose of the diagram below we show a simple FE<->CP setup control plane is to provide an
     example of execution envi-
     ronment for the classical IPv4 service above-mentioned activities with an extension the ultimate goal
     being to do some
     basic QoS egress scheduling and how it fits in this described

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

     model.

                               Control Plane (CP)
                              .------------------------------------
                              |      /^^^^^\      /^^^^^\          |
                              |     |       |    | COPS  |-.       |
                              |     | ospfd |    |  PEP  | |       |
                              |     |       /     \_____/  |       |
                            /--------\_____/          |   |        |
                            | |        |             |   |         |
                            | |______________________|___|_________|
                            |           |            |   |
                           ******************************************
             Forwarding    ************* Netlink  layer ************
             Engine (FE)   *****************************************
              .-------------|-----------|------------|---|-----------
              |       IPv4 forwading    |               /            |
              |       FE Service       /               /             |
              |       Component       /               /              |
              |       ---------------/---------------/---------      |
              |       |             |               /         |      |
       packet |       |     --------|--        ----|-----     |      packet
       in     |       |     |  IPV4    |      | Egress   |    |      out -->--->              |------>|---->|Forwading |----->| QoS      |--->| ---->|---->
              |       |     |          |      | Scheduler|    |      |
              |       |     -----------        ----------     |      |
              |       |                                       |      |
              |        ---------------------------------------       |
              |                                                      |
              -------------------------------------------------------

2.1.1.  Control Plane Components (CPCs)

     Control plane components would encompass signalling protocols with
     diversity ranging from dynamic routing protocols such as OSPF
     [RFC2328] to tag distribution protocols such as CR-LDP [RFC3036].
     Classical Management protocols and activities also fall under this
     category. These include SNMP [RFC1157], COPS [RFC2748] or propri-
     etary CLI/GUI configuration mechanisms.

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

     The purpose of the control plane is to provide an execution envi-
     ronment for the above mentioned activities with the ultimate goal
     being to configure configure and manage the second NE component: Network Element (NE) com-
     ponent: the FE.  The result of the configuration would define defines the way
     that packets travesing traversing the FE are treated.

     In the above diagram, ospfd and COPS are distinct CPCs.

2.1.2.  Forwarding Engine Components (FECs)

     The FE is the entity of the NE that incoming packets (from the net-
     work into the NE) first encounter.

     The FE's service specific service-specific component massages the packet to provide
     it with a treatment to achieve a an IP service service, as defined by the con-
     trol plane components
     Control Plane Components for that IP service.  Different services
     will utilize different FEC. FECs.  Service modules maybe may be chained to
     achieve a more complex service (as shown in (refer to the diagram). Linux FE model,
     described later).  When built for providing a specific service, the
     FE service component will adhere to a Forwading Model.

     In the above diagram, the IPV4 FE component includes both the IPV4 forwarding model.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

2.1.2.1.  Linux IP Forwarding service module as well as the Egress Scheduling service
     module.  Another service might Engine Model

                         ____      +---------------+
                    +->-| FW |---> | TCP, UDP, ... |
                    |   +----+     +---------------+
                    |                   |
                    ^                   v
                    |                  _|_
                    +----<----+       | FW |
                              |       +----+
                              ^         |
                              |         Y
                            To host    From host
                             stack     stack
                              ^         |
                              |_____    |
 Ingress                            ^   Y
 device   ____    +-------+        +|---|--+   ____   +--------+ Egress
 ->----->| FW |-->|Ingress|-->---->| Forw- |->| FW |->| Egress | device
         +----+   |  TC   |        |  ard  |  +----+  |   TC   |-->
                  +-------+        +-------+          +--------+

     The figure above shows the Linux FE model per device.  The only
     mandatory part of the datapath is the Forwarding module, which is
     RFC 1812 conformant.  The different Firewall (FW), Ingress Traffic
     Control, and Egress Traffic Control building blocks are not manda-
     tory in the datapath and may add even be used to bypass the RFC 1812
     module.  These modules are shown as simple blocks in the datapath
     but, in fact, could be multiple cascaded, independent submodules
     within the indicated blocks.  More information can be found at
     [Netfilter] and [Diffserv].

     Packets arriving at the ingress device first pass through a policy forwarder between fire-
     wall module.  Packets may be dropped, munged, etc., by the firewall
     module.  The incoming packet, depending on set policy, may then be
     passed via an Ingress Traffic Control module.  Metering and polic-
     ing activities are contained within the IPV4 forwader Ingress TC module.  Packets
     may be dropped, depending on metering results and policing poli-
     cies, at this module.  Next, the packet is subjected to the QoS egress Scheduler.  A simpler classi-
     cal service would have constituted only
     non-optional module, the IPV4 forwarder. RFC 1812-conformant Forwarding module.
     The packet may be dropped if it is nonconformant (to the many RFCs
     complementing 1812 and 1122).  This module is a juncture point at
     which packets destined to the forwarding NE may be sent up to the
     host stack.

     Packets that are not for the NE may further traverse a policy rout-
     ing submodule (within the forwarding module), if so provisioned.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

     Another firewall module is walked next.  The firewall module can
     drop or munge/transform packets, depending on the configured sub-
     modules encountered and their policies.  If all goes well, the
     Egress TC module is accessed next.

     The Egress TC may drop packets for policing, scheduling, congestion
     control, or rate control reasons.  Egress queues exist at this
     point and any of the drops or delays may happen before or after the
     packet is queued.  All is dependent on configured module algorithms
     and policies.

2.1.3.  IP Services

     An IP Service service is the treatment of an IP packet within the NE.  This
     treatment is provided by a combination of both the CPC and FEC the FEC.

     The time span of the service is from the moment when the packet
     arrives at the NE to the moment that it departs.  In essence essence, an IP ser-
     vice
     service in this context is a Per-Hop Behavior.  A service control/sig-
     naling protocol/management-application (CP  CP components running run-
     ning on NEs defining define the end to end path) unifies end-to-end path control for a service by
     running control/signaling protocol/management-applications.  These
     distributed CPCs unify the end to end end-to-end view of the IP service.  As
     noted above, these CP components then define the behavior of the FE
     (and therefore the NE) to for a described packet.

     A simple example of an IP service is the classical IPv4 Forwarding.
     In this case, control components components, such as routing protocols(OSPF,

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

     RIP etc) protocols (OSPF,
     RIP, etc.) and proprietary CLI/GUI configurations configurations, modify the FE's
     forwarding tables in order to offer the simple service of forward-
     ing packets to the next hop.  Traditionally, NEs offering this sim-
     ple service are known as routers.  In the diagram below, we show a
     simple FE<->CP setup to provide an example of the classical IPv4
     service with an extension to do some basic QoS egress scheduling
     and illustrate how the setup fits in this described model.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

                               Control Plane (CP)
                              .------------------------------------
                              |    /^^^^^^\      /^^^^^^\         |
                              |   |        |    | COPS  |-\       |
                              |   | ospfd  |    |  PEP  |  \      |
                              |   \       /      \_____/    |     |
                            /------\_____/         |       /      |
                            | |        |           |     /        |
                            | |_________\__________|____|_________|
                            |           |          |    |
                           ******************************************
             Forwarding    ************* Netlink  layer ************
             Engine (FE)   *****************************************
              .-------------|-----------|----------|---|-------------
              |       IPv4 forwading    |              |             |
              |       FE Service       /               /             |
              |       Component       /               /              |
              |       ---------------/---------------/---------      |
              |       |             |               /         |      |
       packet |       |     --------|--        ----|-----     |   packet
       in     |       |     |  IPv4    |      | Egress   |    |    out
       -->--->|------>|---->|Forwading |----->| QoS      |--->| ---->|->
              |       |     |          |      | Scheduler|    |      |
              |       |     -----------        ----------     |      |
              |       |                                       |      |
              |        ---------------------------------------       |
              |                                                      |
              -------------------------------------------------------

     The above diagram illustrates ospfd, an OSPF protocol control dae-
     mon, and a COPS Policy Enforcement Point (PEP) as distinct CPCs.
     The IPv4 FE component includes the IPv4 Forwarding service module
     as well as the Egress Scheduling service module.  Another service
     might add a policy forwarder between the IPv4 forwarder and the QoS
     egress scheduler.  A simpler classical service would have consti-
     tuted only the IPv4 forwarder.

     Over the years years, it has become important to add aditional services
     to
     the routers to meet emerging requirements.  More complex services
     extending classical forwarding were have been added and standardized.
     These newer services might go beyond the layer 3 contents of the
     packet header.  However, the name "router", "router," although a misnomer, is
     still used to describe these NEs.  Services (which may look beyond
     the classical L3 service headers) here include firewalling, Qos QoS in Diffserv Diff-
     serv and RSVP, NATs, NAT, policy based routing routing, etc.  Newer control protocols pro-
     tocols or management activities are introduced with these new services. ser-
     vices.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

     One extreme definition of a IP service is something for which a
     service provider would be able to charge for. charge.

3.  Netlink Architecture

     Control of IP services service components control is defined by using templates.

     The FEC and CPC participate to deliver the IP service by communi-
     cating using these templates.  The FEC might continously get
     updates from the control plane component Control Plane Component on how to operate the ser-
     vice (example (e.g., for v4 forwarding or for V4 forwarding, route additions or deletions).

     The interaction between the FEC and the CPC, in the netlink Netlink con-
     text, would define defines a protocol.  Netlink provides the mechanism mechanisms for the CPC
     (residing in user space) and the FEC (residing in kernel space) to
     have their own protocol definition.  Kernel definition--kernel space and user space
     just mean different protection domains.  Therefore  Therefore, a wire protocol
     is needed to communicate.  The wire protocol would be is normally be provided
     by some privileged service that is able to copy between multiple
     protection domains.  We will refer to this service as the
     netlink Netlink
     service.  The Netlink service could can also be necapsulated to encapsulated in a
     different differ-
     ent transport layer layer, if the CPC executes on a different node than
     the FEC.  The FEC and CPC, using netlink Netlink mechanisms, may choose to
     define a reliable protocol between each other.  By default, however, netlink how-
     ever, Netlink provides an unreliable communication.

     Note that the FEC and CPC can both live in the same memory protec-
     tion domain and use the connect() system call to create a path to
     the peer and talk to each other.  We will not discuss this mecha-
     nism further other than to say that it is available as a mechanism.  Through out available.  Throughout
     this

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

     document document, we will refer interchangebly to the FEC to mean kernel- ker-
     nel space and the CPC to mean user-space. user space.  This denomination is not
     meant, however, to restrict the two components to these protection
     domains or to the same compute node.

     Note: Netlink allows participation in IP services by both service
     components.

3.1.  Netlink Logical model Model

     In the diagram below we show a simple FEC<->CPC logical relation-
     ship.  We use the example of IPV4 IPv4 forwarding FEC (NETLINK_ROUTE, which is

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

     discussed further below) as an example.

                               Control Plane (CP)
                              .------------------------------------
                              |    /^^^^^       /CPC-2    /^^^^^\      /^^^^^\          |
                              |   |       |     / CPC-2 \         |
                              |   | CPC-1 |     | COPS  |          |
                              |   | ospfd |     |  PEP  |          |
                              |          /      _____/           |
                              |    _____/           |             |
                              |        |             |             |
                           ****************************************|
                           ************* BROADCAST WIRE  ************
              FE---------- *****************************************.
              |       IPv4 forwading |    |            /           |             |
              |       FEC          |    |           |             |
              |       --------------/-----|-----------|--------       --------------/ ----|-----------|--------     |
              |       |            /      |           |       |     |
              |       |     .-------.  .-------.   .------.   |     |
              |       |     |ingress|     |Ingress|  | IPV4 IPv4  |   |Egress|   |     |
              |       |     |police |  |Forward|   | QoS  |   |     |
              |       |     |_______|  |_______|   |Sched |   |     |
              |       |                             ------    |     |
              |        ---------------------------------------      |
              |                                                     |
               -----------------------------------------------------

     Netlink logically models FECs and CPCs in the form of nodes inter-
     connected to each other via a broadcast wire.

     The wire is specific to a service.  The example above shows the
     broadcast wire belonging to the extended IPV4 IPv4 forwarding service.

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

     Nodes (CPCs or FECs as illustrated above) connect to the wire and
     register to receive specific mes-
     sages. messages.  CPCs may connect to multiple multi-
     ple wires if it helps them to con-
     trol control the service better.  All nodes(CPCs
     nodes (CPCs and FECs) dump packets on the broadcast wire.  Packets could
     can be discarded by the wire if mal-
     formed they are malformed or not specifically formated specifi-
     cally formatted for the wire.  Dropped packets are not seen by any
     of the nodes.  The netlink Netlink service MAY signal an error to the original
     sender if it detects an a malformatted netlink Netlink packet.

     Packets sent on the wire could can be broadcast, multicast multicast, or unicast.
     FECs or CPCs register for and pick specific messages of interest for processing pro-
     cessing or just monitoring purposes.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

     Appendices 1 and 2 have a high level overview of this interaction.

3.2.  The message format  Message Format

     There are three levels to a netlink Netlink message: The general netlink Netlink
     message header, the IP service specific template, and the IP service ser-
     vice specific data.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      |                   Netlink message header                      |
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      |                  IP Service Template                          |
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      |                  IP Service specific data in TLVs             |
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     The netlink Netlink message is used to communicate between the FEC and CPC
     for parametrization of the FECs, asynchoronous event notification
     of FEC events to the CPCs CPCs, and statistics querying/gathering (typi-
     cally by the a CPC).

     The Netlink message header is generic for all
     services services, whereas the
     IP Service Template header is specific to a service.  Each IP Service Ser-
     vice then carries parameterization
     data(CPC->FEC parametrization data (CPC->FEC direction) or
     response (FEC->CPC direction).  These parametrizations are in TLV
     (Type-Length-Value) format and are unique just to the service.

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

3.3.  Protocol Model

     This section expands on how netlink Netlink provides the mechanism for ser-
     vice oriented
     vice-oriented FEC and CPC interaction.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

3.3.1.  Service Addressing

     Access is provided by first connecting to the service on the FE.
     This
     The connection is done achieved by making a socket() system call to the
     PF_NETLINK domain.  Each FEC is identified by a protocol number.
     One may open either SOCK_RAW or SOCK_DGRAM type sockets sockets, although netlink doesnt
     Netlink does not distinguish between the two.  The socket connection connec-
     tion provides the basis for the FE<->CP addressing.

     Connecting to a service is followed (at any point during the life
     of the connection) by issuing either issuing a service specific service-specific command
     mostly for configuration purposes
     (from the CPC to the FEC) FEC, mostly for configuration purposes), issu-
     ing a statistics-collection command, or sub-
     scribing/unsubscribing subscribing/unsubscribing
     to service(s') events, or statistics collec-
     tion.

3.3.1.1.  Sample Service Hierachy

     In service events.  Closing the diagram below we show a simple IP service, foo, and socket terminates the
     interaction it has between CP transaction.
     Refer to Appendices 1 and FE components 2 for examples.

3.3.2.  Netlink Message Header

     Netlink messages consist of a byte stream with one or multiple
     Netlink headers and an associated payload.  If the ser-
     vice(labels 1-3).

     We introduce the diagram below to demonstrate CP<->FE addressing.
     In this section we illustrate only the addressing semantics. In
     section 4, the diagram payload is referenced again too
     big to define the protocol
     interaction between service foo's CPC and FEC (labels 4-10).

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

       CP
      [--------------------------------------------------------.
      |   .-----.                                              |
      |  |                        . -------.                  |
      |  |  CLI   |               /                           |
      |  |        |              | CP protocol                |
      |         /->> -.         |  component  | <-.           |
      |    __ _/      |         |   For       |   |           |
      |                |         | IP service  |   ^           |
      |                Y         |    foo      |   |           |
      |                |          ___________/    ^           |
      |                Y   1,4,6,8,9 /  ^ 2,5,10   | 3,7       |
       --------------- Y------------/---|----------|-----------
                       |           ^    |          ^
                     **|***********|****|**********|**********
                     ************* Netlink  layer ************
                     **|***********|****|**********|**********
             FE        |           |    ^          ^
             .-------- Y-----------Y----|--------- |----.
             |                    |              /     |
             |                    Y            /       |
             |          . --------^-------.  /         |
             |          |FE component/module|/          |
             |          |  for IP Service   |           |
      --->---|------>---|     foo           |----->-----|------>--
             |           -------------------            |
             |                                          |
             |                                          |
              ------------------------------------------

     The control plane protocol for IP service foo does the following to
     connect to its FE counterpart.  The steps below are also numbered
     above in the diagram.

1)   Connect to IP service foo through a socket connect. A typical con-
     nection would be via a call to: socket(AF_NETLINK, SOCK_RAW,
     NETLINK_FOO)

2)   Bind to listen to specific async events for service foo

3)   Bind to listen to specific async FE events

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

3.3.2.  Netlink message header

     Netlink messages consist of a byte stream with one or multiple
     Netlink headers and associated payload. If the payload is too big
     to fit into a single message it can be split over multiple netlink
     messages.  This is called a multipart message. For multipart mes-
     sages fit into a single message it, can be split over multiple
     Netlink messages, collectively called a multipart message.  For
     multipart messages, the first and all following headers have the
     NLM_F_MULTI
     netlink Netlink header flag set, except for the last header
     which has the
     netlink Netlink header type NLMSG_DONE.

     The netlink Netlink message header is shown below.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                    0               1               2             3
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                          Length                             |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |            Type              |           Flags              |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                      Sequence Number                        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                      Process PID ID (PID)                       |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   The fields in the header are:

jhs_hk_ak_ank                                draft-forces-netlink-02.txt                                draft-forces-Netlink-03.txt

          Length: 32 bits
          The length of the message in bytes bytes, including the header.

          Type: 16 bits
          This field describes the message content.
          It can be one of the standard message types:
               NLMSG_NOOP  message  Message is ignored ignored.
               NLMSG_ERROR the The message signals an error and the payload
                           contains a nlmsgerr structure.  This can be looked
                           at as a NACK and typically it is from FEC to CPC.
               NLMSG_DONE  message  Message terminates a multipart message message.

          Individual IP Services services specify more message types, for e.g.,
          NETLINK_ROUTE Service service specifies several types types, such as RTM_NEWLINK,
          RTM_DELLINK, RTM_GETLINK, RTM_NEWADDR, RTM_DELADDR, RTM_NEWROUTE,
          RTM_DELROUTE, etc.

          Flags: 16 bits
          The standard flag bits used in netlink Netlink are
                 NLM_F_REQUEST   Must be set on all request messages (typically
                                 from user space to kernel space)
                 NLM_F_MULTI     Indicates the message is part of a multipart
                                 message terminated by NLMSG_DONE
                 NLM_F_ACK       Request for an acknowledgment on success.
                                 Typical direction of request is from user
                                 space (CPC) to kernel space. space (FEC).
                 NLM_F_ECHO      Echo this request.  Typical direction of
                                 request is from user space (CPC) to kernel space.
                                 space (FEC).

          Additional flag bits for GET requests on config information in
          the FEC.
                 NLM_F_ROOT     Return the complete table instead of a
                                single entry.
                 NLM_F_MATCH    Return all entries matching criteria passed in
                                message content content.
                 NLM_F_ATOMIC   Return an atomic snapshot of the table being
                                referenced.  This may require special privileges
                                because it has the potential to interrupt
                                service in the FE for a longer time.

          Convenience macros for flag bits:
                 NLM_F_DUMP     This is NLM_F_ROOT or'ed with NLM_F_MATCH

          Additional flag bits for NEW requests
                 NLM_F_REPLACE   Replace existing matching config object with
                                 this request.
                 NLM_F_EXCL      Don't replace the config object if it already
                                 exists.

jhs_hk_ak_ank                                draft-forces-netlink-02.txt                                draft-forces-Netlink-03.txt

                                 exists.
                 NLM_F_CREATE    Create config object if it doesn't already
                                 exist.
                 NLM_F_APPEND    Add to the end of the object list.

          For those familiar with BSDish use of such operations in route
          sockets, the equivalent translations are:

                    - BSD ADD operation equates to NLM_F_CREATE or-ed
                      with NLM_F_EXCL
                    - BSD CHANGE operation equates to NLM_F_REPLACE
                    - BSD Check operation equates to NLM_F_EXCL
                    - BSD APPEND equivalent is actually mapped to
                      NLM_F_CREATE

          Sequence Number: 32 bits
          The sequence number of the message.

          Process PID: ID (PID): 32 bits
          The PID of the process sending the message.  The PID is used by the
          kernel to multiplex to the correct sockets.  A PID of zero is used
          when sending messages to user space from the kernel. netlink service
          fills in an appropiate value when zero.

3.3.2.1.  Mechanisms for creating protocols Creating Protocols

     One could create a reliable protocol between an FEC and a CPC by
     using the combination of sequence numbers, ACKs ACKs, and retransmit
     timers.  Both sequence numbers and ACKs are provided by netlink.
     Timers Netlink;
     timers are provided by Linux.

     One could create a heartbeat protocol between the FEC and CPC by
     using the ECHO flags and the NLMSG_NOOP message.

3.3.2.2.  The ACK netlink message Netlink Message

     This message is actually used to denote both an ACK and a NACK.
     Typically
     Typically, the direction is from kernel FEC to user space CPC (in response to an ACK
     request message).  However, user space the CPC should be able to send ACKs
     back to kernel space FEC when requested. This is  The semantics for this are IP service service-
     specific.

jhs_hk_ak_ank                                draft-forces-netlink-02.txt                                draft-forces-Netlink-03.txt

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       0               1               2               3
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                       Netlink message header                  |
      |                       type = NLMSG_ERROR                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                          error                          Error code                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                       OLD Netlink message header              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     Error code: integer (typically 32 bits)

     Error

     An error code of zero indicates that the message is an ACK
     response.  An ACK response message contains the original netlink Netlink
     message
     header that header, which can be used to compare against (sent sequence numbers
     numbers, etc).

     A non-zero error code message is equivalent to a Negative ACK
     (NACK).  In such a situation, the netlink Netlink data that was sent down
     to the kernel is returned appended to the original netlink Netlink message
     header.  An error code printable via the perror() is also set (not
     in the message header, rather in the executing environment state vari-
     able).
     variable).

3.3.3.  FE System services' templates Services' Templates

     These are services that are offered by the system for general use
     by other services.  They include the ability to configure, gather
     statistics and listen to changes in shared resources.  IP address
     management, link events etc events, etc. fit here.  We separate them into create this section here for
     these services for logical purposes separation, despite the fact that they
     are accessed via the NETLINK_ROUTE FEC. The reason that they exist
     within NETLINK_ROUTE is due to historical cruft based on cruft: the fact
     that BSD 4.4 rather narrowly focussed Route
     Sockets implemented them as part of the IPV4 IPv4 forwarding sockets.

3.3.3.1.

Network Interface Service Module

jhs_hk_ak_ank                                draft-forces-netlink-02.txt                                draft-forces-Netlink-03.txt

     This service provides the ability to create, remove remove, or get informa-
     tion infor-
     mation about a specific network interface. The network interface
     could
     can be either physical or virtual and is network protocol inde-
     pendent (example indepen-
     dent (e.g., an x.25 interface can be defined via this mes-
     sage). message).
     The Interface service message template is shown below.

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       0               1               2               3
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   Family    |   Padding   Reserved  |          Device Type              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Interface Index                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      Device Flags                             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      Change Mask                              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

          Family: 8 bits
          This is always set to AF_UNSPEC AF_UNSPEC.

          Device Type: 16 bits
          This defines the type of the link.  The link could be
     ethernet, Ethernet, a tunnel
          tunnel, etc. Although we  We are interested only in IPV4, IPv4, although the link type
          is protocol independent. L3 protocol-independent.

          Interface Index: uniquely 32 bits
          Uniquely identifies interface.

          Device Flags: 32 bits

                 IFF_UP            Interface is running. administrativel up.
                 IFF_BROADCAST     Valid broadcast address set.
                 IFF_DEBUG         Internal debugging flag.
                 IFF_LOOPBACK      Interface is a loopback interface.
                 IFF_POINTOPOINT   Interface is a point-to-point link.
                 IFF_RUNNING       Resources allocated.       Interface is operationally up.
                 IFF_NOARP         No arp ARP protocol needed for this interface.
                 IFF_PROMISC       Interface is in promiscuous mode.
                 IFF_NOTRAILERS    Avoid use of trailers.
                 IFF_ALLMULTI      Receive all multicast packets.
                 IFF_MASTER        Master of a load balancing bundle.
                 IFF_SLAVE         Slave of a load balancing bundle.
                 IFF_MULTICAST     Supports multicast
                 IFF_PORTSEL       Is able to select media type via ifmap.
                 IFF_AUTOMEDIA     Auto media selection active.
                 IFF_DYNAMIC       Interface Address is not permanent.

jhs_hk_ak_ank                                draft-forces-netlink-02.txt was dynamically created.

           Change Mask: 32 bits
           Reserved for future use.  Must be set to 0xFFFFFFFF.

           Applicable attributes:
             attribute            description
             .......................................................
                  Attribute            Description
                  ...........................................................
                  IFLA_UNSPEC          -                  unspecified.          Unspecified.
                  IFLA_ADDRESS         hardware         Hardware address interface L2 address address.
                  IFLA_BROADCAST       hardware       Hardware address L2 broadcast
                                       address.
                  IFLA_IFNAME          ascii          ASCII string device name.
                  IFLA_MTU             MTU of the device.
                  IFLA_LINK            Link type.            ifindex of link to which this device
                                       is bound.
                  IFLA_QDISC           ascii           ASCII string defining Queueing egress root
                                       queueing discipline.
                  IFLA_STATS           Interface Statistics. statistics.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

          Netlink message types specific to this service:
          RTM_NEWLINK, RTM_DELLINK, and RTM_GETLINK

3.3.3.2.  IP Address Service module Module

This service provides the ability to add, remove remove, or receive information
about an IP address associated with an interface.  The Address address provi-
sioning service message template is shown below.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
        0             1              2             3
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |   Family    |     Length    |     Flags     |    Scope      |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                     Interface Index                         |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 Family: 8 bits
 Address Family: AF_INET for IPV4 or IPv4; and AF_INET6 for IPV6. IPV4.

 Length: the 8 bits
 The length of the address mask mask.

 Flags: 8 bits
 IFA_F_SECONDARY for  For secondary address (alias interface), interface).
 IFA_F_PERMANENT for  For a permanent address set by the user as
        opposed to dynamic addresses.
        other flags include: user.
                  When this is not set, it means the address
                  was dynamically created (e.g., by stateless
                  autoconfiguration).
 IFA_F_DEPRECATED which defines Defines deprecated (IPV6) address (IPV4) address.
 IFA_F_TENTATIVE which defines  Defines tentative (IPV6) (IPV4) address (duplicate
                  address detection is still in progress).

 Scope: the 8 bits
 The address scope

jhs_hk_ak_ank                                draft-forces-netlink-02.txt in which the address stays valid.
        SCOPE_UNIVERSE: Global scope.
        SCOPE_SITE (IPv6 only): Only valid within this site.
        SCOPE_LINK: Valid only on this device.
        SCOPE_HOST: Valid only on this host.

     Applicable attributes:
             attribute            description
             .......................................................

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

             Attribute             Description
             .........................................................
                   IFA_UNSPEC      -                      unspecified.      Unspecified.
                   IFA_ADDRESS     raw     Raw protocol address of interface interface.
                   IFA_LOCAL       raw       Raw protocol local address address.
                   IFA_LABEL       ascii       ASCII string name of the interface
                                   reffered to. interface.
                   IFA_BROADCAST   raw   Raw protocol broadcast address.
                   IFA_ANYCAST     raw     Raw protocol anycast address address.
                   IFA_CACHEINFO   cacheinfo   Cache address information.

     Define cacheinfo here -- JHS

     netlink

     Netlink messages specific to this service: RTM_NEWADDR,
     RTM_DELADDR, RTM_GETADDR and RTM_GETADDR.

4.  Sample Protocol for The foo IP service

     Our proverbial IP service "foo" is used again to demonstrate how
     one can deploy a simple IP service control using netlink.

     These steps are continued from the "Sample Service Hierachy" sec-
     tion.

4)   query for current config of FE component

5)   receive response to 4) via channel on 3)

6)   query for current state of IP service foo

7)   receive response to 6) via channel on 2)

9)   register the protocol specific packets you would like the FE to
     forward to you

10)  send specific service foo commands and receive responses for them
     if needed

4.1.  Interacting with other IP services

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

     The last diagram shows another control component configuring the
     same service. In this case, it is a proprietary Command Line Inter-
     face.  The CLI (may or ) may not be using the netlink protocol to
     communicate to the foo component.  If the CLI should issue commands
     that will affect the policy of the FEC for service "foo" then, then
     the "foo" CPC is notified. It could then make algorithmic decisions
     based on this input (example if a policy that foo installed was
     deleted, there might be need to propagate this to all the peers of
     service "foo").

5.  Currently Defined netlink Netlink IP services Services

     Although there are many other IP services defined which that are using
     netlink,
     Netlink, as mentioned earlier, we will talk only mention about a handful of
     those integrated into the kernel
     today (kernel version 2.4.6). 2.4.6.  These are:

          NETLINK_ROUTE,NETLINK_FIREWALL,NETLINK_ARPD,NETLINK_ROUTE6,
          NETLINK_IP6_FW

5.1.

          NETLINK_ROUTE, NETLINK_FIREWALL, and NETLINK_ARPD.

4.1.  IP Service NETLINK_ROUTE

     This service allows CPCs to modify the IPv4 routing table in the
     Forwarding Engine.  It can also be used by CPCs to receive routing
     updates
     updates, as well as collecting to collect statistics.

5.1.1.

4.1.1.  Network Route Service Module

     This service provides the ability to create, remove or receive informa-
tion
     information about a network route.  The service message template is
     shown below.

jhs_hk_ak_ank                                draft-forces-netlink-02.txt                                draft-forces-Netlink-03.txt

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                      0               1               2             3
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   Family    |  Src length   |  Dest length  |     TOS       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Table ID   |   Protocol    |     Scope     |     Type      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                          Flags                              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Family: 8 bits
      Address family of route. Family: AF_INET for IPV4 IPv4; and AF_INET6 for
     IPV6. IPV4.

      Src length: prefix 8 bits
      Prefix length of source IP address.

      Dest length: 8 bits
      Prefix length of destination IP address address.

      TOS: the 8 bit tos bits
      The 8-bit TOS (should be deprecated to make room for DSCP) DSCP).

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

      Table ID: 8 bits
      Table identifier. Upto  Up to 255 route tables are supported.
                    RT_TABLE_UNSPEC    an    An unspecified routing table table.
                    RT_TABLE_DEFAULT   the   The default table table.
                    RT_TABLE_MAIN      the      The main table table.
                    RT_TABLE_LOCAL     the     The local table table.

                    The user may assign arbitary values between
                   RT_TABLE_UNSPEC
                    RT_TABLE_UNSPEC(0) and RT_TABLE_DEFAULT. RT_TABLE_DEFAULT(253).

      Protocol: identifies 8 bits
      Identifies what/who added the route. Described further
     below.
                   protocol
                    Protocol          Route origin.
                    ..............................................
                    RTPROT_UNSPEC     unknown     Unknown.
                    RTPROT_REDIRECT   by   By an ICMP  redirect
                                     (currently unused) redirect.
                    RTPROT_KERNEL     by     By the kernel kernel.
                    RTPROT_BOOT       during boot       During bootup.
                    RTPROT_STATIC     by     By the administrator administrator.

      Values larger than  RTPROT_STATIC RTPROT_STATIC(4) are not interpreted by the ker-
   nel,
      kernel, they are just for user information.  They may be used to
      tag the source of a routing information or to distingush between
      multiple routing daemons.  See <linux/rtnetlink.h> for the
      routing daemon identifiers  which that are already assigned.

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

      Scope: 8 bits
      Route scope (distance (valid distance to destination).
                    RT_SCOPE_UNIVERSE   global route   Global route.
                    RT_SCOPE_SITE       interior       Interior route in the
                                        local autonomous system system.
                    RT_SCOPE_LINK       route       Route on this link link.
                    RT_SCOPE_HOST       route       Route on the local host host.
                    RT_SCOPE_NOWHERE    destination doesn't exist    Destination does not exist.

      The values between RT_SCOPE_UNIVERSE RT_SCOPE_UNIVERSE(0) and RT_SCOPE_SITE RT_SCOPE_SITE(200)
      are avail-
     able available to the user.

      Type: 8 bits
      The type of route.

                    Route type         description
                   -------------------------------------------------        Description
                    ----------------------------------------------------
                    RTN_UNSPEC        unknown route        Unknown route.
                    RTN_UNICAST       a       A gateway or direct route route.
                    RTN_LOCAL         a         A local interface route route.
                    RTN_BROADCAST     a     A local broadcast route

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

                                      (sent as a broadcast) broadcast).
                    RTN_ANYCAST       a local broadcast route
                                     (sent as a  unicast)       An anycast route.
                    RTN_MULTICAST     a     A multicast route route.
                    RTN_BLACKHOLE     a     A silent packet dropping route route.
                    RTN_UNREACHABLE   an   An unreachable destination destination.
                                      Packets dropped and host
                                      unreachable ICMPs are sent to the
                                      originator.
                    RTN_PROHIBIT      a      A packet rejection route route.  Packets
                                      are dropped and communication
                                      prohibited ICMPs are sent to the
                                      originator.
                    RTN_THROW         When used with policy routing,
                                      continue routing lookup in another
                                     table
                                      table.  Under normal routing,
                                      packets are dropped and net
                                      unreachable ICMPs are sent to the
                                      originator.
                    RTN_NAT           a           A network address translation rule
                                      rule.
                    RTN_XRESOLVE      refer      Refer to an external resolver (not
                                     implemented)
                                      implemented).

      Flags: further 32 bits
      Further qualify the route.
                    RTM_F_NOTIFY     if     If the route changes, notify the
                                    user via rtnetlink
                                     user.
                    RTM_F_CLONED     route     Route is cloned from another route route.
                    RTM_F_EQUALIZE   a multicast equalizer (not yet
                                    implemented)   Allow randomization of next hop
                                     path in multi-path routing
                                     (currently not implemented).

      Attributes applicable to this service:

jhs_hk_ak_ank                                draft-forces-netlink-02.txt
                    Attribute       description
                   -----------------------------------------------       Description
                    ---------------------------------------------------
                    RTA_UNSPEC      ignored.      Ignored.
                    RTA_DST         protocol         Protocol address for route
                                    destination address.
                    RTA_SRC         protocol         Protocol address for route source
                                    address.
                    RTA_IIF         Input interface index.
                    RTA_OIF         Output interface index.
                    RTA_GATEWAY     protocol     Protocol address for the gateway of
                                    the route
                    RTA_PRIORITY    Priority of route.
                    RTA_PREFSRC     Preferred source address in cases

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

                                    where more than one source address
                                    could be used.
                    RTA_METRICS     Route metric metrics attributed to route
                                    and associated protocols (e.g.,
                                    RTT, initial TCP window, etc.).
                    RTA_MULTIPATH   Multipath route next hop's
                                    attributes.
                    RTA_PROTOINFO   Firewall based policy routing
                                    attribute.
                    RTA_FLOW        Route realm.
                    RTA_CACHEINFO

     additional netlink   Cached route information.

     Additional Netlink message types applicable to this service:
     RTM_NEWROUTE, RTM_DELROUTE, and RTM_GETROUTE

5.1.2.

4.1.2.  Neighbour Setup Service Module

     This service provides the ability to add, remove remove, or receive infor-
     mation about a neighbour table entry (e.g. (e.g., an ARP entry). entry or an IPv4
     neighbour solicitation, etc.).  The ser-
     vice service message template is
     shown below.

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      0               1               2               3
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   Family    |    Padding    Reserved1  |           Padding           Reserved2           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Interface Index                         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           State             |     Flags     |     Type      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

jhs_hk_ak_ank                                draft-forces-netlink-02.txt                                draft-forces-Netlink-03.txt

      Family: 8 bits
      Address Family Family: AF_INET for IPv4; and AF_INET6 for IPV4.

      Interface Index: 32 bits
      The unique interface index index.

      State: is a 16 bits
      A bitmask of the following states:
                    NUD_INCOMPLETE   a currently resolving cache entry   Still attempting to resolve.
                    NUD_REACHABLE    a    A confirmed working cache entry
                    NUD_STALE        an expired cache entry entry.
                    NUD_DELAY        an entry        Neighbour no longer reachable.
                                     Traffic sent, waiting for a timer
                                     confirmation.
                    NUD_PROBE        a        A cache entry that is currently
                                    reprobed
                                     being re-solicited.
                    NUD_FAILED       an       An invalid cache entry entry.
                    NUD_NOARP        a        A device with no destination cache which does not do neighbor
                                     discovery (ARP).
                    NUD_PERMANENT    a    A static entry entry.

      Flags: one of: 8 bits
                    NTF_PROXY    a        A proxy arp entry ARP entry.
                    NTF_ROUTER   an       An IPv6 router router.

      Attributes applicable to this service:
                    Attributes              description      Description
                    ------------------------------------
                    NDA_UNSPEC      unknown type      Unknown type.
                    NDA_DST         a         A neighbour cache network network.
                                    layer destination address
                    NDA_LLADDR      a      A neighbour cache link layer
                                   address
                                    address.
                    NDA_CACHEINFO   cache   Cache statistics.

     Describe the NDA_CACHEINFO nda_cacheinfo header later --JHS

     additional netlink

     Additional Netlink message types applicable to this service:
     RTM_NEWNEIGH, RTM_DELNEIGH, RTM_GETNEIGH

5.1.3. and RTM_GETNEIGH.

4.1.3.  Traffic Control Service

     This service provides the ability to provision, query or listen to
     events under the auspicies of traffic control.  These include Queueing
disciplines
     queueing disciplines, (schedulers and queue treatment algorithms eg Priority based

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

     algorithms--e.g., priority-based scheduler or the RED algorithm)
     and classifiers.  Linux Traffic Control Ser-
vice Service is very flexible
     and allows for hierachical cascading of the dif-
ferent different blocks for
     traffic resource sharing.

            ++    ++                 +-----+   +-------+   ++     ++ .++
            || .  ||     +------+    |     |-->| Qdisc |-->||     ||  ||
            ||    ||---->|Filter|--->|Class|   +-------+   ||-+   ||  ||
            ||    ||  |  +------+    |     +---------------+| |   ||  ||
            || .  ||  |              +----------------------+ |   || .||
            || .  ||  |  +------+                             |   ||  ||
            ||    ||  +->|Filter|-_  +-----+   +-------+   ++ |   || .||
            || -->||  |  +------+  ->|     |-->| Qdisc |-->|| |   ||->||
            || .  ||  |              |Class|   +-------+   ||-+-->|| .||
     ->dev->||    ||  |  +------+ _->|     +---------------+|     ||  ||
            ||    ||  +->|Filter|-   +----------------------+     || .||
            ||    ||     +------+                                 || .||
            || .  |+----------------------------------------------+|  ||
            ||    |          Parent Queuing discipline             | .||
            || .  +------------------------------------------------+ .||
            || . . .. . . .. . .                 . .. .. .. .      .. ||
            |+--------------------------------------------------------+|
            |                 Parent Queuing discipline                |
            |                  (attached to egress device)             |
            +----------------------------------------------------------+

     The above diagram shows an example of the Egress TC block.  We try
     to be very brief here.  For more information, please refer to
     [Diffserv].  A packet first goes through a filter that is used to
     identify a class to which the packet may belong.  A class is essen-
     tially a terminal queueing discipline and has a queue associated
     with it.  The queue may be subject to a simple algorithm, like
     FIFO, or a more complex one, like RED or a token bucket.  The out-
     ermost queueing discipline, which is refered to as the parent is
     typically associated with a scheduler.  Within this scheduler hier-
     archy, however, may be other scheduling algorithms, making the
     Linux Egress TC very flexible.

     The service message template which that makes this possible is shown
     below.  This template is used in both the ingress and the egress
     queueing disciplines (refer to the egress traffic control model in
     the FE model section).  Each of the specific component components of the
     model has unique attributes which that describe it best.  The common

jhs_hk_ak_ank                                draft-forces-netlink-02.txt
     attributes as well which are described below.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      0               1               2               3
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   Family    |    Padding  Reserved1    |           Padding         Reserved2             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Interface Index                         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      Qdisc handle                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Parent Qdisc                            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                        TCM Info                             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

      Family: 8 bits
      Address Family Family: AF_INET for IPv4; and AF_INET6 for IPV4.

      Interface Index: 32 bits
      The unique interface index index.

      Qdisc handle: unique 32 bits
      Unique identifier for instance of queueing discipline.
 Typically  Typically,
      this is split into major:minor of 16 bits each.  The major number
      would also be the major number of the parent of this instance.

      Parent Qdisc:  This is used 32 bits
      Used in hierarchical layering of queueing disciplines.  If this
      value and the Qdisc handle are the same and equal to TC_H_ROOT TC_H_ROOT,
      then the defined qdisc is the top most layer known as the root
      qdisc.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

      TCM Info: This is set 32 bits
      Set by the FE to 1 typically typically, except when the qdisc Qdisc instance is in
      use, in which case it is set to imply a reference count.

 Attributes  From the
      CPC towards the direction of the FEC, this is typically set to 0
      except when used in the context of filters.  In that case, this
      32-bit field is split into a 16-bit priority field and 16-bit
      protocol field.  The protocol is defined in kernel source
      <include/linux/if_ether.h>, however, the most commonly used one
      is ETH_P_IP (the IP protocol).

      The priority is used for conflict resolution when filters
      intersect in their expressions.

      Generic attributes applicable to this service:

                   Attribute        description        Description
                   ------------------------------------
                   TCA_KIND         canonical         Canonical name of FE component component.
                   TCA_STATS        generic        Generic usage statistics of FEC
                   TCA_RATE         rate estimator being attached to
                                    FEC.  Takes snapshots of stats to
                                    compute rate rate.
                   TCA_XSTATS       specific       Specific statistics of FEC FEC.
                   TCA_OPTIONS      nested FEC-specific attributes

[should we define all      Nested FEC-specific attributes? Seems like a lot attributes.

     Appendix 3 has an example of work
-- jhs]

[We still need configuring an FE component for a FIFO
     Qdisc.

     Additional Netlink message types applicable to talk about classes and filters; later -- jhs]

5.2. this service:
     RTM_NEWQDISC, RTM_DELQDISC, RTM_GETQDISC, RTM_NEWTCLASS, RTM_DELT-
     CLASS, RTM_GETTCLASS, RTM_NEWTFILTER, RTM_DELTFILTER, and RTM_GET-
     TFILTER.

4.2.  IP Service NETLINK_FIREWALL

     This service allows CPCs to receive receive, manipulate, and re-inject
     packets sent by via the IPv4 fire-
     wall firewall service modules in the FE.  A fire-
     wall rule is first inserted to activate packet redirection.  The
     CPC informs the FEC whether it would like to receive just the meta-
     data on the packet or the actual data and, if the metadata is
     desired, what is the maximum data length to be redirected.  The
     redirected packets are still stored in the FEC, waiting a verdict

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

     from the CPC.  The verdict could constitute a simple accept or drop
     decision of the packet, in which case the verdict is imposed on the
     packet still sitting on the FEC.  The verdict may also include a
     modified packet to be sent on as a replacement.

     Two types of messages exist that can be sent from CPC to FEC.
     These are: Mode messages and Verdict messages.  Mode messages are
     sent immediately to the FEC to describe what the CPC would like to
     receive.  Verdict messages are sent to the FEC after a decision has
     been made on the fate of a received packet.  The formats are
     described below.

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

      The Verdict mode message format is as follows described first.

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      0             1               2               3
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                         Value                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                       Packet ID   Mode    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+    Reserved1  |                       Data Length           Reserved2             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                       Payload ...                         Range                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     A ipq_packet_msg

      Mode: 8 bits
      Control information on the packet type is to be sent from the FEC to the CPC.  The
     format is described below ==> We need
      different types are:

             IPQ_COPY_META   Copy only packet metadata to complete this later

5.3.  IP Service NETLINK_ARPD

     This service is used by CPCs for managing the ARP table in FE.

5.4.  IP Service NETLINK_ROUTE6

     This service allows CPCs CPC.
             IPQ_COPY_PACKET Copy packet metadata and packet payloads
                             to modify the IPv6 routing table in CPC.

      Range: 32 bits
      If IPQ_COPY_PACKET, this defines the
     FE.  It can also be used by CPCs maximum length to receive routing updates. copy.

jhs_hk_ak_ank                                draft-forces-netlink-02.txt                                draft-forces-Netlink-03.txt

      A packet and associated metadata received from user space looks
      as follows.

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                    0               1               2               3
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPv6 dst addr                       Packet ID                             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPv6 dst addr                          Mark                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPv6 dst addr                       timestamp_m                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPv6 dst addr                       timestamp_u                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPv6 src addr                          hook                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPv6 src addr                       indev_name                            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPv6 src addr                       outdev_name                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPv6 src addr                          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+           hw_protocol       |                      IPv6 gw addr        hw_type                |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPv6 gw addr         hw_addrlen          |           Reserved            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPv6 gw addr                       hw_addr                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      IPv6 gw addr                       data_len                              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                          Type                      Payload . . .                          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           dst length        |           src length          |

      Packet ID: 32 bits
      The unique packet identifier as passed to the CPC by the FEC.

      Mark: 32 bits
      The internal metadata value set to describe the rule in which
      the packet was picked.

      timestamp_m: 32 bits
      Packet arrival time (seconds)

      timestamp_u: 32 bits
      Packet arrival time (useconds in addition to the seconds in
      timestamp_m)

      hook: 32 bits

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

      The firewall module from which the packet was picked.

      indev_name: 128 bits
      ASCII name of incoming interface.

      outdev_name: 128 bits
      ASCII name of outgoing interface.

      hw_protocol: 16 bits
      Hardware protocol, in network order.

      hw_type: 16 bits
      Hardware type.

      hw_addrlen: 8 bits
      Hardware address length.

      hw_addr: 64 bits
      Hardware address.

      data_len: 32 bits
      Length of packet data.

      Payload: size defined by data_len
      The payload of the packet received.

      The Verdict message format is as follows

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                   0               1               2               3
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                          Metric                         Value                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                          Info                       Packet ID                             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                          Flags                      Data Length                            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Interface Index                      Payload . . .                          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

5.5.

      Value: 32 bits
      This is the verdict to be imposed on the packet still sitting
      in the FEC. Verdicts could be:
              NF_ACCEPT   Accept the packet and let it continue its
                          traversal.
              NF_DROP     Drop the packet.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

      Packet ID: 32 bits
      The packet identifier as passed to the CPC by the FEC.

      Data Length: 32 bits
      The data length of the modified packet (in bytes). If you dont
      modify the packet just set it to 0.

      Payload:
      Size as defined by the Data Length field.

4.3.  IP Service NETLINK_IP6_FW NETLINK_ARPD

     This service allows is used by CPCs to receive packets that failed for managing the IPv6

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

     firewall checks by that module neighbor table in the
     FE.

6.  Security Considerations

     Netlink lives  The message format used between the FEC and CPC is described
     in the section on the Neighbour Setup Service Module.

     The CPC service is expected to participate in neighbor solicitation
     protocol(s).

     A neighbor message of type RTM_NEWNEIGH is sent towards the CPC by
     the FE to inform the CPC of changes that might have happened on
     that neighbour's entry (e.g., a neighbor being perceived as
     unreachable).

     RTM_GETNEIGH is used to solicit the CPC for information on a spe-
     cific neighbor.

5.  Security Considerations

     Netlink lives in a trusted environment of a single host separated
     by kernel and user space.  Linux capabilities ensures ensure that only
     someone with CAP_NET_ADMIN capability (typically (typically, the root user) is
     allowed to open sockets.

7.

6.  References

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

        [RFC1633]  R. Braden, D. Clark, and S. Shenker, "Integrated
     Services in the Internet Architecture: an Overview", RFC 1633,
     ISI, MIT, and PARC, June 1994.

        [RFC1812]  F. Baker, "Requirements for IP Version 4
     Routers", RFC 1812, June 1995.

        [RFC2475]  M. Carlson, W. Weiss, S. Blake, Z. Wang, D.
     Black, and E.  Davies, "An Architecture for Differentiated
     Services", RFC 2475, December 1998.

        [RFC2748] J. Boyle, R. Cohen, D. Durham, S. Herzog, R.
     Rajan, A. Sastry, "The COPS (Common Open Policy Service) Pro-
     tocol", RFC 2748, January 2000.

        [RFC2328] J. Moy, "OSPF Version 2", RFC 2328, April 1998.

        [RFC1157] J.D. Case, M. Fedor, M.L. Schoffstall, C. Davin,
     "Simple Network Management Protocol (SNMP)", RFC 1157, May
     1990.

        [RFC3036] L. Andersson, P. Doolan, N. Feldman, A. Fredette,
     B. Thomas "LDP Specification", RFC 3036, January 2001.

jhs_hk_ak_ank                                draft-forces-netlink-02.txt

        [stevens]

        [Stevens] G.R Wright, W. Richard Stevens.  "TCP/IP Illus-
     trated Volume 2, Chapter 20", June 1995

8.

        [Netfilter] http://netfilter.samba.org

        [Diffserv] http://diffserv.sourceforge.net

7.  Acknowledgements

1)   Andi Kleen Kleen, for man pages on netlink and rtnetlink.

2)   Alexey Kuznetsov is credited for extending netlink Netlink to the IP ser-
     vice delivery model.  The original netlink Netlink character device was

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

     written by Alan Cox.

9.

3)   Jeremy Ethridge for taking the role of someone who did not under-
     stand Netlink and reviewing the document to make sure that it made
     sense.

8.  Author's  Address:

   Jamal Hadi Salim
   Znyx Networks
   Ottawa, Ontario
   Canada
   hadi@znyx.com

   Hormuzd M Khosravi
   Intel
   2111 N.E. 25th Avenue JF3-206
   Hillsboro OR 97124-5961
   USA
   1 503 264 0334
   hormuzd.m.khosravi@intel.com

   Andi Kleen
   SuSE
   Stahlgruberring 28
   81829 Muenchen
   Germany

   Alexey Kuznetsov
   INR/Swsoft
   Moscow
   Russia

9.  Appendix 1: Sample Service Hierachy

     In the diagram below we show a simple IP service, foo, and the
     interaction it has between CP and FE components for the service
     (labels 1-3).

     The diagram is also used to demonstrate CP<->FE addressing.  In
     this section, we illustrate only the addressing semantics.  In
     Apendix 2, the diagram is referenced again to define the protocol
     interaction between service foo's CPC and FEC (labels 4-10).

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

       CP
      [--------------------------------------------------------.
      |   .-----.                                              |
      |  |                        . -------.                  |
      |  |  CLI   |               /                           |
      |  |        |              | CP protocol                |
      |         /->> -.         |  component  | <-.           |
      |    __ _/      |         |   For       |   |           |
      |                |         | IP service  |   ^           |
      |                Y         |    foo      |   |           |
      |                |          ___________/    ^           |
      |                Y   1,4,6,8,9 /  ^ 2,5,10   | 3,7       |
       --------------- Y------------/---|----------|-----------
                       |           ^    |          ^
                     **|***********|****|**********|**********
                     ************* Netlink  layer ************
                     **|***********|****|**********|**********
             FE        |           |    ^          ^
             .-------- Y-----------Y----|--------- |----.
             |                    |              /     |
             |                    Y            /       |
             |          . --------^-------.  /         |
             |          |FE component/module|/          |
             |          |  for IP Service   |           |
      --->---|------>---|     foo           |----->-----|------>--
             |           -------------------            |
             |                                          |
             |                                          |
              ------------------------------------------

     The control plane protocol for IP service foo does the following to
     connect to its FE counterpart.  The steps below are also numbered
     above in the diagram.

1)   Connect to the IP service foo through a socket connect.  A typical
     connection would be via a call to: socket(AF_NETLINK, SOCK_RAW,
     NETLINK_FOO).

2)   Bind to listen to specific asynchronous events for service foo.

3)   Bind to listen to specific asynchronous FE events.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

10.  Appendix 2: Sample Protocol for the Foo IP Service

     Our example IP service foo is used again to demonstrate how one can
     deploy a simple IP service control using Netlink.

     These steps are continued from Appendix 1 (hence the numbering).

4)   Query for current config of FE component.

5)   Receive response to (4) via channel on (3).

6)   Query for current state of IP service foo.

7)   Receive response to (6) via channel on (2).

9)   Register the protocol-specific packets you would like the FE to
     forward to you.

10)  Send service-specific foo commands and receive responses for them,
     if needed.

10.1.  Interacting with Other IP services

     The diagram in Appendix 1 shows another control component configur-
     ing the same service.  In this case, it is a proprietary Command
     Line Interface.  The CLI may or may not be using the Netlink proto-
     col to communicate to the foo component.  If the CLI issues com-
     mands that will affect the policy of the FEC for service foo then,
     then the foo CPC is notified.  It could then make algorithmic deci-
     sions based on this input.  For example, if an FE allowed another
     service to delete policies installed by a different service and a
     policy that foo installed was deleted by service bar, there might
     be a need to propagate this to all the peers of service foo.

11.  Appendix 3: Examples

     In this example, we show a simple configuration Netlink message
     sent from a TC CPC to an egress TC FIFO queue.  This queue algo-
     rithm is based on packet counting and drops packets when the limit
     exceeds 100 packets.  We assume that the queue is in a hierachical
     setup with a parent 100:0 and a classid of 100:1 and that it is to
     be installed on a device with an ifindex of 4.

jhs_hk_ak_ank                                draft-forces-Netlink-03.txt

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                      0               1               2             3
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                          Length (52)                        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Type (RTM_NEWQDISC)           | Flags (NLM_F_EXCL |         |
      |                               |NLM_F_CREATE | NLM_F_REQUEST)|
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      Sequence Number(arbitrary number)      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                        Process ID (0)                       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Family(AF_INET)|  Reserved1    |         Reserved1           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Interface Index  (4)                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      Qdisc handle  (0x1000001)              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Parent Qdisc   (0x1000000)              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                        TCM Info  (0)                        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |            Type (TCA_KIND)   |           Length(4)          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                        Value ("pfifo")                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |            Type (TCA_OPTIONS) |          Length(4)          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                        Value (limit=100)                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+