[Docs] [txt|pdf] [Tracker] [Email] [Nits]

Versions: 00

INTERNET-DRAFT                           Jeff Hilland
draft-hilland-rddp-verbs-00.txt            Hewlett-Packard Company
                                         Paul Culley
                                           Hewlett-Packard Company
                                         Jim Pinkerton
                                           Microsoft Corporation
                                         Renato Recio
                                           IBM Corporation

                                         Expires: October, 2003



   RDMA Protocol Verbs Specification

1  Status of this Memo

   This document is an Internet-Draft and is subject to all provisions
   of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other documents
   at any time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/1id-abstracts.html The list of Internet-Draft
   Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

2  Abstract

   This document describes an abstract interface to a RDMA enabled NIC
   (RNIC). This interface is implemented as a combination of the RNIC,
   its associated firmware, and host software. It provides access to
   the RNIC queuing and memory management resources, as well as the
   underlying networking layers.










   Hilland, et al.       Expires October 2003                  [Page 1]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Table of Contents

   1    Status of this Memo.........................................1
   2    Abstract....................................................1
   3    Introduction................................................7
   4    Glossary....................................................9
   4.1  Abbreviations..............................................19
   5    RNIC Interface.............................................22
   5.1  The RNIC...................................................23
   5.1.1  RNIC Resources...........................................23
   5.1.1.1   Expected Creation Sequence............................24
   5.1.1.2   Expected Destruction Sequence.........................25
   5.1.2  Opening an RNIC..........................................28
   5.1.3  Query RNIC...............................................28
   5.1.4  Closing an RNIC..........................................28
   5.2  Protection Domains.........................................28
   5.2.1  Allocating a PD..........................................29
   5.2.2  Deallocating a PD........................................30
   5.3  Completion Queues..........................................30
   5.3.1  Creating a Completion Queue..............................30
   5.3.2  Querying Completion Queue Attributes.....................31
   5.3.3  Modifying Completion Queue Attributes....................32
   5.3.4  Destroying a Completion Queue............................32
   6    Queue Pairs................................................33
   6.1  Queue Pair Resource Handling...............................34
   6.1.1  Creating a Queue Pair....................................34
   6.1.2  Querying Queue Pair Attributes...........................35
   6.1.3  Modifying Queue Pair Attributes..........................36
   6.1.4  Destroying a Queue Pair..................................39
   6.2  Queue Pair Resource States.................................41
   6.2.1  Idle State...............................................43
   6.2.1.1   Idle to Idle..........................................44
   6.2.1.2   Idle to RTS...........................................44
   6.2.1.3   Idle to Error.........................................46
   6.2.2  RTS (Ready to Send) State................................48
   6.2.2.1   RTS to RTS............................................48
   6.2.2.2   RTS to Closing........................................49
   6.2.2.3   RTS to Terminate......................................49
   6.2.2.4   RTS to Error..........................................50
   6.2.3  Terminate State..........................................53
   6.2.4  Error State..............................................56
   6.2.5  Closing State............................................58
   6.3  Shared Receive Queue.......................................62
   6.3.1  Creating a Shared Receive Queue..........................63
   6.3.2  Modifying a Shared Receive Queue.........................63
   6.3.3  Destroying a Shared Receive Queue........................63
   6.3.4  Associating an S-RQ with a QP............................64
   6.3.5  Shared Receive Queue Processing Model....................64
   6.3.6  S-RQ Error Semantics.....................................66
   6.3.7  S-RQ Resource Sizing.....................................66



   Hilland, et al.        Expires October 2003               [Page 2]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   6.3.8  S-RQ Limit Checking......................................67
   6.4  Stopping QP processing and Sending the Terminate Message...68
   6.5  Outstanding RDMA Read Resource Management..................71
   6.5.1  Example IRD/ORD Negotiation..............................74
   6.6  Connection Management......................................75
   6.6.1  Connection Initialization................................75
   6.6.1.1   Active Connection Initialization after LLP Startup....76
   6.6.1.2   Passive Connection Initialization after LLP Startup...78
   6.6.2  Connection Teardown......................................79
   6.6.2.1   Normal Close..........................................80
   6.6.2.2   ULP Initiated Termination.............................81
   6.6.2.3   ULP Initiated Abortive Teardown.......................82
   6.6.2.4   Remote Termination....................................83
   6.6.2.5   Local Termination, Local Abortive Teardown and Remote
   Abortive Teardown...............................................83
   7    Memory Management..........................................87
   7.1  Memory Management Overview.................................87
   7.2  Steering Tag (STag)........................................88
   7.2.1  STag of zero.............................................90
   7.2.2  Summary of Memory Region STag States.....................91
   7.3  Memory Registration........................................93
   7.3.1  Memory Regions...........................................94
   7.3.1.1   Memory Region Tagged Offset (TO)......................94
   7.3.2  Memory Region Creation and Registration..................94
   7.3.2.1   Allocate Non-Shared Memory Region STag................95
   7.3.2.2   RI-Register Non-Shared Memory Region..................95
   7.3.2.3   RI-Reregister Non-Shared Memory Region................96
   7.3.2.4   Register Shared Memory Region.........................98
   7.3.2.5   Fast-Register Non-Shared Memory Region................99
   7.4  Access to Registered Memory...............................100
   7.4.1  Local Access to Registered Memory.......................101
   7.4.2  Remote Access to Registered Memory......................101
   7.4.3  Multiple Registrations of Memory Regions................103
   7.5  Memory Access Control.....................................104
   7.5.1  Local Access Control....................................105
   7.5.2  Remote Access Control...................................106
   7.6  Addressing................................................106
   7.6.1  Addressing Registered Memory............................106
   7.6.1.1   Addressing with VA based TO..........................107
   7.6.1.2   Addressing with Zero Based TO........................108
   7.6.2  Physical Buffer Lists...................................109
   7.6.2.1   Page Lists...........................................109
   7.6.2.2   Block Lists..........................................110
   7.6.3  Error Checking of Local and Remote Accesses to MRs......110
   7.7  Querying Memory Regions...................................111
   7.8  Invalidating Memory Regions...............................111
   7.9  Deallocation of STag associated with a Memory Region......114
   7.10   Memory Windows..........................................115
   7.10.1  Allocating Memory Windows..............................115
   7.10.2  Binding Memory Windows to Memory Regions...............116



   Hilland, et al.        Expires October 2003               [Page 3]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   7.10.3  Querying Memory Windows................................120
   7.10.4  Invalidating or De-allocating Memory Windows...........120
   7.10.4.1  Invalidating or De-allocating Active Windows.........121
   7.10.5  Summary of Memory Window STag States...................121
   7.10.6  Error Checking during Memory Window Operations.........122
   7.10.6.1  Error Checking at Window Bind Time...................122
   7.10.6.2  Error Checking at Window Access Time.................123
   7.10.6.3  Error Checking at Window Invalidate Time.............123
   8    Work Requests and the WR Processing Model.................125
   8.1  Work Requests.............................................125
   8.1.1  Creating Work Requests..................................125
   8.1.2  Work Request Types......................................125
   8.1.2.1   Send/Receive.........................................125
   8.1.2.2   RDMA.................................................126
   8.1.2.3   Memory...............................................129
   8.1.3  Work Request Contents...................................130
   8.1.3.1   Signaled Completions.................................130
   8.1.3.2   Scatter/Gather List..................................131
   8.1.3.3   RDMA Data Source & Data Sink.........................132
   8.2  Work Request Processing Model.............................133
   8.2.1  Submitting Work Request to a Work Queue.................133
   8.2.2  Work Request Processing.................................134
   8.2.2.1   Memory Management Operation Ordering.................137
   8.2.2.2   Read Fence and Local Fence Indicators................140
   8.2.3  Completion Processing...................................143
   8.2.4  Returning Completed Work Requests.......................144
   8.2.5  Asynchronous Completion Notification....................145
   8.3  Error Handling............................................147
   8.3.1  Immediate Errors........................................148
   8.3.2  Work Completion Errors..................................148
   8.3.3  Asynchronous Errors.....................................150
   9    RNIC Verbs................................................157
   9.1  Consumer Accessibility....................................157
   9.2  RNIC Resource Management..................................158
   9.2.1  RNIC....................................................158
   9.2.1.1   Open RNIC............................................158
   9.2.1.2   Query RNIC...........................................159
   9.2.1.3   Close RNIC...........................................161
   9.2.2  Protection Domain.......................................162
   9.2.2.1   Allocate PD..........................................162
   9.2.2.2   Deallocate PD........................................163
   9.2.3  Completion Queue........................................163
   9.2.3.1   Create CQ............................................163
   9.2.3.2   Query CQ.............................................164
   9.2.3.3   Modify CQ............................................165
   9.2.3.4   Destroy CQ...........................................166
   9.2.4  Shared Receive Queue....................................167
   9.2.4.1   Create S-RQ..........................................167
   9.2.4.2   Query S-RQ...........................................168
   9.2.4.3   Modify S-RQ..........................................169



   Hilland, et al.        Expires October 2003               [Page 4]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   9.2.4.4   Destroy S-RQ.........................................170
   9.2.5  Queue Pair..............................................170
   9.2.5.1   Create QP............................................170
   9.2.5.2   Query QP.............................................174
   9.2.5.3   Modify QP............................................176
   9.2.5.4   Destroy QP...........................................178
   9.2.6  Memory Management.......................................179
   9.2.6.1   Allocate Non-Shared Memory Region STag...............179
   9.2.6.2   Register Non-Shared Memory Region (RI-Register)......180
   9.2.6.3   Query Memory Region..................................182
   9.2.6.4   Deallocate STag......................................183
   9.2.6.5   Reregister Non-Shared Memory Region (RI-Reregister)..184
   9.2.6.6   Register Shared Memory Region........................187
   9.2.6.7   Allocate Memory Window...............................188
   9.2.6.8   Query Memory Window..................................189
   9.3  Work Request Processing...................................190
   9.3.1  QP Operations...........................................190
   9.3.1.1   PostSQ...............................................190
   9.3.1.2   PostRQ...............................................197
   9.3.2  CQ Operations...........................................198
   9.3.2.1   Poll for Completion (Poll CQ)........................198
   9.3.2.2   Request Completion Notification......................200
   9.4  Event Handling............................................200
   9.4.1  Set Completion Event Handler............................200
   9.4.2  Set Asynchronous Event Handler..........................202
   9.5  Result Types..............................................203
   9.5.1  Immediate Status Codes..................................203
   9.5.1.1   RNIC Management Verb Status..........................204
   9.5.1.2   PD Management Verb Status............................204
   9.5.1.3   CQ Management Verb Status............................205
   9.5.1.4   S-RQ Management Verb Status..........................205
   9.5.1.5   QP Management Verb Status............................206
   9.5.1.6   Memory Management Verb Status........................207
   9.5.1.7   Post Verb Status.....................................208
   9.5.1.8   Event Management Verb Status.........................209
   9.5.2  Completion Status Codes.................................210
   9.5.3  Asynchronous Event Identifiers..........................212
   10   Security Considerations...................................217
   11   IANA Considerations.......................................218
   12   References................................................219
   12.1   Normative References....................................219
   12.2   Informative References..................................219
   13   Appendix..................................................220
   13.1   Connection Initialization at LLP Startup................220
   13.2   Graceful Receive Overflow Handling......................221
   14   AuthorÆs Addresses........................................223
   15   Acknowledgments...........................................224
   16   Full Copyright Statement..................................227





   Hilland, et al.        Expires October 2003               [Page 5]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Table of Figures

   Figure 1 - Architectural RNIC & RI Model..........................8
   Figure 2 - Resource Creation Dependency Diagram..................25
   Figure 3 - Resource Destruction Dependency Diagram...............27
   Figure 4 - Allowable QP Attribute Modifications..................37
   Figure 5 - Optional QP Attribute Modifications...................38
   Figure 6 - QP State Diagram......................................42
   Figure 7 - Idle State summary....................................47
   Figure 8 - RTS State summary.....................................52
   Figure 9 - Terminate State summary...............................55
   Figure 10 - Error State summary..................................57
   Figure 11 - Closing State summary................................61
   Figure 12- Terminate Control Field Values........................71
   Figure 13 - An example RDMA Read Resource negotiation............75
   Figure 14 - Connection Initialization after LLP Startup..........76
   Figure 15 - Normal Close on TCP..................................81
   Figure 16 - Abortive Teardown example on TCP.....................86
   Figure 17 - Memory Region and Window State Diagram...............92
   Figure 18 - Valid Combinations of MR Access Rights..............103
   Figure 19 - MR to MW Valid Binding Combinations.................117
   Figure 20 - Valid Combinations of MW & MR Access Rights.........119
   Figure 21 - Valid QP & STag Access Right Combinations...........128
   Figure 22 - Fencing on Prior Operations.........................142
   Figure 23 - Completion Errors with Resulting Terminate Codes....150
   Figure 24 - Affiliated Asynchronous Errors with Terminate Codes.155
   Figure 25 - Unaffiliated Asynchronous Errors with Terminate Code156
   Figure 26 - Memory Management Verbs.............................179
   Figure 27 - PostSQ Input Modifier Validity......................196
   Figure 28 - RNIC Management Verb Status.........................204
   Figure 29 - PD Management Verb Status...........................204
   Figure 30 - CQ Management Verb Status...........................205
   Figure 31 - S-RQ Management Verb Status.........................206
   Figure 32 - QP Management Verb Status...........................207
   Figure 33 - Memory Management Verb Status.......................208
   Figure 34 - Post Verb Status....................................209
   Figure 35 - Event Management Verb Status........................209
   Figure 36 - Completion Status Codes.............................212
   Figure 37 - Asynchronous Event Identifiers......................216
   Figure 39 - Connection Initialization at LLP Startup (using TCP)220













   Hilland, et al.        Expires October 2003               [Page 6]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

3  Introduction

   This document describes an abstract interface to an RDMA aware NIC
   (RNIC). The RNIC implements the RDMA Protocol [RDMAP][DDP] above a
   reliable transport, such as [MPA] over TCP. The Verbs provide the
   Consumer with a semantic definition of the RNIC Interface.

   RDMA provides Verbs Consumers the capability to control data
   placement, eliminate data copy operations, and significantly reduce
   communications overhead and latencies by allowing one Verbs Consumer
   to directly place information in another Verbs Consumer's memory,
   while preserving OS and memory protection semantics. Specification
   of syntactic definitions (API's, hardware registers) and
   implementation details (hardware, firmware, software tradeoffs) are
   beyond the scope of this specification.

   Section 5 of this document defines the semantics of the RNIC
   Interface (RI). This interface is implemented as a combination of
   the RNIC, its associated firmware, and host software. Section 6
   describes Queue Pairs, which represent the focus of interaction with
   the RNIC for work submission. Section 7 describes Memory Management
   and how the RNIC accesses buffers which contain data to be
   transferred. Section 8 describes Work Requests and the WR Processing
   Model, detailing the processing of the units of work from submission
   to completion. Section 9 describes the RNIC Verbs. The Verbs are an
   abstract description of the functionality of an RNIC Interface.
   Section 10 describes security issues associated with implementing an
   RDMA infrastructure.

   A concept frequently encountered in this specification is that of
   the Verbs Consumer, or simply, the Consumer. The precise meaning of
   the phrase varies, as a function of context, but it always means the
   executing entity employing the capabilities of the RNIC to
   accomplish some objective. In some instances the Verb Consumer may
   be an OS kernel thread, in others a non-privileged application, and
   in still others, some special, privileged process. Where the
   difference is important to the correct behavior of an
   implementation, it is defined explicitly.

   Specification of the API used by the Verbs Consumer to access the
   capabilities of the RI is outside of the scope of this
   specification.

   Figure 1 is a conceptual diagram that describes an architectural
   model which includes Privileged Mode consumers, Non-Privileged Mode
   consumers, RNIC components and the RI.







   Hilland, et al.        Expires October 2003               [Page 7]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003




             < Figure 1 did not convert properly from source >
             <  to be corrected in an upcoming version       >















                 Figure 1 - Architectural RNIC & RI Model






























   Hilland, et al.        Expires October 2003               [Page 8]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

4  Glossary

   Access Rights - The Local and Remote Memory Access Rights assigned
       to an STag. This includes Local Read, Local Write, Remote Read,
       Remote Write, Remote Access Flag, and Bind.

   Address List - A list of addresses that represent the physical pages
       or blocks referenced by the Physical Buffer List.

   Advertisement (Advertised, Advertise, Advertisements, Advertises) -
       The act of informing a Remote Peer that a Local Node's Buffer is
       available to it. A Node makes a buffer available for incoming
       RDMA Read Request Message or incoming RDMA Write Message access
       by informing its RDMA/DDP peer of the Tagged Buffer identifiers
       (STag, TO, and buffer length). This advertisement of Tagged
       Buffer information is not defined by RDMA/DDP and is left to the
       ULP. A typical method would be for the Local Peer to embed the
       Tagged Buffer's Steering Tag, TO, and length in a Send Message
       destined for the Remote Peer.

   Affiliated Asynchronous Event - This is an indication from the Verb
       layer to the Consumer that an event has occurred related to a
       specific identifiable RNIC Resource, such as a Completion Queue
       or Queue Pair.

   Affiliated Error - An error that can be directly related back to a
       specific RNIC Resource, such as a QP, S-RQ or CQ, but that
       cannot be returned through a Work Completion.

   Associated QP - The QP on the Remote Peer which is directly
       accessing the other end of the RDMA Stream.

   Asynchronous Error - This is an error that could not be reported
       through immediate or completion error-handling mechanisms at the
       local end. An asynchronous mechanism is necessary as a single
       point of error handling for errors which could not otherwise be
       reported through the normal mechanism since they are not
       associated directly with any single QP, S-RQ or CQ or the QP
       and/or CQ is in a state where an error cannot be reported.
       Asynchronous errors may be Unaffiliated or may be Affiliated
       with a specific QP, CQ or S-RQ.

   Base Tagged Offset (Base TO) - The offset assigned to the first byte
       of a Memory Region or a Memory Window.

   Bind, Binding, Bound - The act of associating an STag, TO, and
       Length within a previously registered Memory Region in order to
       define a Memory Window.





   Hilland, et al.        Expires October 2003               [Page 9]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Block List - A list of physical addresses describing a set of memory
       blocks, which specifies the block size, list of physical
       addresses, and offset to the start of the memory region of the
       first block. Each block has the same length and that length can
       be any value in the range supported by the RNIC. Each block may
       start at a byte granularity address. The starting address for
       the entire list may be an offset into the first block and the
       entire list may have any length.

   Complete (Completed, Completion, Completes) - When the Consumer can
       determine that a particular RDMA Operation has performed all
       functions specified for the RDMA Operation, including Placement
       and Delivery. This can be determined through a Work Completion
       for Signaled Work Requests. For Unsignaled Work Requests, this
       means that the Completion Rules have been met. Note that this is
       a superset of the [RDMAP] definition for RDMA Completion.

   Completion Error - A Processing Error reported through the
       Completion Queue.

   Completion Queue (CQ) - A sharable queue containing one or more
       entries which can contain Completion Queue Entries. A CQ is used
       to create a single point of completion notification for multiple
       Work Queues. The Work Queues associated with a Completion Queue
       may be from different QPs and of differing queue types (SQs or
       RQs).

   Completion Queue Entry (CQE) - The RNIC Interface internal
       representation of a Work Completion.

   Completion Status - The resultant status of a Work Request returned
       as part of a Work Completion.

   Consumer, Verbs Consumer - A software process that communicates
       using RDMA/DDP Verbs. The Consumer typically consists of an
       application program, or an operating system adaptation layer,
       which provides some OS specific API.

   Direct Data Placement Protocol (DDP) - A wire protocol that supports
       Direct Data Placement by associating explicit memory buffer
       placement information with the LLP payload units.

   Data Delivery (Delivery, Delivered, Delivers) - Delivery is defined
       as the process of informing the ULP or Consumer that a
       particular Message is available for use. This is specifically
       different from Data Placement, which may generally occur in any
       order, while the order of Data Delivery is strictly defined.

   Data Placement (Placement, Placed, Places) - A mechanism whereby ULP
       data contained within RDMA/DDP Segments may be put directly into



   Hilland, et al.        Expires October 2003              [Page 10]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       its final destination in memory without processing by the ULP.
       This may occur even when the RDMA/DDP Segments arrive out of
       order. Note that this differs from Data Delivery (see definition
       in this section). From the Verbs viewpoint, Data Placement is
       only confirmed upon Completion.

   Data Sink - The peer receiving a data payload. Note that the Data
       Sink can be required to both send and receive RDMA/DDP Messages
       to transfer a data payload.

   Data Source - The peer sending a data payload. Note that the Data
       Source can be required to both send and receive RDMA/DDP
       Messages to transfer a data payload.

   Event - An indication provided by the RDMAP Layer to the ULP to
       indicate a Completion or other condition requiring immediate
       attention.

   Fabric - The collection of links, switches, and routers that connect
       a set of Nodes with RDMA/DDP protocol implementations.

   First Byte Offset (FBO) - The offset into the first Physical Buffer
       of a Memory Region. The value of the FBO cannot exceed the size
       of the Physical Buffer Entry Size associated with the Memory
       Region.

   Handle - An opaque identifier used to reference an RNIC or an RNIC
       Resource. Whether this is an index, object or some other
       construct is outside the scope of this specification.

   Immediate Error -                   - An error discovered by the RNIC Interface (RI) and
       reported through the RI without affecting the RNIC.

   Inbound RDMA Read Queue Depth (IRD) - The maximum number of incoming
       outstanding RDMA Read Request Messages the RNICÆs QP can handle
       at the Data Source.

   Inbound RDMA Read Request Queue (IRRQ) - The RI internal resource
       which handles incoming RDMA Read Request Messages, queues them
       for processing them by the RI, and then generates the RDMA Read
       Response Messages. This corresponds to Queue Number 1 in [DDP].

   Invalidate STag (Invalidate, Invalidated, etc.) - A mechanism used
       to prevent the Remote Peer from reusing an Advertised STag,
       until the Local Peer transitions the STag to the Valid state.

   Invalidate Local STag - A Work Request that takes an STag which is
       valid within the local RI and performs an Invalidate STag
       operation.




   Hilland, et al.        Expires October 2003              [Page 11]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   iWARP - A suite of wire protocols comprised of [RDMAP] & [DDP]. The
       iWARP protocol suite may be layered above [MPA] and [TCP], or it
       may be layered over [SCTP] or other transport protocols.

   Local Access - The rights used to verify the RNIC's ability to
       access the Data Sink for incoming Untagged Messages, the Data
       Source for outgoing Untagged Messages and the Data Source for
       outgoing RDMA Write Messages.

   Local Fence - To block the current operation from executing until
       all prior local operations submitted on the same Work Queue have
       Completed.

   Local Peer - The RDMA/DDP protocol implementation on the local end
       of the connection. Used to refer to the local entity when
       describing a protocol exchange or other interaction between two
       Nodes.

   Lower Layer Protocol (LLP) - The protocol layer beneath the protocol
       layer currently being referenced. For example, for DDP the LLP
       is SCTP, MPA, or other transport protocols. For RDMA, the LLP is
       DDP.

   LLP Closed (LLP Close)- When the LLP Stream can no longer be used
       for data transmission. If there is a single LLP Stream on an LLP
       Connection, it may also mean that the LLP Connection has been
       torn down. For example, for TCP this could include the states
       TIME_WAIT, CLOSING, LAST-ACK, and CLOSED

   LLP Connection - Corresponds to an LLP transport-level connection
       between the peer LLP layers on two nodes.

   LLP Reset - The abnormal LLP closing mechanism, usually used to
       indicate that the LLP Stream (and possibly Connection) was
       aborted mid-stream. An example of this would be a TCP connection
       being closed due to the reception or transmission of a TCP RST
       on the connection.

   LLP Stream - Corresponds to a single bi-directional LLP transport-
       level association between the peer LLP layers on two Nodes. One
       or more LLP Streams may map to a single transport-level LLP
       Connection. For transport protocols that support multiple
       Streams per connection (e.g. SCTP), a LLP Stream corresponds to
       one transport-level Stream.

   Memory Region (MR) - An area of memory that the Consumer wants the
       RNIC to be able to (locally or locally and remotely) access
       directly in a logically contiguous fashion. A Memory Region is
       identified by an STag, a Base TO, and a length. A Memory Region
       is associated with a Physical Buffer List through the STag.



   Hilland, et al.        Expires October 2003              [Page 12]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Memory Registration (Registration, Register) - The mechanism used to
       enable direct (local or local and remote) access by the RNIC of
       a Consumer Memory Region. The memory registration operation
       associates a Physical Buffer List to the Steering Tag (STag)
       returned.

   Memory Translation and Protection Table(s) (TPT) - The data
       structure(s) used by an RNIC to control buffer access and
       translate STags and Tagged Offsets into local memory addresses
       directly accessible by the local Node.

   Memory Window (MW) -                      - A subset of a Memory Region, which can be
       remotely accessed in a logically contiguous fashion. A Memory
       Window is identified by an STag, a Base TO, and a length, but
       also references an underlying Memory Region and has Access
       Rights.

   Message Sequence Number (MSN) - For the Untagged Buffer Model, it
       specifies a sequence number that is increasing with each DDP
       Message.

   Modifiers - In a Verb definition, the list of input and output
       objects that specify how, and on what, the Verb is to be
       executed.

   Node - A computing device attached to one or more links of a Fabric
       (network). A Node in this context does not refer to a specific
       application or protocol instantiation running on the computer. A
       Node may consist of one or more RNICs installed in a host
       computer.

   Non-Privileged Mode - An operating mode in which Consumers must rely
       on another agent, having a sufficiently high level of privilege,
       to manipulate OS data structures.

   Non-Shared Memory Region - A Memory Region that solely owns the
       Physical Buffer List associated with the Memory Region.
       Specifically, the PBL is not shared, and has never been shared,
       with another Memory Region.

   Outbound RDMA Read Queue Depth (ORD) - The maximum number of
       outstanding RDMA Read Request Messages the RNIC can initiate
       from the SQ at the Data Sink.

   Outstanding - The state of a Work Request after it has been posted
       on a Work Queue, but before the retrieval of the Work
       Completion, or confirmation that the WR has been completed, by
       the Consumer.





   Hilland, et al.        Expires October 2003              [Page 13]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Page List - A list of physical addresses describing a set of memory
       pages, which specifies the page size, list of physical
       addresses, and offset to the start of the memory region of the
       first page. The starting physical addresses of each page is
       aligned on power-of-two addresses and the size of the page is a
       power of two. Note that it is possible for the starting offset
       to be an offset into the first page and to be of a byte
       granularity and the entire list may have an arbitrary length.

   Physical Address - A physical address is used by an RNIC to retrieve
       contents from the local host's memory. Physical addresses are
       determined via the translation of the STag and Tagged Offset by
       the use of the Memory Translation and Protection Table(s).

   Physical Buffer - A set of physically contiguous memory locations
       that can be directly accessed by the RNIC through Physical
       Addresses. A Physical Buffer can either be a block buffer or a
       page buffer, depending on its use as part of a Page List or
       Buffer List.

   Physical Buffer Entry Size - The size, in bytes, of each Physical
       Buffer in the Physical Buffer List. If the Physical Buffer List
       references a Page List, the size is a power of two. If the
       Physical Buffer List references a Block List, the size can have
       any value within the range supported by the RNIC.

   Physical Buffer List (PBL) - A list of Physical Buffers. The
       Physical Buffer List can either be a Block List or a Page List.

   Physical Memory Addresses - The addresses an RNIC uses when
       accessing host system memory.

   Pinning memory - A function supplied by the OS that forces the
       Memory Region to be resident in physical memory and keeps the
       virtual-to-physical address translations constant from the
       RNIC's point of view.

   Place - Also Placed, Placement. See Data Placement.

   Post Receive Queue Work Request (PostRQ) - A Verb that posts a Work
       Request to the Receive Queue of a Queue Pair. This is done to
       indicate the Data Sink Buffers for incoming Send Operation
       Types.

   Post Send Queue Work Request (PostSQ) - A Verb that posts a Work
       Request to the Send Queue of a Queue Pair. This is done to
       initiate all data transfer operations as well as Fast-Register,
       Bind MW and Local Invalidate operations.





   Hilland, et al.        Expires October 2003              [Page 14]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Privileged Mode - A mode in which Consumers operate where they have
       a privilege level sufficient to access OS internal data
       structured directly, and that have the responsibility to control
       access to the RI.

   Processing Error - An error detected below the RNIC Interface during
       the processing of a Work Request or an incoming RDMA operation.

   Protection Domain (PD) - A mechanism for tracking the association of
       Queue Pairs, Memory Windows, and Memory Regions. PDs are
       intended to be set by a Privileged Consumer to provide
       protection of one process from accessing another's memory
       through the use of the RNIC.

   Protection Domain ID (PD ID) - The identifier which represents a
       Protection Domain. It is passed in as an Input Modifier when
       creating QPs, Memory Windows and MRs. The value of PD IDs are
       compared during processing of Work Requests.

   Queue Pair (QP) - The pair of queues that allow the Consumer to
       interact with the RNIC Interface. The two queues are the Send
       Queue and the Receive Queue. Each queue stores a Work Queue
       Element from the time it is posted until the time it is
       completed.

   Queue Pair Context - The collection of information needed by the
       RNIC Interface to perform the RDMA Operations associated with
       the Queue Pair. This includes various pointers to buffers,
       queues, and CQs, as well as LLP specific connection and stream
       information.

   Queue Pair Identifier (QP ID) - An identifier representing a Queue
       Pair.

   Read Fence - To block the current operation from executing until all
       prior RDMA Read Type WRs submitted to the Send Queue have
       Completed.

   Receive Queue (RQ) - One of the two Work Queues associated with a
       Queue Pair. The Receive Queue contains Work Queue Elements that
       describe the Buffers into which data from incoming Send
       Operation Types is placed.

   Remote Access - The Access Rights used to verify the RNIC's ability
       to access the Data Sink for incoming DDP Tagged Messages and the
       Data Source for RDMA Read Request Messages.

   Remote Direct Memory Access (RDMA) - A method of accessing memory on
       a remote system in which the local system specifies the remote
       location of the data to be transferred. Employing an RNIC in the



   Hilland, et al.        Expires October 2003              [Page 15]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       remote system allows the access to take place without
       interrupting the processing of the CPU(s) on the system. Also
       used to indicate the layer implementing the RDMAP wire protocol
       semantics.

   RDMA Message - The sequence of DDP segments which represents an RDMA
       Operation.

   RDMA Operation - A sequence of RDMAP Messages, including control
       Messages, to transfer data from a Data Source to a Data Sink.
       The following RDMA Operations are defined - RDMA Write
       Operation, RDMA Read Operation, Send Operation, Send with
       Invalidate Operation, Send with Solicited Event Operation, Send
       with Solicited Event & Invalidate Operation, and Terminate
       Operation. Note that the various forms of Send Operations are
       defined in [RDMAP] to be called Send Type Operations.

   RDMA Protocol (RDMAP) - A wire protocol that supports RDMA
       Operations to transfer ULP data between a Local Peer and the
       Remote Peer. See [RDMAP].

   RDMA Read Operation - An RDMA Operation that consists of a single
       RDMA Read Request Message and a single RDMA Read Response
       Message. The Data Sink uses this operation to transfer the
       contents of a Data Source buffer from the Remote Peer to the
       Local Peer.

   RDMA Read Request - An RDMA Message used by the Data Sink to request
       the Data Source to transfer the contents of a buffer. The RDMA
       Read Request Message describes both the Data Source and Data
       Sink buffers.

   RDMA Read Response - An RDMA Message used by the Data Source to
       respond to an RDMA Read Request Message.

   RDMA Read Type Work Request - A PostSQ Work Request which specifies
       an operation type of either an RDMA Read or an RDMA Read with
       Invalidate Local STag.

   RDMA Stream - A single bi-directional association between the peer
       RDMA layers on two Nodes over a single LLP Stream.

   RDMA Write Operation - An RDMA Operation that transfers the contents
       of a source buffer from the Local Peer to a destination buffer
       at the Remote Peer using an RDMAP Write Message. The RDMAP Write
       Message only describes the Data Sink's buffer.

   RDMA Network Interface Controller (RNIC) - A network I/O adapter or
       embedded controller with iWARP and Verbs functionality.




   Hilland, et al.        Expires October 2003              [Page 16]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Remote Peer - The RDMA protocol implementation on the opposite end
       of the connection. Used to refer to the remote entity when
       describing protocol exchanges or other interactions between two
       Nodes.

   Remote RDMA Read Operation - a sequence of events that begins upon
       receipt of an incoming RDMA Read Request by the RI and stays in-
       process until the corresponding RDMA Read Response Message has
       been generated. This includes posting the RDMA Read Request to
       the Inbound RDMA Read Request Queue (See Section 6.5 -
       Outstanding RDMA Read Resource Management).

   RNIC Interface (RI) - The presentation of the RNIC to the Verbs
       Consumer as implemented through the combination of the RNIC and
       the RNIC device driver.

   Scatter/Gather Element (SGE) - An individual entry in a
       Scatter/Gather List. Each SGE consists of an STag, Tagged Offset
       and Length.

   Scatter/Gather List (SGL) - A List of Scatter/Gather Elements. The
       list describes one or more ULP Buffers which will have their
       data gathered on transmission or scattered upon reception.

   Send - An RDMA Operation that transfers the contents of an Untagged
       buffer from the Local Peer to an Untagged buffer at the Remote
       Peer.

   Send Operation Types - The set of Send operations that result in the
       consumption of a Receive Queue Work Request at the Data Sink.
       Specifically this includes Send, Send with Invalidate, Send with
       Solicited Event and Send with Solicited Event & Invalidate.

   Send Queue (SQ) - One of the two Work Queues associated with a Queue
       Pair. The Send Queue contains PostSQ Work Queue Elements that
       have specific operation types, such as Send Type, RDMA Write, or
       RDMA Read Type Operations, as well as STag operations such as
       Bind and Invalidate.

   Shared Memory Region - An MR that currently shares, or at one time
       shared, the Physical Buffer List associated with the Memory
       Region. Specifically, the PBL is currently shared or was
       previously shared with another Memory Region.

   Shared Receive Queue - An optional mechanism which allows the
       Receive Queues from multiple QPs to retrieve Receive Queue Work
       Queue Elements from the same shared queue as needed.

   Signaled - A WR which requires that the RNIC generate a Work
       Completion.



   Hilland, et al.        Expires October 2003              [Page 17]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Solicited Event (SE) - A facility by which an RDMA Operation sender
       may cause an Event to be generated at the recipient, if the
       recipient is configured to generate such an Event, when a Send
       with Solicited Event or Send with Solicited Event & Invalidate
       Message is received.

   Steering Tag (STag) - An identifier of a Memory Window or Memory
       Region. STags are composed of two components: an STag Index and
       an STag Key. The Consumer forms the STag by combining the STag
       Index with the STag Key. This specification further refines the
       definitions of STags contained in [RDMAP] and [DDP].

   STag Key - The least significant 8 bit portion of an STag. This
       field of an STag can be set to any value by the Consumer when
       performing a Memory Registration operation, such as Bind Memory
       Window, Fast-Register Memory Region and Register Memory Region.

   STag Index - The most significant 24 bits of an STag. This field of
       the STag is managed by the RI and is treated as an opaque object
       by the Consumer.

   Tagged Buffer - A buffer that can be Advertised to a Remote Peer
       through exchange of an STag, Tagged Offset, and length.

   Tagged Offset (TO) - The offset within a Tagged Buffer.

   Terminate - An RDMA Message used by a Node to pass an error
       indication to the Remote Peer on an RDMA Stream.

   Upper Layer Protocol (ULP) - The protocol layer above the Verb
       layer. An example is SDP.

   ULP Buffer - A buffer owned above the RI that can be represented
       within the RNIC, in whole or in part, by a Memory Window or a
       Memory Region.

   ULP Message - The ULP data that is handed to a specific protocol
       layer for transmission. Data boundaries are preserved as they
       are transmitted through iWARP.

   ULP Payload - The portion of a ULP Message that is contained within
       a single protocol segment or packet (e.g. a DDP Segment).

   Unaffiliated Asynchronous Event - This is an indication from the
       Verb layer to the Consumer that an event has occurred unrelated
       to any single identifiable RNIC Resource.

   Unsignaled - A Work Request which only generates a Work Completion
       if it encounters an error during processing.




   Hilland, et al.        Expires October 2003              [Page 18]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Untagged Buffer - A buffer which is not Advertised to a Remote Peer,
       that has Local Access Rights, and that is referenced by an STag,
       Tagged Offset, and length.

   Verbs - An abstract description of the functionality of an RNIC
       Interface. The OS may expose some or all of this functionality
       via one or more APIs to applications. The OS will also use some
       of the functionality to manage the RNIC Interface.

   Virtual Address - An address represented in the address space of a
       local process on a node. It is generally used to present
       logically contiguous addressability for an underlying and
       possibly non-contiguous list of physical pages.

   Virtual Address Based Tagged Offset (VA Based TO) - The Base TO of
       an MR or MW that starts at a non-zero TO.

   Work Completion (WC) - The output modifiers that the Consumer
       retrieves from a Completion Queue indicating the results of a
       Work Request.

   Work Queue (WQ) - One of either a Send Queue or Receive Queue.

   Work Queue Element (WQE) - The RNIC Interface's internal
       representation of Work Request.

   Work Request (WR) - An elementary object used by Consumers to
       enqueue a requested operation (WQEs) onto the Send and Receive
       Queues of a QP.

   Work Request List (WRL) - A list of Work Requests.

   Zero Based Tagged Offset (Zero Based TO) - The Base TO of an MR or
       MW that starts at TO=0.

4.1  Abbreviations

   CQ - Completion Queue

   CQE - Completion Queue Entry

   DDP - Direct Data Placement Protocol

   FBO - First Byte Offset

   IRD - Inbound RDMA Read Queue Depth

   IRRQ - Inbound RDMA Read Request Queue

   LLP - Lower Layer Protocol



   Hilland, et al.        Expires October 2003              [Page 19]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   MR - Memory Region

   MW - Memory Window

   ORD - Outbound RDMA Read Queue Depth

   PBL - Physical Buffer List

   PD - Protection Domain

   PD ID - Protection Domain Identifier

   QP - Queue Pair

   QP ID - Queue Pair Identifier

   RQ - Receive Queue

   RDMA - Remote Direct Memory Access

   RDMAP - Remote Direct Memory Access Protocol

   RNIC - RDMA NIC

   RI - RNIC Interface

   SGE - Scatter-Gather Element

   SGL - Scatter-Gather List

   SE - Solicited Event

   S-RQ - Shared Receive Queue

   SQ - Send Queue

   STag - Steering Tag

   TO - Tagged Offset

   TPT - Translation & Protection Table

   ULP - Upper Layer Protocol

   WC - Work Completion

   WQ - Work Queue

   WQE - Work Queue Element




   Hilland, et al.        Expires October 2003              [Page 20]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   WR - Work Request

   WRL - Work Request List


















































   Hilland, et al.        Expires October 2003              [Page 21]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

5  RNIC Interface

   The RNIC Interface (RI) is the locus of interaction between the
   Consumer of RNIC services and the RNIC. Semantic behavior of the
   RNIC is specified via Verbs, which enable creation and management of
   Queue Pairs, management of the RNIC, management of Work Requests,
   and transferring error indications from the RI that may be surfaced
   via the Verbs. All these activities must be carried out so as to
   enable Verbs Consumers to expect the same level of protection and
   security as are guaranteed other entities supported by the host
   operating system.

   A fundamental function of the RI is management of RNICs. This
   includes arranging access to them, accessing and modifying their
   attributes, and shutting them down. These activities are described
   below, and details of the corresponding Verbs semantics are given in
   subsequent sections.

   Direct, protected access to Consumer memory is critical to realizing
   the performance potential of the RNIC. This specification describes
   the semantics of memory access defined in this architecture. It
   describes in detail the ideas of Memory Regions and Memory Windows,
   how they are created and managed, Access Rights for local and remote
   access to registered memory, and the semantics of errors that may
   arise.

   The RI is assumed to be a traditional software interface, typically
   synchronous in behavior, while QP interactions are assumed to be
   work requests queued to connection specific, hardware based queues.
   The queue processing model and associated memory protection
   semantics allow QPs to be safely mapped and utilized by both Non-
   Privileged and Privileged routines.

   Queue Pairs (QPs) are a key component required for the operation of
   the RI. They are the RNIC resource used by Consumers to submit Work
   Requests to the RI. A QP is used to interact with an RDMA Stream on
   an RNIC which is running the RDMA Protocol. There may be thousands
   of QPs per RNIC. Each QP provides the Consumer with a single point
   of access to an individual RDMA Stream.

   Work Requests (WRs) provide the mechanism for Consumers to enqueue
   Work Queue Elements (WQEs) onto the Send and Receive queues of a QP.
   The varieties of WRs, and the dynamics of their creation, use, and
   disposition are described in the sections to follow, as are the
   disposition of errors that may arise as WR are processed. Details of
   the WR contents are discussed as well.

   Completion Queues (CQs) provide the mechanism for the Consumer to
   retrieve WR status. In addition, there are notification mechanisms




   Hilland, et al.        Expires October 2003              [Page 22]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   which help a Consumer to efficiently notice when WRs have completed
   processing in the RI. There may be thousands of CQs per RNIC.

   Event Handlers provide the mechanism for Consumers to be notified of
   Asynchronous Events which occur within the RI but which cannot be
   reported through the Completion Queues due to their asynchronous
   nature or the fact that they are not easily associated with a Work
   Completion.

5.1  The RNIC

   Consumers gain access to an RNIC through the RNIC Interface. The
   Verbs allow the Consumer to open the RNIC, retrieve RNIC attributes,
   and close the RNIC.

   All resources MUST be in the scope of the RNIC on which they are
   created. This means that there is no requirement for resources on
   one RNIC to be available, associated with or meaningful to another
   RNIC, even if they are managed by the same RNIC driver. This
   includes all QPs, STags, PDs, CQs, and multiple Completion Event
   Handlers. This also means that any IDs which are created by the RI
   are specific to that RNIC and are not guaranteed to be unique across
   all RNICs.

   An intent of the architecture is to allow an implementation to pass
   Work Requests and Work Completions to and from a Non-Privileged Mode
   Consumer process directly to and from the RNIC. Another intent of
   the architecture is to optimize for a Privileged Mode
   implementation, which shares the Work Request and Work Completion
   requirements of Non-Privileged Mode Consumers but has slightly
   different memory management requirements.

   Because the architecture attempts to optimize for both Privileged
   Mode and Non-Privileged Mode Consumers, there are some Verbs and
   Verb modes which are not allowed to be executed by non-Privileged
   Mode Consumers. An example of this is the use of the STag of zero or
   the ability to do Fast-Register WRs. In addition, there are some
   operations that, while being allowed in kernel mode, are intended to
   be used by Non-Privileged mode applications. An example of this is
   Memory Windows. Any restrictions are clearly specified in this
   document where required.

5.1.1  RNIC Resources

   RNIC Resources can be allocated from a variety of places. They can
   be allocated in host memory on behalf of the Consumer or allocated
   within the RNIC. Where an RNIC allocates resources is implementation
   specific. Consequently, values that the RNIC returns as output
   modifiers when Querying the RNIC indicate the maximum amount of any




   Hilland, et al.        Expires October 2003              [Page 23]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   given resource it can allocate, in the absence of other resource
   allocations.

   For example, an RNIC may allocate QPs, CQs from the same memory
   within the RNIC. If a Consumer allocates the maximum amount of QPs
   before allocating any CQs, it may not be able to allocate any CQs
   due to an insufficient resource condition - even though the RNIC
   indicates that its maximum number of CQs is much larger than the
   number currently allocated.

   The purpose of a handle is to provide a mechanism to lookup a
   specific resource. Resources that have handles associated with them
   are the RNIC, CQ, S-RQ, QP, and Asynchronous Event Handler. Often a
   handle is an address in memory. An identifier or index also
   references a specific resource. An identifier or index is used when
   the value must be used in a comparison operation. The QP ID, PD ID,
   Completion Event Handler Identifier and STag Index fall in this
   category.

   It is expected that a resource manager above the RI will manage RNIC
   resources appropriately for the operating environment.

5.1.1.1  Expected Creation Sequence

   Due to RI Resource interdependencies, there is an ordering sequence
   to the allocation and creation of RNIC resources. The sequence
   indicated below, while not strictly required in all cases, may be
   helpful to the reader.

   1.  Open the RNIC and setup up an Asynchronous Event Handler.

   2.  Prior to initiating a LLP Connection, select the opened RNIC on
       which you will create the connection and create a Protection
       Domain.

   3.  Create one or more Completion Queues.

   4.  Set up one or more Completion Event Handlers.

   5.  Allocate and initialize a Shared Receive Queue, if desired.

   6.  Allocate and initialize one or more QPs.

   7.  Register one or more Memory Regions.

   8.  Allocate Non-Shared Memory Region STags, if desired.

   9.  Allocate Memory Windows, if desired.

   10. Transition the QP through the state diagram to RTS.



   Hilland, et al.        Expires October 2003              [Page 24]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   11. Initiate Work Request Processing through PostSQ, PostRQ and Poll
       CQ.

   Below in Figure 2 is a dependency diagram which may also be helpful
   when determining the order in which resources are created. The
   arrows indicate that the resource the arrow comes from must be
   created or allocated before the item the arrow points to can be
   created or allocated.










             < Figure 2 did not convert properly from source >
             <  to be corrected in an upcoming version       >















              Figure 2 - Resource Creation Dependency Diagram

5.1.1.2  Expected Destruction Sequence

   Due to RI Resource interdependencies, there is an ordering of de-
   allocation and destruction of RNIC resources. The sequence indicated



   Hilland, et al.        Expires October 2003              [Page 25]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   below, while not strictly required in all cases, may be helpful to
   the reader.

   1.  Invalidate all Memory Windows which are in the Valid state
       through a QP WR, if possible.

   2.  Drain the SQ & RQ of WRs and poll the Work Completions through
       the CQ.

   3.  Transition the QP state to Closing.

   4.  When the QP is in the Idle state, Destroy the Memory Windows.

   5.  Destroy the Memory Regions.

   6.  Destroy the Queue Pair.

   7.  Destroy the Shared Receive Queue, if created.

   8.  Destroy the Completion Queues.

   9.  Destroy the Protection Domain.

   10. Close the RNIC.

   Below in Figure 3 is a dependency diagram which may also be helpful
   when determining the order in which resources are destroyed. The
   arrows indicate that the resource the arrow comes from must be
   destroyed or deallocated before the item the arrow points to can be
   destroyed or deallocated. A dashed line means the action should
   occur before the resource can be destroyed. A solid line means the
   action must occur before the resource can be destroyed.





















   Hilland, et al.        Expires October 2003              [Page 26]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003













             < Figure 3 did not convert properly from source >
             <  to be corrected in an upcoming version       >


















            Figure 3 - Resource Destruction Dependency Diagram







   Hilland, et al.        Expires October 2003              [Page 27]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

5.1.2  Opening an RNIC

   The Open RNIC Verb is used to open an RNIC and returns an opaque
   handle to uniquely reference each RNIC so that Consumers can
   distinguish between RNICs in the Local Node.

   Opening an RNIC prepares it for use by the Consumer. Once opened, an
   RNIC cannot be opened again until after it has been closed. At the
   time the RNIC is opened, the RI MUST perform any initialization
   functions required by the RNIC and the RI.

   When the Consumer invokes the Open RNIC Verb, it indicates if this
   RNIC is to be opened in Page Mode or Block Mode. The RI MUST
   initialize the RNIC in either Page Mode or Block Mode, as indicated
   by the Consumer with the input modifier. This will affect all Memory
   Registrations and usage as well as resource consumption on the RNIC.
   Note that while Page Mode MUST be supported, Block Mode is OPTIONAL.
   For more information on Block Mode vs. Page Mode, see Section 7.6.2
   - Physical Buffer Lists.

   Detailed information on the accompanying Verb can be found in
   Section 9.2.1.1 - Open RNIC.

5.1.3  Query RNIC

   Consumers MUST be able to retrieve all of the defined attributes and
   characteristics of the RNIC through the Query RNIC Verb. The full
   list of RNIC Attributes is defined in Section 9.2.1.2 - Query RNIC.

   The maximum values returned when querying the RNIC are values which
   the RI will not exceed. This does not imply that a Consumer can
   allocate all resources to their maximum levels simultaneously.

5.1.4  Closing an RNIC

   Closing the RNIC resets the RNIC and deallocates any resources
   allocated during the RNIC open.

   The RI MUST track all RNIC resources created on behalf of the
   Consumer, such as those allocated within the RI during the creation
   of PDs, QPs, CQs, Memory Windows and MRs. When the Close RNIC verb
   returns, the RI MUST have freed all RNIC resources.

   Detailed information on the accompanying Verb can be found in
   Section 9.2.1.3 - Close RNIC.

5.2  Protection Domains

   A Protection Domain (PD) is the mechanism used to associate Queue
   Pairs with Memory Regions and Memory Windows as a means of enabling



   Hilland, et al.        Expires October 2003              [Page 28]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   and controlling RNIC access to host system memory. A Protection
   Domain is represented by a unique identifier called a Protection
   Domain Identifier (PD ID).

   When the Consumer creates a PD, a PD ID is returned. The Consumer
   then provides the PD ID to the RI when creating QPs, MRs & Memory
   Windows. When a data transfer takes place, if the STag refers to an
   MR, then the PD ID of the MR is validated against the PD ID of the
   QP. If they do not match, the data transfer generates an error and
   no data transfer takes place. If the STag refers to an MW, then the
   PD ID of the MW is validated against the PD ID of the QP when the MW
   is Bound to the QP. When a data transfer takes place, the QP ID of
   the MW is validated against the QP ID of the QP. These rules allow
   the Consumer to ensure that any STag being used on that connection,
   either locally or remotely, has been specifically allowed by the
   Consumer to be used on that connection.

   Each Queue Pair in an RNIC MUST be associated with a single PD ID.
   Multiple Queue Pairs MUST be able to be associated with the same PD
   ID.

   Each Memory Region MUST be associated with a single PD ID. Multiple
   Memory Regions MUST be able to be associated with the same PD ID.

   Each Memory Window MUST be associated with a single PD ID when
   allocated. Multiple Memory Windows MUST be able to be associated
   with the same PD ID.

   The RI MUST be able to associate any PD ID with any MW, MR, QP or S-
   RQ on the RNIC.

   Binding a Memory Window to a Memory Region and Fast-Register are
   performed as Send Queue operations. The Bind operation MUST only be
   allowed if the PD ID of the QP matches the PD ID of the Memory
   Region and the PD ID of the QP matches the PD ID of the Memory
   Window. Similarly, the Fast-Register operation MUST only be allowed
   if the PD ID of the QP matches the PD ID of the STag used as an
   input modifier for the Fast-Register. If the PD ID checks fail for
   either operation, the operation MUST NOT take place and a Completion
   Error MUST be generated.

   Note that S-RQs use PDs as well. PD rules for S-RQs are covered in
   Section 6.3

5.2.1  Allocating a PD

   Protection Domains MUST only be allocated through the RI. A PD ID is
   required to be supplied as an input modifier when creating a Queue
   Pair, registering a Memory Region, or allocating a Memory Window.




   Hilland, et al.        Expires October 2003              [Page 29]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   The RI MUST assign a unique PD ID to each PD allocated by the RI. PD
   ID's MUST be unique per RNIC. PD ID's MAY be unique across multiple
   RNICS which share the same RI.

   Detailed information on the accompanying Verb can be found in
   Section 9.2.2.1 - Allocate PD.

5.2.2  Deallocating a PD

   PDs MUST only be deallocated through the RI. A PD MUST NOT be
   deallocated if it is still associated with any Queue Pair, Shared
   Receive Queue, Memory Region, or Memory Window. If this is
   attempted, the Verbs MUST return an Immediate Error and not allow
   the PD to be deallocated.

   Detailed information on the accompanying Verb can be found in
   Section 9.2.2.2 - Deallocate PD.

5.3  Completion Queues

   The Completion Queue consists of entries to hold Work Completions.
   The RI's internal representations of Work Completions are called
   Completion Queue Entries (CQEs). The RI will post a CQE to the CQ
   when it completes the operation of a Signaled WR. The Consumer then
   Polls the CQ to retrieve the CQE as a Work Completion. When the Work
   Completion is retrieved, the CQE is freed from the CQ and the entry
   is available for another Work Request's Work Completion information.
   For an Unsignaled WR, the RI will not generate a CQE when the WR
   completes successfully. The RI will post a CQE to the CQ when an
   Unsignaled WR completes in an error. For more information on
   Signaled and Unsignaled Completions, see Section 8.1.3.1.

   A Completion Queue (CQ) MUST be the only mechanism used for the
   retrieval of Work Completions.

   A single CQ is used to hold CQEs from one or more Work Queues across
   one or more Queue Pairs on the same RNIC. A CQ MAY have zero or more
   Work Queue associations. Completion Queues MUST be able to service
   Send Queues, Receive Queues or both. Work Queues from multiple QPs
   MUST be able to be associated with a single CQ.

   Completion Queues and Completion Queue Entries are internal to the
   RNIC Interface, and are not directly accessible, nor is the format
   directly visible, by Verb Consumers.

5.3.1  Creating a Completion Queue

   Completion Queues MUST only be created through the RNIC Interface.





   Hilland, et al.        Expires October 2003              [Page 30]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   The RI MUST verify that the Consumer has specified the number of
   CQEs the CQ should hold when creating a Completion Queue. The
   Consumer should ensure that this value is the maximum number of
   Completions the Consumer expects to be outstanding. The RNIC will
   then create the CQ with at least the specified number of entries.
   The number of entries allocated for the CQ by the RI MAY be greater
   than the number requested. If the CQ can be created, the RI MUST
   return the actual number of entries allocated for that CQ to the
   Consumer. If the RI is unable to allocate at least as many entries
   as the Consumer requested, an Immediate Error MUST be returned and
   the CQ MUST NOT be created.

   The RI is NOT REQUIRED to perform CQ overflow detection or
   protection. Therefore, the CQ overflow error codes in this document
   are OPTIONAL. When an overflow occurs, the results are
   indeterminate. Overflow of a CQ MUST NOT affect QPs which do not
   report Work Completions to that CQ and MUST NOT affect other CQs.
   Consequently, when creating the CQ, the Consumer should request
   enough outstanding Work Requests so that if every possible
   outstanding WR were to complete (such as may happen in an error
   case), there would be room for the CQE on the CQ. The RI MUST NOT
   enforce that every WQE on every Work Queue associated with the CQ
   must have a CQE available for the WQE's Work Completion information.

   If the Consumer wishes to have deterministic error behavior, at
   Create/Modify QP, the sum of the maximum number of WQEs associated
   with a single CQ should be less than or equal to the number of
   entries in the CQ. A Consumer can size the CQ smaller, in which case
   the error semantics of a CQ overflow are not deterministic, but
   possible RNIC behavior includes overwriting previous CQEs in whole
   or in part and thus may result in a data integrity issue.

   An additional consideration for sizing the CQ is QP Destruction. Any
   outstanding WRs which were on a Work Queue when it is destroyed may
   occupy entries on the associated CQ. For more information, see
   Section 6.1.4 - Destroying a Queue Pair.

   Detailed information on the accompanying Verb can be found in
   Section 9.2.3.1 - Create CQ.

5.3.2  Querying Completion Queue Attributes

   There are two Completion Queue attributes that can be queried
   through the RI.

   The first of these attributes is the maximum number of entries
   allowed on the CQ. This attribute MUST be able to be retrieved
   through the Query CQ Verb.





   Hilland, et al.        Expires October 2003              [Page 31]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   The other attribute is the Completion Event Handler Identifier,
   which also MUST be able to be retrieved through the Query CQ Verb.

   With one exception, the CQ Verbs do not expose which Work Queues are
   associated with a CQ. The exception is that the QP ID is reported by
   Poll CQ.

   Detailed information on the accompanying Verb can be found in
   Section 9.2.3.2 - Query CQ.

5.3.3  Modifying Completion Queue Attributes

   An implementation MUST support resizing of a CQ through the RI while
   WRs are outstanding. Work Completions MUST NOT be lost due to a CQ
   resize. Resizing the CQ MUST NOT directly generate errors beyond
   Resize CQ Verb Immediate Errors and must either succeed or fail
   atomically. It is understood that this may adversely affect
   performance, and MAY result in connection timeouts. Note that this
   could ultimately result in the connection being torn down. If the
   Consumer wishes to avoid any possibility of a connection being torn
   down during the CQ resize operation, it should quiesce operations to
   the Work Queues associated with the CQ before resizing the CQ. The
   RI MUST NOT allow a CQ to be resized to a size that is smaller than
   the number of CQEs currently on the CQ; if this is attempted, an
   Immediate Error MUST be returned.

   Detailed information on the accompanying Verb can be found in
   Section 9.2.3.3 - Modify CQ.

5.3.4  Destroying a Completion Queue

   CQs MUST only be destroyed through the RI.

   A CQ MUST NOT be destroyed if it is still associated with any Work
   Queue. If this is attempted, the Verbs MUST return an Immediate
   Error and not allow the CQ to be destroyed.

   When the Destroy CQ Verb returns, the RI MUST have returned or
   released any host resources allocated below the RNIC Interface on
   behalf of the Consumer that are related to the specified CQ. After
   the Destroy CQ Verb returns, the RI MUST NOT return any more Work
   Completions that are associated with the destroyed CQ.

   Detailed information on the accompanying Verb can be found in
   Section 9.2.3.4 - Destroy CQ.








   Hilland, et al.        Expires October 2003              [Page 32]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

6  Queue Pairs

   Queue Pairs (QP) are the RNIC resource used by Consumers to submit
   operations to the RNIC. A QP consists of a pair of Work Queues (Send
   and Receive) as well as a posting mechanism for each queue. The Send
   Queue (SQ) and Receive Queue (RQ) are each Work Queues, in that the
   Consumer posts Work Requests (WR) to them in order to get the RI to
   perform operations. In addition, there are resources that make up
   the QP with which the Consumer does not directly interact. These
   include the Inbound RDMA Read Request Queue and the Work Queue
   Elements (WQEs).

   Work Queue Elements are the representation of Work Requests inside
   of the RI, once the Work Requests have been posted to the QP.

   An internal Inbound RDMA Read Request Queue (IRRQ) MUST be
   associated with a Queue Pair when the QP is created or modified to
   support greater than zero incoming RDMA Read Request Messages. The
   IRRQ enqueues incoming RDMA Read Request Messages and processes them
   in order, sending RDMA Read Response Messages as a result. The depth
   of this queue MUST be specified when the QP is created and is set
   with the IRD Input Modifier.

   A QP is created by the RI at the request of a Consumer. The
   resources required by the RI to create the Work Queues and get them
   to transmit and receive resources are allocated at this time. The
   memory needed may be allocated from system memory, memory associated
   within the RNIC, or any other resources accessible through the
   Verbs.

   Certain QP attributes may be changed after QP creation. A Modify QP
   Verb is provided to modify the attributes. The details of this Verb
   are defined in Section 6.1.3 - Modifying Queue Pair Attributes.

   The Consumer should instruct the RI to destroy a QP that is no
   longer in use. The semantics for destruction of a QP are provided in
   this Section 6.1.4 - Destroying a Queue Pair.

   The Verbs Post Send Queue Work Request (Section 9.3.1.1 PostSQ) and
   Post Receive Queue Work Request (Section 9.3.1.2 PostRQ) provide a
   posting mechanism for the Consumer to indicate to the RI that there
   is work for the RI to perform and that there is a new WR,
   represented within the RI by a WQE, on the Work Queue. Details of
   Work Request handling are defined in Section 8 - Work Requests and
   the WR Processing Model.








   Hilland, et al.        Expires October 2003              [Page 33]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

6.1  Queue Pair Resource Handling

6.1.1  Creating a Queue Pair

   Queue Pairs are created through the RI. When a QP is created, the RI
   MUST verify that the Consumer has specified a complete set of
   initial attributes. The attributes that need to be defined when the
   QP is created are specified in Section 9.2.5.1 - Create QP.

   Two of the attributes that must be initialized when a QP is created
   is the maximum number of Outstanding WRs on the SQ and the maximum
   number of Outstanding WRs on the RQ. This number represents the
   maximum number of WRs which have been submitted but which have not
   Completed at any given time. This is really the maximum depth of the
   SQ or RQ and not the number of WRs on the Work Queue at the moment.
   The RI MUST support Consumers specifying the maximum number of
   outstanding WRs on the SQ and on the RQ and allow the maximum number
   of outstanding WRs on the SQ to be different from that on the RQ.
   The Consumer requests a maximum number of outstanding WRs on the SQ
   and on the RQ. The RI MUST return the maximum number of outstanding
   WRs allocated on the SQ and on the RQ, and each of these numbers MAY
   be greater than the number requested. For information on determining
   when WRs are completed, see Section 8.1.3.1 - Signaled Completions.
   Note that if the QP uses an S-RQ for incoming Untagged Messages, the
   maximum number of Outstanding WRs on the RQ is not needed.

   Each Work Queue in a QP MUST be associated with one and only one CQ
   when that QP is created.

   Since both WQEs and CQEs are implemented below the RI and the
   implementations are outside the scope of this specification, they
   may be implemented using a variety of mechanisms, including in the
   Local Host virtual memory address space. The RI MAY require that the
   Work Queues be in the same memory space as the corresponding
   Completion Queues or the creation of the QP will fail. Therefore the
   Consumer should assume that the CQ & QP share the same address
   space. If the RI detects that QP and CQ are inaccessible to each
   other, creation of the QP MAY fail.

   Other attributes that MUST be initialized when a QP is created are
   whether or not this QP will support the Fast-Register Non-Shared
   Memory Region operation and whether the QP supports an STag of zero.
   These attributes must only be enabled on QPs used by Privileged Mode
   Consumers. See Section 7.2.1 - STag of zero for an explanation of
   the STag of Zero. For an explanation of the Fast-Register Non-Shared
   Memory Region operation, see Section 7.3.2.5 - Fast-Register Non-
   Shared Memory Region.

   When a QP is created it MUST be associated with a PD. This is done
   by specifying the PD ID as an Input Modifier to Create QP.



   Hilland, et al.        Expires October 2003              [Page 34]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   An attribute that MUST be created within the RI when the Consumer
   invokes the Create QP Verbs is a Queue Pair Identifier (QP ID). The
   QP ID MUST be used by the RI to uniquely identify this QP within
   this RNIC to the Consumer. The QP ID is used when trying to
   determine if a Memory Window is Bound to the QP, as discussed in
   Section 7.10.2 - Binding Memory Windows to Memory Regions. The QP ID
   value MUST be returned as part of the Create QP, Query QP and Poll
   CQ Verbs.

   Create QP MUST NOT associate an LLP Connection or LLP Stream with
   the QP. No data will flow until the QP is Associated with another QP
   through an LLP Stream and the QP state is changed to RTS. For more
   details, see Section 6.6.1 - Connection Initialization.

   A QP can exist in one of several states. For the details of the QP
   states, see Section 6.2 - Queue Pair Resource States. The following
   list summarizes the valid QP states:

   *   Idle state - No LLP Stream is associated with the QP.

   *   RTS state - An LLP Stream is associated with the QP and normal
       data transfer can occur.

   *   Closing state - An error free LLP Close has begun but has not
       finished. It was initiated by either the Remote Peer or Local
       Peer.

   *   Terminate state - An error occurred. A Terminate Message was
       either sent or received, and the QP is waiting for either a LLP
       Close or LLP Reset before automatically transitioning the QP to
       the Error state.

   *   Error state - An error occurred. No LLP Stream is associated
       with the QP. A Terminate Message will be available through
       QueryQP if the QP transitioned through the Terminate state
       before entering the Error state. If the transition was from the
       Closing state to the Error state, a Terminate Message may be
       available.

   When the QP is created, it is initialized to the Idle state.

   Detailed information on the accompanying Verb can be found in
   Section 9.2.5.1 - Create QP.

6.1.2  Querying Queue Pair Attributes

   Queue Pairs have attributes that can be retrieved through the Query
   QP Verb. The RI MUST support the complete list of QP attributes as
   described in Section 9.2.5.2 - Query QP.



   Hilland, et al.        Expires October 2003              [Page 35]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

6.1.3  Modifying Queue Pair Attributes

   Certain QP attributes may be modified after the QP has been created.
   If the Consumer invokes Modify QP without specifying all Required
   Attributes as defined in Figure 4, the RI MUST NOT modify any of the
   QP attributes and MUST return an Immediate Error. The RI MUST allow
   the Consumer to request a change for the Allowed Additional
   Attributes as described in Figure 4, for the QP state transitions
   also shown in the figure. On Consumer request, the RI MAY change the
   allowed Additional Attributes as described in Figure 5, for the QP
   state transitions shown in the figure, if the RI indicates through
   Query RNIC that the attribute in question is allowed to be changed.
   The Modify QP Verb output modifiers can be used to determine if the
   changes are actually made.

   If any of the QP attributes requested to be modified are invalid or
   the requested state transition is invalid, the RI MUST NOT modify
   any of the QP attributes and an Immediate Error MUST be returned.
   Note that the table is heavily dependent upon the QP state. For
   further information on the QP state, see Section 6.2 - Queue Pair
   Resource States.
































   Hilland, et al.        Expires October 2003              [Page 36]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

    Transition   Attributes that        Attributes that the RI must
                 Consumer is Required   Support and the Consumer may
                 to Supply for the      Supply for the State
                 State Transition       Transition

    Idle->Idle   Next state             ORD

    Idle->RTS    Next state,            Stream message buffer,
                 LLP Stream Handle      ORD

    Idle->Error  Next state             None

    RTS->RTS     Next state             ORD
    (Footnote 1)

    RTS->CLOSING Next state             None

    RTS->TERM    Next state             None


    RTS->Error   Next state             None

    Error->Idle  Next state             None

              Figure 4 - Allowable QP Attribute Modifications




















   Footnote 1: Changing these parameters in RTS requires care to avoid
   race conditions to prevent errors.




   Hilland, et al.        Expires October 2003              [Page 37]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

      Transition   Attributes that the RI Optionally Supports
                   and the Consumer may Supply for the State
                   Transition

      Idle->Idle   Max Number of SQ WQE,
                   Max Number of RQ WQE (Footnote 2),
                   IRD,
                   QP's RQ Limit,
                   QP's RQ Limit Armed

      Idle->RTS    Max Number of SQ WQE,
                   Max Number of RQ WQE (Footnote 2),
                   IRD,
                   QP's RQ Limit,
                   QP's RQ Limit Armed

      Idle->Error  None

      RTS->RTS     Max Number of SQ WQE,
      (Footnote 3) Max Number of RQ WQE (Footnote 2),
                   IRD

      RTS->CLOSING None

      RTS->TERM    None

      RTS->Error   None

      Error->Idle  None

              Figure 5 - Optional QP Attribute Modifications

   It is possible to modify the QP attributes in Figure 4 and Figure 5
   with Work Requests outstanding on the QP. Depending on the
   modification, any Work Requests outstanding on the specified QP
   might not execute properly when the attributes are changed.

   An RNIC MAY allow the Consumer to change the maximum number of
   outstanding WRs on the SQ and on the RQ. The RNIC MUST indicate to
   the Consumer if it supports the ability to change the number of


   Footnote 2: Note that changing the Max Number of RQ WQEs has no
   effect if the QP uses an S-RQ

   Footnote 3: Changing these parameters in RTS requires care to avoid
   race conditions to prevent errors.




   Hilland, et al.        Expires October 2003              [Page 38]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   outstanding WRs on a QP. If the RNIC supports it, it MUST allow the
   number of outstanding WRs on both the SQ and the RQ to be changed
   while WRs are still outstanding. In addition, the RI MUST support
   the ability to change this on every QP if it indicates an ability to
   change the outstanding number of WRs.

   It is understood that changing the number of WRs that a Work Queue
   may have outstanding may adversely affect performance. Resizing the
   QP MUST NOT cause Immediate, Completion or Asynchronous Errors, with
   the exception of Immediate Errors returned by the Modify Queue Pair
   Verb and possible LLP time-outs. It is expected that the resize
   operation MAY adversely affect the Associated QP attempting to
   communicate with the Local QP during the resize operation in the
   form of LLP time-outs and retries which could result in LLP Stream
   teardown (which would result in an Asynchronous Error). It is
   suggested that the Consumer only perform this resize operation when
   activity on the connections has been quiesced to minimize the risk
   of transitioning Associated QPs to the Error state as a result of
   LLP time-outs.

   If the number of requested outstanding WRs is smaller than the
   actual number of outstanding WRs currently on the Work Queue(s),
   then the modification of the QP MUST fail with an Immediate Error
   and the QP MUST remain in the original state.

   For information on performing a Modify QP and modifying the value of
   IRD and/or ORD, see Section 6.5 - Outstanding RDMA Read Resource
   Management.

   When the Modify QP Verb completes, any state change requested MUST
   have occurred or an Immediate Error MUST be returned, in which case
   the QP state and accompanying modifier changes MUST remain as they
   were prior to the Modify QP Verb being invoked.

   The LLP Stream and the LLP Stream Message Buffer Input Modifiers for
   Modify QP are covered in Section 6.6.1.

   Detailed information on the accompanying Verb can be found in
   Section 9.2.5.3 - Modify QP.

6.1.4  Destroying a Queue Pair

   Queue Pairs MUST only be destroyed through the RNIC Interface.

   Successful destruction of a QP MUST release all resources allocated
   by the RI for the QP on behalf of the Consumer. The RI MUST have
   destroyed the QP when the Destroy QP Verb has successfully
   completed. If the LLP Stream is still associated with the QP, a
   Destroy QP MUST include disassociating the LLP resources from the
   QP, and MAY include an LLP Reset. After a Destroy QP finishes, the



   Hilland, et al.        Expires October 2003              [Page 39]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   QP ID will be immediately available for use on any subsequently
   created QP. The QP will cease processing all WRs, and no additional
   CQEs resulting from any outstanding WRs on this QP will be posted to
   the CQ.

   The RI MUST not allow a QP to be destroyed if there are still Memory
   Windows Bound to the QP. If the Consumer attempts to destroy a QP
   with Memory Windows Bound to the QP, an Immediate Error MUST be
   returned by the RI.

   The RI MUST allow the Destroy QP Verb to succeed regardless of the
   QP's state, provided there are no MWs Bound to it. For more
   information on the resource destruction and deallocation sequence,
   see Section 5.1.1.2 - Expected Destruction Sequence.

   It is RECOMMENDED that before a Consumer attempts to destroy a Queue
   Pair, it should cleanly complete all outstanding Work Requests and
   invalidate all Memory Windows which are Bound to the QP. It is
   recommended that ULPs and Consumers provide a graceful termination
   mechanism, return all Advertised STags to a known state, submit WRs
   to Invalidate all outstanding Memory Windows and then move through
   the Closing state. The Consumer should then retrieve all outstanding
   Work Completions through the CQ(s) associated with the QP's SQ & RQ.
   Only then should the Consumer destroy the QP.

   A QP is allowed to have Work Requests outstanding on both Work
   Queues when a request to destroy the QP is made.

   Any outstanding WRs posted to the QP but not yet processed by the RI
   MAY result in CQEs that MAY be retrievable by the Consumer. Note
   that even in the case where CQEs were generated it might not be
   possible for the Consumer to retrieve them after the QP has been
   destroyed. Since it is implementation dependent as to whether CQEs
   are consumed for outstanding WRs on a QP after that QP is destroyed,
   for the purposes of CQ overflow prevention, the Consumer should
   consider each outstanding WR to have consumed an entry in the CQ.
   There are three ways to free the CQE consumed within the CQ. Any
   method is acceptable and they are not mutually exclusive. The three
   methods are:

   *   the Consumer polls the CQ (See Section 9.3.2.1 - Poll for
       Completion (Poll CQ)) until the CQ is empty, or

   *   the Consumer retrieves a WC for a WR which was submitted to a
       Work Queue associated with the same CQ and that WR was submitted
       after the previous QP was destroyed, or

   *   the Consumer polls (See Section 9.3.2.1 - Poll for Completion
       (Poll CQ) a number of Work Completions equal to the total number
       of entries that the CQ can hold.



   Hilland, et al.        Expires October 2003              [Page 40]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Detailed information on the accompanying Verb can be found in
   Section 9.2.5.4 - Destroy QP.

6.2  Queue Pair Resource States

   The RI MUST restrict the QP to only be in one of the five Resource
   States (or just "states") as shown in Figure 6. The RI MUST NOT
   support transitions between QP states that are not shown in Figure
   6.

   During any state in which iWARP processing is done, it is possible
   for errors to be detected by the RNIC. When this occurs, the QP
   state will eventually transition to the Error state.

   State transitions must only be initiated by the Modify QP Verb,
   except where otherwise explicitly stated in the state descriptions.

   Creation of a QP causes the QP to enter the state diagram in the
   Idle state. Destruction of a QP causes the QP to exit the state
   diagram.

   Below, in Figure 6, is the QP State diagram. It shows the five QP
   states and the allowed transitions between states as well as the
   events and methods which cause those transitions. The individual
   states and transitions are described in the following sections in
   detail.



























   Hilland, et al.        Expires October 2003              [Page 41]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003












             < Figure 6 did not convert properly from source >
             <  to be corrected in an upcoming version       >













                        Figure 6 - QP State Diagram





   Hilland, et al.        Expires October 2003              [Page 42]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

6.2.1  Idle State

   The QP MUST be in the Idle state following QP creation or when moved
   to this state with Modify QP. In this state, Send or Receive WRs MAY
   be posted but they MUST NOT be processed and CQEs MUST NOT be
   generated.

   Note that whether or not the Consumer posts WRs to the Send Queue
   when the QP is in the Idle state depends on the method chosen for
   connection initialization (see Section 6.6.1 - Connection
   Initialization).

   While in the Idle state the RI MUST NOT associate an LLP stream to
   the QP.

   The RI MUST return an Immediate Error if the Consumer attempts to
   transition the QP from the Idle state to the Terminate state or to
   the Closing state.

   A short summary table describing the state changes for Idle state is
   shown in Figure 7. The following are detailed descriptions of those
   changes.

   Note that under certain conditions the Consumer might be required to
   flush Work Requests from a prior RDMAP Stream when in the Idle
   state. This can be done by transitioning the QP from the Idle to
   Error state (the Error state flushes all WRs) and then back to the
   Idle state. This may be necessary if when the Idle state is reached
   automatically (i.e. no Consumer intervention) from the RTS state at
   the Local Peer, which will occur if:

   *   the QP is currently in the RTS state, and the Consumer is
       actively posting Work Requests (PostSQ or PostRQ),

   *   the Remote Peer initiates an LLP Close (e.g. for TCP, it
       generates a FIN segment),

   *   the Local RI receives the LLP Close request, and immediately
       transitions to Closing state,

   *   the RI automatically creates an LLP Close acknowledgement (i.e.
       for TCP, it generates a FIN ACK segment), thus finishing the LLP
       Close from the Local PeerÆs perspective,

   *   the RI flushes all WRs, and no errors occurred during the LLP
       Close or flush,

   *   the RI automatically transitions the QP to the Idle state,





   Hilland, et al.        Expires October 2003              [Page 43]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   the Consumer is not aware of the transition to Idle, and posts a
       Work Request thinking it can still transmit or receive data.

   Note that a normal close should only be done by a ULP after an end-
   to-end synchronization to ensure all outstanding Work Requests have
   been flushed end-to-end. This is because RDMAP does not provide a
   graceful close. Thus if the Consumer performed a PostSQ, it is an
   error made by the Consumer. However, if the ULP posted an extra
   PostRQ buffer, it is arguable whether this is an error made by the
   Consumer or not. In either case, to recover the resources before
   reusing the QP, the Consumer should cause the QP to transition to
   Error state to flush the WQEs on the SQ and RQ, and then transition
   the QP back to the Idle state.

6.2.1.1  Idle to Idle

   The Modify QP Verb MUST allow a transition of the QP from the Idle
   state to the Idle state. This is to allow certain Queue Pair Context
   attributes to be modified in this state before an association with a
   Remote Peer's QP has been established.

6.2.1.2  Idle to RTS

   The Modify QP Verb MUST allow a transition from the Idle state to
   the RTS state. This is to support LLP Stream establishment. For this
   transition, the Modify QP Verb requires an LLP Stream Handle, and
   allows a Stream Message Buffer as well as other Input Modifiers. In
   order to transition from Idle to RTS, the LLP must be in its
   "Established" state, able to send and receive data. If not, the
   Modify QP Verb MUST return an Immediate Error. For more details on
   LLP Stream establishment, see Section 6.6.1 - Connection
   Initialization.

   The RI performs the following actions in the Idle to RTS transition,
   which MAY be performed in order:

   1.  The RI resets the RDMAP, DDP and MPA layers to the initial
       conditions specified in the appropriate specifications. For
       example, the DDP Untagged Message Sequence Numbers (MSN) for the
       Receive queue & IRRQ, and the MPA marker position must be reset
       as described in [RDMAP], [DDP], and [MPA].

   2.  If the Modify QP Verb includes a Stream Message Buffer to send,
       it is RECOMMENDED that the RI performs the following list in
       order:

      1. The implementation should stop receiving messages from the LLP
         Stream.





   Hilland, et al.        Expires October 2003              [Page 44]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

      2. The RI should transmit the specified message buffer to the
         Remote Peer in streaming mode.

      3. The RI should associate the LLP Stream with the RDMAP, DDP,
         and MPA layers and the RI should enable the QP to receive and
         transmit iWARP messages.

      4. The implementation should resume receiving messages from the
         LLP Stream.

   3.  If the Modify QP Verb does not include a Stream Message Buffer
       to send, the RI should associate the LLP Stream with the RDMA,
       DDP, and MPA layers and the RI should enable the QP to receive
       and transmit iWARP messages.

   4.  The RI moves the QP to the RTS state and begins normal
       operation.

   The RI MAY implement the Verb in other ways, but the end result
   MUST:

   1. Associate RDMAP and Lower layers with the QP;

   2. While in streaming mode, transmit any Stream Message Buffer that
      was included in the Modify QP;

   3. Ensure that the QP enables reception and transmission of iWARP
      messages; and,

   4. That regardless of how quickly the remote side returns the first
      iWARP message, ensure that messages MUST NOT be lost.

   For example, if the Verb did not stop the LLP receive side, the
   following race condition MUST be handled properly:

   1. The Associated QP transitions to RTS,

   2. It begins transmitting RDMA packets,

   3. Then the rapid arrival of an iWARP message from the Remote Peer
      occurs while the Local Peer is transitioning, but not completed
      the transition, to the RTS state.

   Note that the Modify QP Idle to RTS transition that includes a
   Stream Message Buffer to send may take a significant amount of time
   to complete. This is due to the requirement to reliably transmit the
   stream message.






   Hilland, et al.        Expires October 2003              [Page 45]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

6.2.1.3  Idle to Error

   The Modify QP Verb MUST allow the Consumer to modify the QP from the
   Idle state to the Error state.

   If it becomes necessary to remove WQEs posted to the queues in the
   Idle state, the Consumer may Modify the QP to the Error state, and
   then back to Idle. Any WQEs on the SQ & RQ will be Completed with a
   Flushed status by this procedure. This procedure will not change the
   Completion Status of CQEs already Completed on the CQ. The Consumer
   can then Poll for Completion on the Completion Queue and examine the
   Completion Status to determine which WRs were flushed.

   Note there is no effect on the LLP since no LLP Stream has been
   associated with the QP at this point.






































   Hilland, et al.        Expires October 2003              [Page 46]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Event                        Action                       Next
                                                             State

   PostSQ, PostRQ               Enqueue WQE                  Idle

   WQE is present on or added   WQE is NOT processed         Idle
   to the tail of the SQ

   Modify QP->Idle (Footnote 4)                              Idle

   Modify QP->RTS and Stream    Reset RDMAP and Lower layers RTS
   Message Buffer included      to their initial conditions.
                                Associate RDMAP and Lower
                                layers with QP. Transmit the
                                specified Stream Message
                                buffer in Streaming mode and
                                enable iWARP mode as
                                described in 6.2.1.2.

   Modify QP->RTS with NO       Reset RDMAP and lower layers RTS
   Stream Message Buffer        to their initial conditions.
   included                     Associate RDMAP and lower
                                layers with QP. Enable iWARP
                                mode.

   Modify QP->Error                                          Error

   PostSQ/PostRQ error          Return an Immediate Error    Idle

   Modify QP, results in error  Return an Immediate Error    Idle

                       Figure 7 - Idle State summary













   Footnote 4: This transition allows changing QP parameters as defined
   in Figure 4.




   Hilland, et al.        Expires October 2003              [Page 47]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

6.2.2  RTS (Ready to Send) State

   The RTS state is the main operational state for iWARP operation. All
   normal message processing, both incoming and outgoing, occurs in
   this state.

   The QP MUST be in the RTS state to begin transmitting and receiving
   any messages. Prior to moving to this state, the LLP Connection &
   LLP Stream MUST be fully established.

   Once in this state, any WQEs already posted on the Send Queue will
   begin processing. Any new WQEs posted MUST be added to the tail of
   the queue, (and begin processing, if the queue is empty). Once in
   this state, valid incoming iWARP Messages MUST be processed, placed
   and Completed. In this state, posted Receive WRs will be added to
   the Receive Queue (or S-RQ), processed when a Send Operation Type
   arrives, and Completed as described in Section 8.2.4 - Completed
   Work Requests.

   The RTS state MAY be left automatically by any of a variety of
   processing Errors, which will cause a transition to either the
   Terminate or Error state. See Section 8.3 - Error Handling for
   details on which errors result in transitioning to which state.

   The RI MUST return an Immediate Error if the Consumer attempts to
   transition the QP from the RTS state to the Idle state.

   A short summary table describing the state changes for RTS state is
   shown in Figure 8. Following are detailed descriptions of those
   changes.

6.2.2.1  RTS to RTS

   The Modify QP Verb MUST allow the Consumer to modify the QP from the
   RTS state to the RTS state. This allows certain QP parameters to be
   changed while the QP is Associated with another QP through an LLP
   Stream.

   Among the parameters that MAY be changed are IRD and ORD, the
   maximum number of WQEs supported by the SQ or RQ. A Consumer should
   take care when making changes to these parameters in order to
   prevent potential race conditions between the Modify operation, the
   posting of operations on the Send and Receive Queue, and incoming
   messages. For example, reducing the size of the Send or Receive
   Queue can only be done when there are fewer WQEs present on the
   queue than the new size. It is the responsibility of the consumer to
   track the number of outstanding WR on the SQ and RQ if it intends to
   modify the size of the SQ or the RQ. For IRD and ORD details, see
   Section 6.5 - Outstanding RDMA Read Resource Management.




   Hilland, et al.        Expires October 2003              [Page 48]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

6.2.2.2  RTS to Closing

   If the Remote Peer begins an LLP Close operation that does not
   include a Terminate Message (e.g. for TCP a FIN was received), the
   RI MUST cause the QP to leave the RTS state automatically. If all
   Send Queue Work Requests and Remote RDMA Read Operations (i.e.
   incoming RDMA Read Request Messages and associated RDMA Read
   Response Messages) are completed, the QP MUST transition to the
   Closing state; If this is not true, or a Terminate Message was
   received, the QP MUST transition to the Terminate state (see
   following section). In all of the above cases the RI MUST create an
   Affiliated Asynchronous Event to report the transition.

   The Modify QP Verb MUST allow the Consumer to modify the QP from the
   RTS state to the Closing state, to begin an LLP Close operation
   (e.g. for TCP a FIN segment is generated), and MUST NOT generate an
   Affiliated Asynchronous Event. See Section 6.6.2.1 - Normal Close
   for more details. When doing a Modify QP to Closing, all Send Queue
   Work Requests should have been previously Completed, any Remote RDMA
   Read Operations should have been previously finished, and the
   Consumer should have stopped posting PostSQ operations, so that no
   work remains for the QP to do. If this is not the case, the RI MUST
   ensure that either of the following actions are taken:

   *   The Modify QP MAY cause a transition to the Closing state which
       is immediately followed by a transition to the Error state (due
       to the SQ being non-empty).

   *   The Modify QP MAY cause a transition to the Closing state
       followed by a transition to the Idle state (because the SQ was
       originally empty, the LLP Close completed, causing the
       transition to the Idle state, and yet the Consumer was still
       posting SQ operations).

   If this Modify QP Verb completes without error, the QP has
   successfully transitioned to the Closing state (although it may have
   already transitioned out of the Closing state).

6.2.2.3  RTS to Terminate

   The Modify QP Verb MUST allow the Consumer to modify the QP from the
   RTS state to the Terminate state. This enables the Consumer to
   inform the Remote Peer that an Abnormal ULP Termination of the
   connected stream is being done. The Modify QP will result in the
   Error Code subfield of the Terminate Control Field of the Terminate
   Message (See [RDMAP]) having a value of 0x0000: Local Catastrophic
   Error. The Terminate Buffer will then be available to the Local node
   via Query QP and to the Remote Peer through Query QP (provided the
   Terminate Message arrives at and is processed by the Remote Peer).




   Hilland, et al.        Expires October 2003              [Page 49]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   When this Verb completes, the QP is in the Terminate state. For more
   details, see 6.6.2.2 - ULP Initiated Termination.

   The RTS to Terminate state transition MUST occur automatically
   following: a locally detected error; a Remote Peer beginning an LLP
   Close (e.g. for TCP a FIN was received) with either local Send Queue
   WQEs incomplete, or local Remote RDMA Read Operations incomplete;
   operation error; or any other error that would cause the RI to
   generate a Terminate Message. If the transition to the Terminate
   state is due to other locally detected errors, the RI MUST create
   the appropriate Asynchronous Error Event reporting that error. See
   Section 8.3.3 - Asynchronous Errors.

   The WR, if any, which caused the QP to enter into the Terminate
   state MUST be completed with the correct Completion Error Code for
   the error through the CQ associated with the WQ that experienced the
   error.

   If a remote Terminate Message is received, the Terminate state MUST
   be automatically entered and an Asynchronous Error Event MUST be
   reported with a status of "Termination Message Received". In this
   case, the RI MUST NOT send a Terminate Message back to the Remote
   Peer. Note that if TCP is the LLP, depending upon implementation of
   LLP Close, the RI may immediately transition to the Error state or
   it may wait for a TCP ACK before the transition.

6.2.2.4  RTS to Error

   The Modify QP Verb MUST allow the Consumer to modify the QP from the
   RTS state to the Error state. This enables the Consumer to perform
   an Abnormal ULP initiated Abortive Teardown (for more details, see
   Section 6.6.2.3 - ULP Initiated Abortive Teardown).

   An LLP failure that prevents further transmissions will also cause
   the RTS to Error transition.

   When the QP transitions from the RTS state to the Error state, the
   LLP stream MUST NOT be associated with the QP.

   The following are done prior to entering Error state:

   *   The RI MUST stop processing SQ WRs, Remote RDMA Read Operations,
       and any incoming iWARP Segments targeting the QP. See Section
       6.4 - Stopping QP processing and Sending the Terminate Message
       for additional information.

   *   If the LLP Stream has not closed, an LLP Reset MUST occur

   *   The LLP Stream resources MUST no longer be associated with the
       QP once the LLP actions, if any, are taken.



   Hilland, et al.        Expires October 2003              [Page 50]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   If this transition is due to a failure of the LLP, the RI MUST
       create an Asynchronous Error event reporting the error.

   When the prior items complete, the QP MUST be transitioned to the
   Error state.
















































   Hilland, et al.        Expires October 2003              [Page 51]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Event                        Action                     Next
                                                           State

   PostSQ, PostRQ               Enqueue WQE                RTS
   Valid iWARP Segment Arrives  Process Segment            RTS
   WQE is present on or added   Process WQE(s) and send    RTS
   to Send Queue                data (as necessary)
   Modify QP->Closing           Begin LLP Graceful Close   Closing
   Modify QP->RTS               Modify QP parameters as    RTS
                                document in Section 6.5
   Modify QP->Error             Stop QP processing,LLP     Error
                                Reset & LLP Disassociated
   Modify QP->Terminate         Generate Terminate Message  Terminate
   PostSQ/PostRQ error          Return an Immediate Error  RTS

   Modify QP, resulting in an   Return an Immediate Error  RTS
   Immediate Error
   LLP Failure that prevents    Stop QP processing, LLP    Error
   transmission of the          Reset, LLP Disassociated,
   Terminate Message            Create Asynchronous Error
   LLP Failure that allows      Generate Terminate Message Terminate
   transmission of the
   Terminate Message
   Local incoming RDMA Message  Generate Terminate Message Terminate
   processing error (RDMA Read
   Request, RDMA Read Response,
   or RDMA Write handling)
   Local incoming Send Type     Generate Terminate Message Terminate
   Message Processing Error
   Local WQ processing error    Complete WR as necessary,  Terminate
                                Generate Terminate Message
   Received Terminate Message                              Terminate
   LLP Close Received AND (SQ   Generate Terminate Message Terminate
   NOT empty OR IRRQ NOT empty)
   LLP Close Received AND SQ                               Closing
   empty AND IRRQ empty

                       Figure 8 - RTS State summary



   Hilland, et al.        Expires October 2003              [Page 52]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

6.2.3  Terminate State

   The Terminate state is used to send the final Terminate Message and
   begin an LLP Close if an error has occurred, or as a staging ground
   to perform an LLP Close if a Terminate Message was received from the
   Remote Peer. This state is transitory. The duration is limited by
   the time to finish the LLP Close operation or a final timeout in LLP
   Close (which would cause an LLP Reset).

   When the Terminate state is exited to the Error state, the LLP
   Stream MUST no longer be associated with the QP and the LLP Stream
   MUST be in either a condition of LLP Closed or LLP Reset.

   It is possible to examine the Terminate Message buffer while in this
   state by using Query QP (Section 9.2.5.2) to retrieve the Terminate
   Message.

   A short summary table describing the state changes for the Terminate
   state is shown in Figure 9. The following are detailed descriptions
   of those changes.

   While in the Terminate state, the following are done:

   *   The RI MUST stop processing SQ WRs, Remote RDMA Read Operations
       and any new incoming iWARP Segments targeting the QP. For
       additional information, see Section 6.4 - Stopping QP processing
       and Sending the Terminate Message.

   *   The RNIC MUST attempt to send the RDMAP Terminate Message,
       indicating the cause of error, except when the Terminate state
       is entered due to reception of a remote Terminate Message. Note
       that sending the Terminate Message may not be successful if an
       LLP Reset occurs.

   *   The RI MUST begin an LLP Close operation.

   *   If the current stream is the last (or only) active LLP Stream on
       the LLP Connection, or the LLP is in a state where all streams
       are unable to operate, the LLP Close MUST cause the LLP
       Connection to be closed. (For example, in [TCP] the FIN is sent
       and the close sequence is done.)

   *   If an LLP error occurs during the sending of the Terminate
       Message (including reception of an incoming LLP Reset, between
       the time the Terminate state is entered and the LLP Close
       sequence is completed), or due to an LLP final timeout while the
       LLP Close operation is not finished, then an LLP Reset MUST
       occur and its resources MUST no longer be associated with the
       QP. Note that the LLP MUST use a timeout to detect errors, so
       that the QP is in the Terminate state for a bounded time.



   Hilland, et al.        Expires October 2003              [Page 53]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   At some point in the Terminate state, the RI MUST begin to
       return an Immediate Error for any attempt to post a WR to a Work
       Queue; prior to that point, WQEs MUST be enqueued (and
       eventually flushed) or result in an Immediate Error.

   *   The RI MAY begin to flush any incomplete WRs on the SQ or RQ.
       Please see the Section 6.2.4 - Error State for further
       requirements about flushing incomplete WRs.

   *   When the prior actions are done:

       1.  If the transition to the Terminate state is due to the
           Modify QP Verb, the RI MUST NOT create an Asynchronous Error
           Event reporting "Error State Entered". If the transition to
           the Terminate state is due to the Modify QP Verb, but an LLP
           error occurred while in the Terminate state, then the RI
           MUST generate an Asynchronous Error reporting "Bad Close".

       2.  If the transition to the Terminate state is due to an error
           that is reported in a Work Completion, the RI MUST NOT
           create an Asynchronous Error. See Section 8.3.2 - Work
           Completion Errors. If the transition to the Terminate state
           is due to an error that is reported in a Completion, but an
           LLP error occurred while in the Terminate state, then the RI
           MUST generate an Asynchronous Error reporting "Bad Close".

   When the actions listed above are complete, and the LLP Close is
   finished, the QP state MUST move automatically to the Error state.

   When the LLP Close is finished or an LLP Reset occurs, the RI MUST
   disassociate the QP from the LLP Stream, including any LLP Stream
   context and any resources associated with it. Disassociating the LLP
   Stream from the QP means that it becomes possible for the QP to be
   transitioned to Idle and to RTS with a new LLP Stream.



















   Hilland, et al.        Expires October 2003              [Page 54]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Any attempt to perform a Modify QP in the Terminate state MUST
   return with an Immediate Error.

   Event                        Action                      Next
                                                            State

   On entry                     Stop QP processing,         Terminate
                                Send & attempt to complete
                                Terminate Message if one
                                wasn't received.
                                LLP Close Initiated

   LLP Close complete           Create Asynchronous Event   Error
                                if necessary,
                                LLP Disassociated from QP

   LLP Failure that prevents    LLP Reset and create        Error
   transmission of the          Asynchronous Event if
   Terminate Message            necessary,
                                LLP Disassociated from QP

   Valid IWARP Segment Arrives  Ignore Segment              Terminate

   PostSQ/PostRQ error          Return an Immediate Error   Terminate

   Modify QP                    Return an Immediate Error   Terminate

   WQE is present on or added   WQE is NOT processed and is Terminate
   to a Work Queue              eventually flushed.

                    Figure 9 - Terminate State summary




















   Hilland, et al.        Expires October 2003              [Page 55]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

6.2.4  Error State

   The Error state provides an indication that the QP has experienced
   an error (or transitioned to the Error state through the use of a
   Modify QP) and has stopped operations. On entry to the Error state,
   the LLP Stream MUST NOT be associated with the QP.

   The RI MUST return an Immediate Error if the Consumer attempts to
   transition the QP from the Error state to the RTS, Terminate, or
   Closing state.

   The following is done on entry into the Error state:

   *   The RI MUST flush any incomplete WRs on the SQ or RQ. All WQEs
       on the SQ and RQ, except for the WQE that caused the error (if
       any), MUST be returned with the Flushed Error Completion Status
       through the Completion Queue associated with the WQ. Note that
       the WQE which caused the error may not be at the head of the
       Work Queue. The Consumer should expect in some cases to retrieve
       Work Completions with the Flushed Error Completion Status, as
       well as potential successful completions, before retrieving the
       WC for the WR which caused the error. The RI MUST NOT return
       more than one Work Completion with a Work Completion Status set
       to something other than the Flushed Completion Status or the
       Success Completion Status.

   *   At some point in the execution of the flushing operation, the RI
       MUST begin to return an Immediate Error for any attempt to post
       a WR to a Work Queue; prior to that point, any WQEs posted to a
       Work Queue MUST be enqueued and then flushed as described above
       (e.g. The PostSQ is done in Non-Privileged Mode and the Non-
       Privileged Mode portion of the RI has not yet been informed that
       the QP is in the Error state).

   If a Terminate Message was sent or received, the RI MUST allow the
   Consumer to retrieve it through the Query QP Verb (Section 9.2.5.2).

   Following entry to the Error state, and before Destroying the QP or
   restarting the QP by going through Idle to RTS, it may be necessary
   to clean up some of the resources associated with the QP.

   *   Work Completions should be reaped by using Poll for Completion
       (Poll CQ) (see Section 9.3.2.1) before destroying the QP,
       otherwise they may become inaccessible.

   *   Memory Window resources MUST be deallocated by using Deallocate
       STag (see Section 9.2.6.4). This is necessary since in the Valid
       state they are associated with the QP. QP destruction will fail
       when Memory Windows which are in the Valid state are still Bound
       to the QP.



   Hilland, et al.        Expires October 2003              [Page 56]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   Memory Regions can be invalidated by posting an Invalidate Local
       STag WR to other SQs in the same PD, or they can be deallocated
       by using Deallocate STag. If left in the Valid state, the
       associated memory may be at risk of unexpected remote access.

   If the QP is transitioning to the Error state, or has not yet
   finished flushing the Work Queues, a Modify QP request to transition
   to the IDLE state MUST fail with an Immediate Error. If none of the
   prior conditions are true, a Modify QP to the Idle state MUST take
   the QP to the Idle state. No other state transitions out of Error
   are supported. Any attempt to transition the QP to a state other
   than Idle MUST result in an Immediate Error.

   A short summary table describing the state changes for Error state
   is shown in Figure 10.

   Event                        Action                       Next
                                                             State

   On Entry                     Flush any incomplete WQEs

   Modify QP->Idle                                           Idle
   (no outstanding WRs and
    not in transition to Error)

   Modify QP->Idle              Return an Immediate Error    Error
   (outstanding WRs or
    in transition to Error)

   Post WR                      Post WQE, and then Flush it, Error
                                OR
                                Return an Immediate Error

   Modify QP, resulting in an   Return an Immediate Error    Error
   error

                      Figure 10 - Error State summary














   Hilland, et al.        Expires October 2003              [Page 57]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

6.2.5  Closing State

   This state is used to wait for the LLP to complete the LLP Close, if
   no errors occurred. For some LLPs or some RI implementations, moving
   a QP from the RTS state to the Idle state can require an end-to-end
   acknowledgement or require the Remote Peer to close their half of
   the LLP Stream before the LLP Close is finished. This may take a
   significant amount of time. Thus the Closing state is provided so
   that these operations are done in a fashion that is visible to the
   Consumer. Note that some RI implementations may require the LLP
   Stream to be completely closed before transitioning to the Idle
   state. This can be in the order of tens of seconds (e.g. an RI
   implementation on TCP may require TCP to be in the CLOSED state,
   possibly waiting in the TIME-WAIT state for a significant amount of
   time).

   If the LLP Close operation does not require the LLP to transmit
   messages (e.g. for SCTP there is no mechanism to close a single LLP
   Stream, thus when one LLP Stream is closed and other LLP Streams
   remain active, there is no end-to-end handshake required), then the
   RI MAY transition rapidly through this state.

   When the Closing state is exited to Idle, the LLP Stream MUST NOT be
   associated with the QP.

   Any attempt to perform a Modify QP in the Closing state MUST return
   an Immediate Error.

   Errors detected by the RI when the QP is in the Closing state result
   in a transition to the Error state; for LLP failures, this is
   indicated with the specific Asynchronous Event "LLP Connection
   Lost".

   A short summary table describing the state changes for the Closing
   state is shown in Figure 11. Following are detailed descriptions of
   those changes.

   The following are done prior to exiting Closing state:

   *   The RI MUST stop processing SQ WRs and Remote RDMA Read
       Operations targeting the QP.

   *   The RI MUST stop processing any incoming segments, though the RI
       MAY process any arriving Terminate Messages.

   *   At some point in the Closing state the RI MUST begin to return
       an Immediate Error for any attempt to post a WR to a Work Queue;
       prior to that point, WQEs MUST be enqueued or result in an
       Immediate Error.




   Hilland, et al.        Expires October 2003              [Page 58]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   The RI MUST flush all incomplete WQEs on the RQ. All WQEs on the
       RQ MUST be returned with the Flushed Error Completion Status
       through the Completion Queue associated with the RQ. If RQ WQEs
       are enqueued, the RI MUST flush the WQE with the Flushed Error
       Completion Status through the Completion Queue associated with
       the RQ.

   *   If no errors have been detected (see next bullet), an LLP Close
       MUST occur. If the LLP Stream is the last or only active stream
       for the LLP Connection, the LLP Connection MUST be attempted to
       be closed gracefully. (For example, in [TCP] the FIN is sent and
       close sequence is done.).

   *   The RI MUST generate an Asynchronous Error if:

       o   Any SQ WQEs were on the SQ at any time during the Closing
           state. Note, this condition may happen if the PostSQ is done
           in Non-Privileged Mode and the Non-Privileged Mode portion
           of the RI has not yet been informed that the QP is in the
           Closing state. Also, the Error state will flush all SQ WQEs.

       o   Any incoming data arrives during the LLP Close. If the
           incoming data is a Terminate Message, the RI MAY allow the
           Consumer to retrieve the Terminate Message through the Query
           QP Verb.

       o   Any Remote RDMA Read Operations are in process.

       o   An LLP Stream failure (e.g. LLP Stream is lost) occurs
           during the LLP Close. Note that the RI MUST use a timeout
           mechanism to detect LLP errors during the LLP Close, so that
           the QP is in the Closing state for a bounded time. If the
           LLP detects a final timeout, it MUST be considered an error.

   *   If the RI generates an Asynchronous Error, the following MUST
       occur in order:

       o   An LLP Reset MUST occur and the LLP resources MUST no longer
           be associated with the QP.

       o   The QP MUST be transitioned to the Error state.

       o   The RI MUST generate an Asynchronous Event

   *   If no error occurs during the LLP Close operation:

       o   When all RQ WRs have been flushed and the LLP Close has
           finished, the LLP Stream MUST be disassociated with the QP,
           the RI MUST generate an Asynchronous Event "LLP Close
           Complete".



   Hilland, et al.        Expires October 2003              [Page 59]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   When the prior items complete, the QP MUST be transitioned
           to the Idle state.

   When the LLP Close is finished or an LLP Reset occurs, the RI MUST
   disassociate the QP from the LLP Stream, including any LLP Stream
   context and any resources associated with it. Disassociating the LLP
   Stream from the QP means that it becomes possible for the QP to be
   transitioned to Idle and to RTS with a new LLP Stream.

   Note that it is possible for the Consumer to post WRs while the
   automatic transition from RTS to Closing to Idle is occurring. See
   Section 6.2.1 - Idle State for additional details.









































   Hilland, et al.        Expires October 2003              [Page 60]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Event                       Action                        Next
                                                             State

   On Entry                    Stop QP processing, start LLP Closing
                               Close, and start Flushing any
                               incomplete WQEs on Receive
                               queues.

   LLP Close complete, all RQ  Create Asynchronous Event:    Idle
   WQEs flushed, and no SQ     "LLP Close Complete", LLP
   WQEs on the SQ              Disassociated from QP

   At least one SQ WQE on the  Perform LLP Reset, Create     Error
   SQ or Remote RDMA Read      Asynchronous Event: "Bad
   Operation in progress.      Close", LLP Disassociated
                               from QP

   LLP Connection Failure      Perform LLP Reset, Create     Error
                               Asynchronous Event: "LLP
                               Connection Lost", LLP
                               Disassociated from QP

   Segment Arrives and is not  Perform LLP Reset, Segment is Error
   a Terminate Message         not processed. Create
                               Asynchronous Event: "Bad
                               Close", LLP Disassociated
                               from QP

   Segment Arrives and is a    Perform LLP Reset, MAY create Error
   Terminate Message           Async Event: "Bad Close"; MAY
                               allow examination of
                               Terminate Message, LLP
                               Disassociated from QP

   PostSQ/PostRQ with          Return an Immediate Error     Closing
   Immediate Error

   Modify QP                   Return an Immediate Error     Closing

   PostRQ without Immediate    Enqueue and flush             Closing
   Error

   PostSQ without Immediate    Enqueue & Flush, Perform LLP  Error
   Error                       Reset, Create Async Event
                               "Bad Close", LLP
                               Disassociated from QP.

                     Figure 11 - Closing State summary




   Hilland, et al.        Expires October 2003              [Page 61]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

6.3  Shared Receive Queue

   The Verbs support a Shared Receive Queue (S-RQ). Support for the
   Shared Receive Queue is OPTIONAL. The Query RNIC Verb MUST indicate
   whether the RNIC supports the Shared Receive Queue.

   A Shared Receive Queue is an RNIC resource which allows multiple RQs
   to retrieve WQEs from the same shared queue on an as needed basis.
   This allows a Consumer to post WRs to the S-RQ instead of the RQ.
   When a message arrives, the RI uses a WQE from the S-RQ and makes it
   appear as if the WQE has been copied from the S-RQ to the QP's RQ. A
   CQE for an incoming message which result in a WQE being consumed
   from an S-RQ MUST be posted to the CQ associated with the QP's RQ.

   The RI MUST return the maximum number of S-RQs supported by the RI
   as an output modifier of Query RNIC, and the value MUST be zero if
   the RI does not support S-RQs.

   The RI MUST return the maximum number of outstanding WRs on an S-RQ
   as an output modifier of Query RNIC, and the value MUST be zero if
   the RI does not support S-RQs.

   Each S-RQ MUST be associated with a single PD ID. Multiple S-RQs
   MUST be able to be associated with the same PD ID.

   The SQ of a QP associated with an S-RQ MUST operate no differently
   than the SQ of a QP which is not associated with an S-RQ.

   When using an S-RQ, the RI MUST allow Work Requests to be posted to
   the S-RQ and MUST NOT allow WRs to be posted to an RQ of a QP
   associated with the S-RQ.

   If the RI supports an S-RQ, then it MUST:

   *   support the Create S-RQ Verb (See Section 9.2.4.1),

   *   support the Query S-RQ Verb (See Section 9.2.4.2),

   *   support the Modify S-RQ Verb (See Section 9.2.4.3),

   *   support the Destroy S-RQ Verb (See Section 9.2.4.4),

   *   support the S-RQ Handle as an Input Modifier for Create QP (See
       Section 9.2.5.1), and

   *   support an S-RQ Limit Event and a QP RQ Limit Event (See Section
       6.3.8),

   *   support the S-RQ Handle as an Input Modifier for PostRQ,




   Hilland, et al.        Expires October 2003              [Page 62]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   support the S-RQ Handle as an Asynchronous Event Handler routine
       parameter.

6.3.1  Creating a Shared Receive Queue

   When the S-RQ is created, it MUST be associated with a PD ID, and
   the maximum number of WRs which can be posted at any time must be
   provided as an Input Modifier. Note that the number of WQEs on the
   S-RQ at any given moment is dependent upon the completion semantics
   described below.

6.3.2  Modifying a Shared Receive Queue

   The RI MAY allow the Consumer to change the maximum number of
   outstanding WRs on the S-RQ. If the RI supports the ability to
   change the number of outstanding WRs on a SQ and RQ, and the RI
   supports S-RQs, then it MUST:

   *   allow the maximum number of outstanding WRs on the S-RQ to be
       changed;

   *   allow the maximum number of outstanding WRs to be changed while
       WRs are still outstanding; and

   *   support the ability to change this on every S-RQ.

   It is understood that changing the number of WRs that an S-RQ may
   have outstanding MAY adversely affect performance. Resizing the S-RQ
   MUST NOT cause Immediate, Completion or Asynchronous Errors, with
   the exception of Immediate Errors returned by the Modify S-RQ Verb
   and possible LLP time-outs. It is expected that the resize operation
   MAY adversely affect the Associated QPs attempting to communicate
   with the QPs associated with the S-RQ during the resize operation
   possibly resulting in LLP time-outs and retries which could result
   in LLP Stream teardown (which would result in an Asynchronous
   Error). It is suggested that the Consumer only perform this resize
   operation when activity on the connections has been quiesced to
   minimize the risk of transitioning Associated QPs to the Error state
   as a result of LLP time-outs.

   If the number of requested outstanding WRs is smaller than the
   actual number of outstanding WRs currently on the S-RQ, then the
   modification of the S-RQ MUST fail with an Immediate Error and the
   S-RQ MUST remain in the original state.

6.3.3  Destroying a Shared Receive Queue

   The Verbs provide a Destroy S-RQ Verb to allow a Consumer to destroy
   an S-RQ that is no longer needed. The RI MUST only allow an S-RQ to
   be destroyed when all the QPs associated with that S-RQ have been



   Hilland, et al.        Expires October 2003              [Page 63]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   destroyed. The RI MUST allow an S-RQ to be destroyed when there are
   WRs still posted to the S-RQ. Note that it is recommended that a
   Consumer drain the S-RQ or track all WRs posted to the S-RQ before
   destroying it so that no WRs are lost. For example, a WR which was
   Posted to the S-RQ but which was never Completed would still be on
   the S-RQ when the S-RQ was destroyed so the Consumer would never be
   notified that the buffers associated with the WR were available
   again.

   After the Destroy S-RQ returns to the Consumer, the RI:

   *   MUST have freed all RI resources associated with Receive Work
       Requests that were not Completed and were posted on that S-RQ,
       and

   *   MUST ensure that it will no longer reference any Consumer
       resources associated with Receive Work Requests that were not
       Completed and were posted on that S-RQ.

6.3.4  Associating an S-RQ with a QP

   A Shared Receive Queue MUST only be associated with a QP when the QP
   is created. When the QP is created, the RI MUST ignore the maximum
   number of outstanding RQ WRs Input Modifier.

6.3.5  Shared Receive Queue Processing Model

   If a QP is associated with an S-RQ, the RI MUST allow WRs to be
   posted to the S-RQ using PostRQ, specifying the S-RQ Handle instead
   of the QP Handle. If the QP is associated with an S-RQ, the RI MUST
   NOT allow WRs to be posted to the Local RQ through PostRQ and MUST
   return an Immediate Error if Posting to the Local RQ is attempted by
   the Consumer.

   The RI MUST ensure that S-RQs follow the rules for Work Queues with
   respect to the posting rules and completion rules defined in Section
   8.2.1 - Submitting Work Request to a Work Queue and Section 8.2.3 -
   Completion Processing. This means the RI MUST prevent a Consumer
   from overflowing the S-RQ using the PostRQ.

   When an incoming Untagged Message arrives on a QP, the RI determines
   if the QP is associated with an S-RQ. If it is, the RI must make it
   appear as if the WQE has been dequeued from the S-RQ and queued to
   the QP's local RQ. This does not guarantee that the S-RQ WQE is
   free. The S-RQ WQE is considered to be part of the S-RQ until the
   Work Completion associated with the S-RQ WQE has been retrieved or
   the S-RQ is destroyed.

   The RI MAY dequeue or use the S-RQ WQEs in any order. Since the WQEs
   are in an implementation specific order, the Consumer should not



   Hilland, et al.        Expires October 2003              [Page 64]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   depend on S-RQ post order in any way. The RI should support one of
   the following two models: sequential order or arrival order.

   *   In sequential ordering, the RI dequeues S-RQ WQEs as messages
       arrive. If messages arrive out of order, in addition to
       dequeueing the WQE required to place the data for that message,
       the RI also dequeues a WQE for each message with an MSN lower
       than the out-of-order message that has not arrived and does not
       yet have an associated WQE.

   *   In arrival ordering, the RI dequeues S-RQ WQEs as the messages
       arrive. If messages arrive out of order, only the WQE required
       to place the out of order message will be dequeued from the S-
       RQ. WQEs required to place data for the messages with an MSN
       lower than the out of order message will be dequeued from the S-
       RQ when those messages arrive.

   The RI MUST Complete incoming Send Message Types in the order they
   were Posted to the Associated QP's Send Queue. This means Work
   Completions retrieved from the CQ for any individual QP will be
   retrieved only in Message Sequence Number (MSN) order (see [DDP] for
   details). The RI MUST dequeue only one WQE from the S-RQ to place
   any message represented by a single MSN. Note that the Work
   Completions are not necessarily in the order in which the Send
   Message Types arrived, nor in the order the WQEs were posted to the
   S-RQ, nor in the order the WQEs were dequeued from the S-RQ.

   When a Work Completion which represents a WR originally submitted to
   an S-RQ has been returned to the Consumer via the Poll for
   Completion Verb, the RI MUST allow the Consumer to be able to post
   another Work Request to the S-RQ immediately.

   All QPs that use an S-RQ MUST be able to consume S-RQ WQEs, as long
   as the S-RQ has unconsumed WQEs available. If there are no S-RQ WQEs
   when an Untagged Message arrives on a QP which is associated with
   that S-RQ, then the LLP Stream MAY be Terminated. If the LLP Stream
   is not terminated, the reader should see Section 13.2 - Graceful
   Receive Overflow Handling for one implementation option.

   Protection Domain checking rules are slightly different for an S-RQ.
   An S-RQ MUST have a PD ID assigned as an Input Modifier for Create
   S-RQ. When an Untagged Message arrives and the QP has been
   determined to use an S-RQ for its incoming Untagged Message WQEs,
   then the PD ID of the STags in the WQEs MUST be validated against
   the PD ID of the S-RQ and MUST NOT be validated against the PD ID of
   the QP.

   Note that due to the Protection Domain checking rule above, the
   Consumer will not be able to invalidate an STag used by the S-RQ
   unless the S-RQ's PD is the same as the QP's PD, even if the QP uses



   Hilland, et al.        Expires October 2003              [Page 65]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   the S-RQ. This is because the PD used for comparison in Invalidation
   operations is that of the QP, not the S-RQ.

   The use of the STag of zero as part of a SGE in a WR MUST be
   validated by the RI based on the QP's attribute which indicates if
   it is allowed on the QP. If use of the STag of zero is not permitted
   on the QP and a WQE referencing STag zero is processed on the QP,
   the RI MUST return a Completion Error. Consequently, if the Consumer
   uses the STag of zero in S-RQ Work Requests and the S-RQ is accessed
   by QPs that have the use of STag of zero enabled as well as QPs that
   do not have the use of STag of zero enabled, then the QPs that do
   not have the use of STag of zero enabled will transition to the
   Error state as soon as they retrieve a WQE which contains an STag of
   zero.

6.3.6  S-RQ Error Semantics

   All errors encountered MUST be reported through Work Completions
   where possible. This is due to the semantic requiring the WQE to
   appear as if it had been on the QP's RQ. The exception is that a
   catastrophic S-RQ error MUST be reported as an Affiliated
   Asynchronous Error.

   Errors related to a connection for a QP associated with an S-RQ MUST
   NOT affect the S-RQ. Any WQEs already consumed by the QP from the S-
   RQ will be completed in error or flushed in the case of an LLP
   Stream error. Any other QPs associated with the S-RQ MUST remain
   unaffected by a local QP error.

   Errors related to a Work Request on an S-RQ will be posted to the CQ
   associated with the QP's RQ if they are processing errors, or
   returned as Verb results if they are Immediate Errors.

   In the case of a catastrophic S-RQ failure, any QP associated with
   the S-RQ will transition to the Terminate state when the QP attempts
   to dequeue a WQE from the S-RQ when handling an incoming Send Type
   Message. The resource ID returned by the Asynchronous Event Handler
   MUST be the QP ID. All outstanding WQEs on the QP will be flushed
   and an Affiliated Asynchronous Event: "S-RQ error on a QP" MUST be
   generated as part of the Terminate state transition.

   The RI MUST NOT flush the WQEs on an S-RQ which have not been used
   to Place incoming Untagged Messages when any associated QP
   transitions to the Terminate, Error or Closing states.

6.3.7  S-RQ Resource Sizing

   The Consumer is responsible for sizing the S-RQ and the CQs
   associated with the QP's RQs appropriately. The RI MUST ignore the
   sizing information provided for the QP's RQ when the QP uses an S-



   Hilland, et al.        Expires October 2003              [Page 66]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   RQ. The Consumer should note this fact when invoking the Create QP
   Verb using an S-RQ handle. In addition, S-RQs are subject to the
   Completion confirmation rules defined in Section 8.2.3 - Completion
   Processing. This means that the WR MUST be considered to be in the
   scope of the RI, and thus using a WQE on the S-RQ until the Work
   Completion has been retrieved. In addition, the RI MUST allow any
   single RQ to utilize all of the WQEs posted to an S-RQ . Note also
   that the RI is not required to perform CQ overflow detection.

   The RQ size Input Modifier is not used when a QP is associated with
   an S-RQ. In this case, the RQ has no defined size. It can be up to
   the size of the S-RQ. If the S-RQ is resized, any QP MUST be able to
   utilize all of the WQEs posted to the S-RQ. It is up to the
   implementation to process multiple messages in progress at one time.
   Note that the number of messages that can be in progress at once is
   limited by the S-RQ size, the LLP receive window, and possibly other
   factors.

6.3.8  S-RQ Limit Checking

   An RI that supports the S-RQ MUST support an S-RQ Limit
   Notification. An RI that supports S-RQ MUST support an S-RQ Limit
   input modifier on the Create S-RQ and Modify S-RQ Verbs to establish
   the value of the Limit. The S-RQ Limit detection MUST be armed by
   the RI upon creation of the S-RQ, if non-zero. This is only used for
   generation of the Affiliated Asynchronous Event and MUST NOT
   otherwise disrupt the QP operation. When the number of available (or
   unused) WQEs posted to the S-RQ drops below the S-RQ Limit, the RI
   MUST generate an Asynchronous Event and provide the S-RQ Handle as
   the Resource ID. This event will only be triggered once after it is
   armed and will not generate another event until the Consumer re-arms
   the event. The RI MUST allow the Consumer to re-arm this event
   through the use of Modify S-RQ. The RI MUST arm this event when the
   S-RQ is created if the S-RQ Limit is greater than zero. The RI MUST
   allow an already armed S-RQ Limit to be armed again. If the S-RQ
   Limit is armed for an S-RQ and the maximum number of outstanding WRs
   on the S-RQ is modified below S-RQ Limit, then the RI MUST return an
   Immediate Error indicating that an invalid Input Modifier was
   provided.

   An RI that supports the S-RQ MUST support a QP RQ Limit Notification
   for QPs associated with an S-RQ. The QP RQ Limit detection MUST be
   armed by the RI upon creation of the QP, if non-zero. The Consumer
   specifies the QP RQ Limit as part of either Create QP or Modify QP.
   This is only used for generation of the Affiliated Asynchronous
   Event and MUST NOT otherwise disrupt the QP operation. When the
   number of messages in progress on the QP (which is defined as
   messages being Placed, and thus have WQEs associated with them, but
   which have not yet had CQEs generated for the WQEs and thus have not
   been Delivered to the Consumer) exceeds the QP's RQ Limit, the RI



   Hilland, et al.        Expires October 2003              [Page 67]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   MUST generate an Asynchronous Event and provide the QP ID as the
   Resource ID. This event will only be triggered once after it is
   armed and will not generate another event until the Consumer re-arms
   the event. The RI MUST allow the Consumer to re-arm this event
   through the use of Modify QP. The RI MUST arm this event when the QP
   is created if the QP's RQ Limit is greater than zero. The RI MUST
   allow an already armed S-RQ Limit to be armed again. If the S-RQ
   Limit specified in the Create S-RQ or Modify S-RQ is greater than
   the maximum number of outstanding WRs on the S-RQ, then the RI MUST
   return an Immediate Error indicating that an invalid Input Modifier
   was provided.

   Note that neither Limit Notification forces Work Completions to be
   retrieved by the Consumer. Only retrieving the Work Completions
   allows the Consumer to Post additional WQEs to the S-RQ.
   Consequently, if separate Consumers are allowed to share an S-RQ,
   then one Consumer could consume all or part of the S-RQ entries if
   it does not retrieve Work Completions.

6.4  Stopping QP processing and Sending the Terminate Message

   Certain conditions require that QP operations be stopped, and a
   final Terminate Message be sent. Stopping WR processing on the QP
   and transmission of a Terminate Message are associated with QP state
   changes; the specific QP state transitions that require this are
   described in Section 6.2 - Queue Pair Resource States. When a QP
   must be stopped, either by a Modify QP Verb, or by QP state change
   due to an error, the following notes apply:

   1.  For Errors that do not impact the integrity of an outbound DDP
       Segment or for Modify QP Verb invocations that require stopping
       the QP, outbound processing MUST be stopped only on DDP Segment
       boundaries, in the absence of LLP errors. Any Terminate Message
       (if required) MUST be filled out as described in [RDMAP] and
       MUST be sent after the last complete outbound DDP Segment.

       For Errors that impact the integrity of an outbound DDP Segment
       that require stopping the QP:

       o   If the RI has not begun sending the DDP Segment, then
           outbound processing MUST be stopped before the DDP Segment
           is sent; and the Terminate Message and error code MUST be
           sent instead of the erroneous DDP Segment.

       o   If the RI has begun sending the DDP Segment, then outbound
           processing MUST be stopped immediately on the byte that
           experienced the error and the LLP Stream MUST be Reset.

   2.  For Errors or Modify QP Verbs (except for RTS to Closing
       transitions) that require stopping the QP, the RI MUST cease to



   Hilland, et al.        Expires October 2003              [Page 68]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       process inbound DDP Segments, at least by the time that any
       currently in-process DDP Segment has completed processing.

       The semantics of stopping QP processing and handling incoming
       DDP segments for Modify QP Verbs that require the transition
       from RTS to Closing are discussed at length in Section 6.2.5.

       Subsequent inbound DDP Segments (if any) are ignored and any
       inbound DDP Segments that have been Placed but not Delivered are
       never Delivered.

   3.  For Modify QP Verbs that require stopping the QP, the RI SHOULD
       stop outbound QP processing prior to sending any current DDP
       Segment to the LLP and MUST stop outbound QP processing at least
       by the time that any currently in-process outbound message has
       completed processing.

   4.  For Errors detected while creating RDMA Write, Send Type, or
       RDMA Read Type Work Requests, the RI MUST stop outbound QP
       processing prior to sending the current DDP Segment to the LLP.
       The Terminate Message and Error code MUST be sent instead of the
       original message (or DDP Segment). In this case, the [RDMAP]
       Terminate Message's Terminate Control Field is set to represent
       RDMA and the Error Type is set to represent Local Catastrophic
       Error.

   5.  For Errors detected while creating RDMA Read Responses to a
       Remote RDMA Read Operation, the RI MUST stop outbound QP
       processing prior to sending the erroneous DDP Segment to the
       LLP. The Terminate Message and Error code are sent instead of
       the erroneous RDMA Read Response Message.

   6.  For Errors detected while creating CQEs, or other reasons not
       directly associated with creating an outbound DDP Segment, the
       RI SHOULD stop outbound QP processing prior to sending any
       current DDP Segment to the LLP and MUST stop outbound QP
       processing at least by the time that any currently in-process
       outbound DDP message has completed processing. In this case,
       [RDMAP] Terminate Message's Terminate Control Field's Header
       Control Bits are all zero.

   7.  If an error is detected by an iWARP implementation while an
       incoming DDP Segment data is being Placed, the error actions
       (changing state, stopping the QP, etc.) MUST be delayed until
       after the segment is actually delivered by the LLP. If more than
       one error is detected on incoming segments, then the first DDP
       Segment Delivered with a detected error MUST result in the error
       actions. The first detected error MAY have been detected by the
       LLP, DDP Layer, or RDMA Layer. If, while waiting for Delivery of
       an incoming segment that contains an error, another error is



   Hilland, et al.        Expires October 2003              [Page 69]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       detected that is not associated with incoming segments (for
       example, an LLP error, Send Queue or RDMA Read Response
       processing error), then the RI MUST perform the actions for that
       error without waiting for Delivery of any other segments.

   8.  For errors detected on incoming DDP Segments (after they have
       been Delivered by the LLP), the Terminate Message MUST include a
       copy of the iWARP header from the DDP Segment in error (see
       [RDMA]).

   Below, in Figure 12, is a table which should indicate the values for
   the fields in the Terminate Control Field of the Terminate Message
   in [RDMAP].

                         Layer EType   Error HdrCt DDP   Term  Term
                                       Code        Seg.  DDP   RDMA
                                                   Lgt   Hdr.  Hdr.

For Modify QP from RTS   No Terminate Message is sent.
to Error

For Modify QP from RTS   RDMA  Local   None  000b  All   All   All
to Terminate             (0x)  Catast. (0x)        zeros zeros zeros
                                (0x)

For Errors detected      RDMA  Local   None  000b  All   All   All
while creating RDMA      (0x)  Catast. (0x)        zeros zeros zeros
Write, Send Type, or            (0x)
RDMA Read Request
Messages

For Errors detected      RDMA  Local   None  000b  All   All   All
while creating           (0x)  Catast. (0x)        zeros zeros zeros
completions, or other           (0x)
reasons not directly
associated with
creating an outbound
DDP Segment

For Errors detected      Depends on error, see [RDMAP] specification
processing or Placing    and/or Sections 8.3.2 & 8.3.3.
incoming Send Type,
RDMA Write, RDMA Read
Request or RDMA Read
Response Messages







   Hilland, et al.        Expires October 2003              [Page 70]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

                         Layer EType   Error HdrCt DDP   Term  Term
                                       Code        Seg.  DDP   RDMA
                                                   Lgt   Hdr.  Hdr.

For Errors detected      Depends on error, see [RDMAP] specification
while creating RDMA      and/or Sections 8.3.2 & 8.3.3.
Read Response Messages

For LLP layer errors     No Terminate Message is sent.
detected by an iWARP
implementation (e.g.
incoming LLP Reset
while QP in RTS)

Incoming Terminate Msg   No Terminate Message is sent.


                 Figure 12- Terminate Control Field Values

6.5  Outstanding RDMA Read Resource Management

   RDMA allows multiple RDMA Read Request Messages to be outstanding on
   a single LLP Stream. To enable this feature, the RNIC provides
   resources associated with both the inbound and outbound stream. For
   each outbound RDMA Read Request Message, the RNIC has some resources
   to track the request until a local Completion occurs. Similarly, for
   each inbound RDMA Read Request Message, the RNIC has an Inbound RDMA
   Read Request Queue (IRRQ) (associated with the DDP Queue Number of
   1) to store the state of the request until it has been satisfied by
   sending all of the requested data in the RDMA Read Response Message.
   The Input Modifier that specifies this value is called the Inbound
   RDMA Read Queue Depth (IRD).

   The Outbound RDMA Read Queue Depth (ORD) is the allocated number of
   outstanding RDMA Read Request Messages the RNIC is allowed to have
   outstanding at the Data Sink of an RDMA Read Operation. This is the
   resource used to track the request until a local Completion occurs.

   The Inbound RDMA Read Queue Depth (IRD) is the allocated number of
   incoming RDMA Read Request Messages a QP can support at the Data
   Source for an RDMA Read Operation. This is the resource used to
   track inbound RDMA Read Request Messages.

   An RNIC MUST implement these resources as either per QP resources,
   or shared per RNIC resources. Per QP means that the resources are
   tied to the QP and are most likely part of the QP Context. Per RNIC
   resources implies that the RNIC has a pool of such resources
   internally that it assigns to the QP based on the values of IRD and
   ORD associated with the QP.



   Hilland, et al.        Expires October 2003              [Page 71]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Query RNIC MUST return the type of resources the specified RNIC
   supports. The results are returned in the following Output Modifiers
   for Query RNIC:

   *   The maximum number of Inbound RDMA Read Request Queue messages
       that can be outstanding per RNIC. This is the per RNIC parameter
       that corresponds to IRD. This value is Zero if the resources for
       handling Inbound RDMA Read Requests are not shared between QPs.

   *   The maximum number of Outbound RDMA Read Request messages that
       can be outstanding per RNIC. This is the per RNIC parameter that
       corresponds to ORD. This value is Zero if the outstanding RDMA
       Read Requests are not shared between QPs.

   *   The maximum number of inbound RDMA Read Request Messages that
       the Inbound RDMA Read Request Queue can store per QP
       (corresponds to IRD).

   *   The maximum number of outbound RDMA Read Request Messages that
       can be outstanding per QP (corresponds to ORD).

   The Consumer is responsible for setting the RDMA Read Data Sink QP's
   ORD so that it does not exceed the Associated QP's IRD at the Data
   Source.

   If the Consumer attempts to set IRD or ORD to one or greater, and
   there are not enough resources to allow this, the Create QP or
   Modify QP Verb MUST fail with an Immediate Error. This can happen
   because the maximum amount of IRD/ORD resources returned by Query
   RNIC MAY be affected by consumption of unrelated resources, so that
   not all of the reported resources may actually be available
   simultaneously.

   If the IRD and ORD resources are not shared between QPs (e.g. fixed
   per QP instead of allocated out of a pool for the RNIC), then the
   ULP need only negotiate the values for IRD and ORD. But if the IRD
   and ORD resources are shared across the RNIC, then some function of
   the Consumer or Consumer's environment (such as a resource manager)
   must determine how to allocate the resources among the QPs in
   addition to negotiating the IRD and ORD values.

   The RNIC MUST ensure that it does not issue more RDMA Read Request
   Messages than is specified by the QP's ORD value. However, the RI
   MUST allow the Consumer to post as many RDMA Read Type Work Requests
   as it can, within the limit of the total Work Requests the Send
   Queue can support. The RI MUST delay processing of an RDMA Read Type
   Work Request posted to the SQ which would result in exceeding the
   QP's ORD value until a prior RDMA Read Type Work Request Completes.





   Hilland, et al.        Expires October 2003              [Page 72]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   The rules in Section 8.2.2 enable subsequent Work Requests to be
   executed before the RDMA Read Type Work Request Completes. If
   however, a delay in processing occurs due to waiting for a prior
   RDMA Read Type Work Completion, this will effectively prevent
   subsequent Work Requests from being executed until the delay is over
   (i.e. stall Send Queue processing). If the Consumer wants to avoid
   this type of delay in Send Queue processing, it can issue up to as
   many RDMA Read Work Requests as supported by the value of ORD for
   that QP, and when each one Completes, then add an additional RDMA
   Read Type Work Request.

   The Consumer should manage the number of RDMA Read Request Messages
   outstanding, either by correctly setting the QP's ORD value to be
   less than or equal to the Associated QP's IRD value, or by limiting
   the number of RDMA Read Type Work Requests the Consumer posts on the
   Send Queue at any one time to be less than or equal to the
   Associated QP's IRD value. If this is not done correctly, the Local
   Peer may attempt to send more RDMA Read Request Messages than the
   Remote Peer can accept, which will result in an error from the
   Remote Peer that Terminates the RDMAP Stream (See Section 6.6.2.4 -
   Remote Termination).

   The RDMA Read Resources (IRD and ORD) MUST be initialized at QP
   creation (Create QP). The RDMA Read Resources MAY be changed while
   the QP is in Idle state and when the QP is in the RTS state. If the
   Consumer changes the resources while the QP is in the RTS state, the
   Consumer should ensure that no RDMA Read Operations are outstanding
   for the affected direction (outbound for ORD, inbound for IRD). If
   the Consumer modifies the RDMA Read Resources when RDMA Read
   Operations are outstanding, the QP state MAY be indeterminate and
   the RI MUST NOT adversely affect any other QPs supported by the RI.
   Changing RDMA Read Resources when RDMA Read Operations are not
   outstanding is easily done if IRD and ORD are set before any RDMA
   Read Work Requests are posted by either Peer. If RDMA Read Work
   Requests have already been posted, it is up to the Consumer to
   ensure that they have all Completed before changing IRD or ORD or
   the QP may be in an indeterminate state.

   The following semantics are required of the RI:

   *   All RNICs MUST allow the Consumer to reduce the ORD in the IDLE
       and RTS states.

   *   It is OPTIONAL for an RI to allow the Consumer to increase IRD
       or ORD after the QP has been created.

   *   It is OPTIONAL for an RI to accept reductions of IRD from the
       Consumer after the QP has been created.





   Hilland, et al.        Expires October 2003              [Page 73]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   The RNIC MUST support a total number of inbound RDMA Read
       Request Messages and outbound RDMA Read Request Messages, so
       that each is at least equal to the total number of QPs supported
       by the RNIC. The RNIC thus MUST be able to support at least
       IRD=1 and ORD=1 for each QP.

   *   RNICs that implement shared "per RNIC" RDMA Read Resources for
       IRD and ORD, MUST have enough so that all of the QPs can be
       assigned a value of one for IRD and one for ORD. It is up to the
       resource manager to allocate these resources fairly, so that
       applications that need RDMA Read Resources can be assured of
       their availability.

   Note that the maximum amount of resources returned by Query RNIC may
   be adversely affected by consumption of unrelated resources, so that
   not all of the reported number may actually be available
   simultaneously.

   If the Consumer attempts to set either IRD or ORD to one or greater,
   and there are not enough resources to allow this, the Create QP or
   Modify QP Verb MUST fail with an Immediate Error.

   Note that when using "per RNIC" resources, the Create or Modify QP
   IRD and ORD values are also limited by the "per QP" resources.

6.5.1  Example IRD/ORD Negotiation

   The example in Figure 13 shows one possible negotiation for a single
   direction (if the ULP uses RDMA Read Operations in both directions
   on the RDMA Stream, it must also do the same thing in reverse). Note
   that the last step may be omitted if the ULP is not interested in
   reducing the resources used at the left side of the connection when
   the right side supports less.




















   Hilland, et al.        Expires October 2003              [Page 74]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003








             < Figure 13 did not convert properly from source >
             <  to be corrected in an upcoming version        >






           Figure 13 - An example RDMA Read Resource negotiation

6.6  Connection Management

6.6.1  Connection Initialization

   RDMA Stream initialization can occur as the transport connection is
   created or sometime thereafter. In the latter case, the connection
   may require a ULP supplied end-to-end handshake before iWARP is
   initialized. Either the active or passive side of the connection may
   initiate turning on iWARP.

   In either case, the ULP must know, before iWARP mode is to begin,
   which model of operation is to be used by the ULP.

   An RI MUST support RDMA Stream initialization sometime after the
   transport connection is established and some streaming mode data has
   been sent.

   An RI MAY support RDMA Stream startup along with the transport
   connection, with no streaming mode data sent. This option is more
   completely described in Section 13.1 - Connection Initialization at
   LLP Startup.

   Once iWARP initialization is complete, the RI MUST allow only iWARP
   messages to be sent across the LLP connection until the RDMA Stream
   is torn down.





   Hilland, et al.        Expires October 2003              [Page 75]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Section 6.6.1.1 and 6.6.1.2 provide informative examples of methods
   for the ULP to transition to RDMA mode. Other implementations are
   possible.

6.6.1.1  Active Connection Initialization after LLP Startup

   For this discussion, the Active side goes to iWARP mode first. In
   the figures below, the thin lines represent TCP Streaming mode and
   the thick lines represent iWARP mode.







             < Figure 14 did not convert properly from source >
             <  to be corrected in an upcoming version        >

















          Figure 14 - Connection Initialization after LLP Startup

   Below is the sequence for an active side iWARP startup. Note that
   the dotted line arrows above indicate messages that may not be
   needed for some implementations.

   1.  The ULP establishes the LLP Connection and LLP Stream.

   2.  The active side ULP ensures that the passive side is able to
       enter iWARP mode via some negotiation or other mechanism, which
       is outside the scope of this specification.




   Hilland, et al.        Expires October 2003              [Page 76]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   3.  The active side Consumer creates a QP, setting up the CQ, PD
       etc., and registers memory for buffers. Note that in some
       instances, this may have been done at some previous time during
       the initialization process.

   4.  The active side Consumer posts receive buffers via PostRQ that
       are appropriate for the expected traffic. A first message may
       arrive quickly after the transition to RTS.

   5.  The active side Consumer moves the QP to the RTS state. The
       Consumer includes the LLP Stream Handle in the Modify QP Verb,
       and a single message buffer which contains the last streaming
       mode message to be sent to the Remote Peer. The RI uses the
       presence of this message buffer to recognize the Active startup
       sequence. For information on implementing this state transition,
       see Section 6.2.1.2 - Idle to RTS.

   6.  When the active side Consumer receives the first RDMA/DDP
       Message from the passive side (e.g. a Send type message), the
       active side Consumer is free to post additional Work Requests to
       the Send Queue. The active side Consumer should not have posted
       any SQ WRs while the QP was in the Idle state, or while the QP
       is in the RTS state. The active side consumer should not post
       any SQ WRs until the first RDMA/DDP Message is received. If the
       Consumer posts SQ WRs during either of these times, the Remote
       Peer is likely to improperly synchronize to the LLP Stream and
       to Terminate the LLP Stream. One way that the Consumer can
       determine that the message arrives is to have the initial
       message sent from the Associated QP have the Solicited Event bit
       set, thus generating an event at the Local Peer.

   7.  If the local Consumer intends to perform RDMA Read Operations,
       the local Consumer obtains, by some ULP defined message, the
       number of Incoming RDMA Read Request Messages that the Remote
       Peer can have outstanding (IRD). If the Remote Peer's IRD is
       smaller than the local Peer's ORD, the local Consumer should
       also perform a Modify QP Verb with the Remote Peer's IRD placed
       into the local ORD prior to posting the first RDMA Read Type WR.
       The local Consumer may also transmit, in some ULP defined
       message, the number of Outbound RDMA Read Request Messages that
       the Local Peer can have outstanding (ORD).

   8.  If the local ULP intends the QP to be a target of RDMA Read
       Operations, the local Consumer provides, in some ULP defined
       mechanism, the number of Inbound RDMA Read Request Messages that
       the Local Peer can have outstanding (IRD). The Consumer may also
       receive, by some ULP defined mechanism, the Number of Outbound
       RDMA Read Request Messages that the Remote Peer can have
       outstanding (ORD). If the Remote Peer's ORD is smaller than the
       Local Peer's IRD and the Local RNIC supports IRD reduction, the



   Hilland, et al.        Expires October 2003              [Page 77]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       local Consumer could perform a Modify QP Verb with the Remote
       Peer's ORD placed into the local IRD prior to posting the first
       RDMA Read Type WR.

6.6.1.2  Passive Connection Initialization after LLP Startup

   Below is the sequence for a passive side iWARP startup:

   1.  The passive side ULP establishes the LLP Connection and LLP
       Stream.

   2.  The passive side ULP informs the active side that it is able to
       enter iWARP mode via some negotiation.

   3.  The passive side ULP waits for the Active side to send a last
       streaming mode message to indicate that it should enter RDMA
       mode and that the remote node is in RDMA mode. When that message
       arrives, and if it indicates that iWARP mode is desired, the
       passive side Consumer continues with the items below.

   4.  The passive side Consumer creates a QP, setting up the CQ, PD
       etc. Note that this may have been done previously.

   5.  The passive side Consumer posts receive buffers appropriate for
       the expected traffic to the RQ.

   6.  The passive side Consumer posts at least one Send type Work
       Request that is used by the active side to complete the
       negotiation. The WR may contain any data that the ULP needs to
       communicate.

       Note: the passive side Consumer may delay the posting of buffers
       and Work Requests until after the transition to RTS, described
       below.

   7.  The passive side Consumer moves the QP to RTS state, specifying
       the LLP Stream Handle. The passive side Consumer does not
       include a last streaming mode message buffer in the Modify QP
       Verb; if it does, the Remote Peer is likely to improperly
       synchronize to the RDMA Stream and be forced to terminate the
       LLP Stream.

   8.  The passive side Consumer may now begin posting additional Work
       Requests.

   9.  If the local Consumer intends to perform RDMA Read Operations,
       the local Consumer obtains, in some ULP defined message, the
       number of incoming RDMA Read Request Messages that the Remote
       Peer can have outstanding (IRD). If the Remote Peer's IRD is
       smaller than the local Peer's ORD, the local Consumer should



   Hilland, et al.        Expires October 2003              [Page 78]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       also perform a Modify QP Verb with the Remote Peer's IRD placed
       into the local ORD prior to posting the first RDMA Read Type WR.
       The local Consumer may also transmit, in some ULP defined
       message, the number of outgoing RDMA Read Request Messages that
       the Local Peer can have outstanding (ORD).

   10. If the local Consumer intends the QP to be a target of RDMA Read
       Operations, the Consumer provides, in some ULP defined message,
       the number of incoming RDMA Read Request Messages that the Local
       Peer can have outstanding (IRD). The Consumer may also receive,
       in some ULP defined message, the number of outgoing RDMA Read
       Request Messages that the Remote Peer can have outstanding
       (ORD). If the Remote Peer's ORD is smaller than the Local Peer's
       IRD, the local Consumer may also perform a Modify QP Verb with
       the Remote Peer's ORD value placed into the local IRD prior to
       posting the first RDMA Read Type WR, if the RI supports IRD
       reduction.

6.6.2  Connection Teardown

   Five types of iWARP and LLP connection teardown mechanisms are
   supported:

   *   A normal close is an LLP Close that finishes with no errors (see
       Section 6.2.5 - Closing State, for a list of possible errors).
       This is used when the Consumers on both sides of the connection
       have sent their last message and wish to close the LLP Stream
       (see Section 6.6.2.1 - Normal Close).

   *   A ULP initiated Termination is used when the ULP desires to
       perform an LLP Close with an error message to the Associated QP
       (see Section 6.6.2.2 - ULP Initiated Termination).

   *   A ULP initiated Abortive Teardown is used when the ULP wishes to
       perform an LLP Reset with no error message to the Associated QP
       (see Section 6.6.2.3 - ULP Initiated Abortive Teardown).

   *   Remote Termination occurs when the RI receives a Terminate
       Message from the Associated QP, and the LLP Close process has
       begun (see Section 6.6.2.4 - Remote Termination).

   *   Local Termination, Local Abortive Teardown and Remote Abortive
       Teardown occur when the RI or LLP Stream detects an error and a
       Terminate Message is sent prior to an LLP Close or an LLP Reset
       is initiated (see Section 6.6.2.5 - Local Termination, Local
       Abortive Teardown and Remote Abortive Teardown).

   Sections 6.6.2.1 through 6.6.2.5 provide informative examples of
   methods for the ULP to terminate an RDMA Stream. Other
   implementations are possible.



   Hilland, et al.        Expires October 2003              [Page 79]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

6.6.2.1  Normal Close

   A normal close is provided as a mechanism for the ULP to cease
   activity, flush any receive buffers that have been posted to the RQ,
   and disassociate the LLP Stream from the QP. It requires that no
   errors occur during the close process. If an error occurs, it is now
   an abnormal close, which would cause the QP to transition to the
   Error state.

   The Consumer initiates a normal close, either locally or remotely,
   when both sides of a LLP Stream agree to the close.

   When the Consumer desires a normal close, the following items must
   be done:

   1.  The Consumer waits for all outstanding Work Requests on the Send
       Queues on both sides of the LLP Stream to be Completed. Note:
       the Completion on the remote WQ can be inferred by the arrival
       of a SEND message from the ULP that indicates that it intends to
       do no more work.

   2.  One of the Consumers moves the QP state to Closing with the
       Modify QP Verb, resulting in the following actions:

       o   If any WQEs are present on the Send Queue, or if any RDMA
           Read Operations are incomplete on the IRRQ, an error will
           result (for more information, see Section 6.2.5 - Closing
           State).

       o   The RI stops QP processing and flushes all incomplete WQEs
           on the Receive Queue by Completing them with the Flushed
           Completion Status.

       o   The RI performs an LLP Close. If this QP was using the last
           LLP Stream on the LLP Connection, the RI closes the LLP
           Connection.

       o   When the LLP Close actions are complete, the RI
           automatically moves the QP to the Idle state and an
           Affiliated Asynchronous Event: "LLP Close Complete" is
           created.

   3.  The Consumer may re-use the QP for a new LLP Stream or it may
       destroy the QP (see Section 6.1.3 - Modifying Queue Pair
       Attributes and Section 6.1.4 - Destroying a Queue Pair).

   The normal close may also be initiated remotely (e.g. for TCP a FIN
   segment is received). If the Send Queue is empty and the IRRQ is
   empty, the RI moves the QP state to the Closing state and an




   Hilland, et al.        Expires October 2003              [Page 80]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Asynchronous Event: "LLP Close Complete" will be generated. If this
   is the last LLP Stream, the LLP Connection will be closed.








             < Figure 15 did not convert properly from source >
             <  to be corrected in an upcoming version        >













                      Figure 15 - Normal Close on TCP

6.6.2.2  ULP Initiated Termination

   A ULP initiated Termination is usually used when the Consumer (such
   as the OS) detects an error. The ULP needs to perform an LLP Close,
   but would like to let the Remote Peer know that an error occurred.
   Note that an ULP initiated termination may entail loss of data.

   When the ULP desires a ULP initiated Termination, the following
   items must be done:

   1.  The Consumer modifies the QP to the Terminate state.

       o   Before returning from the Modify QP -> Terminate, the RI
           stops QP processing, formats a Terminate Message containing
           the termination code: "Local Catastrophic Error" and sends
           it to the Remote Peer.

       o   The RI performs an LLP Close. If the LLP cannot deliver the
           Terminate Message, an LLP Reset is performed, and the RI
           generates an Asynchronous Error Event: "Bad Close".



   Hilland, et al.        Expires October 2003              [Page 81]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   2.  After returning from the Modify QP -> Terminate, the Consumer
       waits for the QP to automatically be moved to the Error state.
       This is signaled by an Asynchronous Error Event: "Error State
       Entered".

   3.  Once in the Error state, the RI flushes all incomplete WQEs on
       both the Send and Receive Queues by completing them with the
       Flushed Completion Status. The Consumer would presumably reap
       all of the Work Completions to ensure all resources are cleaned
       up. Once the Consumer believes all Work Completions have been
       reaped, it should attempt to transition the QP to the Idle state
       by performing a Modify QP. If the transition is successful, the
       Consumer knows it can either re-use the QP for another LLP
       Stream or call Destroy QP (see Section 6.1.3 - Modifying Queue
       Pair Attributes and Section 6.1.4 - Destroying a Queue Pair). If
       the Modify QP returns with an error (presumably because Work
       Requests are still being flushed), the Consumer must try at a
       later time to transition to the Idle state. The Consumer might
       arm a timeout. If the Consumer is unable to transition to the
       Idle state after some amount of time, it should destroy the QP
       (presumably because the QP can not recover from an internal
       error).

6.6.2.3  ULP Initiated Abortive Teardown

   A ULP initiated Abortive Teardown is usually used when the Consumer
   (such as the OS) detects an error, and the ULP needs to tear down
   the entire LLP Stream immediately (i.e. perform an LLP Reset). Note
   that a ULP initiated abortive teardown may entail loss of data.

   When the ULP desires an Abnormal ULP initiated Abortive Teardown,
   the following items must be done:

   1.  The Consumer modifies the QP to the Error state.

       o   The RI stops QP processing and performs an LLP Reset.

   2.  Once in the Error state, the RI flushes all incomplete WQEs on
       both the Send and Receive Queues by completing them with the
       Flushed Completion Status. The Consumer would presumably reap
       all of the Work Completions to ensure all resources are cleaned
       up. Once the Consumer believes all Work Completions have been
       reaped, it should attempt to transition the QP to the Idle state
       by performing a Modify QP. If the transition is successful, the
       Consumer knows it can either re-use the QP for another LLP
       Stream or it can invoke Destroy QP (see Section 6.1.3 -
       Modifying Queue Pair Attributes and Section 6.1.4 - Destroying a
       Queue Pair). If the Modify QP returns with an error (presumably
       because Work Requests are still being flushed), the Consumer
       must try at a later time to transition to the Idle state. The



   Hilland, et al.        Expires October 2003              [Page 82]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       Consumer might arm a timeout. If the Consumer is unable to
       transition to the Idle state after some amount of time, it
       should destroy the QP (presumably because the QP can not recover
       from an internal error).

6.6.2.4  Remote Termination

   Remote Termination occurs when the Associated QP sends a Terminate
   Message to the Local Peer. Note that remote termination may entail
   loss of data.

   When the Remote Peer sends a Terminate Message, and it is locally
   received, the following sequence occurs:

   1.  The RI stops QP processing.

   2.  The RI moves the QP automatically to the Terminate state. The RI
       then generates an Asynchronous Error Event: "Terminate Message
       Received".

   3.  The RI performs an LLP Close, or if an LLP final timeout occurs,
       an LLP Reset.

   4.  The RI moves the QP to the Error state.

   5.  Once in the Error state, the RI flushes all incomplete WQEs on
       both the Send and Receive Queues by completing them with the
       Flushed Completion Status. The Consumer would presumably reap
       all of the Work Completions to ensure all resources are cleaned
       up. Once the Consumer believes all Work Completions have been
       reaped, it should attempt to transition the QP to the Idle state
       by performing a Modify QP. If the transition is successful, the
       Consumer knows it can either re-use the QP for another LLP
       Stream or it can invoke Destroy QP (see Section 6.1.3 -
       Modifying Queue Pair Attributes and Section 6.1.4 - Destroying a
       Queue Pair). If the Modify QP returns with an error (presumably
       because Work Requests are still being flushed), the Consumer
       must try at a later time to transition to the Idle state. The
       Consumer might arm a timeout. If the Consumer is unable to
       transition to the Idle state after some amount of time, it
       should destroy the QP (presumably because the QP can not recover
       from an internal error).

6.6.2.5  Local Termination, Local Abortive Teardown and Remote Abortive
         Teardown

   iWARP defines an abortive teardown mechanism which is invoked if a
   catastrophic iWARP error is encountered locally. iWARP attempts to
   send a Terminate Message, but depending upon the condition of the
   LLP, it is possible a Terminate Message can not be sent or can not



   Hilland, et al.        Expires October 2003              [Page 83]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   be successfully delivered to the Associated QP. If an LLP Stream
   error occurs, it is possible for the LLP Stream or LLP Connection to
   be torn down before a) iWARP is aware of the error, b) before iWARP
   is able to send the Terminate Message, or c) after iWARP has posted
   the Terminate Message to the LLP, but it is still in the LLP send
   queue. Thus the Consumer at the Remote Peer may or may not be able
   to retrieve a valid Terminate reason for some forms of abortive
   teardown. The Consumer at the Remote Peer can retrieve the Terminate
   Message, if available, using the Query QP when the QP has
   transitioned to the Error state. The Consumer at the Local Peer
   should always be able to retrieve the Terminate Message that was
   sent (if the QP transitioned through the Terminate state),
   regardless of whether it was successfully delivered to the Remote
   Peer.

   Note that an abortive teardown may entail loss of data. The RI will
   complete all outstanding (incomplete) iWARP messages in error. In
   general, when an abortive teardown occurs it is impossible to tell
   for sure what iWARP messages were successfully placed and delivered
   at the Remote Peer. Thus even completed messages on the Send Queue
   should be treated as incomplete unless a ULP Acknowledge has been
   received. Note that Completed RDMA Read Type Work Requests act as a
   ULP Acknowledgement, in that any prior RDMA Write Messages, Send
   Type Messages, RDMA Read Operations and the RDMA Read Request
   Message itself are required to have arrived at the Remote Peer
   before the RDMA Read Response Message can be generated at the Remote
   Peer to Complete the RDMA Read Type Work Request.

   When iWARP detects a local error the following items are done:

   1.  If the LLP Stream is still functional, the RI moves the QP to
       the Terminate state. If the error was not reported in a CQE, the
       RI generates an Asynchronous Error Event, with an appropriate
       error code (see 8.3.3 - Asynchronous Errors). Then the RI stops
       QP processing.

       If the LLP Stream is not functional, the RI performs an LLP
       Reset and moves the QP to the Error state. If the error was not
       reported in a CQE, the RI generates an Asynchronous Error Event,
       with an appropriate error code (see 8.3.3 - Asynchronous
       Errors). The RI skips steps 2 and 3 below.

   2.  The RI formats a Terminate Message with an appropriate
       termination error code and sends it to the Remote Peer.

   3.  The RI performs an LLP Close. If the LLP could not successfully
       perform the LLP Close (e.g. for TCP, transitioning through the
       normal closing states incurred a final timeout), an LLP Reset
       occurs. Once either the LLP Close or LLP Reset is finished, the
       RI transitions the QP to the Error state.



   Hilland, et al.        Expires October 2003              [Page 84]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   4.  Once in the Error state, the RI flushes all incomplete WQEs on
       both the Send and Receive Queues by completing them with the
       Flushed Completion Status. The Consumer would presumably reap
       all of the Work Completions to ensure all resources are cleaned
       up. Once the Consumer believes all Work Completions have been
       reaped, it should attempt to transition the QP to the Idle state
       by performing a Modify QP. If the transition is successful, the
       Consumer knows it can either re-use the QP for another LLP
       Stream or it can invoke Destroy QP (see Section 6.1.3 -
       Modifying Queue Pair Attributes and Section 6.1.4 - Destroying a
       Queue Pair). If the Modify QP returns with an error (presumably
       because Work Requests are still being flushed), the Consumer
       must try at a later time to transition to the Idle state. The
       Consumer might arm a timeout. If the Consumer is unable to
       transition to the Idle state after some amount of time, it
       should destroy the QP (presumably because the QP can not recover
       from an internal error).

   Figure 16 is an example of how the abortive teardown might occur.
   Other sequences of events are possible. For example, the TCP FIN
   could be sent in a separate TCP segment. Another example is the
   Remote Peer RI might not transition from the Terminate state when
   the LLP can no longer be used for data transmission (i.e. the TCP
   FIN ACK segment is sent). Instead it waits for TCP finite state
   machine to reach the Closed state. If the latter implementation is
   used, QP resources may not be able to be recycled until after TCP
   finishes transitioning through the TIME-WAIT state, which takes a
   considerable amount of time. See Section 10, Security
   Considerations, for potential security issues with this approach.
























   Hilland, et al.        Expires October 2003              [Page 85]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003









             < Figure 16 did not convert properly from source >
             <  to be corrected in an upcoming version        >











               Figure 16 - Abortive Teardown example on TCP
























   Hilland, et al.        Expires October 2003              [Page 86]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

7  Memory Management

7.1  Memory Management Overview

   There are two basic methods for enabling memory to be accessed by an
   RNIC. These are Memory Regions and Memory Windows. Memory Regions
   are used to assign an STag to a Physical Buffer List, associate it
   with a starting Tagged Offset and length, and assign it Memory
   Access Rights. Memory Windows are used to assign an STag to a
   portion, or window, of a Memory Region.

   Fundamental to Memory Management is the definition of an STag (see
   Section 7.2 - Steering Tag (STag)) and the Tagged Offset (TO)
   associated with it (see Section 7.3.1.1 - Memory Region Tagged
   Offset (TO) and Section 7.6.1 - Addressing Registered Memory). Also
   fundamental is the concept of a Physical Buffer List (PBL), which
   contains the physical address mappings for the memory used in the
   Memory Region, as discussed in Section 7.6.2 - Physical Buffer
   Lists.

   An STag can be associated with either a Memory Region or a Memory
   Window. While both Memory Regions and Memory Windows can be used for
   data transfer operations, they differ with respect to the Verbs used
   to manipulate them. These distinctions are covered in great detail
   in this section.

   There are three mechanisms for associating a Memory Region's STag
   with a Physical Buffer List. A Consumer can allocate an STag with
   the PBL in one step, as is done with RI-Register Non-Shared Memory
   Region. A Consumer can also allocate an STag and then use a Fast-
   Register WR to associate the PBL with the STag. Finally, a Consumer
   can create a new STag that is associated with an existing Memory
   Region through the Register Shared Memory Region. For more
   information on Memory Region creation, see Section 7.3.2 - Memory
   Region Creation and Registration.

   There are two types of Memory Regions. These are Non-Shared MR and
   Shared MR. A Non-Shared MR has a PBL that is not shared with other
   MRs. A Shared MR has a PBL that may be shared with other MRs. A Non-
   Shared MR becomes a Shared MR through the Register Shared Memory
   Region operation. For more information on Shared Memory Regions, see
   Section 7.3.2.4 - Register Shared Memory Region. MR (without any
   qualifiers) is used to refer to both Non-Shared MR and Shared MRs.

   Before use, Memory Windows must first be allocated and then Bound to
   a Memory Region. The allocation is a RI Verbs call, but the Bind
   operation is a WR. For more information on Memory Windows, see
   Section 7.10 - Memory Windows.





   Hilland, et al.        Expires October 2003              [Page 87]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Memory registration enables access to a Memory Region by a specific
   RNIC. Binding a Memory Window enables the specific RNIC to access
   memory represented by that Memory Window. STags are specific to an
   RNIC and the RI is NOT REQUIRED to grant access to the Memory Region
   by other local RNICs.

   Mechanisms are provided for Re-registering Non-Shared Memory
   Regions. These are discussed in Sections 7.3.2.3 - RI-Reregister
   Non-Shared Memory Region. In addition, the Verbs provide mechanisms
   for Registering Memory Regions which share PBL mappings. These are
   discussed in Section 7.3.2.4 - Register Shared Memory Region.

   Architecturally, only Bind Memory Window and Fast-Register Non-
   Shared Memory Region are anticipated to be optimized for
   performance. The rest of the Memory Registration mechanisms are not
   anticipated to be performance optimized.

   All Memory Regions MUST have Access Rights associated with them to
   indicate if local read, local write, remote read and remote write
   accesses are allowed. This is discussed in Section 7.4 - Access to
   Registered Memory. All Memory Windows MUST have Access Rights
   associated with them to indicate if remote read and remote write
   accesses are allowed. This is discussed in Section 7.4 - Access to
   Registered Memory.

   Non-Shared Memory Regions and Memory Windows have to be invalidated
   before they can have their PBL associations changed. This has other
   benefits as well, such as preventing remote accesses using that
   STag. This is discussed is Section 7.8 - Invalidating Memory Regions
   and 7.10.4 - Invalidating or De-allocating Memory Windows.

   The RI also provides Verbs for retrieving STag attributes, as
   discussed in Section 7.7 - Querying Memory Regions and 7.10.3 -
   Querying Memory Windows. The Verbs also define the destruction and
   deallocation of Memory Windows and Memory Regions in Section 7.9 -
   Deallocation of STag associated with a Memory Region and in Section
   7.10.4 - Invalidating or De-allocating Memory Windows, respectively.

7.2  Steering Tag (STag)

   All local and remote memory accesses through the Verbs require the
   use of an STag. For local access, the STag, along with a Tagged
   Offset (TO) is used by the RI, when processing a Work RequestÆs SGE,
   to identify a memory location within a specific Memory Region. For
   remote access, the STag, along with a TO, is used by the RI when
   handling RDMA operations to identify a memory location within a
   specific Memory Region or Memory Window.

   An STag is a 32-bit identifier that has two sub-fields: a Consumer
   provided STag Key and an RI provided STag Index. The STag Key is the



   Hilland, et al.        Expires October 2003              [Page 88]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   8 least significant bits of the STag. The STag Index is the 24 most
   significant bits of the STag.

   The 8 bit STag Key is provided by the Consumer. The Consumer can use
   the STag Key in any way it desires. For example, it can be used as
   an incrementing value to help discover application errors by using a
   different value with each registration. As a general rule, the
   Consumer provides the STag Key to the RI whenever the consumer
   causes the transition of an STag to the Valid state, or when the
   STag is being Invalidated. In the Invalid state, only the STag Index
   is meaningful.

   There is no default value for the STag Key. The RI MUST use the STag
   Key provided by the Consumer for the following Verbs:

   *   Register Non-Shared Memory Region,

   *   Register Shared Memory Region,

   *   Reregister Non-Shared Memory Region,

   *   PostSQ Verb Fast-Register Non-Shared Memory Region operation,
       and

   *   PostSQ Verb Bind operation,

   *   PostSQ Invalidate Local STag.

   The RI MUST return the value of the STag Index sub-field on an
   invocation of the following:

   *   Allocate Non-Shared Memory Region STag,

   *   Allocate Memory Window,

   *   Register Non-Shared Memory Region,

   *   Register Shared Memory Region, and

   *   Reregister Non-Shared Memory Region.

   The RI MUST use the same STag Index sub-field as was passed in by
   the Consumer, on an invocation of the following:

   *   Query Memory Region,

   *   Query Memory Window,

   *   Register Shared Memory Region,




   Hilland, et al.        Expires October 2003              [Page 89]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   Reregister Non-Shared Memory Region,

   *   PostSQ Fast-Register Non-Shared Memory Region,

   *   PostSQ Bind Memory Window,

   *   PostSQ Invalidate Local STag, and

   *   Deallocate STag.

   Implementation Note: To guarantee that the immediately previous STag
   is no longer valid, the Consumer may change the STag Key field each
   time the STag is bound. The use of a suitable random number with
   each binding can provide a valuable interface check and diagnostic
   tool.

7.2.1  STag of zero

   The STag of zero (STag with a value of zero) is a special STag. It
   has a fixed value for the STag Index and STag Key. The STag Key is
   composed of all zeros and the STag Index is composed of all zeros.
   It has no PD associated with it and it cannot be used for Remote
   Access operations.

   The purpose of an STag of zero is to allow Privileged Mode Consumers
   to be able to reference a Physical Buffer in a WR without first
   registering the buffer with the RI. This approach has the advantage
   of reduced overhead. It has the potential disadvantage that the
   buffer is represented by only a single SGE and therefore must be
   contiguous. Note that buffers which are not contiguous can be
   represented by multiple SGEs in this case, but all SGLs have a
   finite limit of the number of entries allowed by the RI. If the
   buffer is not physically contiguous, any access to the non-existent
   memory may result in an access error.

   Using an STag of zero as part of a Scatter/Gather Element tells the
   RNIC that it MUST interpret the TO portion of the SGE as a physical
   address on the local node. Note the RI MUST never generate an STag
   Index of zero. The RI MUST NOT allow the Consumer to associate an
   STag Key with the STag of zero.

   The STag of zero has the following semantics, which are different
   than the semantics of any other STag:

   1.  The RNIC MUST NOT perform any PD checks on an STag of zero.

   2.  When accessing an STag of zero on a given QP, the RNIC MUST
       assure access to the STag of zero is enabled on that QP. If
       allowing an STag of zero is not enabled, then the operation MUST
       result in a protection error.



   Hilland, et al.        Expires October 2003              [Page 90]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   3.  The RNIC MUST NOT permit any remote access that references STag
       of zero and any attempt to do so MUST result in a protection
       error. The RI MUST grant STag of zero Local Read and Local Write
       Access Rights.

   4.  The RNIC MUST NOT allow Memory Windows to be Bound to STag of
       zero. Any attempt to do so MUST result in an error.

   5.  The RNIC MUST NOT allow a Local or Remote Invalidation of the
       STag of zero. Any attempt to do so MUST result in an error. The
       STag of zero MUST always be in the Valid state.

   6.  The RNIC MUST NOT allow an STag of zero to be an input modifier
       of an RI-Reregister Non-Shared Memory Region, Register Shared
       Memory Region, Query Memory Region, Query Memory Window, Bind
       Memory Window, Deallocate STag, Invalidate STag or Fast-Register
       and MUST return an Immediate Error if a Consumer attempts to do
       so.

   7.  The RI MUST NOT return a value of zero as an STag Index for RI-
       Register Non-Shared Memory Region, RI-Reregister Non-Shared
       Memory Region, Register Shared Memory Region, Allocate Non-
       Shared Memory Region STag and Allocate Memory Window.

7.2.2  Summary of Memory Region STag States

   The STag associated with a Non-Shared Memory Region has two states.
   They are Invalid and Valid. Memory accesses MUST NOT be allowed if
   the STag is in the Invalid state.

   Below in Figure 17 is the Memory Region and Memory Window state
   diagram. It indicates the state transitions required to change
   Memory Regions and Memory Windows from the Valid state to and from
   the Invalid state. In addition, it denotes the effects of the
   Register Shared Memory Region Verb on a Memory Region.


















   Hilland, et al.        Expires October 2003              [Page 91]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003













             < Figure 17 did not convert properly from source >
             <  to be corrected in an upcoming version        >


























            Figure 17 - Memory Region and Window State Diagram

   For a Non-Shared Memory Region, the following bulleted list
   indicates the state, if memory access is allowed in that state, and
   what Verbs are used to enter and exit the specified state.



   Hilland, et al.        Expires October 2003              [Page 92]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   Invalid - May not be used to access a memory location.

       o   Entered through: Allocate Non-Shared Memory Region STag,
           PostSQ Invalidate STag, incoming Send with Invalidate STag
           Message, incoming Send with Solicited Event and Invalidate
           STag Message, or local RDMA Read with Invalidate Local STag
           WR.

       o   Exited through: RI-Register Non-Shared Memory Region, RI-
           Reregister Non-Shared Memory Region, Fast-Register Non-
           Shared Memory Region WR, or Deallocate STag.

   *   Valid - May be used to access a memory location.

       o   Entered through: RI-Register Non-Shared Memory Region, RI-
           Reregister Non-Shared Memory Region, Fast-Register Non-
           Shared Memory Region WR.

       o   Exited through: PostSQ Invalidate STag, incoming Send with
           Invalidate STag Message, incoming Send with Solicited Event
           and Invalidate STag Message, local RDMA Read with Invalidate
           Local STag WR, or Deallocate STag.

   Note: Deallocate STag exits the state logic captured above, as does
   RI-Reregister Non-Shared Memory Region (if a different STag is
   returned).

   The STag associated with a Shared Memory Region MUST always be in
   the Valid state. Note that the Register Shared Memory Region Verb
   does two things - it returns a new Shared Memory Region STag for an
   existing Memory Region's Physical Buffer List (either Shared or Non-
   Shared), and if the input STag is for a Non-Shared MR, the Non-
   Shared MR is permanently converted into a Shared MR (See Section
   7.3.2.4 - Register Shared Memory Region). The following bulleted
   list indicates what Verbs are used to enter and exit the Valid state
   for a Shared Memory Region.

   *   Valid - May be used to access a memory location.

       o   Entered through: Register Shared Memory Region.

       o   Exited through: Deallocate STag.

   Note: Deallocate STag of a Non-Shared MR MUST exit the state logic
   captured above.

7.3  Memory Registration

   Memory Registration provides mechanisms that allow Consumers to
   describe a set of virtually contiguous memory locations or a set of



   Hilland, et al.        Expires October 2003              [Page 93]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   physically contiguous memory locations to the RI in order to allow
   the RNIC to access either as a virtually contiguous buffer using the
   STag and Tagged Offset.

   Memory Registration provides the RNIC with a mapping between a STag
   and Tagged Offset and a Physical Memory Address. It also provides
   the RNIC with a description of the access control associated with
   the memory location.

   Before using a data buffer with the RI, all Consumers MUST
   explicitly register with the RI the memory locations associated with
   the data buffer, except when using an STag of zero. Local or remote
   attempts to access unregistered memory MUST result in a protection
   error. Thus every WR simply uses an STag, TO and length to reference
   a buffer.

   Memory Registration MAY fail due to the RNICÆs inability to find
   resources to hold information needed by the RNIC to record the
   registration. Memory MUST NOT be registered in this case and MUST
   NOT consume any RI resources if the Registration fails.

7.3.1  Memory Regions

   A set of memory locations that have been registered are referred to
   as a Memory Region (MR).

   The RNIC uses two values to identify a memory location within a
   Memory Region: Steering Tag (STag) and Tagged Offset (TO).

7.3.1.1  Memory Region Tagged Offset (TO)

   The base of the TO field is specified by the Consumer when the
   Memory Region is registered through RI-Register Non-Shared Memory
   Region, RI-Reregister Non-Shared Memory Region, or Fast-Register
   Non-Shared Memory Region. Two bases MUST be supported by the RNIC:
   Virtual Address (VA) based TO and zero based TO. For a VA based TO,
   the TO of the first memory location associated with the Memory
   Region equals the VA value passed as an input modifier of the Verb
   or WR used to register the Memory Region. For a zero based TO, the
   TO of the first memory location associated with the Memory Region
   equals zero.

7.3.2  Memory Region Creation and Registration

   Before the RNIC can use a Memory Region, the resources associated
   with a Memory Region must be allocated and the Memory Region must be
   registered with the RNIC. The RI defines the following mechanisms
   for providing these functions through the Verbs interface: Allocate
   Non-Shared Memory Region STag, Register Shared Memory Region, RI-




   Hilland, et al.        Expires October 2003              [Page 94]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Register Non-Shared Memory Region, RI-Reregister Non-Shared Memory
   Region, and Fast-Register Non-Shared Memory Region.

   When registering a Memory Region, the Consumer specifies whether
   Memory Windows may be Bound to the Memory Region or not.

7.3.2.1  Allocate Non-Shared Memory Region STag

   This Verb allocates memory registration resources in the RI. When
   the Verb completes, the STag Index will be allocated as described
   below and provided as an output modifier.

   When allocating an STag:

   *   the RI MUST verify the Consumer specified maximum Physical
       Buffer List Size is less than or equal to the size allowed by
       the RI. The RI MUST return the Physical Buffer List (PBL) size
       allocated, which MUST be greater than or equal to the size
       requested. The RI MUST also return the allocated STag Index. If
       the Consumer specified a maximum PBL Size greater than the size
       allowed by the RI, the RI MUST return an Immediate Error.

   *   the RI MUST verify and use the Consumer specified Input Modifier
       called the Remote Access Flag to indicate if Remote Access is
       enabled with the STag. If the Remote Access Flag is enabled, the
       RI MUST be able to allow remote reads or remote writes that
       reference the STag. Otherwise, the RI MUST NOT allow the STag to
       be used in remote read or remote write operations.

   An STag created through the Allocate Non-Shared Memory Region STag
   Verb MUST be able to be used in an RI-Reregister or a Fast-Register
   Non-Shared Memory Region.

   When the Allocate Non-Shared Memory Region STag Verb returns control
   to the Consumer and the Verb has completed successfully, the
   returned STag is in the Invalid state. The STag MUST be placed in
   the Valid state before it can be used by a local or remote operation
   to access a memory location. See Section 7.2.2 - Summary of Memory
   Region STag States for the requirements on transitioning the STag to
   the Valid state.

   For a description of the Verb which Allocates an STag, see Section
   9.2.6.1 - Allocate Non-Shared Memory Region STag.

7.3.2.2  RI-Register Non-Shared Memory Region

   When the RI-Register Non-Shared Memory Region Verb returns, it has
   allocated the appropriate memory registration resources on the RNIC
   and has registered a Non-Shared Memory Region. When the RI-Register
   Non-Shared Memory Region Verb is invoked:



   Hilland, et al.        Expires October 2003              [Page 95]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   The RI MUST accept and use any STag Key passed in by the
       Consumer for the Memory Registration.

   *   The RI MUST use the Physical Buffer List passed in by the
       Consumer.

   *   The RI MUST verify and use the Consumer specified modifier which
       indicates if Remote Access is enabled with the STag. If Remote
       Access is enabled, the RI MUST allow remote reads or remote
       writes that reference the STag. Otherwise, the RI MUST NOT allow
       the STag to be used in remote read or remote write operations.

   When the RI-Register Non-Shared Memory Region Verb completes
   successfully:

   *   the RI MUST have Registered the Non-Shared Memory Region with
       the RNIC,

   *   the RI MUST return the STag Index associated with the Non-Shared
       Memory Region to the Consumer,

   *   the RI MUST return the number of Physical Buffer List Entries in
       the allocated Physical Buffer List, which may be larger than the
       requested size, and

   *   the returned STag MUST be in the Valid state.

   See Section 9.2.6.2 - Register Non-Shared Memory Region (RI-
   Register) for a description of the RI-Register Non-Shared Memory
   Region Verb.

7.3.2.3  RI-Reregister Non-Shared Memory Region

   This Verb conceptually performs the functional equivalent of
   Deallocate STag followed by RI-Register Non-Shared Memory Region.
   Where possible, resources below the Verb layer are expected to be
   reused instead of deallocated and reallocated. This Verb may be used
   to change the Access Rights and/or PD ID of a Region, as well as
   changing the memory locations that are registered.

   When the RI-Reregister Non-Shared Memory Region Verb is invoked:

   *   The STag MUST be the STag of a Non-Shared Memory Region.

   *   The STag MUST be in either the Invalid or Valid state.

   *   The RI MUST accept and use any STag Key passed in by the
       Consumer for the Memory Reregistration.





   Hilland, et al.        Expires October 2003              [Page 96]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   The RI MUST ensure that no Memory Windows are Bound to the STag
       Index passed in by the Consumer. If any Memory Windows are Bound
       to it, an Immediate Error is returned.

   *   The STag passed in by the Consumer MAY have an original PBL size
       that is smaller than the new PBL size to be associated with that
       STag. If the PBL passed in by the Consumer is greater than the
       PBL associated with the STag, the RI MAY return an error
       indicating it had insufficient resources to complete the
       request.

   If the RI-Reregister Non-Shared Memory Region Verb does not complete
   successfully:

   *   If the RI returns an "Invalid RNIC handle", "Invalid STag Index"
       or "One or more Memory Windows is still Bound to the Region"
       Immediate Error, the RI MUST make no changes to the current
       registration (assuming that it even exists).

   *   If the RI returns any error other than "Invalid RNIC handle",
       "Invalid STag Index" or "One or more Memory Windows is still
       Bound to the Region", the RI MUST Deallocate the Memory Region
       associated with the STag Index used as an Input Modifier and
       ensure that no new Memory Region is registered.

   When the RI-Reregister Non-Shared Memory Region Verb completes
   successfully:

   *   the RI MUST have registered the Non-Shared Memory Region with
       the RNIC;

   *   the RI MAY return a different STag Index than the one passed in
       by the Consumer. If a different STag Index is returned, all
       resources associated with the prior STag MUST have been
       effectively Deallocated (e.g. transition to the Deallocated
       state);

   *   the RI MUST return the number of Physical Buffer List Entries in
       the allocated Physical Buffer List, which may be larger than the
       requested size,

   *   the RI MUST use and set the Remote Access Rights and Remote
       Access Flag for the STag as indicated with the Input Modifier,
       and

   *   the returned STag MUST be in the Valid state. This STag can be
       used to access a memory location.

   The Consumer should note that since the STag Index returned MAY be
   different than the STag Index provided to the Verb, any attempt to



   Hilland, et al.        Expires October 2003              [Page 97]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   use the previous STag Index in this case would result in a memory
   protection error.

   The RI-Reregister Non-Shared Memory Region Verb can be used to
   modify the attributes of a Memory Region created through the RI-
   Register Non-Shared Memory Region, RI-Reregister Non-Shared Memory
   Region, or an Allocate Non-Shared Memory Region STag Verb. A Memory
   Region MUST be allowed to be reregistered an arbitrary number of
   times provided the PBL length is less than or equal to the original
   PBL length.

   For the error case where a Remote Peer is accessing a Non-Shared
   Memory Region while it is in the process of being reregistered,
   implementations MUST present the same semantics as a deallocate or
   invalidate operation followed by a separate registration operation.

   For information on the Verb to Reregister a Memory Region, see
   Section 9.2.6.5 - Reregister Non-Shared Memory Region (RI-
   Reregister).

7.3.2.4  Register Shared Memory Region

   Shared Memory Regions provide a way for the Consumer to obtain a new
   STag Index for a Memory Region that has already been registered.
   This allows optimization of RNIC resources because returning a new
   STag Index allows the Consumer to assign different Access Rights,
   change the VA Base, change if the Region is VA Based or Zero Based,
   assign an STag Key and use a different PD, but use the same Physical
   Buffer List as a previously registered Memory Region. Thus an
   optimized implementation is possible where the new STag can use the
   previous PBL for memory translation but has new STag properties for
   Access Rights and Protection Domain checks.

   When the Shared Memory Region Verb is invoked:

   *   If the STag Index, passed in by the Consumer, is associated with
       a Non-Shared Memory Region, the RI MUST verify that the Memory
       Region STag Index passed in is in the Valid state. Note that
       Shared Memory Regions are always in the Valid state.

   *   Any Memory Windows that are currently bound to the MR,
       associated with the STag Index passed in by the Consumer, MUST
       be unaffected.

   *   The RI MUST verify that the STag Key of the existing MR matches
       the STag Key supplied as an input modifier by the Consumer.

   *   The RI MUST accept and use any STag Key passed in by the
       Consumer for the Shared Memory Registration.




   Hilland, et al.        Expires October 2003              [Page 98]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   If the STag Index passed in by the Consumer references a VA
       based TO, the RI MUST verify that the VA passed in by the
       Consumer produces an FBO that matches the FBO of the PBL that is
       associated with the STag Index passed in by the Consumer.

   When the Shared Memory Region Verb completes successfully:

   *   the RI MUST have registered the new Shared Memory Region with
       the RNIC;

   *   the RI MUST return a different STag Index that is associated
       with the same or identical PBL as the PBL referenced by the STag
       Index passed in by the Consumer;

   *   The RI MUST allow the new Shared Memory Region to have different
       Access Rights, change the VA Base, change if the Region is VA
       Based or Zero Based, assign an STag Key and a different PD; and

   *   if the STag Index passed in by the Consumer is associated with a
       Non-Shared Memory Region, the RI MUST convert the Non-Shared
       Memory Region to a Shared Memory Region but MUST NOT change any
       other attributes of the Memory Region being converted.

   The returned STag, which references the new, Shared Memory Region,
   is in the Valid state. The STag can be used to access a memory
   location.

7.3.2.5  Fast-Register Non-Shared Memory Region

   Fast-Register provides a mechanism for the Consumer to use the
   PostSQ Verb to invoke an asynchronous memory registration. Fast-
   Register Non-Shared Memory Region MUST support registration using
   STags that were created with the Allocate Non-Shared Memory Region
   STag, RI-Register Non-Shared Memory Region Verb or RI-Reregister
   Non-Shared Memory Region Verb and have not subsequently been
   converted to a Shared Memory Region.

   When the Fast-Register Non-Shared Memory Region mechanism is
   invoked:

   *   The RI MUST accept and use any STag Key passed in by the
       Consumer for the Fast-Register operation.

   *   The RI MUST use the STag Index passed in by the Consumer to
       register a Non-Shared Memory Region with the RNIC.

   *   The RI MUST verify that the STag Index passed in by the Consumer
       is in the same PD as the QP. The RI MUST verify that the STag
       Index passed in by the Consumer is not the STag of zero. The RI
       MUST verify that the STag Index passed in by the consumer is not



   Hilland, et al.        Expires October 2003              [Page 99]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       the STag of a Memory Window. If the STag Index is not in the
       same PD as the QP or the STag is that of a Memory Window or the
       STag is the STag of zero, the RI MUST return an error.

   *   The STag MUST be in the Invalid state at the time the Fast-
       Register Non-Shared Memory Region is processed. See Section
       7.2.2 - Summary of Memory Region STag States for more details.
       If the STag is not in the Invalid state at the time the Fast-
       Register Non-Shared Memory Region WR is processed, the RI MUST
       return an error.

   *   If the Non-Shared Memory Region referenced by the STag does not
       have a maximum PBL size greater than or equal to the PBL size
       passed in the Fast-Register Non-Shared Memory Region, the RI
       MUST return an error.

   *   The RI MUST prevent an STag with the Remote Access Flag disabled
       from having its Access Rights changed to include remote Access
       Rights. The RNIC MUST assure an STag with the Remote Access Flag
       enabled can have its Access Rights changed to include remote and
       local, or local only Access Rights. Note that the Remote Access
       Flag cannot be changed except by the RI-Reregister Non-Shared
       Memory Region Verb. If Remote Access Rights are requested and
       the Remote Access Flag is not enabled, the RI MUST return an
       error.

   *   The RI MUST verify that Fast-Register access is enabled on the
       QP that is processing the Fast-Register Non-Shared Memory Region
       operation. Note that this is intended to prevent a Non-
       Privileged Mode application from accessing physical memory
       without Privileged Mode intervention. If Fast-Register is not
       enabled on the QP, the RI MUST return an error.

   The Fast-Register operation MUST take place within the RI at any
   time between when the Work Request is posted and before execution of
   the Work Request immediately after the Fast-Register operation.

   When the Fast-Register Non-Shared Memory Region operation completes
   successfully, the associated STag MUST be in the Valid state. The
   STag can be used to access a memory location.

   For a description of the Fast-Register Non-Shared Memory Region
   mechanism, see Section 9.3.1.1 - PostSQ.

7.4  Access to Registered Memory

   The RI MUST support four distinct Memory Region Access Rights: Local
   Read, Local Write, Remote Read, and Remote Write. The Access Rights
   of the Memory Region MUST apply to each memory location within the
   Memory Region. The RI MUST allow changing Access Rights from local



   Hilland, et al.        Expires October 2003             [Page 100]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   to local and remote only through an RI-Reregister or through a
   Deallocate followed by an Allocate or RI-Register.

   The RI MUST support a Remote Access Flag. It can be supplied as an
   Input Modifier for the Allocate STag, RI-Register and RI-Reregister
   Verbs. If the Remote Access Flag is enabled, the RI MUST allow the
   remote Access Rights to be set on the STag. If the Remote Access
   Flag is disabled, the RI MUST not allow the remote Access Rights to
   be set on the STag.

   When performing local and remote data transfer operations, the RI
   MUST validate all 32 bits of the STag used to represent the data
   transfer.

7.4.1  Local Access to Registered Memory

   The RI MUST allow the Consumer to assign one or both of the Local
   Access Rights to a given Memory Region. If the Consumer does not
   assign one of the local Access Rights, the RI MUST return an error.

   If the RI assigns Local Read Access to a Memory Region, the RNIC is
   allowed to use the STag and Tagged Offset to read any location
   within the Memory Region. If the RI assigns Local Write Access to a
   Memory Region, the RNIC is allowed to use the STag and Tagged Offset
   to write any location within the Memory Region.

   Work Requests may require the Consumer to supply a locally
   accessible data buffer. Locally accessible data buffers are
   described by the STag associated with that Memory Region, a Tagged
   Offset that points to a location within a Memory Region, and the
   quantity of bytes in the buffer that may be used by the Work
   Request.

   The RI MUST enforce that Scatter Gather Elements used in Send
   Operation Type and RDMA Write Work Requests posted to the SQ have
   Local Read Access enabled or a Completion Error will result.

   The RI MUST enforce that Scatter Gather Elements used in Receive
   Work Requests posted to the Receive Queue or Shared-Receive Queue
   have Local Write Access enabled or a Completion Error will result.

   The RI MUST use only Local Access Rights when determining the Access
   Rights for Scatter/Gather Elements. The RI MUST NOT use Remote
   Access Rights when determining the Access Rights for Scatter/Gather
   Elements.

7.4.2  Remote Access to Registered Memory

   The Consumer may, in addition to the Local Access Rights, request
   the RI to assign one or both of the Remote Access Rights to a given



   Hilland, et al.        Expires October 2003             [Page 101]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Memory Region. The RI MUST NOT allow the Consumer to assign Remote
   Write to an MR that has not been assigned Local Write. The RI MUST
   NOT allow the Consumer to assign Remote Read to an MR that has not
   been assigned Local Read.

   If the Consumer assigns Remote Read Access to a Memory Region, the
   RNIC is allowed to use the STag and Tagged Offset to read any subset
   of the Memory Region when processing an incoming RDMA Read Request
   Message. If the Consumer assigns Remote Write Access to a Memory
   Region, the RNIC is allowed to use the STag and Tagged Offset to
   write any subset of the Memory Region when processing an incoming
   RDMA Write or RDMA Read Response Message. For more information, see
   [RDMAP].

   The RI MUST enforce that Tagged Buffers at the Data Sink targeted by
   incoming RDMA Write Messages have Remote Write Access enabled or an
   Asynchronous Error will result at the Data Sink.

   The RI MUST enforce that Tagged Buffers whose contents are retrieved
   by RDMA Read Request Messages have Remote Read Access enabled or an
   Asynchronous Error will result at the Data Source.

   The RI MUST enforce that Tagged Buffers consumed by RDMA Read
   Response Messages have Remote Write Access enabled or an
   Asynchronous Error will result at the Data Sink. The access control
   on the Local Address is not verified until a remote access is
   attempted through the RDMA Read Response Message.

   Remote Access Rights MUST only be used by the RI when determining
   the Access Rights for incoming Tagged and remote Invalidation
   operations. The RI MUST NOT allow an STag with only Local Access
   Rights to be Invalidated by an incoming remote Invalidation
   operation or a protection error will result.

   Figure 18 summarizes local and remote Access Rights and the validity
   of their combinations that the RI MUST enforce:

















   Hilland, et al.        Expires October 2003             [Page 102]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

           Local              Remote          Valid Access Combination

            None               None                      No

            None               Read                      No

            None               Write                     No

            None          Read and Write                 No

            Read               None                     Yes

            Read               Read                     Yes

            Read               Write                     No

            Read          Read and Write                 No

           Write               None                     Yes

           Write               Read                      No

           Write              Write                     Yes

           Write          Read and Write                 No

       Read and Write          None                     Yes

       Read and Write          Read                     Yes

       Read and Write          Write                     Yes

       Read and Write     Read and Write                Yes

            Figure 18 - Valid Combinations of MR Access Rights

7.4.3  Multiple Registrations of Memory Regions

   The same set of memory locations may be registered multiple times,
   resulting in multiple STags. There are two methods for doing this in
   the architecture. The first is the Shared Memory Region, which is
   discussed in Section 7.3.2.4 - Register Shared Memory Region. The
   second is to simply register a set of memory locations a second time
   using the same, similar or overlapping Physical Buffer List.
   Regardless of the method, each resulting STag represents a separate
   and distinct Memory Region and may be independently associated with
   any PD and have distinct Access Rights.




   Hilland, et al.        Expires October 2003             [Page 103]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   The RI MUST support registration of Non-Shared Memory Regions that
   have partially or completely overlapping Physical Buffer Lists and
   return a different STag Index for each.

   In cases where multiple registrations that use the same memory
   locations is desired, provision for optimizing the use of RI
   resources is provided. This Verb is called Register Shared Memory
   Region and is discussed in Section 7.3.2.4 - Register Shared Memory
   Region and the Verb is discussed in Section 9.2.6.6 - Register
   Shared Memory Region.

   Given an existing Non-Shared Memory Region, a Shared Memory Region
   Verb creates a new Shared Memory Region associated with the same
   Physical Memory Addresses, with the intention that the new Shared
   Memory Region shares RNIC mapping resources to the extent possible.
   This also turns the existing Non-Shared Memory Region into a Shared
   Memory Region. Through repeated calls to the Register Shared Memory
   Region Verb, an arbitrary number of Shared Memory Regions can
   potentially share the same RNIC mapping resources, all associated
   with the same Physical Memory Addresses. The Base TO, VA (if the
   input STag Index references a VA Based TO), PD ID, and Access Rights
   specified for the new Shared Memory Region need not be the same as
   those of the existing Memory Region. For a VA Based TO, the RI MUST
   verify that the VA passed in by the Consumer produces a FBO that
   matches the FBO of the PBL that is associated with the STag Index
   passed in by the Consumer. The lengths are by definition the same.

7.5  Memory Access Control

   Only a Privileged Mode Consumer can invoke an RI-Register, RI-
   Reregister, or Allocate Non-Shared Memory Region STag Verb. In
   general, the OS is responsible for determining and enforcing access
   control policy for memory registrations it does on behalf of Non-
   privileged Consumers. For instance, it is anticipated, but not
   required, that operating systems will enforce policies similar to
   the following:

   *   A Non-Privileged Mode Consumer has control over which of its
       memory areas can be accessed by local and remote RNIC data
       transfer operations.

   *   A Non-Privileged Mode Consumer can enable any local memory area
       it has access to for access by RNIC data transfer operations.

   *   A Non-Privileged Mode Consumer cannot enable RNIC read access to
       memory areas that the Consumer itself doesnÆt have read access
       to.






   Hilland, et al.        Expires October 2003             [Page 104]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   A Non-Privileged Mode Consumer cannot enable RNIC write access
       to memory areas that the Consumer itself doesnÆt have write
       access to.

   When a Consumer creates QPs or CQs (through the appropriate Verbs),
   the RI automatically allocates and pins any local memory needed for
   the associated RI internal control structures. Access by the RNIC to
   these control structures is implicitly enabled. Access by the
   Consumer to these control structures is supported only indirectly
   through Verbs. Any STags used within the RI that are used for the
   control structures (if they exist) MUST NOT be exposed to the
   Consumer.

   A Consumer controls which Memory Regions and Memory Windows are
   accessible by each QP through the use of PDs. Prior to creating any
   QPs, registering any Memory Regions, or allocating any Memory
   Windows, the Consumer should allocate one or more PDs. When
   registering Memory Regions or allocating Memory Windows, the
   Consumer specifies the PD ID to associate to each. For information
   on the use of PDs, see Section 5.2 - Protection Domains.

7.5.1  Local Access Control

   With Send Type, RDMA Write, and Receive Queue WRs, the Consumer
   explicitly specifies the data buffers to be accessed through the
   local Scatter Gather Elements (SGEs) that the Consumer posts with
   the associated Work Requests.

   When registering a Memory Region, a Privileged Consumer can
   generally specify the following local Access Rights for the Region:
   read only, write only, read and write.

   The Consumer can access the Memory Region through the STag. This
   STag grants the Consumer local Access Rights for the entire Memory
   Region as bounded by the base TO and byte length and the granularity
   of the access control is enforced at the byte level.

   The following list defines the local Access Rights requirements for
   SGEs used in local operations:

   *   Local read access MUST be specified for Gather Elements used in
       Send Type WRs and RDMA Write WRs,

   *   Local Write access MUST be specified for Scatter Elements used
       in Receive WRs, and

   *   For RDMA Read Type WRs, Local Access Rights are not used to
       verify the Local Address or Remote Address.





   Hilland, et al.        Expires October 2003             [Page 105]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

7.5.2  Remote Access Control

   When a Consumer wants to allow Remote Peers to access its local
   memory using RDMA Writes or RDMA Read Operations, the Consumer
   should explicitly enable remote access and Advertise an appropriate
   STag to the Remote Peer for it to use when initiating these RDMA
   Operations targeting the ConsumerÆs (local) memory.

   A Consumer can use either of two mechanisms to enable remote access
   to its memory. The first mechanism consists of using a Memory Region
   that has remote Access Rights. The second mechanism consists of
   allocating and binding Memory Windows. Either results in an STag
   with associated remote Access Rights for the memory referenced by
   the STag.

   Two types of remote access - read and write - are supported. RDMA
   Write requires Remote Write Access at the Remote Peer. The RDMA
   Protocol converts an RDMA Read Type WR into an RDMA Read Operation
   that uses two RDMAP Messages: RDMA Read Request and RDMA Read
   Response. Remote Read Access MUST be enabled for Memory Regions read
   by a remote RDMA Read Request Message. Remote Write Access MUST be
   enabled for Memory Regions written by a remote RDMA Read Response
   Message. If the Memory Region does not have the appropriate Access
   Rights, a protection error occurs.

   For RDMA Read Operations, during the processing of a RDMA Read Type
   WR, the RNIC is responsible for generating one RDMA Read Request
   Message that contains a description of the Local Address and Remote
   Address. Local Access Rights are not used to verify the Local
   Address or Remote Address. The Remote Access Rights of the Local
   Address is not verified until an incoming RDMA Read Response Message
   is received. The Remote Access Rights of the Remote Address are
   verified when the Remote Peer processes the RDMA Read Request
   Message.

   In order to set either Remote Access control types in a Fast-
   Register operation, when the Non-Shared Memory Region STag was
   created, it MUST have been created with the Remote Access Flag
   enabled.

7.6  Addressing

   The Tagged Offset field is used by local and remote operations to
   address registered Memory Regions.

7.6.1  Addressing Registered Memory

   The RI MUST support two mechanisms for specifying the offset within
   Memory Regions: VA Based TO and Zero Based TO. At the time the
   Memory Region is registered, the RI MUST allow the Consumer to



   Hilland, et al.        Expires October 2003             [Page 106]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   choose between these two mechanisms. A Virtual Address Base Tagged
   Offset (VA Based TO) is one that has a Tagged Offset base that
   starts at a non-zero Virtual Address. A Zero Based Tagged Offset
   (Zero Based TO) is one that has a Tagged Offset base that starts at
   zero.

7.6.1.1  Addressing with VA based TO

   The Virtual Addresses that Consumers manipulate and pass as input
   modifiers are referred to simply as Virtual Addresses in this
   specification. The size of the Virtual Addresses used to specify a
   Memory Region to be registered is implementation dependent. The size
   of the TO MUST be 64 bits. The TO passed in the SGE defines the VA
   of the first byte of the SGE.

   A Memory Region is specified by a Virtual Address that points to the
   first byte, which is specified by the First Byte Offset of the
   Physical Buffer List, and by the length of the set in bytes. The
   Physical Buffer size that backs the Region depends on the host
   system hardware and host operating system.

   The RI MUST allow a Consumer to specify an arbitrary alignment and
   length of the virtually contiguous buffer to be registered through a
   RI-Register Non-Shared Memory Region Verb, RI-Reregister Non-Shared
   Memory Region Verb, or Fast-Register Non-Shared Memory Region.

   The following operations should be performed before registering a VA
   Based TO Non-Shared Memory Region:

   *   Translate the set of virtually contiguous memory locations that
       are associated with the Non-Shared Memory Region into a Physical
       Buffer List.

   *   Pin the Physical Buffers in the Physical Buffer List.

   While a Memory Region is Valid, every Physical Buffer within the
   Region must be pinned down in physical memory. This guarantees to
   the RNIC that the Memory Region is physically resident (not paged
   out) and that the virtual to physical address translation remains
   fixed while the Region is registered. The RI is NOT REQUIRED to
   verify that the Physical Buffers in the Physical Buffer List are
   pinned.

   When the Consumer registers a Non-Shared Memory Region addressed
   through the VA based TO mechanism, the following input modifiers are
   passed to the RI (along with additional input modifiers - see
   Section 9.2.6):

   *   Virtual Address - The VA Physical Buffer offset portion of the
       VA defines the offset into the first Physical Buffer of the Non-



   Hilland, et al.        Expires October 2003             [Page 107]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       Shared Memory Region. The RI checks that the VA modulo Physical
       Buffer Size equals the FBO.

   *   Physical Buffer size - Size of all Physical Buffers referenced
       by the Non-Shared Memory Region.

   *   First Byte Offset (FBO) - Offset into the first Physical Buffer
       of the Non-Shared Memory Region

   When a RI-Register Non-Shared Memory Region Verb, RI-Reregister Non-
   Shared Memory Region Verb, Register Shared Memory Region or Fast-
   Register Non-Shared Memory Region is processed, the RI MUST verify
   that the Base TO modulo the Physical Buffer Size is equal to the VA
   modulo the Physical Buffer Size.

7.6.1.2  Addressing with Zero Based TO

   A zero based contiguous set of memory locations is specified by the
   length of the set in bytes. The RI MUST associate a TO that has a
   value of zero with the First Byte Offset in the Physical Buffer
   List.

   The following operations must be performed before registering a zero
   Based TO Non-Shared Memory Region:

   *   Translate the set of virtually contiguous memory locations
       associated with the Non-Shared Memory Region into a Physical
       Buffer List.

   *   Pin the Physical Buffers in the Physical Buffer List.

   While a Memory Region is Valid, every Physical Buffer within the
   Region must be pinned down in physical memory. This guarantees to
   the RNIC that the Memory Region is physically resident (not paged
   out) and that the virtual to physical address translation remains
   fixed while the Region is registered. The RI is NOT REQUIRED to
   verify that the Physical Buffers in the Physical Buffer List are
   pinned.

   When the Consumer registers a Non-Shared Memory Region addressed
   through the Zero Based TO mechanism, the following input modifiers
   are passed to the RI (along with additional input modifiers - see
   Section 9.2.6):

   *   First Byte Offset - Offset into the first Physical Buffer of the
       Non-Shared Memory Region

   *   Buffer size - Size of all Physical Buffers referenced by the
       Non-Shared Memory Region.




   Hilland, et al.        Expires October 2003             [Page 108]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   When a RI-Register Non-Shared Memory Region Verb, RI-Reregister Non-
   Shared Memory Region Verb, Register Shared Memory Region Verb or
   Fast-Register Non-Shared Memory Region WR is processed for a Zero
   base TO MR, the base TO MUST be set to zero.

   Note that a Memory Window cannot be bound to a Zero base TO MR.

7.6.2  Physical Buffer Lists

   Two Physical Buffer types are defined in this specification: Page
   and Block. The RI MUST support the Page Physical Buffer type.
   Support for the Block Physical Buffer type by the RI is OPTIONAL. If
   the RI supports Block Mode, the RI MUST support the ability to place
   the RNIC into either Block Mode or Page Mode when the RNIC is
   opened. The RI MUST support a mechanism for querying the RNIC to
   determine if the Block Physical Buffer type is supported.

   Memory that is part of a Physical Buffer List should remain pinned
   while the RI has any reference to it. It is not safe for the
   Consumer to assume that when an STag is deallocated that the
   Physical Buffer can be unpinned, since another STag may still have a
   reference to that resource. It is the responsibility of the Consumer
   to determine if and when the Physical Buffers should be unpinned.

7.6.2.1  Page Lists

   A Page List is defined by the following attributes:

   *   Page size - The size, in bytes, of each page in the list.

   *   Address List - A list of addresses that point to the physical
       pages referenced by the Page List. The Address List has the
       following attributes:

       o   All pages in the list have the same size, and that size MUST
           be a power of two.

       o   Page addresses MUST be an integral number of page size. In
           other words, each address in the Address List modulo page
           size MUST equal zero.

   *   First Byte Offset (FBO) - Byte offset to start of Memory Region
       within the first page.

   *   Length - Total length in bytes of the Memory Region.

   When a Page List is used to register a Non-Shared Memory Region that
   has a VA based TO, the RI MUST check that the VA modulo the Page
   Size equals the FBO.




   Hilland, et al.        Expires October 2003             [Page 109]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

7.6.2.2  Block Lists

   A Block List is defined by the following attributes:

   *   Block size - The size, in bytes, of each block in the list.

   *   Address List - A list of addresses that point to the physical
       blocks referenced by the Block List. The Address List has the
       following attributes:

       o   The RI MUST interpret each block referenced in the Address
           List as having the same size.

       o   The RI MUST allow Block Addresses to have an arbitrary byte
           alignment.

   *   First Byte Offset (FBO) - Byte offset to start of Memory Region
       within the first block.

   *   Length - Total length in bytes of the Memory Region.

   When a Block List is used to register a Non-Shared Memory Region
   that has a VA based TO, the RI MUST check that the VA modulo the
   Block Size equals the FBO.

7.6.3  Error Checking of Local and Remote Accesses to MRs

   When a local or remote operation attempts to access a registered
   Memory Region, the RI MUST ensure that:

   *   The Access Rights of the Memory Region allow the type of access
       being performed by the operation,

   *   The Access Rights of the QP allow the type of access being
       performed by the operation,

   *   For a QP not associated with an S-RQ, the PD ID associated with
       the Memory Region matches the PD ID associated with the QP that
       is processing the operation,

   *   For a QP that is associated with an S-RQ:

       o   On an incoming Send Operation Type, the PD ID associated
           with the Memory Region matches the PD ID associated with the
           S-RQ that is processing the operation, and

       o   On an outbound Send or RDMA Write, or any incoming RDMA
           Message, the PD ID associated with the Memory Region matches
           the PD ID associated with the QP that is processing the
           operation,



   Hilland, et al.        Expires October 2003             [Page 110]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   The memory access as specified by the TO & length is within the
       base and bounds of the Memory Region. The RI MUST enforce this
       with a byte level granularity.

   If the length of the access is zero, the RI MUST NOT perform any of
   the above checks on the Memory Region.

7.7  Querying Memory Regions

   Memory Regions have attributes that can be retrieved through the
   Query Memory Region Verb. The RI MUST support the complete list of
   QP attributes as described in Section 9.2.6.3 - Query Memory Region.

7.8  Invalidating Memory Regions

   When access to a Non-Shared Memory Region by an RI is no longer
   required, but the Consumer wants to retain the STag for use in
   future Fast-Register Non-Shared Memory Region and RI-Reregister Non-
   Shared Memory Region Verb invocations, the Consumer may directly
   invalidate access to the Non-Shared Memory Region through an
   Invalidate Local STag WR or an RDMA Read with Invalidate Local STag
   WR. Additionally, an STag may be invalidated by a remote Consumer
   through the use of a Send with Invalidate Message or a Send with
   Solicited Event and Invalidate Message.

   Multiple Memory Regions can represent memory locations that have
   been registered multiple times. The invalidation of a single STag
   prevents RNIC access to those memory locations via the STag
   associated with that Memory Region. Access to the memory locations
   via STags associated with other Memory Regions other than the STag
   being Invalidated MUST NOT be affected. Invalidating an STag
   associated with a Memory Region that partially or completely overlap
   other Memory Regions MUST NOT cause the RI to affect the
   registration of those other Memory Regions.

   The requirements for unpinning the physical buffers associated with
   deallocated Memory Regions are covered in Section 7.6.2 - Physical
   Buffer Lists.

   Invalidating an STag associated with a Shared Memory Region MUST
   result in an Completion Error. Consequently, using an STag
   associated with a Shared Memory Region under the following
   conditions will cause a Completion Error at the Data Sink that
   results in the LLP Stream being torn down after the data transfer
   operation takes place:

   *   As the STag specified in an Invalidate Local STag WR.

   *   As the Data Sink STag for an RDMA Read with Invalidate Local
       STag WR.



   Hilland, et al.        Expires October 2003             [Page 111]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   As the STag to be Invalidated for a Send with Invalidate or Send
       with SE & Invalidate Message.

   When a local Invalidate Local STag WR, a local RDMA Read with
   Invalidate Local STag WR, an incoming Send with Invalidate, or an
   incoming Send with Solicited Event and Invalidate completes
   successfully, the RNIC MUST place the associated STag in the Invalid
   state. For more information, see Section 8.2.2.1 - Memory Management
   Operation Ordering.

   An Invalidated STag retains associated RI resources, such as the PD,
   and the Remote Access Flag, and the number of Physical Buffer List
   entries but the contents of the Address List Entries become
   indeterminate when the Memory Region is in the Invalid state.

   The RI MUST fail Local Work Requests or Remote Operations that
   attempt to access memory locations in a Non-Shared Memory Region
   that has had its STag Invalidated with a protection error. The RNIC
   MUST NOT be able to access any memory locations through an STag that
   is in the Invalid state.

   For Non-Shared Memory Regions created through the RI-Register Non-
   Shared Memory Region Verb, when an STag is Invalidated, the RNIC
   MUST retain:

   *   The Maximum Physical Buffer List (PBL) size and entries used:

       o   When the RI-Register Non-Shared Memory Region was invoked,
           if an RI-Reregister Verb has not been invoked on the Non-
           Shared Memory Region; or

       o   On the last RI-Reregister Non-Shared Memory Region that used
           the Non-Shared Memory Region.

   *   The state of the Remote Access Flag.

   *   The PD associated with the Non-Shared Memory Region.

   For Non-Shared Memory Regions created through the Allocate Non-
   Shared Memory Region STag Verb, when an STag is Invalidated, the
   RNIC MUST retain:

   *   The Maximum Physical Buffer List size and entries used:

       o   When the STag was created for a Non-Shared Memory Region, if
           an RI-Reregister Verb has not been invoked on the Non-Shared
           Memory Region; or

       o   On the last RI-Reregister Non-Shared Memory Region that used
           the Non-Shared Memory Region.



   Hilland, et al.        Expires October 2003             [Page 112]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   The state of the Remote Access Flag.

   *   The PD associated with the Non-Shared Memory Region.

   For Memory Regions created through the RI-Reregister Non-Shared
   Memory Region Verbs, when an STag is Invalidated, the RNIC MUST
   retain:

   *   The Maximum Physical Buffer List (PBL) size and entries used:

       o   When the RI-Register Non-Shared Memory Region was invoked,
           if an RI-Reregister Verb has not been invoked on the Non-
           Shared Memory Region; or

       o   On the last RI-Reregister Non-Shared Memory Region that used
           the Non-Shared Memory Region.

   *   The PD associated with the Non-Shared Memory Region.

   If a Fast-Register is invoked after an RI-Register Memory Region ,
   Allocate Non-Shared Memory Region STag or RI-Reregister Memory
   Region, the Consumer is guaranteed that the RNIC can register a Non-
   Shared Memory Region with a PBL size that is equal to or smaller
   than the original PBL size returned when the Non-Shared Memory
   Region was created or allocated.

   An STag is allowed to already be in the Invalid state, when the RNIC
   performs the STag Invalidation.

   In order to perform an Invalidation Operation on a given QP, either
   through a Local Invalidation operation or an incoming Send with
   Invalidate or Send with Solicited Event and Invalidate, the
   following checks MUST be performed by the RI:

   *   The STag MUST be Non-Shared and in the Valid or Invalid state.

   *   The STag MUST NOT be the STag of zero.

   *   If the STag is that of a Non-Shared Memory Region, the PD ID of
       the STag MUST equal the PD ID of the QP.

   *   If the STag is that of a Non-Shared Memory Region, there MUST
       NOT be any Memory Windows Bound to it.

   *   The STag Key supplied by the Invalidate Operation must be
       validated against the STag Key associated with the Memory Region
       when moving the STag to the Invalid state.

   *   If the Invalidation Operation is due to an Incoming Send with
       Invalidate or Send with Solicited Event & Invalidate, the RI



   Hilland, et al.        Expires October 2003             [Page 113]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       MUST ensure that the QP has either of the remote Access Rights
       enabled and the STag has either of the remote Access Rights
       enabled.

   If any of the above checks fail, a Protection Error MUST result
   unless the STag is in the Deallocated state, in which case an
   Operation Error MUST result. If the operation was initiated by a
   Local Invalidation, a Completion Error MUST result. If the operation
   was initiated by an incoming Invalidation operation, a processing
   error MUST result and the Queue Pair will enter the Terminate state.

   For descriptions of the Work Requests that Invalidate STags
   (Invalidate STag, Send with Invalidate, Send with Solicited Event
   and Invalidate and RDMA Read with Invalidate Local STag), see
   Section 9.3.1.1 - PostSQ.

7.9  Deallocation of STag associated with a Memory Region

   The Consumer can reverse the allocation or registration process that
   created the STag by invoking the Deallocate STag Verb. The process
   of deallocating an STag MUST revoke all RNIC Access Rights
   associated with that STag.

   The RI MUST verify that the STag Index used as an Input Modifier is
   a valid STag on the specified RNIC.

   Multiple Memory Regions can represent memory locations that have
   been registered multiple times. The deallocation of a single STag
   prevents RNIC access to those memory locations via the STag
   associated with that Memory Region. Access to memory locations using
   STags associated with other Memory Regions MUST NOT be affected.
   Deallocating an STag associated with a Memory Region that partially
   or completely overlaps other Memory Regions MUST NOT cause the RI to
   affect the registration of those other Memory Regions. Deallocating
   an STag associated with a Shared Memory Region MUST NOT cause the RI
   to affect the registration of any other Shared Memory Region.

   The requirements for unpinning the physical buffers associated with
   deallocated Memory Regions are covered in Section 7.6.2 - Physical
   Buffer Lists.

   When the Deallocate STag Verb is invoked, any in-process Local or
   Remote Operations that are actively referencing memory locations by
   using the STag being deallocated, MUST fail with a protection error.
   Local or Remote Operations attempting to access memory locations in
   a Memory Region with a deallocated STag MUST fail with a protection
   error.






   Hilland, et al.        Expires October 2003             [Page 114]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Before the Deallocate Verb returns, the RI MUST free all resources
   associated with the STag and revoke the right to use the STag in
   Local or Remote Operations.

   When a Deallocate STag is invoked, the RI MUST NOT:

   *   check the state of the associated STag. That is, an STag
       associated with a Non-Shared MR can be in either the Valid or
       Invalid state when the Deallocate STag is invoked.

   *   check the STag Key portion of the STag. Note that the Deallocate
       Verb does not have an STag Key Input Modifier.

   If any Memory Windows are Bound to the Memory Region and the
   Consumer invokes the Deallocate STag Verb, the RI MUST return an
   Immediate Error and MUST NOT deallocate the Memory Region. Memory
   Windows can reverse the Bind process through deallocation or
   invalidation.

   For a description of the Deallocate Memory Region mechanism, see
   Section 9.2.6.4 - Deallocate STag.

7.10 Memory Windows

   When a Consumer needs more flexible control over remote access to
   its memory, the Consumer can use Memory Windows. Memory Windows are
   intended for situations where:

   *   A Non-Privileged Mode Consumer wants to grant and revoke remote
       Access Rights to a registered Region in a dynamic fashion with
       less of a performance penalty than using
       deallocation/registration or invalidation/re-registration.

   *   A Consumer wants to grant different remote Access Rights to
       different Remote Peers and/or grant those rights over different
       ranges within a registered Region.

   To use a Memory Window, the Consumer allocates a Memory Window and
   then Binds it to a specified TO range of an existing Memory Region
   that is enabled for use with Memory Windows. The range can include
   the entire Memory Region or any subset of the Memory Region.

   See Section 9.2.6 - Memory Management for a description of the Verbs
   used to manage Memory Windows.

7.10.1 Allocating Memory Windows

   The Allocate Memory Window Verb is used to allocate a Memory Window.
   When the Verb returns, it must have allocated Memory Window
   resources on the RNIC, associated the STag with the PD ID supplied



   Hilland, et al.        Expires October 2003             [Page 115]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   as an Input Modifier by the Consumer, and returned the STag
   associated with the allocated Memory Window. The RI MUST ensure that
   the returned STag is in the Invalid state. The RI MUST NOT allow the
   returned STag to be used with RI-Reregister Non-Shared Memory
   Region, Register Shared Memory Region, Query Memory Region or Fast-
   Register Non-Shared Memory Region. For allocating a Memory Window,
   see Section 9.2.6.7 - Allocate Memory Window.

7.10.2 Binding Memory Windows to Memory Regions

   The PostSQ Verb is used to Bind a Memory Window to a previously
   registered Memory Region. After the WR that Binds the MW is
   processed, the STag associated with the Memory Window is in the
   Valid state.

   The RI MUST allow a MW to Bind to a Non-Shared Memory Region. The RI
   MUST allow a MW to Bind to a Shared Memory Region. The RI MUST allow
   all allocated MWs to be Bound to a single MR. The RI MUST allow all
   allocated MWs to be Bound to a single QP.

   If the STag representing the Memory Region to which the Memory
   Window will Bind has an STag of zero, the Verb MUST return either an
   Immediate Error or a Completion Error.

   During the processing of PostSQ Bind Memory Window Verb, the RNIC
   MUST ensure that the PD ID of the Memory Window equals the PD ID of
   the Memory Region and with the PD ID of the QP that is processing
   the PostSQ Bind Memory Window Verb. If the three PD IDs are equal,
   the Memory Window is Bound to the Memory Region and is associated
   with the QP that processed the PostSQ Bind Memory Window Verb.
   Otherwise an invalid PD Completion Error is returned to the
   Consumer. When a Memory Window is Bound to a QP at this point, it is
   conceptually equivalent to having the PD ID of the Memory Window
   replaced with the QP ID of the QP. Thus, instead of performing a PD
   check upon validating the STag for incoming RDMA operations, the QP
   ID of the Memory Window MUST be equal to the QP ID of the QP where
   the incoming RDMA operation arrived.

   The RI MUST check that the QP has the ability to Bind Memory Windows
   enabled.

   When Binding a Memory Window, the RI MUST ensure that the memory
   locations being associated with the Memory Window are within the
   base TO and length of the associated Memory Region. The RI MUST
   support Memory Windows with a Zero Based TO. The RI MUST support
   Memory Windows with a VA Based TO. The RI MUST allow Memory Windows
   to bind to Memory Regions with a VA based TO. If the Memory Window
   has a VA based TO, the RNIC MUST ensure that the value assigned for
   the base of the Memory Window be between the MR's base VA, and the
   MR's Base VA plus the MR's length.



   Hilland, et al.        Expires October 2003             [Page 116]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   When the Bind MW WR completes successfully:

   *   The RI MUST have Bound the MW to the Non-Shared Memory Region.

   *   The RI MUST have Bound the MW to the QP that processed the Bind
       WR, by associating the QP's QP ID to the MW.

   *   The RI MUST have set the MW STag's access rights as requested by
       the Consumer.

   *   The RI MUST accept and use the STag Key passed in by the
       Consumer for the Bind operation.

   *   The RI MUST have set the MW Address Type as requested by the
       Consumer.

   *   If the Address Type of the MW was requested as VA Based, the RI
       MUST have set the Virtual Address as requested by the Consumer.

   *   The RI MUST have placed the MW STag in the Valid State.

   Figure 19 indicates which MR to MW Binding combinations are valid.
   Note that the figure is based on the Base TO type of the Memory
   Region and Memory Window. If the Consumer attempts to Bind a MW to a
   Zero-based TO MR, the RI MUST return an error. The Underlying Memory
   Region in this case may be either a Non-Shared Memory Region or a
   Shared Memory Region.

     Underlying Memory   Memory Window TO base    Valid combination
       Region TO base

         Zero based            Zero based                 No

         Zero based             VA based                  No

          VA based             Zero based                Yes

          VA based              VA based                 Yes

              Figure 19 - MR to MW Valid Binding Combinations

   When a remote access references a Bound Memory Window, the RNIC MUST
   ensure that the QP ID associated with the Memory Window matches the
   QP ID associated with the remote access' RDMA Stream. The RNIC MUST
   also ensure that the memory locations being referenced by the remote
   access are within the base TO and length of the associated Bound
   Memory Window. The RI MUST enforce this with a byte level
   granularity.



   Hilland, et al.        Expires October 2003             [Page 117]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   When Binding a Memory Window, a Consumer can request any combination
   of remote Access Rights for the Window. However, if the associated
   Memory Region does not have local write access enabled and the
   Consumer requests remote write for the Window, implementations MUST
   return a Completion Error.

   Memory Windows MUST support two distinct remote Access Rights:
   Remote Read and Remote Write. Bind Memory Window WRs must specify
   one or both of these rights. Memory Windows with Remote Write Access
   MUST be bound to Memory Regions that have Local Write Access
   Enabled. Memory Windows with Remote Read access MUST be bound to
   Memory Regions that have Local Read Access Enabled.

   A Consumer is allowed and commonly expected to enable remote Access
   Rights when Binding a Window that it may not have enabled when it
   registered the underlying Region - provided it doesnÆt violate the
   above rule regarding local access. For example, a Consumer might
   register a Region with no remote Access Rights, and later Bind one
   or more Windows to that Region that would grant remote Access
   Rights.

   Figure 20 summarizes the access right mappings between Memory
   Regions and Memory Windows and if the Memory Window Access Right
   requested is allowable or not. The RI MUST validate Memory Windows
   Access Right requests according to Figure 20 and if the Access Right
   requested is not allowed, the Bind operation must result in a
   Completion Error.

     Underlying Memory      Requested Remote    Access Right Requested
       Region's Local       Access Rights for          allowed:
       Access Rights         Memory Window

         Local Read           Remote Write                No

         Local Read            Remote Read                Yes

         Local Read       Remote Read and Write           No

         Local Write           Remote Write               Yes

         Local Write           Remote Read                No

         Local Write      Remote Read and Write           No

    Local Read and Write       Remote Write               Yes

    Local Read and Write       Remote Read                Yes





   Hilland, et al.        Expires October 2003             [Page 118]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

     Underlying Memory      Requested Remote    Access Right Requested
       Region's Local       Access Rights for          allowed:
       Access Rights         Memory Window

    Local Read and Write  Remote Read and Write           Yes

            None                   Any                    No

            Any                   None                    No

          Figure 20 - Valid Combinations of MW & MR Access Rights

   Allocating or de-allocating a Memory Window requires a Privileged
   mode transition for a Non-Privileged Consumer, and thus incurs the
   associated software overhead. Binding a Memory Window is performed
   with a Work Request posted to a Send Queue, and thus incurs far less
   software overhead.

   An STag used in a PostSQ Bind Memory Window Verb MUST be in the
   Invalid state.

   Each time a Memory Window is Bound, the Consumer passes the STag Key
   portion of the STag to the RI. The RI MUST use the STag Key provided
   by the Consumer. Additionally, the RI MUST NOT change the STag Index
   portion of the STag passed in by the Consumer. Note that the Bind
   Memory Window WR has unique ordering rules which are detailed in
   Section 8.2.2.1 - Memory Management Operation Ordering. Once the
   Bind operation has completed processing, RNIC implementations MUST
   guarantee that no additional accesses on this Memory Window can be
   performed with any STag Key other than the one used in the last Bind
   operation.

   If the RNIC detects an error with the Bind operation, it MUST put
   the QP into the Error state.

   Multiple Windows can be Bound to the same Memory Region, each with
   arbitrary remote Access Rights, and their associated areas can be
   overlapping or disjoint.

   For a description of the error conditions checked during MW Bind and
   MW access, see Section 7.10.6 - Error Checking during Memory Window
   Operations.

   For a description of the Bind Memory Window operation, see Section
   9.3.1.1 - PostSQ.






   Hilland, et al.        Expires October 2003             [Page 119]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

7.10.3 Querying Memory Windows

   Memory Windows have attributes that can be retrieved through the
   Query Memory Window Verb. The RI MUST support the complete list of
   QP attributes as described in Section 9.2.6.8 - Query Memory Window.

7.10.4 Invalidating or De-allocating Memory Windows

   When access to a Memory Window by the RI is no longer required, but
   the Consumer wants to retain the STag for use in future PostSQ Bind
   Memory Window Verb invocations, the Consumer may directly invalidate
   access to the Memory Window through either an Invalidate Local STag
   WR or an RDMA Read with Invalidate Local STag WR. Additionally, an
   STag associated with a Memory Window may be invalidated by a remote
   Consumer through the use of a Send with Invalidate Message or a Send
   with Solicited Event and Invalidate Message. For more information on
   these Verbs, see Section 7.8 - Invalidating Memory Regions.

   Memory Windows are Deallocated in a fashion similar to Memory
   Regions: with the Deallocate STag Verb. For more information, see
   Section 7.9 - Deallocation of STag associated with a Memory Region.

   When processing an Invalidate operation on an MW STag:

   *   and the MW is in the Valid state, the RI MUST check and enforce
       that the QP ID associated with the MW is equal to the QP ID of
       the QP processing the Invalidate Local STag WR. If the QP IDs
       match, the RNIC MUST place the specified local STag in the
       Invalid state. If the QP IDs do not match, the RI MUST return an
       error.

   *   and the MW is in the Invalid state, the RI MUST check and
       enforce that the PD ID associated with the MW is equal to the PD
       ID associated with the QP processing the Invalidate Local STag
       WR. If the PD IDs do not match, the RI MUST return an error.

   When a local Invalidate Local STag WR, local RDMA Read with
   Invalidate Local STag WR, an incoming Send with Invalidate Message,
   or an incoming Send with Solicited Event and Invalidate Message
   completes successfully, the RNIC MUST:

   *   transition the associated STag to the Invalid state,

   *   change the association of the newly invalidated STag from the QP
       to the PD of the QP that processed the STag Invalidation,

   *   retain the Memory Window resources associated with the STag,

   *   remove the association of the Memory Window with the underlying
       Memory Region.



   Hilland, et al.        Expires October 2003             [Page 120]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   An invalidated STag which was either Invalidated as described above,
   or in the Invalid state because it was created through the Allocate
   Memory Window Verb but never used, can be used as the MW in a PostSQ
   Bind Memory Window WR.

   Once an STag associated with a MW is successfully Invalidated, the
   RI MUST associate the STag with the PD associated with the QP
   processing the Invalidate Local STag WR.

   For information on Invalidating Memory Windows through the
   Invalidate Local STag or RDMA Read with Invalidate Local STag WR,
   see Section 9.3.1.1 - PostSQ. For information on Invalidating Memory
   Windows through Send with Invalidate or Send with Solicited Event &
   Invalidate WR, see Section 9.3.1.1 - PostSQ. For a description of
   the Verb to deallocate a Memory Window, see Section 9.2.6.4 -
   Deallocate STag.

7.10.4.1 Invalidating or De-allocating Active Windows

   Under normal operation, it is improper for a Consumer to deallocate
   or Invalidate the STag of the Memory Window while it is being used
   in an incoming, remote operation. However, this can occur if the
   Remote Consumer misbehaves, or it can occur under error recovery
   circumstances.

   Any Remote Operations that are in-process and actively using a
   Memory Window when its STag is Invalidated MUST fail with a
   protection error. Once the Completion of the Invalidate operation
   has been determined by the Consumer, the RI MUST guarantee that no
   additional accesses can be performed under the previous binding.

   Any Remote Operations that are in-process and actively using a
   Memory Window when it is deallocated MUST fail with a protection
   error. Once the de-allocation Verb completes, RNIC implementations
   MUST guarantee that no additional accesses can be performed through
   that Memory Window.

   An STag is allowed to already be in the Invalid state, when the RNIC
   performs the STag Invalidation.

7.10.5 Summary of Memory Window STag States

   An STag associated with a Memory Window has two states:

   *   Invalid - May not be used to access a memory location.

       o   Entered through: Allocate Memory Window, PostSQ Invalidate
           STag WR, incoming Send with Invalidate STag Message,
           incoming Send with Solicited Event and Invalidate STag
           Message, or local RDMA Read with Invalidate Local STag WR.



   Hilland, et al.        Expires October 2003             [Page 121]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   Exited through: PostSQ Bind Memory Window WR or Deallocate
           STag.

   *   Valid - May be used to access a memory location.

       o   Entered through: PostSQ Bind Memory Window WR.

       o   Exited through: PostSQ Invalidate STag MW, incoming Send
           with Invalidate STag Message, incoming Send with Solicited
           Event and Invalidate STag Message, local RDMA Read with
           Invalidate Local STag WR, or Deallocate STag.

   Note: Deallocate STag exits the state logic captured above.

7.10.6 Error Checking during Memory Window Operations

7.10.6.1 Error Checking at Window Bind Time

   The RI MUST check for the following error conditions during the
   Memory Window Bind operation and, if any error is detected the RI
   MUST return a Completion Error.

   *   The RNIC MUST check and enforce that the MW STag is an MW STag
       and is in the Invalid state.

   *   The RNIC MUST check and enforce that the QP has Memory Window
       Binding enabled.

   *   The RNIC MUST check and enforce that the STag of the MR is an MR
       STag and is in the Valid state and is not the STag of zero.

   *   The RNIC MUST check and enforce that the Memory Window, Memory
       Region, and QP belong to the same PD.

   *   The RNIC MUST check and assure that the Memory Region has Window
       binding enabled.

   *   The RNIC MUST check and enforce that the Memory Window Access
       Rights are compatible with the Access Rights of the underlying
       Memory Region. (See Figure 19).

   *   The RNIC MUST check and enforce that the Memory Region is not a
       Zero based TO MR.

   *   The RNIC MUST check and enforce that the Memory Window base TO
       and bounds is within the base TO and bounds of the underlying
       Memory Region. The RI MUST enforce this with a byte level
       granularity.





   Hilland, et al.        Expires October 2003             [Page 122]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

7.10.6.2 Error Checking at Window Access Time

   The following conditions MUST be checked for each incoming RDMAP
   Tagged Message targeting an STag that is associated with a Memory
   Window:

   *   The RNIC MUST check and enforce that the MW STag is in the Valid
       state.

   *   The RNIC MUST check and enforce that the QP ID associated with
       the Memory Window is equal to the QP ID associated with the
       incoming remote operation that is accessing the Memory Window.

   *   The RNIC MUST check and enforce the incoming memory access as
       represented by the TO and length is within the TO base and
       bounds of the Memory Window. The RI MUST enforce this with a
       byte level granularity.

   *   The RNIC MUST check and enforce the Access Rights associated
       with the Memory Window.

   *   The RNIC MUST NOT check or enforce the Access Rights associated
       with the Memory Region to which the Memory Window is Bound.

   *   The RI MUST check that the appropriate MW and QP Remote Access
       Rights are enabled for the incoming RDMA Message. For example,
       if the incoming RDMA Message is an RDMA Write targeting a MW,
       the RI must check that the MW and the QP have Remote Write
       Access Rights enabled.

   If any of the above checks fail, the RI MUST not allow the memory
   access to take place and a protection error MUST be generated.

   If the length of the access is zero, the RI MUST NOT perform any of
   the above checks on the Memory Window.

   Note that the QP attributes must be verified as well. For more
   information, see Section 8.1.2.2.

7.10.6.3 Error Checking at Window Invalidate Time

   The following conditions MUST be checked on a PostSQ Invalidate
   Local STag WR, RDMA Read with Invalidate Local STag WR, incoming
   Send with Invalidate Message, or incoming Send with Solicited Event
   and Invalidate Message that accesses a Memory Window:

   *   If the Memory Window is in the Valid state, the RNIC MUST check
       and enforce that the QP ID associated with the Memory Window is
       equal to the QP ID associated with the QP processing the
       Invalidate Local STag WR.



   Hilland, et al.        Expires October 2003             [Page 123]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   If the Memory Window is in the Invalid state, the RNIC MUST
       check and enforce that the PD ID associated with the Memory
       Window is equal to the PD ID associated with the QP processing
       the Invalidate Local STag WR.

   If any of the above checks fail, the RI MUST NOT allow the
   invalidation to take place and the operation MUST result in an
   error.













































   Hilland, et al.        Expires October 2003             [Page 124]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

8  Work Requests and the WR Processing Model

8.1  Work Requests

   A Work Request is the fundamental unit of work used by the Consumer
   to indicate to the RNIC that there is data to transfer and control
   operations to process on a specific QP. The following sections
   describe the creation of Work Requests, types of Work Requests and
   Work Request Contents.

8.1.1  Creating Work Requests

   Work Requests MUST be the only mechanism available to Consumers to
   submit work to the Work Queues. The Work Requests Verbs MUST be used
   only to pass operations from the Consumer to the RI. Specifically,
   these Verbs are PostSQ (Section 9.3.1.1) and PostRQ (Section
   9.3.1.2).

   Work Requests can only be posted to the SQ or RQ of a specific QP,
   or, if the QP is associated with an S-RQ, to the S-RQ associated
   with the QP.

   Work Requests are created by the Consumer above the RI and submitted
   through the Verbs to the RI for processing. The format of Work
   Requests within the RI is not defined. Its structure is opaque to
   the Consumer and is not part of this specification. WRs are only
   valid during the Posting process. WRs are then represented by WQEs
   until Completed.

   The RNIC MUST support the submission of multiple WRs to the RI as a
   list of individual Work Requests. The intention of this requirement
   is to allow for optimizations in the RNIC such that the RI can
   inform the RNIC of WQEs in the most efficient manner for that
   individual RNIC.

8.1.2  Work Request Types

   There are three basic Work Request types. These are those dealing
   with Send/Receive, RDMA, and Memory.

8.1.2.1  Send/Receive

   The Send/Receive model supports the Untagged Buffer Model in the
   RDMAP/DDP specifications. The Send/Receive model uses a one-to-one
   correspondence between outgoing Sends Operation Type WRs and
   incoming Receive Queue WRs. Successful Send Type Work Requests MUST
   result in the consumption of a Receive Queue Work Request at the
   Associated QP. Receive Queue Work Requests should be posted to the
   RQ before the incoming Send Message Type arrives. If a WQE is not
   available on the RQ to describe the Untagged Buffer for the incoming



   Hilland, et al.        Expires October 2003             [Page 125]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Send Message Type, then the LLP Stream MAY be terminated. If the LLP
   Stream is not terminated, the reader should see Section 13.2 -
   Graceful Receive Overflow Handling for one implementation option.

   The RI MUST allow Send Work Requests to only be posted to a Send
   Queue. This includes all Send Operation Types, which are: Send, Send
   with Solicited Event, Send with Invalidate and Send with Solicited
   Event & Invalidate. The RI MUST allow only Receive Work Requests to
   be posted to a Receive Queue or Shared Receive Queue.

   A Receive Queue Scatter/Gather List Work Request MUST contain at
   least enough buffer space to place the incoming Send Message Type.
   If it does not, a Completion Error MUST be returned. The length of
   the buffer represented by the Scatter/Gather List of a Receive Queue
   Work Request MAY be greater than the length of the incoming data.
   The length of incoming data MUST be returned by the RI as part of
   the Work Completion. In the case of any Completion Error, the value
   of the length in the Work Completion MUST be considered
   indeterminate.

   Since segmentation and reassembly is provided by DDP, Send Operation
   Types and corresponding Receives can be larger than the EMSS (See
   [RDMAP][DDP]). The maximum data transfer length supported by the
   architecture is 2^32-1 octets of data. Note that for any given
   message, the length of the buffers represented by the WRs posted to
   the RQ MAY have a total length that is smaller than the maximum data
   transfer length. It is up to the Consumer to negotiate the maximum
   receive buffer size with the Remote Peer.

   The Data Source of Send Operation Types MUST be a local
   Scatter/Gather List. See Section 8.1.3.2 for a description of
   Scatter/Gather List.

   The Data Sink of Receive operations MUST be a local Scatter/Gather
   List.

8.1.2.2  RDMA

   RDMA Write WRs, RDMA Read WRs, and RDMA Read with Invalidate Local
   STag WRs  MUST NOT result in the consumption of a Receive Queue Work
   Request at the Remote Peer.

   The Data Source of an RDMA Write Work Request MUST be a
   Scatter/Gather List consisting of local buffers.

   The Data Sink used in an RDMA Read Type WR MUST be in the local
   node's address space as represented by the TO, STag and Length
   contained in the RDMA Read Type WR. The STag MUST be Bound to either
   a Memory Region or a Memory Window containing the buffer represented
   by the TO and length.



   Hilland, et al.        Expires October 2003             [Page 126]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   The Data Source for an RDMA Read Type WR and the Data Sink for an
   RDMA Write WR MUST be in the Remote Peer's address space as
   represented by the TO, STag and Length contained in the Work
   Request. The STag MUST represent either a Memory Region or a Memory
   Window containing the buffer represented by the STag, TO and length.

   Queue Pairs have RDMA Read enable and RDMA Write enable attributes.
   Memory Regions and Memory Windows have Remote Read and Remote Write
   attributes as well. Memory Regions also have Local Read and Local
   Write attributes. RDMA transfers MUST only take place when the
   appropriate QP RDMA attribute is enabled and the appropriate STag
   attribute is enabled where the STag represents either a Memory
   Region or a Memory Window. If the STag is that of a Memory Window,
   the attributes of the Memory Region do not apply at memory access
   time. These attributes are checked at the node where the target
   memory is located. After the STag Access Rights and QP Access Rights
   have been verified, the RI MUST verify that the STag Access Rights
   match the QP Access Rights. If the RI detects an invalid Access
   Rights combination, the operation MUST result in a protection error.
   The combinations of QP Access Rights and STag Access Rights which
   will allow the data transfer to take place are shown in Figure 21.
































   Hilland, et al.        Expires October 2003             [Page 127]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   STag Used as   QP Attribute        STag Attribute(5)   Access
                                                          Allowed?

   RDMA Read Type Inbound RDMA Read:  Remote Read Access:
   Data Source
                   Enabled             Enabled             Yes

                   Disabled            Either              No

                   Either              Disabled            No

   RDMA Write or  Inbound RDMA Write  Remote Write
   RDMA Read Type and inbound RDMA    Access:
   Data Sink      Read Response:

                   Enabled             Enabled             Yes

                   Disabled            Either              No

                   Either              Disabled            No

   RDMA Write or                      Local Read Access:
   Send Type Data  Either
   Source                              Enabled             Yes

                                       Disabled            No

   Receive Data                       Local Write Access:
   Sink            Either
                                       Enabled             Yes

                                       Disabled            No


           Figure 21 - Valid QP & STag Access Right Combinations

   The RDMA Read with Invalidate Local STag WR behaves similar to an
   RDMA Read Work Request which is then immediately followed by a
   Invalidate Local STag WR on the STag in the Local Address. The
   slight difference in behavior is in this case the Invalidate will
   not occur until after the RDMA Read Operation is complete; while
   with two separate WRs, the Invalidate operation could begin
   processing before the RDMA Read Type WR Completes. Work Requests
   subsequent to an RDMA Read with Invalidate Local STag WR may begin


   Footnote 5: The STag may have additional Access Rights, but only the
   rights listed effect the allowed access.




   Hilland, et al.        Expires October 2003             [Page 128]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   processing before the RDMA Read with Invalidate Local STag WR
   Completes. See Section 8.2.2.1 - Memory Management Operation
   Ordering for more details.

8.1.2.3  Memory

   The following Memory Operations can be posted to the SQ: Bind Memory
   Window, Fast-Register Non-Shared Memory Region, Invalidate Local
   STag and RDMA Read with Invalidate Local STag.

8.1.2.3.1   Bind Memory Windows

   The Bind Memory Window WR associates a previously allocated MW to a
   specified Tagged Offset (TO) range within an existing MR, as well as
   sets the MW's RDMA remote Access Rights.

   Bind operations MUST be posted to the SQ as a Work Request. Binds
   only affect local RNIC mapping resources and MUST NOT cause any
   segment to be issued to the LLP. No resources at the associated QP
   are directly affected.

   For more information on the Memory Window Bind operation, see
   Section 7.10.2 - Binding Memory Windows to Memory Regions.

8.1.2.3.2   Fast-Register Non-Shared Memory Region

   The Fast-Register Non-Shared Memory Region WR associates an MR STag
   that is in the Invalid state to a specified Physical Buffer List
   (For more information on Invalidating STags, see Section 7.8 -
   Invalidating Memory Regions). For information on the STag types
   allowed, see Section 7.3.2.5 - Fast-Register Non-Shared Memory
   Region.

   Fast-Register Non-Shared Memory Region operations MUST be posted to
   the Send Queue. Fast-Register Non-Shared Memory Region operations
   only affect local RNIC mapping resources and do not cause any data
   transfer. No resources at the Associated QP are directly affected.

8.1.2.3.3   Invalidate Local STag

   The Invalidate Local STag and RDMA Read with Invalidate Local STag
   WRs use the STag supplied as the target for the invalidation and
   transition the STag to the Invalid state.

   The STag which is the target of an Invalidate Local STag or RDMA
   Read with Invalidate Local STag WR MUST be associated with a Non-
   Shared Memory Region (i.e. created by Allocate Non-Shared Memory
   Region STag, RI-Register Non-Shared Memory Region, RI-Reregister
   Non-Shared Memory Region and has not transitioned to a Shared Memory
   Region) or MW (i.e. created by Allocate Memory Window).



   Hilland, et al.        Expires October 2003             [Page 129]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   For information on Invalidating STags associated with a Non-Shared
   MR, see Section 7.8 - Invalidating Memory Regions. For information
   on Invalidating STags associated with MWs, see Section 7.10.4 -
   Invalidating or De-allocating Memory Windows.

   Invalidate Local STag operations MUST be posted to the Send Queue as
   a Work Request. The Invalidate Local STag operations only affect
   local RNIC mapping resources and MUST NOT cause any data transfer.
   No resources at the Associated QP are directly affected.

   The initiation of an Invalidate Local STag operation must remain
   ordered with respect to other Work Requests on the same QP and the
   operation must take effect before any subsequent WRs can begin
   processing by the RNIC, as defined in the ordering rules in Section
   8.2.2.1 and Section 8.2.2.2.

8.1.3  Work Request Contents

   Every Work Request submitted through the Verbs contains all of the
   information required to perform the requested operation. The exact
   WR contents are covered in the Section 9.3.1.1 - PostSQ and 9.3.1.2
   - PostRQ. The characteristics of two of the Post Send Request Verb
   modifiers are discussed below.

8.1.3.1  Signaled Completions

   Signaled Completions refer to Work Requests that result in a Work
   Completion. Unsignaled Completions provide a mechanism where Work
   Requests posted to the Send Queue do not generate a Work Completion
   in the associated Completion Queue if the operations complete
   successfully. The RI MUST support PostSQ WRs with Unsignaled
   Completions on every QP.

   Every WR posted to the RQ MUST result in a Work Completion.
   Consequently, all RQ WRs are considered Signaled WRs.

   The Consumer can indicate that it does not need a Signaled
   Completion by setting the Unsignaled Completion indicator in a Work
   Request posted to the SQ.

   When an error is encountered on an Unsignaled or Signaled WR, a CQE
   will be generated for that WR with the appropriate error code. In
   addition, the RI MUST Complete all subsequent WRs with a Flushed
   Error Completion Status regardless of their signaling type. The
   Consumer is safe in assuming that all WRs prior to the one resulting
   in an error were completed successfully.

   An Unsignaled WR is defined as completed successfully when all of
   the following rules are met:




   Hilland, et al.        Expires October 2003             [Page 130]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   A Work Completion is retrieved from the CQ associated with the
       SQ where the unsignaled Work Request was posted,

   *   that Work Completion corresponds to a subsequent Work Request on
       the same Send Queue as the unsignaled Work Request, and

   *   the subsequent Work Request is ordered after the unsignaled Work
       Request as per the ordering rules. Depending on the Work Request
       used, this may require using the Local Fence indicator in order
       to guarantee ordering.

   When an unsignaled WQE completes successfully:

   *   The RI MUST free up any resources associated with the Unsignaled
       WQE,

   *   The Consumer MAY consider the WQE as having completed
       successfully, and

   *   The Consumer MAY re-use any resources associated with the
       Unsignaled WQE.

   The Consumer should ensure that in the event that a WQE with an
   Unsignaled Completion indicator results in an error that the CQ will
   not overflow as stated in Section 5.3.1. This is because the WQE
   will cause a CQE and every WQE after it will cause a CQE as well
   since they result in CQEs with the Flushed status.

8.1.3.2  Scatter/Gather List

   The RI MUST allow each Scatter/Gather List (SGL) to contain one or
   more Scatter/Gather Elements (SGE). The SGE references a buffer via
   an STag, TO, and length. The STag specified in the SGE MUST be
   Registered with the RI prior to submission, except for the STag of
   zero. These buffers referenced by the STag MUST be considered to be
   in the scope of the RI from the time they are submitted to a Work
   Queue until Completion of the Work Request has been confirmed.

   If a Memory Window STag is used in an SGE in a PostRQ or PostSQ Send
   Operation Type or the Data Source for an RDMA Write WR, the RI MUST
   Complete the Work Request with a Completion Error.

   The sum total of all of the buffer lengths in an SGL MUST NOT exceed
   the maximum message payload size specified for RDMAP. This is 2^32-1
   bytes. If an SGE has a length of zero, the STag MUST NOT be
   validated by the RI. For PostSQ WRs, the sum of the Length field in
   all of the SGEs MUST be the total length of that RDMAP operation.
   This value MUST be able to be zero.





   Hilland, et al.        Expires October 2003             [Page 131]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   An RI MAY support more than one Scatter/Gather Element per
   Scatter/Gather List. The exact number of Scatter/Gather Elements per
   Scatter/Gather List supported by the RNIC MUST be returned via the
   Query RNIC Verb (Section 9.2.1.2) where there is one value for Send
   Operation Type WR for Data Source buffers (which also applies to
   PostRQ buffers) and one value for RDMA Write WR Data Source buffers.
   The Consumer can specify the maximum number of Scatter/Gather
   Elements per Scatter/Gather List for each Work Queue as an input
   modifier to the Create QP (Section 9.2.5.1). The RI MUST return an
   Immediate Error if the value in Create QP exceeds the value
   supported by the RNIC.

   An RI MUST support at least four Scatter/Gather Elements per
   Scatter/Gather List when the Scatter/Gather List refers to the Data
   Source of a Send Operation Type or the Data Sink of a Receive
   Operation. An RI is NOT REQUIRED to support more than one
   Scatter/Gather Element per Scatter/Gather List when the
   Scatter/Gather List refers to the Data Source of an RDMA Write.

8.1.3.2.1   STag of zero Usage

   The ability to use the reserved STag of zero MUST NOT be allowed for
   Non-Privileged Mode accessible QPs. The RI must generate an
   Affiliated Asynchronous Error if an RDMAP Tagged message is received
   with an STag of zero. If the STag of zero is used in an outgoing
   RDMA Read Type WR or as the Data Sink of an RDMA Write WR, the RI
   MUST return a Completion Error. Thus the Consumer should not
   Advertise the STag of zero, since an error will result.

8.1.3.3  RDMA Data Source & Data Sink

   For RDMA Read Type Work Requests, the RI MUST support the Data
   Source Local Address as an input modifier to PostSQ. The structure
   representing this information is known as a Data Source Address. A
   Data Source Address consists of an STag, Tagged Offset and Length.
   An RI MUST support exactly one Data Source Address for RDMA Read
   Type Work Requests.

   For RDMA Write Work Requests, the RI MUST support the Data Source
   Scatter/Gather List as an input modifier to PostSQ.

   For RDMA Write and RDMA Read Type Work Requests, the RI MUST support
   the Data Sink Remote Address as an input modifier to PostSQ. The
   structure representing this information is known as a Data Sink
   Address. A Data Sink Address consists of an STag, Tagged Offset and
   Length. An RI MUST support exactly one Data Sink Address for RDMA
   Read Type Work Requests and RDMA Write Work Requests.






   Hilland, et al.        Expires October 2003             [Page 132]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

8.2  Work Request Processing Model

   The Work Request processing model describes how requests are sub-
   mitted, processed by the RNIC, and the results returned to the
   Consumer.

8.2.1  Submitting Work Request to a Work Queue

   Work Requests are submitted to the RNIC through the Verbs. They are
   represented within the RI as Work Queue Elements. Work Queue
   Elements are abstract. This means they are not accessible directly
   by the Consumer of the RNIC Interface.

   Work Requests can be submitted to the RNIC as a list of Work
   Requests. Each Work Request in the Work Request List which is
   successfully inserted into the Work Queue MUST result in the
   consumption of one WQE on the Work Queue, and each Work Request MUST
   be submitted to the Work Queue in the order specified in the Work
   Request List. When a list of WRs containing more than one WR is
   posted on an SQ, RQ, or an S-RQ, the first Immediate Error in
   processing a WR MUST stop processing of the Work Request List and
   MUST NOT enqueue the subsequent WRs in the list onto the Work Queue.
   All Work Requests prior to the Work Request in error MUST be
   inserted into the Work Queue. The RI MUST return to the Consumer the
   number of successfully posted WRs and the verbs result MUST indicate
   the Immediate Error associated with the WR that resulted in the
   first error.

   The intent of supporting a WR List is to allow some implementations
   to reduce the number of Consumer to RI interactions when the
   Consumer has multiple WRs to post, and to reduce the number of
   interactions between the RI and RNIC due to alerting the RNIC of
   additional work to perform.

   One of the intentions of the architecture is to allow an
   implementation to pass Work Requests from a Non-Privileged Mode
   Consumer directly to the RNIC. Consequently, certain Verbs are
   designed to be invoked in either Privileged Mode or Non-Privileged
   Mode while others are designed to be invoked only in Privileged
   Mode. The Verbs that are intended to be invoked in either Privileged
   Mode or Non-Privileged Mode are: PostSQ, PostRQ, Poll for Completion
   and Request Completion Notification.

   The RI MUST return control to the Consumer immediately after a WR or
   WR List has been submitted to the SQ, RQ or S-RQ and the RNIC has
   been notified that a new WR or WR List is ready to process.

   The RI MUST ensure that the space occupied by a Work Request in
   either the Send or Receive Work Queue is not made available for
   posting a new Work Request until:



   Hilland, et al.        Expires October 2003             [Page 133]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   In the case where the WR was Signaled, the associated Completion
       has been reaped.

   *   In the case where the WR is Unsignaled, one of the following is
       true:

       o   The WR has Completed processing successfully, OR

       o   The associated Completion has been reaped for the WR if the
           Unsignaled WR Completed in error, OR

       o   A Completion associated with a subsequently posted WR to the
           same WQ has been reaped.

   If space is not available on a Work Queue, then an RI MUST return an
   Immediate Error.

   The Unsignaled WR confirmation rules dictate that the Consumer must
   post a WR with the Signaled Completion indicator set with a
   frequency less than or equal to the maximum number of WQEs on the
   SQ. In other words, if X equals the maximum number of WQEs on the
   SQ, then the Consumer must post at least one Signaled Completion
   Work Request every X Work Requests. In addition, the Consumer must
   retrieve a Work Completion of a Signaled Completion with a frequency
   less than or equal to the maximum number of WQEs on the SQ. This is
   done in order to force confirmation that prior Unsignaled WRs are
   Completed. If the Consumer does not follow these rules, a situation
   may arise where the Consumer is unable to post WRs to the SQ. A ULP
   reply based on the data that was in a SQ WR is insufficient for
   determining if the WR has completed, since hardware resources may be
   held in use until the WCs are polled from the CQ.

   The QP can accept Work Requests only when the QP is in a state that
   allows Work Requests to be submitted.

   For details on the Verbs which submit Work Requests, see Sections
   9.3.1.1 - PostSQ and 9.3.1.2 - PostRQ.

8.2.2  Work Request Processing

   Processing of Work Requests submitted to a Work Queue is initiated
   and processed according to the rules in this section.

   It is important to understand the difference between Placement and
   Delivery ordering since RDMAP provides different semantics for the
   two.

   Note that many current protocols, both as used in the Internet and
   elsewhere, assume that data is both Placed and Delivered in order.
   This allowed applications to take a variety of shortcuts that



   Hilland, et al.        Expires October 2003             [Page 134]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   depended on in-order Placement and Delivery. For RDMAP, many of
   these shortcuts are no longer safe to use, and could cause
   application failure. To ensure reliable operation, applications need
   to take the rules described below into account.

   The following rules apply to implementations of the RDMAP protocol:

   1.  Send Type, RDMA Write, and RDMA Read Type Work Requests
       submitted to a Send Queue MUST be initiated and sent in the
       order submitted to the Send Queue.

   2.  Work Requests submitted to a single Send Queue or Receive Queue
       MUST be Completed by the RI in the same order as the Work
       Requests were submitted. Note that this does not apply to WRs
       posted to S-RQs.

   3.  Ordering guarantees for processing and Completion notifications
       exist only between Work Requests submitted to the same Work
       Queue. The RI is NOT REQUIRED to provide ordering guarantees
       across multiple local SQ to remote RQ pairs.

   4.  RDMA Messages MAY be Placed in any order while in the scope of
       the RI. If an application uses overlapping buffers (points
       different Messages or portions of a single Message at the same
       buffer), then it is possible that the last incoming write to the
       Data Sink buffer will not be the last outgoing data sent from
       the Data Source.

   5.  For a Send Type Operation, the contents of the Receive Queue
       Buffer at the Data Sink MAY be indeterminate until the Receive
       Queue Work Request is Completed at the Data Sink.

   6.  For an RDMA Write Operation, the contents of the buffer at the
       Data Sink MUST be considered indeterminate until a subsequent
       Send Type Message is Completed by consuming a Receive Queue WQE
       at the Data Sink.

   7.  For an RDMA Read Operation, the contents of the buffer at the
       Data Sink MUST be considered indeterminate until the RDMA Read
       Type Work Request has been Completed.

       Statements 5, 6, and 7 imply no peeking at the data in a buffer
       to see if all of the data has arrived. It is possible for some
       data to arrive before logically earlier data does, and peeking
       may cause unpredictable application failure

   8.  Except for Unsignaled WRs that complete successfully, the
       resources associated with a Work Request must be considered to
       be in the scope of the RI from the time the Work Request is sub-
       mitted to a Work Queue until the associated Work Completion has



   Hilland, et al.        Expires October 2003             [Page 135]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       been returned. For Unsignaled WRs that complete successfully,
       refer to Section 8.1.3.1 for a description of when the resources
       associated with the Unsignaled WR are freed.

   9.  If the Consumer or Application modifies the contents of Data
       Sink Buffers while the buffers are in the scope of the RI, the
       state of the Data Sink Buffers is indeterminate.

   10. If the Consumer or Application modifies the contents of Data
       Source Buffers while the buffers are in the scope of the RI, the
       state of the Data Sink buffers is indeterminate.

   11. The RI is NOT REQUIRED to guarantee that the Completion of an
       RDMA Write or Send Type WR at the Local Peer means that the ULP
       Message has: reached the Remote Peer, reached the Remote Peer
       ULP Buffer, or been examined by the Remote Peer ULP.

   12. Incoming Untagged RDMAP Messages (sent in FIFO and MSN order)
       MUST use RQ or S-RQ Buffers and Complete through the RQ's CQ, in
       the same order as the Send Message Type Work Requests are posted
       to the Associated QP's Send Queue.

   13. Upon local Completion of an incoming Untagged RDMAP Message the
       RI MUST guarantee that any prior Send or RDMA Write Messages
       from the same Associated QP have also Completed at the Data
       Sink.

   14. If the Consumer overlaps its Data Sink buffers for different
       operations, subsequent Operations MAY cause the RI to overwrite
       the data in those buffers before the Consumer receives and
       processes the Completion.

   15. The RI MAY begin processing subsequent Work Requests posted to
       the Send Queue (except for operations which are affected by a
       fence - see Section 8.2.2.2), before Completing a prior RDMA
       Read Type Work Request (including zero-length RDMA Read Type
       Work Requests). Therefore, when an application does an RDMA Read
       Type Work Request followed by an RDMA Write or Send Type WR
       targeting the same buffer, it MAY return the data from the later
       RDMA Write or Send Type WR in the RDMA Read Operation Data Sink
       buffer, even though the operations Complete in order on the Send
       Queue's Completion Queue. If this behavior is not desired, the
       Local Peer Consumer must set the Read Fence indicator on the
       later RDMA Write or Send Type Work Request.

   16. Before an Inbound RDMA Read Request Message is processed (the
       specified buffer is read), the RI MUST have delivered all prior
       incoming RDMAP Messages initiated from the same Remote Peer's
       Send Queue. Therefore, when an application does an RDMA Write or
       Send Type Work Request followed by an RDMA Read Type Work



   Hilland, et al.        Expires October 2003             [Page 136]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       Request targeting the same remote buffer, the RDMA Read Type WR
       MUST return the data as modified by the prior operations.

   17. The RI MAY Complete incoming Send Message Types before the RI
       has finished generating RDMA Read Response Messages for an
       incoming RDMA Read Request Message (initiated from the same
       Remote Peer's Send Queue). Therefore, indeterminate results may
       occur if an application does an RDMA Read Type Work Request
       followed by a Send Type Work Request, and uses the Work
       Completion on the Associated QP's RQ Completion Queue (for the
       incoming Send Type Message) as an indicator that the inbound
       RDMA Read Operation processing has finished. If this behavior is
       not desired, the Local Peer Consumer must set the Read Fence
       indicator on the later RDMA Write (or Send Type) Work Request.

   18. If more RDMA Read Type Work Requests are posted to the Send
       Queue than are indicated by the ORD QP Attribute, the RI MUST
       pause the processing of the Send Queue until at least one prior
       RDMA Read Type WR Completes. If zero outbound RDMA Read Request
       Messages are supported on the QP, and the Consumer posts an RDMA
       Read Type Work Request, the RI MUST Complete the Work Request in
       error.

   Access by the RNIC to Memory Regions or Memory Windows are NOT
   REQUIRED to be cache-coherent. If an RNIC caches some portion of
   memory buffers during the time that the buffers are being processed
   by the RNIC, there is no requirement that updates to these buffers
   by any entity be seen by the RNIC. Also, any updates to these
   buffers by the RNIC are implementation dependent and may not be
   immediately seen by the system processor, other IO devices, or other
   RNICs.

8.2.2.1  Memory Management Operation Ordering

   This section defines the ordering constraints imposed on Work
   Requests. The next section defines additional ordering constraints
   that can be placed by using the Read Fence or Local Fence indicator.

   Because one of the objectives of DDP is to enable placement of
   incoming out-of-order DDP segments into the buffer provided by the
   Consumer, ordering semantics can not be guaranteed for certain
   operation combinations. If the Work Request sends payload to the
   Remote Peer, just because a Work Request Completes locally does not
   necessarily mean that the Remote Peer has received the data, or that
   subsequent DDP Segment payload can not overwrite the current data if
   targeting the same Remote Peer buffer.

   Thus, for example, an RDMA Write Message, containing payload1
   immediately followed by an RDMA Write Message containing payload2 to
   the same Remote Peer buffer location may result in the remote buffer



   Hilland, et al.        Expires October 2003             [Page 137]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   containing either payload1, payload2, or some combination of
   payload1 and payload2. Thus a programming model that does multiple
   RDMA Write WRs into the same Remote Peer buffer location without an
   end-to-end synchronization mechanism is NOT RECOMMENDED.

   1.  An Incoming Remote Invalidate (the Invalidate portion of the
       Send with Invalidate or Send with Solicited Event & Invalidate
       operation) MUST be performed after the Send Message payload is
       delivered to the appropriate Receive Queue Entry buffer, and
       before the Associated RQ WR Completes.

       Note: Send with Invalidate is usually used by Remote Peers to
       invalidate STags that were enabled for remote access and
       advertised to the Remote Peer. The expected usage is:

       a.  Local Peer Consumer creates a Send WR containing a command
           to be remotely executed and an STag enabled for Remote
           access and posts it to the Send Queue.

       b.  Remote Consumer gets the Send Message through a Completion
           of an RQ buffer, and does one or more accesses to the STag's
           buffers via RDMA Read Type WRs and/or RDMA Write WRs.

       c.  Remote Consumer creates a Send with Invalidate or Send with
           SE and Invalidate WR with the status from the Consumer's
           operation and the original STag to be invalidated as an
           input modifier. Note that the Read Fence indicator would
           most likely be set on the Send with Invalidate or Send with
           SE and Invalidate WR if the remote buffer to be Invalidated
           was accessed using an RDMA Read or RDMA Read with Invalidate
           Local STag WR.

       d.  RI at Local Peer gets the Send with Invalidate or Send with
           SE and Invalidate Message, places the data according to the
           RQ WQE, Invalidates the STag, and creates a CQE on the
           Receive Queue's Completion Queue, which also contains the
           Invalidated STag as part of the CQE.

       e.  Local Consumer checks that the Invalidate STag output
           modifier from the Work Completion is the same as was
           originally sent (as a check on the remote Consumer). If it
           was not, and the Consumer wishes to prevent remote access,
           the Consumer should post an Invalidate Local STag WR for the
           STag.

   2.  RDMA Read with Invalidate Local STag
       The Invalidate portion of the RDMA Read with Invalidate Local
       STag Work Request MUST be performed after the RDMA Read Response
       Message is delivered to the Data Sink buffers, and before a Work
       Completion is retrieved for the RDMA Read with Invalidate Local



   Hilland, et al.        Expires October 2003             [Page 138]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       STag WR. As with RDMA Read, subsequent operations MUST be
       allowed to begin executing before the Invalidate takes place,
       unless the subsequent operations have the Read Fence indicator
       set.

   3.  Fast-Register
       The RI MUST ensure that the Fast-Register operation takes effect
       prior to the execution of any subsequent Work Requests.

   4.  Bind
       The RI MUST ensure that the Bind Memory Window operation takes
       effect prior to the execution of any subsequent Work Requests.

   5.  Invalidate Local STag
       The Invalidate Local STag Work Request MUST take effect prior to
       the execution of any subsequent Work Requests.

   The RI MAY perform Fast-Register WRs, Bind WRs and Invalidate Local
   STag WRs at any time between the posting of the Work Request and the
   execution of a subsequent Work Request. Consequently, it is up to
   the Consumer to ensure that the posting of the Invalidate Work
   Request takes place after the STag is no longer in use.

   SQ processing of Memory Management Operations (Fast-Register, Bind
   and Invalidate Local STag) does not usually require the prior
   operation to Complete before the current operation begins execution.
   Thus it is possible to have an Invalidate Local STag operation be
   applied to an RDMA Write WR Data Source buffer before the RDMA Write
   Message payload has been completely sent. To ensure that this does
   not occur, the Local Fence indicator may be set to require that all
   prior operations Complete first (See Section 8.2.2.2).

   Note that performing a Fast-Register on an already registered
   region, or a Bind on a Window that is already Bound, will result in
   a Completion Error. As such, it is up to the application to ensure
   that the STag is in the Invalid state before the Fast-Register or
   Bind Memory Window Work Request is posted.

   The rules for Invalidate and Fast-Register or Bind Memory Window
   above are based on the following usage model:

       a.  Allocate an STag (through either Allocate Non-Shared Memory
           Region STag or Allocate Memory Window).

       b.  Fast-Register or Bind the STag

       c.  Use the STag in a manner compatible with its Access Rights.






   Hilland, et al.        Expires October 2003             [Page 139]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       d.  Wait for the Completion of the operations using the STag.
           This ensures that the STag and its related buffer is no
           longer in use.

       e.  Invalidate the STag

       f.  Loop to (b) as long as the STag is still needed; otherwise,
           Deallocate the STag.

8.2.2.2  Read Fence and Local Fence Indicators

   Two types of fence indicators are defined in Verbs -                                                      - a Read Fence
   indicator for RDMA Write or Send Type WRs, and a Local Fence
   indicator for Invalidate Local STag WRs. The Read Fence ensures that
   the current WR does not execute until all prior RDMA Read Type WRs
   Complete. The Local Fence indicator ensures that all prior
   operations Complete before the Invalidate Local STag WR is executed.

   Note that in the Verbs specification, a fence indicates that some
   set of prior operations have completed before the current operation
   begins. A different concept is operations that are required to
   Complete before future operations in the SQ can be executed -
   specifically Bind, Fast-Register, and Invalidate Local STag WR. By
   default, these operations do not ensure prior operations have
   completed before they execute. For Invalidate Local STag, if the
   Local Fence indicator is set, it can ensure that all prior SQ
   operations Complete before it executes.

   Note that RDMAP does not provide any end-to-end acknowledgement
   except for an RDMA Read Operation. Thus in general an end-to-end
   fence is not possible without using an RDMA Read Operation, unless
   an explicit ULP exchange of messages is done. Some operations are
   local only operations - specifically PostSQ Invalidate Local STag,
   Bind Memory Window and PostSQ Fast-Register. For combinations of
   these operations and the local buffers which they operate on (the
   Data Source for an RDMA Write and Send Type Operation, or the Data
   Sink for an RDMA Read Operation), it is possible to ensure that a
   current operation is not executed until prior operation which
   operate on the referenced local buffer are Completed.

   Figure 22 shows the fencing semantics when one operation is followed
   by another, and whether that operation will not execute until all
   prior operations have Completed, some prior operations have
   completed, or potentially no prior operations have completed. The
   rows are the first operation, and the columns are the second
   operation. The fields are defined as follows:

   *   NA-1 - a fence is not applicable. An Invalidate must precede
       Bind or Fast-Register. Thus in terms of potential WRs in the SQ,




   Hilland, et al.        Expires October 2003             [Page 140]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       it is the Invalidate Local STag operation that must be fenced to
       ensure proper operation.

   *   NA-2 - A fence is not applicable. This is because RDMAP allows
       RDMA Write Message payloads and Send Type Message payloads to be
       Placed out-of-order. Thus a local Completion of prior WRs does
       not ensure the payload has been Placed at the Remote Peer.

   *   Not Needed - A fence is not needed, because RDMAP requires that
       the RDMA Read Request Message at the Data Source (i.e. the
       Remote Peer) must be executed in order. Note that RDMAP does not
       ensure that operations which are sent after the RDMA Read
       Request Message occur after the RDMA Read Type WR Completes.
       Thus the need for the Read Fence Indicator for RDMA Write and
       Send Type WRs.

   *   Yes, Full - If the Local Fence indicator is set on the
       Invalidate Local STag WR, then the operation and subsequent
       operations will not be executed until all prior operations
       Complete. Note that this can effectively cause a pipeline stall
       in transmission of RDMAP Messages, and should be used
       judiciously.

   *   Yes, Partial - If the Read Fence indicator is set on the RDMA
       Write or Send Type WR, then all prior RDMA Read Type WRs must
       Complete before the current operation can begin execution.


























    Hilland, et al.        Expires October 2003             [Page 141]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   PostSQ     Send    RDMA    RDMA       Bind Fast-    Invalidate
   Work       Type    Write   Read            Register
   Request

   Send Type  NA-2    NA-2    Not Needed NA-1 NA-1     Yes, full

   RDMA Write NA-2    NA-2    Not Needed NA-1 NA-1     Yes, full

   RDMA Read  Yes,    Yes,    Not Needed NA-1 NA-1     Yes, full
               Partial Partial

   Bind       NA-2    NA-2    Not Needed NA-1 NA-1     Yes, full

   Fast-      NA-2    NA-2    Not Needed NA-1 NA-1     Yes, full
   Register

   Invalidate NA-2    NA-2    Not Needed NA-1 NA-1     Yes, full


                  Figure 22 - Fencing on Prior Operations

   The following paragraphs provide the rules which dictate the above
   behavior.

   Read Fence - set in RDMA Write or Send Type Work Requests to ensure
       all prior RDMA Read Type WRs have been processed by the RI.
       The RI MUST provide a Read Fence indicator for Send Type Work
       Requests and RDMA Write Work Requests. This indicator MUST cause
       the RI to pause before the execution of the Read Fenced Work
       Request if all prior RDMA Read Type Work Requests are not
       complete. Once all prior RDMA Read Type Work Requests are
       complete the RI MUST resume SQ processing.

   Local Fence - set in Invalidate Local STag Work Requests to ensure
       all prior operations have been processed by the RI.
       The RI MUST provide a Local Fence indicator for the Invalidate
       Local STag Work Request. This Indicator MUST cause the RI to
       wait until all prior Work Requests on the Send Queue Complete.
       Once all prior WRs on the SQ complete, the RI MUST resume SQ
       processing.

       Note: This indicator may be used by the Consumer when there are
       insufficient STags available to allow them to remain in use
       until the Consumer can process the Completions for Work Requests
       using those STags. For example, the following sequence could be
       used:

       a.  Allocate an STag (either Allocate Non-Shared Memory Region
           or Allocate Memory Window)



   Hilland, et al.        Expires October 2003             [Page 142]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       b.  Fast-Register or Bind the STag

       c.  Use the STag in a manner compatible with its Access Rights.

       d.  Invalidate the STag using an Invalidate Local STag Work
           Request with the with Local Fence indicator set.

       e.  Loop to (b) as long as the STag is still needed; otherwise,
           Deallocate the STag by invoking the Deallocate STag Verb.

       Using this model, the application can reuse an STag multiple
       times without having to wait for the prior Work Request to
       Complete before posting the next Work Request. Using the Local
       Fence indicator may require the RI to stall before processing
       the Invalidate Local STag Work Request, reducing the rate of
       Send Queue processing.

   Implementation of an end-to-end fence - using an RDMA Write WR
       followed by an RDMA Read Type WR.
       An end-to-end fence ensures that all outstanding operations have
       been flushed from the network fabric prior to the next operation
       executing. [RDMAP] enables an application to use an RDMA Read
       Operation to ensure that all RDMA Write Operations and Send Type
       Operations prior to the RDMA Read Operation on the same RDMAP
       Stream have made it to remote memory and can be read back by any
       other RDMAP Stream connecting through the same remote RNIC with
       access to the remote memory. The RDMA Read Operation need not be
       to any of the data written, and can even be a zero length RDMA
       Read Operation (which does not even require a valid Data Source
       STag) to have this effect. This enables the Consumer to
       implement an end-to-end fence by waiting for a RDMA Read WR
       Completion to determine that data is up to date at the Remote
       Peer.

       If the requirement, for example, is to ensure, from the Data
       Source, that one RDMA Write Message has been Placed at the
       Remote Peer before another RDMA Write Message occurs, the
       following sequence can be used by the Consumer:

       a.  Perform one (or more) RDMA Write WR(s).

       b.  Perform an RDMA Read Type WR (zero length is acceptable)

       c.  Perform a second RDMA Write WR with the Read Fence indicator
           enabled on the Work Request.

8.2.3  Completion Processing

   A CQE is an internal representation of the Work Completion. The
   results from a Work Request operation are placed in a Completion



   Hilland, et al.        Expires October 2003             [Page 143]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Queue Entry (CQE) on the CQ associated with the Work Queue when the
   request has completed. A CQE MUST be generated for each WQE that
   results in a Work Completion.

8.2.4  Returning Completed Work Requests

   All Work Completions are abstracted through the Verbs. The only
   method of retrieving a Work Completion MUST be through the Poll for
   Completion Verb. The RI MUST enable the Consumer to be able to
   retrieve WCs resulting from WRs posted to QPs which are in any valid
   QP state. Note that a destroyed QP is not in a valid QP state. See
   Section 6.1.4.

   A Work Request is confirmed Complete when the associated Work
   Completion is retrieved from its CQ. The RI MUST NOT return a Work
   Completion for an Unsignaled Work Request that completed
   successfully. When the RI returns a single WC through Poll for
   Completion, it MUST free at least one CQE. Note that more than one
   CQE may be freed due to Unsignaled Completions. See Section 8.1.3.1,
   Signaled Completions, for the rules on determining when Unsignaled
   Work Requests have Completed.

   When a Work Request has Completed, any Scatter/Gather Elements or
   other information associated with the original WR are no longer in
   the domain of the RI. The RI MUST NOT access any memory locations
   referenced by the Scatter/Gather Elements, Local Address or Remote
   Address for a WR that has Completed. The RI MUST provide Work
   Completions through the Poll for Completion Verb no more than once
   per Work Request. Note that if Destroy QP is invoked with Work
   Requests pending, the Work Completion may be lost.

   The Work Completion contents are specified in 9.3.2.1 - Poll for
   Completion.

   A Consumer is able to find out if a Work Completion is available by
   polling or notification.

   Work Completions MUST be returned when the Consumer polls the CQ in
   the following cases:

   *   On Completion of a Work Request submitted to a Send Queue with a
       Signaled Completion.

   *   On Completion of a Work Request submitted to a Send Queue that
       completed in error.

   *   On Completion of a Work Request submitted to a Receive Queue.

   When the Consumer desires to know if a QP has had all of its WRs
   retrieved and the Work Queues are empty, but there may be only



   Hilland, et al.        Expires October 2003             [Page 144]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Unsignaled Work Requests on the Send Queue, the Consumer can
   transition the QP to the Error state (See Section 6.2.4) and then to
   the Idle state. This will guarantee that all WRs have been
   Completed. In order to ensure that the WQEs have been freed and the
   entries on the CQ have been made available, the Consumer should free
   any associated CQEs, if any are consumed. There are three methods
   for a Consumer to free the CQE consumed within the CQ. They are:

   *   for the Consumer to poll the CQ (See Section 9.3.2.1 - Poll for
       Completion (Poll CQ)) until the CQ is empty, or

   *   the Consumer retrieves a WC for a WR submitted to a Work Queue
       associated with the same CQ where the former WR was submitted
       and the new WR was submitted after the previous QP was
       destroyed, or

   *   the Consumer polls (See Section 9.3.2.1 - Poll for Completion
       (Poll CQ) a number of Work Completions equal to the total number
       of entries that the CQ can hold.

8.2.5  Asynchronous Completion Notification

   A Consumer of a CQ may request asynchronous notification of when
   CQEs have been added to a Completion Queue by invoking the Request
   Completion Notification Verb. The Verbs architecture assumes a
   Privileged Mode intermediary will process Asynchronous CQ Events for
   CQs. The Verbs architecture allows this intermediary to register one
   or more CQ Event Handlers for Asynchronous CQ Events by invoking the
   Set Completion Event Handler Verb. It is the responsibility of this
   intermediary to create the asynchronous completion notification to
   the Consumer that called the Request Completion Notification Verb.

   A Completion Event Handler Identifier delineates each Completion
   Event Handler. The Set Completion Event Handler is invoked once per
   supported Completion Event Handler. Note that the maximum number of
   supported Completion Event Handlers is returned by Query RNIC.

   Each Set Completion Event Handler invocation can be used to:

   *   Return a Completion Event Handler Identifier that is used as an
       input modifier to Create CQ (to associate a CQ with a Completion
       Event Handler).

   *   Clear a Completion Event Handler associated with the Completion
       Event Handler Identifier.

   *   Modify the address of the Completion Event Handler for the
       Completion Event Handler Identifier.





   Hilland, et al.        Expires October 2003             [Page 145]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   The RI is NOT REQUIRED to disassociate CQs from CQ Event Handlers
   when those CQ Event Handlers associated with the Completion Event
   Handler Identifiers are cleared. If a CQ Event Handler is cleared
   and the Consumer still has CQs associated with that CQ Event Handler
   (through the CQ Event Handler Identifier), and a Completion occurs
   which would have invoked the CQ Event Handler, behavior of the RI is
   indeterminate. The Consumer should keep this in mind before clearing
   the association to prevent indeterminate behavior, such as possible
   race conditions.

   The Request Completion Notification Verb is set on a per CQ basis.
   When armed, the RI MUST generate at most one notification until the
   notification has been rearmed by invoking Request Completion
   Notification Verb. Once Completion Notifications have been enabled,
   additional Request Completion Notification calls have no effect. The
   Completion Event Handler will be called only once when the next CQE
   is added to the CQ. The RI MUST invoke the Completion Event Handler
   associated with the CQ Event Handler Identifier which is associated
   with the CQ where the CQE was added. Once the Completion Event
   Handler routine has been invoked, the Consumer should call Request
   Completion Notification again to be notified when a new entry is
   added to the CQ, since the notification is a "one shot" mechanism.

   Existing CQEs on the CQ at the time the notification is enabled do
   not result in a call to the Completion Event Handler. The Completion
   Event Handler MUST be called when the next CQE is added to the CQ
   after the Request Completion Notification has been set.

   The RI MUST provide the ability for the Consumer to specify whether
   the Completion Event Handler is invoked for either:

   *   the next Solicited Completion Event only, or

   *   the next Completion Event.

   If the local Consumer requests the next solicited Completion in the
   Request Completion Notification Verb, the RI MUST generate a
   Completion Event when:

   *   an incoming Send with Solicited Event or Send with SE and
       Invalidate successfully causes a Receive Queue's WQE to be
       consumed, and thus a CQE to be added to a CQ, or

   *   a Work Completion for a Work Request which Completed in error is
       added to a CQ.

   If the Consumer requested an event for the next completion in the
   Request Completion Notification Verb, the RI MUST generate a
   Completion Event when any incoming Send operation type or Signaled
   Local SQ WR completes.



   Hilland, et al.        Expires October 2003             [Page 146]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   If multiple calls to Request Completion Notification have been made
   for the same CQ and at least one of the requests set the type to the
   next Work Completion, the RI MUST invoke the CQ event handler when
   the next CQE is added to that CQ. The CQ Event Handler MUST be
   called only once, even if multiple CQ notification requests were
   made prior to the Completion Event for the specified CQ.

   The RI MUST ensure that the following sequence of events will not
   result in a Completion Notification being missed. Therefore, the
   following sequence of calls should be used by the Consumer when
   using Request Completion Notification in order to ensure that a new
   CQE is not missed for the specified CQ:

   *   Call Poll for Completion to dequeue all existing CQ entries

   *   Call Request Completion Notification.

   *   Call Poll for Completion to dequeue all of the CQ entries that
       were added between the time the last Poll for Completion was
       called and the notification was enabled.

   When the Completion Event Handler is invoked, the RI MUST supply the
   CQ handle of the CQ which generated the Completion notification.

   The Consumer is responsible for polling the CQ to retrieve the Work
   Completion. This function MUST NOT be performed automatically by the
   RI when the notification occurs.

   For details on the Asynchronous Completion Verbs, refer to Section
   9.4.1 - Set Completion Event Handler and Section 9.3.2.2 - Request
   Completion Notification.

8.3  Error Handling

   The following section details many of the errors that can occur when
   using the RNIC, and the responsibilities of the RNIC and the
   Consumer.

   Errors are returned to the Consumer by one of three mechanisms:
   Immediate Errors, Work Completions, or Asynchronous Error Events.
   Immediate Errors are returned immediately as an Output Modifier of a
   Verb. Work Completions are used when the error can be related
   directly to the Work Request in progress. Asynchronous Error Events
   are used when the error can only be localized to the QP, CQ or RNIC
   but are not directly attributable to any single Work Request. Each
   of these errors is described below.







   Hilland, et al.        Expires October 2003             [Page 147]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

8.3.1  Immediate Errors

   Immediate Errors are those surfaced as Verb results provided to the
   Consumer via Output Modifiers. The individual Immediate Errors are
   documented within each Verb in Section 9 - RNIC Verbs. A summary of
   all of the Immediate Errors are covered in Section 9.5.1 - Immediate
   Status Codes.

   When the RI returns an Immediate Error, the RI MUST NOT affect the
   RI Resource that is the subject of the verb for which the Immediate
   Error is being returned, except for RI-Reregister Non-Shared Memory
   Region (which has slightly different rules). That is, for an
   Immediate Error returned on any verb that has the:

   - RI as the subject, the RI remains unchanged;

   - CQ as the subject, the CQ remains unchanged;

   - QP as the subject, the QP remains unchanged;

   - S-RQ as the subject, the S-RQ remains unchanged;

   - STag as the subject, the STag remains unchanged (except certain
   rules for RI-ReRegister Memory Region);

   - PD as the subject, the PD remains unchanged;

   - Asynchronous Event handling as the subject, Asynchronous Events
   must not be lost.

8.3.2  Work Completion Errors

   The following errors can be associated with a specific Work Request.
   The RI MUST return a Completion Error via a Work Completion on the
   Completion Queue associated with the Send or Receive Queue on which
   the Work Request was posted for the errors defined in Figure 23. The
   Work Completion's Completion Status field contains the Error
   information. In each case, the QP MUST be moved to the Terminate
   state and a Terminate Message is sent with the indicated Terminate
   code (see Section 6.6.2.5 - Local Termination, Local Abortive
   Teardown and Remote Abortive Teardown). On any Work Completion that
   includes the sending of a Terminate Message, the Terminate Message
   Buffer MUST be available for examination while the QP is in the
   Terminate state or Error state using Query QP. The Terminate Message
   may contain useful diagnostic information, depending on the error.
   For information on the format of the Terminate Message, see [RDMAP].







   Hilland, et al.        Expires October 2003             [Page 148]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

Error                            Terminate Action
                                 Code

Receive Queue Work Request Errors - These errors are probably due to a
local Consumer error.

Invalid WQE format,              0x0000    The RI Terminates the LLP
Invalid STag in SGE,                        Stream with Local
Base and bounds violation                   Catastrophic Error and the
(including length errors),                  QP transitions to the
Access Rights violation,                    Terminate state.
Invalid PD ID,
Wrap error (TO & Segment Length
caused an address to wrap).

Receive Queue Remote Protection Errors - These errors may be due to a
Consumer error at either end.

Invalidate STag Invalid.         0x0100    The RI Terminates the LLP
                                            Stream with the indicated
Invalidate STag Access Rights.   0x0102     Error and the QP
                                            transitions to the
Invalidate STag Invalid PD ID.   0x0103     Terminate state.
or STag not Bound to QP.

Invalidate MR STag had Bound MW. 0x0109

Send Queue Work Request Errors - These errors are probably due to a
local Consumer error.

Invalid WQE format,              0x0000    The RI Terminates the LLP
Zero ORD.                                   Stream with Local
                                            Catastrophic Error and the
                                            QP transitions to the
                                            Terminate state.

Local SQ Protection Errors -     0x0000    The RI Terminates the LLP
Send Types, RDMA Writes, and                Stream with Local
RDMA Read Types:                            Catastrophic Error and the
Invalid STag,                               QP transitions to the
Base and bounds violation                   Terminate state.
(including length errors),
Access Rights violation,
Invalid PD ID,
Wrap error.







   Hilland, et al.        Expires October 2003             [Page 149]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

              Error              Terminate Action
                                 Code

SQ Fast-Register errors:         0x0000    The RI Terminates the LLP
QP not in Privileged Mode,                  Stream with Local
Invalid Region STag,                        Catastrophic Error and the
Invalid Physical Buffer Size,               QP transitions to the
Physical Buffer List too long,              Terminate state.
STag not in Invalid state,
Invalid PD ID,
Invalid Access Rights Specified,
Invalid Virtual Address,
Invalid FBO,
Invalid Length

SQ Bind errors:                  0x0000    The RI Terminates the LLP
Invalid Region STag                         Stream with Local
Invalid Window STag                         Catastrophic Error and the
Base and bounds violation                   QP transitions to the
Access Rights violation                     Terminate state.
STag not in Invalid state
MR not in Valid state
Invalid PD ID

SQ Invalidate errors             0x0000    The RI Terminates the LLP
 (Footnote 6):                              Stream with Local
Invalid STag                                Catastrophic Error and the
Invalid PD ID (or QP ID)                    QP transitions to the
Invalidate MR STag had Bound MW             Terminate state.

       Figure 23 - Completion Errors with Resulting Terminate Codes

8.3.3  Asynchronous Errors

   The Consumer may register an Asynchronous Event Handler to be called
   when an Asynchronous Event occurs which is not associated with an
   individual CQE by using the Set Asynchronous Event Handler Verb.

   An input modifier to the Set Asynchronous Event Handler Verb is the
   address of the event handler routine. This is a Consumer routine
   that is invoked when an Asynchronous Event is generated. When the
   handler routine is invoked, an indication of the origin of the
   error, called an Event Record, is provided.




   Footnote 6: This includes RDMA Read and Invalidate.




   Hilland, et al.        Expires October 2003             [Page 150]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   The errors defined in Figure 24 are returned to the Consumer via an
   Event Record in the Asynchronous Event Handler.

   There is only one Asynchronous Event Handler per RNIC. If Set
   Asynchronous Event Handler Verb is called more than once, the new
   handler MUST replace the previous handler. The RI MUST turn off
   Asynchronous Event Notification if the Asynchronous Event Handler's
   address is zero.

   After the Asynchronous Event Handler is registered, all subsequent
   asynchronous events not associated with a CQE MUST result in a call
   to the handler. Until an Asynchronous Event Handler is registered,
   asynchronous events will be lost.

   For more information, see Section 9.4.2 - Set Asynchronous Event
   Handler and Section 9.5.3 - Asynchronous Event Identifiers.

   The following table covers the errors that can be associated with a
   QP, thus the Event Record should include the QP ID when the error is
   associated with a specific QP. On any Asynchronous Error Event that
   includes the reception or sending of a Terminate Message, the
   Terminate Message Buffer is available for examination while the QP
   is in the Terminate or Error state by retrieving it through Query
   QP. Note that Terminate Messages generated locally as well as
   Terminated Messages received from the Associated QP are available
   through Query QP. The Terminate Message may contain useful
   diagnostic information, depending on the error. For information on
   the format of the Terminate Message, see [RDMAP].

              Error               Terminate Action
                                   Code

Remotely detected Errors

"Terminate Message Received"       None      QP -> Terminate state. See
An incoming Terminate Message has            6.6.2.4 Remote Termination
arrived.















   Hilland, et al.        Expires October 2003             [Page 151]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

              Error               Terminate Action
                                   Code

LLP Errors - Errors on incoming RDMAP Segments or Messages probably due
to the Remote Peer or fabric corruption.

"LLP Connection Lost" -            None      QP -> Error state. See
Usually caused by Timeout or Too             6.6.2.4 Remote Termination
many Retries at the LLP.

"LLP Connection Reset" -           None      QP -> Error state. See
Caused by an incoming Reset at the           6.6.2.4 Remote Termination
LLP.

"LLP Integrity Error: Segment size 0x1000    If this cannot be
invalid" -                                   corrected by the LLP (drop
The incoming segment is too small            and retry etc.), then
to contain a valid RDMAP header,             QP -> Terminate state.
or larger than supported by this             The RI Terminates the LLP
implementation.                              Stream with the indicated
                                             error. See 6.6.2.5.
" LLP Integrity Error: Invalid     0x0202
CRC" -
The incoming segment had a bad LLP
CRC.

"Bad FPDU" -The incoming segment   0x0203
Received MPA marker and 'Length'
fields do not agree on the start
of a FPDU






















   Hilland, et al.        Expires October 2003             [Page 152]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

              Error               Terminate Action
                                   Code

Remote Operation Errors - Protocol Errors on incoming RDMAP Segments or
Messages probably due to the Remote Peer.

Invalid DDP version                0x1206    QP -> Terminate state. The
                                             RI Terminates the LLP
Invalid RDMA version               0x0205    Stream with the indicated
                                             error. See 6.6.2.5.
Unexpected Opcode                  0x0206

Invalid DDP Queue Number           0x1201

Invalid RDMA Read Request - RDMA   0x1201
Read not enabled

No 'L' bit when expected           0x0207

Remote Protection Errors (not associated with the RQ) - Protection
Errors on incoming DDP Segments or RDMAP Messages that are not RDMA
Read Request Messages, probably due to the Remote Peer's Consumer.

Invalid STag                       0x1100    QP -> Terminate state. The
                                             RI Terminates the LLP
Base and bounds violation          0x1101    Stream with the indicated
                                             error. See 6.6.2.5.
Access Rights violation            0x1102

Invalid PD ID                      0x1102

Wrap error - TO and segment length 0x1103
caused an address wrap past
0xFFFFFFFFFFFFFFFF


















   Hilland, et al.        Expires October 2003             [Page 153]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

              Error               Terminate Action
                                   Code

Remote Closing Error - Probably due to Consumer not properly
synchronizing the ULP close operation.

Bad Close - QP in Closing state    None      QP -> Error state.
and: Segment arrives, at least one
SQ WQE on the SQ, or RDMA in
progress.

Bad LLP Close - LLP Close received 0x0207    QP -> Terminate state. The
AND (the Send Queue was NOT empty            RI Terminates LLP Stream
OR the IRRQ was NOT empty)                   with indicated error. See
(Footnote 7)                                 6.6.2.5.

Remote Protection Errors associated with the Receive Queue - Protection
Errors on incoming RDMAP Segments or Messages probably due to the
Remote Peer's Consumer.

Invalid MSN - MSN range not valid  0x1202    QP -> Terminate state. The
                                             RI Terminates LLP Stream
                                             with indicated error. See
                                             6.6.2.5.

Invalid MSN - gap in MSN           0x1202    QP -> Terminate state. The
                                             RI Terminates LLP Stream
                                             with indicated error. See
                                             6.6.2.5.

IRRQ Protection Errors - Error processing an incoming RDMA Read Request
and generating the outgoing RDMA Read Response.

Invalid STag                       0x0100    QP -> Terminate state. The
                                             RI Terminates the LLP
Base and bounds violation(includes 0x0101    Stream with the indicated
RDMA Read Request larger than                error. See 6.6.2.5.
supported by the Data Source STag)

Access Rights violation            0x0102

Invalid PD ID                      0x0103




   Footnote 7: For TCP this would be a 1/2 close and a Terminate
   Message could be sent. For SCTP, no Terminate Message is sent.



    Hilland, et al.        Expires October 2003             [Page 154]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

              Error               Terminate Action
                                   Code

Wrap error - TO and length caused  0x0104
an address wrap past
0xFFFFFFFFFFFFFFFF

Invalid MSN - too many RDMA Read   0x1203
Request Messages in process

Invalid MSN - gap in MSN (RDMA     0x1203
Messages found missing when LLP
claims a Message is delivered.)

Invalid MSN - MSN range is not     0x1203
valid (MSN is unreasonably beyond
the end of the queue.)

Local Errors

CQ/SQ error - An error occurred on 0x0207    QP -> Terminate state. The
the CQ during a SQ completion.               CQ number itself must be
CQ Overflow error                            determined by using Query
CQ Operation error                           QP. The RI Terminates the
                                             LLP Stream with the
CQ/RQ error - An error occurred on 0x0207    indicated error. See
the CQ during a RQ completion.               6.6.2.5.
CQ Overflow error
CQ Operation error

S-RQ error on a QP - An error      0x0207    QP-> Terminate state. The
occurred while attempting to pull            S-RQ can be determined by
a WQE from the S-RQ associated               using Query QP. The RI
with the QP.                                 Terminates the LLP Stream
                                             with the indicated error.
                                             See 6.6.2.5.

Local QP Catastrophic Error - An   0x0207    The RI will attempt to
error related to the QP occurred             move the QP to the Error
while processing (probably a                 state. The QP is most
problem with the RNIC).                      likely unusable and should
                                             be destroyed.

      Figure 24 - Affiliated Asynchronous Errors with Terminate Codes

   Figure 25 indicates errors that cannot be associated with a QP; the
   Asynchronous Event Record MUST contain the additional information as
   indicated in the table.



   Hilland, et al.        Expires October 2003             [Page 155]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

              Error               Terminate Action
                                  Code

Locally detected Catastrophic Errors

CQ Operation Error - An error     None      The Asynchronous Event
occurred on the CQ unrelated to             Record includes the CQ
a specific QP completion.                   handle. All completions on
                                            the CQ are in an undefined
                                            state. It may be necessary
                                            to destroy any QPs
                                            targeting the CQ and
                                            destroy the CQ.

Shared Receive Queue              None      The Asynchronous Event
Catastrophic Failure - A problem            Record includes the S-RQ
occurred with the RNIC or its               handle. All WRs on the S-RQ
driver that renders the RNIC                are in an undefined state.
unable to use the S-RQ.                     It may be necessary to
                                            destroy any QPs using the
                                            S_RQ and destroy the S-RQ.

RNIC Catastrophic failure - A     0x0208    The Asynchronous Event
problem occurred with the RNIC              Record does not include any
or its driver that renders the              additional information. If
RNIC unable to reliably                     possible, the RI Terminates
function. All RNIC/QP/CQ state              all LLP Connections with
is indeterminate. The only                  Global Catastrophic Error.
recovery is to close the RNIC               See 6.6.2.5
(and reopen it if desired).

     Figure 25 - Unaffiliated Asynchronous Errors with Terminate Codes




















   Hilland, et al.        Expires October 2003             [Page 156]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

9  RNIC Verbs

   The Verbs described in this chapter provide an abstract definition
   of the functionality provided to a host by a RI. Host RIs that are
   compliant with this specification MUST exhibit the semantic behavior
   described by the Verbs.

   Since the Verbs define the behavior of the host RI, they may
   influence the design of software constructs, such as application
   programming interfaces (APIs), which provide access to the host RI.
   However, this specification explicitly does not define any such API.
   In particular, there is no requirement that an API used with a
   compliant host RI be semantically identical to, or expose the
   semantics of, the Verbs. For example, whether the input modifiers
   referenced in the Verbs are pass-by-reference or pass-by-value is
   outside the scope of this specification.

   It is OPTIONAL for an RI to implement Block Lists. It is OPTIONAL
   for an RI to implement S-RQs. Support for S-RQs can be discovered
   using Query RNIC. Support for Block Lists can be discovered by
   attempting to open the RNIC in Block Mode. If the Verb fails with
   the error "Block List Not Supported", the RNIC does not support
   Block Mode.

   The RI MUST use the values and information provided in the Input
   Modifiers when processing the requests and operations instantiated
   in the Verbs for mandatory features. The RI MUST use the values and
   information provided in the Input Modifiers when processing the
   requests and operations instantiated in the Verbs for optional
   features if the RI supports that optional feature.

9.1  Consumer Accessibility

   Verb Consumers are the direct users of the Verbs, and are sub-
   divided into two classes, Privileged and Non-Privileged.

   Privileged Consumers are typically those Consumers that operate at a
   privilege level sufficient to access OS internal data structures
   directly, and have the responsibility to control access to the RNIC
   Interface. All Verbs are available for use by Privileged Consumers.

   Non-Privileged Mode Consumers are those Consumers that must rely on
   another agent, having a sufficient high level of privilege, to
   manipulate OS data structures. Only those Verbs specifically labeled
   as such are available to be used by Non-Privileged Mode Consumers.
   Conceptually, the intent is that Non-Privileged Mode Consumers are
   not allowed to manipulate RI resources that could affect a QP in a
   different Protection Domain. Any manipulation of resources that can
   affect another Protection Domain, such as registering physical




   Hilland, et al.        Expires October 2003             [Page 157]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   memory, are assumed to be done by a trusted intermediary, or
   Privileged Consumer.

   The Protection Domain provides a mechanism to detect when a Consumer
   is posting WRs to QPs with which it is not associated. The RI also
   usually provides a mechanism to help prevent posting WRs to QPs not
   directly owned by the Consumer (e.g. a multi-Consumer application
   which shares the same PD). But it may still be possible to post a WR
   to a QP that is not owned by the Consumer in some environments.
   Preventing access to memory structures such as QPs not directly
   created by that Consumer can be partially provided by the Local
   HostÆs operating environment through the use of the virtual memory
   subsystem and mapping of RNIC resources. Since this is
   implementation and environment dependent, the mechanism describing
   it is outside the scope of the architecture.

   All Verbs can be accessed by Privileged Mode Consumers. To maintain
   the access control over RI resources, the host environment MUST
   provide Non-Privileged Mode Consumers with direct access to only the
   following Verbs:

   *   PostSQ

   *   PostRQ

   *   Poll for Completion

   *   Request Completion Notification

9.2  RNIC Resource Management

9.2.1  RNIC

9.2.1.1  Open RNIC

   Description:

       Opens the specified RNIC.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 5.1.2 - Opening an RNIC.

   Input Modifiers:

   *   The unique identifier for this RNIC. The naming scheme is
       implementation dependent.






   Hilland, et al.        Expires October 2003             [Page 158]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   The Physical Block List mode of the RNIC. This MUST either be
       Block List mode or Page List mode. Block List mode is only valid
       if the RNIC supports it.

   Output Modifiers:

   *   If the operation completed successfully:

       o   RNIC Handle.

   *   Verb Results:

       o   Operation completed successfully.

       o   Insufficient resources to complete request.

       o   Invalid Modifier (RNIC name).

       o   Block List mode not supported.

       o   RNIC in use.

9.2.1.2  Query RNIC

   Description:

       Returns the attributes for the specified RNIC.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 5.1.3 - Query RNIC.

   Input Modifiers:

   *   RNIC handle.

   Output Modifiers:

   *   RNIC Attributes & Values, if the operation completed
       successfully:

       o   Vendor specific information. This could, but is not required
           to, include information such as a vendor identifier, part
           number and/or hardware version.

       o   The maximum number of QPs supported by this RNIC.

       o   The maximum number of outstanding Work Requests on any Send
           Queue or Receive Queue supported by this RNIC.




   Hilland, et al.        Expires October 2003             [Page 159]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   The maximum number of outstanding Work Requests on any S-RQ
           supported by this RNIC. If S-RQs are not supported by this
           RNIC, this number is zero.

       o   The maximum number of Scatter/Gather Elements per Send
           Operation Type Work Request supported by this RNIC. This
           value also applies to the maximum number of Scatter/Gather
           Elements for WRs posted to Receive Queues as well as those
           posted to Shared-Receive Queues.

       o   The maximum number of Scatter/Gather Elements per RDMA Write
           Work Request supported by this RNIC.

       o   The maximum number of CQs supported by this RNIC.

       o   The maximum number of entries in each CQ supported by this
           RNIC.

       o   The maximum number of CQ Event Handlers supported by this
           RNIC.

       o   The maximum number of Memory Regions supported by this RNIC.

       o   The maximum number of Physical Buffer Entries per Physical
           Buffer List.

       o   The maximum number of Protection Domains supported by this
           RNIC.

       o   The maximum number of inbound RDMA Read Request Messages
           that can be in the IRRQ per RNIC. This is the per RNIC
           parameter that represents the maximum total value of IRD for
           all QPs. This value MUST be Zero if the resources used to
           handle Inbound RDMA Read Requests are not shared between
           QPs. (For more information, see Section 6.5 - Outstanding
           RDMA Read Resource Management)

       o   The maximum number of outbound RDMA Read Request Messages
           that can be outstanding per RNIC. This is the per RNIC
           parameter that represents the maximum total value of ORD for
           all QPs. This value is Zero if the resources used to handle
           outstanding Outbound RDMA Read Request Messages are not
           shared between QPs.

       o   The maximum number of inbound RDMA Read Request Messages
           that can be in the IRRQ per QP. This represents the maximum
           value for IRD for any QP.






   Hilland, et al.        Expires October 2003             [Page 160]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   The maximum number of outbound RDMA Read Request Messages
           that can be outstanding per QP. This represents the maximum
           value for ORD for any QP.

       o   Ability of this RNIC to support modifying IRD after the QP
           has been created.

       o   Ability of this RNIC to support increasing ORD after the QP
           has been created.

       o   The maximum number of Memory Windows supported by this RNIC.

       o   The ability of this RNIC to support modifying the maximum
           number of outstanding Work Requests per QP. (For more
           information, see Section 6.1.3 - Modifying Queue Pair
           Attributes)

       o   The Physical Block List mode of the RNIC. This MUST either
           be Block List Mode or Page List Mode.

       o   If Block List Mode is supported:

           +   The Physical Buffer Entry range of sizes supported by
               this RNIC.

       o   If Page List Mode is supported:

           +   The List of Page sizes supported by this RNIC.

       o   The ability of this RNIC to support Shared Receive Queues.

       o   The ability of this RNIC to perform CQ Overflow detection.

       o   If Shared Receive Queues are supported:

           +   The maximum number of Shared Receive Queues supported by
               this RNIC.

           +   The dequeuing model the RNIC supports: arrival order or
               sequential order.

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC handle.

9.2.1.3  Close RNIC

   Description:



   Hilland, et al.        Expires October 2003             [Page 161]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       Closes and resets the specified RNIC.

       This Verb is responsible for de-allocating resources allocated
       by the RI and to make the RNIC unavailable for use by the
       Consumer.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 5.1.4 - Closing an RNIC.

   Input Modifiers:

   *   RNIC handle.

   Output Modifiers:

   *   Verb Results

       o   Operation completed successfully.

       o   Invalid RNIC handle.

9.2.2  Protection Domain

9.2.2.1  Allocate PD

   Description:

       Allocates an unused Protection Domain.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 5.2.1 - Allocating a PD.

   Input Modifiers:

   *   RNIC Handle.

   Output Modifiers:

   *   If the operation completed successfully:

       o   PD ID.

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC handle.




   Hilland, et al.        Expires October 2003             [Page 162]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   Insufficient resources to complete request.

9.2.2.2  Deallocate PD

   Description:

       Deallocates a previously Allocated Protection Domain.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 5.2.2 - Deallocating a PD.

       The Protection Domain MUST NOT be deallocated if it is still
       associated with any Queue Pair, Non-Shared Memory Region, Shared
       Memory Region, Shared Receive Queue, Bound Memory Window or
       Invalidated Memory Window.

   Input Modifiers:

   *   RNIC Handle.

   *   PD ID.

   Output Modifiers:

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid PD ID.

       o   Invalid RNIC handle.

       o   Protection Domain is in use.

9.2.3  Completion Queue

9.2.3.1  Create CQ

   Description:

       Creates a CQ on the specified RNIC. In addition, a Completion
       Event Handler may be registered for the created CQ.

       The Consumer must specify the minimum number of entries in the
       CQ. The number of allocated entries for CQEs on the specified
       CQ, which might be different than the number requested, is
       returned on successful creation. The number returned differs
       only when the number of actual entries is greater than the
       number that the Consumer requested. If the maximum number of



   Hilland, et al.        Expires October 2003             [Page 163]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       entries the RNIC supports is less than the Consumer requested,
       an Immediate Error is returned and the CQ is not created.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 5.3.1 - Creating a Completion Queue.

   Input Modifiers:

   *   RNIC handle.

   *   The minimum number of entries in the CQ.

   *   Completion Event Handler Identifier - An opaque handle used to
       identify a Completion Event Handler. If the identifier is set to
       zero, then there is no Completion Event Handler associated with
       this CQ. Completion Event Handler Identifiers are obtained via
       the Set Completion Event Handler Verb.

   Output Modifiers:

   *   If the operation completed successfully:

       o   The handle of the newly created CQ.

       o   The allocated number of entries in the CQ.

   *   Verb Results:

       o   Operation completed successfully.

       o   Insufficient resources to complete request.

       o   Invalid RNIC handle.

       o   Number of CQ entries requested exceeds RNIC capability.

       o   Invalid Completion Event Handler Identifier

9.2.3.2  Query CQ

   Description:

       Returns the number of entries in the specified CQ.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 5.3.2 - Querying Completion Queue Attributes.

   Input Modifiers:



   Hilland, et al.        Expires October 2003             [Page 164]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   RNIC handle.

   *   CQ handle.

   Output Modifiers:

   *   If the operation completed successfully:

       o   The allocated number of entries in the CQ.

       o   The Completion Event Handler Identifier.

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC handle.

       o   Invalid CQ handle.

9.2.3.3  Modify CQ

   Description:

       Resizes the CQ.

       A CQ must be able to be resized with outstanding Work
       Completions on the CQ and Work Requests on queues associated
       with the specified CQ. If the requested minimum number of
       entries in the CQ is insufficient to hold the current number of
       entries on the CQ, an Immediate Error will result.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 5.3.3 - Modifying Completion Queue Attributes.

   Input Modifiers:

   *   RNIC handle.

   *   CQ handle.

   *   The minimum number of entries in the CQ.

   Output Modifiers:

   *   If the operation completed successfully:

       o   The allocated number of entries in the CQ.




   Hilland, et al.        Expires October 2003             [Page 165]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   Verb Results:

       o   Operation completed successfully.

       o   Insufficient resources to complete request.

       o   Invalid RNIC handle.

       o   Invalid CQ handle.

       o   Number of CQ entries requested exceeds RNIC capability.

       o   An Attempt to shrink the size of the queue failed because
           too many Completion Queue Entries were still present on the
           Completion Queue.

9.2.3.4  Destroy CQ

   Description:

       Destroys the specified CQ.

       The CQ cannot be destroyed if any Work Queue is still associated
       with the CQ.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 5.3.4 - Destroying a Completion Queue.

   Input Modifiers:

   *   RNIC handle.

   *   CQ handle.

   Output Modifiers:

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC handle.

       o   Invalid CQ handle.

       o   One or more Work Queues is still associated with the CQ.







   Hilland, et al.        Expires October 2003             [Page 166]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

9.2.4  Shared Receive Queue

9.2.4.1  Create S-RQ

   Description:

       Creates an S-RQ for the specified RNIC.

       A set of initial S-RQ attributes must be specified by the
       Consumer. If any of the required initial attributes are illegal
       or missing, an error is returned and the S-RQ is not created.

       The RI MUST support this Verb if the Query RNIC Output Modifier
       indicates support for an S-RQ and MUST support all of the Input
       & Output Modifiers in this case, except where noted. For more
       information, see Section 6.3.1 - Creating a Shared Receive
       Queue.

   Input Modifiers:

   *   RNIC handle.

   *   The maximum number of outstanding Work Requests the Consumer
       expects to submit to the Shared Receive Queue.

   *   The S-RQ Limit. The S-RQ Limit detection is armed by the RI upon
       creation of the S-RQ, if the S-RQ Limit is non-zero.

   *   The maximum number of Scatter/Gather Elements the Consumer can
       specify in a Work Request.

   *   PD ID.

   Output Modifiers:

   *   If the operation completed successfully:

       o   The S-RQ Handle.

       o   The allocated number of outstanding Work Requests the
           Consumer can submit to the Shared Receive Queue.

       o   The allocated number of scatter/gather elements that can be
           specified in Work Requests. If an error is not returned,
           this is guaranteed to be greater than or equal to the number
           requested.

   *   Verb Results:

       o   Operation completed successfully.



   Hilland, et al.        Expires October 2003             [Page 167]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   Insufficient resources to complete request.

       o   Invalid RNIC handle.

       o   Maximum number of Work Requests requested exceeds RNIC
           capability.

       o   Maximum number of scatter/gather elements per Receive Queue
           Work Request requested exceeds RNIC capability.

       o   Invalid PD ID.

       o   S-RQ Limit out of range.

9.2.4.2  Query S-RQ

   Description:

       Returns the attribute list and current values for the specified
       S-RQ.

       The RI MUST support this Verb if the Query RNIC Output Modifier
       indicates support for an S-RQ and MUST support all of the Input
       & Output Modifiers in this case, except where noted.

   Input Modifiers:

   *   RNIC Handle.

   *   S-RQ Handle.

   Output Modifiers:

   *   The S-RQ attributes, if the operation completed successfully.
       The list of attributes returned by the query are:

       o   The allocated number of outstanding Work Requests supported
           on the Shared Receive Queue.

       o   The allocated number of Scatter/Gather Elements supported on
           Work Requests submitted to the Shared Receive Queue.

       o   PD ID.

       o   The S-RQ Limit.

       o   S-RQ Limit Armed Indicator.

   *   Verb Results:




   Hilland, et al.        Expires October 2003             [Page 168]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   Operation completed successfully.

       o   Invalid RNIC handle.

       o   Invalid S-RQ handle.

9.2.4.3  Modify S-RQ

   Description:

       Modifies the attributes for the specified S-RQ.

       The RI MUST support this Verb if the Query RNIC Output Modifier
       indicates support for an S-RQ and MUST support all of the Input
       & Output Modifiers in this case, except where noted. For more
       information, see Section 6.3.2 - Modifying a Shared Receive
       Queue.

   Input Modifiers:

   *   RNIC Handle.

   *   S-RQ Handle.

   *   The S-RQ attributes to modify and their new values. The S-RQ
       attributes that can be modified after the S-RQ has been created
       are:

       o   The maximum number of outstanding Work Requests the Consumer
           expects to submit to the Shared Receive Queue (if changing
           is supported by the RNIC).

       o   The S-RQ Limit.

       o   Re-arm the S-RQ Limit Asynchronous Event.

   Output Modifiers:

   *   If the operation completed successfully:

       o   The allocated number of outstanding Work Requests supported
           on the Shared Receive Queue.

   *   Verb Results:

       o   Operation completed successfully.

       o   Insufficient resources to complete request.

       o   Invalid RNIC handle.



   Hilland, et al.        Expires October 2003             [Page 169]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   Invalid S-RQ handle.

       o   Maximum number of Shared Receive Queue Work Requests
           requested exceeds RNIC capability.

       o   An Attempt to shrink the size of the queue failed because
           too many elements were still present.

       o   S-RQ Limit out of range.

       o   Invalid Input Modifier.

9.2.4.4  Destroy S-RQ

   Description:

       Destroys the specified S-RQ.

       The RI MUST support this Verb if the Query RNIC Output Modifier
       indicates support for an S-RQ and MUST support all of the Input
       & Output Modifiers in this case, except where noted.

       For more information, see Section 6.3.3 - Destroying a Shared
       Receive Queue.

   Input Modifiers:

   *   RNIC handle.

   *   S-RQ handle.

   Output Modifiers:

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC handle.

       o   Invalid S-RQ handle.

       o   QPs still associated with the S-RQ.

9.2.5  Queue Pair

9.2.5.1  Create QP

   Description:

       Creates a QP for the specified RNIC.



   Hilland, et al.        Expires October 2003             [Page 170]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       A set of initial QP attributes must be specified by the
       Consumer. If any of the required initial attributes are illegal
       or missing, an error is returned and the Queue Pair is not
       created.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 6.1.1 - Creating a Queue Pair.

   Input Modifiers:

   *   RNIC handle.

   *   The QP attributes that must be specified at QP create time are:

       o   The CQ handle of the CQ to be associated with the Send
           Queue.

       o   The CQ handle of the CQ to be associated with the Receive
           Queue. (Note that this may be the same CQ that is associated
           with the Send Queue, or it may be a different CQ than the
           one associated with the Send Queue).

       o   The maximum number of outstanding Work Requests the Consumer
           expects to submit to the Send Queue.

       o   The maximum number of outstanding Work Requests the Consumer
           expects to submit to the Receive Queue. This value is
           ignored if the QP is associated with an S-RQ.

       o   If the QP's RQ will be associated with an S-RQ:

           +   S-RQ Handle.

           +   QP RQ Limit Indicator, as discussed in Section 6.3.8 -
               S-RQ Limit Checking. The QP RQ Limit detection is armed
               by the RI upon creation of the QP, if non-zero.

       o   Inbound RDMA Read enable.

       o   Inbound RDMA Write and inbound RDMA Read Response enable.

       o   Bind Memory Windows enable.

       o   The maximum number of scatter/gather elements the Consumer
           can specify in a Send Operation Type Work Request submitted
           to the Send Queue.






   Hilland, et al.        Expires October 2003             [Page 171]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   The maximum number of scatter/gather elements the Consumer
           can specify in a RDMA Write Work Request submitted to the
           Send Queue.

       o   The maximum number of scatter/gather elements the Consumer
           can specify in a Work Request submitted to the Receive
           Queue. This value is not returned if the QP is associated
           with an S-RQ.

       o   ORD (Requested) - The requested maximum number of
           outstanding Outgoing RDMA Read Request Messages the RNIC can
           initiate from the SQ.

       o   IRD (Requested) - The requested maximum number of
           outstanding Incoming RDMA Read Request Messages (e.g. IRRQ
           depth) the RNIC can handle for this QP.

       o   PD ID.

       o   Enable or disable the Use of the STag of zero and Fast-
           Register Non-Shared Memory Region Operations. This MUST only
           be allowed to be enabled for Privileged Mode Consumers.

   Output Modifiers:

   *   If the operation completed successfully:

       o   The QP Handle.

       o   The QP ID.

       o   The allocated number of outstanding Work Requests supported
           on the Send Queue. If an error is not returned, this is
           guaranteed to be greater than or equal to the number
           requested. (This may require the Consumer to increase the
           size of the CQ.)

       o   The allocated number of outstanding Work Requests supported
           on the Receive Queue. If an error is not returned, this is
           guaranteed to be greater than or equal to the number
           requested. (This may require the Consumer to increase the
           size of the CQ.) This value is not returned if the QP is
           associated with an S-RQ.

       o   The allocated number of scatter/gather elements that can be
           specified in Work Requests submitted to the Send Queue. If
           an error is not returned, this is guaranteed to be greater
           than or equal to the number requested.





   Hilland, et al.        Expires October 2003             [Page 172]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   The allocated number of Scatter/Gather Elements supported on
           RDMA Write Work Requests submitted to the Send Queue. If an
           error is not returned, this is guaranteed to be greater than
           or equal to the number requested.

       o   The allocated number of Scatter/Gather Elements that can be
           specified in Work Requests submitted to the Receive Queue.
           If an error is not returned, this is guaranteed to be
           greater than or equal to the number requested. This value is
           not returned if the QP is associated with an S-RQ.

       o   ORD (allocated) - The allocated number of outstanding RDMA
           Read Request Messages the RNIC can initiate from the SQ at
           the Data Sink. This number MUST be between zero and the
           number requested, inclusive. If the Consumer requested a
           non-zero number and the RI was unable to provision at least
           one then an Immediate Error MUST be returned.

       o   IRD (allocated) - The allocated number of incoming
           outstanding RDMA Read Request Messages (e.g. IRRQ depth) the
           RNICÆs QP can handle at the Data Source. If the Consumer
           requested a non-zero number and the RI was unable to
           provision at least one then an Immediate Error MUST be
           returned.

   *   Verb Results:

       o   Operation completed successfully.

       o   Insufficient resources to complete request.

       o   Invalid RNIC handle.

       o   Invalid CQ handle.

       o   Invalid S-RQ handle.

       o   The value requested for ORD exceeds RNIC capability.

       o   The value requested for IRD exceeds RNIC capability.

       o   Maximum number of Send Queue Work Requests requested exceeds
           RNIC capability.

       o   Maximum number of Receive Queue Work Requests requested
           exceeds RNIC capability

       o   Maximum number of scatter/gather elements per Send Queue
           Work Request requested exceeds RNIC capability.




   Hilland, et al.        Expires October 2003             [Page 173]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   Maximum number of scatter/gather elements per Receive Queue
           Work Request requested exceeds RNIC capability.

       o   Invalid Protection Domain.

       o   QP RQ Limit Out of Range.

9.2.5.2  Query QP

   Description:

       Returns the attribute list and current values for the specified
       QP.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 6.1.2 - Querying Queue Pair Attributes.

   Input Modifiers:

   *   RNIC Handle.

   *   QP Handle.

   Output Modifiers:

   *   The QP attributes, if the operation completed successfully. The
       list of attributes returned by the query are:

       o   Handle of the Completion Queue associated with the Send
           Queue.

       o   Handle of the Completion Queue associated with the Receive
           Queue.

       o   Handle of the S-RQ. This value is only returned if the QP is
           associated with an S-RQ.

       o   The allocated number of outstanding Work Requests supported
           on the Send Queue.

       o   The allocated number of outstanding Work Requests supported
           on the Receive Queue. This value is not returned if the QP
           is associated with an S-RQ.

       o   The actual number of Scatter/Gather Elements supported on
           Send Operation Type Work Requests submitted to the Send
           Queue.





   Hilland, et al.        Expires October 2003             [Page 174]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   The allocated number of Scatter/Gather Elements supported on
           RDMA Write Work Requests submitted to the Send Queue.

       o   The allocated number of Scatter/Gather Elements supported on
           Work Requests submitted to the Receive Queue. This value is
           not returned if the QP is associated with an S-RQ.

       o   ORD - The allocated number of outstanding RDMA Read Request
           Messages the RNIC can initiate from the SQ at the Data Sink.

       o   IRD - The allocated number of outstanding incoming RDMA Read
           Request Messages (e.g. IRRQ depth) the RNICÆs QP can handle
           at the Data Source.

       o   Current QP state.

       o   PD ID.

       o   QP ID.

       o   Use of the STag of zero and Fast-Register Non-Shared Memory
           Region Operations enabled.

       o   Inbound RDMA Read enable.

       o   Inbound RDMA Write and inbound RDMA Read Response enable.

       o   Bind Memory Windows enable.

       The following attributes are not defined unless the QP is in the
       Terminate or Error states.

       o   A buffer containing the Terminate Message that was received
           or sent (if possible).

       o   An indicator to state if the Terminate Message was generated
           locally or by the Associated QP.

       The following attributes are only defined if the QP is
       associated with a Shared Receive Queue.

       o   Current QP's RQ Limit.

       o   QP's RQ Limit armed indicator.

       The following attributes are only defined if the QP is not in
       the Idle state.

       o   LLP Stream Handle.




   Hilland, et al.        Expires October 2003             [Page 175]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC handle.

       o   Invalid QP handle.

9.2.5.3  Modify QP

   Description:

       Modifies the attributes for the specified QP then causes the QP
       to transition to the specified QP state. Only a subset of the QP
       attributes can be modified in each of the QP states.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 6.1.3 - Modifying Queue Pair Attributes.

   Input Modifiers:

   *   RNIC Handle.

   *   QP Handle.

   *   The QP attributes to modify and their new values. The QP
       attributes that can be modified after the QP has been created
       are:

       o   Next QP state. If the current state is specified, only the
           QP attributes will be modified.

       o   ORD - The requested number of outstanding RDMA Read Request
           Messages the RNIC can initiate from the SQ at the Data Sink.

       o   IRD - The requested number of incoming outstanding RDMA Read
           Request Messages (e.g. IRRQ depth) the RNICÆs QP can handle
           at the Data Source.

       o   The maximum number of outstanding Work Requests the Consumer
           expects to submit to the Send Queue (if changing is
           supported by the RNIC).

       o   The maximum number of outstanding Work Requests the Consumer
           expects to submit to the Receive Queue (if changing is
           supported by the RNIC). This value is not allowed if the QP
           is associated with an S-RQ.





   Hilland, et al.        Expires October 2003             [Page 176]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       The following attributes are only defined if the QP is
       associated with a Shared Receive Queue.

       o   QP's RQ Limit, as described in Section 6.3.8 - S-RQ Limit
           Checking.

       o   Re-arm the QP's RQ Limit, as described in Section 6.3.8 - S-
           RQ Limit Checking. The RI MUST allow an already armed S-RQ
           limit to be armed.

       Valid only when moving from Idle to RTS.

       o   LLP Stream Handle

       o   Stream Message Buffer.

   Output Modifiers:

   *   If the operation completed successfully:

       o   The allocated number of outstanding Work Requests supported
           on the Send Queue.

       o   The allocated number of outstanding Work Requests supported
           on the Receive Queue. This value is not returned if the QP
           is associated with an S-RQ.

       o   ORD - The allocated number of outstanding RDMA Read Request
           Messages the RNIC can initiate from the SQ at the Data Sink.
           This number MUST be between zero and the number requested,
           inclusive. If the Consumer requested a non-zero number and
           was unable to provision at least one then an Immediate Error
           will be returned.

       o   IRD - The allocated number of incoming outstanding RDMA Read
           Request Messages (e.g. the IRRQ depth) the RNICÆs QP can
           handle at the Data Source. If the Consumer requested a non-
           zero number and was unable to provision at least one then an
           Immediate Error will be returned.

   *   Verb Results:

       o   Operation completed successfully.

       o   Insufficient resources to complete request.

       o   Invalid RNIC handle.

       o   Invalid QP handle.




   Hilland, et al.        Expires October 2003             [Page 177]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   Cannot change QP attribute.

       o   Invalid QP state change requested.

       o   Maximum number of Send Queue Work Requests requested exceeds
           RNIC capability.

       o   Maximum number of Receive Queue Work Requests requested
           exceeds RNIC capability.

       o   The value requested for ORD exceeds RNIC capability.

       o   The value requested for IRD exceeds RNIC capability.

       o   An Attempt to shrink the size of the queue failed because
           too many elements were still present.

       o   Invalid LLP Stream Handle.

       o   Invalid Modifier.

       o   RI still flushing WQEs.

       o   RQ Limit Out of Range.

9.2.5.4  Destroy QP

   Description:

       Destroys the specified QP.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. The QP cannot be
       destroyed if any Memory Windows are still Bound to the QP.

       For more information, see Section 6.1.4 - Destroying a Queue
       Pair.

   Input Modifiers:

   *   RNIC handle.

   *   QP handle.

   Output Modifiers:

   *   Verb Results:

       o   Operation completed successfully.




   Hilland, et al.        Expires October 2003             [Page 178]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   Invalid RNIC handle.

       o   Invalid QP handle.

       o   Memory Windows still Bound to QP.

9.2.6  Memory Management

   Memory Management Verbs are used to manage Memory Regions and Memory
   Windows. The following table describes what each of the Memory
   Management Verbs manage and where the Verb appears to performed:

                    Verb                 Used to manage  Performed by
                                           MR vs. MW     RI vs. RNIC

    Allocate Non-Shared Memory Region    MR              RI
    STag

    Register Non-Shared Memory Region    MR              RI
    (RI-Register)

    Reregister Non-Shared Memory Region  MR              RI
    (RI-Reregister)

    Register Shared Memory Region        MR              RI

    Fast-Register Non-Shared Memory      MR              RNIC
    Region (PostSQ)

    Query Memory Region                  MR              RI

    Invalidate Local STag (PostSQ)       MR or MW        RNIC

    Deallocate STag                      MR or MW        RI

    Allocate Memory Window               MW              RI

    Query Memory Window                  MW              RI

    Bind Memory Window (PostSQ)          MW              RNIC

                    Figure 26 - Memory Management Verbs

9.2.6.1  Allocate Non-Shared Memory Region STag

   Description:

       Allocates memory registration resources on the RNIC.



   Hilland, et al.        Expires October 2003             [Page 179]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 7.3.2.1 - Allocate Non-Shared Memory Region STag.

   Input Modifiers:

   *   RNIC Handle.

   *   Requested Physical Buffer List size to be allocated.

   *   PD ID.

   *   Remote Access Flag. If set, Local and Remote Access is enabled.
       Otherwise only Local access is enabled.

   Output Modifiers:

   *   If the operation completed successfully:

       o   STag Index - used for local and, if specified by the input
           modifiers, remote access.

       o   The actual number of Physical Buffer List Entries in the
           allocated Physical Buffer List. Note that this MAY be
           greater than the number requested.

   *   Verb Results:

       o   Operation completed successfully.

       o   Insufficient resources to complete request.

       o   Invalid RNIC handle.

       o   Invalid PD ID.

9.2.6.2  Register Non-Shared Memory Region (RI-Register)

   Description:

       Registers a Non-Shared Memory Region for use by an RNIC.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 7.3.2.2 - RI-Register Non-Shared Memory Region.

   Input Modifiers:

   *   RNIC Handle.




   Hilland, et al.        Expires October 2003             [Page 180]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   Physical Buffer Entry size - The size, in bytes, of each
       Physical Buffer in the list. Note: If the Physical Buffer List
       references a Page List, the size MUST be a power of two. If the
       Physical Buffer List references a Block List, the size MAY have
       a byte alignment.

   *   Address List - A list of addresses that point to the Physical
       Buffers referenced by the Physical Buffer List. All Physical
       Buffers in the list have the same size.

   *   Address List Length - the number of entries in the Address list.

   *   First Byte Offset (FBO) - Offset to start of Non-Shared Memory
       Region on first Physical Buffer.

   *   Length - Total length of the Non-Shared Memory Region (can be of
       arbitrary byte-aligned length).

   *   Addressing type. The Addressing type MUST be one of the
       following:

       o   VA Based TO

       o   Zero Based TO

   *   The following input modifier is only valid if the Addressing
       type is VA Based TO:

       o   Virtual Address - The VA address of the first byte in the
           Non-Shared Memory Region.

   *   PD ID.

   *   STag Key.

   *   Remote Access Flag.

   *   Access Control - The following MAY be selected in any
       combination except as noted:

       o   Enable Local Write Access.

       o   Enable Remote Write Access. Remote Write Access requires
           Local Write Access to be enabled.

       o   Enable Local Read Access.

       o   Enable Remote Read Access. Remote Read Access requires Local
           Read Access to be enabled.




   Hilland, et al.        Expires October 2003             [Page 181]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   Enable Memory Window Binding.

   Output Modifiers:

   *   If the operation completed successfully:

       o   STag Index - used for local and, if specified by the input
           modifiers, remote access. Note: the RNIC associates the STag
           Key passed in as an input modifier to STag associated with
           the registered Non-Shared Memory Region.

       o   The actual number of Physical Buffer List Entries in the
           allocated Physical Buffer List. Note that this MAY be
           greater than the number requested.

   *   Verb Results:

       o   Operation completed successfully.

       o   Insufficient resources to complete request.

       o   Invalid RNIC handle.

       o   Invalid PD ID.

       o   Invalid Virtual Address.

       o   Invalid length.

       o   Invalid First Byte Offset.

       o   Invalid Access Rights requested.

       o   Invalid Physical Buffer List entry.

       o   Invalid Physical Buffer size.

9.2.6.3  Query Memory Region

   Description:

       Retrieves information about a specific Memory Region.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 7.7 - Querying Memory Regions.

   Input Modifiers:

   *   RNIC Handle.



   Hilland, et al.        Expires October 2003             [Page 182]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   STag Index - as originally returned from an Allocate Non-Shared
       Memory Region STag, RI-Register Non-Shared Memory Region, RI-
       Reregister Non-Shared Memory Region or Register Shared Memory
       Region Type Verb.

   Output Modifiers:

   *   If the operation completed successfully:

       o   STag Key - Current STag Key associated with the Memory
           Region, if it is in the Valid state.

       o   Remote Access Flag.

       o   PD ID.

       o   STag State: Valid or Invalid.

       o   STag Type: Shared or Non-Shared.

       o   The actual number of Physical Buffer List Entries in the
           allocated Physical Buffer List. Note that this MAY be
           greater than the number requested.

       o   Access Control settings for the registered Region. The
           following MAY be set in any combination except as noted:

           +   Local Write Access Enabled.

           +   Remote Write Access Enabled. Remote Write Access
               requires Local Write Access to be enabled.

           +   Local Read Access Enabled.

           +   Remote Read Access Enabled. Remote Read Access requires
               Local Read Access to be enabled.

           +   Memory Window Binding Enabled.

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC handle.

       o   Invalid STag Index.

9.2.6.4  Deallocate STag

   Description:



   Hilland, et al.        Expires October 2003             [Page 183]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       Removes an STag created through an Allocate Non-Shared Memory
       Region STag, RI-Register Non-Shared Memory Region, RI-Reregister
       Non-Shared Memory Region, Register Shared Memory Region or
       Allocate Memory Window from the RNIC.

       Work Requests or Remote Operation requests that are in-process
       and actively referencing memory locations associated with the
       STag being deallocated must fail with a protection error.

       If the STag references a Memory Region which has Memory Windows
       Bound to it, an immediate Error MUST be returned and the Memory
       Region must not be destroyed or modified.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 7.9 - Deallocation of STag associated with a Memory
       Region and Section 7.10.4 - Invalidating or De-allocating Memory
       Windows.

   Input Modifiers:

   *   RNIC Handle.

   *   STag Index - as originally returned from an Allocate Non-Shared
       Memory Region STag, Allocate Memory Window, or RI-Register Non-
       Shared Memory Region, RI-Reregister Non-Shared Memory Region or
       Register Shared Memory Region Verb.

   Output Modifiers:

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC handle.

       o   Invalid STag Index.

       o   One or more Memory Windows is still Bound to the Memory
           Region. Applies only if the STag is associated with a Memory
           Region.

9.2.6.5  Reregister Non-Shared Memory Region (RI-Reregister)

   Description:

       Modifies the attributes of an existing Non-Shared Memory Region.

       The STag output modifier from this Verb must be used in place of
       any previously issued for this Non-Shared Memory Region.



   Hilland, et al.        Expires October 2003             [Page 184]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       If the STag references a Non-Shared Memory Region which has
       Memory Windows Bound to it, an immediate Error MUST be returned
       and the Non-Shared Memory Region must not be destroyed or
       modified.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 7.3.2.3 - RI-Reregister Non-Shared Memory Region.

   Input Modifiers:

   *   RNIC Handle.

   *   Physical Buffer Entry size - The size, in bytes, of each
       Physical Buffer Entry in the list. Note: If the Physical Buffer
       List references a Page-List, the size MUST be a power of two. If
       the Physical Buffer List references a Block-List, the size MAY
       have a byte alignment.

   *   Address List - A list of addresses that point to the Physical
       Buffers referenced by the Physical Buffer List. All Physical
       Buffers in the list MUST have the same size.

   *   Address List Length - the number of entries in the Address list.

   *   First Byte Offset (FBO) - Offset to start of Non-Shared Memory
       Region on first Physical Buffer.

   *   Length - Total length of Non-Shared Memory Region (can be of
       arbitrary byte-aligned length).

   *   Addressing type. The addressing type MUST be one of the
       following:

       o   VA Based TO

       o   Zero Based TO

   *   The following input modifier is only valid if the Addressing
       type is VA Based TO:

       o   Virtual Address - The VA address of the first byte in the
           Non-Shared Memory Region.

   *   PD ID.

   *   STag Index.

   *   STag Key (not the existing STag Key, but the new STag Key).




   Hilland, et al.        Expires October 2003             [Page 185]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   Remote Access Flag.

   *   Access Control - The following MAY be selected in any
       combination except as noted:

       o   Enable Local Write Access.

       o   Enable Remote Write Access. Remote Write Access requires
           Local Write Access to be enabled.

       o   Enable Local Read Access.

       o   Enable Remote Read Access. Remote Read Access requires Local
           Read Access to be enabled.

       o   Enable Memory Window Binding.

   Output Modifiers:

   *   If the operation completed successfully:

       o   STag Index - used for local and, if specified by the input
           modifiers, remote access. Note: the RNIC associates the STag
           Key passed in as an input modifier to STag associated with
           the registered Non-Shared Memory Region. If the output STag
           index differs from the input STag index, the old STag index
           was Deallocated.

       o   The actual number of Physical Buffer List Entries in the
           allocated Physical Buffer List. Note that this MAY be
           greater than the number requested.

   *   Verb Results:

       o   Operation completed successfully.

       o   Insufficient resources to complete request.

       o   Invalid RNIC handle.

       o   Invalid STag Index.

       o   Invalid Virtual Address.

       o   Invalid Length.

       o   Invalid PD ID.

       o   Invalid First Byte Offset.




   Hilland, et al.        Expires October 2003             [Page 186]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   Invalid Access Rights request.

       o   One or more Memory Windows is still Bound to the Region.

       o   Invalid Physical Buffer List entry.

       o   Invalid Physical Buffer size.

9.2.6.6  Register Shared Memory Region

   Description:

       Registers a new Shared Memory Region which shares RNIC mapping
       resources with a previously registered Memory Region, thus
       returning a new STag. Note that other than the change of the
       original Memory Region to a Shared Memory Region, the original
       Memory Region remains unaffected by this operation.

       The Base TO,VA (if the input STag Index references a VA Based
       TO), PD ID, and Access Rights specified for the new Memory
       Region need not be the same as those of the existing Memory
       Region. The lengths are by definition the same.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 7.4.3 - Multiple Registrations of Memory Regions.

   Input Modifiers:

   *   RNIC Handle.

   *   STag Index of the existing Memory Region. If the existing Memory
       Region is Non-Shared, successful completion of this verb will
       convert the existing Non-Shared Memory Region to a Shared Memory
       Region.

   *   Addressing type. The addressing type MUST be one of the
       following:

       o   VA Based TO

       o   Zero Based TO

   *   The following modifier is only valid if the Addressing type of
       the existing region is VA Based TO:

       o   Virtual Address - The VA address of the first byte in the
           Memory Region.

   *   PD ID.



   Hilland, et al.        Expires October 2003             [Page 187]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   STag Key of the new STag.

   *   Remote Access Flag.

   *   Access Control - The following MAY be selected in any
       combination except as noted:

       o   Enable Local Write Access.

       o   Enable Remote Write Access. Remote Write Access requires
           Local Write Access to be enabled.

       o   Enable Local Read Access.

       o   Enable Remote Read Access. Remote Read Access requires Local
           Read Access to be enabled.

       o   Enable Memory Window Binding.

   Output Modifiers:

   *   If the operation completed successfully:

       o   STag Index - used for local and, if specified by the input
           modifiers, remote access. Note: the RNIC associates the STag
           Key passed in as an input modifier to STag associated with
           the registered Shared Memory Region.

   *   Verb Results:

       o   Operation completed successfully.

       o   Insufficient resources to complete request.

       o   Invalid RNIC handle.

       o   Invalid STag Index.

       o   Invalid Virtual Address.

       o   Invalid PD ID.

       o   Invalid Access Rights requested.

9.2.6.7  Allocate Memory Window

   Description:






   Hilland, et al.        Expires October 2003             [Page 188]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       This Verb allocates a memory window and associates it with a
       Protection Domain. It is not inherently associated with any
       Memory Region when allocated.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 7.10.1 - Allocating Memory Windows.

   Input Modifiers:

   *   RNIC Handle.

   *   PD ID.

   Output Modifiers:

   *   If the operation completed successfully:

       o   STag Index - an unbound STag for use in specifying the
           Window when invoking a Bind Work Request through the Post
           Send Verb.

   *   Verb Results:

       o   Operation completed successfully.

       o   Insufficient resources to complete request.

       o   Invalid RNIC handle.

       o   Invalid PD ID.

9.2.6.8  Query Memory Window

   Description:

       This Verb returns the attributes associated with the specified
       memory window.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 7.10.3 - Memory Windows.

   Input Modifiers:

   *   RNIC Handle.

   *   STag Index - the current STag associated with the Memory Window.

   Output Modifiers:



   Hilland, et al.        Expires October 2003             [Page 189]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   If the operation completed successfully:

       o   STag Key - current value of the STag Key, if the STag is in
           the Valid state.

       o   STag State: Valid or Invalid.

       o   PD ID.

       o   Access Rights. The following may be set in any combination
           except as noted.

           +   Remote Write Access Enabled. If set Remote Write Access
               is enabled.

           +   Remote Read Access Enabled. If set Remote Read Access is
               enabled.

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC handle.

       o   Invalid STag Index.

9.3  Work Request Processing

9.3.1  QP Operations

9.3.1.1  PostSQ

   Description:

       Builds a WQE on the Send Queue of the specified QP for each
       entry in the Work Request List submitted by the Consumer. This
       WQE is added to the end of the Send Queue and the RNIC is
       notified that a new WQE is ready to be processed.

       Note that not all Input Modifiers are valid for all operations.
       If Input Modifiers are specified that are not valid for a
       particular operation, they are ignored.

       Following the Verbs is a Work Request table which contains a
       List of the Operation Types and the Input Modifiers which are
       required for each of those Operation Types.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 8.2.1 - Submitting Work Request to a Work Queue.



   Hilland, et al.        Expires October 2003             [Page 190]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Input Modifiers:

   *   RNIC Handle

   *   QP Handle.

   *   A list of Work Requests. Each Work Request MUST contain the
       following information:

       o   A user defined 64-bit Work Request ID

       o   Operation type. The operation type MUST be one of the
           following:

           +   Send

           +   Send with Solicited Event

           +   Send with Invalidate

           +   Send with Solicited Event & Invalidate

           +   RDMA Write

           +   RDMA Read

           +   RDMA Read with Invalidate Local STag

           +   Bind Memory Window

           +   Fast-Register Non-Shared Memory Region

           +   Invalidate Local STag

       o   Completion Notification Type: Signaled or Unsignaled.

       o   The following list of modifiers are only valid for Send
           Operation Types and RDMA Write WRs to represent the Local
           Buffer:

           +   Scatter/Gather List. The Scatter/Gather List can contain
               zero or more Scatter/Gather Elements. This list is
               specified only for Send and RDMA type operations.

           +   Number of Scatter/Gather Elements.

           +   Note that the length is determined by adding up the
               Length field in the SGEs of the SGL.

           +   Read Fence indicator.



   Hilland, et al.        Expires October 2003             [Page 191]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   The following list of modifiers are only valid for RDMA Read
           Type operations to represent the Local Buffer:

           +   Local Address. This is a contiguous buffer represented
               by a TO, an STag, and a Length to be read.

       o   The following list of modifiers are only valid for RDMA
           Write or RDMA Read Type WRs to represent the Remote Buffer:

           +   Remote Address. This is a contiguous buffer represented
               by a TO and an STag.

       o   The following modifier is only valid for the Send with
           Invalidate and Send with Solicited Event & Invalidate
           operations:

           +   Remote STag. This is the STag to be Invalidated at the
               Remote Peer.

       o   The following list of modifiers are only valid for Bind
           Memory Window operations:

           +   STag Index for the Memory Window.

           +   STag Key for the Memory Window.

           +   STag for the Memory Region that the Memory Windows is to
               be associated with. This parameter includes both the
               STag Index and STag Key.

           +   Length or range to be Bound in number of octets.

           +   Addressing type. The addressing type MUST be one of the
               following:

               *   VA Based TO

               *   Zero Based TO

           +   Virtual Address - The VA address of the first byte into
               the Memory Region. This may be different than the
               starting address of the Memory Region.

           +   Access Control - either or both of the following must be
               selected:

               *   Enable Remote Write Access. Requires the Memory
                   Region to have Local Write Access.





   Hilland, et al.        Expires October 2003             [Page 192]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

               *   Enable Remote Read Access. Requires the Memory
                   Region to have Local Read Access.

       o   The following list of modifiers are only valid for Fast-
           Register Non-Shared Memory Region operations:

           +   Physical Buffer Entry size - The size, in bytes, of each
               Physical Buffer in the list. Note: If the Physical
               Buffer List references a Page-List, the size MUST be a
               power of two. If the Physical Buffer List references a
               Block-List, the size MUST be an RNIC supported size (see
               Section 9.2.1.2 - Query RNIC).

           +   Address List - A list of addresses that point to the
               Physical Buffers referenced by the Physical Buffer List.
               All Physical Buffers in the list MUST have the same
               size.

           +   Address List Length - the number of entries in the
               Address list.

           +   First Byte Offset (FBO) - Offset to start of Non-Shared
               Memory Region on first Physical Buffer.

           +   Length - Total length of Non-Shared Memory Region (can
               be any value supported by the RNIC).

           +   Addressing type. The addressing type MUST be one of the
               following:

               *   VA Based TO

               *   Zero Based TO

           +   The following modifier is only valid if the Addressing
               type is VA Based TO:

               *   Virtual Address - The VA address of the first byte
                   in the Non-Shared Memory Region

           +   STag Index.

           +   STag Key.

           +   Access Control - The following may be selected in any
               combination except as noted:

               *   Enable Local Write Access.





   Hilland, et al.        Expires October 2003             [Page 193]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

               *   Enable Remote Write Access. Remote Write Access
                   requires Local Write Access to be enabled. The STag
                   Index MUST have the Remote Access Flag enabled.

               *   Enable Local Read Access.

               *   Enable Remote Read Access. Remote Read Access
                   requires Local Read Access to be enabled. The STag
                   Index MUST have the Remote Access Flag enabled.

               *   Enable Memory Window Binding.

       o   The following list of modifiers are only valid for
           Invalidate Local STag operations:

           +   STag to be the target of the Invalidate operation.

           +   Local Fence indicator.

   Below, in Figure 27, is a matrix of the Input Modifiers for PostSQ
   and the Operation Types. The intersection of the matrix indicates
   that the Input Modifier is required for that Operation Type by
   specifying "Yes".

  Opcode-> Send Send Send Send RDMA  RDMA RDMA Bind Fast- Inv.
  Input         w/   w/   w/   Write Read Read MW   Reg.  Local
  Modifier      SE   Inv. SE &             w/       NS MR STag
                          Inv.            Inv.

  WR ID    Yes  Yes  Yes  Yes  Yes   Yes  Yes  Yes  Yes   Yes

  Compltn. Yes  Yes  Yes  Yes  Yes   Yes  Yes  Yes  Yes   Yes
  Notif.
  Type

  SGL      Yes  Yes  Yes  Yes  Yes

  SGE No.  Yes  Yes  Yes  Yes  Yes

  Read     Yes  Yes  Yes  Yes  Yes
  Fence

  Local                                                   Yes
  Fence

  Local                              Yes  Yes
  Address





   Hilland, et al.        Expires October 2003             [Page 194]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

  Opcode-> Send Send Send Send RDMA  RDMA RDMA Bind Fast- Inv.
  Input         w/   w/   w/   Write Read Read MW   Reg.  Local
  Modifier      SE   Inv. SE &            w/        NS MR STag
                          Inv.            Inv.

  Remote                       Yes   Yes  Yes
  Address

  Remote             Yes  Yes
  STag

  MW STag                                      Yes
  Key

  MW STag                                      Yes
  Index

  MW's                                         Yes
  MR STag

  MW                                           Yes
  Length

  Addr                                         Yes  Yes
  Type

  VA, if                                       Yes  Yes
  VA Based
  TO

  Acs                                               Yes
  Ctrl:
  Local Rd

  Acs                                          Yes  Yes
  Ctrl:
  Remote
  Rd

  Acs                                               Yes
  Ctrl:
  Local Wt

  Acs                                          Yes  Yes
  Ctrl:
  Remote
  Wt




   Hilland, et al.        Expires October 2003             [Page 195]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

  Opcode-> Send Send Send Send RDMA  RDMA RDMA Bind Fast- Inv.
  Input         w/   w/   w/   Write Read Read MW   Reg.  Local
  Modifier      SE   Inv. SE &            w/        NS MR STag
                          Inv.            Inv.

  Acs                                               Yes
  Ctrl:
  Bind
  Enable

  PBLE                                              Yes
  Size

  PBL                                               Yes

  FBO                                               Yes

  STag                                              Yes   Yes
  Index

  STag Key                                          Yes   Yes


                Figure 27 - PostSQ Input Modifier Validity

   Output Modifiers:

   *   Number of WRs posted.

   *   Verb Results:

       o   Operation completed successfully

       o   Invalid RNIC Handle

       o   Invalid QP Handle

       o   Too many Work Requests posted.

       o   Invalid operation type.

       o   Invalid QP state.

       o   Invalid Scatter/Gather list format.

       o   Invalid Scatter/Gather list length.

       o   Invalid Modifier.




   Hilland, et al.        Expires October 2003             [Page 196]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

9.3.1.2  PostRQ

   Description:

       Builds a WQE on the Receive Queue of the specified QP for each
       entry in the Work Request List submitted by the Consumer. This
       WQE is added to the end of the Receive Queue and the RNIC is
       notified that a new WQE is ready to be processed.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 8.2.1 - Submitting Work Request to a Work Queue.

   Input Modifiers:

   *   RNIC Handle.

   *   QP Handle, for QP's not associated with an S-RQ.

   *   S-RQ Handle, for QP's associated with an S-RQ.

   *   A list of Work Requests. Each Work Request MUST contain the
       following information.

       o   A user defined 64-bit Work Request ID.

       o   Scatter/Gather List. The scatter/gather list can contain one
           or more Data Segments.

       o   Number of Scatter/Gather List elements.

   Output Modifiers:

   *   Number of WRs posted.

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC handle.

       o   Invalid QP handle.

       o   Invalid S-RQ handle.

       o   Too many Work Requests posted.

       o   Invalid QP state.

       o   Invalid Scatter/Gather list format.



   Hilland, et al.        Expires October 2003             [Page 197]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       o   Invalid Scatter/Gather list length.

       o   Invalid Modifier.

       o   RQ Associated with S-RQ.

9.3.2  CQ Operations

9.3.2.1  Poll for Completion (Poll CQ)

   Description:

       Polls the specified CQ for a Work Completion.

       If a CQE is present, the CQE at the head of the CQ MUST be
       returned to the Consumer as a Work Completion. Note that the
       resources used are expected to be directly accessible by a Non-
       Privileged Mode Consumer.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 8.2.4 - Returning Completed Work Requests.

   Input Modifiers:

   *   RNIC Handle

   *   CQ Handle.

   Output Modifiers:

   *   The Work Completion. If an entry is present on the CQ and if the
       operation completed successfully, this contains information
       relating to a completed Work Request. If the status of the
       operation that generates the Work Completion is anything other
       than success, the contents of the Work Completion are undefined
       except as noted below. The contents of a Work Completion are:

       o   The 64-bit Work Request ID set by the Consumer in the asso-
           ciated Work Request. This is always valid, regardless of the
           status of the operation.

       o   The operation type specified in the completed Work Request.
           The valid operation types are:

           +   Send (for WRs posted to the Send Queue)

           +   Send with Solicited Event (for WRs posted to the Send
               Queue)




   Hilland, et al.        Expires October 2003             [Page 198]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

           +   Send with Invalidate (for WRs posted to the Send Queue)

           +   Send with Solicited Event & Invalidate (for WRs posted
               to the Send Queue)

           +   RDMA Write (for WRs posted to the Send Queue)

           +   RDMA Read (for WRs posted to the Send Queue)

           +   RDMA Read with Invalidate Local STag (for WRs posted to
               the Send Queue)

           +   Memory Window Bind (for WRs posted to the Send Queue)

           +   Fast-Register Non-Shared Memory Region (for WRs posted
               to the Send Queue)

           +   Invalidate Local STag (for WRs posted to the Send Queue)

           +   Receive (for WRs posted to the Receive Queue)

       o   The number of bytes transferred. This is only valid if the
           operation type was a Receive.

       o   The Completion Status of the operation. This modifier MUST
           be as specified in Section 9.5.2 - Completion Status Codes.

       o   STag Invalidated Indicator. This indicates that the incoming
           Untagged Message destined for the RQ was a Send with
           Invalidate or Send with Solicited Event & Invalidate, and
           thus the STag Invalidated field is valid.

       o   STag Invalidated. This contains the STag which was
           Invalidated. This is only valid when the Invalidated STag
           Indicator is set.

       o   QP ID. This is the QP ID of the QP where the WR which
           generated this completion was posted.

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC handle.

       o   Invalid CQ handle.

       o   CQ empty.





   Hilland, et al.        Expires October 2003             [Page 199]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

9.3.2.2  Request Completion Notification

   Description:

       Requests the CQ event handler be called when the next CQE of the
       specified type is added to the specified CQ.

       A CQ event handler must be specified prior to calling this
       routine (see Section 9.4.1 - Set Completion Event Handler). If
       the CQ event handler has not been registered when the event is
       generated, the handler will not be called.

       Once the handler routine has been invoked, the Consumer must
       call Request Completion Notification again to be notified when a
       new entry is added to that CQ.

       It is the responsibility of the Consumer to call the Poll for
       Completion Verb to retrieve a Work Completion after the handler
       is called.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 8.2.5 - Asynchronous Completion Notification.

   Input Modifiers:

   *   RNIC Handle.

   *   CQ Handle.

   *   Completion notification type. This MUST be either the next
       completion event or the next solicited completion event.

   Output Modifiers:

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC handle.

       o   Invalid CQ handle.

9.4  Event Handling

9.4.1  Set Completion Event Handler

   Description:





   Hilland, et al.        Expires October 2003             [Page 200]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

       A RNIC MUST support one CQ Event Handler, and MAY support
       additional Completion Event Handlers. Each Completion Event
       Handler address is maintained by the RI and delineated by an
       opaque handle called a Completion Event Handler Identifier. The
       consumer uses the Set Completion Event Handler to register
       individual Completion Event Handlers and obtain a unique
       Completion Event Handler Identifier. The Completion Event
       Handler Identifier is used in Create CQ to associate a CQ with a
       specific Completion Event Handler.

       This call does not automatically request a notification on a
       completion event. The Request Completion Notification Verb must
       be called in order to request notification.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 8.2.5 - Asynchronous Completion Notification.

   Input Modifiers:

   *   RNIC Handle

   *   Completion Event Handler Address. If set to zero, then the Set
       Completion Handler Verb is being used to clear the associated
       Completion Event Handler address identified by the Completion
       Event Handler Identifier. The Completion Event Handler will be
       invoked when an appropriate Completion occurs with the following
       input parameters passed in to it:

       o   RNIC Handle.

       o   CQ Handle.

   *   Completion Event Handler Identifier - An opaque handle used to
       identify a Completion Event Handler address.

       o   If set to zero, the Set Completion Event Handler verb is
           being used to register a new Completion Event Handler
           address and the verb will return a new Completion Event
           Handler Identifier.

       o   If set to non-zero, then the Set Completion Event Handler is
           being used:

           +   to clear the associated Completion Event Handler address
               for the specified Completion Event Handler Identifier,
               if the Completion Event Handler address is zero;

           +   to modify the associated Completion Event Handler
               address for the specified Completion Event Handler



   Hilland, et al.        Expires October 2003             [Page 201]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

               Identifier, if the Completion Event Handler address is
               non-zero.

   Output Modifiers:

   *   Completion Event Handler Identifier - Only returned if the Set
       Completion Event Handler verb is being used to register a new
       Completion Event Handler address.

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC Handle.

       o   Invalid Completion Event Handler Identifier.

       o   Insufficient Resources.

9.4.2  Set Asynchronous Event Handler

   Description:

       Registers the asynchronous event handler. Only one asynchronous
       event handler can be registered per RNIC. Additional calls to
       this Verb will overwrite the handler routine to be called.
       Additional calls will not generate an additional handler
       routine. If the new handler address is zero, there will be no
       Asynchronous Event Handler associated with the RNIC.

       The RI MUST support this Verb and MUST support all of the Input
       & Output Modifiers, except where noted. For more information,
       see Section 8.3.3 - Asynchronous Errors.

   Input Modifiers:

   *   RNIC Handle

   *   Asynchronous Event Handler Address. This routine will be invoked
       with the following input parameters passed in:

       o   RNIC Handle.

       o   Event Record. This contains information which indicates the
           resource type and identifier as well as which event
           occurred:

           +   Resource Indicator. This indicates the type of resource
               to which the Resource Identifier refers. This must be
               one of the following values:



   Hilland, et al.        Expires October 2003             [Page 202]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

               *   QP

               *   CQ

               *   RNIC

               *   S-RQ

           +   Resource Identifier. This value is the QP Handle, CQ
               Handle, S-RQ Handle or RNIC Handle for the Asynchronous
               Event.

           +   Event Identifier. This indicates the event which caused
               the Asynchronous Event to be generated. The possible
               list of Event Identifiers can be found in Section 9.5.3
               - Asynchronous Event Identifiers.

   Output Modifiers:

   *   Verb Results:

       o   Operation completed successfully.

       o   Invalid RNIC Handle.

9.5  Result Types

   The following section is a summary of Verb results detailed in
   Sections 9.2 - 9.4)

9.5.1  Immediate Status Codes

Operation completed successfully - The Verb was      All Verbs
executed successfully.


















   Hilland, et al.        Expires October 2003             [Page 203]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

9.5.1.1  RNIC Management Verb Status

Insufficient resources to complete request - An      Open RNIC, Query
error was detected due to insufficient resources.    RNIC

Invalid Modifier - One of the parameters were        Open RNIC
invalid.

Block List mode not supported - The RNIC does not    Open RNIC
support Block List mode and Block List mode was
requested.

RNIC in use - The RNIC was already in use.           Open RNIC

Invalid RNIC handle - An invalid RNIC handle was     Query RNIC, Close
specified.                                           RNIC

                  Figure 28 - RNIC Management Verb Status

9.5.1.2  PD Management Verb Status

Insufficient resources to complete request - An      Allocate PD
error was detected due to insufficient resources.

Invalid RNIC handle - An invalid RNIC handle was     Allocate PD,
specified.                                           Deallocate PD

Invalid PD ID - An invalid PD was specified.         Deallocate PD

Protection Domain is in use - The PD was currently   Deallocate PD
in use by a QP, Memory Region, or Memory Window.

                   Figure 29 - PD Management Verb Status


















   Hilland, et al.        Expires October 2003             [Page 204]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

9.5.1.3  CQ Management Verb Status

Insufficient resources to complete request - An      Create CQ, Modify
error was detected due to insufficient resources.    CQ

Number of CQE requested exceeds RNIC capability -    Create CQ, Modify
Too many CQ entries for this RNIC were requested.    CQ

An Attempt to shrink the size of the queue failed    Modify CQ
because too many elements were still present.

Invalid RNIC handle - An invalid RNIC handle was     Create CQ, Query
specified.                                           CQ, Modify CQ,
                                                     Destroy CQ, Poll
                                                     CQ

Invalid CQ handle- An invalid CQ handle was          Query CQ, Modify
specified.                                           CQ, Destroy CQ,
                                                     Poll CQ

CQ In Use - One or more QPs is still tied to the CQ. Destroy CQ

CQ empty - There were no Work Completions available  Poll CQ
to be retrieved.

Invalid Completion Event Handler Identifier - An     Create CQ
invalid identifier was specified.

                   Figure 30 - CQ Management Verb Status

9.5.1.4  S-RQ Management Verb Status

Insufficient resources to complete request - An      Create S-RQ,
error was detected due to insufficient resources.    Modify S-RQ

Invalid RNIC handle - An invalid RNIC handle was     Create S-RQ,
specified.                                           Query S-RQ,
                                                     Modify S-RQ,
                                                     Destroy S-RQ

Invalid PD ID - An invalid PD was specified.         Create S-RQ

Maximum number of Work Requests requested exceeds    Create S-RQ,
RNIC capability.                                     Modify S-RQ







   Hilland, et al.        Expires October 2003             [Page 205]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

Maximum number of scatter/gather elements per        Create S-RQ
Receive Queue Work Request requested exceeds RNIC
capability.

S-RQ Limit out of range                              Create S-RQ,
                                                     Modify S-RQ

Invalid S-RQ handle                                  Query S-RQ,
                                                     Modify S-RQ,
                                                     Modify S-RQ

An attempt to shrink the size of the queue failed    Modify S-RQ
because too many elements were still present

QPs still associated with the S-RQ                   Modify S-RQ

Invalid Input Modifer                                Modify S-RQ

                  Figure 31 - S-RQ Management Verb Status

9.5.1.5  QP Management Verb Status

Insufficient resources to complete request - An    Create QP, Modify QP
error was detected due to insufficient resources.

Invalid RNIC handle - An invalid RNIC handle was   Create QP, Query QP,
specified.                                         Modify QP, Destroy
                                                   QP

Invalid CQ handle - An invalid CQ handle was       Create QP
specified.

Value requested for ORD exceeds RNIC capability.   Create QP, Modify QP

Value requested for IRD exceeds RNIC capability.   Create QP, Modify QP

Maximum number of Work Requests requested exceeds  Create QP, Modify QP
RNIC capability.

Maximum number of scatter/gather elements          Create QP, Modify QP
requested per Work Request exceeds RNIC
capability.

Invalid PD ID - The PD ID provided was not valid   Create QP

Invalid QP ID - An invalid QP handle was           Query QP, Modify QP,
specified.                                         Destroy QP




   Hilland, et al.        Expires October 2003             [Page 206]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

Cannot change QP attribute - An attempt was made   Modify QP
to modify an attribute which is not allowed by the
RNIC (for example, number of WQEs)

An Attempt to shrink the size of the queue failed  Modify QP
because too many elements were still present.

Invalid state - An invalid QP state was specified. Modify QP

Invalid LLP Stream handle                          Modify QP

Invalid Modifier - One of the modifiers was        Modify QP
invalid or was not allowed to be modified in the
current state or state transition.

RI Still flushing WQEs - The QP is in the Error    Modify QP
state and a request to transition to the Idle
state but the RI is still flushing WQEs and
therefore cannot transition.

Invalid S-RQ handle                                Create QP

QP RQ Limit Out of Range.                          Create QP, Modify QP

Memory Windows still Bound to QP                   Destroy QP

                   Figure 32 - QP Management Verb Status

9.5.1.6  Memory Management Verb Status

Insufficient resources to complete     Allocate NS MR STag, RI-
request - An error was detected due to Register, RI-Reregister,
insufficient resources.                Register Shared MR, Allocate MW

Invalid RNIC handle - An invalid RNIC  Allocate NS MR STag, RI-
handle was specified.                  Register,Query MR, Deallocate
                                       STag, RI-Reregister, Register
                                       Shared MR, Allocate MW, Query MW

Invalid PD ID - An invalid PD ID was   Allocate NS MR STag, RI-
specified.                             Register, RI-Reregister,
                                       Register Shared MR, Allocate MW

Invalid Virtual Address - An invalid   RI-Register, RI-Reregister,
Memory Address or Offset was           Register Shared MR
specified.





   Hilland, et al.        Expires October 2003             [Page 207]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

Invalid Length - An invalid Length was RI-Register, RI-Reregister
specified. Too many pages or the MR
length was too long.

Invalid Access Rights requested - An   RI-Register, RI-Reregister,
invalid Access Control specifier was   Register Shared MR
specified.

Invalid Physical Buffer List entry.    RI-Register, RI-Reregister

Invalid Physical Buffer size - The     RI-Register, RI-Reregister
Physical Buffer size
(Page/Block)_requested was not
supported by the RNIC.

Invalid STag Index - An invalid Memory Query MR, RI-Reregister,
Region STag Index was specified.       Deallocate STag,
                                       Register Shared MR, Query MW

Invalid FBO - the FBO is larger than   RI-register, RI-Reregister
the physical buffer size

One or more Memory Windows is still    Deallocate STag, RI-Reregister,
Bound to the Region.

                 Figure 33 - Memory Management Verb Status

9.5.1.7  Post Verb Status

Invalid RNIC handle - An invalid RNIC handle was     PostSQ, PostRQ
specified.

Invalid QP handle - An invalid QP handle was         PostSQ, PostRQ
specified.

Invalid S-RQ handle - An invalid S-RQ handle was     PostRQ
specified.

Too many Work Requests posted.                       PostSQ, PostRQ

Invalid Operation type                               PostSQ

Invalid QP state.                                    PostSQ, PostRQ

Invalid Scatter/Gather list format                   PostSQ, PostRQ

Invalid Scatter/Gather list length - The Work        PostSQ, PostRQ




   Hilland, et al.        Expires October 2003             [Page 208]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

Request specified more Scatter/Gather elements than
the QP can support.

RQ Associated with S-RQ - This QP is associated with PostRQ
an S-RQ and therefore the QP Handle cannot be used
to post receive Work Requests. The S-RQ handle
should be used instead.

Invalid Modifier - One of the parameters were        PostSQ, PostRQ
invalid.

                       Figure 34 - Post Verb Status

9.5.1.8  Event Management Verb Status

Invalid RNIC handle - An invalid RNIC      Request Completion
handle was specified.                      Notification, Set Completion
                                           Event Handler, Set
                                           Asynchronous Event Handler

Invalid CQ handle - An invalid CQ handle   Request Completion
was specified.                             Notification

Invalid Notify Type - An invalid CQ        Request Completion
Notification type was specified.           Notification

Invalid Completion event handler           Set Completion Event Handler
identifier -           - An invalid identifier was
specified while attempting to clear a
Completion Event Handler address.

Insufficient Resources - The RI did not    Set Completion Event Handler
have sufficient resources to complete the
request, such as when the Consumer
requests another Completion Event Handler
Identifier but has already set an amount
equal to the value returned in Query RNIC.

                 Figure 35 - Event Management Verb Status












   Hilland, et al.        Expires October 2003             [Page 209]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

9.5.2  Completion Status Codes

Success - The RNIC Operation was           Send Operation Types,
successful.                                Receive, RDMA Write, RDMA
                                           Read, RDMA Read with
                                           Invalidate Local STag, Bind,
                                           Fast-Register, Invalidate
                                           Local STag

Flushed - The Work Request was incomplete  Send Operation Types,
when the QP entered the Error state.       Receive, RDMA Write, RDMA
                                           Read, RDMA Read with
                                           Invalidate Local STag, Bind,
                                           Fast-Register, Invalidate
                                           Local STag

Invalid WQE - The Work Request Element     Send Operation Types,
contained a format error.                  Receive, RDMA Write, RDMA
                                           Read, RDMA Read with
                                           Invalidate Local STag, Bind,
                                           Fast-Register, Invalidate
                                           Local STag

Local QP Catastrophic Error - An error     Send Operation Types,
related to the QP occurred while           Receive, RDMA Write, RDMA
processing the Work Request.               Read, RDMA Read with
                                           Invalidate Local STag, Bind,
                                           Fast-Register, Invalidate
                                           Local STag

Remote Termination Error - A Terminate     Send Operation Types, RDMA
Message was received from the Remote Peer  Write, RDMA Read, RDMA Read
that appears to be related to the          with Invalidate Local STag
execution of this Work Request. The error
type can be examined by looking at the
Terminate Message buffer via Query QP.

Invalid STag - An invalid STag was found   Send Operation Types,
in the local SGL. The STag was either not  Receive, RDMA Write, RDMA
found allocated, bound, or registered in   Read, RDMA Read with
the RI, or an STag of zero was specified   Invalidate Local STag, Bind,
for a QP without Privileged rights, or     Fast-Register, Invalidate
referred to a Shared Memory Region, or the Local STag
type of STag supplied was not allowed to
be used in the specified operation.







   Hilland, et al.        Expires October 2003             [Page 210]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

Base & Bounds Violation - The local SGL    Send Operation Types,
referenced an address beyond the limits    Receive, RDMA Write, RDMA
specified for the MR or MW. This includes  Read, RDMA Read with
length errors. For a Bind, the MW was not  Invalidate Local STag, Bind
wholly contained in the MR.

Access Violation - The RNIC attempted to   Send Operation Types,
read or write to a local SGL MR or MW that Receive, RDMA Write, RDMA
did not provide appropriate Access Rights. Read, RDMA Read with
For a Bind, the MW Access Rights were not  Invalidate Local STag, Bind
compatible with the MR Access Rights.

Invalid PD ID - For one of the STags       Send Operation Types,
specified in the Work Request the PD of    Receive, RDMA Write, RDMA
the MR STag was not the same as the PD of  Read, RDMA Read with
the QP, or, the QP of the MW STag was not  Invalidate Local STag, Bind,
the same as QP.                            Fast-Register, Invalidate
                                           Local STag

Wrap Error - The specified Address or      Send Operation Types,
offset (TO or MO) added to the length of   Receive, RDMA Write, RDMA
the operation resulted in a wrap beyond    Read, RDMA Read with
the machine-supported address.             Invalidate Local STag, Bind,
                                           Fast-Register

STag to Invalidate had Invalid PD or       Receive
Access Rights - The Invalidate STag on a
Receive did not have a PD ID that matched
the PD ID of the QP (for a MR) or a QP ID
that matched the QP ID of the QP (for a
MW). Or the STag did not have Access
Rights to be invalidated remotely.

Zero RDMA Read Resources - The QP ORD      RDMA Read, RDMA Read with
value was set to zero.                     Invalidate Local STag

QP Not In Privileged Mode - The QP is not  Fast-Register
enabled to perform the Privileged WR.

STag Not In Invalid state - The STag was   Bind,
already registered or bound, when          Fast-Register
attempting to Register or Bind it.

Invalid Page Size - The page size          Fast-Register
requested was not supported by the RNIC.

Invalid Physical Buffer Size - size not    Fast-Register
supported by the RNIC.




   Hilland, et al.        Expires October 2003             [Page 211]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

Invalid Physical Buffer List entry - for   Fast-Register
page mode, the entry must start on page
size boundaries.

Invalid FBO - the FBO is larger than the   Fast-Register
physical buffer size.

Invalid length - requested length is       Fast-Register
larger than supported by the buffer list.

Invalid Access Rights specified.           Fast-Register

Physical Buffer List too long.             Fast-Register

Invalid Virtual Address - VA and FBO are   Fast-Register
not consistent.

Invalid Region - The STag specified for    Bind
the MR in the BIND request was invalid.

Invalid Window - The STag specified for    Bind
the MW in the BIND request was invalid.

Invalid Length - The total size of the     Send, Receive, RDMA Write,
data to be moved as specified by the sum   RDMA Read, RDMA Read with
of the SGL elements, was larger than that  Invalidate Local STag
supported by the RNIC.

                    Figure 36 - Completion Status Codes

9.5.3  Asynchronous Event Identifiers

   The following table contains the list of Event Identifiers and
   Resource Indicators that the RNIC MUST support as Asynchronous Event
   Identifiers to be returned by the Asynchronous Event Handler. Note
   that the Resource Indicator dictates that the appropriate Resource
   Identifier corresponding to that Resource Indicator MUST be returned
   as well. For more information, see Section 9.4.2 - Set Asynchronous
   Event Handler.












   Hilland, et al.        Expires October 2003             [Page 212]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

             Event Identifier and Description.             Resource
                                                            Indicator

   LLP Close Complete - The RDMA Stream has completed     QP ID
   Closing and no SQ WQEs were flushed.

   Terminate Message Received                             QP ID

   LLP Connection Reset - An incoming LLP Reset (e.g. RST QP ID
   on TCP) was received.

   LLP Connection Lost                                    QP ID

   LLP Integrity Error: Segment size invalid              QP ID

   LLP Integrity Error: Invalid CRC                       QP ID

   LLP Integrity Error: Bad FPDU - Received MPA marker    QP ID
   and 'Length' fields do not agree on the start of a
   FPDU

   Remote Operation Error: Invalid DDP version - caused   QP ID
   by an inbound segment.

   Remote Operation Error: Invalid RDMA version - caused  QP ID
   by an inbound segment.

   Remote Operation Error: Unexpected Opcode - caused by  QP ID
   an inbound segment.

   Remote Operation Error: Invalid DDP Queue Number -     QP ID
   caused by an inbound segment.

   Remote Operation Error: Invalid RDMA Read Request      QP ID
   Message, RDMA Read not enabled - caused by an inbound
   segment.

   Remote Operation Error: Invalid RDMA Write or RDMA     QP ID
   Read Response Message, RDMA Write & RDMA Read Response
   not enabled - caused by an inbound segment.

   Remote Operation Error: Invalid RDMA Read Request      QP ID
   Message, message size too small or Offset non-zero -
   caused by an inbound segment.

   Remote Operation Error: No 'L' bit when expected -     QP ID
   caused by an inbound segment.





   Hilland, et al.        Expires October 2003             [Page 213]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Protection Error: Invalid STag - caused by an inbound  QP ID
   Tagged DDP segment not valid for this QP. This
   includes using the STag of zero, the STag was not
   associated with the QP or the STag was in the Invalid
   state.

   Protection Error: Tagged Base and bounds violation -   QP ID
   caused by an inbound Tagged segment attempted to
   access memory outside the limits assigned to the STag.

   Protection Error: Tagged Access Rights violation -     QP ID
   caused by an inbound segment referencing a Tagged
   Buffer which did not have the necessary memory Access
   Rights for the requested operation.

   Protection Error: Tagged Invalid PD - caused by an     QP ID
   inbound segment referencing a Tagged Buffer which was
   not allowed to be referenced by QP.

   Protection Error: Wrap error - caused by an inbound    QP ID
   segment not targeting the RQ.

   Bad Close - The QP was in the Closing state when a     QP ID
   Segment arrived.

   Bad LLP Close - An attempt was made to close the RDMA  QP ID
   Stream with work in progress.

   RQ Protection Error - Invalid MSN - MSN range not      QP ID
   valid. Caused by an inbound segment targeting the RQ.
   Possibly due to Receive Queue being empty.

   RQ Protection Error - Invalid MSN - gap in MSN. Caused QP ID
   by an inbound segment targeting the RQ.

   IRRQ Protection Error: Invalid MSN - too many RDMA     QP ID
   Read Request Messages in progress - caused by an
   inbound segment not targeting the IRRQ.

   IRRQ Protection Error: Invalid MSN - gap in MSN -      QP ID
   caused by an inbound segment not targeting the RQ.

   IRRQ Protection Error: Invalid MSN - range is not      QP ID
   valid - caused by an inbound segment not targeting the
   RQ.

   IRRQ Protection Error: Invalid STag - Data Source STag QP ID
   determined to be invalid during RDMA Read Response




   Hilland, et al.        Expires October 2003             [Page 214]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   processing.

   IRRQ Protection Error: Tagged Base and bounds          QP ID
   violation - This includes RDMA Read Request of a
   message larger than supported by the RNIC. It is
   detected accessing the Data Source during RDMA Read
   Response processing.

   IRRQ Protection Error: Tagged Access Rights violation  QP ID
   - Data Source Access Rights violation detected during
   RDMA Read Response processing.

   IRRQ Protection Error: Tagged Invalid PD - Data Source QP ID
   PD violation detected during RDMA Read Response
   processing.

   IRRQ Protection Error: Wrap error - detected during    QP ID
   RDMA Read Response processing.

   CQ/SQ Error: CQ Overflow Error - An error occurred on  QP ID
   the CQ during a SQ completion.

   CQ/RQ Error: CQ Operation error - An error occurred on QP ID
   the CQ during a RQ completion.

   S-RQ error on a QP - An error occurred while           QP ID
   attempting to pull a WQE from the S-RQ associated with
   the QP.

   Local QP Catastrophic Error - occurred during          QP ID
   processing.

   CQ Overflow Detected - An overflow of the Completion   CQ Handle
   Queue has been detected. This Error Code is OPTIONAL.

   CQ Operation Error - An error occurred on the CQ       CQ Handle
   unrelated to a specific QP completion.

   Shared Receive Queue Limit reached - The Limit value   S-RQ Handle
   established for the Shared Receive Queue has been
   reached.

   QP RQ Limit Reached - The Limit value established for  QP ID
   the QP's RQ has been reached.

   Shared Receive Queue Catastrophic Failure - A problem  S-RQ Handle
   occurred with the RNIC or its driver that renders the
   RNIC unable to use the S-RQ.




   Hilland, et al.        Expires October 2003             [Page 215]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   RNIC Catastrophic Failure - A problem occurred with    RNIC Handle
   the RNIC or its driver that renders the RNIC unable to
   reliably function.

                Figure 37 - Asynchronous Event Identifiers















































   Hilland, et al.        Expires October 2003             [Page 216]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

10 Security Considerations

   Security Considerations are necessary for the RDMA Protocols and
   this specification.  An Internet Draft is under development.

















































   Hilland, et al.        Expires October 2003             [Page 217]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

11 IANA Considerations

   If DDP was enabled a priori for a ULP by connecting to a well-known
   port, this well-known port would be registered for the DDP with
   IANA.
















































   Hilland, et al.        Expires October 2003             [Page 218]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

12 References

12.1 Normative References

   [RFC2026] Bradner, S., "The Internet Standards Process -- Revision
       3", BCP 9, RFC 2026, October 1996.

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
       Requirement Levels", BCP 14, RFC 2119, March 1997.

   [MPA] P. Culley et al., "Markers with PDU Alignment", RDMA
       Consortium Draft Specification draft-cully-iwarp-mpa-00.doc,
       October 2002

   [DDP] H. Shah et al., "Direct Data Placement over Reliable
       Transports", RDMA Consortium Draft Specification draft-shah-
       iwarp-ddp-00.txt, October 2002

   [RDMAP] R. Recio et al., "RDMA Protocol Specification", RDMA
       Consortium Draft Specification draft-recio-iwarp-00, October
       2002

   [SCTP] R. Stewart et al., "Stream Control Transmission Protocol",
       RFC 2960, October 2000.

   [TCP] Postel, J., "Transmission Control Protocol", STD 7, RFC 793,
       September 1981.

12.2 Informative References

    [IPSEC] Atkinson, R., Kent, S., "Security Architecture for the
       Internet Protocol", RFC 2401, November 1998.





















   Hilland, et al.        Expires October 2003             [Page 219]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

13 Appendix

13.1 Connection Initialization at LLP Startup

   The purpose of an initialization at LLP Startup is to enable iWARP
   using the minimum number of messages possible. Note that not all
   RNIC/OS implementations are required to support this.







             < Figure 39 did not convert properly from source >
             <  to be corrected in an upcoming version        >










     Figure 39 - Connection Initialization at LLP Startup (using TCP)


   Below is an example sequence for an iWARP startup that accomplishes
   this (other sequences are possible). The Sequence applies equally to
   either the active or passive side.

   *   The Consumer establishes the LLP Connection using a non-Verbs
       interface.

   *   The Consumer creates a QP, setting up the CQ, PD, etc., and
       registers memory for buffers.

   *   The Consumer posts buffers to the RQ appropriate for the
       expected traffic.

   *   If the ULP intends to transmit first, the Consumer could Post
       one or more Work Request(s) on the SQ (usually a SEND message)
       that will be sent after the QP is placed in the RTS state.







   Hilland, et al.        Expires October 2003             [Page 220]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   *   The Consumer moves the QP state to RTS. The Modify QP Verb for
       this includes the LLP Stream Handle, and does not include a
       streaming message buffer.

   *   If the local Consumer intends to perform RDMA Read Type WRs, the
       local Consumer obtains, in some ULP defined message, the number
       of incoming RDMA Read Request Messages that the Remote Peer can
       have outstanding (IRD). If the Remote Peer's IRD is smaller than
       the local Peer's ORD, the local Consumer should also perform a
       Modify QP Verb with the Remote Peer's IRD value placed into the
       local ORD value prior to posting the first RDMA Read Type WR.
       The local Consumer may also transmit, in some ULP defined
       message, the number of outgoing RDMA Read Request Messages that
       the Local Peer can have outstanding (ORD).

   *   If the local Consumer intends the QP to be the Data Source of
       RDMA Read Operations, the Consumer provides, in some ULP defined
       message, the number of incoming RDMA Read Request Messages (e.g.
       IRRQ depth) that the Local Peer can have outstanding (IRD). The
       Consumer may also receive, in some ULP defined message, the
       number of outgoing RDMA Read Request Messages that the Remote
       Peer can have outstanding (ORD). If the Remote Peer's ORD is
       smaller than the Local Peer's IRD, the local Consumer may also
       perform a Modify QP Verb with the Remote Peer's ORD value placed
       into the local IRD value prior to posting the first RDMA Read
       Type WR.

   This specification does not define which side of the connection
   sends the first message, the active or passive side; the ULP is
   responsible for determining this. In addition, this specification
   does not preclude the use of Active/Active connections.

   RNIC Implementers note: Since there is no integration between the RI
   and the LLP Connection startup sequence, as defined above, it is
   possible that some data may arrive over the transport before the
   RNIC is in iWARP mode. It is the responsibility of the RI to accept
   this data and interpret it as iWARP data. Alternately, the Consumer
   (or other service that establishes the LLP Connection) can ensure
   that no data will be received prior to moving the QP to RTS state.
   If neither of these methods is available, then iWARP startup with
   the LLP is not available.

13.2 Graceful Receive Overflow Handling

   A valid implementation option is to gracefully handle Receive Queue
   or Shared-Receive Queue overflow. In a strictly layered model, this
   may be difficult but in an RNIC implementation, this should be
   feasible.





   Hilland, et al.        Expires October 2003             [Page 221]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   In the current architecture, if there are no Receive Queue Work
   Queue Elements available when an Untagged Message arrives then the
   connection is dropped. This is true if there is a Shared Receive
   Queue or a dedicated receive queue.

   In this case, the implementation (RI/RNIC), which is not relying on
   an external LLP, may choose to handle this gracefully through LLP
   mechanisms. In this case, the RI will choose to not drop the
   connection and instead appear to pause receive queue processing
   until more WQEs have been posted to the RQ or S-RQ.

   How the RNIC decides to perform this function is left up to
   implementation. One example mechanism which may be used to
   gracefully handle receive overflow is for the implementation to drop
   incoming packets when there are no WQEs on the RQ or S-RQ. This type
   of mechanism may have side effects, such as causing back-off
   algorithms to be invoked, but this type of mechanism is still a
   valid implementation option.



































   Hilland, et al.        Expires October 2003             [Page 222]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

14 AuthorÆs Addresses

   Jeff Hilland
   Hewlett-Packard Company
   20555 SH 249
   Houston, TX 77070-2698 USA
   Phone: +1 (281) 514-9489
   Email: jeff.hilland@hp.com

   Paul R. Culley
   Hewlett-Packard Company
   20555 SH 249
   Houston, TX 77070-2698 USA
   Phone: +1 (281) 514-5543
   Email: paul.culley@hp.com

   James Pinkerton
   Microsoft Corporation
   One Microsoft Way
   Redmond, WA. 98052 USA
   Phone: +1 (425) 705-5442
   Email: jpink@windows.microsoft.com

   Renato Recio
   IBM Corporation
   11501 Burnett Road
   Austin, TX 78758 USA
   Phone: +1 (512) 838-1365
   Email: recio@us.ibm.com
























   Hilland, et al.        Expires October 2003             [Page 223]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

15 Acknowledgments

   John Carrier
   Adaptec, Inc.
   691 S. Milpitas Blvd.
   Milpitas, CA 95035 USA
   Phone: +1 (360) 378-8526
   Email: john_carrier@adaptec.com

   Hari Ghadia
   Adaptec, Inc.
   691 S. Milpitas Blvd.,
   Milpitas, CA 95035 USA
   Phone: +1 (408) 957-5608
   Email: hari_ghadia@adaptec.com

   Patricia Thaler
   Agilent Technologies, Inc.
   1101 Creekside Ridge Drive, #100
   M/S-RG10
   Roseville, CA 95678
   Phone: +1 (916) 788-5662
   email: pat_thaler@agilent.com

   Mike Penna
   Broadcom Corporation
   16215 Alton Parkway
   Irvine, California 92619-7013 USA
   Phone: +1 (949) 926-7149
   Email: MPenna@Broadcom.com

   Uri Elzur
   Broadcom Corporation
   16215 Alton Parkway
   Irvine, California 92619-7013 USA
   Phone: +1 (949) 585-6432
   Email: Uri@Broadcom.com

   Ted Compton
   EMC Corporation
   Research Triangle Park, NC 27709, USA
   Phone: +1 (919) 248-6075
   Email: compton_ted@emc.com

   Dwight Barron
   Hewlett-Packard Company
   20555 SH 249
   Houston, TX 77070-2698 USA
   Phone: +1 (281) 514-2769
   Email: Dwight.Barron@Hp.com



   Hilland, et al.        Expires October 2003             [Page 224]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Mallikarjun Chadalapaka
   Hewlett-Packard Company
   8000 Foothills Blvd.
   Roseville, CA 95747-5668, USA
   Phone: +1 (916) 785-5621
   Email: cbm@rose.hp.com

   Dave Garcia
   Hewlett-Packard Company
   19333 Vallco Parkway
   Cupertino, Ca. 95014 USA
   Phone: +1 (408) 285-6116
   Email: dave.garcia@hp.com

   Mike Krause
   Hewlett-Packard Company, 43LN
   19410 Homestead Road
   Cupertino, CA 95014 USA
   Phone: +1 (408) 447-3191
   Email: krause@cup.hp.com

   Jim Wendt
   Hewlett-Packard Company
   8000 Foothills Boulevard
   Roseville, CA 95747-5668 USA
   Phone: +1 (916) 785-5198
   Email: jim_wendt@hp.com

   John L. Hufferd
   IBM Corp.
   650 Harry Rd.
   San Jose CA
   Phone: +1 (408) 256-0403
   Email: hufferd@us.ibm.com

   Mike Ko
   IBM Corp.
   650 Harry Rd.
   San Jose, CA 95120, USA
   Phone: +1 (408) 927-2085
   Email: mako@us.ibm.com

   Ellen Deleganes
   Intel Corporation
   MS JF5-355
   2111 NE 25th Ave.
   Hillsboro, OR 97124 USA
   Phone: +1 (503) 712-4173
   Email: ellen.m.deleganes@intel.com




   Hilland, et al.        Expires October 2003             [Page 225]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

   Frank Berry
   Intel Corporation
   2111 NE 25th Ave.
   Hillsboro, OR 97124 USA
   Phone: +1 (503) 712-3897
   Email: frank.berry@intel.com

   Howard C. Herbert
   Intel Corporation
   MS CH7-404
   5000 West Chandler Blvd.
   Chandler, AZ 85226 USA
   Phone: +1 (480) 554-3116
   Email: howard.c.herbert@intel.com

   Dave Minturn
   Intel Corporation
   MS JF1-210
   5200 North East Elam Young Parkway
   Hillsboro, OR 97124 USA
   Phone: +1 (503) 712-4106
   Email: dave.b.minturn@intel.com

   Hemal Shah
   Intel Corporation
   MS PTL1
   1501 South Mopac Expressway, #400
   Austin, TX 78746 USA
   Phone: +1 (512) 732-3963
   Email: hemal.shah@intel.com

   James Livingston
   NEC Solutions (America), Inc.
   7525 166th Ave. N.E., Suite D210
   Redmond, WA 98052-7811
   Phone: +1 (425) 897-2033
   Email: james.livingston@necsam.com

   Tom Talpey
   Network Appliance
   375 Totten Pond Road
   Waltham, MA 02451 USA
   Phone: +1 (781) 768-5329
   Email: thomas.talpey@netapp.com









   Hilland, et al.        Expires October 2003             [Page 226]

   Internet-Draft      RDMA Verbs Specification            25 Apr 2003

16 Full Copyright Statement

   This document and the information contained herein is provided on an
   ææAS ISÆÆ basis and ADAPTEC INC., AGILENT TECHNOLOGIES INC., BROADCOM
   CORPORATION, CISCO SYSTEMS INC., DELL COMPUTER CORPORATION, EMC
   CORPORATION, HEWLETT-PACKARD COMPANY, INTERNATIONAL BUSINESS
   MACHINES CORPORATION, INTEL CORPORATION, MICROSOFT CORPORATION, NEC
   SOLUTIONS (AMERICA), INC., NETWORK APPLIANCE INC., THE INTERNET
   SOCIETY, AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL
   WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY
   WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE
   ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
   FOR A PARTICULAR PURPOSE.

   Copyright (c) 2002, 2003 ADAPTEC INC., BROADCOM CORPORATION, CISCO
   SYSTEMS INC., DELL COMPUTER CORPORATION, EMC CORPORATION, HEWLETT-
   PACKARD COMPANY, INTERNATIONAL BUSINESS MACHINES CORPORATION, INTEL
   CORPORATION, MICROSOFT CORPORATION, NETWORK APPLIANCE INC., All
   Rights Reserved.


































   Hilland, et al.        Expires October 2003             [Page 227]


Html markup produced by rfcmarkup 1.107, available from http://tools.ietf.org/tools/rfcmarkup/