Registration Protocols Extensions M. Loffredo Internet-Draft M. Martinelli Intended status: Standards Track IIT-CNR/Registro.it Expires:August 5,October 13, 2019 S. Hollenbeck Verisign LabsFebruary 1,April 11, 2019 Registration Data Access Protocol (RDAP) Query Parameters for Result Sorting and Pagingdraft-ietf-regext-rdap-sorting-and-paging-00draft-ietf-regext-rdap-sorting-and-paging-01 Abstract The Registration Data Access Protocol (RDAP) does not include core functionality for clients to provide sorting and paging parameters for control of large result sets. This omission can lead to unpredictable server processing of queries and client processing of responses. This unpredictability can be greatly reduced if clients can provide servers with their preferences for managing response values. This document describes RDAP query extensions that allow clients to specify their preferences for sorting and paging result sets. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire onAugust 5,October 13, 2019. Copyright Notice Copyright (c) 2019 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Conventions Used in This Document . . . . . . . . . . . . 4 2. RDAP Query Parameter Specification . . . . . . . . . . . . . 4 2.1. Sorting and Paging Metadata . . . . . . . . . . . . . . . 4 2.2. "count" Parameter . . . . . . . . . . . . . . . . . . . . 6 2.3. "sort" Parameter . . . . . . . . . . . . . . . . . . . . 7 2.3.1. Representing Sorting Links . . . . . . . . . . . . . 11 2.4. "limit" and "offset" Parameters . . . . . . . . . . . . . 12 2.4.1. RepresentingPagingOffset Pagination Links . . . . . . . . 13 2.5. "cursor" Parameter . . . . . .13. . . . . . . . . . . . . 14 2.5.1. Representing Cursor Pagination Links . . . . . . . . 16 3. Negative Answers . . . . . . . . . . . . . . . . . . . . . .1417 4. RDAP Conformance . . . . . . . . . . . . . . . . . . . . . .1518 5. Implementation Considerations . . . . . . . . . . . . . . . .15 5.1. Considerations about Paging Implementation . . . . . . . 1618 6. Implementation Status . . . . . . . . . . . . . . . . . . . . 19 6.1. IIT-CNR/Registro.it . . . . . . . . . . . . . . . . . . .1920 6.2. Google Registry . . . . . . . . . . . . . . . . . . . . .1920 7. Security Considerations . . . . . . . . . . . . . . . . . . . 20 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . .2021 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . .2021 10. References . . . . . . . . . . . . . . . . . . . . . . . . .2021 10.1. Normative References . . . . . . . . . . . . . . . . . . 21 10.2. Informative References . . . . . . . . . . . . . . . . . 22 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . .2425 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . .2425 1. Introduction The availability of functionality for result sorting and paging provides benefits to both clients and servers in the implementation of RESTful services [REST]. These benefits include: o reducing the server response bandwidth requirements; o improving server response time; o improving query precision and, consequently, obtaining more reliable results; o decreasing server query processing load; o reducing client response processing time. Approaches to implementing features for result sorting and paging can be grouped into two main categories: 1. Sorting and paging are implemented through the introduction of additional parameters in the query string (i.e. ODATA protocol [OData-Part1]); 2. Information related to the number of results and the specific portion of the result set to be returned, in addition to a set of ready-made links for the result set scrolling, are inserted in the HTTP header of the request/response. However, there are some drawbacks associated with use of the HTTP header. First, the header properties cannot be set directly from a web browser. Moreover, in an HTTP session, the information on the status (i.e. the session identifier) is usually inserted in the header or in the cookies, while the information on the resource identification or the search type is included in the query string. The second approach is therefore not compliant with the HTTP standard [RFC7230]. As a result, this document describes a specification based on use of query parameters. Currently the RDAP protocol [RFC7482] defines two query types: o lookup: the server returns only one object; o search: the server returns a collection of objects. While the lookup query does not raise issues in the management of large result sets, the search query can potentially generate a large result set that could be truncated according to the limits of the server. In addition, it is not possible to obtain the total number of the objects found that might be returned in a search query response [RFC7483]. Lastly, there is no way to specify sort criteria to return the most relevant objects at the beginning of the result set. Therefore, the client could traverse the whole result set to find the relevant objects or, due to truncation, could not find them at all. The protocol described in this specification extends RDAP query capabilities to enable result sorting and paging, by adding new query parameters that can be applied to RDAP search path segments. The service is implemented using the Hypertext Transfer Protocol (HTTP) [RFC7230] and the conventions described in RFC 7480 [RFC7480]. The implementation ofthesethe new parameters is technically feasible, as operators for counting, sorting and paging rows are currently supported by the major RDBMSs. 1.1. Conventions Used in This Document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 2. RDAP Query Parameter Specification The new query parameters are OPTIONAL extensions of path segments defined in RFC 7482 [RFC7482]. They are as follows: o "count": a boolean value that allows a client to request the total number of objects found (that due to truncation can be different from the number of returned objects); o "sort": a string value that allows a client to request a specific sort order for the result set; o "limit" and "offset": numeric values that allow a client to request a specific portion of the entire result set; o "cursor": a string value representing a pointer to a specific fixed size portion of the result set. It is alternate to "limit" and "offset". Augmented Backus-Naur Form (ABNF) [RFC5234] is used in the following sections to describe the formal syntax of these new parameters. 2.1. Sorting and Paging Metadata According to most advanced principles in REST design, collectively known as HATEOAS (Hypermedia as the Engine of Application State) ([HATEOAS]), a client entering a REST application through an initial URI should use the server-provided links to dynamically discover available actions and access the resources it needs. In this way, the client is not requested to have prior knowledge of the service and, consequently, to hard code the URIs of different resources. This would allow the server to make URI changes as the API evolves without breaking the clients. Definitively, a REST service should be as self-descriptive as possible. Therefore,the implementation ofservers implementing the query parameters described in this specificationrecommends servers toSHOULD provide additional information in their responses about both the available sorting criteria and the possible pagination. Such information is collected in two new data structures named, respectively, "sorting_metadata" and "paging_metadata". Obviously, both the new data structures are OPTIONAL because their presence in the response not only depends on the implementation of sorting and paging query capabilities but also on some situations related to the results. For example, it is quite natural to expect that the "paging_metadata"sectionelement will not be present at the last result page when the server implements only the forward pagination. The "sorting_metadata" structure contains the followingfields:properties: o "currentSort": "String" (OPTIONAL) either the value of sortparameter"parameter" as specified in the querystring;string or the sort applied by default, if any; o "availableSorts": "AvailableSort[]" (OPTIONAL) an array of objects each one describing an alternate available sortingcriterion:criterion. Members are: * "property": "String" (REQUIRED) the name that can be used by the client to request the sorting criterion; * "default": "Boolean" (REQUIRED) whether the sorting criterion is applied by default; * "jsonPath": "String" (OPTIONAL) the JSON Path of the RDAP field corresponding to the property; *"default": whether the sorting criterion is applied by default; *"links": "Link[]" (OPTIONAL) an array of links as described in RFC 8288 [RFC8288] containing the query string that applies the sorting criterion.BothAt least one between "currentSort" and "availableSorts"are OPTIONAL fields of the "sorting_metadata" structure. In particular, the "currentSort" field is provided when the query string contains a valid value for sort parameter, while the "availableSorts" field SHOULDMUST beprovided when the sort parameter is missing in the query string or when it is present and the server implements more than a sorting criterion for the RDAP object. At least the "property" field is REQUIRED in each item of the "availableSorts" array while the other fields are RECOMMENDED.present. The "paging_metadata" structure contains the following fields: o "totalCount": "Numeric" (OPTIONAL) a numeric value representing the total number of objectsfound;found. It is provided if the query string contains the "count" parameter; o "pageCount": "Numeric" (OPTIONAL) a numeric value representing the number of objects returned in the currentpage; o "offset": a numeric value identifyingpage. It is provided when thestarttotal number ofcurrentobjects exceeds the pageinsize. This property is redundant for clients because theresult set; o "nextOffset": a numeric value identifyingpage size can be derived from thestartlength of thenext page in the result set or nullsearch results array but it can be helpful if theresult set has been completely scrolled;end user interacts with the server through a web browser; o "links": "Link[]" (OPTIONAL) an array of links as described in RFC 8288 [RFC8288] containing the reference to next page.Only the "pageCount" field is REQUIRED in the "paging_metadata" structure. The other fields appear when pagination occurs.In this specification, only the forward pagination is dealt because it is considered satisfactory in order to traverse the result set.If a server should also implement backward pagination, an appropriate field (e.g. "prevOffset") identifying the startExamples of additional references are to: the previouspage is RECOMMENDED. Finally, the "totalCount" field is provided if the query string containspage, thecount parameter. FOR DISCUSSION: Shouldfirst page, themetadata described in this specificationlast page. At least one between "totalCount" and "links" MUST bepart of a more general "metadata" section including other contents (e.g rate limits, information about the server, information about the response, metadata information related to other parameters)?present. 2.2. "count" Parameter Currently the RDAP protocol does not allow a client to determine the total number of the results in a query response when the result set is truncated. This is rather inefficient because the user cannot evaluate the query precision and, at the same time, cannot receive information that could be relevant. Thecount"count" parameter provides additional functionality (Figure 1) that allows a client to request information from the server that specifies the total number of elements matching a particular search pattern. https://example.com/rdap/domains?name=*nr.com&count=true Figure 1: Example of RDAP query reporting thecount"count" parameter The ABNF syntax is the following: count = "count" EQ ( trueValue / falseValue ) trueValue = ("true" / "yes" / "1") falseValue = ("false" / "no" / "0") EQ = "=" A trueValue means that the server MUST provide the total number of the objects in the "totalCount" field of the "paging_metadata"sectionelement (Figure 2). A falseValue means that the server MUST NOT provide this number. { "rdapConformance": [ "rdap_level_0", "paging_level_0" ], ... "paging_metadata": { "totalCount": 73 }, "domainSearchResults": [ ... ] } Figure 2: Example of RDAP response with "paging_metadata"sectionelement containing the "totalCount" field 2.3. "sort" Parameter The RDAP protocol does not provide any capability to specify results sort criteria. A server could implement a default sorting scheme according to the object class, but this feature is not mandatory and might not meet user requirements. Sorting can be addressed by the client, but this solution is rather inefficient. Sorting and paging features provided by the RDAP server could help avoid truncation of relevant results and allow for scrolling the result set using subsequent queries. Thesort"sort" parameter allows the client to ask the server to sort the results according to the values of one or more properties and according to the sort direction of each property. The ABNF syntax is the following: sort = "sort" EQ sortItem *( "," sortItem ) sortItem = property-ref [":" ( "a" / "d" ) ] "a" means that the ascending sort MUST be applied, "d" means that the descending sort MUST be applied. If the sort direction is absent, an ascending sort MUST be applied (Figure 3). In thesort"sort" parameter ABNF syntax, property-ref represents a reference to a property of an RDAP object. Such a reference could be expressed by using a JSON Path. The JSON Path in a JSON document [RFC8259] is equivalent to the XPath [W3C.CR-xpath-31-20161213] in a XML document. For example, the JSON Path to select the value of the ASCII name inside an RDAP domain object is "$.ldhName", where $ identifies the root of the document (DOM). Another way to select a value inside a JSON document is the JSON Pointer [RFC6901]. While JSON Path or JSON Pointer are both standard ways to select any value inside JSON data, neither is particularly easy to use (e.g. "$.events[?(@.eventAction='registration')].eventDate" is the JSON Path expression of the registration date inaan RDAP domain object). Therefore, this specification provides a definition of property-ref in terms of RDAP properties. However, not all the RDAP properties are suitable to be used in sort criteria, such as: o properties providing service information (e.g. links, notices, remarks, etc.); o multivalued properties (e.g. status, roles, variants, etc.); o properties modeling relationships to other objects (e.g. entities). On the contrary, some properties expressed as values of other properties (e.g. registration date) could be used in such a context. In the following, a list ofthe proposedpropertiesfor sort criteriaan RDAP server MAY implement is presented. The properties are divided in two groups: object common properties and object specific properties. o Object common properties. Object common properties are derived from the merge of the "eventAction" and the "eventDate" properties. The following values of thesort"sort" parameter are defined: * registrationDate * reregistrationDate * lastChangedDate * expirationDate * deletionDate * reinstantiationDate * transferDate * lockedDate * unlockedDate o Object specific properties. With regard to the specific properties, some of them are already defined among the query paths. In the following the list of the proposed sorting properties, grouped by objects, is shown: * Domain: ldhName * Nameserver: ldhName, ipV4, ipV6. * Entity: fn, handle, org, email, voice, country, city. In the following, the correspondence between the sorting properties and the RDAP fields is shown (Table 1): +------------+------------+---------------+------------+------------+ | Object | Sorting | RDAP property | Reference | Reference | | class | property | | in RFC | in RFC | | | | | 7483 | 6350 | +------------+------------+---------------+------------+------------+ | Searchable | Common | eventAction | 4.5. | | | objects | properties | values | | | | | | suffixed by | | | | | | "Date" | | | | | | | | | | Domain | ldhName | ldhName | 5.3. | | | | | | | | | Nameserver | ldhName | ldhName | 5.2. | | | | ipV4 | v4 ipAddress | 5.2. | | | | ipV6 | v6 ipAddress | 5.2. | | | | | | | | | Entity | handle | handle | 5.1. | | | | fn | vcard fn | 5.1. | 6.2.1 | | | org | vcard org | 5.1. | 6.6.4 | | | voice | vcard tel | 5.1. | 6.4.1 | | | | with | | | | | | type="voice" | | | | | email | vcard email | 5.1. | 6.4.2 | | | country | country name | 5.1. | 6.3.1 | | | | in vcard adr | | | | | city | locality in | 5.1. | 6.3.1 | | | | vcard adr | | | +------------+------------+---------------+------------+------------+ Table 1: Sorting properties definition With regard to the definitions in Table 1, some further considerations must be made to disambiguate cases where the RDAP property is multivalued: o Even if a nameserver can have multiple IPv4 and IPv6 addresses, the most common configuration includes one address for each IP version. Therefore, the assumption of having a single IPv4 and/or IPv6 value for a nameserver cannot be considered too stringent. o With the exception of handle values, all the sorting properties defined for entity objects can be multivalued according to the definition of vCard as given in RFC6350 [RFC6350]. When more than a value is reported, sorting can be applied to the preferred value identified by the parameter pref="1". Each RDAP provider MAY define other sorting properties than those shown in this document. The "jsonPath" field in the "sorting_metadata" section is used to clarify the RDAP field the sorting property refers to. In the following, the mapping between the sorting properties and the JSON Paths of the RDAP fields is shown (Table 2). The JSON Paths are provided according to the Goessner v.0.8.0 specification ([GOESSNER-JSON-PATH]): +-------+-------------+---------------------------------------------+ | Objec | Sorting | JSON Path | | t | property | | | class | | | +-------+-------------+---------------------------------------------+ | Searc | registratio | "$.domainSearchResults[*].events[?(@.eventA | | hable | nDate | ction=="registration")].eventDate | | objec | | | | ts | | | | | reregistrat | "$.domainSearchResults[*].events[?(@.eventA | | | ionDate | ction=="reregistration")].eventDate | | | lastChanged | "$.domainSearchResults[*].events[?(@.eventA | | | Date | ction=="lastChanged")].eventDate | | | expirationD | "$.domainSearchResults[*].events[?(@.eventA | | | ate | ction=="expiration")].eventDate | | | deletionDat | "$.domainSearchResults[*].events[?(@.eventA | | | e | ction=="deletion")].eventDate | | | reinstantia | "$.domainSearchResults[*].events[?(@.eventA | | | tionDate | ction=="reinstantiation")].eventDate | | | transferDat | "$.domainSearchResults[*].events[?(@.eventA | | | e | ction=="transfer")].eventDate | | | lockedDate | "$.domainSearchResults[*].events[?(@.eventA | | | | ction=="locked")].eventDate | | | unlockedDat | "$.domainSearchResults[*].events[?(@.eventA | | | e | ction=="unlocked")].eventDate | | | | | | Domai | ldhName | $.domainSearchResults[*].ldhName | | n | | | | | | | | Names | ldhName | $.nameserverSearchResults[*].ldhName | | erver | | | | | ipV4 | $.nameserverSearchResults[*].ipAddresses.v4 | | | | [0] | | | ipV6 | $.nameserverSearchResults[*].ipAddresses.v6 | | | | [0] | | | | | | Entit | handle | $.entitySearchResults[*].handle | | y | | | | | fn | $.entitySearchResults[*].vcardArray[1][?(@[ | | | | 0]="fn")][3] | | | org | $.entitySearchResults[*].vcardArray[1][?(@[ | | | | 0]="org")][3] | | | voice | $.entitySearchResults[*].vcardArray[1][?(@[ | | | | 0]=="tel" && @[1].type=="voice")][3] | | | email | $.entitySearchResults[*].vcardArray[1][?(@[ | | | | 0]=="email")][3] | | | country | $.entitySearchResults[*].vcardArray[1][?(@[ | | | | 0]=="adr")][3][6] | | | city | $.entitySearchResults[*].vcardArray[1][?(@[ | | | | 0]=="adr")][3][3] | +-------+-------------+---------------------------------------------+ Table 2: Sorting properties - JSON Path Mapping If thesort"sort" parameter reports an allowed sorting property, it MUST be provided in the "currentSort" field of the "sorting_metadata" structure. https://example.com/rdap/domains?name=*nr.com&sort=ldhName https://example.com/rdap/domains?name=*nr.com&sort=registrationDate:d https://example.com/rdap/domains?name=*nr.com&sort=lockedDate,ldhName Figure 3: Examples of RDAP query reporting thesort"sort" parameter 2.3.1. Representing Sorting Links An RDAP server MAY use the "links" array of the "sorting_metadata" section to provide ready-made references [RFC8288] to the available sort criteria (Figure 4). Each link represents a reference to an alternate view of the results. { "rdapConformance": [ "rdap_level_0", "sorting_level_0" ], ... "sorting_metadata": { "currentSort": "ldhName", "availableSorts": [ { "property": "registrationDate", "jsonPath": "$.domainSearchResults[*].events[?(@.eventAction==\"registration\")].eventDate", "default": false, "links": [ { "value": "https://example.com/rdap/domains?name=*nr.com &sort=ldhName", "rel": "alternate", "href": "https://example.com/rdap/domains?name=*nr.com &sort=registrationDate", "title": "Result Ascending Sort Link", "type": "application/rdap+json" }, { "value": "https://example.com/rdap/domains?name=*nr.com &sort=ldhName", "rel": "alternate", "href": "https://example.com/rdap/domains?name=*nr.com &sort=registrationDate:d", "title": "Result Descending Sort Link", "type": "application/rdap+json" } ] }, "domainSearchResults": [ ... ] } Figure 4: Example of a "sorting_metadata" instance to implement result sorting 2.4. "limit" and "offset" Parameters An RDAP query could return a response with hundreds of objects, especially when partial matching is used. For that reason, two parameters addressing result pagination are defined to make responses easier to handle: o "limit": means that the server MUST return the first N objects of the result set in the response; o "offset": means that the server MUST skip the first N objects and MUST return objects starting from position N+1. The ABNF syntax is the following: EQ = "=" limit = "limit" EQ positive-number offset = "offset" EQ positive-number positive-number = non-zero-digit *digit non-zero-digit = "1" / "2" / "3" / "4" / "5" / "6" / "7" / "8" / "9" digit = "0" / non-zero-digit When limit and offset are used together, they allow implementation of result pagination. The following examples illustrate requests to return, respectively, the first 5 objects, the set of objects starting from position 6, and first 5 objects starting from position 11 of the result set (Figure 5). https://example.com/rdap/domains?name=*nr.com&limit=5 https://example.com/rdap/domains?name=*nr.com&offset=5 https://example.com/rdap/domains?name=*nr.com&limit=5&offset=10 Figure 5: Examples of RDAP query reporting thelimit"limit" andoffset"offset" parameters 2.4.1. RepresentingPagingOffset Pagination Links An RDAP serverMAYSHOULD use the "links" array of the "paging_metadata"sectionelement to provide a ready-made reference [RFC8288] to the next page of the result set (Figure 6). Examples of additional "rel" values a server MAY implements are "first", "last", "prev". { "rdapConformance": [ "rdap_level_0", "paging_level_0" ], ... "notices": [ { "title": "Search query limits", "type": "result set truncated due to excessive load", "description": [ "search results for domains are limited to 10" ] } ], "paging_metadata": { "totalCount": 73, "pageCount": 10,"offset": 10, "nextOffset": 20,"links": [ { "value": "https://example.com/rdap/domains?name=*nr.com", "rel": "next", "href": "https://example.com/rdap/domains?name=*nr.com&limit=10 &offset=10", "title": "Result Pagination Link", "type": "application/rdap+json" } ] }, "domainSearchResults": [ ... ] } Figure 6: Example of a "paging_metadata" instance to implement result pagination based on offset and limit3. Negative Answers2.5. "cursor" Parameter Thevalue constraints foruse of limit and offset operators represents theparametersmost common way to implement results pagination. However, when offset has a very high value, scrolling the result set could take some time. In addition, offset pagination may return inconsistent pages when data aredefined by their ABNF syntax. Therefore, each request providing an invalid valuefrequently updated (i.e. real-time data) but this is not the case of registration data. An alternative approach to offset pagination is the keyset pagination, a.k.a. seek-method [SEEK] or cursor pagination. This method has been taken as the basis for the implementation of a "cursor" parameterSHOULD obtain[CURSOR] by some REST API providers (e.g. [CURSOR-API1],[CURSOR-API2]). The cursor is anHTTP 400 (Bad Request) response code. The same response SHOULD be returned ifopaque URL-safe string representing a logical pointer to theclient provides an unsupported value forfirst result of thesort parameter in both single and multi sort. The server can provide a different response when it supportsnext page (Figure 7). https://example.com/rdap/domains?name=*nr.com&cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M= Figure 7: An example of RDAP query reporting thelimit and/or"cursor" parameter However cursor pagination raises some drawbacks with respect to offsetparameterspagination: o it needs at least one key field; o it does not allow to sort by any field and paginate theclient submits values that are out ofresults because sorting has to be made on thevalid ranges. The possible cases are:key field; oIf the client submits a value forit does not allow to skip pages because they have to be scrolled in sequential order starting from thelimit parameter that is greater thaninitial page; o it makes very hard thenumbernavigation ofobjectsthe result set in both directions because all comparison and sort operations have to beprocessed, it is RECOMMENDED that server returns a response including onlyreversed. Furthermore, in theprocessed objects.RDAP context, some additional considerations can be made: oIf the client submitsan RDAP object is avalueconceptual aggregation of information collected from more than one data structure (e.g. table) and this makes even harder for theoffset parameter that is greater thandevelopers thenumberimplementation ofobjects to be processed, it is RECOMMENDEDthe seek-method thatserver returns an HTTP 404 (Not Found) response code. Optionally,is already quite difficult. In fact, for example, theresponse MAY include additionalentity object can gather informationregarding the negative answer infrom different data structures (registrars, registrants, contacts, resellers, and so on), each one with its own key field mapping theHTTP entity body. 4.RDAPConformance Servers returningentity handle; o depending on the"paging_metadata" section in their responses MUST include "paging_level_0" innumber of therdapConformance arraypage results as well asservers returningthe"sorting_metadata" section MUST include "sorting_level_0". 5. Implementation Considerations The implementationnumber and the complexity of thenew parametersproperties of each RDAP object in the response, the time required by offset pagination to skip the previous pages could be much faster than the processing time needed to build the current page. In fact, RDAP objects are usually formed by information belonging to multiple data structures and containing multivalued properties (i.e. arrays) and, therefore, data selection istechnically feasible, as operators for counting,a time consuming process. This situation occurs even though the data selection process makes use of indexes; o depending on the access levels defined by each RDAP operator, the increase of complexity and the decrease of flexibility of cursor pagination with respect to the offset pagination could be considered impractical. Finally, the cursor pagination seems to be a bit inconsistent with some capabilities described in this document like sorting and the implementation of pagingare currently supported bylinks other than themajor RDBMSs. Inusual "next page" link. But, while thefollowing,provisioning of more paging links can be superfluous, dropping thematch betweensorting capability seems quite unreasonable. Ultimately, both pagination methods have benefits and drawbacks. Therefore, since thenew defined parameters"paging_metadata" element keeps the same structure and clients merely make use of theSQL operators is shown (Table 3): +----------------+--------------------------------------------------+ | New query | SQL operator | | parameter | | +----------------+--------------------------------------------------+ | count | count(*) query without offset, limit and order | | | by | | | [MYSQL-COUNT],[POSTGRES-COUNT],[ORACLE-COUNT] | | | | | sort | order by | | | [MYSQL-SORT],[POSTGRES-SORT],[ORACLE-SORT] | | | | | limit | limit n (in MySql [MYSQL-LIMIT] and Postgres | | | [POSTGRES-LIMIT]) | | | FETCH FIRST n ROWS ONLY (in Oracle | | | [ORACLE-LIMIT]) | | | | | offset | offset m (in Postgres) | | | OFFSET m ROWS (in Oracle) | | | | | limit + offset | limit n offset m (in MySql and Postgres) | | | OFFSET m ROWS FETCH NEXT n ROWS ONLY (in Oracle) | +----------------+--------------------------------------------------+ Table 3: New query parameters vs. SQL operators With regard to Oracle, Table 3 reports only one of the three methods that can be used to implement limit and offset parameters. The others are described in [ORACLE-ROWNUM] and [ORACLE-ROW-NUMBER]. In addition, similar operatorspaging links as they arecompletely or partially supported by the most known NoSQL databases (MongoDB, CouchDB, HBase, Cassandra, Hadoop) so the implementation of the new parameters seems to be practicable by servers working without the use of an RDBMS. 5.1. Considerations about Paging Implementation The use of limit and offsetprovided, this RDAP specification lets operatorsrepresents the most common wayto implementresults pagination. However, when offset hasahigh value, scrollingmethod according to their needs, theresult set could take some time. In addition,user access levels, the submitted queries. 2.5.1. Representing Cursor Pagination Links The same considerations made for offset paginationmay return inconsistent pages when datalinks arefrequently updated (i.e. real-time data) but this is not the case of registration data. An alternative approachapplied tooffset pagination is the keyset pagination, a.k.a. seek-method [SEEK] orcursorbased pagination. This method has been taken as the basis for the implementation of a cursor parameter [CURSOR] by some REST API providers (e.g. [CURSOR-API1],[CURSOR-API2]). The cursor parameter is an opaque URL-safe string representing a logical pointer to the first result of the next pagepagination links (Figure7).8). { "rdapConformance": [ "rdap_level_0", "paging_level_0" ], ... "notices": [ { "title": "Search query limits", "type": "result set truncated due to excessive load", "description": [ "search results for domains are limited to 10" ] } ], "paging_metadata": { "totalCount": 73, "pageCount": 10, "links": [ { "value": "https://example.com/rdap/domains?name=*nr.com", "rel": "next", "href":"https://example.com/rdap/domains?name=*nr.com&limit=10"https://example.com/rdap/domains?name=*nr.com &cursor=wJlCDLIl6KTWypN7T6vc6nWEmEYe99Hjf1XY1xmqV-M=", "title": "Result Pagination Link", "type": "application/rdap+json" } ] }, "domainSearchResults": [ ... ] } Figure7:8: Example of a"paging_metadata" instance to implement keyset pagination But keyset pagination raises some drawbacks with respect to offset pagination: o it needs at least one key field; o"paging_metadata" instance to implement cursor pagination 3. Negative Answers The value constraints for the parameters are defined by their ABNF syntax. Therefore, each request providing an invalid value for a parameter SHOULD obtain an HTTP 400 (Bad Request) response code. The same response SHOULD be returned if the client provides an unsupported value for the "sort" parameter in both single and multi sort. The server can provide a different response when itdoes not allow to sort by any fieldsupports the "limit" and/or "offset" parameters andpaginatetheresults because sorting has to be made onclient submits values that are out of thekey field;valid ranges. The possible cases are: oit does not allow to skip pages because they have to be scrolled in sequential order starting fromIf theinitial page; o it makes very hardclient submits a value for thenavigation of"limit" parameter that is greater than theresult set in both directions because all comparison and sort operations havenumber of objects to bereversed. Furthermore, inprocessed, it is RECOMMENDED that server returns a response including only theRDAP context, some additional considerations can be made:processed objects. oan RDAP object isIf the client submits aconceptual aggregation of information collected from more than one data structure (e.g. table) and this makes even hardervalue for thedevelopers"offset" parameter that is greater than theimplementationnumber of objects to be processed as well as an invalid value for theseek-method that"cursor" parameter, it isalready quite difficult. In fact, for example,RECOMMENDED that server returns an HTTP 404 (Not Found) response code. Optionally, theentity object can gatherresponse MAY include additional informationfrom different data structures (registrars, registrants, contacts, resellers, and so on), each one with its own key field mappingregarding the negative answer in theRDAPHTTP entityhandle; o depending onbody. 4. RDAP Conformance Servers returning thenumber of"paging_metadata" element in their responses MUST include "paging_level_0" in thepage resultsrdapConformance array as well as servers returning thenumber and the complexity of the properties"sorting_metadata" element MUST include "sorting_level_0". 5. Implementation Considerations The implementation ofeach RDAP object in the response,thetime requirednew parameters is technically feasible, as operators for counting, sorting and paging are currently supported byoffset pagination to skip the previous pages could be much faster than the processing time needed to buildthecurrent page.major RDBMSs. Infact, RDAP objects are usually formed by information belonging to multiple data structures and containing multivalued properties (e.g. arrays) and, therefore, data selection is a time consuming process. This situation occurs even thoughthedata selection process makes use of indexes; o depending onfollowing, theaccess levelsmatch between the new definedby each RDAP operator,parameters and theincrease of complexitySQL operators is shown (Table 3): +----------------+--------------------------------------------------+ | New query | SQL operator | | parameter | | +----------------+--------------------------------------------------+ | count | count(*) query without offset, limit and order | | | by | | | [MYSQL-COUNT],[POSTGRES-COUNT],[ORACLE-COUNT] | | | | | sort | order by | | | [MYSQL-SORT],[POSTGRES-SORT],[ORACLE-SORT] | | | | | limit | limit n (in MySql [MYSQL-LIMIT] and Postgres | | | [POSTGRES-LIMIT]) | | | FETCH FIRST n ROWS ONLY (in Oracle | | | [ORACLE-LIMIT]) | | | | | offset | offset m (in Postgres) | | | OFFSET m ROWS (in Oracle) | | | | | limit + offset | limit n offset m (in MySql andthe decrease of flexibility of keyset pagination with respectPostgres) | | | OFFSET m ROWS FETCH NEXT n ROWS ONLY (in Oracle) | +----------------+--------------------------------------------------+ Table 3: New query parameters vs. SQL operators With regard tothe offset pagination could be considered impractical. Finally, the keyset pagination is not fully compliant with the additional RDAP capabilities proposed by this document. In fact, the presenceOracle, Table 3 reports only one ofa possible cursor parameter does not seem to be consistent with both the sorting capability andthepossibilitythree methods that can be used to implementadditional ready-made links besides"limit" and "offset" parameters. The others are described in [ORACLE-ROWNUM] and [ORACLE-ROW-NUMBER]. In addition, similar operators are completely or partially supported by theclassic "next page" link. But, whilemost known NoSQL databases (MongoDB, CouchDB, HBase, Cassandra, Hadoop) so theprovisioningimplementation ofmore paging links can be superfluous, dropping the sorting capability seems quite unreasonable. If pagination is implemented by using a cursor, both "offset" and "nextOffset" fields MUST not be included inthe"paging_metadata" section. FOR DISCUSSION: Should RDAP specification reports both offset and cursornew parametersand let operators to implement pagination accordingseems totheir needs, the user access levels,be practicable by servers working without thesubmitted queries?use of an RDBMS. 6. Implementation Status NOTE: Please remove this section and the reference to RFC 7942 prior to publication as an RFC. This section records the status of known implementations of the protocol defined by this specification at the time of posting of this Internet-Draft, and is based on a proposal described in RFC 7942 [RFC7942]. The description of implementations in this section is intended to assist the IETF in its decision processes in progressing drafts to RFCs. Please note that the listing of any individual implementation here does not imply endorsement by the IETF. Furthermore, no effort has been spent to verify the information presented here that was supplied by IETF contributors. This is not intended as, and must not be construed to be, a catalog of available implementations or their features. Readers are advised to note that other implementations may exist. According to RFC 7942, "this will allow reviewers and working groups to assign due consideration to documents that have the benefit of running code, which may serve as evidence of valuable experimentation and feedback that have made the implemented protocols more mature. It is up to the individual working groups to use this information as they see fit". 6.1. IIT-CNR/Registro.it Responsible Organization: Institute of Informatics and Telematics of National Research Council (IIT-CNR)/Registro.it Location: https://rdap.pubtest.nic.it/ Description: This implementation includes support for RDAP queries using data from the public test environment of .it ccTLD. The RDAP server implements both offset and cursorbasedpagination (the latter only whensort"sort" andoffset"offset" parameters are not present in the query string). Level of Maturity: This is a "proof of concept" research implementation. Coverage: This implementation includes all of the features described in this specification. Contact Information: Mario Loffredo, mario.loffredo@iit.cnr.it 6.2. Google Registry Responsible Organization: Google Registry Location: https://www.registry.google/rdap/ Description: This implementation includes support for RDAP queries for TLDs such as .GOOGLE, .HOW, .SOY, and .xn--q9jyb4c . The RDAP server implements cursorbased pagination (the number of objects per page is fixed so the limit parameter is not available).pagination. The link used to request the next page is included in the notice section of the response. Level of Maturity: Production. Coverage: This implementation includes thecursor"cursor" parameter described in this specification. Contact Information: Brian Mountford, mountford@google.com 7. Security Considerations Security services for the operations specified in this document are described in RFC 7481 [RFC7481]. Search query typically requires more server resources (such as memory, CPU cycles, and network bandwidth) when compared to lookup query. This increases the risk of server resource exhaustion and subsequent denial of service due to abuse. This risk can be mitigated by either restricting search functionality and limiting the rate of search requests. Servers can also reduce their load by truncating the results in the response. However, this last security policy can result in a higher inefficiency if the RDAP server does not provide any functionality to return the truncated results. The new parameters presented in this document provide the RDAP operators with a way to implement a secure server without penalizing its efficiency. The "count" parameter gives the user a measure to evaluate the query precision and, at the same time, return a significant information. Thesort"sort" parameter allows the user to obtain the most relevant information at the beginning of the result set. In both cases, the user doesn't need to submit further unnecessary search requests. Finally, thelimit"limit" andoffset"offset" parameters enable the user to scroll the result set by submitting a sequence of sustainable queries according to the server limits. 8. IANA Considerations This document has no actions for IANA. 9. Acknowledgements The authors would like to acknowledge Brian Mountford for his contribution to the development of this document. 10. References 10.1. Normative References [ISO.3166.1988] International Organization for Standardization, "Codes for the representation of names of countries, 3rd edition", ISO Standard 3166, August 1988. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>. [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", RFC 5226, DOI 10.17487/RFC5226, May 2008, <https://www.rfc-editor.org/info/rfc5226>. [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, DOI 10.17487/RFC5234, January 2008, <https://www.rfc-editor.org/info/rfc5234>. [RFC6350] Perreault, S., "vCard Format Specification", RFC 6350, DOI 10.17487/RFC6350, August 2011, <https://www.rfc-editor.org/info/rfc6350>. [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing", RFC 7230, DOI 10.17487/RFC7230, June 2014, <https://www.rfc-editor.org/info/rfc7230>. [RFC7480] Newton, A., Ellacott, B., and N. Kong, "HTTP Usage in the Registration Data Access Protocol (RDAP)", RFC 7480, DOI 10.17487/RFC7480, March 2015, <https://www.rfc-editor.org/info/rfc7480>. [RFC7481] Hollenbeck, S. and N. Kong, "Security Services for the Registration Data Access Protocol (RDAP)", RFC 7481, DOI 10.17487/RFC7481, March 2015, <https://www.rfc-editor.org/info/rfc7481>. [RFC7482] Newton, A. and S. Hollenbeck, "Registration Data Access Protocol (RDAP) Query Format", RFC 7482, DOI 10.17487/RFC7482, March 2015, <https://www.rfc-editor.org/info/rfc7482>. [RFC7483] Newton, A. and S. Hollenbeck, "JSON Responses for the Registration Data Access Protocol (RDAP)", RFC 7483, DOI 10.17487/RFC7483, March 2015, <https://www.rfc-editor.org/info/rfc7483>. [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data Interchange Format", STD 90, RFC 8259, DOI 10.17487/RFC8259, December 2017, <https://www.rfc-editor.org/info/rfc8259>. [RFC8288] Nottingham, M., "Web Linking", RFC 8288, DOI 10.17487/RFC8288, October 2017, <https://www.rfc-editor.org/info/rfc8288>. 10.2. Informative References [CURSOR] Nimesh, R., "Paginating Real-Time Data with Keyset Pagination", July 2014, <https://www.sitepoint.com/ paginating-real-time-data-cursor-based-pagination/>. [CURSOR-API1] facebook.com, "facebook for developers - Using the Graph API", July 2017, <https://developers.facebook.com/docs/ graph-api/using-graph-api>. [CURSOR-API2] twitter.com, "Pagination", 2017, <https://developer.twitter.com/en/docs/ads/general/guides/ pagination.html>. [GOESSNER-JSON-PATH] Goessner, S., "JSONPath - XPath for JSON", 2007, <http://goessner.net/articles/JsonPath/>. [HATEOAS] Jedrzejewski, B., "HATEOAS - a simple explanation", 2018, <https://www.e4developer.com/2018/02/16/ hateoas-simple-explanation/>. [MYSQL-COUNT] mysql.com, "MySQL 5.7 Reference Manual, Counting Rows", October 2015, <https://dev.mysql.com/doc/refman/5.7/en/ counting-rows.html>. [MYSQL-LIMIT] mysql.com, "MySQL 5.7 Reference Manual, SELECT Syntax", October 2015, <https://dev.mysql.com/doc/refman/5.7/en/select.html>. [MYSQL-SORT] mysql.com, "MySQL 5.7 Reference Manual, Sorting Rows", October 2015, <https://dev.mysql.com/doc/refman/5.7/en/ sorting-rows.html>. [OData-Part1] Pizzo, M., Handl, R., and M. Zurmuehl, "OData Version 4.0. Part 1: Protocol Plus Errata 03", June 2016, <http://docs.oasis- open.org/odata/odata/v4.0/errata03/os/complete/part1- protocol/ odata-v4.0-errata03-os-part1-protocol-complete.pdf>. [ORACLE-COUNT] Oracle Corporation, "Database SQL Language Reference, COUNT", March 2016, <http://docs.oracle.com/database/122/SQLRF/COUNT.htm>. [ORACLE-LIMIT] Oracle Corporation, "Database SQL Language Reference, SELECT, Row limiting clause", March 2016, <http://docs.oracle.com/database/122/SQLRF/SELECT.htm>. [ORACLE-ROW-NUMBER] Oracle Corporation, "Database SQL Language Reference, SELECT, ROW_NUMBER", March 2016, <http://docs.oracle.com/database/122/SQLRF/ ROW_NUMBER.htm#SQLRF06100>. [ORACLE-ROWNUM] Oracle Corporation, "Database SQL Language Reference, SELECT, ROWNUM Pseudocolumn", March 2016, <http://docs.oracle.com/database/122/SQLRF/ ROWNUM-Pseudocolumn.htm#SQLRF00255>. [ORACLE-SORT] Oracle Corporation, "Database SQL Language Reference, SELECT, Order by clause", March 2016, <http://docs.oracle.com/database/122/SQLRF/SELECT.htm>. [POSTGRES-COUNT] postgresql.org, "PostgresSQL, Aggregate Functions", September 2016, <https://www.postgresql.org/docs/9.6/static/ functions-aggregate.html>. [POSTGRES-LIMIT] postgresql.org, "PostgresSQL, LIMIT and OFFSET", September 2016, <https://www.postgresql.org/docs/9.6/static/ queries-limit.html>. [POSTGRES-SORT] postgresql.org, "PostgresSQL, Sorting Rows", September 2016, <https://www.postgresql.org/docs/9.6/static/ queries-order.html>. [REST] Fredrich, T., "RESTful Service Best Practices, Recommendations for Creating Web Services", April 2012, <http://www.restapitutorial.com/media/ RESTful_Best_Practices-v1_1.pdf>. [RFC6901] Bryan, P., Ed., Zyp, K., and M. Nottingham, Ed., "JavaScript Object Notation (JSON) Pointer", RFC 6901, DOI 10.17487/RFC6901, April 2013, <https://www.rfc-editor.org/info/rfc6901>. [RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running Code: The Implementation Status Section", BCP 205, RFC 7942, DOI 10.17487/RFC7942, July 2016, <https://www.rfc-editor.org/info/rfc7942>. [SEEK] EverSQL.com, "Faster Pagination in Mysql - Why Order By With Limit and Offset is Slow?", July 2017, <https://www.eversql.com/faster-pagination-in-mysql-why- order-by-with-limit-and-offset-is-slow/>. [W3C.CR-xpath-31-20161213] Robie, J., Dyck, M., and J. Spiegel, "XML Path Language (XPath) 3.1", World Wide Web Consortium CR CR-xpath- 31-20161213, December 2016, <https://www.w3.org/TR/2016/CR-xpath-31-20161213>. Appendix A. Change Log 00: Initial working group version ported from draft-loffredo-regext- rdap-sorting-and-paging-05 01: Removed both "offset" and "nextOffset" to keep "paging_metadata" consistent between the pagination methods. Renamed "Considerations about Paging Implementation" section in ""cursor" Parameter". Removed "FOR DISCUSSION" items. Provided a more detailed description of both "sorting_metadata" and "paging_metadata" objects. Authors' Addresses Mario Loffredo IIT-CNR/Registro.it Via Moruzzi,1 Pisa 56124 IT Email: mario.loffredo@iit.cnr.it URI: http://www.iit.cnr.it Maurizio Martinelli IIT-CNR/Registro.it Via Moruzzi,1 Pisa 56124 IT Email: maurizio.martinelli@iit.cnr.it URI: http://www.iit.cnr.it Scott Hollenbeck Verisign Labs 12061 Bluemont Way Reston, VA 20190 USA Email: shollenbeck@verisign.com URI: https://www.verisignlabs.com/