[Docs] [txt|pdf] [Tracker] [Email] [Diff1] [Diff2] [Nits]

Versions: 00 01 02 03 04

IETF NNTP Working Group                            N. Ballou, Microsoft
Internet Draft                                    B. Hernacki, Netscape
Document: draft-ballou-nntpsrch-04.txt                  September, 1997


                    NNTP Full-text Search Extension


Status of this Memo

   This document is an Internet Draft. Internet Drafts are working
   documents of the Internet Engineering Task Force (IETF), its Areas,
   and its Working Groups. Note that other groups may also distribute
   working documents as Internet Drafts.

   Internet Drafts are draft documents valid for a maximum of six
   months. Internet Drafts may be updated, replaced, or obsoleted by
   other documents at any time. It is not appropriate to use Internet
   Drafts as reference material or to cite them other than as a
   "working draft" or "work in progress".

   To learn the current status of any Internet-Draft, please check the
   1id-abstracts.txt listing contained in the Internet-Drafts Shadow
   Directories on ds.internic.net, nic.nordu.net, ftp.isi.edu, or
   munnari.oz.au.

   A revised version of this draft document will be submitted to the
   RFC editor as a Proposed Standard for the Internet Community.
   Discussion and suggestions for improvement are requested. This
   document will expire before March 1998.  Distribution of this draft
   is unlimited.


1. Abstract

   This document describes a set of enhancements to the Network News
   Transport Protocol [NNTP-977] that allows full-text searching of
   news articles in multiple newsgroups.  The proposed SEARCH command
   supports functionality similar to the [IMAP4] SEARCH command, minus
   user specific search keys (i.e., ANSWERED, DRAFT, FLAGGED, KEYWORD,
   NEW, OLD, RECENT, SEEN) and minus search keys based on headers that
   do not exist in news (i.e., CC, BCC, TO).

   The availability of the extensions described here will be advertised
   by the server using the extension negotiation-mechanism described in
   the new NNTP protocol specification currently being developed [NNTP-
   NEW].


2. Conventions used in this document

   In examples, "C:" and "S:" indicate lines sent by the client and
   server respectively.
                    NNTP Full-text Search Extension     September 1997


   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC-2119].


3. Introduction

   The NNTP SEARCH command is sent from the client to the server to
   specify and initiate a full-text search on articles in one or more
   newsgroups. The NNTP SEARCH command is similar to the [IMAP4] SEARCH
   command, with user property and mail-specific header search keys not
   present in NNTP SEARCH.  The results of an NNTP Search is OVER data
   as specified in [NNTP-NEW] for each article that satisfies the
   search criteria.

   In addition, the PAT command is extended so that it can be used to
   full-text search articles within a single newsgroup. Both the
   headers and the body of the articles are searched.


3.1. New and Enhanced NNTP Commands

   There are four new NNTP commands: two new options to the existing
   LIST command, and enhancements to one existing command.

   *  SEARCH
   *  LIST SRCHFIELDS
   *  LIST SEARCHABLE
   *  PAT

   The SEARCH command runs a one-time search, returning overview-like
   data.

   The LIST SRCHFIELDS command returns the fields that the server
   allows in full-text searches.

   The LIST SEARCHABLE command allows the client to determine which
   newsgroups are full-text searchable.

   The PAT command allows the pseudo-header ":TEXT".  This specifies a
   full-text (headers and body) search of the articles in a single
   newsgroup.


4. Use of NNTP Extension Mechanism

   The NNTP extension mechanism allows a server to describe its
   capabilities. The following extensions are used to describe the
   capabilities described in this document.


4.1. SEARCH Extension
                    NNTP Full-text Search Extension     September 1997


   The SEARCH extension means that the server supports the following
   commands: SEARCH, LIST SEARCHABLE, LIST SRCHFIELDS.


4.2. PATTEXT Extension

   The PATTEXT extension means that the server supports the :TEXT
   header in the PAT command, as described by this document.


5. SEARCH Command

   Arguments: optional newsgroup specification
              searching criteria (one or more)

   Responses: 224 overview information follows
              412 no news group selected
              462 error performing search
              480 authentication required
              501 command syntax error
              502 no permission

   The SEARCH command searches the newsgroups for articles that match
   the given searching criteria.  Searching criteria consist of one or
   more search keys. If there are articles that match the search
   criteria, the server responds with code 224 and returns OVER data
   for each matching article in a similar format as described in [NNTP-
   NEW] with one exception. The one change from [NNTP-NEW] OVER format
   is to change the article number field to a format that supports
   searches over multiple newsgroups. The article ID field for SEARCH
   OVER data will use the format newsgroup:art-ID rather than just an
   article number as defined in [NNTP-NEW] (note: this is the same
   format used by the Xref header).

   A response of 501 indicates a syntax error in the search criteria. A
   response of 502 indicates that the user does not have permission to
   search one or more of the specified newsgroups. If the search
   criteria did not specify a newsgroup, and there is no current
   newsgroup (i.e., set using the NNTP GROUP command), then the server
   returns the error code 412, indicating that no newsgroup has been
   specified. A response of 462 indicates that the server encountered
   an error when processing the search.

   When multiple keys are specified, the result is the intersection
   (AND function) of all the messages that match those keys. For
   example, the criteria FROM "SMITH" SINCE 1-Feb-1994 refers to all
   articles from Smith that were placed in the newsgroup since February
   1, 1994. A search key may also be a parenthesized list of one or
   more search keys (e.g. for use with the OR and NOT keys).

   Server implementations MAY exclude [MIME-1] body parts with terminal
   content types other than TEXT and MESSAGE from consideration in
   SEARCH matching.
                    NNTP Full-text Search Extension     September 1997



   The optional newsgroup specification consists of the word "IN"
   followed by either a wildcard character "*" - indicating a search
   over all newsgroups - or a list of newsgroup names separated by a
   comma. A newsgroup name can end with the wildcard string ".*"
   indicating a search over a sub-hierarchy of the newsgroup name
   space. If no newsgroup specification is given, the search is over
   the current newsgroup. If there is no current newsgroup, the server
   returns the 412 error code.

   The ON, BEFORE, and SINCE search criteria use the same date as used
   in the NNTP NEWNEWS command in [NNTP-NEW] - the date the article
   arrived on the server. A server indicates support for the ON,
   BEFORE, and SINCE search criteria by listing :Date in the LIST
   SRCHFIELDS response.

   The defined search keys are as follows.  Refer to the Formal Syntax
   section for the precise syntactic definitions of the arguments.

      <message range> Articles with article numbers corresponding to
                      the specified range.

      ALL             All Articles in the current newsgroup; the
                      default initial key for ANDing.

      BEFORE <date>   Articles whose server arrival date is earlier
                      than the specified date.

      BODY <string>   Articles that contain the specified string in
                      the body of the message.

      FROM <string>   Articles that contain the specified string in
                      the article structure's FROM field.

      HEADER <field-name> <string>
                      Articles that have a header with the specified
                      field-name (as defined in [RFC-822]) and that
                      contains the specified string in the [RFC-822]
                      field-body.
      LARGER <n>      Articles with an size larger than the specified
                      number of octets.

      NOT <search-key>
                      Articles that do not match the specified search
                      key.

      ON <date>       Articles whose server arrival date is within
                      the specified date.

      OR <search-key1> <search-key2>
                      Articles that match either search key.

      SENTBEFORE <date>
                    NNTP Full-text Search Extension     September 1997


                      Articles whose [RFC-822] Date: header is
                      earlier than the specified date.

      SENTON <date>   Articles whose [RFC-822] Date: header is within
                      the specified date.

      SENTSINCE <date>
                      Articles whose [RFC-822] Date: header is within
                      or later than the specified date.

      SINCE <date>    Articles whose server arrival date is within or
                      later than the specified date.

      SMALLER <n>     Articles with a size smaller than the specified
                      number of octets.

      SUBJECT <string>
                      Articles that contain the specified string in
                      the envelope structure's SUBJECT field.

      TEXT <string>   Articles that contain the specified string in
                      the header or body of the message.

      Example: C: SEARCH FROM "Smith" SINCE 1-Feb-1994
               S: 224 overview information follows
               S: comp.object:573 \t RE: object-oriented langs \t \
                  "John Smith" <JSmith@xyz.com> \t Sun, 03 Nov 1996 \
                  14:25:05 -0800 \t <01cbc9d5f3c70$eab9a2cd@xyz.com> \
                  \t 4080 \t 33
               S: .

      Note: each field in OVER response is separated by a tab - shown
            as a \t in the example above.


5.1.1. Search Formal Syntax

   The search query syntax is derived from the search syntax defined
   for the IMAP4 protocol. It is somewhat different because of the way
   international character sets need to be encoded.

   The following syntax specification uses the augmented Backus-Naur
   Form (BNF) as described in [ABNF].

   Except as noted otherwise, all alphabetic characters are case-
   insensitive. The use of upper or lower case characters to define
   token strings is for editorial clarity only. Implementations MUST
   accept these strings in a case-insensitive fashion.

     astring       ::= atom / string

     atom          ::= 1*ATOM_CHAR
                    NNTP Full-text Search Extension     September 1997


     ATOM_CHAR     ::= <any CHAR except atom_specials>

     atom_specials ::= "," / "(" / ")" / SPACE / CTL / "*" /
                       quoted_specials

     CHAR          ::= <any ASCII character except NUL,
                        0x01 - 0x7f>

     CTL           ::= <any ASCII control character and DEL,
                        0x00 - 0x1f, 0x7f>

     date          ::= date_text / <"> date_text <">

     date_day      ::= 1*2digit
                   ;; Day of month

     date_month    ::= "Jan" / "Feb" / "Mar" / "Apr" / "May" /
                       "Jun" / "Jul" / "Aug" / "Sep" / "Oct" /
                       "Nov" / "Dec"

     date_text     ::= date_day "-" date_month "-" date_year

     date_year     ::= 4digit

     digit         ::= "0" / digit_nz

     digit_nz      ::= "1" / "2" / "3" / "4" / "5" / "6" / "7" /
                       "8" / "9"

     header_fld_name ::= sstring

     mstring       ::= A MIME encoded string surrounded by double
                       quotes

     newsgroup     ::= atom [ ".*"]

     newsgroups    ::= "*" / newsgroup_list

     newsgroup_list ::= newsgroup [ "," newsgroup_list]

     number        ::= 1*digit
                   ;; Unsigned 32-bit integer
                   ;; (0 <= n < 4,294,967,296)

     nz_number     ::= digit_nz *digit
                   ;; Non-zero unsigned 32-bit integer
                   ;; (0 < n < 4,294,967,296)

     QUOTED_CHAR   ::= <any TEXT_CHAR except quoted_specials> /
                   "\" quoted_specials

     quoted_specials ::= <"> / "\"
                    NNTP Full-text Search Extension     September 1997


     range         ::= nz_number / nz_number "-" [ nz_number ]
                   ;; Identifies a range of Articles.

     search        ::= "SEARCH" SPACE
                       ["IN" SPACE newsgroups SPACE]
                       1#search_key

     search_key    ::= "ALL" / "BODY" SPACE sstring /
                       "FROM" SPACE sstring / "ON" SPACE date /
                       "SINCE" SPACE date / "BEFORE" SPACE date /
                       "SUBJECT" SPACE sstring / "TEXT" SPACE sstring /
                       "HEADER" SPACE header_fld_name SPACE sstring /
                       "LARGER" SPACE number / "NOT" SPACE search_key /
                       "OR" SPACE search_key SPACE search_key /
                       "SENTBEFORE" SPACE date / "SENTON" SPACE date /
                       "SENTSINCE" SPACE date /
                       "SMALLER" SPACE number / range /
                       "(" 1#search_key ")"

     SPACE         ::= 1*<ASCII SP, space, 0x20>

     sstring       ::= astring / mstring

     string        ::= <"> *QUOTED_CHAR <">

     TEXT_CHAR     ::= <any CHAR except CR and LF>


5.2. LIST SRCHFIELDS Command

   Arguments: none

   Responses: 224 data follws

   The LIST SRCHFIELDS command returns a list of which fields can be
   specified in full-text search queries on the server. The response is
   a list of searchable fields, one per line. A "." on its own line
   terminates the list.  The fields are either newsgroup headers, or
   non-header fields supported by the query syntax.

   The three currently defined non-header fields are ":Body", ":Text",
   and ":Date". ":Text" means all the searchable text in the article,
   and indicates that the "TEXT" keyword is supported in the search
   query language. ":Body" means the body of the article, excluding the
   headers, and indicates that the "BODY" keyword is supported in the
   search query language. ":Date" means the date at which an article
   arrived on a server - similar to the date used in the NNTP NEWNEWS
   command - and indicates that the "ON", "SINCE", and "BEFORE"
   keywords are supported in the search query language.

   The "TEXT" and "BODY" search query fields are optional, but the
   server must indicate whether they are supported or not in the LIST
   SRCHFIELDS response.
                    NNTP Full-text Search Extension     September 1997



     Example: C: LIST SRCHFIELDS
              S: 224 Data follows.
              S: From
              S: Date
              S: Subject
              S: :Text
              S: .


5.3. LIST SEARCHABLE Command

   Arguments: none

   Responses: 224 Data Follows

   The LIST SEARCHABLE command returns a list of strings that define
   which new groups are being indexed by the news server and are thus
   available for searching. In addition, the character sets allowed for
   each group is returned.

   When there are newsgroups indexed it will return 224, followed by
   each portion of the tree that is indexed. If all groups are indexed,
   a line with "*" is returned. If only some parts of the newsgroup
   hierarchy are indexed, they are identified in the form <indexed-
   hierarchy>.*. Clients should not assume that these will always be
   top level hierarchies.  A "." on its own line terminates the list.

     Example: C: LIST SEARCHABLE
              S: 224 Data follows.
              S: alt.*
              S: comp.lang.*
              S: mcom.*
              S: .


5.4. PAT Command Enhancement

   Arguments: header range|<message-id> [pat [pat...]]

   Responses: <same as PAT - see [NNTP-NEW]>

   The PAT command is enhanced in a simple way: The new value ":TEXT"
   will be supported as a header when invoking the command. The :TEXT
   header requests a full-text search the body and all headers of the
   specified articles. Other than adding a new header name, the PAT
   command arguments are the same as specified in [NNTP-NEW].

   If :TEXT isn't specified as the header, the response is the same as
   it always has been for PAT, with each result line containing the
   article number and the value of the header that matched the pattern.
                    NNTP Full-text Search Extension     September 1997


   If the :TEXT header is specified, the constant string "TEXT" is
   returned in place of the value of the header that matched the
   pattern.

    Example: C: PAT :TEXT 1000-2000 searchtext
             S: 221 Header follows
             S: 1021 TEXT
             S: 1024 TEXT
             S:.


6. Security Considerations

   The search commands must be implemented in a way that does not allow
   access to articles in newsgroups that a client is otherwise
   restricted from reading due to access control rules.


9. References

   [ABNF], DRUMS working group, Dave Crocker Editor, "Augmented BNF for
   Syntax Specifications: ABNF", draft-drums-abnf-02.txt (work in
   progress), Internet Mail Consortium, April 1997

   [IMAP4] IMAP4 INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1. M
   Crispin, Request for Comment (RFC) 2060, December 1994

   [MIME-1] Borenstein N., and N. Freed, MIME (Multipurpose Internet
   Mail Extensions) Part One: Format of Internet Message Bodies,
   Request for Comment (RFC) 2045, December 1996.

   [NNTP-977] Network News Transfer Protocol. B. Kantor, Phil Lapsley,
   Request for Comment (RFC) 977, February 1986.

   [NNTP-NEW] Network News Transfer Protocol. S. Barber INTERNET DRAFT,
   draft-ietf-nntpext-base-02.txt, September 1997.

   [RFC-2119], Bradner, S, "Key words for use in RFCs to Indicate
   Requirement Levels", RFC 2119, Harvard University, March 1997

10.  Acknowledgments

   TBD

11. Author's Addresses

   Nathaniel Ballou
   Microsoft
   One Microsoft Way
   Redmond, WA 98052
   Phone: +1 425-703-0574
   Email: NatBa@Microsoft.com
                    NNTP Full-text Search Extension     September 1997


   Brian Hernacki
   Netscape Communications
   501 E. Middlefield Rd.
   Mountain View, CA 94043
   Phone: (650) 937-6738
   Email: bhern@netscape.com


Html markup produced by rfcmarkup 1.129d, available from https://tools.ietf.org/tools/rfcmarkup/