[Docs] [txt|pdf] [Tracker] [Email] [Nits]

Versions: 00 draft-ietf-avt-rtp-jpeg2000

INTERNET-DRAFT                                              Eric Edwards
draft-edwards-avt-rtp-jpeg2000-00.txt                    Satoshi Futemma
                                                        Eisaburo Itakura
                                                       Takahiro Fukuhara
                                                        Sony Corporation
                                                       November 14, 2001
                                                    Expires: May 13 2002


             RTP Payload Format for JPEG 2000 Video Streams

Status of this memo

    This document is an Internet-Draft and is in subject to all
    provisions of Section 10 of RFC2026.

    Internet-Drafts are working documents of the Internet Engineering
    Task Force (IETF), its areas, and its working groups. Note that
    other groups may also distribute working documents as
    Internet-Drafts.

    Internet-Drafts are draft documents valid for a maximum of six
    months and may be updated, replaced, or obsoleted by other documents
    at any time. It is inappropriate to use Internet-Drafts as reference
    materials or to cite them other than as "work in progress."

    The list of current Internet-Drafts can be accessed at
    http://www.ietf.org/1id-abstracts.html

    The list of Internet-Drafts Shadow Directories can be accessed at
    http://www.ietf.org/shadow.html.


Abstract

    This document describes a payload format for transporting JPEG 2000
    video streams using RTP (Real-time Transport Protocol).
    JPEG 2000 video streams are formed as a continuous series of JPEG
    2000 still images which is next-generation still image coding.
    The JPEG 2000 payload format described in this document has three
    features: (1) Improvement of robustness to packet loss by
    fragmenting JPEG 2000 packet units intelligently, (2) Persistency of
    main header to minimize loss effect, (3) Priority information field
    for scalable delivery from the same codestream.
    These will allow the scalability and robustness of JPEG 2000 to be
    maximized in streaming applications.


1. Introduction

    This document specifies payload formats for JPEG 2000 video streams
    over the Real-time Transport Protocol (RTP).
    JPEG-2000 is the international standardization system for
    next-generation still image encoding and its basic encoding
    technology is described in [1].

Edwards, et al.                                                 [Page 1]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001


    In JPEG 2000 part 3, Motion JPEG 2000 is defined[2]. However, this
    defines only the file format but not the transmission format for
    streaming on the Internet. For this reason, it is necessary to
    define the RTP format for JPEG 2000 video streams.

    JPEG 2000 supports many features over the current JPEG
    standard[3][4][5].

        o Higher compression efficiency than JPEG with less visual loss
          especially at bit rates less than 0.25bpp for grayscale images.

        o A single codestream that offers both lossy and superior
          lossless compression.

        o Transmission over noisy environments.  The JPEG 2000
          codestream can be built with markers to boost its error
          resilience and recovery.  The JPEG 2000 codestream is very
          robust to bit errors as it has been designed to avoid
          catastrophic decoding failure due to bit errors.

        o Progressive transmission by pixel accuracy and resolution:
          Progressive transmission that allows images to be
          reconstructed with increasing pixel accuracy or spatial
          resolution is essential for many applications.  This feature
          allows the reconstruction of images with different resolutions
          and pixel accuracy, as needed or desired, for different target
          devices. The image architecture provides for the efficient
          delivery of image data in many applications such as
          client/server applications.

        o Random codestream access and processing.  There are parts of
          an image which maybe more important than others.  Specific
          regions of the codestream can be defined to be less distorted
          than other areas.  Access to any specific area of an image is
          handled efficiently without the need to completely decompress
          the codestream.  Simple image transforms (rotating,
          translation, filtering) can be done with compressed
          codestream.


    First, the JPEG 2000 algorithm is briefly explained below.
    Fig. 1 shows a block diagram of JPEG 2000  encoder.

                                                  +-----+
                                                  | ROI |
                                                  +-----+
                                                     |
                                                     V
                    +----------+   +----------+   +------------+
                    |DC, comp. |   | Wavelet  |   |            |
        raw image==>|transform-|==>|transform-|==>|Quantization|==+
                    |  ation   |   |  ation   |   |            |  |
                    +----------+   +----------+   +------------+  |

Edwards, et al.                                                 [Page 2]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001

                                                                  |
                 +-------------+   +----------+   +------------+  |
                 |             |   |          |   |            |  |
    JPEG 2000 <==|Data ordering|<==|Arithmetic|<==|Coefficient |<=+
    codestream   |             |   |  coding  |   |bit modeling|
                 +-------------+   +----------+   +------------+

             Fig. 1: Block diagram of the JPEG2000 encoder


       First, the image will go through component separation, if it is a
    color image.  Split into RGB, YUV, or various other colorspaces.  It
    can also further be sectioned into tiles within the image for
    processing.

    Each color component or tile is transformed into the wavelet
    coefficients.  The component or tile is sampled into various levels
    usually subsampled vertically and horizontally from high frequencies
    (which contains all the sharp details) to the low frequencies (which
    contains all the flat areas).  These wavelet coefficients are
    categorized into different frequencies called subbands.  Subband HH
    has the high frequency information, then HL and LH are the contains
    the middle frequencies, and the lowest frequencies and most
    important coefficients are in the LL subband.

    Quantization is performed on the coefficients within each subband.
    The wavelet coefficient is divided by the quantization step size and
    the result is truncated.  This can happen iteratively to produce an
    accurate target bitrate.

    After quantization, code-blocks are formed from within the precincts
    within the tiles.  Precincts are a finer separation than tiles and
    code-blocks are the smallest separation of the image data.  Entropy
    coding is performed within each code-block and arithmetically
    encoded by bitplane.  There are 3 passes for the code-block:
    significance propagation pass, magnitude refinement pass, and
    cleanup pass.

    After the coefficients of all code-blocks have been coded into a
    short bitstream, a header is added turning it into a packet.  The
    header has all the information needed to decompress the packet into
    code-blocks.  A group of packets is called layers.

    For additional features in transmitting, a re-ordering of the formed
    packets is necessary.  The standard has four ways to transmit and
    decode a compressed image by: resolution, quality, location, or
    component.  As there are many markers builtin to the codestream of
    JPEG 2000, a parser can go through the bitstream and get the proper
    order of packets to transmit and decode.

    This is only to serve as an introduction to JPEG 2000 to aid in
    understanding the rest of this document.  Further details of the
    encoder can be found in various texts on JPEG 2000.


Edwards, et al.                                                 [Page 3]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001

    To decompress a JPEG 2000 codestream, one would follow the reverse
    order of the encoding order, minus the quantization step.  It is
    outside the scope of this document to describe in detail this
    procedure.  Please refer to various JPEG 2000 texts for details.


2. JPEG 2000 video features

    As described above, JPEG 2000 has the following features.

        o Higher compression efficiency than existing JPEG and yet less
          SNR deterioration
          (improved compression efficiency over JPEG with dramatic
           improvements at low bitrates)

        o Random codestream access and processing

        o Both lossless and compression and lossy compression can be
          performed by the same algorithm.

        o Optional spatial resolution and SNR progressive can be easily
          taken out from a single codestream.
          (NOTE)SNR means Signal to Noise Ratio. This is the factor to
                define the quality.

        o Parts of an image can have more bits for more detail. (ROI
          (Region of Interest) function)

        o Various levels of error resilience functionality.


    JPEG 2000 video streams are formed as a continuous series of JPEG
    2000 still images, so the above features of JPEG 2000 can be used
    effectively. JPEG 2000 video stream has the following merits.

        o SNR is improved at a low bit rate.  The formation can be used
          as a video stream format at a low band.

        o This is a Full Intra format in which each frame is
          independently compressed has a low encoding and decoding
          delay. This is suitable for interactive video communication.
          Even if a packet loss occurs in any part  of the frame, error
          is not propagated to subsequent frames.  Moreover, each frame
          can be handled independently this facilitates video editing.

        o JPEG 2000 has  flexible and accurate rate control.  This is
          suitable for traffic control and congestion control at the
          Internet transmission.

        o JPEG 2000 can provide within its own codestream error
          resilience markers to aid in codestream recovery.  An encoder
          can insert a resynchronization marker at the beginning of a
          JPEG 2000 packet and a segmentation symbol at the end of the
          bit plane to aid in recovery within a frame.

Edwards, et al.                                                 [Page 4]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001



3. Requirements for RTP payload format of JPEG 2000 video streams

    To provide a payload format that makes the most of the merits of
    JPEG 2000 video stream, described in the previous section, the
    following must be taken into consideration.

    - Provisions for packet loss

        On the Internet, 5% packet loss is common and this percentage
        may sometimes come to 20% or more.  To split JPEG 2000 video
        streams into RTP packets, efficient packetization of the
        codestream is required to minimize the effects of disabled
        decoding due to missing code-blocks over error prone
        environments.  If the main header is lost in transmission, the
        decoding ability is lost.  Accordingly, a system to compensate
        for the loss of the main header as much as possible is required.

    - A packetizing scheme that permits making the most of the JPEG 2000
      functionality.

        A packetizing scheme so that an image can be progressively
        transmitted and reconstructed progressively by the receiver using
        JPEG 2000 functionality.  Maximizing performance over various
        network conditions and various computing power of receiving
        platforms.


4. Proposal for an RTP payload format for JPEG 2000 video streams

4.1 RTP fixed header usage

    For each RTP packet, the RTP fixed header is followed by the JPEG
    2000 payload header, which is followed by JPEG 2000 codestream.
    The RTP header fields that have a meaning specific to the JPEG 2000
    video are described as follows:

    Payload type (PT): The payload type is dynamically assigned by means
    outside the scope of this document. A payload type in the dynamic
    range SHALL be chosen by means of an out of band signaling
    protocol (e.g., RTSP, SIP, etc).

    Marker bit (M): The marker bit of the RTP fixed header is set to 1
    on the last RTP packet of a video frame, and otherwise, must be 0.
    When transmission is performed by multiple RTP sessions, the bit is
    set in the last packet of the frame in each session.

    Timestamp: The RTP timestamp is in units of 90 KHz. The same
    timestamp must appear in each fragment of a given frame. The
    initial value of the timestamp is random (unpredictable) to make
    known-plaintext attacks on encryption more difficult, even if the
    source itself does not encrypt, because the packets may flow
    through a translator that does.

Edwards, et al.                                                 [Page 5]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001



4.2 RTP Payload header format

    The RTP payload header format for JPEG 2000 video stream is as
    follows:

     0               1               2               3
     0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |     type      | type-specific |   priority    |X|rsvd | mh_id |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                        fragment offset                        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

            Fig. 2:  RTP payload header format for JPEG 2000

    type : 8 bits

        The type field shows which part of JPEG 2000 codestream is
        included. The details of the type are described later.

    type-specific : 8 bits

        Interpretation depends on the value of the type field.
        This field is defined for future usage.
        This bit must be set to 0 when not used.
        i.e. Tile specific priority number (general idea)

    priority : 8 bits

        The priority field shows the importance of the JPEG 2000
        packet included in the given RTP packet. Typically, the higher
        priority is set at the packet which contains the JPEG 2000
        packets of the lower layers and the lower subbands.

    X : 1 bit

        extension bit. This bit must be set to 1 when
        JPEG 2000 optional payload header follows the JPEG 2000
        payload header, and otherwise set to 0.
        The details of the optional payload header is described later.

    rsvd : 3 bits

        These bits are reserved for future use and must be set to 0.

    mh_id : 4 bit

        identification of the main header of JPEG 2000. The same mh_id
        is used as long as the coding parameters described in the main
        header remain unchanged.

    fragment offset : 32 bits

Edwards, et al.                                                 [Page 6]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001


        Because JPEG 2000 frames are typically larger than
        underlying network's maximum transfer units (MTU), frames may
        often be fragmented into several packets. The fragment offset is
        the data offset in bytes of the current packet from the first
        byte in the JPEG 2000 codestream. This field helps the receiver
        to reassemble JPEG 2000 codestream.

        To perform scalable video delivery by using multiple RTP
        sessions, the offset value from the first byte of the same frame
        is set for fragment offset.  Accordingly, in scalable video
        delivery using multiple RTP sessions, maybe the fragment offset
        will not be started with 0 in some RTP sessions even if the
        packet is the first one of the frame.


5. Fragmentation of JPEG 2000 codestream and Type Field

    Fig. 2 shows the construction of the JPEG 2000 codestream.  The JPEG
    2000 codestream consists of a main header beginning with the SOC
    marker, one or more tiles (only one tile for no tile division), and
    the EOC marker to indicate the end of the codesteam.  Each tile
    consists of a tile-part header starts with the SOT marker and ending
    with the SOD marker, and a bit stream (a series of JPEG 2000
    packets) of the bit stream.

         +--  +------------+
   Main  |    |    SOC     |  Required as the first marker.
   header|    +------------+
         |    |    main    |  Main header marker segments
         +--  +------------+
         |    |    SOT     |  Required at the beginning of each tile-part
   Tile- |    +------------+  header.
   part  |    |   T0,TP0   |  Tile 0, tile-part 0 header marker segments
   header|    +------------+
         |    |    SOD     |  Required at the end of each tile-part header
         +--  +------------+
              | bit stream |  Tile-part bit stream.
         +--  +------------+  Might include SOP and EPH
         |    |    SOT     |
   Tile- |    +------------+
   part  |    |   T1,TP0   |
   header|    +------------+
         |    |    SOD     |
         +--  +------------+
              | bit stream |
              +------------+
              |    EOC     |  Required as the last marker in the codestream
              +------------+

            Fig. 3: Construction of the JPEG 2000 codestream


    JPEG 2000 video streams are typically larger than underlying

Edwards, et al.                                                 [Page 7]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001

    network's maximum transfer units (MTU), video sequence may often be
    fragmented into several IP packets at the network layer. the JPEG
    2000 video streams are fragmented into RTP packets according to the
    following basic rule.

    The JPEG 2000 construction consists of a main header, tile-part
    headers, and JPEG 2000 packets.  When we packetize the JPEG 2000
    codestream, these construction units from the codestream should be
    maintained.  Each RTP packet should consist of a main header,
    tile-part header, or JPEG 2000 packet.

    If the sender understands JPEG 2000 codestream and can read the JPEG
    2000 packets from the codestream.  (i.e. the sender is intelligent)
    JPEG 2000 packets should be packed into RTP payload packets in the
    following way:

        1. If the JPEG 2000 packets are smaller than the MTU size, the
           sender should put as many whole JPEG 2000 packets into a
           single RTP packet.  That is, the JPEG 2000 payload data
           should begin with one of the SOC marker, SOT marker, or SOP
           marker (if it exists).

        2. If the JPEG 2000 packets are larger than the MTU size, the
           sender should segment the JPEG 2000 packets at the largest
           possible MTU size but without JPEG 2000 packets overlapping.

    If the server does not understand JPEG 2000 codestream (i.e. the
    sender is not intelligent,) it should pack JPEG 2000 codestream in
    the largest possible MTU data size for the RTP packet.  JPEG 2000
    codestream will be segmented along arbitrary lengths by the sender
    into RTP packets.

    Regardless of the sender's capabilities, the receiver must be able
    to handle RTP packets of any size.

    If we do not fragment at the sender, any packets larger than the MTU
    size, will be fragmented into multiple smaller IP packets than the
    MTU size by the IP layer.  If one fragmented IP packet is lost
    during transmission, it is recognized as a loss of the whole RTP
    packet because the receiving host cannot reassemble the RTP packet.
    The segmentation of the JPEG 2000 codestream into RTP packets,
    should fit within the RTP payload size.

    In the following, all the possible packetization cases are described
    with diagrams.  For each case, the type field value shown in Fig. 2
    is also indicated.


5.1 Separation at arbitrary lengths

    In this case, a JPEG 2000 codestream is split into several
    fragments at arbitrary byte-position.
    The type value of the RTP packet is set to 0.


Edwards, et al.                                                 [Page 8]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001

    +---+---+---+----------------------+
    |RTP|PL |SOC| jpeg 2000 codestream |        type = 0
    |hdr|hdr|   | fragment (1)         |
    +---+---+---+----------------------+
    +---+---+--------------------------+
    |RTP|PL |   jpeg 2000 codestream   |        type = 0
    |hdr|hdr|   fragment (2)           |
    +---+---+--------------------------+
                ...
    +---+---+----------------------+---+
    |RTP|PL | jpeg 2000 codestream |EOC|        type = 0
    |hdr|hdr| fragment (N)         |   |
    +---+---+----------------------+---+
    *PL hdr = payload header


    Such RTP packetization scheme is not recommended from the standpoint
    of error resilience.  It is desirable to use it only in some limited
    environments shown below.

        - The sender finds it difficult to distinguish the main header,
          tile header, and JPEG 2000 packets from one another.  There is
          no SOP marker in the JPEG 2000 codestream.  The sender is not
          intelligent.

        - The network environment is error free.

        - If the JPEG 2000 error resilience markers (TLM, PLM, PLT, PPM,
          and PPT markers) are present in the codestream.  Error
          resilience will be handled outside of RTP.  Its description is
          not within the scope of this document.  Using these markers
          may improve the error resilience.



5.2  General JPEG 2000 RTP packet types

   (1) JPEG 2000 main header(SOC marker) must come first of the RTP
       payload (just after the RTP payload header). The type value of
       the RTP packets which contain the whole main header (not
       fragmented) is 4,

       (1-a) The RTP packet only contains the complete main header.
       +---+---+------+
       |RTP|PL |Main  |                         type = 4
       |hdr|hdr|header|
       +---+---+------+

       (1-b) The main header and the first tile-part header are packed
       into one RTP packet.
       +---+---+------+---------+
       |RTP|PL |Main  |Tile-part|               type = 4
       |hdr|hdr|header|header   |
       +---+---+------+---------+

Edwards, et al.                                                 [Page 9]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001


       (1-c) The main header, the first tile-part header and JPEG 2000
       packet(s) are packed into one RTP packet.
       +---+---+------+---------+---------+-----+---------+
       |RTP|PL |Main  |Tile-part|jpeg 2000| ... |jpeg 2000|     type = 4
       |hdr|hdr|header|header   |packet   |     |packet   |
       +---+---+------+---------+---------+-----+---------+


       (1-d) The main header is split into the several RTP packets.

       If the main header is larger than one RTP packet,  then it may be
       split into several RTP packets.
       In this case, the RTP packets must contain only a piece of the
       main header. The type value of the RTP packets which contain the
       first piece of the main header is type 5, and the last piece is
       type 7 and the middle pieces are all type 6.

       +---+---+--------------+
       |RTP|PL |Main Header(1)|                 type = 5
       |hdr|hdr|              |
       +---+---+--------------+
       +---+---+--------------+
       |RTP|PL |Main Header(2)|                 type = 6
       |hdr|hdr|              |
       +---+---+--------------+
       +---+---+--------------+
       |RTP|PL |Main Header(3)|                 type = 6
       |hdr|hdr|              |
       +---+---+--------------+
                 ...                              ...
       +---+---+--------------+
       |RTP|PL |Main Header(N)|                 type = 7
       |hdr|hdr|              |
       +---+---+--------------+


       (Note) When the main header is split into multiple RTP packets,
       the first  tile-part header must not be included in the RTP
       packet containing the last fragment.
       +---+---+--------------+---------+
       |RTP|PL |Main Header(N)|Tile-part|       This packetization is
       |hdr|hdr|              |header   |       not allowed.
       +---+---+--------------+---------+



   (2) Tile-part headers (SOT marker) must come first of the RTP payload
       (just after the RTP payload header), except for the first
       tile-part header just after the main header.
       The first tile-part header may either be packed with the main
       header, or be separated to another RTP packet.
       The type value of the RTP packet which begins with the
       tile-part header is 8.

Edwards, et al.                                                [Page 10]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001


       (2-a) The RTP packet only contains the complete tile-part header.
       +---+---+----------+
       |RTP|PL |Tile-part |                     type = 8
       |hdr|hdr|Header    |
       +---+---+----------+

       (2-b) The tile-part header and JPEG 2000 packet(s) are packed
       into one RTP packet.
       +---+---+----------+---------+-----+---------+
       |RTP|PL |Tile-part |jpeg 2000| ... |jpeg 2000|   type = 8
       |hdr|hdr|Header    |packet   |     |         |
       +---+---+----------+---------+-----+---------+


       (2-c) The tile-part header is split into the several RTP
       packets.

       If the tile-part header is larger than one RTP
       packet, it may be split into several RTP packets. In this
       case, the RTP packets contain only a piece of the tile-part
       header.
       The RTP packets which contain the first piece of the tile-part
       header is type 9, and the last piece is type 11, and the middle
       pieces are all type 10.

       +---+---+-------------------+
       |RTP|PL |Tile-part header   |            type = 9
       |hdr|hdr|fragment(1)        |
       +---+---+-------------------+
       +---+---+-------------------+
       |RTP|PL |Tile-part header   |            type = 10
       |hdr|hdr|fragment(2)        |
       +---+---+-------------------+
       +---+---+-------------------+
       |RTP|PL |Tile-part header   |            type = 10
       |hdr|hdr|fragment(3)        |
       +---+---+-------------------+
                 ...
       +---+---+-------------------+
       |RTP|PL |Tile-part header   |            type = 11
       |hdr|hdr|fragment(N)        |
       +---+---+-------------------+

       (Note) When the tile-part header is split into multiple RTP
       packets, the JPEG 2000 packet must not be included in the RTP
       packet containing the last fragment.
       +---+---+-------------------+---------+
       |RTP|PL |Tile-part header   |jpeg 2000|  This packetization is
       |hdr|hdr|fragment(N)        |packet   |  not allowed.
       +---+---+-------------------+---------+




Edwards, et al.                                                [Page 11]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001

   (3) The JPEG 2000 packet must be packed by itself, except for JPEG
       2000 packets just after the tile-part header. Also several JPEG
       2000 packets may be packed into the one RTP packet.
       If SOP(Start of Packet) marker is used for error resilience,
       SOP marker shall be placed at the beginning of the RTP payload.
       (When the SOP marker is used, it is placed at the beginning of
       the RTP packet.)
       The type value of the RTP packet, which contains only jpeg 2000
       packet(s) is 12.

       (3-a) More than one jpeg 2000 packets are packed into one RTP packet.
       +---+---+---------+-----+---------+
       |RTP|PT |jpeg 2000| ... |jpeg 2000|      type = 12
       |hdr|hdr|packet   |     |packet   |
       +---+---+---------+-----+---------+

       (3-b) The jpeg 2000 packet is split into the several RTP
       packets

       If the JPEG 2000 packet is larger than one RTP packet, then it
       may be split into two or more RTP packets. In this case, the RTP
       packets contain only a piece of the jpeg 2000 packet.

       The RTP packet with the first piece of JPEG 2000 packet is type
       13, and the last piece is type 15, and the middle pieces are
       all type 14.

       +---+---+-------------------+
       |RTP|PT |jpeg 2000 packet   |            type = 13
       |hdr|hdr|fragment(1)        |
       +---+---+-------------------+
       +---+---+-------------------+
       |RTP|PT |jpeg 2000 packet   |            type = 14
       |hdr|hdr|fragment(2)        |
       +---+---+-------------------+
       +---+---+-------------------+
       |RTP|PT |jpeg 2000 packet   |            type = 14
       |hdr|hdr|fragment(3)        |
       +---+---+-------------------+
                 ...                               ...
       +---+---+-------------------+
       |RTP|PT |jpeg 2000 packet   |            type = 15
       |hdr|hdr|fragment(N)        |
       +---+---+-------------------+

       (Note) When the JPEG 2000 packet is split into multiple RTP
       packets, another JPEG 2000 packet must not be included in the RTP
       packet containing the last fragment.
       +---+---+-------------------+---------+
       |RTP|PT |jpeg 2000 packet   |jpeg 2000|  This packetization is
       |hdr|hdr|fragment(N)        |packet   |  not allowed.
       +---+---+-------------------+---------+



Edwards, et al.                                                [Page 12]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001

6. Scalable Delivery and Priority field

    JPEG 2000 codestream has rich functionality built into it so
    decoders can easily handle scalable delivery or progressive
    transmission.  Progressive transmission that allows images to be
    reconstructed with increasing pixel accuracy or spatial resolution
    is essential for many applications. This feature allows the
    reconstruction of images with different resolutions and pixel
    accuracy, as needed or desired, for different target devices.  The
    largest image source devices can provide a codestream that is easily
    processed for the smallest image display device.

    The JPEG 2000 packets contain all compressed image data from a
    specific layer, a specific component, a specific resolution level,
    and a specific precinct. The order in which these packets are found
    in the codestream is called the "progression order". The ordering
    of the packets can progress along four axes: layer, component,
    resolution level and precinct.

    Providing priority field to show importance of data contained in a
    given RTP packet makes the most of JPEG 2000 progressive/scalable
    functions.

    In resolution progression order, the higher decomposition level is
    more important.  The priority field of the RTP packet that contains
    the higher decomposition level is set to the higher priority.
    When transmitted in spatial resolution order, LL0 components data
    is set to the highest priority.


6.1 Priority mapping table

    For the progression order, the priority value to be given to each
    JPEG 2000 packet is defined by the priority mapping table.  The
    higher the importance, the smaller the priority value.  The priority
    mapping table can define the priority values for spatial resolution,
    layer, color component, or precinct level. This priority table is
    sent from the sender to a receiver through another protocol (RTSP,
    SIP, etc.) outside of RTP.  To change the priority mapping table, a
    new priority mapping table must be sent from the sender to the
    receiver as needed.

    If there is no priority mapping table, the priority value of the RTP
    packet must be set to '0xff'.

    For example, the priority table can be sent to the receiver from the
    sender but the receiver will determine its own level of priority RTP
    packets to receive using the priority table as a guideline.

    The priority value of 1 has the highest priority in the priority
    mapping table.  As the priority value increases, the priority
    becomes lower.  If transmission is performed without attaching any
    priority mapping table, 0xff (255) must be set in the priority
    field.

Edwards, et al.                                                [Page 13]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001


    For RTP packets that only consist of a whole or fragmented main or
    tile header and containing no JPEG 2000 packets , priority 0 must be
    set by the sender if a priority mapping table is used.  (If a
    priority mapping table is not used, the priority value must be 0xff
    for the same RTP packets.)

    The sender may transmit each priority using separate multiple RTP
    sessions defined by the priority value. For example, different
    priority may be allocated to other multicast groups. The sender may
    also transmit all priority valued RTP packets using a single RTP
    session.

    When multiple JPEG 2000 packets are included in a single RTP packet,
    the higher priority value of JPEG 2000 packets is set for the whole
    RTP packet by the sender.

    In the following, an example of priority mapping table is shown.
    The component based priority should be used when there is a higher
    priority component like Y in YUV components.


6.1.1 Layer based priority

    This is an example of priority mapping table in the progression
    order in which SNR is improved progressively.  The JPEG 2000 packet
    of layer 0 and resolution 0 has the highest priority.  The JPEG 2000
    packets with layer 0 and resolution 1 or more are next in priority.
    As the layer number increases, the priority becomes lower.

        L  R  C P | priority
      ------------+-------------
        0  0  - - |     1
        0 >0  - - |     2
        1  -  - - |     3
         ....     |   ....


6.1.2 Resolution level based priority

    This is an example of priority mapping table in the progression
    order in which the spatial resolution is increased.  The JPEG 2000
    packet with layer 0 and resolution 0 has the highest priority and
    the JPEG 2000 packets with later 1 or more and resolution 0 are next
    in priority.  As the resolution level increases, the priority
    becomes lower.

        L  R C P  | priority
      ------------+-------------
        0  0 - -  |     1
       >0  0 - -  |     2
        -  1 - -  |     3
         ....     |   ....


Edwards, et al.                                                [Page 14]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001


6.1.3 Component based priority

    The priority mapping table for component progression is used only
    when there is priority order among components.  This example is for
    YUV components.  The JPEG 2000 packet with layer 0, resolution 0,
    and component 0 has the highest priority.  The JPEG packets with
    layer 1 or more, resolution 0, and component 0 are next in priority.
    The JPEG 2000 packets with resolution 0 and component 0 are the
    third in priority.  As the resolution increases, the priority
    becomes lower.

        L  R C P  | priority
      ------------+-------------
        0  0 0 -  |     1
       >0  0 0 -  |     2
          >0 0 -  |     3
        -  - 1 -  |     4
         ....     |   ....


6.2 Sender's Actions

    Priority is given in accordance with the priority mapping table.
    The priority field is only a hint for the receiver but never forces
    the receiver to use any specific processing method.  If the priority
    mapping table is not used, '0xff' must be set.


6.3 Receiver's Action

    Progressive transmission that allows images to be reconstructed with
    increasing pixel accuracy or spatial resolution is essential for
    many applications. This feature allows the reconstruction of images
    with different resolutions and pixel accuracy, as needed or desired,
    for different target devices.  The image architecture provides for
    the efficient delivery of image data in many applications such as
    client/server applications.  The receiver should decode packets
    above a certain priority to obtain maximum performance depending on
    the receiver's platform.

    The receiver can determine on its own (using or not using the
    mapping table and several other variables) the priority value level
    the RTP packets it should decode.

    For example, when the CPU power is incompetent or the terminal has
    only a low-resolution display, decoding only RTP packets below a
    certain priority permits obtaining optimal performance.

    If any high-priority RTP packet is not received when a packet loss
    occurs, frame(s) can be skipped  because visual loss may be
    remarkable even if decoding can be successfully performed.

    When any uninterpretable or unexpected priority is received, the

Edwards, et al.                                                [Page 15]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001

    receiver must interpret packets as no priority (i.e. priority=
    0xff.)

7. JPEG 2000 main header compensation

    The JPEG 2000 image main header describes various encode parameters
    and the decoder decodes by using the parameters described in the main
    header.  If the RTP packet that contains the main header is lost,
    the corresponding JPEG 2000 codestream cannot be decoded.  In an
    extremely rare case, if the main header has dropped and all the
    remainder JPEG 2000 packets has been received successfully, the
    receiver cannot decode the frame.  Even when the main header is
    lost, it can be recovered to a certain level using the following
    method.

    A recovery of the main header that has been lost is very simple.  In
    the case of JPEG 2000 video, it is common that encode parameters
    will not greatly change in each frame.  Even if the RTP packet
    including the main header of a frame has dropped, decoding
    processing can be performed by using the main header of the previous
    frame if this previous frame is already encoded by the same encode
    parameters.

    The mh_id field of the payload header is used to recognize whether
    the encoding parameters of the main header are the same as the
    encoding parameters of the previous frame. The same value is set in
    mh_id of the RTP packet in the same frame.  mh_id and encode
    parameters are not associated with each other as 1:1 but they are
    used to recognize whether the encode parameters of the previous
    frame are the same or not.

    The mh_id field is saved from previous frames to be used to recover
    the current frame's main header, if lost.  If the mh_id of the
    current frame has the same value as the mh_id value of the previous
    frame, the previous frame's main header can be used to decode the
    current frame, in case the main header lost.


7.1 Sender processing

    The sender transmits RTP packets with the same mh_id value unless
    the encoder parameters are different from the previous frame.  The
    encode parameters are the fixed information marker segment (SIZ
    marker) and functional marker segments (COD, COC, RGN, QCD, QCC, and
    POC) specified in JPEG 2000 Part 1 Annex. A.  If the encode
    parameters have been changed, the sender transmits RTP packets by
    incrementing the mh_id value by one.  The initial mh_id value is 1.
    When the mh_id value exceeds 15, the value returns to 1 again.

    If the mh_id field is set to 0, the receiver must not save the main
    header and must not compensate for lost headers using the above
    method.



Edwards, et al.                                                [Page 16]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001

7.2 Receiver processing

    When the receiver has received the main header correctly, the RTP
    sequence number, the mh_id and main header are saved except when the
    mh_id value is 0.  Only the last main header that was received
    correctly is saved.  That is, if there has been a saved main header,
    the previous one is deleted and the new main header is saved.

    When the main header could not be received, the receiver compares
    the current mh_id value (this mh_id can be known by receiving at
    least one RTP packet) with the saved mh_id value.  When the values
    are the same, decoding is performed by using the saved main header.

    The main header of mh_id = 0 is an indication from the sender  to
    not compensate for lost headers or to save any headers.  .


8. Optional Payload Header

    When the extension bit of the JPEG 2000 payload header is 1, the
    payload header is followed by an optional payload header.  The JPEG
    2000 video stream payload comes after the optional payload header.
    The figure shows a general format of the optional payload header.

     0               1               2               3
     0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |    optype   |X|     length    |                              |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                              |
    |                   option specific format .....               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

        Fig. JPEG 2000 video stream optional payload header generic format

    optype : 7 bits

        optype shows the optional payload header type.

    X : 1bit

        more extension bit. This must be set to 1 if another optional
        payload header follows this optional payload header; otherwise
        it must be set to 0.

    length : 8 bits

        length of optional header in bytes.


    The receiver performs processing for the optional header when the
    extension bit of the JPEG 2000 payload header is 1.
    When having received an optype that cannot be interpreted, the
    receiver will skip the amount specified in the length field and not
    process the optional payload header..

Edwards, et al.                                                [Page 17]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001


    When the more extension bit of the optional header is 1, another
    optional payload header will come immediately after this optional
    payload header.


8.1 Quantization Optional Header

    As one of optional payload headers, the quantization optional header
    is defined.  If only the QCD and/or QCC information has been
    changed, this optional payload header conveys the information.  One
    optional payload header for QCD and another optional payload header
    for the QCC information.  Both changes must not be conveyed in a
    single optional payload header.

    If the receiver having received the quantization optional header but
    the main header of the current frame is lost; the receiver can
    replace the QCD and QCC information in the saved main header using
    the current QCD or QCC optional header only if the mh_id value of
    the current frame and previous frame differ by 1. The receiver
    should interpret this optional payload header only when the mh_id
    value changes.

    This header is supposed to be used when an adjustment is made by
    quantization size in order to keep the amount of compressed JPEG
    2000 image data at a constant level.

    The quantization optional header format is shown in the figure
    below.

     0               1               2               3
     0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |   optype=1  |X|     length    |Q|    cindex   |  decomp level |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |     style     |                                               |
    +-+-+-+-+-+-+-+-+                                               +
    |               quantization step size value  .....             |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

        Fig. Quantization Optional Header format

    Each field is explained below.

    optype : 1 bit

        The optype value of the quantization optional header is 1.

    Q : 1 bit

        This indicates whether the information is of QCD or of QCC.  If
        the information is of QCD, 0 is set.  If the information is of
        QCC, 1 is set.


Edwards, et al.                                                [Page 18]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001

    cindex : 7 bits

        When the information is of QCC, this represents a component
        number.

    decomp level : 8 bits

        This indicates the decomposition level of the corresponding frame.

    style : 8 bits

        This indicates the quantization style specified in the QCD and
        QCC marker segments.  (Refer to JPEG 2000 Part I: Annex A Table
        A-28.)

    quantization step size value : variable length

        This is followed by the quantization stop size value specified
        by style.  (Refer to JPEG 2000 Part I: Annex A Table A-29 and
        A-30.)


9. Security Consideration

    RTP packets using the payload format defined in this specification
    are subject to the security considerations discussed in the RTP
    specifications[3]. This implies that confidentiality of the media
    streams is achieved by encryption. Because the data compression
    used with this payload format is applied end-to-end, encryption
    may be performed on the compressed data so there is no conflict
    between the two operations.


10. Author's Address

    Eric Edwards
    Sony Corporation
    Media Processing Division
    Network & Software Technology Center of America
    3300 Zanker Road, MD: SJ2C4
    San Jose, CA 95134
    Phone: +1 408 955 6462
    Fax: +1 408 955 5724
    Email: Eric.Edwards@am.sony.com

    Satoshi Futemma
    Sony Corporation
    6-7-35 Kitashinagawa Shinagawa-ku
    Tokyo 141-0001 JAPAN
    Phone: +81 3 5448 4373
    Fax: +81 3 5448 4622
    Email: satosi-f@sm.sony.co.jp

    Eisaburo Itakura

Edwards, et al.                                                [Page 19]

INTERNET-DRAFT    draft-edwards-avt-rtp-jpeg2000-00.txt    November 2001

    Sony Corporation
    6-7-35 Kitashinagawa Shinagawa-ku
    Tokyo 141-0001 JAPAN
    Phone: +81 3 5448 3096
    Fax: +81 3 5448 4622
    Email: itakura@sm.sony.co.jp

    Takahiro Fukuhara
    Sony Corporation
    1-11-1 Osaki Shinagawa-ku
    Tokyo 141-0032 JAPAN
    Phone: +81 3 5435 3665
    Fax: +81 3 5435 3891
    Email: fukuhara@av.crl.sony.co.jp


11. References

    [1] ISO/IEC JTC1/SC29/WG1: "JPEG 2000 Part I Final Draft
        International Standard", September 2000.

    [2] ISO/IEC JTC1/SC29/WG1: "Motion JPEG 2000 Committee Draft
        1.0", http://www.jpeg.org/public/cd15444-3.pdf,
        December 2000.

    [3] A. N. Skodras, C. A. Christopoulos and T. Ebrahimi: "JPEG2000:
        The Upcoming Still Image Compression Standard", In Proc. of the
        11th Portuguese Conference on Pattern Recognition, pp. 359-366,
        Porto, Portugal, May 2000.

    [4] ISO/IEC JTC1/SC29/WG1: "JPEG2000 requirements and profiles
        version 6.3", draft in progress,
        http://www.jpeg.org/public/wg1n1803.pdf, July 2000.

    [5] Diego Santa-Cruz, Touradj Ebrahimi, Joel Askelof, Mathias
        Larsson and Charilaos Christopoulos: "JPEG 2000 still image
        coding versus other standards", In Proc. of SPIE's 45th annual
        meeting, Applications of Digital Image Processing XXIII,
        vol. 4115, pp. 446-454, July 2000.

    [6] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson "RTP:
        A Transport Protocol for Real Time Applications", RFC 1889,
        January 1996.












Edwards, et al.                                                [Page 20]


Html markup produced by rfcmarkup 1.107, available from http://tools.ietf.org/tools/rfcmarkup/