[Docs] [txt|pdf] [Tracker] [Email] [Nits]

Versions: 00

Network Working Group                                          T. Davies
Internet-Draft                                                     Cisco
Intended status: Standards Track                        October 19, 2015
Expires: January 7, 2016


              Interpolated reference frames for video coding
                      draft-davies-netvc-irfvc-00

Abstract

   This document describes the use of interpolated reference frames in
   video coding in general, and in the Thor video codec in particular.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on January 7, 2016.

Copyright Notice

   Copyright (c) 2015 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.










Davies                   Expires January 7, 2016                [Page 1]


Internet-Draft                   IRFVC                      October 2015


Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Definitions . . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.1.  Requirements Language . . . . . . . . . . . . . . . . . .   3
     2.2.  Terminology . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  The interpolation process . . . . . . . . . . . . . . . . . .   3
     3.1.  Interpolation framework . . . . . . . . . . . . . . . . .   3
     3.2.  Motion estimation process . . . . . . . . . . . . . . . .   4
     3.3.  Complexity considerations . . . . . . . . . . . . . . . .   5
   4.  Coding using interpolated reference frames  . . . . . . . . .   6
   5.  Compression performance . . . . . . . . . . . . . . . . . . .   6
   6. IANA Considerations  . . . . . . . . . . . . . . . . . . . . .   8
   7. Security Considerations  . . . . . . . . . . . . . . . . . . .   8
   8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . .   8
   9. Normative References . . . . . . . . . . . . . . . . . . . . .   8
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .   8

1.  Introduction

   This document describes a method of generating synthetic reference
   frames for video coding using a simplified frame interpolation
   method. The aim is to create a reference frame that is temporally
   co-located with the current frame being predicted, leveraging the
   motion information already present in the previously-coded frames,
   and removing the need for techniques such as motion vector scaling
   in motion vector prediction.

   Since the decoder will have to generate the same interpolated
   reference frame as the encoder, complexity considerations are a
   paramount concern. The interpolation process is therefore a highly
   simplified block-matching algorithm and uses only pixel-accurate
   motion vectors, for example. Worst-case complexity can be managed by
   controlling the number of matches per block, per region and per
   frame as well as the total vertical excursion to manage memory
   bandwidth.

   The method gives most gain in Thor at high quantisation (QP) levels
   i.e. low bitrates. Overall, Bjontegaard delta-rate (BDR) reductions
   across QP ranges 22-37 are on average 5.2% for a range of HD test
   sequences.  For higher QP (32-44) the reductions gains are larger:
   8.8% on average.


   Interpolated reference frames are enabled by default in the high
   complexity random access (RA) and High Delay B (HDB) configurations
   in the Thor repository github.com/cisco/thor.




Davies                   Expires January 7, 2016                [Page 2]


Internet-Draft                   IRFVC                      October 2015


   Section 3 describes the interpolation process, which is based on
   a simplified hierarchical motion estimation (HME). Section 4
   describes the modifications to the Thor syntax coding processes.
   Section 5 provides details of compression performance.

2.  Definitions

2.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].


2.2.  Terminology

   This document frequently uses the following terms.

      MV: Motion vector - a horizontal and vertical vector displacement
      (x,y)

      ME: Motion Estimation

      HME: Hierarchical ME

      SAD: Sum of Absolute Differences. A metric defined for a pair of
      equal dimension blocks of numerical vaules consisting of the sum
      of the absolute differences of the corresponding values in each
      location in the blocks

      QP: quantisation parameter

      BDR: Bjontegaard Delta-Rate

3.  The interpolation process

3.1.  Interpolation framework

  Consider two frames R0 and R1 and a frame F equidistant in time
  between them which is to be interpolated (Figure 1). Image data must
  be created for each block in F by combining information from R0 and
  R1 using a linear model for the block motion.









Davies                   Expires January 7, 2016                [Page 3]


Internet-Draft                   IRFVC                      October 2015




    ______________________________|_______|__________________________ R0
                                  |   /\  |
                                      /
                                     /
                                    / mv0
                                   /
                                  /
    ________________________|____/__|________________________________ F
                            |   /   |
                               /
                              /
                             / mv1
                            /
                           /
    __________________|___\/__|______________________________________ R1
                      |       |


            Figure 1: forward and backward motion pairs for a block

   For each block in the frame F there is an associated motion vector
   mv0 pointing at a displaced block in R0 and a corresponding motion
   vector mv1 which is equal to -mv0 pointing at R1.

   Where F is not equidistant from the reference frames the linear model
   can simply be scaled appropriately.

   If both blocks fall within the reference frames, then the
   interpolated block is just the average of the two reference blocks.
   At the edges of the frames one of the reference blocks may fall off
   the edge - here the other reference only is used instead.

3.2.  Motion estimation process

   Since F does not exist the motion estimation process consists of
   matching blocks B+mv0 in R0 with blocks B+mv1 in R1. A basic block
   size of 8x8 is used but the bulk of the motion estimation is done for
   16x16 blocks. For UHD resolutions, perhaps a larger basic block size
   would be better. The overall approach is to use hierarchical motion
   estimation (HME), as this is amenable to limiting both average and
   worst-case complexity.

   In the HME scheme each reference frame is down-scaled vertically and
   horizontally by a factor 2, using a (1/2,1/2) filter. This is done
   repeatedly to get a series R0(n) and R1(n) of reference frames. Then
   motion estimation is done very simply on each resolution layer n, but



Davies                   Expires January 7, 2016                [Page 4]


Internet-Draft                   IRFVC                      October 2015


   using candidates from next layer (n+1) as well as spatial neighbours.
   The block sizes are the same at each layer, so each block at layer
   n+1 corresponds to 4 blocks at layer n.

   For each layer, the ME stages are as follows:

      1.  For each 16x16 block in raster order:
         a.  Check if ME can be bypassed.
         b. If not bypassed, determine candidates from lower layer
            blocks and from neighbour blocks in raster order
         c. Perform an adaptive cross search around each candidate
            vector and determine the best vector

      2.  For each 8x8 block in raster order, find the best merge
          candidate, i.e. choose which MV to use: the original 16x16
          block vector, or one of 4 neighbouring block 16x16 vectors
          (above, below, left or right)

   The majority of blocks bypass ME at step 1a. Here a skip candidate
   is generated as:

   skipmv = argmin{mvx in {mv0,mv1,mv2}: sum_{i=0}^{2} |mvx-mvi|}

   where mv0,mv1,and mv2 are the motion vectors for blocks above, left
   and above-right the current block. If the cost for this vector is
   below a fixed value for each 8x8 sub-block, no further ME is done.

   In step 1c, the ranges of the cross search are restricted to just 2
   steps (max 8 matches) for each candidate, if the search is not at
   the lowest resolution layer. This is because vector candidates from
   the lower layer or from neighbours will already be highly accurate by
   this point.

   In step 1, the cost metric is a combination of luma SAD and a fixed
   multiple of the sum of abolute motion vector difference between the
   vector mvx and the four neighbours mv0,mv1,mv2,mv3 to the left,
   right, above and above right, i.e.

   sum_{i=0}^{3} |mvx-mvi|

   This helps make the motion estimation process less sensitive to noise
   and spurious matches.

   In step 2 the cost metric is SAD alone.

3.3.  Complexity considerations

   The ME process is not that sensitive to the selection of candidates,



Davies                   Expires January 7, 2016                [Page 5]


Internet-Draft                   IRFVC                      October 2015


   at least in terms of the impact on coding performance. If the
   interpolated frames are used directly this might not be so, but in
   effect the interpolated blocks are only going to be used for
   prediction if they are interpolated well: therefore effort refining
   bad matches is generally wasted, so should be avoided.

   This means that the ME process can be quite truncated. The only
   candidates considered are up to three neighbour block candidates and
   one from the layer below. The majority of motion estimation is
   skipped, and so only requires a single match. For HW applications
   the total number of matches would still require a hard limit, as
   well as limits for the matches per block and possibly per region.
   Vertical motion vector limits could also be imposed to reduce memory
   bandwidth costs.

4.  Coding using interpolated reference frames

   In the Thor implementation, when an interpolated reference frame is
   used it is inserted at the beginning of the reference pictures list
   and is given the same frame number as the current frame. Typically
   use of the interpolated reference frame causes a considerable
   increase in uni-pred prediction, often with no residual to code, and
   a reduction of bi-prediction modes. This changes the probability of
   the various supermode values used in Thor. Therefore in such frames
   it makes sense to modify the supermode coding to reflect this, and
   this contributes a small amount to coding gains. Full details are in
   [Fuld1].

5.  Compression performance

  Luma PSNR BDR percentage gains for standard QP ranges (22,27,32,37)
  are given in Table 1. For high QP (32,36,40,44), the results are in
  Table 2.


















Davies                   Expires January 7, 2016                [Page 6]


Internet-Draft                   IRFVC                      October 2015



  -------------------------------------------------------
  1920x1080
  -------------------------------------------------------
  Kimono                                             -3.5
  ParkScene                                          -3.1
  Cactus                                             -4.9
  BasketballDrive                                    -2.1
  BQTerrace                                          -1.9
  ChangeSeats                                        -5.8
  HeaAndShoulder                                     -6.6
  TelePresence                                       -6.6
  WhiteBoard                                         -7.5
  -------------------------------------------------------
  1280x720
  -------------------------------------------------------
  FourPeople                                         -7.0
  Johnny                                             -6.2
  KristenAndSara                                     -7.0
  -------------------------------------------------------
  Average                                            -5.2

        Table 1: BDR reductions for standard QPs

  -------------------------------------------------------
  1920x1080
  -------------------------------------------------------
  Kimono                                             -6.6
  ParkScene                                          -7.0
  Cactus                                             -8.9
  BasketballDrive                                    -5.5
  BQTerrace                                          -4.7
  ChangeSeats                                       -12.1
  HeaAndShoulder                                    -10.1
  TelePresence                                      -11.0
  WhiteBoard                                        -12.4
  -------------------------------------------------------
  1280x720
  -------------------------------------------------------
  FourPeople                                         -9.1
  Johnny                                             -8.0
  KristenAndSara                                     -9.9
  -------------------------------------------------------
  Average                                            -8.8

        Table 2: BDR reductions for high QPs





Davies                   Expires January 7, 2016                [Page 7]


Internet-Draft                   IRFVC                      October 2015



6.  IANA Considerations

   This document has no IANA considerations.

7.  Security Considerations

   This document has no security considerations.

8.  Acknowledgements

   The author would like to thank Arild Fuldseth for assistance with
   experimental investigations, and Mo Zanaty for reviewing this
   document.


9.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [Fuld1]    Fuldseth, A., Bjontegaard, G., Zanaty, M. "The Thor video
              codec", draft-fuldseth-netvc-thor-01, October 2015.

Authors' Addresses

   Thomas Davies
   Cisco
   Feltham
   UK

   Email: thdavies@cisco.com



















Davies                   Expires January 7, 2016                [Page 8]

Internet-Draft                   IRFVC                      October 2015


Html markup produced by rfcmarkup 1.124, available from https://tools.ietf.org/tools/rfcmarkup/