[Docs] [txt|pdf|xml] [Tracker] [Email] [Diff1] [Diff2] [Nits]

Versions: 00 01

Network Working Group                                    P. Hallam-Baker
Internet-Draft                                         Comodo Group Inc.
Intended status: Informational                             July 13, 2018
Expires: January 14, 2019


                Data At Rest Encryption: DARE Container
                  draft-hallambaker-dare-container-01

Abstract

   This document describes DARE Container, a message and file syntax
   that allows a sequence of data frames to be represented with
   cryptographic integrity, signature and encryption enhancements to be
   constructed in an append only format.  The format supports data
   integrity checks using digest chains and Merkle trees.  The simplest
   supports efficient append only write operations and efficient read
   operations in either the forward or reverse direction.  Support for
   efficient random-access reads may be provided through the use of
   binary trees or index records appended to the end of the file.

   This document is also available online at
   http://mathmesh.com/Documents/draft-hallambaker-dare-container.html
   [1] .

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on January 14, 2019.

Copyright Notice

   Copyright (c) 2018 IETF Trust and the persons identified as the
   document authors.  All rights reserved.





Hallam-Baker            Expires January 14, 2019                [Page 1]


Internet-Draft             DARE Data Container                 July 2018


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Definitions . . . . . . . . . . . . . . . . . . . . . . . . .   5
     2.1.  Related Specifications  . . . . . . . . . . . . . . . . .   5
     2.2.  Requirements Language . . . . . . . . . . . . . . . . . .   5
   3.  Container Format  . . . . . . . . . . . . . . . . . . . . . .   5
     3.1.  Container Profile . . . . . . . . . . . . . . . . . . . .   6
       3.1.1.  Index Profiles  . . . . . . . . . . . . . . . . . . .   7
       3.1.2.  Content Profiles  . . . . . . . . . . . . . . . . . .   7
     3.2.  Payload Encryption  . . . . . . . . . . . . . . . . . . .   8
   4.  Index Mechanisms  . . . . . . . . . . . . . . . . . . . . . .   8
     4.1.  Tree  . . . . . . . . . . . . . . . . . . . . . . . . . .   8
     4.2.  Position Index  . . . . . . . . . . . . . . . . . . . . .   9
     4.3.  Metadata Index  . . . . . . . . . . . . . . . . . . . . .   9
   5.  Integrity Mechanisms  . . . . . . . . . . . . . . . . . . . .   9
     5.1.  Digest Chain calculation  . . . . . . . . . . . . . . . .   9
     5.2.  Binary Merkle tree calculation  . . . . . . . . . . . . .  10
   6.  Reference . . . . . . . . . . . . . . . . . . . . . . . . . .  10
     6.1.  Container Headers . . . . . . . . . . . . . . . . . . . .  10
       6.1.1.  Structure: ContainerHeaderFirst . . . . . . . . . . .  10
       6.1.2.  Structure: ContainerHeader  . . . . . . . . . . . . .  11
     6.2.  Content Metadata Structure  . . . . . . . . . . . . . . .  12
       6.2.1.  Structure: ContentMeta  . . . . . . . . . . . . . . .  12
     6.3.  Index Structures  . . . . . . . . . . . . . . . . . . . .  12
       6.3.1.  Structure: ContainerIndex . . . . . . . . . . . . . .  12
       6.3.2.  Structure: IndexPosition  . . . . . . . . . . . . . .  13
       6.3.3.  Structure: KeyValue . . . . . . . . . . . . . . . . .  13
       6.3.4.  Structure: IndexMeta  . . . . . . . . . . . . . . . .  13
     6.4.  Signature . . . . . . . . . . . . . . . . . . . . . . . .  13
   7.  Further Work  . . . . . . . . . . . . . . . . . . . . . . . .  14
     7.1.  Fast open with random access  . . . . . . . . . . . . . .  14
     7.2.  Partitioning of very large data sets across hosts . . . .  15
     7.3.  Filtering and redaction . . . . . . . . . . . . . . . . .  15
     7.4.  Encryption of large data blocks . . . . . . . . . . . . .  15
     7.5.  Concurrent Writes . . . . . . . . . . . . . . . . . . . .  16
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .  16
   9.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  16



Hallam-Baker            Expires January 14, 2019                [Page 2]


Internet-Draft             DARE Data Container                 July 2018


   10. Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  16
   11. Appendix A: Examples and Test Vectors . . . . . . . . . . . .  16
     11.1.  Simple container . . . . . . . . . . . . . . . . . . . .  16
     11.2.  Payload and chain digests  . . . . . . . . . . . . . . .  17
     11.3.  Merkle Tree  . . . . . . . . . . . . . . . . . . . . . .  17
     11.4.  Signed container . . . . . . . . . . . . . . . . . . . .  17
     11.5.  Encrypted container  . . . . . . . . . . . . . . . . . .  17
   12. Appendix B  . . . . . . . . . . . . . . . . . . . . . . . . .  19
   13. References  . . . . . . . . . . . . . . . . . . . . . . . . .  19
     13.1.  Normative References . . . . . . . . . . . . . . . . . .  19
     13.2.  Informative References . . . . . . . . . . . . . . . . .  20
     13.3.  URIs . . . . . . . . . . . . . . . . . . . . . . . . . .  20
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  20

1.  Introduction

   DARE Container is a message and file syntax that allows a sequence of
   data frames to be represented with cryptographic integrity,
   signature, and encryption enhancements to be constructed in an append
   only format.  DARE Container was developed in response to needs that
   arose out of the design of the Mathematical Mesh
   [draft-hallambaker-mesh-architecture] . It is built on the binary
   encodings of JSON data objects, JSON-B and JSON-C
   [draft-hallambaker-jsonbcd] and the DARE Message format
   [draft-hallambaker-dare-message] .

   The high level requirements supported include:

   o  Recording Mesh transactions in persistent storage.

   o  Synchronizing transaction logs between hosts.

   o  Representing message archives (aka mail spool)

   o  Signing and encrypting single data items.

   The features supported by DARE Container include:

   o  The format is append only, thus providing for rapid write
      operations and enabling the use of technologies that provide
      atomic transactions.

   o  All length and index values support the use of integers of at
      least 64 bits.

   o  Data frames may be of variable length.





Hallam-Baker            Expires January 14, 2019                [Page 3]


Internet-Draft             DARE Data Container                 July 2018


   o  Data frames may be read in either direction.  This allows the last
      n frames to be read as efficiently as the first n frames.

   o  Appending a data frame to an existing file is efficient taking no
      more than log2 (n) operations.

   o  A binary tree index MAY be constructed on an incremental basis,
      allowing random access to the nth record in the file in log2 (n)
      operations.

   o  An index MAY be appended to an existing container to allow random
      access to the nth record in the file in log2 (n) operations

   o  Permits the use of modern data encodings (e.g.  JSON [RFC7159] ).

   o  Supports digital signature and public key operations on the
      payloads of individual data frames.

   o  Data frame content (i.e. payload data) may be overwritten without
      invalidating the integrity of any other frame.  This allows
      content to be expunged in exigent circumstances (court order,
      regulatory, confidentiality breach, etc.) without compromising the
      integrity of the rest of the data in the container.

   Many file proprietary formats are in use that support some or all of
   these capabilities but only a handful have public, let alone open,
   standards.  DARE Container is designed to provide a superset of the
   capabilities of existing message and file syntaxes, including:

   o  Cryptographic Message Syntax [RFC5652] defines a syntax used to
      digitally sign, digest, authenticate, or encrypt arbitrary message
      content.

   o  The.ZIP File Format specification [ZIPFILE] developed by Phil
      Katz.

   o  The BitCoin Block chain [BLOCKCHAIN] .

   o  JSON Web Encryption and JSON Web Signature

   Attempting to make use of these specifications in a layered fashion
   would require at least three separate encoders and introduce
   unnecessary complexity.

   Every data format represents a compromise between different concerns,
   in particular:

   Data Storage  The space required to record data in the encoding.



Hallam-Baker            Expires January 14, 2019                [Page 4]


Internet-Draft             DARE Data Container                 July 2018


   Memory Overhead  The additional volatile storage (RAM) required to
      maintain indexes etc. to support efficient retrieval operations.

   Number of Operations  The number of operations required to retrieve
      data from or append data to an existing encoded sequence.

   Number of Disk Seek Operations  Optimizing the response time of
      magnetic storage media to random access read requests has
      traditionally been one of the central concerns of database design.
      The DARE Container format is designed to the assumption that this
      will cease to be a concern as solid state media replaces magnetic.

   While the cost of storage of all types has declined rapidly over the
   past decades, so has the amount of data to be stored.  DARE Container
   represents a pragmatic balance of these considerations for current
   technology.  In particular, since payload volumes are likely to be
   very large, memory and operational efficiency are considered higher
   priorities than data volume.

2.  Definitions

2.1.  Related Specifications

   DARE Container makes use of the following related standards and
   specifications.

   Encoding  Content frame headers are encoded using JavaScript Object
      Notation (JSON) [RFC7159] , JSON-B or JSON-C
      [draft-hallambaker-jsonbcd] .

   Cryptography  The encryption and signature schemes used are based on
      JSON Web Signature [RFC7515] and JSON Web Encryption [RFC7516] .

2.2.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119] .

3.  Container Format

   A DARE Container consists of a sequence of JBCD Frames containing up
   to three ordered JBCD records as follows:

   Header Record [required]  Metadata of any type including container
      metadata, payload metadata and DARE message headers.





Hallam-Baker            Expires January 14, 2019                [Page 5]


Internet-Draft             DARE Data Container                 July 2018


   Payload Record [optional]  The frame data.  Payload records are
      either complete or incremental.  A complete payload contains a
      complete unit of whatever data is being written to the log.  An
      incremental payload contains a part of a unit that has been split
      across multiple records.

   Guard Record [optional]  An opaque data record with a known value
      written to the end of a frame to allow a writer to avoid
      corruption of the container framing data by detecting an
      incomplete append state.

   A DARE frame consists of a forward length indicator, the framed data
   and a reverse length indicator.  The reverse length indicator is
   written out backwards to allow the frame to be read in the reverse
   direction:

   [[This figure is not viewable in this format.  The figure is
   available at http://mathmesh.com/Documents/draft-hallambaker-dare-
   container.html [2].]]


   JBCD Bidirectional Frame

   When first reading an existing file, an application will typically
   read the first frame and the last frame (if the container has more
   than one frame).  This allows the reader to quickly determine the
   format(s) used by the container, the number of frames in the
   container and the location of any index frames (if present).

   The container format is designed to support creation of write-once
   and append-only file formats.  Each frame SHOULD be written as an
   atomic operation.

   The first frame in a container and the first record in a frame have
   special roles that are described in this document.

   o  The first frame in a container describes the container format
      options and defaults.  These include the range of encoding options
      for frame metadata supported and the container profiles to which
      the container conforms.

   o  The first record in a frame MUST NOT contain payload data

3.1.  Container Profile

   A key objective of the DARE Container format is that the simplest
   possible reader be capable of reading any container file albeit with
   possibly reduced performance.



Hallam-Baker            Expires January 14, 2019                [Page 6]


Internet-Draft             DARE Data Container                 July 2018


   A Container MAY conform to one or more profiles.  Conforming to a
   profile typically requires a writer to provide additional information
   when writing a file but does not require a reader to interpret it
   unless use of a feature (e.g. authentication) that depends on the
   additional information is required.

3.1.1.  Index Profiles

   The following profiles are currently defined:

   Tree  Frame headers contain IndexPosition entries that specify the
      start position of previous frames.  This enables efficient random
      access to any frame in the file.

   Digest  Frame headers contain PayloadDigest entries that specify the
      digest value of the corresponding payload data in that frame.

   Chain  Frame headers contain ChainDigest entries that link each frame
      to the preceding frame.

   Merkle  Frame headers contain TreeDigestPartial and TreeDigestFinal
      entries linking all the frames in the container in a binary Merkle
      Tree.

   The use of Chain and Merkle Trees for integrity checks is described
   below.

   The use of Tree and Index frames is described below.

3.1.2.  Content Profiles

   The following profiles are currently defined:

   Singleton  A container with exactly one content frame.  A container
      declared as a singleton frame cannot have additional content
      frames appended (but metadata frames may be)

   Multi  A container whose payload data is limited to content frames.
      A container declared as a multi container may contain 0, 1 or more
      content frames.

   Archive  A multi-container whose payload data is limited to content
      frames whose last frame contains a metadata index for the content
      frames in the container.

   Unitary  A multi-container in which each frame represents exactly one
      payload object.




Hallam-Baker            Expires January 14, 2019                [Page 7]


Internet-Draft             DARE Data Container                 July 2018


   Serial  A multi-container in which payload objects MAY be split
      across multiple consecutive frames.

   Interleaved  A multi-container in which payload objects MAY be split
      across multiple frames which may in turn be interleaved with
      frames containing other payload objects in complete or partial
      form.

3.2.  Payload Encryption

   DARE container payloads MAY be encrypted as DARE Message Enhanced
   Data Sequences [draft-hallambaker-dare-message] . This specification
   builds on JSON Web Encryption (JWE) [RFC7516] to provide a flexible
   framework allowing a single key exchange to be applied to encrypt
   multiple data sequences.

   The DARE Container and DARE Message format are designed to compliment
   each other:

   o  DARE Message Format allows a Master Key established in a single
      key exchange to be applied to multiple DARE related messages.

   o  DARE Container Format allows sets of DARE related messages to be
      organized so that individual messages and the keying material
      required to interpret them can be efficiently retrieved.

4.  Index Mechanisms

   An index may be appended to an existing file at any time.  Since the
   use of bidirectional frames makes reading the last record is as
   efficient as reading the first, the last record in an indexed file is
   usually either the index itself or a pointer to the last index.

   An index frame consists of a frame header

   Use of index frames provides read access to any record in the file in
   O(1) operations but attempting to compiling a complete index with
   every write incurs an O(n) penalty on write for both operations and
   storage.  Accordingly, random read access to a file while it is being
   written is better supported using an index tree.

4.1.  Tree

   Binary search is supported by means of the TreePosition parameter
   specified in the FrameHeader.  This parameter specifies the value of
   the immediately preceding apex.





Hallam-Baker            Expires January 14, 2019                [Page 8]


Internet-Draft             DARE Data Container                 July 2018


   Calculation of the immediately preceding apex is most easily
   described by representing the array index in binary with base of 1
   (rather than 0).  An array index that is a power of 2 (2, 4, 8, 16,
   etc.) will be the apex of a complete tree.  Every other array index
   has the value of the sum of a set of powers of 2 and the immediately
   preceding apex will be the value of the next smallest power of 2 in
   the sum.

   For example, to find the immediately preceding apex for frame 5, we
   add 1 to get 6. 6 = 4 + 2, so we ignore the 2 and the preceding frame
   is 4.

   The values of Tree Position are shown for the first 8 frames in
   figure xx below:

   [[This figure is not viewable in this format.  The figure is
   available at http://mathmesh.com/Documents/draft-hallambaker-dare-
   container.html [3].]]


   Merkle Tree Integrity check

   An algorithm for efficiently calculating the immediately preceding
   apex is provided in Appendix C.

4.2.  Position Index

   Contains a table of index, position pairs pointing to prior locations
   in the file.

4.3.  Metadata Index

   Contains a list of IndexMeta entries.  Each entry contains a metadata
   description and a list of frame indexes (not positions) of frames
   that match the description.

5.  Integrity Mechanisms

   Frame sequences in a DARE container MAY be protected against a frame
   insertion attack by means of a digest chain, a binary Merkle tree or
   both.

5.1.  Digest Chain calculation

   A digest chain is simple to implement but can only be verified if the
   full chain of values is known.  Appending a frame to the chain has
   O(1) complexity but verification has O(n) complexity:




Hallam-Baker            Expires January 14, 2019                [Page 9]


Internet-Draft             DARE Data Container                 July 2018


   [[This figure is not viewable in this format.  The figure is
   available at http://mathmesh.com/Documents/draft-hallambaker-dare-
   container.html [4].]]


   Hash chain integrity check

   The value of the chain digest for the the first frame (frame 0) is
   H(IV+H(Payload0)), where IV is an initialization vector consisting of
   a string of zero bytes and payloadn is the sequence of payload data
   bytes for frame n

   The value of the chain digest for frame n is H(H(Payloadn-1
   +H(Payloadn)), where A+B stands for concatenation of the byte
   sequences A and B.

5.2.  Binary Merkle tree calculation

   The tree index mechanism describe earlier may be used to implement a
   binary Merkle tree.  The value TreeDigest specifies the apex value of
   the tree for that node.

   Appending a frame to the chain has O(log2n) complexity provided that
   the container format supports at least the binary tree index.
   Verifying a chain has O(log2 n) complexity, provided that the set of
   necessary digest inputs is known.

   To calculate the value of the tree digest for a node, we first
   calculate the values of all the sub trees that have their apex at
   that node and then calculate the digest of that value and the
   immediately preceding local apex.

6.  Reference

   TBS stuff

6.1.  Container Headers

   TBS stuff

6.1.1.  Structure: ContainerHeaderFirst

   Inherits: ContainerHeader

   Inherits: ContainerHeader






Hallam-Baker            Expires January 14, 2019               [Page 10]


Internet-Draft             DARE Data Container                 July 2018


   DataEncoding: String (Optional)  Specifies the data encoding for the
      header section of for the following frames.  This value is ONLY
      valid in Frame 0 which MUST have a header encoded in JSON.

6.1.2.  Structure: ContainerHeader

   Inherits: DAREHeader

   Describes a container header.  A container header MAY contain any
   DARE Message header.

   Index: Integer (Optional)  The record index within the file.  This
      MUST be unique and satisfy any additional requirements determined
      by the ContainerType.

   ContainerType: String (Optional)  Specifies the container type for
      the following records.

   IsMeta: Boolean (Optional)  If true, the current frame is a meta
      frame and does not contain a payload.

      Note: Meta frames MAY be present in any container.  Applications
      MUST accept containers that contain meta frames at any position in
      the file.  Applications MUST NOT interpret a meta frame as a data
      frame with an enpty payload.

   UniqueID: String (Optional)  Unique object identifier

   ContentMeta: ContentMeta (Optional)  Content meta data.

   TreePosition: Integer (Optional)  Position of the frame containing
      the apex of the preceding sub-tree.

   IndexPosition: Integer (Optional)  Specifies the position in the file
      at which the last index entry is to be found

   ExchangePosition: Integer (Optional)  Specifies the position in the
      file at which the key exchange data is to be found

   ContainerIndex: ContainerIndex (Optional)  An index of records in the
      current container up to but not including this one.

   PayloadDigest: Binary (Optional)  If present, contains the digest of
      the Payload.

   ChainDigest: Binary (Optional)  If present, contains the digest of
      the PayloadDigest values of this frame and the frame immediately
      preceding.



Hallam-Baker            Expires January 14, 2019               [Page 11]


Internet-Draft             DARE Data Container                 July 2018


   TreeDigest: Binary (Optional)  If present, contains the Binary Merkle
      Tree digest value.

   Event: String (Optional)  Unique object identifier

   Labels: String [0..Many]  List of labels that are applied to the
      payload of the frame.

   KeyValues: KeyValue [0..Many]  List of key/value pairs describing the
      payload of the frame.

   First: Integer (Optional)  Frame number of the first object instance
      value.

   Previous: Integer (Optional)  Frame number of the immediately prior
      object instance value

6.2.  Content Metadata Structure

   TBS stuff

6.2.1.  Structure: ContentMeta

   Information describing the object instance

   ContentType: String (Optional)  The content type field as specified
      in JWE

   Paths: String [0..Many]  List of filename paths for the payload of
      the frame.

   UniqueID: String (Optional)  Unique object identifier

   Created: DateTime (Optional)  Initial creation date.

   Modified: DateTime (Optional)  Date of last modification.

6.3.  Index Structures

   TBS stuff

6.3.1.  Structure: ContainerIndex

   A container index

   Full: Boolean (Optional)  If true, the index is complete and contains
      position entries for all the frames in the file.  If absent or
      false, the index is incremental and only contains position entries



Hallam-Baker            Expires January 14, 2019               [Page 12]


Internet-Draft             DARE Data Container                 July 2018


      for records added since the last frame containing a
      ContainerIndex.

   Positions: IndexPosition [0..Many]  List of container position
      entries

   Metas: IndexMeta [0..Many]  List of container position entries

6.3.2.  Structure: IndexPosition

   Specifies the position in a file at which a specified record index is
   found

   Index: Integer (Optional)  The record index within the file.

   Position: Integer (Optional)  The record position within the file
      relative to the index base.

6.3.3.  Structure: KeyValue

   Specifies a key/value entry

   Key: String (Optional)  The key

   Value: String (Optional)  The value corresponding to the key

6.3.4.  Structure: IndexMeta

   Specifies the list of index entries at which a record with the
   specified metadata occurrs.

   Index: Integer [0..Many]  List of record indicies within the file
      where frames matching the specified criteria are found.

   ContentType: String (Optional)  Content type parameter

   Paths: String [0..Many]  List of filename paths for the current
      frame.

   Labels: String [0..Many]  List of labels that are applied to the
      current frame.

6.4.  Signature

   Payload data MAY be signed using a JWS [RFC7515] as applied in the
   DARE Message format [draft-hallambaker-dare-message] .





Hallam-Baker            Expires January 14, 2019               [Page 13]


Internet-Draft             DARE Data Container                 July 2018


   Signatures are specified by the Signatures parameter in the content
   header.  The data that the signature is calculated over is defined by
   the typ parameter of the Signature as follows.

   Payload  The frame payload data.

   PayloadDigest  The value of the PayloadDigest parameter

   ChainDigest  The value of the ChainDigest parameter

   TreeDigestFinal  The value of the TreeDigestFinal parameter

   If the typ parameter is absent, the value Payload is implied.

   A frame MAY contain multiple signatures created with the same signing
   key and different typ values.

   The use of signatures over chain and tree digest values permit
   multiple frames to be validated using a single signature verification
   operation.

7.  Further Work

   The container format is intended to be the basis of future work to
   support:

   o  Very large container sizes (larger than the size of the host's
      memory).

   o  Partitioning of very large data sets across multiple hosts with
      parallel append.

   o  Fault tolerance

7.1.  Fast open with random access

   The container format is designed to be capable of supporting
   efficient random access to frames in containers considerably larger
   than the processing memory of the host computer without the need to
   pre-load indexes.

   A combination of the following strategies is being considered:

   o  Use memory mapped file views to container data to optimize random
      access times while controlling memory use and time taken to
      construct memory views.





Hallam-Baker            Expires January 14, 2019               [Page 14]


Internet-Draft             DARE Data Container                 July 2018


   o  When the container is first bound, use the binary tree index data
      in TreePosition parameters to support random access operations
      until index building is complete.

   o  Perform Index building operations as a non-blocking background
      task.

7.2.  Partitioning of very large data sets across hosts

   While storage devices capable of storing tends of Tb of data with
   RAID redundancy are commonplace, it is generally desirable that there
   be at least as many CPU cores as disks.  Thus, partitioning of data
   sets across multiple hosts becomes desirable for throughput even if a
   single host could handle the storage requirement.

7.3.  Filtering and redaction

   In the types of applications envisaged in the Mesh, almost every data
   set may be reduced to collections that are bound to a single account.
   While it is obviously desirable that a user's mail messages (for
   example) be replicated across multiple machines to provide fault
   tolerance, fragmenting the copies of this data set across multiple
   machines should be avoided unless the data volumes are so large as to
   require it.

7.4.  Encryption of large data blocks

   The encoding scheme is 64-bit clean throughout and thus supports
   containers and frames as large as 18 petabytes.  Larger data volumes
   could be supported through use of 128-bit integer pointers but even
   if the technology to support such data volumes were developed, it is
   highly unlikely anyone would want to represent data sets anywhere
   near this size in a serial format.

   Due to limitations in the design of the encryption schemes that may
   be used (e.g.  AES-GCM), the maximum encrypted frame size is 64GB.
   While this is not currently a major concern for encryption of
   individual data files, it is easy to see situations in which an
   archive of encrypted files could exceed that amount.  One possibility
   would be to define a modification to AES -GCM which caused the
   encryption key to be incremented by a fixed amount after encrypting a
   certain amount of data, though this might well present implementation
   challenges unless the maximum data block size was chosen to be
   deliberately small so as to force code paths to be exercised.
   Another possibility would be to limit the size of encrypted data
   frames by requiring the frame pointer to be no larger than 32 bits
   and require larger data items to be represented as a sequence of
   frames.



Hallam-Baker            Expires January 14, 2019               [Page 15]


Internet-Draft             DARE Data Container                 July 2018


7.5.  Concurrent Writes

   The container format deliberately avoids support for concurrent write
   operations.  Should this be desirable, some mechanism must be
   provided to cache write fragments to an intermediate file and then
   consolidate them for writing to the master log.

8.  Security Considerations

9.  IANA Considerations

10.  Acknowledgements

11.  Appendix A: Examples and Test Vectors

   The data payloads in all the following examples are identical, only
   the authentication and/or encryption is different.

   o  Frame 1..n consists of 300 bytes being the byte sequence 00, 01,
      02, etc.  repeating after 256 bytes.

   For conciseness, the wire format is omitted for examples after the
   first, except where the data payload has been transformed, (i.e.
   encrypted).

11.1.  Simple container

   Here the simple container:

   f4 5b
   f0 59
     7b 0a 20 20 22 49 6e 64   65 78 22 3a 20 30 2c 0a
     20 20 22 43 6f 6e 74 61   69 6e 65 72
     ...
     7d 2c 0a 20 20 22 44 61   74 61 45 6e 63 6f 64 69
     6e 67 22 3a 20 22 4a 53   4f 4e 22 7d
   5b f4


                                 Figure 1

   The header values are:

   Frame 0







Hallam-Baker            Expires January 14, 2019               [Page 16]


Internet-Draft             DARE Data Container                 July 2018


   {
     "Index":0,
     "ContainerType":"List",
     "ContentMeta":{},
     "DataEncoding":"JSON"}

                                 Figure 2

11.2.  Payload and chain digests

   Frame 0

   {
     "Index":0,
     "ContainerType":"Chain",
     "ContentMeta":{},
     "DataEncoding":"JSON"}

                                 Figure 3

11.3.  Merkle Tree

   Frame 0

   {
     "Index":0,
     "ContainerType":"Merkle",
     "ContentMeta":{},
     "DataEncoding":"JSON"}

                                 Figure 4

11.4.  Signed container

11.5.  Encrypted container

   The following example shows a container in which all the frame
   payloads are encrypted under the same master secret established in a
   key agreement specified in the first frame.

   Frame 0










Hallam-Baker            Expires January 14, 2019               [Page 17]


Internet-Draft             DARE Data Container                 July 2018


   {
     "enc":"A256CBC",
     "Salt":"H7RSZXq75pcQY9P1FypMsg",
     "recipients":[{
         "kid":"MDD7L-HSRZY-6DYKQ-7FHIW-PS6I5-VOC4F-A",
         "epk":{
           "PublicKeyDH":{
             "kid":"MCRRC-JW7RV-NETIF-MXOXV-NMXRN-KWNWB-A",
             "Domain":"YE6bnq1MlX5ojaJto6PLP_PEwA",
             "Public":"viI6d03KCERI9vaEazTc4lJeRIIXYRVPWtdHUuWvgmiJNih
     T5-II3nfYzYEWxw1g7bbw_TuhPNV1oABPrLwGThoPQ_F957nuWsub_wbk5x0I-Bkx
     7dE8D1mkQFavx1YnlyRCG8yCXQpS6kdq4pNkHzG5AdS6EhmCguoaFfrGPVuTE9Taw
     ypZvL6K4sSdoQtDyUduyRo4f88ApW9OT0NdmnCzM6_g462BX1DlIKfIEtabjeUjTt
     Dmk44_anDH7HIOVYXO1YWCRtRvElAlRyb85GVhn_QTAFV00OrOZlN5d_7-nAXJRPM
     0eV7tnLv0kjD-QC9N_IfEeMLBtFu0n2DKFA"}},
         "wmk":"vbrTqge3bqMuOunkYugDDC5kOT_MUNOof3Ddr9M0D3gO-I_hMckY7w"}
       ],
     "Index":0,
     "ContainerType":"List",
     "ContentMeta":{},
     "DataEncoding":"JSON"}

                                 Figure 5

   Here are the container bytes.  Note that the content is now encrypted
   and has expanded by 25 bytes.  These are the salt (16 bytes), the AES
   padding (4 bytes) and the JSON-B framing (5 bytes).

   f5 03 2f
   f1 03 27
     7b 0a 20 20 22 65 6e 63   22 3a 20 22 41 32 35 36
     43 42 43 22 2c 0a 20 20   22 53 61 6c
     ...
     7d 2c 0a 20 20 22 44 61   74 61 45 6e 63 6f 64 69
     6e 67 22 3a 20 22 4a 53   4f 4e 22 7d
   2f 03 f5


                                 Figure 6

   The following example shows a container in which all the frame
   payloads are encrypted under separate key agreements specified in the
   payload frames.








Hallam-Baker            Expires January 14, 2019               [Page 18]


Internet-Draft             DARE Data Container                 July 2018


12.  Appendix B

   public long PreviousFrame (long Frame) {
       long x2 = Frame + 1;
       long d = 1;

       while (x2 > 0) {
           if ((x2 & 1) == 1) {
               return x2 == 1 ? (d / 2) - 1 : Frame - d;
               }
           d = d * 2;
           x2 = x2 / 2;
           }
       return 0;
       }

                                 Figure 7

13.  References

13.1.  Normative References

   [draft-hallambaker-dare-message]
              Hallam-Baker, P., "Data At Rest Encryption Part 1: DARE
              Message", draft-hallambaker-dare-message-00 (work in
              progress), May 2018.

   [draft-hallambaker-jsonbcd]
              Hallam-Baker, P., "Binary Encodings for JavaScript Object
              Notation: JSON-B, JSON-C, JSON-D", draft-hallambaker-
              jsonbcd-12 (work in progress), May 2018.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997.

   [RFC7159]  Bray, T., "The JavaScript Object Notation (JSON) Data
              Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March
              2014.

   [RFC7515]  Jones, M., Bradley, J., and N. Sakimura, "JSON Web
              Signature (JWS)", RFC 7515, DOI 10.17487/RFC7515, May
              2015.

   [RFC7516]  Jones, M. and J. Hildebrand, "JSON Web Encryption (JWE)",
              RFC 7516, DOI 10.17487/RFC7516, May 2015.





Hallam-Baker            Expires January 14, 2019               [Page 19]


Internet-Draft             DARE Data Container                 July 2018


13.2.  Informative References

   [BLOCKCHAIN]
              Chain.com, "Blockchain Specification".

   [draft-hallambaker-mesh-architecture]
              Hallam-Baker, P., "Mathematical Mesh: Architecture",
              draft-hallambaker-mesh-architecture-04 (work in progress),
              September 2017.

   [RFC5652]  Housley, R., "Cryptographic Message Syntax (CMS)", STD 70,
              RFC 5652, DOI 10.17487/RFC5652, September 2009.

   [ZIPFILE]  PKWARE Inc, "APPNOTE.TXT - .ZIP File Format
              Specification", October 2014.

13.3.  URIs

   [1] http://mathmesh.com/Documents/draft-hallambaker-dare-
       container.html

   [2] http://mathmesh.com/Documents/draft-hallambaker-dare-
       container.html

   [3] http://mathmesh.com/Documents/draft-hallambaker-dare-
       container.html

   [4] http://mathmesh.com/Documents/draft-hallambaker-dare-
       container.html

Author's Address

   Phillip Hallam-Baker
   Comodo Group Inc.

   Email: philliph@comodo.com















Hallam-Baker            Expires January 14, 2019               [Page 20]


Html markup produced by rfcmarkup 1.127, available from https://tools.ietf.org/tools/rfcmarkup/