[Docs] [txt|pdf] [Tracker] [WG] [Email] [Diff1] [Diff2] [Nits]
Versions: (draft-thurlow-nfsv4-repl-mig-design)
00
Network Working Group Robert Thurlow
Internet Draft December 2002
Document: draft-ietf-nfsv4-repl-mig-design-00.txt
Server-to-Server Replication/Migration Protocol Design Principles
Status of this Memo
This document is an Internet-Draft and is subject to all provisions
of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet- Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/1id-abstracts.html
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
Discussion and suggestions for improvement are requested. This
document will expire in June, 2003. Distribution of this draft is
unlimited.
Abstract
NFS Version 4 [RFC3010] provided support for client/server
interactions to support replication and migration, but left
unspecified how replication and migration would be done. This
document discusses the nature of a protocol to be used to transfer
filesystem data and metadata for use with replication and migration
services for NFS Version 4.
Expires: June 2003 [Page 1]
Title Replication/Migration Design Principles December 2002
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Definitions of terms . . . . . . . . . . . . . . . . . . . 3
1.1.1. Replication . . . . . . . . . . . . . . . . . . . . . . 3
1.1.2. Migration . . . . . . . . . . . . . . . . . . . . . . . 3
1.2. Current practice . . . . . . . . . . . . . . . . . . . . . 4
1.3. The problem . . . . . . . . . . . . . . . . . . . . . . . 4
1.3.1. NFS clients today . . . . . . . . . . . . . . . . . . . 4
1.3.2. NFS Version 4 . . . . . . . . . . . . . . . . . . . . . 5
1.4. The need for a transfer protocol . . . . . . . . . . . . . 5
2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1. Interoperability . . . . . . . . . . . . . . . . . . . . . 5
2.2. Transparency . . . . . . . . . . . . . . . . . . . . . . . 5
2.3. Security . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4. Efficiency . . . . . . . . . . . . . . . . . . . . . . . . 6
2.5. Scalability . . . . . . . . . . . . . . . . . . . . . . . 6
3. Non-requirements . . . . . . . . . . . . . . . . . . . . . . 6
4. Design considerations . . . . . . . . . . . . . . . . . . . 7
4.1. Basic structure . . . . . . . . . . . . . . . . . . . . . 7
4.2. Administrative Control . . . . . . . . . . . . . . . . . . 7
4.3. Basic environment . . . . . . . . . . . . . . . . . . . . 7
4.4. Handling file changes . . . . . . . . . . . . . . . . . . 7
4.5. Replication model . . . . . . . . . . . . . . . . . . . . 8
5. Security considerations . . . . . . . . . . . . . . . . . . 8
6. Implementation considerations . . . . . . . . . . . . . . . 8
6.1. Filehandle preservation . . . . . . . . . . . . . . . . . 8
6.2. Data transfer phases . . . . . . . . . . . . . . . . . . . 9
6.3. Operation on filesystem subsets . . . . . . . . . . . . . 9
7. Difficult issues . . . . . . . . . . . . . . . . . . . . . 10
7.1. Transparency violations . . . . . . . . . . . . . . . . 10
7.2. Directory access . . . . . . . . . . . . . . . . . . . . 10
8. Bibliography . . . . . . . . . . . . . . . . . . . . . . . 11
9. Author's Address . . . . . . . . . . . . . . . . . . . . . 12
Expires: June 2003 [Page 2]
Title Replication/Migration Design Principles December 2002
1. Introduction
Though used in different circumstances, replication of data and
migration of data share a common problem: how to accurately transfer
data (which may be in use by applications) from one location to
another with reasonable bandwidth usage and in reasonable time.
Years ago, this was done by taking storage offline (or at least
preventing write access), making a tape copy of the data files, and
walking it to the new machine, after warning the twenty or so people
who cared about it. Networks reduced wear on sneakers, but many of
the data formats we use for filesystem copies tend to be little
improved - they are either lowest-common-denominator standards like
"tar" and "cpio" or internal dump formats which are non-standard.
Today, with distributed filesystems like NFS Version 4, richer
metadata including Access Control Lists (ACLs) and extended
attributes, and potential users all over the enterprise and the
Internet, we need something better - a standard, complete and
extensible protocol to transfer filesystems. Functionality like this
has been available in AFS and DCE/DFS for many years, though the
protocols have not been published.
Though data replication and transfer are needed in many areas, this
document will focus primarily on solving the problem of providing
replication and migration support between NFS Version 4 servers. It
is assumed that the reader has familiarity with NFS Version 4
[RFC3010].
1.1. Definitions of terms
1.1.1. Replication
Filesystem replication is the creation of a functionally identical
copy of a filesystem, usually to enhance availability or provide for
redundancy or disaster recovery. For example, a company may set up
replicas of a customer database accessed by employees in different
geographies. The data sets are often read-only, and initial creation
of a replica is not as interesting a problem as maintaining the
replica efficiently over time via incremental updates, which will
likely be set up to push automatically.
1.1.2. Migration
Filesystem migration is the moving of a filesystem to another server
for load balancing purposes or because a user or server has moved.
For example, a user may have moved from one building to another, or
across the country, and want his home directory to follow him, or it
may just be time to decommission an old server and move data to a new
Expires: June 2003 [Page 3]
Title Replication/Migration Design Principles December 2002
one. Only one data transfer is done, and it is important for this to
be done efficiently and with the lowest possible impact on users.
1.2. Current practice
System administrators typically have several options available to
them to replicate or migrate files, but none of them cover the
problem space:
o The pax, cpio and tar tape archivers as defined by IEEE 1003.1
or ISO/IEC 9945-1 are often used without tape over a network for
data transfer; these support only generic Unix-specific metadata
and do not support ACLs or extended attributes
o The rdist (http://www.magnicomp.com/rdist) and rsync
(http://samba.anu.edu.au/rsync) applications focus on
propagating changes to replicas, but are documented only by
source code, are not available on all platforms, and do not
support more than generic Unix-specific metadata
o "cp -r" or its equivalent over NFS Version 4 could work in cases
where capabilities of servers were the same, but if the
destination did not support ACLs or extended attributes, would
it do what the user wanted?
o Most server filesystems have a "dump" format of some kind, which
can preserve all data and metadata as long as there are no
architectural differences in the servers
o Most server vendors have products which can keep replicas in
sync by monitoring changes at the block level below the server
filesystem, which are again inherently tied to one architecture
o Most of the above tools are not set up to properly deal with
exotic metadata which may be present on filesystems like MacOS's
HFS or Windows' NTFS, which can result in loss of data even when
transferring to the same platform
1.3. The problem
1.3.1. NFS clients today
Replication and migration events both cause problems for NFS clients,
which may have applications operating on data when the event occurs.
Past versions of NFS did not provide any support in protocol for the
client, and typical clients did not even attempt to find another
Expires: June 2003 [Page 4]
Title Replication/Migration Design Principles December 2002
replica which might provide service.
1.3.2. NFS Version 4
NFS Version 4 [RFC3010] introduced some extra error codes and
attributes to improve this situation. For replication, the new
"fs_locations" attribute could be retrived by the client to determine
if multiple locations were available, so that when a server became
unavailable, the client could fail over to a new location without
hoping updated information was available in its name service. For
migration and in the case of a decommissioned replica, the
NFS4ERR_MOVED error would inform a client that it should consult
"fs_locations" and make contact with a new server responsible for the
data. In both cases, a client is required to establish a
relationship with a new server, which may involve state recovery and
using saved pathname information to discover new filehandles.
1.4. The need for a transfer protocol
To support NFS Version 4, a method is needed to transfer functionally
complete filesystem data from one server to another. The
shortcomings listed previously in the common tools in use demonstrate
that there is value in a standard protocol to transfer filesystem
data.
2. Requirements
The requirements for a replication and migration protocol are to be
addressed in a separate document, but are approximately these:
2.1. Interoperability
The replication/migration protocol must first and foremost be one
which can potentially be implemented on any server. Several vendors
already have a replication mechanism in their product lines which
takes advantage of known properties of their servers to replicate at
the block level, but this is inherently tied to one system.
2.2. Transparency
When a client has been using a file which has been migrated, it
should be able to detect this and recover the file state on the new
server without applications needing to take action. Similarly, when
a client has availability problems with a particular replica, it
should be able to adapt to the use of the new replica without
application involvement. This implies that, as far as possible, the
replication/migration protocol must copy all filesystem data, as much
Expires: June 2003 [Page 5]
Title Replication/Migration Design Principles December 2002
metadata as possible, and all non-recoverable transient state such as
outstanding lock and delegation state, completely and correctly. It
is acceptable that the client must recover some state as occurs in
the event of a server reboot.
2.3. Security
NFS Version 4 supported strong mandatory-to-implement security
mechanisms to protect the integrity and privacy of file data and
metadata. The replication/migration protocol must specify
mandatory-to-implement security to protect data in transit, and
provide a security payload and an encryption mechanism to ensure
strong security for each message. It is expected that the security
mechanisms will correlate well with NFS Version 4 [RFC3010].
2.4. Efficiency
The replication/migration protocol must get the job of data movement
done as efficiently as possible in terms of both bandwidth and time.
Components of this are:
o the protocol will conserve bandwidth by streaming data in large
blocks with limited header overhead
o the protocol will transfer changed regions in files rather than
complete files whenever possible
o the protocol will permit restart in the event of a server
failure or lost connection
2.5. Scalability
The replication/migration protocol must be able to handle both huge
files and huge filesystems, while maintaining low enough overhead to
work well with small filesystems as well.
3. Non-requirements
There have been discussions about the things a good replication
protocol could do which are not considered part of the scope of this
work at this time. Some of these things could be specified by future
RFCs, so we do not wish to preclude them from being done later on.
These non-requirements include:
o being an "rdist" or "rsync" replacement
o being a tool to permit unprivileged users to copy file trees
Expires: June 2003 [Page 6]
Title Replication/Migration Design Principles December 2002
o being used for replication of other types of data
4. Design considerations
4.1. Basic structure
For best performance, a replication/migration protocol should be able
to move large amounts of data without frequent small packets in the
direction of data movement. Use of RPC [RFC1831] and XDR [RFC1832]
will be subject to analysis of their overhead for this purpose.
However they are formatted, groups of messages would probably
include:
o Initialization and negotiation messages
o Filesystem information messages
o Data transfer messages
o Finalization messages
4.2. Administrative Control
The replication and migration protocol should include nothing
specifying how an administrative user contacts a server to initiate
replication or migration. A separate document should define a
mechanism suitable for this purpose.
4.3. Basic environment
The replication/migration protocol should be available to a
privileged context on a well-known TCP port on an NFSv4 server, able
to authenticate and act on control messages from administration
clients and general messages from other servers.
4.4. Handling file changes
For replication, it should be possible to handle large files changed
in small ways without transferring the entire file. The protocol
needs to be able to express changes to byte ranges within a file;
ideally, the server will be able to extract such changes from some
kind of change log or from internal filesystem data. However, this
may not be practical. The existence of "rdist" shows that a
bidirectional protocol can determine differences in files at a
reasonable bandwidth cost, and it would be good for the
replication/migration protocol to be able to operate in this mode.
Expires: June 2003 [Page 7]
Title Replication/Migration Design Principles December 2002
4.5. Replication model
Replication is usually set up as a series of read-only replicas, with
the master copy of the filesystem generally unaccessible to the
client or accessible through a different mount point. It is possible
to envision a case where, along with several read-only replicas, a
single writer is available and "marked" as such in the fs_locations
attribute. The client would have to ensure that all reads and writes
were directed to the writable copy from the time a particular file on
the filesystem was first written to the time the client ceased caring
about the file. This is considered beyond our current scope at this
time.
5. Security considerations
NFS Version 4 is the primary impetus behind a replication/migration
protocol, so this protocol should mandate a strong security scheme
and security negotiation in a manner compatible with NFS Version 4.
Since NFS Version 4 specifies RPCSEC_GSS [RFC2203], which in turn
builds on GSS-API [RFC2078], it makes sense for a
replication/migration protocol to specify RPCSEC_GSS if it is based
on RPC, and GSS-API if it is not based on RPC. Kerberos Version 5
will be used as described in [RFC1964] to provide one security
framework. The LIPKEY GSS-API mechanism described in [RFC2847] will
be used to provide for the use of user password and server public
key. An initial message exchange will permit security negotiation.
The replication/migration protocol will also specify a NULL security
mechanism to optimize its performance when used with strong host-
based security mechanism such as SSH and IPSec.
6. Implementation considerations
6.1. Filehandle preservation
Filahandles are the basic shorthand used by clients to perform most
operations on files. The are opaque to the client, but are usually
derived from:
o the fsid of the filesystem
o the fileid or "inode number" of the directory shared by the
server
o the fileid or "inode number" of the file
o the "generation number", an internal field to support inode
reuse.
Expires: June 2003 [Page 8]
Title Replication/Migration Design Principles December 2002
It is, in some circumstances, desireable to preserve persistant
filehandles across a replication or migration event. The most likely
circumstance for this is when both servers are of the same
architecture, and when the destination server can assign values to
these fields as data is accepted. To support this case, the
filehandle should be available as an attribute which can be passed to
the new server. Some operating environments will not have interfaces
to support access to this data or a way to recreate it anew, so this
should be negotiated so that this data is not sent unnecessarily.
Even if a server implementation can transfer and accept persistent
filehandles, it must ensure that the client is not falsely promised
that this will happen. [RFC3010] specifies that a server may migrate
a filesystem with persistent filehandles as long as the new server
also uses persistent filehandles and the same filehandles will
correspond to the same files after migration. In the general case,
the decision to migrate a filesystem, perhaps to a heterogeneous
server with different filehandles, will be made after clients have
accessed filesystems and learned of the value of the "fh_expire_type"
attribute. Thus it seems necessary that servers return an
"fh_expire_type" of at least FH4_VOL_MIGRATION so that clients will
always store partial pathnames for later use. It is possible for
clients to attempt to use pre-event filehandles with the new server
in the hope that persistent filehandles would have been transferred
intact, but there is no way for the server to promise this unless it
will never transfer to a server of a different implementation.
6.2. Data transfer phases
For both replication and migration, transfer most generally happens
in two phases: first, the bulk of the data is copied to the target
while access to the source filesystem continues, and second, changes
made since the start of the first phase are transferred while write
access to the source filesystem is curtailed. This reduces the
window during which clients will see restrictions, at the cost of
needing a method to lock out writes to files in the file tree. For
replication, it would be possible to bypass locking by the use of
multiple point-in-time copies ("snapshots"), since the delta
represented by each snapshot could be used to update the replicas.
6.3. Operation on filesystem subsets
When NFSv4 clients discover that they must react to a replication or
migration event, [RFC3010] states that they will recover at the
granularity of an entire filesystem, i.e. a set of files sharing the
same "fsid" attribute. It is possible that this protocol could be
useful for splitting up of large filesystems to permit them to be
replicated and migrated separately. This can most easily be done if
Expires: June 2003 [Page 9]
Title Replication/Migration Design Principles December 2002
the server can arrange to return distinct "fsid"s for subdirectories
of what it manages as a single filesystem.
7. Difficult issues
7.1. Transparency violations
When being used between servers that are sufficiently different, it
may be impossible for the new server to support some metadata
enumerated in the data stream, or it may be that metadata critical to
the new server are not supported on the old. When this happens, the
client may notice and react badly to the loss of transparency.
Sources of this kind of problem include:
o Filename encoding differences
o Attributes supported on one server and not the other
o A failure of atomicity during transfer
o Incomplete or no transfer of locking, delegation and other state
7.2. Directory access
When a directory is read, a series of RPCs is used to get the entries
in small parts. The sequence of RPCs is tied together by a "cookie"
returned by the server in each reply and used by the client in the
next request. The sequence can be interrupted by a replication or
migration event, which can lead to NFS4ERR_BAD_COOKIE on the new
server, even if the servers are the same architecture, due to
different orders of creation of the directory entries and compaction.
Expires: June 2003 [Page 10]
Title Replication/Migration Design Principles December 2002
8. Bibliography
[RFC1831]
R. Srinivasan, "RPC: Remote Procedure Call Protocol Specification
Version 2", RFC1831, August 1995.
[RFC1832]
R. Srinivasan, "XDR: External Data Representation Standard", RFC1832,
August 1995.
[RFC3010]
S. Shepler, B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M.
Eisler, D. Noveck, "NFS version 4 Protocol", RFC3010, December 2000.
[RDIST]
MagniComp, Inc., "The RDist Home Page",
http://www.magnicomp.com/rdist.
[RSYNC]
The Samba Team, "The rsync web pages", http://samba.anu.edu.au/rsync.
Expires: June 2003 [Page 11]
Title Replication/Migration Design Principles December 2002
9. Author's Address
Address comments related to this memorandum to:
nfsv4-wg@sunroof.eng.sun.com
Robert Thurlow
Sun Microsystems, Inc.
500 Eldorado Boulevard, UBRM05-171
Broomfield, CO 80021
Phone: 877-718-3419
E-mail: robert.thurlow@sun.com
Expires: June 2003 [Page 12]
Html markup produced by rfcmarkup 1.127, available from
https://tools.ietf.org/tools/rfcmarkup/