[Docs] [txt|pdf] [Tracker] [WG] [Email] [Diff1] [Diff2] [Nits]

Versions: 00 01 02 03 04 05 RFC 2525

Network Working Group                                  V. Paxson, Editor
Internet Draft                                                 M. Allman
                                                               S. Dawson
                                                              I. Heavens
                                                                 B. Volz
Expiration Date: May 1998                                  November 1997


                   Known TCP Implementation Problems
                    <draft-ietf-tcpimpl-prob-02.txt>


1. Status of this Memo

   This document is an Internet  Draft.   Internet  Drafts  are  working
   documents  of  the Internet Engineering Task Force (IETF), its areas,
   and its working groups.  Note that other groups may  also  distribute
   working documents as Internet Drafts.

   Internet Drafts are draft  documents  valid  for  a  maximum  of  six
   months, and may be updated, replaced, or obsoleted by other documents
   at any time.  It is inappropriate to use Internet Drafts as reference
   material or to cite them other than as ``work in progress''.

   To learn the current status of any Internet Draft, please  check  the
   ``1id-abstracts.txt'' listing contained in the Internet Drafts shadow
   directories  on  ftp.is.co.za   (Africa),   nic.nordu.net   (Europe),
   munnari.oz.au  (Pacific  Rim),  ds.internic.net  (US  East Coast), or
   ftp.isi.edu (US West Coast).

   This memo provides information for the Internet community.  This memo
   does  not  specify an Internet standard of any kind.  Distribution of
   this memo is unlimited.


2. Introduction

   This memo catalogs a number of  known  TCP  implementation  problems.
   The  goal  in  doing  so  is  to  improve  conditions in the existing
   Internet by enhancing the quality of current TCP/IP  implementations.
   It  is  hoped  that  both  performance  and correctness issues can be
   resolved by making implementors  aware  of  the  problems  and  their
   solutions.   In  the  long term, it is hoped that this will provide a
   reduction  in  unnecessary  traffic  on  the  network,  the  rate  of
   connection  failures  due  to  protocol  errors,  and load on network
   servers due to time spent processing  both  unsuccessful  connections
   and  retransmitted  data.   This will help to ensure the stability of
   the global Internet.



Paxson, Editor                                                  [Page 1]


ID                 Known TCP Implementation Problems       November 1997


   Each problem is defined as follows:


   Name The name associated with the problem.  In this memo, the name is
        given as a subsection heading.

   Classification
        One or more problem categories for which the problem is  classi-
        fied.   Categories  used  so far: "congestion control", "perfor-
        mance", "reliability", "resource  management".   Others  antici-
        pated: "security", "interoperability", "configuration".

   Description
        A definition of the problem, succinct  but  including  necessary
        background material.

   Significance
        A brief summary of the sorts of environments for which the prob-
        lem is significant.

   Implications
        Why the problem is viewed as a problem.

   Relevant RFCs
        Brief discussion of the RFCs with respect to which  the  problem
        is viewed as an implementation error.

   Trace file demonstrating the problem
        One or more ASCII trace  files  demonstrating  the  problem,  if
        applicable.   These  may  in the future be replaced with URLs to
        on-line traces.

   Trace file demonstrating correct behavior
        One or more examples of how correct behavior appears in a trace,
        if applicable.  These may in the future be replaced with URLs to
        on-line traces.

   References
        References that further discuss the problem.

   How to detect
        How to test an implementation to see if it exhibits the problem.
        This  discussion may include difficulties and subtleties associ-
        ated with causing the  problem  to  manifest  itself,  and  with
        interpreting  traces  to  detect the presence of the problem (if
        applicable).  In the future, this may include URLs for  diagnos-
        tic tools.




Paxson, Editor                                                  [Page 2]


ID                 Known TCP Implementation Problems       November 1997


   How to fix
        For known causes of the problem, how to correct the  implementa-
        tion.

   Implementation specifics
        If it is viewed as beneficial to document particular implementa-
        tions exhibiting the problem, and if the corresponding implemen-
        tors approve, then this section gives  the  specifics  of  those
        implementations,  along with a contact address for the implemen-
        tors.


3. Known implementation problems


3.1. No initial slow start

Classification
     Congestion control

Description
     When a TCP begins transmitting data, it is required  by  RFC  1122,
     4.2.2.15,  to  engage in a "slow start" by initializing its conges-
     tion window, cwnd, to one packet (one segment of the maximum size).
     It  subsequently  increases  cwnd  by  one  packet  for each ACK it
     receives for new data.  The minimum  of  cwnd  and  the  receiver's
     advertised  window  bounds  the highest sequence number the TCP can
     transmit.  A TCP that fails to initialize  and  increment  cwnd  in
     this fashion exhibits "No initial slow start".

Significance
     In congested environments, detrimental to the performance of  other
     connections, and possibly to the connection itself.

Implications
     A TCP failing to slow start when beginning a connection results  in
     traffic  bursts  that  can stress the network, leading to excessive
     queueing delays and packet loss.

     Implementations exhibiting this problem might do  so  because  they
     suffer  from  the  general  problem  of  not including the required
     congestion window.  These implementations will also suffer from "No
     slow start after retransmission timeout".

     There are different shades of "No initial slow  start".   From  the
     perspective  of  stressing  the  network, the worst is a connection
     that simply always sends based on the receiver's advertised window,
     with  no  notion of a separate congestion window.  Some other forms



Paxson, Editor                                                  [Page 3]


ID                 Known TCP Implementation Problems       November 1997


     are described in "Uninitialized CWND" and "Initial CWND of 2  pack-
     ets".


Relevant RFCs
     RFC 1122 requires use of slow start.  RFC 2001 gives the  specifics
     of slow start.

Trace file demonstrating it
     Made using tcpdump/BPF recording at the connection  responder.   No
     losses reported.

     10:40:42.244503 B > A: S 1168512000:1168512000(0) win 32768
                             <mss 1460,nop,wscale 0> (DF) [tos 0x8]
     10:40:42.259908 A > B: S 3688169472:3688169472(0)
                             ack 1168512001 win 32768 <mss 1460>
     10:40:42.389992 B > A: . ack 1 win 33580 (DF) [tos 0x8]
     10:40:42.664975 A > B: P 1:513(512) ack 1 win 32768
     10:40:42.700185 A > B: . 513:1973(1460) ack 1 win 32768
     10:40:42.718017 A > B: . 1973:3433(1460) ack 1 win 32768
     10:40:42.762945 A > B: . 3433:4893(1460) ack 1 win 32768
     10:40:42.811273 A > B: . 4893:6353(1460) ack 1 win 32768
     10:40:42.829149 A > B: . 6353:7813(1460) ack 1 win 32768
     10:40:42.853687 B > A: . ack 1973 win 33580 (DF) [tos 0x8]
     10:40:42.864031 B > A: . ack 3433 win 33580 (DF) [tos 0x8]

     After the third packet, the connection is established.  A, the con-
     nection responder, begins transmitting to B, the connection initia-
     tor.  Host A quickly sends 6 packets comprising  7812  bytes,  even
     though  the SYN exchange agreed upon an MSS of 1460 bytes (implying
     an initial congestion window  of  1  segment  corresponds  to  1460
     bytes), and so A should have sent at most 1460 bytes.

     The ACKs sent by B to A in the last two lines  indicate  that  this
     trace  is  not a measurement error (slow start really occurring but
     the corresponding ACKs having been dropped by the packet filter).

     A second trace confirmed that the problem is repeatable.


Trace file demonstrating correct behavior

     Made using tcpdump/BPF recording at the connection originator.   No
     losses reported.

     12:35:31.914050 C > D: S 1448571845:1448571845(0) win 4380 <mss 1460>
     12:35:32.068819 D > C: S 1755712000:1755712000(0) ack 1448571846 win 4096
     12:35:32.069341 C > D: . ack 1 win 4608



Paxson, Editor                                                  [Page 4]


ID                 Known TCP Implementation Problems       November 1997


     12:35:32.075213 C > D: P 1:513(512) ack 1 win 4608
     12:35:32.286073 D > C: . ack 513 win 4096
     12:35:32.287032 C > D: . 513:1025(512) ack 1 win 4608
     12:35:32.287506 C > D: . 1025:1537(512) ack 1 win 4608
     12:35:32.432712 D > C: . ack 1537 win 4096
     12:35:32.433690 C > D: . 1537:2049(512) ack 1 win 4608
     12:35:32.434481 C > D: . 2049:2561(512) ack 1 win 4608
     12:35:32.435032 C > D: . 2561:3073(512) ack 1 win 4608
     12:35:32.594526 D > C: . ack 3073 win 4096
     12:35:32.595465 C > D: . 3073:3585(512) ack 1 win 4608
     12:35:32.595947 C > D: . 3585:4097(512) ack 1 win 4608
     12:35:32.596414 C > D: . 4097:4609(512) ack 1 win 4608
     12:35:32.596888 C > D: . 4609:5121(512) ack 1 win 4608
     12:35:32.733453 D > C: . ack 4097 win 4096


References
     This problem is documented in [Paxson97].

How to detect
     For implementations always manifesting this problem,  it  shows  up
     immediately  in  a  packet trace or a sequence plot, as illustrated
     above.

How to fix
     If the root problem is that the implementation lacks a notion of  a
     congestion  window,  then  unfortunately  this requires significant
     work to fix.  However, doing so is important, as  such  implementa-
     tions also exhibit "No slow start after retransmission timeout".


3.2. No slow start after retransmission timeout

Classification
     Congestion control

Description
     When a TCP experiences a retransmission timeout, it is required  by
     RFC  1122,  4.2.2.15, to engage in "slow start" by initializing its
     congestion window, cwnd, to one packet (one segment of the  maximum
     size).   It  subsequently increases cwnd by one packet for each ACK
     it  receives  for  new  data  until  it  reaches  the   "congestion
     avoidance"  threshold,  ssthresh,  at  which  point  the congestion
     avoidance algorithm for updating the window takes over.  A TCP that
     fails  to  enter  slow start upon a timeout exhibits "No slow start
     after retransmission timeout".

Significance



Paxson, Editor                                                  [Page 5]


ID                 Known TCP Implementation Problems       November 1997


     In congested environments, severely detrimental to the  performance
     of other connections, and also the connection itself.

Implications
     Entering slow start upon timeout forms one of the  cornerstones  of
     Internet  congestion  stability,  as  outlined in [Jacobson88].  If
     TCPs fail to do so,  the  network  becomes  at  risk  of  suffering
     "congestion collapse" [RFC896].

Relevant RFCs
     RFC 1122 requires use of slow start after loss.  RFC 2001 gives the
     specifics  of  how  to  implement  slow  start.   RFC 896 describes
     congestion collapse.

     The retransmission timeout discussed here should  not  be  confused
     with  the  separate  "fast  recovery" retransmission mechanism dis-
     cussed in RFC 2001.


Trace file demonstrating it
     Made using tcpdump/BPF recording at the sending TCP (A).  No losses
     reported.

     10:40:59.090612 B > A: . ack 357125 win 33580 (DF) [tos 0x8]
     10:40:59.222025 A > B: . 357125:358585(1460) ack 1 win 32768
     10:40:59.868871 A > B: . 357125:358585(1460) ack 1 win 32768
     10:41:00.016641 B > A: . ack 364425 win 33580 (DF) [tos 0x8]
     10:41:00.036709 A > B: . 364425:365885(1460) ack 1 win 32768
     10:41:00.045231 A > B: . 365885:367345(1460) ack 1 win 32768
     10:41:00.053785 A > B: . 367345:368805(1460) ack 1 win 32768
     10:41:00.062426 A > B: . 368805:370265(1460) ack 1 win 32768
     10:41:00.071074 A > B: . 370265:371725(1460) ack 1 win 32768
     10:41:00.079794 A > B: . 371725:373185(1460) ack 1 win 32768
     10:41:00.089304 A > B: . 373185:374645(1460) ack 1 win 32768
     10:41:00.097738 A > B: . 374645:376105(1460) ack 1 win 32768
     10:41:00.106409 A > B: . 376105:377565(1460) ack 1 win 32768
     10:41:00.115024 A > B: . 377565:379025(1460) ack 1 win 32768
     10:41:00.123576 A > B: . 379025:380485(1460) ack 1 win 32768
     10:41:00.132016 A > B: . 380485:381945(1460) ack 1 win 32768
     10:41:00.141635 A > B: . 381945:383405(1460) ack 1 win 32768
     10:41:00.150094 A > B: . 383405:384865(1460) ack 1 win 32768
     10:41:00.158552 A > B: . 384865:386325(1460) ack 1 win 32768
     10:41:00.167053 A > B: . 386325:387785(1460) ack 1 win 32768
     10:41:00.175518 A > B: . 387785:389245(1460) ack 1 win 32768
     10:41:00.210835 A > B: . 389245:390705(1460) ack 1 win 32768
     10:41:00.226108 A > B: . 390705:392165(1460) ack 1 win 32768
     10:41:00.241524 B > A: . ack 389245 win 8760 (DF) [tos 0x8]




Paxson, Editor                                                  [Page 6]


ID                 Known TCP Implementation Problems       November 1997


     The first packet indicates the ack point is 357125.  130 msec after
     receiving  the  ACK,  A  transmits  the packet after the ACK point,
     357125:358585.  640 msec after this  transmission,  it  retransmits
     357125:358585,  in  an  apparent  retransmission  timeout.  At this
     point, A's cwnd should be one MSS, or 1460 bytes, as A enters  slow
     start.  The trace is consistent with this possibility.

     B replies with an ACK of 364425, indicating that  A  has  filled  a
     sequence  hole.   At  this  point, A's cwnd should be 1460*2 = 2920
     bytes, since in slow start receiving an ACK advances cwnd  by  MSS.
     However,  A  then  launches 19 consecutive packets, which is incon-
     sistent with slow start.

     A second trace confirmed that the problem is repeatable.


Trace file demonstrating correct behavior
     Made using tcpdump/BPF recording at the sending TCP (C).  No losses
     reported.

     12:35:48.442538 C > D: P 465409:465921(512) ack 1 win 4608
     12:35:48.544483 D > C: . ack 461825 win 4096
     12:35:48.703496 D > C: . ack 461825 win 4096
     12:35:49.044613 C > D: . 461825:462337(512) ack 1 win 4608
     12:35:49.192282 D > C: . ack 465921 win 2048
     12:35:49.192538 D > C: . ack 465921 win 4096
     12:35:49.193392 C > D: P 465921:466433(512) ack 1 win 4608
     12:35:49.194726 C > D: P 466433:466945(512) ack 1 win 4608
     12:35:49.350665 D > C: . ack 466945 win 4096
     12:35:49.351694 C > D: . 466945:467457(512) ack 1 win 4608
     12:35:49.352168 C > D: . 467457:467969(512) ack 1 win 4608
     12:35:49.352643 C > D: . 467969:468481(512) ack 1 win 4608
     12:35:49.506000 D > C: . ack 467969 win 3584

     After C transmits the first packet shown to D, it takes  no  action
     in  response  to  D's  ACKs  for  461825,  because the first packet
     already reached the advertised window limit  of  4096  bytes  above
     461825.    600   msec   after  transmitting  the  first  packet,  C
     retransmits  461825:462337,  presumably  due  to  a  timeout.   Its
     congestion window is now MSS (512 bytes).

     D acks 465921, indicating that C's retransmission filled a sequence
     hole.   This  ACK advances C's cwnd from 512 to 1024.  Very shortly
     after, D acks 465921 again in order to update  the  offered  window
     from  2048 to 4096.  This ACK does not advance cwnd since it is not
     for new data.  Very shortly after, C responds to the newly enlarged
     window  by  transmitting  two packets.  D acks both, advancing cwnd
     from 1024 to 1536.  C in turn transmits three packets.



Paxson, Editor                                                  [Page 7]


ID                 Known TCP Implementation Problems       November 1997


References
     This problem is documented in [Paxson97].


How to detect
     Packet loss is common enough in the Internet that generally  it  is
     not  difficult to find an Internet path that will force retransmis-
     sion due to packet loss.

     If the effective window prior to loss  is  large  enough,  however,
     then  the  TCP  may  retransmit using the "fast recovery" mechanism
     described in RFC 2001.  In a packet trace, the  signature  of  fast
     recovery  is  that  the packet retransmission occurs in response to
     the receipt of three duplicate ACKs, and subsequent duplicate  ACKs
     may  lead to the transmission of new data, above both the ack point
     and the highest sequence transmitted so far.  An absence  of  three
     duplicate  ACKs  prior  to  retransmission  suffices to distinguish
     between timeout and fast recovery retransmissions.  In the face  of
     only  observing  fast recovery retransmissions, generally it is not
     difficult to repeat the data transfer  until  observing  a  timeout
     retransmission.

     Once armed with a trace exhibiting a timeout retransmission, deter-
     mining  whether the TCP follows slow start is done by computing the
     correct progression of cwnd and comparing it to the amount of  data
     transmited by the TCP subsequent to the timeout rtransmission.


How to fix
     If the root problem is that the implementation lacks a notion of  a
     congestion  window,  then  unfortunately  this requires significant
     work to fix.  However, doing so is critical, for  reasons  outlined
     above.


3.3. Inconsistent retransmission

Classification
     Reliability

Description
     If, for a given sequence number, a  sending  TCP  retransmits  dif-
     ferent  data  than previously sent for that sequence number, then a
     strong possibility arises that the receiving TCP will reconstruct a
     different  byte  stream  than that sent by the sending application,
     depending on which instance of  the  sequence  number  it  accepts.
     Such a sending TCP exhibits "Inconsistent retransmission".




Paxson, Editor                                                  [Page 8]


ID                 Known TCP Implementation Problems       November 1997


Significance
     Critical for all environments.

Implications
     Reliable delivery of data is a fundamental property of TCP.

Relevant RFCs
     RFC 793, section 1.5, discusses the central role of reliability  in
     TCP operation.

Trace file demonstrating it
     Made using tcpdump/BPF recording at  the  receiving  TCP  (B).   No
     losses reported.

     12:35:53.145503 A > B: FP 90048435:90048461(26) ack 393464682 win 4096
                                          4500 0042 9644 0000
                      3006 e4c2 86b1 0401 83f3 010a b2a4 0015
                      055e 07b3 1773 cb6a 5019 1000 68a9 0000
     data starts here>504f 5254 2031 3334 2c31 3737*2c34 2c31
                      2c31 3738 2c31 3635 0d0a
     12:35:53.146479 B > A: R 393464682:393464682(0) win 8192
     12:35:53.851714 A > B: FP 90048429:90048463(34) ack 393464682 win 4096
                                          4500 004a 965b 0000
                      3006 e4a3 86b1 0401 83f3 010a b2a4 0015
                      055e 07ad 1773 cb6a 5019 1000 8bd3 0000
     data starts here>5041 5356 0d0a 504f 5254 2031 3334 2c31
                      3737*2c31 3035 2c31 3431 2c34 2c31 3539
                      0d0a

     The sequence numbers shown in  this  trace  are  absolute  and  not
     adjusted to reflect the ISN.  The 4-digit hex values show a dump of
     the packet's IP and TCP headers, as well as payload.  A first sends
     to  B  data  for  90048435:90048461.  The corresponding data begins
     with hex words 504f, 5254, etc.

     B responds with a RST.  Since the recording location was  local  to
     B, it is unknown whether A received the RST.

     A then sends 90048429:90048463, which includes six  sequence  posi-
     tions  below the earlier transmission, all 26 positions of the ear-
     lier transmission, and two additional sequence positions.

     The retransmission disagrees starting just after sequence 90048447,
     annotated  above  with  a leading '*'.  These two bytes were origi-
     nally transmitted as hex 2c34 but retransmitted as hex 2c31.   Sub-
     sequent positions disagree as well.

     This behavior has been observed in other traces involving different



Paxson, Editor                                                  [Page 9]


ID                 Known TCP Implementation Problems       November 1997


     hosts.  It is unknown how to repeat it.

     In this instance, no corruption would occur, since  B  has  already
     indicated it will not accept further packets from A.

     A second example illustrates a slightly different instance  of  the
     problem.   The  tracing  again  was  made  with  tcpdump/BPF at the
     receiving TCP (D).

     22:23:58.645829 C > D: P 185:212(27) ack 565 win 4096
                                          4500 0043 90a3 0000
                      3306 0734 cbf1 9eef 83f3 010a 0525 0015
                      a3a2 faba 578c 70a4 5018 1000 9a53 0000
     data starts here>504f 5254 2032 3033 2c32 3431 2c31 3538
                      2c32 3339 2c35 2c34 330d 0a
     22:23:58.646805 D > C: . ack 184 win 8192
                                          4500 0028 beeb 0000
                      3e06 ce06 83f3 010a cbf1 9eef 0015 0525
                      578c 70a4 a3a2 fab9 5010 2000 342f 0000
     22:31:36.532244 C > D: FP 186:213(27) ack 565 win 4096
                                          4500 0043 9435 0000
                      3306 03a2 cbf1 9eef 83f3 010a 0525 0015
                      a3a2 fabb 578c 70a4 5019 1000 9a51 0000
     data starts here>504f 5254 2032 3033 2c32 3431 2c31 3538
                      2c32 3339 2c35 2c34 330d 0a

     In this trace, sequence numbers are relative.  C sends 185:212, but
     D only sends an ACK for 184 (so sequence number 184 is missing).  C
     then sends 186:213.  The packet payload is identical to the  previ-
     ous  payload, but the base sequence number is one higher, resulting
     in an inconsistent retransmission.

     Neither trace exhibits checksum errors.


Trace file demonstrating correct behavior
     (Omitted, as presumably correct behavior is obvious.)

References
     None known.

How to detect
     This problem unfortunately can be very difficult to  detect,  since
     available  experience  indicates  it is quite rare that it is mani-
     fested.  No "trigger" has been  identified  that  can  be  used  to
     reproduce the problem.

How to fix



Paxson, Editor                                                 [Page 10]


ID                 Known TCP Implementation Problems       November 1997


     In the absence of a known "trigger", we cannot always assess how to
     fix the problem.

     In one implementation (not the one illustrated above), the  problem
     manifested  itself  when  (1) the sender received a zero window and
     stalled; (2) eventually an ACK arrived that offered a window larger
     than  that  in  effect  at  the  time  of the stall; (3) the sender
     transmitted out of the buffer of data it held at the  time  of  the
     stall,  but (4) failed to limit this transfer to the buffer length,
     instead using the newly advertised  (and  larger)  offered  window.
     Consequently,  in  addition  to  the valid buffer contents, it sent
     whatever garbage values followed the end of the buffer.  If it then
     retransmitted  the corresponding sequence numbers, at that point it
     sent the correct data, resulting in an inconsistent retransmission.
     Note  that  this  instance  of  the problem reflects a more general
     problem, that of initially transmitting incorrect data.


3.4. Failure to retain above-sequence data

Classification
     Congestion control, performance

Description
     When a TCP receives an "above sequence" segment, meaning one with a
     sequence  number  exceeding  RCV.NXT  but below RCV.NXT+RCV.WND, it
     SHOULD queue the segment for later delivery (RFC  1122,  4.2.2.20).
     A  TCP  that  fails  to do so is said to exhibit "Failure to retain
     above-sequence data".

     It may sometimes be appropriate for a TCP to discard above-sequence
     data  to  reclaim memory.  If they do so only rarely, then we would
     not consider them to exhibit this problem.  Instead, the particular
     concern is with TCPs that always discard above-sequence data.


Significance
     In environments prone to packet loss, detrimental  to  the  perfor-
     mance of both other connections and the connection itself.

Implications
     In times of congestion, a failure  to  retain  above-sequence  data
     will lead to numerous otherwise-unnecessary retransmissions, aggra-
     vating the congestion and potentially  reducing  performance  by  a
     large factor.

Relevant RFCs
     RFC 1122 revises RFC 793 by upgrading the latter's MAY to a  SHOULD



Paxson, Editor                                                 [Page 11]


ID                 Known TCP Implementation Problems       November 1997


     on this issue.

Trace file demonstrating it
     Made using tcpdump/BPF recording at the receiving TCP.   No  losses
     reported.

     B is the TCP sender, A the receiver.  A exhibits failure to  retain
     above sequence data:

     10:38:10.164860 B > A: . 221078:221614(536) ack 1 win 33232 [tos 0x8]
     10:38:10.170809 B > A: . 221614:222150(536) ack 1 win 33232 [tos 0x8]
     10:38:10.177183 B > A: . 222150:222686(536) ack 1 win 33232 [tos 0x8]
     10:38:10.225039 A > B: . ack 222686 win 25800

     Here B has sent up to (relative) sequence 222686 in-sequence, and A
     accordingly acknowledges.

     10:38:10.268131 B > A: . 223222:223758(536) ack 1 win 33232 [tos 0x8]
     10:38:10.337995 B > A: . 223758:224294(536) ack 1 win 33232 [tos 0x8]
     10:38:10.344065 B > A: . 224294:224830(536) ack 1 win 33232 [tos 0x8]
     10:38:10.350169 B > A: . 224830:225366(536) ack 1 win 33232 [tos 0x8]
     10:38:10.356362 B > A: . 225366:225902(536) ack 1 win 33232 [tos 0x8]
     10:38:10.362445 B > A: . 225902:226438(536) ack 1 win 33232 [tos 0x8]
     10:38:10.368579 B > A: . 226438:226974(536) ack 1 win 33232 [tos 0x8]
     10:38:10.374732 B > A: . 226974:227510(536) ack 1 win 33232 [tos 0x8]
     10:38:10.380825 B > A: . 227510:228046(536) ack 1 win 33232 [tos 0x8]
     10:38:10.387027 B > A: . 228046:228582(536) ack 1 win 33232 [tos 0x8]
     10:38:10.393053 B > A: . 228582:229118(536) ack 1 win 33232 [tos 0x8]
     10:38:10.399193 B > A: . 229118:229654(536) ack 1 win 33232 [tos 0x8]
     10:38:10.405356 B > A: . 229654:230190(536) ack 1 win 33232 [tos 0x8]

     A now receives 13 additional packets  from  B.   These  are  above-
     sequence because 222686:223222 was dropped.  The packets do however
     fit within the offered window of 25800.  A does  not  generate  any
     duplicate ACKs for them.

     The trace contributor (V. Paxson) verified that  these  13  packets
     had valid IP and TCP checksums.

     10:38:11.917728 B > A: . 222686:223222(536) ack 1 win 33232 [tos 0x8]
     10:38:11.930925 A > B: . ack 223222 win 32232

     B times out for 222686:223222 and retransmits it.   Upon  receiving
     it,  A  only acknowledges 223222.  Had it retained the valid above-
     sequence packets, it would instead have ack'd 230190.

     10:38:12.048438 B > A: . 223222:223758(536) ack 1 win 33232 [tos 0x8]
     10:38:12.054397 B > A: . 223758:224294(536) ack 1 win 33232 [tos 0x8]



Paxson, Editor                                                 [Page 12]


ID                 Known TCP Implementation Problems       November 1997


     10:38:12.068029 A > B: . ack 224294 win 31696

     B retransmits two more packets, and A only acknowledges them.  This
     pattern  continues  as  B retransmits the entire set of previously-
     received packets.

     A second trace confirmed that the problem is repeatable.


Trace file demonstrating correct behavior
     Made using tcpdump/BPF recording at  the  receiving  TCP  (C).   No
     losses reported.

     09:11:25.790417 D > C: . 33793:34305(512) ack 1 win 61440
     09:11:25.791393 D > C: . 34305:34817(512) ack 1 win 61440
     09:11:25.792369 D > C: . 34817:35329(512) ack 1 win 61440
     09:11:25.792369 D > C: . 35329:35841(512) ack 1 win 61440
     09:11:25.793345 D > C: . 36353:36865(512) ack 1 win 61440
     09:11:25.794321 C > D: . ack 35841 win 59904

     A sequence hole occurs because 35841:36353 has been dropped.

     09:11:25.794321 D > C: . 36865:37377(512) ack 1 win 61440
     09:11:25.794321 C > D: . ack 35841 win 59904
     09:11:25.795297 D > C: . 37377:37889(512) ack 1 win 61440
     09:11:25.795297 C > D: . ack 35841 win 59904
     09:11:25.796273 C > D: . ack 35841 win 61440
     09:11:25.798225 D > C: . 37889:38401(512) ack 1 win 61440
     09:11:25.799201 C > D: . ack 35841 win 61440
     09:11:25.807009 D > C: . 38401:38913(512) ack 1 win 61440
     09:11:25.807009 C > D: . ack 35841 win 61440
     (many additional lines omitted)
     09:11:25.884113 D > C: . 52737:53249(512) ack 1 win 61440
     09:11:25.884113 C > D: . ack 35841 win 61440

     Each additional, above-sequence packet C receives from D elicits  a
     duplicate ACK for 35841.

     09:11:25.887041 D > C: . 35841:36353(512) ack 1 win 61440
     09:11:25.887041 C > D: . ack 53249 win 44032

     D retransmits 35841:36353 and C acknowledges receipt  of  data  all
     the way up to 53249.


References
     This problem is documented in [Paxson97].




Paxson, Editor                                                 [Page 13]


ID                 Known TCP Implementation Problems       November 1997


How to detect
     Packet loss is common enough in the Internet that generally  it  is
     not  difficult  to  find  an Internet path that will result in some
     above-sequence packets arriving.  A TCP that exhibits  "Failure  to
     retain  ..."  may  not  generate  duplicate ACKs for these packets.
     However, some TCPs that do retain above-sequence data also  do  not
     generate  duplicate ACKs, so failure to do so does not definitively
     identify the problem.  Instead, the key observation is whether upon
     retransmission  of  the  dropped  packet,  data that was previously
     above-sequence is acknowledged.

     Two considerations in detecting this problem using a  packet  trace
     are  that  it  is  easiest  to  do  so with a trace made at the TCP
     receiver, in order to unambiguously determine which packets arrived
     successfully,  and  that  such  packets may still be correctly dis-
     carded if they arrive with checksum  errors.   The  latter  can  be
     tested  by  capturing the entire packet contents and performing the
     IP and TCP checksum algorithms to verify  their  integrity;  or  by
     confirming  that the packets arrive with the same checksum and con-
     tents as that with which they were sent, with  a  presumption  that
     the  sending  TCP correctly calculates checksums for the packets it
     transmits.

     It is considerably easier to verify that an implementation does NOT
     exhibit this problem.  This can be done by recording a trace at the
     data sender, and observing that sometimes  after  a  retransmission
     the  receiver  acknowledges a higher sequence number than just that
     which was retransmitted.


How to fix
     If the root problem is that the implementation lacks  buffer,  then
     then unfortunately this requires significant work to fix.  However,
     doing so is important, for reasons outlined above.


3.5. Excessively short keepalive connection timeout


Classification
     Reliability


Description
     Keep-alive is a mechanism for checking whether an  idle  connection
     is  still  alive.   According to RFC-1122, keepalive should only be
     invoked in server applications that might otherwise  hang  indefin-
     itely  and  consume  resources unnecessarily if a client crashes or



Paxson, Editor                                                 [Page 14]


ID                 Known TCP Implementation Problems       November 1997


     aborts a connection during a network failure.

     RFC-1122 also specifies that if a keep-alive  mechanism  is  imple-
     mented  it  MUST  NOT  interpret failure to respond to any specific
     probe as a dead connection.  The RFC does not specify a  particular
     mechanism  for timing out a connection when no response is received
     for keepalive probes.  However, if the  mechanism  does  not  allow
     ample  time  for recovery from network congestion or delay, connec-
     tions may be timed out unnecessarily.


Significance
     In congested networks, can lead to unwarranted termination of  con-
     nections.


Implications
     It is possible for the network connection between two peer machines
     to  become  congested  or to exhibit packet loss at the time that a
     keep-alive probe is  sent  on  a  connection.   If  the  keep-alive
     mechanism  does  not  allow sufficient time before dropping connec-
     tions in the face of  unacknowledged  probes,  connections  may  be
     dropped even when both peers of a connection are still alive.


Relevant RFCs
     RFC 1122 specifies that the keep-alive mechanism may  be  provided.
     It  does  not  specify a mechanism for determining dead connections
     when keepalive probes are not acknowledged.


Trace file demonstrating it
     Made using the Orchestra tool at the  peer  of  the  machine  using
     keep-alive.   After  connection establishment, incoming keep-alives
     were dropped by Orchestra to simulate a dead connection.

     22:11:12.040000 A > B: 22666019:0 win 8192 datasz 4 SYN
     22:11:12.060000 B > A: 2496001:22666020 win 4096 datasz 4 SYN ACK
     22:11:12.130000 A > B: 22666020:2496002 win 8760 datasz 0 ACK
     (more than two hours elapse)
     00:23:00.680000 A > B: 22666019:2496002 win 8760 datasz 1 ACK
     00:23:01.770000 A > B: 22666019:2496002 win 8760 datasz 1 ACK
     00:23:02.870000 A > B: 22666019:2496002 win 8760 datasz 1 ACK
     00:23.03.970000 A > B: 22666019:2496002 win 8760 datasz 1 ACK
     00:23.05.070000 A > B: 22666019:2496002 win 8760 datasz 1 ACK

     The initial three packets  are  the  SYN  exchange  for  connection
     setup.   About  two  hours later, the keepalive timer fires because



Paxson, Editor                                                 [Page 15]


ID                 Known TCP Implementation Problems       November 1997


     the connection has been idle.  Keepalive probes are  transmitted  a
     total  of  5  times,  with a 1 second spacing between probes, after
     which the connection is dropped.  This is problematic because  a  5
     second network outage at the time of the first probe results in the
     connection being killed.


Trace file demonstrating correct behavior
     Made using the Orchestra tool at the  peer  of  the  machine  using
     keep-alive.   After  connection establishment, incoming keep-alives
     were dropped by Orchestra to simulate a dead connection.

     16:01:52.130000 A > B: 1804412929:0 win 4096 datasz 4 SYN
     16:01:52.360000 B > A: 16512001:1804412930 win 4096 datasz 4 SYN ACK
     16:01:52.410000 A > B: 1804412930:16512002 win 4096 datasz 0 ACK
     (two hours elapse)
     18:01:57.170000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK
     18:03:12.220000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK
     18:04:27.270000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK
     18:05:42.320000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK
     18:06:57.370000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK
     18:08:12.420000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK
     18:09:27.480000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK
     18:10:43.290000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK
     18:11:57.580000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK
     18:13:12.630000 A > B: 1804412929:16512002 win 4096 datasz 0 RST ACK

     In this trace, when  the  keep-alive  timer  expires,  9  keepalive
     probes  are sent at 75 second intervals.  75 seconds after the last
     probe is sent, a final RST segment is sent indicating that the con-
     nection  has  been  closed.   This  implementation  waits  about 11
     minutes before timing out the connection, while the first implemen-
     tation shown allows only 5 seconds.


References
     This problem is documented in [Dawson97].


How to detect
     For implementations manifesting this problem,  it  shows  up  on  a
     packet  trace  after  the keepalive timer fires if the peer machine
     receiving the keepalive does not respond.   Usually  the  keepalive
     timer  will  fire  at least two hours after keepalive is turned on,
     but it may be sooner if the timer value has been configured  lower,
     or  if  the  keepalive  mechanism  violates  the specification (see
     Insufficient interval between keepalives problem).  In  this  exam-
     ple,  suppressing  the response of the peer to keepalive probes was



Paxson, Editor                                                 [Page 16]


ID                 Known TCP Implementation Problems       November 1997


     accomplished using the Orchestra toolkit, which can  be  configured
     to  drop  packets.  It could also have been done by creating a con-
     nection, turning on keepalive, and disconnecting the  network  con-
     nection at the receiver machine.


How to fix
     This problem can be fixed by using a different  method  for  timing
     out keepalives that allows a longer period of time to elapse before
     dropping the connection.  For example, the algorithm for timing out
     on dropped data could be used.  Another possibility is an algorithm
     such as the one shown in the trace above, which sends 9  probes  at
     75  second  intervals and then waits an additional 75 seconds for a
     response before closing the connection.


3.6. Insufficient interval between keepalives


Classification
     Reliability


Description
     Keep-alive is a mechanism for checking whether an  idle  connection
     is  still alive.  According to RFC-1122, keep-alive may be included
     in an implementation.  If it  is  included,  the  interval  between
     keep-alive  packets  MUST  be  configurable, and MUST default to no
     less than two hours.


Significance
     In congested networks, can lead to unwarranted termination of  con-
     nections.


Implications
     According to RFC-1122, keep-alive is not  required  of  implementa-
     tions  because  it  could:  (1) cause perfectly good connections to
     break during transient Internet failures; (2)  consume  unnecessary
     bandwidth  ("if  no one is using the connection, who cares if it is
     still good?"); and (3) cost money for an Internet path that charges
     for  packets.   Regarding this last point, we note that in addition
     the presence of dial-on-demand links in the route can greatly  mag-
     nify  the  cost penalty of excess keepalives, potentially forcing a
     full-time connection on a link that would otherwise  only  be  con-
     nected a few minutes a day.




Paxson, Editor                                                 [Page 17]


ID                 Known TCP Implementation Problems       November 1997


     If keepalive is provided the RFC states that  the  required  inter-
     keepalive  distance  MUST default to no less than two hours.  If it
     does not, the probability of connections  breaking  increases,  the
     bandwidth used due to keepalives increases, and cost increases over
     paths which charge per packet.


Relevant RFCs
     RFC 1122 specifies that the keep-alive mechanism may  be  provided.
     It  also  specifies  the  two hour minimum for the default interval
     between keepalive probes.


Trace file demonstrating it
     Made using the Orchestra tool at the  peer  of  the  machine  using
     keep-alive.   Machine  A was configured to use default settings for
     the keepalive timer.

     11:36:32.910000 A > B: 3288354305:0      win 28672 datasz 4 SYN
     11:36:32.930000 B > A: 896001:3288354306 win 4096  datasz 4 SYN ACK
     11:36:32.950000 A > B: 3288354306:896002 win 28672 datasz 0 ACK

     11:50:01.190000 A > B: 3288354305:896002 win 28672 datasz 0 ACK
     11:50:01.210000 B > A: 896002:3288354306 win 4096  datasz 0 ACK

     12:03:29.410000 A > B: 3288354305:896002 win 28672 datasz 0 ACK
     12:03:29.430000 B > A: 896002:3288354306 win 4096  datasz 0 ACK

     12:16:57.630000 A > B: 3288354305:896002 win 28672 datasz 0 ACK
     12:16:57.650000 B > A: 896002:3288354306 win 4096  datasz 0 ACK

     12:30:25.850000 A > B: 3288354305:896002 win 28672 datasz 0 ACK
     12:30:25.870000 B > A: 896002:3288354306 win 4096  datasz 0 ACK

     12:43:54.070000 A > B: 3288354305:896002 win 28672 datasz 0 ACK
     12:43:54.090000 B > A: 896002:3288354306 win 4096  datasz 0 ACK

     The initial three packets  are  the  SYN  exchange  for  connection
     setup.   About  13 minutes later, the keepalive timer fires because
     the connection is idle.  The keepalive  is  acknowledged,  and  the
     timer  fires again in about 13 more minutes.  This behavior contin-
     ues indefinitely until the connection is closed, and is a violation
     of the specification.


Trace file demonstrating correct behavior
     Made using the Orchestra tool at the  peer  of  the  machine  using
     keep-alive.   Machine  A was configured to use default settings for



Paxson, Editor                                                 [Page 18]


ID                 Known TCP Implementation Problems       November 1997


     the keepalive timer.

     17:37:20.500000 A > B: 34155521:0       win 4096 datasz 4 SYN
     17:37:20.520000 B > A: 6272001:34155522 win 4096 datasz 4 SYN ACK
     17:37:20.540000 A > B: 34155522:6272002 win 4096 datasz 0 ACK

     19:37:25.430000 A > B: 34155521:6272002 win 4096 datasz 0 ACK
     19:37:25.450000 B > A: 6272002:34155522 win 4096 datasz 0 ACK

     21:37:30.560000 A > B: 34155521:6272002 win 4096 datasz 0 ACK
     21:37:30.570000 B > A: 6272002:34155522 win 4096 datasz 0 ACK

     23:37:35.580000 A > B: 34155521:6272002 win 4096 datasz 0 ACK
     23:37:35.600000 B > A: 6272002:34155522 win 4096 datasz 0 ACK

     01:37:40.620000 A > B: 34155521:6272002 win 4096 datasz 0 ACK
     01:37:40.640000 B > A: 6272002:34155522 win 4096 datasz 0 ACK

     03:37:45.590000 A > B: 34155521:6272002 win 4096 datasz 0 ACK
     03:37:45.610000 B > A: 6272002:34155522 win 4096 datasz 0 ACK

     The initial three packets  are  the  SYN  exchange  for  connection
     setup.   Just  over  two  hours  later,  the  keepalive timer fires
     because the connection is idle.  The keepalive is acknowledged, and
     the  timer  fires  again  just over two hours later.  This behavior
     continues indefinitely until the connection is closed.


References
     This problem is documented in [Dawson97].


How to detect
     For implementations manifesting this problem,  it  shows  up  on  a
     packet trace.  If the connection is left idle, the keepalive probes
     will arrive closer together than the two hour minimum.


3.7. Stretch ACK Violation


Classification
     Congestion Control/Performance


Description
     To improve efficiency (both computer and network) a  data  receiver
     may  refrain  from  sending  an  ACK  for  each  incoming  segment,



Paxson, Editor                                                 [Page 19]


ID                 Known TCP Implementation Problems       November 1997


     according to [RFC1122].  However, an ACK should not be  delayed  an
     inordinate  amount  of  time.   Specifically, ACKs MUST be sent for
     every second full-sized segment that arrives.  If  a  second  full-
     sized  segment  does  not arrive within a given timeout (of no more
     than 0.5  seconds),  an  ACK  must  be  transmitted,  according  to
     [RFC1122].  A TCP receiver which does not generate an ACK for every
     second full-sized segment exhibits a "Stretch ACK Violation".


Significance
     TCP receivers exhibiting this behavior will cause  TCP  senders  to
     generate  burstier  traffic, which can degrade performance in cong-
     ested environments.  In addition, generating fewer  ACKs  increases
     the  amount  of time needed by the slow start algorithm to open the
     congestion window to an appropriate point, which diminishes perfor-
     mance   in   environments   with  large  bandwidth-delay  products.
     Finally, generating fewer ACKs may  cause  needless  retransmission
     timeouts  in  lossy  environments,  as it increases the possibility
     that an entire window of ACKs is  lost,  forcing  a  retransmission
     timeout.


Implications
     When not in loss recovery, every  ACK  received  by  a  TCP  sender
     triggers  the transmission of new data segments.  The burst size is
     determined by the number of previously unacknowledged segments each
     ACK  covers.  Therefore, a TCP receiver ACKing more than 2 segments
     at a time causes the sending TCP to  generate  a  larger  burst  of
     traffic  upon  receipt of the ACK.  This large burst of traffic can
     overwhelm an intervening gateway, leading to higher drop rates  for
     both the connection and other connections passing through the cong-
     ested gateway.

     In addition, the TCP slow start algorithm increases the  congestion
     window  by  1 segment for each ACK received.  Therefore, increasing
     the ACK interval (thus  decreasing  the  rate  at  which  ACKs  are
     transmitted)  increases  the  amount of time it takes slow start to
     increase the congestion window to an appropriate  operating  point,
     and  the  connection consequently suffers from reduced performance.
     This is especially true for connections using large windows.


Relevant RFCs
     RFC 1122 outlines delayed ACKs as a recommended mechanism.


Trace file demonstrating it
     Trace file taken using tcpdump at host B, the  data  receiver  (and



Paxson, Editor                                                 [Page 20]


ID                 Known TCP Implementation Problems       November 1997


     ACK  originator).   The advertised window (which never changed) and
     timestamp options have been omitted for  clarity,  except  for  the
     first packet sent by A:

     12:09:24.820187 A.1174 > B.3999: . 2049:3497(1448) ack 1
         win 33580 <nop,nop,timestamp 2249877 2249914> [tos 0x8]
     12:09:24.824147 A.1174 > B.3999: . 3497:4945(1448) ack 1
     12:09:24.832034 A.1174 > B.3999: . 4945:6393(1448) ack 1
     12:09:24.832222 B.3999 > A.1174: . ack 6393
     12:09:24.934837 A.1174 > B.3999: . 6393:7841(1448) ack 1
     12:09:24.942721 A.1174 > B.3999: . 7841:9289(1448) ack 1
     12:09:24.950605 A.1174 > B.3999: . 9289:10737(1448) ack 1
     12:09:24.950797 B.3999 > A.1174: . ack 10737
     12:09:24.958488 A.1174 > B.3999: . 10737:12185(1448) ack 1
     12:09:25.052330 A.1174 > B.3999: . 12185:13633(1448) ack 1
     12:09:25.060216 A.1174 > B.3999: . 13633:15081(1448) ack 1
     12:09:25.060405 B.3999 > A.1174: . ack 15081

     This portion of the trace clearly shows that the receiver (host  B)
     sends  an  ACK for every third full sized packet received.  Further
     investigation of this implementation found that the  cause  of  the
     increased  ACK interval was the TCP options being used.  The imple-
     mentation sent an ACK after it was holding 2*MSS  worth  of  unack-
     nowledged  data.   In  the above case, the MSS is 1460 bytes so the
     receiver transmits an ACK after it is holding at least  2920  bytes
     of  unacknowledged  data.   However,  the length of the TCP options
     being used [RFC1323] took 12 bytes away from the  data  portion  of
     each  packet.  This produced packets containing 1448 bytes of data.
     But the additional bytes used by the options in the header were not
     taken into account when determining when to trigger an ACK.  There-
     fore, it took 3 data segments before the data receiver was  holding
     enough  unacknowledged  data  (>= 2*MSS, or 2920 bytes in the above
     example) to transmit an ACK.


Trace file demonstrating correct behavior

     Trace file taken using tcpdump at host B, the  data  receiver  (and
     ACK  originator), again with window and timestamp information omit-
     ted except for the first packet:

     12:06:53.627320 A.1172 > B.3999: . 1449:2897(1448) ack 1
         win 33580 <nop,nop,timestamp 2249575 2249612> [tos 0x8]
     12:06:53.634773 A.1172 > B.3999: . 2897:4345(1448) ack 1
     12:06:53.634961 B.3999 > A.1172: . ack 4345
     12:06:53.737326 A.1172 > B.3999: . 4345:5793(1448) ack 1
     12:06:53.744401 A.1172 > B.3999: . 5793:7241(1448) ack 1
     12:06:53.744592 B.3999 > A.1172: . ack 7241



Paxson, Editor                                                 [Page 21]


ID                 Known TCP Implementation Problems       November 1997


     12:06:53.752287 A.1172 > B.3999: . 7241:8689(1448) ack 1
     12:06:53.847332 A.1172 > B.3999: . 8689:10137(1448) ack 1
     12:06:53.847525 B.3999 > A.1172: . ack 10137

     This trace shows the TCP receiver (host  B)  ack'ing  every  second
     full-sized packet, according to [RFC1122].  This is the same imple-
     mentation shown above, with slight  modifications  that  allow  the
     receiver to take the length of the options into account when decid-
     ing when to transmit an ACK.


References
     This problem is documented in [Allman97] and [Paxson97].


How to detect
     Stretch ACK violations show up immediately in receiver-side  packet
     traces  of  bulk transfers, as shown above.  However, packet traces
     made on the sender side of the TCP connection may lead to  ambigui-
     ties  when  diagnosing  this problem due to the possibility of lost
     ACKs.


3.8. Failure to send FIN notification promptly


Classification
     Performance


Description
     When an application closes  a  connection,  the  corresponding  TCP
     should  send  the  FIN  notification  promptly  to its peer (unless
     prevented by the  congestion  window).   If  a  TCP  implementation
     delays  in sending the FIN notification, for example due to waiting
     until unacknowledged data has been acknowledged, then it is said to
     exhibit "Failure to send FIN notification promptly".

     Also, while not strictly required, FIN segments should include  the
     PSH  flag  to  ensure expedited delivery of any pending data at the
     receiver.


Significance
     The greatest impact occurs for short-lived connections,  since  for
     these  the  additional time required to close the connection intro-
     duces the greatest relative delay.




Paxson, Editor                                                 [Page 22]


ID                 Known TCP Implementation Problems       November 1997


     The additional time can be significant in the common  case  of  the
     sender waiting for an ACK that is delayed by the receiver.


Implications
     Can diminish total throughput as seen  at  the  application  layer,
     because connection termination takes longer to complete.


Relevant RFCs
     RFC 793 indicates that a receiver should treat an incoming FIN flag
     as implying the push function.


Trace file demonstrating it
     Made using tcpdump (no losses reported).

     10:04:38.68 A > B: S 1031850376:1031850376(0) win 4096
                     <mss 1460,wscale 0,eol> (DF)
     10:04:38.71 B > A: S 596916473:596916473(0) ack 1031850377
                     win 8760 <mss 1460> (DF)
     10:04:38.73 A > B: . ack 1 win 4096 (DF)
     10:04:41.98 A > B: P 1:4(3) ack 1 win 4096 (DF)
     10:04:42.15 B > A: . ack 4 win 8757 (DF)
     10:04:42.23 A > B: P 4:7(3) ack 1 win 4096 (DF)
     10:04:42.25 B > A: P 1:11(10) ack 7 win 8754 (DF)
     10:04:42.32 A > B: . ack 11 win 4096 (DF)
     10:04:42.33 B > A: P 11:51(40) ack 7 win 8754 (DF)
     10:04:42.51 A > B: . ack 51 win 4096 (DF)
     10:04:42.53 B > A: F 51:51(0) ack 7 win 8754 (DF)
     10:04:42.56 A > B: FP 7:7(0) ack 52 win 4096 (DF)
     10:04:42.58 B > A: . ack 8 win 8754 (DF)

     Machine B in the trace above does not send out a  FIN  notification
     promptly  if  there  is any data outstanding.  It instead waits for
     all unacknowledged data to be acknowledged before sending  the  FIN
     segment.  The connection was closed at 10:04.42.33 after requesting
     40 bytes to be sent.  However,  the  FIN  notification  isn't  sent
     until  10:04.42.51,  after  the (delayed) acknowledgement of the 40
     bytes of data.


Trace file demonstrating correct behavior
     Made using tcpdump (no losses reported).

     10:27:53.85 C > D: S 419744533:419744533(0) win 4096
                     <mss 1460,wscale 0,eol> (DF)
     10:27:53.92 D > C: S 10082297:10082297(0) ack 419744534



Paxson, Editor                                                 [Page 23]


ID                 Known TCP Implementation Problems       November 1997


                     win 8760 <mss 1460> (DF)
     10:27:53.95 C > D: . ack 1 win 4096 (DF)
     10:27:54.42 C > D: P 1:4(3) ack 1 win 4096 (DF)
     10:27:54.62 D > C: . ack 4 win 8757 (DF)
     10:27:54.76 C > D: P 4:7(3) ack 1 win 4096 (DF)
     10:27:54.89 D > C: P 1:11(10) ack 7 win 8754 (DF)
     10:27:54.90 D > C: FP 11:51(40) ack7 win 8754 (DF)
     10:27:54.92 C > D: . ack 52 win 4096 (DF)
     10:27:55.01 C > D: FP 7:7(0) ack 52 win 4096 (DF)
     10:27:55.09 D > C: . ack 8 win 8754 (DF)


References
     This problem is documented in [Dawson97].


How to detect
     For implementations manifesting this problem,  it  shows  up  on  a
     packet trace.  If the connection is left idle, the keepalive probes
     will arrive closer together than the two hour minimum.

     Here, Machine D sends a FIN with 40 bytes of data even  before  the
     original 10 octets have been acknowledged. This is correct behavoir
     as it provides for the highest performance.


3.9. Failure to send a RST after Half Duplex Close


Classification
     Resource management

     [[ editor's comment: this is a new classification category ]]


Description
     RFC 1122 4.2.2.13 states that a TCP SHOULD send a RST  if  data  is
     received  after "half duplex close", i.e. if it cannot be delivered
     to the application.  A TCP that fails to do so is said  to  exhibit
     "Failure to send a RST after Half Duplex Close".


Significance
     Potentially serious for TCP endpoints that manage large numbers  of
     connections,  due  to  exhaustion  of memory available for managing
     connection state.





Paxson, Editor                                                 [Page 24]


ID                 Known TCP Implementation Problems       November 1997


Implications
     Failure to send the RST can lead to permanently  hung  TCP  connec-
     tions.   This problem has been demonstrated when HTTP clients abort
     connections, common when users move on to a  new  page  before  the
     current  page  has finished downloading.  The HTTP client closes by
     transmitting a FIN while the server is transmitting  images,  text,
     etc.   The  server  TCP receives the FIN,  but its application does
     not close the  connection  until  all  data  has  been  queued  for
     transmission.   Since  the server will not transmit a FIN until all
     the preceding data has been transmitted, deadlock  results  if  the
     client  TCP does not consume the pending data or tear down the con-
     nection: the window decreases to zero, since the client cannot pass
     the  data  to the application, and the server sends probe segments.
     The client acknowledges the probe segments with a zero  window.  As
     mandated  in  RFC1122  4.2.2.17, the probe segments are transmitted
     forever.  Server connection state remains in CLOSE_WAIT, and  even-
     tually server processes are exhausted.

     Note that there are two bugs.   First,  probe  segments  should  be
     ignored  if  the window can never subsequently increase.  Second, a
     RST should be sent when data is received after half  duplex  close.
     Fixing the first bug, but not the second, results in the probe seg-
     ments eventually timing out the connection, but the server  remains
     in CLOSE_WAIT for a significant and unnecessary period.


Relevant RFCs
     RFC 1122 sections 4.2.2.13 and 4.2.2.17.


Trace file demonstrating it
     Made using an unknown network analyzer.  No drop information avail-
     able.

     client.1391 > server.8080: S 0:1(0) ack: 0 win: 2000 <mss: 5b4>
     server.8080 > client.1391: SA 8c01:8c02(0) ack: 1 win: 8000 <mss:100>
     client.1391 > server.8080: PA
     client.1391 > server.8080: PA 1:1c2(1c1) ack: 8c02 win: 2000
     server.8080 > client.1391: [DF] PA 8c02:8cde(dc) ack: 1c2 win: 8000
     server.8080 > client.1391: [DF] A 8cde:9292(5b4) ack: 1c2 win: 8000
     server.8080 > client.1391: [DF] A 9292:9846(5b4) ack: 1c2 win: 8000
     server.8080 > client.1391: [DF] A 9846:9dfa(5b4) ack: 1c2 win: 8000
     client.1391 > server.8080: PA
     server.8080 > client.1391: [DF] A 9dfa:a3ae(5b4) ack: 1c2 win: 8000
     server.8080 > client.1391: [DF] A a3ae:a962(5b4) ack: 1c2 win: 8000
     server.8080 > client.1391: [DF] A a962:af16(5b4) ack: 1c2 win: 8000
     server.8080 > client.1391: [DF] A af16:b4ca(5b4) ack: 1c2 win: 8000
     client.1391 > server.8080: PA



Paxson, Editor                                                 [Page 25]


ID                 Known TCP Implementation Problems       November 1997


     server.8080 > client.1391: [DF] A b4ca:ba7e(5b4) ack: 1c2 win: 8000
     server.8080 > client.1391: [DF] A b4ca:ba7e(5b4) ack: 1c2 win: 8000
     client.1391 > server.8080: PA
     server.8080 > client.1391: [DF] A ba7e:bdfa(37c) ack: 1c2 win: 8000
     client.1391 > server.8080: PA
     server.8080 > client.1391: [DF] A bdfa:bdfb(1) ack: 1c2 win: 8000
     client.1391 > server.8080: PA

     [ HTTP client aborts and enters FIN_WAIT_1 ]

     client.1391 > server.8080: FPA

     [ server ACKs the FIN and enters CLOSE_WAIT ]

     server.8080 > client.1391: [DF] A

     [ client enters FIN_WAIT_2 ]

     server.8080 > client.1391: [DF] A bdfa:bdfb(1) ack: 1c3 win: 8000

     [ server continues to try to send its data ]

     client.1391 > server.8080: PA < window = 0 >
     server.8080 > client.1391: [DF] A bdfa:bdfb(1) ack: 1c3 win: 8000
     client.1391 > server.8080: PA < window = 0 >
     server.8080 > client.1391: [DF] A bdfa:bdfb(1) ack: 1c3 win: 8000
     client.1391 > server.8080: PA < window = 0 >
     server.8080 > client.1391: [DF] A bdfa:bdfb(1) ack: 1c3 win: 8000
     client.1391 > server.8080: PA < window = 0 >
     server.8080 > client.1391: [DF] A bdfa:bdfb(1) ack: 1c3 win: 8000
     client.1391 > server.8080: PA < window = 0 >

     [ ... repeat ad exhaustium ... ]



Trace file demonstrating correct behavior
     Made using an unknown network analyzer.  No drop information avail-
     able.

     client > server D=80 S=59500 Syn Seq=337 Len=0 Win=8760
     server > client D=59500 S=80 Syn Ack=338 Seq=80153 Len=0 Win=8760
     client > server D=80 S=59500 Ack=80154 Seq=338 Len=0 Win=8760

     [ ... normal data omitted ... ]

     client > server D=80 S=59500 Ack=14559 Seq=596 Len=0 Win=8760
     server > client D=59500 S=80 Ack=596 Seq=114559 Len=1460 Win=8760



Paxson, Editor                                                 [Page 26]


ID                 Known TCP Implementation Problems       November 1997


     [ client closes connection ]

     client > server D=80 S=59500 Fin Seq=596 Len=0 Win=8760
     server > client D=59500 S=80 Ack=597 Seq=116019 Len=1460 Win=8760

     [ client sends RST (RFC1122 4.2.2.13) ]

     client > server D=80 S=59500 Rst Seq=597 Len=0 Win=0
     server > client D=59500 S=80 Ack=597 Seq=117479 Len=1460 Win=8760
     [[ editor's comment: why doesn't the RST result in the
        connection being torn down at this point ??]]
     client > server D=80 S=59500 Rst Seq=597 Len=0 Win=0
     server > client D=59500 S=80 Ack=597 Seq=118939 Len=1460 Win=8760
     client > server D=80 S=59500 Rst Seq=597 Len=0 Win=0
     server > client D=59500 S=80 Ack=597 Seq=120399 Len=892 Win=8760
     client > server D=80 S=59500 Rst Seq=597 Len=0 Win=0
     server > client D=59500 S=80 Ack=597 Seq=121291 Len=1460 Win=8760
     client > server D=80 S=59500 Rst Seq=597 Len=0 Win=0


How to detect
     The problem can be detected by inspecting packet traces of a large,
     interrupted bulk transfer.


4. Security Considerations

   This version of this  memo  does  not  discuss  any  security-related
   implementation problems.  Futures versions most likely will, so secu-
   rity considerations will require revisiting.


5. Acknowledgements

   Thanks to numerous correspondents on the tcp-impl  mailing  list  for
   their  input: Steve Alexander, Mark Allman, Larry Backman, Jerry Chu,
   Alan Cox, Kevin Fall, Richard Fox, Jim Gettys,  Rick  Jones,  Allison
   Mankin,  Neal  McBurnett,  Perry  Metzger,  der Mouse, Thomas Narten,
   Andras Olah, Steve Parker, Francesco  Potorti`,  Luigi  Rizzo,  Allyn
   Romanow,  Jeff  Semke, Al Smith, Jerry Toporek, Joe Touch, and Curtis
   Villamizar.

   Thanks also to Josh Cohen for the traces documenting the "Failure  to
   send a RST after Half Duplex Close" problem.







Paxson, Editor                                                 [Page 27]


ID                 Known TCP Implementation Problems       November 1997


6. References



[Allman97]
     Mark Allman, "Fixing Two BSD TCP Bugs," Technical Report CR-204151,
     NASA       Lewis      Research      Center,      October      1997.
     http://gigahertz.lerc.nasa.gov/~mallman/papers/bug.ps

[RFC1122]
     R. Braden, Editor, "Requirements for Internet Hosts  --  Communica-
     tion Layers," Oct. 1989.

[Dawson97]
     S. Dawson, F. Jahanian, and T. Mitton, "Experiments on Six  Commer-
     cial TCP Implementations Using a Software Fault Injection Tool," to
     appear in Software Practice & Experience, 1997.  A technical report
     version     of     this     paper     can     be     obtained    at
     ftp://rtcl.eecs.umich.edu/outgoing/sdawson/CSE-TR-298-96.ps.gz.

[Jacobson88]
     V. Jacobson, "Congestion Avoidance and Control," Proc. SIGCOMM '88.
     ftp://ftp.ee.lbl.gov/papers/congavoid.ps.Z

[RFC896]
     J. Nagle, "Congestion Control in IP/TCP Internetworks," Jan. 1984.

[Paxson97]
     V. Paxson, "Automated Packet  Trace  Analysis  of  TCP  Implementa-
     tions,"       Proc.      SIGCOMM      '97,      available      from
     ftp://ftp.ee.lbl.gov/papers/vp-tcpanaly-sigcomm97.ps.Z.

[RFC793]
     J. Postel, Editor, "Transmission Control Protocol," Sep. 1981.

[RFC2001]
     W. Stevens, "TCP Slow Start, Congestion Avoidance, Fast Retransmit,
     and Fast Recovery Algorithms," Jan. 1997.


7. Authors' Addresses

   Vern Paxson <vern@ee.lbl.gov>
   Network Research Group
   Lawrence Berkeley National Laboratory
   Berkeley, CA 94720
   USA
   Phone: +1 510/486-7504



Paxson, Editor                                                 [Page 28]


ID                 Known TCP Implementation Problems       November 1997


   Mark Allman <mallman@lerc.nasa.gov>
   NASA Lewis Research Center/Sterling Software
   21000 Brookpark Road
   MS 54-2
   Cleveland, OH 44135
   USA
   Phone: +1 216/433-6586

   Scott Dawson <sdawson@eecs.umich.edu>
   Real-Time Computing Laboratory
   EECS Building
   University of Michigan
   Ann Arbor, MI  48109-2122
   USA
   Phone: +1 313/763-5363

   Ian Heavens <ian@spider.com>
   Spider Software Ltd.
   8 John's Place, Leith
   Edinburgh EH6 7EL
   UK
   Phone: +44 131/475-7015

   Bernie Volz <volz@process.com>
   Process Software Corporation
   959 Concord Street
   Framingham, MA 01701
   USA
   Phone: +1 508/879-6994






















Paxson, Editor                                                 [Page 29]


Html markup produced by rfcmarkup 1.129c, available from https://tools.ietf.org/tools/rfcmarkup/