Fish slow uploads - Final patch

[buc]

  Several days ago I have sent a patch which tried to solve too slow 
FISH uploads problem. The idea also has been discussed in RedHat 
bugzilla <https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=140750> . 
Now the solution has changed, and it is the best variant which I could find.
 
  Full description of the issue:

  Midnight Commander has a FISH virtual filesystem (FIle transfer over 
SHell). There is one old problem in MC implementation of it -- uploads 
onto remote system.
  To do such an upload, MC should invoke on the remote side a command, 
which reads appropriate number of bytes from stdin and store them in the 
target file. Initially, `dd' command had been chosen for this purpose.

  The decision was such:

( dd bs=4096 count=<size/4096> ; dd bs=<size%4096> count=1 ) | ( cat 
>target_file ; cat >/dev/null )

  (Note: additional cat to /dev/null is needed to flush input on write 
errors).
  But unfortunately, this variant appeared unreliable. The design of 
`dd' does not assume full filling of input buffers on read (see, for 
example, "conv=sync" description in `dd' manual). The design of `ssh' is 
those that sometimes data can be written into pipe by portions of 
different size. Therefore the part of data could remain not read by `dd' 
...
   In the current version, the problem is solved as:

dd bs=1 count=<size> | ..... 

i.e., `dd' reads input byte-by-byte. It is robast, but is very slow and 
grabs a lot of cpu time on the remote system.
                                                                               

  The better decision is possible.
  There is `head' command. It has `-c <number>' option, which does what 
we really need. The necessary amount of data is read from stdin reliably.
  Unfortunately, the remote system (any *NIX-like) may not have `head' 
command, or have another implementation of it. But for such systems we 
can still use `dd' .

  The first solution was:

( head -c <size> -q - || dd bs=1 count=<size> ) | ( cat >target_file ; 
cat >/dev/null )

  Either fast `head' is used, or `dd' as a fallback. Using `-q' and `-'  
we separate GNU `head' from another `head' implementations, which might 
be unreliable (for example, AIX4 has `-c' option but add extra newline 
byte after EOF).

  But "dd bs=1" seems to be too slow, even for fallback...

  The better solution has been found:
1. First, try GNU `head', as the fastest variant:

 ( head -c %lu -q - ) | ( cat >/%s ; cat >/dev/null )

2. Second, use `dd'-based fallback:

  rest=%lu
  while [ $rest -gt 0 ]
  do
    cnt=`expr \( $rest + 255 \) / 256`
    n=`dd bs=256 count=$cnt | tee -a /%s | wc -c`
    rest=`expr $rest - $n`
  done

  Notes:
    - `bs=256' is used, because more large buffer does not give real effect.
    - we do not read tail (dd bs=<remaining> count=1) explicitly, 
because MC never send more bytes until receives a status code.
    - `expr' is used as most portable way for expressions.
  In ideal world, only one `dd' invokation is needed. Unfortunately, 
`dd'  can receive less bytes than `bs' is, and does not report exact 
byte number (only full/partial block counts are reported).
Therefore we use `tee' and then `wc' to count bytes properly.

   `dd'-based variant is fast enough, but using of `head' (when 
possible) can reduce remote cpu time in several times. Therefore it is 
better to try `head' first, anyway.

   The final script is as this:

  >/%s
  res=`exec 3>&1; ( head -c %lu -q - || echo DD >&3 ) | ( cat >/%s; cat >/dev/null)`
  [ "$res" = DD ] && {
     rest=%lu
     while [ $rest -gt 0 ]
     do
       cnt=`expr \( $rest + 255 \) / 256`
       n=`dd bs=256 count=$cnt | tee -a /%s | wc -c`
       rest=`expr $rest - $n`
     done
  }

  and similar for file append case, with some little changes.

  `echo DD' etc. looks ugly, but it is portable :-)

 
    I have tested both head and fallback successfully with remote host 
under Linux, and fallback with remote AIX4. I also checked GNU  and some 
old (1984) UNIX sources of `dd', `tee', `wc' -- all things should be OK.

-- 
		Dmitry Butskoj <buc AT odusz.so-cdu.ru>
		Saint-Petersburg, Russia
		Red Hat Certified Engineer 809003662809495

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.midnight-commander.org/pipermail/mc-devel/attachments/20041208/076f034f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mc-4.6.1-fish-upload.patch
Type: text/x-patch
Size: 2900 bytes
Desc: not available
URL: <http://lists.midnight-commander.org/pipermail/mc-devel/attachments/20041208/076f034f/attachment.bin>