
lurkftp v0.92

What it is: This is the ultimate ftp site lurker program.  It is a simple
program to monitor changes in ftp sites and either just report changes or
mirror changes into a local directory.

So why another mirror program?  Because I couldn't get any of the mirror
programs on sunsite to do what I wanted half-way reliably & efficiently.

lurkftp -h will provide a tersely documented list of options; read the man
page for more details.

Since I'm too lazy to write FTP site handling stuff myself, I took the
easiest-to-find C-based library for that purpose: ftplib-v2 (970315).  I
made a few changes (auto-use of PORT command if PASV isn't supported and
external callability of a few functions so I could rewrite the get and list
commands for my purposes), which are encapsulated in ftplib.patch.
Of course since ftplib is under GPL, I guess that puts lurkftp under GPL,
too unless I can get a usable ftplib as a shared library, or write one myself.

***************************************************************

Changelog

  0.9 - first release 4/23/97

  0.91 - second release 4/25/97
	man page cleaned up & updated
	-L (listing timeout) removed; use -T instead
	-P option output file corruption fixed
	-u (non-anonymous FTP) option added
	-z/Z (auto gzip/gunzip) options added
	-L (use remote ls-lR[.(Z|gz)] file) option added
	-U (detect moved files) option added
	-n (stamp only) option added
	-A (append) option added
	general code cleanups

  0.92 - third release 4/27/97
        more examples added to man page (thanks to Andy Wick)
	Solaris fixes [i.e. removal of BSD calls] (thanks to Andy Wick &
          Tobias Oetiker)
	Fixed -S option's "any output" detection (hopefully)

***************************************************************

KNOWN BUGS

[+: may want to fix; *: definitely want to fix; -: may never fix]

* Writes out new ls file before transfer verified
* Doesn't handle non-UNIX remote sites
* Probably plenty of nasty hidden bugs
+ Some fixed-sized buffers may overflow
- Doesn't handle spaces in dir names very well
- Groups & user names aren't mirrored
- Sockets aren't mirrored
- Exact time isn't used for comparison (only accurate to what ls gives)
- All options in external program option group are obsolete

***************************************************************

More information than you really wanted to know:

I am an ftp site lurker.  I for some reason or another tend to want to know
what the latest & greatest of any software I'm using is at all times.  At
first I just logged into sunsite every day and did an ls -trl on the
Incoming directory, and read interesting-looking .lsm's.  Since I couldn't
always tell what I had seen or not, chkftp.sh was born.  It read a simple
configuration file (~/.chksites: site/dir pairs) and ran the ls -tr's
through ncftp for me, keeping the updated results in "ls files"
(~/.chkls.<site>) and reporting any changes.  The program then automatically
rescheduled itself with at, and at mailed me the results.  When sunsite's
Incoming directory's owner & permissions started changing erratically, I
decided to make a directory filter (parsels, still mostly there as a single
function in ftpsupt.c) to produce normalized listings.  This also allowed me
to monitor larger directory trees.  Around this time someone else wanted to
see the results, so I made chkftp.sh mail to users listed on the command
line.  Then I wanted to download an entire directory for some reason or
another, and after not getting decent results from any of the mirror
programs on sunsite, I modified chkftp.sh slightly to produce domir, which
mirrored a remote directory to a single local directory by diffing the
output of remote dir|parsels and local ls -tRl|parsels.  It then deleted all
excess local files, generated soft links, and piped a long string of get
commands to ncftp to retrieve the missing files.

Now I wanted to no longer be dependent on ncftp, as well as having better
more sophisticated timeouts and timeout recovery, better file retrieval
monitoring and recovery, more customization at the command line level, and
the ability to mail new .lsm files to users.  So, I obtained the nearest
working C-based ftp library (ftplib-v2), and grew parsels.c+chkftp.sh into
ftpsupt.c+lurkftp.c, and integrated the mirror functions.  Thus was born
lurkftp v0.9, and after some more testing and perhaps enhancing, it will
become v1.0.

----------------------------------------------------------------------------
 Thomas J. Moore, Hacker/SysAdmin  | Must ... Kill ... Bugs ... AAARRRGGHHH
 inet:  dark@mama.indstate.edu     | unix ada asm c/c++ mcu ee amiga bored
----------------------------------------------------------------------------
