
embedded in application snntpbatch :

                       Features and Options of nntpclient

This software connects to news servers via NNTP and fetches messages which
it stores to the filesystem. Image files in formats uuencode and
MIME-multipart are extracted, decoded, and stored separately.

The program can also upload simple messages with at most one binary file
attached to each of them. Nevertheless more than one file can be given
to produce an according number of messages. The name of each binary file
is attached to the subject line of the message.

The directory structure for storing downloaded data is given by a base
directory, the servername, the portnumber, and the group name:
 $HOME/snntpbatch_download/${servername}.port${portnumber}/${groupname}
The base directory (default: $HOME/snntpbatch_download) can be set with
the option -basedir .
This directory structure can be created by option -subscribe if the
base directory already exists.

Message texts are stored as "msg${messagenumber}.html" in their group's
directory. ${messagenumber} is padded with zeros to 8 digits.
Binary files (images) are stored in a subdirectory called "bin".

The image filenames are read from the message content. If they collide
with a filename which is already in the target directory, then they are
extended by "_" and a 7 digit number. A suffix after the last dot in
the filename is preserved.

Messages can be selected directly via their message id or by filtering
with regular expressions (see man regex, man grep, man ex ...)
If a message filename is already in the target directory, then the message
including all images gets omitted if not "-overwrite on" is set.

Content summaries (generated by -list_interval) are stored in the group's
directory as files with name  list_${first_number}_${last_number}.html.
The file "list_start" contains the first number. The file "list_newest" 
contains the number of the last message header actually read. In case of
connection problems this number can be smaller than original ${last_number}.
The list file is renamed accordingly then. A copy of the most recently
successfully created list file is available as  list_00000000_00000000.html .

Some simple HTML features are inserted into lists and messages.
With your browser bookmark the file  main_index.html  in the basedirectory.
This index of servers and groups is automatically maintained by -subscribe.

Each link of the main index leads to a group index  list_index.html  which
lists links to content summaries. Since the links lead to single message
files, such a summary is called message index.

A message index gets a TITLE which tells about server, group and desired
message range. A header part provides links upward to the main index and
the group index. Another header link points to the first message of the
list which has actually been downloaded.
Also there is a hyperlink (HREF) and a jumplabel (NAME) to the filename
of each message listed. Usually the message files get downloaded later
so these links do not work immediately after creating the message index.

Messages get a TITLE which tells their filename. Three hyperlinks at start
and end of the message text serve for navigation.
"up" points to list_00000000_00000000.html#{messagefilename}.
The messages which are fetched by a single -fetch_interval form a chain:
"previous" points to the  message fetched before.
"next" points to the message which was fetched afterwards. This link gets
its final content only after downloading of this following message started.
Messages downloaded by -fetch simply point to the neighbouring message
numbers wether they exist or not.Naturally, images are embedded by IMG tags.

If a message contains references to other messages, the last one is checked
wether it points to a downloaded message in the same group. If this is true,
then a link to that message is inserted in the header of the one which
is freshly downloaded. Also a link back is inserted into the referenced
message.

The final link structure is generated by option -adjust_links after the
desired messages have been loaded. This operation also generates the group
index (all available message indice in the group). It should be performed as
often as bunch of new messages has been downloaded from a group.

This software is copyright 2001, Thomas Schmitt stic-source@gmx.net and
provided to you without any warranty under an open source BSD license.
(see file COPYING)


 ---------------------------------------------------------------------------
                              Getting Started
 ---------------------------------------------------------------------------

                      --- Setup of Servers and Groups ---

Create the base directory, the directory for your server and the
directories for holding the message id list :

    $ basedir=$HOME/snntpbatch_download
    $ newsserver=INSERT_NEWSERVER_ADDRESS_HERE
    $ mkdir $basedir
    $ cd $basedir
    $ snntpbatch -subscribe $newsserver 119 -

You may want to start your browser and bookmark the main index page. It is
still quite empty yet :

    $ netscape $HOME/snntpbatch_download/main_index.html &
---------------------------------------------------------------------------
no groups subscribed
---------------------------------------------------------------------------

If your news service requires a user-id and a password, you should edit the
main startup-file and insert a line  -login <username> <password>

    $ touch $HOME/.snntpbatchrc
    $ chmod og-rw $HOME/.snntpbatchrc
    $ vi $HOME/.snntpbatchrc

a typical login line would look like

-login JOHNDOE 8HM1RX

If you use more than one news server then you may put this line in a local
startup file (see below). But for a first try, $HOME/.snntpbatchrc is ok.

The following action may last some time and you may skip that until you
really need it.
A list of available newsgroups can be obtained by this command :
    $ snntpbatch -server $newsserver -list_groups
Read the result file in the server's directory
    $ view ${newsserver}.port119/grouplist
presented in raw format as sent from the server. Four fields per line :
        groupname   last_msg_number   first_msg_number   posting_yes_or_no

Create the directories and files needed for the first news group :

    $ newsgroup=alt.binaries.pictures.fantasy-sci-fi
    $ snntpbatch -subscribe $newsserver 119 $newsgroup

If you reload the bookmarked main index, you will see that group listed now.
---------------------------------------------------------------------------
                Serverdirectory YOUR_NEWSSERVER.port119
---------------------------------------------------------------------------
Group alt.binaries.pictures.fantasy-sci-fi
first message |group index |most recent index (start message) |last message
---------------------------------------------------------------------------
The links are still dead until a first -list_interval and -fetch_interval
has been performed.

                            --- Daily usage ---

Create a message index. It will contain some header information about those
messages which have been added since the last -list_interval in $newsgroup
on server $newsserver :

    $ snntpbatch -server $newsserver \
                 -group  $newsgroup \
                 -list_interval new last

This creates a HTML file list_????????_????????.html in the group directory
which can be accessed also as copy  list_00000000_00000000.html .
On the main index page, the link "most recent index" of the group should
be alive now. Also "group index".

Fetch all recently listed articles with subject about "Royo"
but not about "Repost" . After downloading, the HTML hyperlinks in
the messages and index files are adjusted :

    $ snntpbatch -server $newsserver \
                 -group  $newsgroup \
                 -filter subject: '[Rr][Oo][Yy][Oo]' \
                         -and -not subject: '[Rr][Ee][Pp][Oo][Ss][Tt]' \
                 -fetch_interval list_start list_end \
                 -adjust_links - - -

If there is no '-filter' then all messages in the given range get fetched.
As soon as the first message has been downloaded, it is accessible via
the link "(start message)" on the main index page. The "next" link
within a message page gets adjusted when the download of the next message
begins. So wait until that happens before you enter a message page. Or
reload the page if the "next" link is still dead.
The links "first message" and "last message" will get alive only after
the -adjust_links command.

If there are many Royo images or if you do not like them, you may cancel
the run by creating an empty file  nntpclient_stop  in the group directory
    $ touch $basedir/${newsserver}.port119/$newsgroup/nntpclient_stop
this stops the whole program run after the current article is downloaded.
A less rigid way is to set the control index to a ridiculous high value.
    $ echo 2000000000 \
           > $basedir/${newsserver}.port119/$newsgroup/nntpclient_index
this will only end the -fetch_interval command and not the following ones.
You may resume the download with just the same snntpbatch command again.

Usually the indexing, fetching and relinking is done within shell scripts.
So updating all groups can be a matter of single command without any
arguments.

If you see interesting messages in the list which are not fetched by
your automats, then you can get them by simple calls like these two.

Fetch the single message 110233 :

    $ snntpbatch -server $newsserver \
                 -group  $newsgroup \
                 -fetch  110233

Fetch messages 110200 to 110399 . Get only those with subject about
"Space" or "space" :

    $ snntpbatch -server $newsserver \
                 -group  $newsgroup \
                 -filter subject: '[Ss]pace' \
                 -fetch_interval 110200 110399

A filter to prevent double downloading of files, if the names can be guessed
from the subject line. It accepts subjects with '[Ss]paceship' or
'[Qq][Mm][Aa][Nn]'.
If the subject texts of the 'Qman's contain a filename with more than 4
characters before the first dot, this name is checked. The message is only
accepted if the filename (case independend) doesn't already exists in the
hashed directory $filerecord . After the file is accepted, an empty
file with this name is created within the directory. The hashing structure
is created automatically, but at least the parent directory of $filerecord
has already to exist.
For details, see the the explanation of option -filter below and keep in
mind, that a test after -and is only performed if the test before this
-and was true. So only 'Qman's which are not already known get recorded
by "subjectfilerec:".

                 -filter \
                 subject: '[Ss]paceship' \
                 -or -sub \
                    subject: '[Qq][Mm][Aa][Nn]' \
                    -and -not -sub \
                       subjectfile: '......*\..*' \
                       -and subjectfilefound: "$filerecord" \
                    -subend \
                    -and subjectfilerec: "$filerecord -return_true" \
                 -subend \

Complicated filters should be written into a separate file and employed
by -read_from . In that case one does not need \ at line ends and has not to
be aware of most characters which are special to the shell. Quotation marks
'' and "" are interpreted like with the shell. No variable substition is
performed, so you have to insert $filerecord as a constant text :

                 -filter
                 subject: '[Ss]paceship'
                 -or (
                    subject: '[Qq][Mm][Aa][Nn]'
                    -and -not (
                       subjectfile: '......*\..*'
                       -and subjectfilefound: "INSERT_$FILERECORD"
                    )
                    -and subjectfilerec: "INSERT_$FILERECORD -return_true"
                 )

If you expect a lot of downloaded data and your $HOME filesystem is small
you may consider to copy $HOME/snntpbatch_download to a larger filesystem.
Either you may set the new basedirectory by
  -basedir {new_name_of_snntpbatch_download}
at the very start of $HOME/.snntpbatchrc or you may establish a link from
$HOME/snntpbatch_download to the new location. One may also move some
subtrees of the base directory to other filesystems if needed. Here, links
are the only means to keep it working.
A happy snntbatch installion can easily download hundreds of megabytes per
day. So keep an eye on your disk usage.


                            --- Disk Cleaning ---

Remove from the newsgroup directory all messages and binary files which are
older than 14 days. Remove message index files which only contain expired
messages. Also this option adjusts the links of messages and index lists.

   $ snntpbatch -remove_old_files $newsserver 119 $newsgroup -14d

Remove message ids from the directories below  msg_idlist  which are older
than 30 days. These files form a database which prevents multiple
downloading of identical messages. Also they contain some header information

   $ snntpbatch -remove_old_ids  -30d

When creating new directories, -subscribe imposes a limit of at least 3 days
for groups and message ids (via file  sfile_remove_old_files_allowed ).


                              --- Posting ---

For posting messages, a server and a group have to be set. Usually some
id informations are set before posting is performed.

Post three images from stic's sample collection to a test newsgroup :

    $ cd $(cat $HOME/.stic_main_dir)
    $ newsgroup=alt.test.binary.posting
    $ snntpbatch -server  $newsserver \
                 -group   $newsgroup \
                 -post_id my_fantasy_host my_fantasy_name my_organization \
                 -post    "please ignore - just a test" "some text" \
                          images/birds/*

This will result in three messages with subject lines :
    please ignore - just a test - 00000010.jpg (10K)
    please ignore - just a test - 00000055.jpg (15K)
    please ignore - just a test - 00000058.jpg (17K)
and the body texts will just consist of the words "some text".

After each posted message the program waits for confirmation by the server.
    message sent. waiting for confirmation ...
    ... accepted.
In case of failure, you will get a reaction like this
    message sent. waiting for confirmation ...
    ... rejected.
    status = 441 Invalid domain in from-header
this server does not like my_fantasy_host, give it a more realistic one.

Since some servers do send error message without CR LF at the end, the
program might get stuck in this phase. The received part of the error
message should be visible at the terminal's bottom line then. Sorry to say,
there is no remedy against this effect yet.

You will have to -subscribe $newsserver 119 $newsgroup to see the result.
It may last some time until the messages appear on your news server.

Post a reply to the message with id <38B1C855.3D17F85A@home> :

    $ snntpbatch -server   $newsserver \
                 -group    $newsgroup \
                 -post_id  my_fantasy_host my_fantasy_name my_organization \
                 -post_ref '<38B1C855.3D17F85A@home>' \
                 -post     "please ignore - just a test" \
                           "testing follow-ups" \
                           images/misc/00000059.jpg

It is also ok to give the message id without brackets 38B1C855.3D17F85A@home

Cancel the message with id <09400808BE5838D17FCF8088@my_fantasy_host> :

    $ snntpbatch -server   $newsserver \
                 -group    $newsgroup \
                 -post_id  my_fantasy_host my_fantasy_name my_organization \
                 -cancel   '<09400808BE5838D17FCF8088@my_fantasy_host>'

Cancelling does not work with all news servers.
Especially if -post_own_msgid is set to "on".


 ---------------------------------------------------------------------------
                                 Options
 ---------------------------------------------------------------------------

If the first option is # or ! then the whole list of options is ignored.
(Useful for remarks in jobfiles)
Options are processed strictly sequentially. Presets like -server or
-filter only apply to the options that follow them. When in doubt, use the
sequence of options presented here. (i.e -timeout before -server)

Intervals of messages are given by the lowest and highest message number.
The special codes "first" (or "start") and "last" (or "end") are
replaced by the first and last message number as obtained when opening the
group.
The code "list_start" and "list_end" give the first and last number of
the most recent -list_interval run in the group. The code "new" gives
the first unlisted message number (i.e. list_end+1).

Some options require timedefinitions.
Absolute timepoints are defined by the input format MMDDhhmm[[CC]YY][.ss]]
of the UNIX command 'date'. For example:
   021619302000  =  16 Feb 2000 19:30:00 local time
Relative times are preceded by '-' or '+'. They are added to the current
time. '-' therefore points to the past, '+' points to the future. The basic
unit is 1 second. It can be modified by a letter at the end of the number:
   seconds 1s=1 , hours 1h=3600s , days 1d=24h ,
   weeks 1w=7d , months 1m=31d , years y : Xy=X*365.25d+1d
Example:
  -3w  =  3 weeks in the past  =  -1814400s 
Please note that 'm' and 'y' are possibly slightly larger than the actual
calendar time range.

 -basedir directoryname   set the basedir for downloading messages
                      (default: $HOME/snntpbatch_download)

 -login username password  set the login parameters for news servers which
                      require authentication. This option should be stored
                      in the file .snntpbatchrc in the server directory
                      (e.g. $basedir/news.isp.port119/.snntpbatchrc ).
                      This file should have read permission only for its
                      owner. If you do not trust your superuser, store the
                      file on a floppy and make .snntpbatchrc a symbolic link
                      to the floppyfile's name. A missing link target has
                      the same effect as a missing .snntpbatchrc , i.e. none.

 -subscribe servername portnumber groupname
                      create the directories necessary as targets for down-
                      loading. If portnumber is 0 or negative, 119 is used.
                      If the group directory did not already exist, then
                      also create some controlfiles with initial values.
                      If the group name is '-' , then no group directories
                      are created. If the servername is '-' , then no server
                      directory is created (just the basedir and the
                      msg_idlist directories). Directories are created only
                      if they did not already exist. No files are removed.

 -timeout seconds     set the number of seconds to wait for reply
                      After this limit is exceeded, the program closes the
                      connection to the server and tries to re-establish it.
                      Timeout numbers should be larger than the time needed
                      to fetch a large article since an interrupted article
                      has to be re-read from start. Default is 120
 -timeout_bell on|off
                      enable|disable the acustic signal on timout.
                      Default is "on"
 -baudlimit baudrate  restricts the average read bandwith to the given rate.
                      This is done by waiting after too many bytes have been
                      read. Since a few kB are buffered, the bandwidth at
                      the communications port may vary.

 -port portnumber     use the given port at the server. If a server aready
                      has been specified, then the connection is closed and
                      re-established.
                      Usually this option is not necessary since port 119
                      is the default and should work fine.
 -server servername   connect to the given server at the preset port.
                      This should be a very early option since nothing can
                      be fetched and no group can be selected without -server

 -list_groups         write a list of groups into the file grouplist in the
                      server+port's directory
 -group groupname     choose the group on the connected server. This should
                      be an early option since nothing can be fetched 
                      without -group
 -filter   -not | -or | -and | '(' | -sub | ')' | -subend |
           from: | date: | subject: | subjectfile: regular_expression
           subjectfilefound: | subjectfilerec:  option_list
                      set a filter for -fetch_interval  or  -list_interval .
                      The default filter does not reject anything.
                      Basically a filter consists of tests which use regular
                      expressions (like grep and sed). An expression can
                      test one of the message header properties
                         "from:" "date:" "subject:"
                      so a test actually consists of two arguments.
                      Special tests with pseudo properties are:
                         "subjectfile:"
                               try to read a filename from "subject:"
                               and test wether it matches the expression
                         "subjectfilefound:"
                               try to read a filename from "subject:"
                               option_list is a list of directories
                               separated by blanks. Nevertheless it has
                               to be a single shell argument. Use quotation
                               marks to protect the blanks. The filename is
                               searched in these directories. The test is
                               not case sensitive unless "-case" is
                               given at start of the directory list.
                               This test can handle hashed directories.
                         "subjectfilerec:"
                               guess a filename from the subject line and
                               create an empty file in each of the hashed
                               directories listed in option_list. This is
                               done at the end of downloading if a file
                               with the guessed name was actually extracted.
                               Case sensitivity is tiggered with -case
                               and -nocase. The result of the test is
                               'true' if a subject filename could be guessed
                               else it is 'false'. It also can be forced by
                               -return_true resp. -return_false within the
                               list of directories.
                               If a directory does not exist or is not
                               hashed, then it is made a hashed directory.
                               Usually the hashed directories are tested
                               by "subjectfilefound:" to avoid duplicates.
                      Tests can be combined by the logical operations -or,
                      -and, -not and can be grouped by brackets or pairs of
                      -sub and -subend. Precedence: -not, -and, -or .
                      Regular expressions, option lists and brackets should
                      always be surrounded by quotes to prevent the shell
                      parser from interpreting them. With -read_from quotes
                      are only needed to include blank characters.
                      The tests are performed in strict left-to-right order.
                      An important fact concerning speed as well as the
                      effects of "subjectfilerec:" :
                        The right side of -and is only performed if the left
                        side is true. The right side of -or is only
                        performed if the left side is false.
 -overwrite on|off    enable|disable overwriting of already downloaded
                      messages. Image files don't get overwritten but the
                      newly loaded images get new names. Default is "off"
 -html  on|off        enable|disable insertion of HTML tags in lists and
                      messages. Default is "on"
 -group_stat_limits min_articles min_total_size min_max_article_size
                      sets limits for the options -group_statistics and
                      -grouplist_statistics . A group's overview result gets
                      only reported if the group contains at least as many
                      articles as given by parameter min_articles.
                      The total sum of article sizes has to be at least
                      min_total_size. The size of the largest article has
                      to be at least min_max_article_size.
 -group_statistics target_file
                      print a summarized overview of the current group to
                      standard output. If target_file is not '-' or '.' then
                      the output is also appended to this file.
                      The result has the following form
                        Group : ${groupname}
                        Age   : ${oldest}  to  ${youngest}  days
                        Size  : ${x}  bytes in ${y} articles with ${z} lines
                        Avg.  : ${average}  bytes per article
                        Max.  : ${max_size}  bytes in article #{msg_no}
                        Server: ${server}  Port: ${portnumber}  GMT: ${time}
                      If a group contains no articles, then the lines 'Age'
                      'Avg.' and 'Max.' are omitted.
                      If a filter is set, then messages get counted only
                      if they pass that filter.
 -grouplist_statistics start_expression filter_expression target_file
                      print a -group_statistics report for each group in the
                      grouplist (to be generated by -list_groups).
                      Reading and reporting does start with the first group
                      which matches the regular expression start_expression.
                      To be read, a group has to match the regular
                      expression filter_expression and (if set) the limits
                      given by option -group_stat_limits.
                      Both expressions may be inverted by the prefix '-not:'

 -list_interval start_number end_number
                      create a new message index. This file will contain
                      all headers of messages with numbers between
                      start_number and end_number (both included). Test the
                      headers with the current filter (which matches
                      everything if not restricted by option -filter ).
 -fetch message_id    download the message with the given id from the preset
                      server and group. The id may be the message number or
                      the unique message id (looks like a mail address).
 -fetch_interval start_number end_number
                      fetch all messages with numbers between start_number
                      and end_number (both included). Test the messages with
                      the current filter (which matches everything if not
                      restricted by option -filter ). 

 -adjust_links servername portnumber groupname
                      adjust the links in the group's message files to avoid
                      all skipped messages and to to connect messages from
                      different download runs. Also generate the group index
                      list_index.html in the group's directory.
                      If servername, portnumber or groupname are '-' then
                      the settings of the most recent -server, -port and
                      -group options are used.
 -remove_old_files  servername portnumber groupname timedefinition
                      check the files related to the group wether they are
                      older than the defined time and eventually remove
                      them. This option finally runs  -adjust_links .
                      If servername, portnumber or groupname are '-' then
                      the settings of the most recent -server, -port and
                      -group options are used.
                      The group's directory has to contain a file with name
                       sfile_remove_old_files_allowed . This file may contain
                      a timedefinition which prevents remove runs with more
                      recent timepoints. If empty, no limit is imposed.
 -remove_old_ids  timedefinition
                      check the files in the idlist directories wether they
                      are older than the given time.
                      The directory  msg_idlist  has to contain a file with
                      name  sfile_remove_old_files_allowed . This file may
                      contain a timedefinition which prevents remove runs
                      with more recent timepoints. No limit imposed if empty.
 -release_all_locks   above operations are permitted only once at a time.
                      If the program aborts while such an operation, a lock
                      file 'maintenance_lock' may remain and prevent all
                      further actions. This option removes all lock files.

 -post_id  hostname  username  organization
                      set the components of the sender information in posted
                      messages. These will be used with the header lines
                      From , Organization and eventually Message-ID .
                      The default is "somewhere", "anonymous", "none"
                      Personal data from system files or mail settings will
                      not be published by this program.
 -post_own_msgid  on|off
                      if -post_own_msgid is set to "on" then posted
                      messages will get a message id generated from local
                      properties and using the hostname given with -post_id
                      If set to "off" posted messages will rely on the
                      id generator of your news server. Default is "off".
 -post_ref  message_id
                      set the message id for the "Reference:" header line.
                      The following messages will be posted as follow-ups to
                      the message with this message id.
 -post_style  flat|indent|stair
                      set the posting style for multiple binary files.
                      In style "flat" all messages are refering to the one
                      defined by -post_ref (or to none without -post_ref).
                      Style "indent" lets only the first message refer
                      like in flat style. All other messages refer to this
                      first one.
                      Style "stair" lets each message refer to the previous
                      message. The first one refers like in flat style.
 -post  subject message_text [ binary_file [ ... binary_file ]]
 -post_end
                      post messages to the preset server and group. At least
                      the two arguments subject and message_text have to
                      be given. All further arguments up to "-post_end"
                      are considered to be names of binary files.
                      -post_end is not necessary if no other options follow
                      the binary list.
                      If there is more than one binary file given, copies of
                      the message subject and message_text are posted
                      together with these additional files.
                      The format complies to RFC850 (pre-MIME). Binary files
                      are attached as uuencoded lines after message_text.
 -cancel message_id   remove the depicted message from the news servers.
                      -post_id has to be the same as used when posting the
                      article.
 -post_control control_text
                      posts a message with  control_text  as content of the
                      header lines Subject: and Control: and also as message
                      body. Use this option only after reading RFC850 and
                      its successors to learn about the meanings of various
                      control texts.

 -hashfilerec directory file [... file] [-hashfilerec_end]
                      records the given files in the hashdirectories below
                      the given directory. This can be used for -filter test
                      "subjectfilefound:"

 -read_from filename  read options from file. Empty lines and lines which
                      start with '#' or '!' are ignored. All others have to
                      contain at least one option and all its additional
                      arguments.

 -help               Print this text.


 ---------------------------------------------------------------------------
                                 Control Files
 ---------------------------------------------------------------------------

The run can be controled by files in the base directory or in the
group's directory :

  nntpclient_stop ..... if this file exists, the run ends before the next
                        message is fetched or the next option is executed.
                        The file gets removed by nntpclient.
  nntpclient_stop_all . similar to nntpclient_stop but this file does not
                        get removed by nntpclient. Do not forget to remove
                        this file when not needed any more.
  nntpclient_baudlimit  contains a number which limits the bandwidth used
                        by the client. See option -baudlimit above.
                        If a baudlimit is set, this file is checked after
                        each transmission at the communication port.
                        If no limit is set then this file is checked only
                        before fetching messges or executing options. Just
                        like the other files.

The following files are only working in the group's directory :

  nntpclient_pause .... a decimal number in the file tells how many seconds
                        to pause. The number is re-read once in a second.
                        If the file is removed externally then the
                        run is resumed. If the time elapsed is larger than
                        the current number in the file, then the file
                        gets removed and the run is resumed.
  nntpclient_index .... a decimal number in the file sets the current
                        message number in the current -fetch_interval or
                        -list_interval. Useful to skip nasty messages. But
                        use this option with great care.
  nntpclient_multi_bin_tee
                        if this file exists and the subject line matches the
                        regular expression .*[[(]1/[2-9][])].* then the
                        binary data are also stored undecoded in a file in
                        the group's bin directory. The file's name is
                        the subject line with some characters replaced by a
                        '+' and their ASCII-Code in hex. This is done for
                        those characters which are illegal or inconvenient
                        for file names. This filename gets the additional
                        ending .tee .
                        If the subject matches .*[[(][2-9]/[2-9][])].* then
                        the whole article body gets stored in such a .tee
                        file. The intention is to gather all parts of a
                        divided article's binary and put them together with
                        a pipe of cat and uudecode.

This file works in the server+port directory :

  nntpclient_no_xover   disables the use of NNTP extension XOVER which saves
                        a lot of time with -list_interval and filtered
                        -fetch_interval . XOVER might not be available on
                        all servers and the xover_*_* cache files occupy
                        some disk space.

 ---------------------------------------------------------------------------
                               Directory Structure
 ---------------------------------------------------------------------------

Root of the tree is the base directory. Usually $HOME/snntpbatch_download .

   main_index.html
                HTML index of all subscribed newsgroups
   msg_idlist   Directory which memorizes unique message ids to prevent
                duplicate downloading.
      00 .. FA  subdirectories which are used as fields of a hashtable
      sfile_remove_old_files_allowed
                Gives permission to run -remove_old_ids and may contain a
                time limit which protects young files.

   ${servername}.port${portnumber}
                Directory with all server specific data
      grouplist
                Text file with list of groups (result of NNTP command LIST)
      ${groupname}
                Directory with all data specific to a single group
         bin    Directory with extracted binary files

         list_index.html
                Group index. Links to message index files.
         list_00000000_00000000.html
                Copy of newest message index.
         list_${startnumber}_${endnumber}.html
                Message index to messages within number range shown by name
         list_newest
                File containing the number of the last indexed message
         list_start
                File containing the number of the first message in the most
                recent message index
         maintenance_lock
                indicates that a maintenance process (-list_interval
                -adjust_links, -remove_old_files) is in progress and
                no other may be started. The file's content tells about
                hostname,process-id,client-id and the operation in progress.
         msg_${messagenumber}.html
                File containing the message text and HTML links to binary
                files which have been extracted.
         msg_first.html
                Copy of the first message in the directory
         msg_new.html
                Copy of the first message of newest message index
         msg_last.html
                Copy of the last message in the directory
         sfile_remove_old_files_allowed
                Gives permission to run -remove_old_files and may contain a
                time limit which protects young files.

 ---------------------------------------------------------------------------

----------------------------------------------------------------------------
Startupfiles with name '.snntpbatchrc' :
----------------------------------------------------------------------------

On start the program looks in the users $HOME directory for the above
file. If it exists then it is executed like with option -read_from .
After this the same is done within the current working directory's parent
and afterwards with the current working directory itself.
The option -subscribe creates such a file in the group's directory if it
does not already exist. So if the program starts in a group's directory
it automatically connects to the appropriate server and group and has
its base directory set in a suitable way.
Login name and password for a news server may be defined in the server's
directory. Deny read permission to anyone else than yourself.
(see above: option -login).
If a global option is needed (like a different -timeout) then $HOME
is a good place to define it.

