Newsx - An NNTP client for posting and fetching news
====================================================
Version 1.6
Written by Egil Kvaleberg
------
Newsx is an NNTP client for Unix. It will connect to a remote NNTP server
and post outgoing articles batched by the news system, as well as fetch
incoming articles.
It provides the NNTP capabilities required for small local news spools on
installations with NNTP access only through limited ISP accounts. It works
well via a dialup SLIP/PPP connection.
Newsx is also well suited for large spools with normal feeds, being used for
pulling newsgroups from specific NNTP servers that are not distributed in
the usual manner. Since newsx obeys the normal news spool configuration file
and requires little or no specific configuration, the administrative burden
should be minimized.
- Compatible with C News and INN local news servers.
- Configuration relies on standard C News and INN mechanisms for setting
up newsservers and newsgroups, making administration easy.
- Comprehensive error recovery for posting as well as for fetching. Failed
postings will be retried at the next opportunity, while fetching will
resume at the point where it stopped.
- Logging of errors that occurred, as well as article transfer statistics.
Optional log file for actual articles posted, and collection of posted
articles in folders.
- Has been designed to be nice, so as not to overload the remote news
server.
- Uses only standard RFC-977 functions, so should be compatible with the
vast majority of news servers.
- Refers to news history database to prevent fetching of articles already
in the local spool. Will not fetch crossposted articles more than once.
Does not rely on Xref, since many sites does not provide it.
- Interfaces to spam filters like cleanfeed.
Newsx was originally written in C under Linux, but due to the magic of
Gnu automake and autoconf it should now be very portable. Unassisted
system configuration is done by means of said Gnu autoconf.
PULLING NEWS
------------
News transfer via fetching (sucking, slurping, pulling) to local news
spools is sometimes claimed to be an inefficient way of transferring news
compared to connecting newsreaders directly to the remote NNTP server.
The fact of the matter is that if set up and used correctly, exactly the
opposite is the case.
A local news spool allows news transfer to occur at off-peak hours,
thereby decreasing the host server load in the critical period. This will
tie up less modems at peak hours, as well as decreasing the connect time
since the actual transfer will run much quicker.
A fetch-based system is also easier to administrate then the feeding kind,
since every site decides for itself what groups it will exchange with whom.
The major down-side is that article propagation times will be longer. For
sites that are leaf nodes in the news distribution tree, this is usually not
very important.
Much of the problems connected with news pulling is connected to the use
of the RFC-977 NEWNEWS command. Most news servers has implemented this
command in an inefficient manner, which will put severe loads on the
newsserver. Newsx does not use NEWNEWS.
ARCHIVE
-------
In this archive you should find the following files:
README To you what `Drink Me' was to Alice.
INSTALL Documentation re. configuration and installation
COPYING The GNU General Public License
FAQ Frequently asked questions
NEWS Versions
configure For making a `Makefile'
Makefile.in Input to `configure'
README.in README original - make changes here
autogen.sh For rebuild everything maintained by autoconf and automake
configure.ac For autoconf
config.guess For autoconf
config.h.in For autoconf, generated by autoheader
install-sh For autoconf
mkinstalldirs For autoconf
missing For autoconf
acconfig.h For autoconf
aclocal.m4 For autoconf
newsx.spec For building a rpm: rpm -bb newsx.spec
doc Directory for man-pages and misc. documentation
dbz Directory for database source code
src Directory for newsx source code
lib Directory for misc. source code
test Directory for test code
CONFIGURE, COMPILE, AND INSTALL
-------------------------------
1. Set up your local news server first. Newsx auto-configuration depends
on the news server files and directories it finds and where.
2. Glance at ./INSTALL to see if you need any special configuration
options. Run ./configure with the options you prefer.
3. Type `make', and hopefully you will get the newsx executable.
4. Type `su' to become root, and then `make install' to install the
executables and the man pages.
QUICK START -- GENERAL
----------------------
Newsx moves articles between servers, so you need to have a working local
news server with at least one newsgroup in order to test newsx. There are
two basic tests:
1. Can an article locally posted to the newsgroup and queued by the news
server for delivery to the remote server be successfully sent to the
remote server, and
2. Can articles from the remote server be fetched and placed in the
local server's "in" queue.
The "QUICK START WITH INN/C-NEWS" instructions below walk you through these
tests.
By default, an in.hosts/$server file is built automatically and will fetch
all articles in all groups listed in active (except as limited by the sys
file or command line options). It is sometimes worthwhile to set up this
file by hand rather than accept the default.
Special case #1:
* you've been reading news DIRECTLY from one or more REMOTE servers,
* you're now setting up a local server, AND
* you'd like to avoid downloading articles you've already seen.
You can copy your $HOME/.newsrc file and use it as the initial
in.hosts/$server file! The (remote server's) article numbers will be
extracted, and if you use "--forget-inactive" the first time you call
`newsx', groups that were in .newsrc that you've unsubscribed to will be
eliminated from in.hosts/$server, and current and future downloading
will be limited to only the groups you're subscribed to at the time.
(This does not "track" newsgroup subscription changes you make later.)
Special case #2:
* you've been running a local server for some time and have lots of
newsgroups in 'active' that you don't want fetched.
Use the 'active' file or a .newsrc file, remove the article numbers
(assuming they're the local server's article numbers), trim the list
down to the groups you want, and use that as the in.hosts/$server
file.
WARNING:
Newsgroups can have LOTS of articles. Please look at the newsx man page
descriptions of --maxnew, --syncnew, and --maxart before running newsx
to pull articles from the remote server the first time.
QUICK START WITH INN
--------------------
We assume that you already have INN installed. What you have to do is:
SETUP:
1. Do all maintenance of the news system as user `news'. Ensure
that the `.profile' of this user defines a path that includes
`/usr/lib/newsbin'. E.g.:
export PATH="$PATH:/usr/lib/newsbin"
2. Ensure that an outgoing batch has been configured in
`/usr/lib/news/newsfeeds'. If the NNTP host is named `news.acme.net',
a minimum configuration might be:
ME:::
news.acme.net/newsx:*,!junk*,!control*:Tf,Wf:
In many cases, it will be a good to add the last line shown above
to an existing configuration file.
The outgoing batch will be named `news.acme.net' - replace
`news.acme.net' with what is appropriate for your configuration.
The name of the batch does not HAVE the same as the name of the server,
but it is often convenient.
All groups that you do NOT want to exchange with the host in question
should be listed with an ! before it, the `junk' group being a
standard fitment.
NOTE: Depending on how you use the spool, you may want to replace
the special `newsx' in the example above with the particular exclude
string for the newsserver in question.
3. If you haven't done so already, create one or more newsgroups in the
`/usr/lib/news/active' file that you will be fetching from the remote
newsserver:
ctlinnd newgroup acme.test
FETCH TEST:
4. Dry-run test fetching news articles for these groups by doing:
newsx -ddddd -n --maxnew 100 news.acme.net
If the output looks correct, try running it for real (using your
preference of --maxnew, --maxart, and --syncnew options):
newsx -dd --maxnew 100 news.acme.net
5. The articles should now appear in the incoming batch
`/var/spool/news/incoming'. If you invoke:
rnews -U
the incoming articles should appear in the news spool. In the
example, you may look for them in `/var/spool/news/acme/test'.
POST TEST:
6. Using a newsreader, post an article to a suitable test group. Try to
use a group with as small distribution as possible, preferably local
to the external newsserver you are using. If you haven't done so
already, the group must be in the `/usr/lib/news/active' file.
The next time that `rnews' is invoked, the articles will be output to
the spool file `/var/spool/news/outgoing/acme' specified in the
`/usr/lib/news/newsfeeds' file.
7. Run:
newsx -ddddd -n acme news.acme.net
to do a `dry test' to ensure that everything is set up correctly. No
external connection is required.
8. Now, fire up the connection and do it for real:
newsx -dd acme news.acme.net
Check that everything runs smoothly. The `-dd' may be removed when you
are sure everything works as expected. To maintain a log of posted
articles, as well as a folder containing all articles, you can do:
newsx -l posted.log -f posted acme news.acme.net
9. You may want to arrange for automatic news exchange by using `crontab'.
It is usually good to invoke `newsrun' before and after the newsx
session.
10. For more information about how to set up news and other subjects related
to use of ordinary ISP accounts, you might want to consult:
http://www.kvaleberg.com/ISP-Hookup-HOWTO.html
ftp://ftp.sn.no/user/egilk/ISP-Hookup-HOWTO.txt
QUICK START WITH C News
-----------------------
We assume that you already have C News installed. What you have to do is:
SETUP:
1. Do all maintenance of the news system as user `news'. Ensure
that the `.profile' of this user defines a path that includes
`/usr/lib/newsbin'. E.g.:
export PATH="$PATH:/usr/lib/newsbin"
2. Ensure that an outgoing batch has been configured in `/usr/lib/news/sys'.
If the NNTP host is named `news.acme.net', a minimum configuration
might be:
ME:all/all::
acme/newsx:all,!junk,!control/all:FL:
The filename of the outgoing batch will be the server's name, `acme' in
the example above. Replace `acme' with what is appropriate for your
configuration.
All groups that you do NOT want to exchange with the host in question
should be listed with an `!' before it, the `junk' group being a
standard fitment.
The L-flag is a double insurance that only actual local postings will
be sent out.
NOTE: Depending on how you use the spool, you may want to replace the
special `newsx' in the example above with the particular exclude string
for the newsserver in question. This is particularly true if you are
sending and receiving news with multiple servers or if the local system
names you're using aren't the name the servers use for themselvers in
"Path:".
3. If you haven't done so already, create one or more newsgroups in the
`/usr/lib/news/active' file that you will be fetching from the remote
newsserver:
addgroup acme.test y
FETCH TEST:
4. Dry-run test fetching news articles for these groups by doing:
newsx -ddddd -n --maxnew 100 acme news.acme.net
If the output looks correct, try running it for real (using your
preference of --maxnew, --maxart, and --syncnew options):
newsx -dd --maxnew 100 acme news.acme.net
5. The articles should now appear in the incoming batch
`/var/spool/news/incoming'. If you invoke:
newsrun
the incoming articles should appear in the news spool. For newsgroup
acme.test, articles would be placed in `/var/spool/news/acme/test/'.
POST TEST:
6. Using a newsreader, post an article to a suitable test group. Try to
use a group with as small distribution as possible, preferably local
to the external newsserver you are using. If you haven't done so
already, the group must be in the `/usr/lib/news/active' file.
The next time that `newsrun' is invoked, the articles will be output
to the spool file `/var/spool/news/outgoing/acme/togo' specified in the
`/usr/lib/news/sys' file.
7. Continue with point 7 in the INN explanation above.
MAILING LISTS
-------------
If you want to be informed about future newsx versions and receive
possible bug alerts, you can send an email message to:
majordomo@kvaleberg.no
with the following line in the message body:
subscribe newsx-announce
If you want to join the newsx discussion list, you can send an
email message to the same address with the following line in the
message body:
subscribe newsx
The Usenet newsgroup news.software.nntp is also a suitable forum for
discussing newsx.
LEGAL MATTER
------------
Copyright 2003 Egil Kvaleberg <egil@kvaleberg.no>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
ACKNOWLEDGMENTS
---------------
The DBZ database manager is copyright 1988 Jon Zeeff, and has been
modified by a number of people.
The NNTP header file is copyright 1994 David Alden and The Ohio State
University.
The getopt library is copyright 1987, 88, 89, 90, 91, 92, 93, 94, 95
Free Software Foundation, Inc.
Autoconf is copyright 1992, 1993, 1994, 1996, 1997 Free Software
Foundation, Inc.
Automake is copyright 1994, 1995, 1996 1997 Free Software Foundation,
Inc.
The wildmat function is written by Rich $alz in 1986.
Ansi2knr is copyright 1989, 1991, 1993, 1994, 1995, 1996 Aladdin Enterprises.
The hashing function used for the INN 2.0 history database is derived
from the RSA Data Security, Inc. MD5 Message-Digest Algorithm, and is
copyright 1990 RSA Data Security, Inc.
INN 2.0 is Copyright (c) 1996 by Internet Software Consortium.
The setenv function is Copyright (c) 1987 Regents of the University of
California.
Please refer to the respective source files for a full description of the
terms that apply.
Thanks to the following individuals for constructive comments and
improvements:
Greg Wooledge <wooledge@kellnet.com>
Riku Saikkonen <rjs@isil.lloke.dna.fi>
Simon J Mudd <sjmudd@redestb.es>
Laurent Frigault <frigault@isicom.fr>
Gerhard Zuber <zuber@berlin.snafu.de>
Bruce Fisher <bruce@smtl.co.uk>
Tim Tuck <tim_tuck@yes.optus.com.au>
Stanislav Protassov <st@sw.org.sg>
Brian Buhrow <buhrow@cats.ucsc.edu>
Michael Faurot <mfaurot@phzzzt.atww.org>
Rachel Polanskis <r.polanskis@nepean.uws.edu.au>
Stoty <stoty@legba.tvnet.hu>
Arne Georg Gleditsch <arnegl@ifi.uio.no>
Randall Shutt <rshutt@extacy.ravenet.com>
Rene Hoejbjerg Larsen <renehl@post1.tele.dk>
J. Richard Sladkey <jrs@foliage.com>
Filip Lingier <filip@filtronix.eunet.be>
Joacim Persson <sp2joap1@ida.his.se>
Andreas Jaeger <aj@arthur.pfalz.de>
Janne Snabb <snabb@niksula.hut.fi>
Frank Tarczynski <ftarz@mindspring.com>
Jason Brown <jbrown@interalpha.co.uk>
Knut Anders Hatlen <kahatlen@riksnett.no>
Adrian Bridgett <adrian.bridgett@poboxes.com>
Paul Tomblin <ptomblin@xcski.com>
Helmut Heller <heller@altoetting-online.de>
Michele Bini <mbini@dada.it>
Per Hedeland <per@erix.ericsson.se>
Uli Zappe <uli@zappe.de>
Nikolay Grigoriev <shadow@aanet.ru>
Peter Maydell <pmaydell@chiark.greenend.org.uk>
Steinar Haug <sthaug@nethelp.no>
Stefan Huelswitt <huels@iname.com>
Winston Edmond <wbe@psr.com>
Petter Gustad <petter@gustad.com>
G. Paul Ziemba <paul@treehouse.napa.ca.us>
Bernhard R. Erdmann <be@berdmann.de>
Briggs, John H <John.H.Briggs@team.telstra.com>
Carlo Fusco <messer_samvise@libero.it>
Gorka Olaizola <gorka@escomposlinux.org>
Niels Heinen <zillion@snosoft.com>
Jochen Schmitt <Jochen@herr-schmitt-de>
Andreas Metzler <ametzler@logic.univie.ac.at>
CONTACT INFORMATION
-------------------
All comments, changes you had to make for specific systems, bug
reports, bug fixes, etc. can be reported to:
http://www.kvaleberg.com/bug/
There is also a mailing list available, as described before.
Although not a requirement of the license terms, you are according
to ancient news software tradition encouraged to send a postcard to
the following address if you find the program useful:
Egil Kvaleberg
Husebybakken 14A
NO-0379 Oslo
Norway
$Id: README.in,v 1.59.2.15 2003/01/22 13:15:25 egil Exp $
