Newsx - Frequently Asked Questions ================================== Fetching new groups seem to take forever. If you are connected to a server with a big news spool, many newsgroups may contain thousands of news articles. If you start fetching a new newsgroup, newsx will try to fetch all of these. It is often convenient to limit the number of articles fetched in previously unseen groups, for instance by giving the option: --maxnew 100 How do I find out about future newsx versions? Send an email message to majordomo@kvaleberg.no with the following message body: subscribe newsx-announce This should add your name to the newsx announcement list, where future versions and possible bug alerts will be announced. I fetch most of my news from one provider, but in addition I want to pick up `linux.*' from another source. Is it possible? I can't find any newsx configuration file where I can specify this. That is because newsx relies on the configuration file already present in your news spool. This conveniently coordinates the actions of the outbound stream prepared by the news server itself with the inbound stream handled by newsx. Assuming you already has your main newsserver set up (see README), you have to define the second news source. For C News, you would then add to the `sys' file: foo/foo.bar:linux.all/all:FL: Similarily for the INN `newsfeeds' file: foo/foo.bar:linux.*:Tf,Wf: This will tell newsx to only pick up the linux hierarchy to the foo spool, and tell the local news server to place local postings for the `linux' hierarchy in the foo spool. Assuming news.acme.net don't carry these groups, you will never get any linux news from news.acme.net. But to prevent postings to be sent there in vain, and to avoid attempts at fetching these groups, the acme entry needs to be modified: acme/newsx:all,!linux.all,!junk,!control/all:FL: And for INN: acme/newsx:*,!linux.*,!junk,!control:Tf,Wf: To fetch from both sources, you would need to invoke newsx twice: newsx acme news.acme.net & newsx foo news.foo.bar & How do I configure a local newsgroups? If you define newsgroups that should be local to your own machine, you need to exclude these from the external feeds. Assuming these groups are called `local.something', the entry in the C News sys file should be: acme/newsx:all,!local.all,!junk,!control/all:FL: Similarily for the INN newsfeeds file: acme/newsx:*,!local.*,!junk,!control:Tf,Wf: With regards to INN, notice that the trailing `.*' is required. The `local.*' could be written as `local*' but this would then also include any `localities' hierarchy, for instance. I have tried to set up a local newsgroup, but newsx insists on fetching the local group from other servers. This surely means that there is something wrong with the configuration in your sys or newsfeeds file. Ensure that the entry for every external news server excludes the local groups. To find out what actually happens, run newsx with a sufficient debug level (e.g. -ddddd). You should see something like: group local.something disallowed by newsfeeds Should I use the distribution field for limiting the distribution of local newsgroups? In general, the answer is a definitive no. The distribution field is a mechanism that is really best suited for limiting distribution of specific news articles. How can I automatically ensure that my local spool carries all available groups, subject to the limitations of my sys/newsfeeds file? You can make newsx produce a list of newsgroups that the remote newsserver carries, and feed that list to a small script that handles the actual task of local subscriptions. This can be combined with normal news fetching by adding the option: --newlist "|addgroup.all" This will pipe the list of new groups to the script specified, which incidently is available in the lib subdirectory in the newsx distribution. On servers with many newsgroups this does take some time, so it will be useful not to do it too often. A standalone effort may thus be convenient, e.g.: newsx acme news.acme.net --newlist "|addgroup.all" \ --no-fetch --no-post --locks none How can newsx be set up to handle the increasing problem of spamming? Any of the following methods are suggested: A. Use a news source that does spam filtering. B. Do spam filtering by way of external filters like cleanfeed. C. Use NoCeM to stop spam. D. Do spam filtering in the newsreader by use of kill-files and article scoring mechanisms. newsx can be set up to interface with a filter like cleanfeed by use of the --filter option. I've set up filtering, but I've noticed that the articles being filtered out still are transferred. It surely makes sense not to pick up the spam article in the first place? The answer to that is not as obvious as it might seem, and all depends on how large percentage of the message stream that would be stopped by an automated spam filter. The streaming mode of operation used by newsx gives a very significant transfer speed increase over other methods. It does, however, require that articles are transferred in their entirety. Many of the more useful filtering techniques relies on access to the article body anyway. Filtering by article header only means that article transfers must be handled in a less efficient manner, either by using the overview mechanism (which is not entirely standardized and universally available anyway), or by reading the article header and body separately. This extra overhead means that the ratio of spam content must be very high to warrant such a scheme in newsx. There is one exception to this, and that is methods involving spam killing based on the message ID alone. The --filterm option is provided for this purpose. How can newsx be set up to take advantage of the NoCeM effort? The c-nocem package from Olaf Titz can be used with both INN and C News. It is available via: http://www.inka.de/~bigred/sw/c-nocem.txt c-nocem will delete articles that already has reached the spool. Just as importantly, it will mark articles not arrived yet in the history database. Since newsx refers to the history database for every article, this does in fact mean that newsx will NOT fetch any articles that has already been flagged by NoCeM. This method will work better the fresher NoCeM info you can get. It will be sensible to fetch and handle the NoCeM newsgroups before starting a new ordinary transfer. Note that the volume of the NoCeM newsgroups is pretty huge. For a small spool, this will mean that the total news transfer volume will most probably INCREASE when using NoCeM. NoCeM may still serve a useful purpose in reducing the number of undesired articles popping up in the newsreader. How can I make newsx run quicker? You can try to increase the size of the window. The default value is 10. In some circumstances, it may help to increase this, e.g: --window 25 I want to limit the number of groups that I actually fetch from a particular server. Should I use the -e mechanism for that? It depends. It will usually be better to set up your requirements in the sys/newsfeeds file. For instance by splitting into several incoming spools for a single news source. But by all means, the -e approach does work. You must manually edit and add a tag line in the server active file found in `/var/spool/news/in.hosts/spool', where `spool' is the name of the spool that appears in the sys/newsfeeds file. Rearrange it so that all groups you want appears before the tag, and all those that you don't want appears after it. The tag line could look like this: :end Remember to use the corresponding command line option: -e end The thing with the -e option is that since you have limited the number of groups, newsx won't be allowed to start fetching new groups that appear in the active file. So you can't simply add groups with addgroup, you will also have to manually create an entry for them in the in.hosts file. Why did the configure script fail to find the news home directory? The configure script goes through a list of probable directory names in some order of priority to find the one you are using, looking for the either an `bin/config' (C News) or `innshellvars' (INN) file. One reason can be that the directory name you are using is not on that list. See the variable VISIT in configure.ac If your system is using another location, run configure with the option --with-newshome="/my/news/home". Or set the environment variable NEWSHOME before you invoke configure. Please alse send the relevant information to egil@kvaleberg.no, so that future versions of the newsx configure can be updated accordingly. Another reason can be that the configuration file is missing altogether, which must be corrected because it will cause trouble with other news software. It could also be that access to the news home directory has been prohibited. Why did the configure script come up with another news home directory than the one I am using? Presumably, the script have found some traces of a previous news installation, or perhaps some symbolic link to the actual news directory. Try specifying the news home explicitly with the --with-newshome configure option. In the latter case, you may get away with keeping the configuration as it is. Why did the configure say that the configure file is `inaccessible'? Usually because it *is* inaccessible. It is commonplace to protect access to this configure file from other users than he actual news user. The solution here is to install the software as user news, or extend the read permission of the configuration file. Why did the configure script say I am using INN when I am actually using C News, or vice versa? Presumably, the script have found some traces of a previous news installation, and has become confused. Why did the configure script come up with another news spool directory than the one my system is in fact using? The news spool name is read from the news config file. In C News, the file is `bin/config' and the variable is NEWSARTS, In INN, the file is `innshellvars' and the variable is SPOOL. Either way, if this doesn't match the actual directory you are using, there is something wrong with the setup of your news server software. Why did the configure script come up with another history file than the one my system is in fact using? The reason is most probably that the history database is in a non- standard location. If you are running INN, this should have been defined properly by the news configuration file, so there must be something wrong with news server setup. If you are using C News, you can either supply the configure script with the --with-history=FILE option, append a (non-standard) definition of HISTORY to `bin/config', or solve this by installing appropriate symbolic links for the files `history', `history.dir' and `history.pag' from the news home directory. Newsx compiled fine, but doesn't work? Run newsx from the command line, adding the option `-dddddd'. You will then be told what newsx does step by step, and finding out what happens should be self-explanatory ;-). How do I install the binary and manual together with the news binaries? By default, newsx is installed in /usr/local/bin and /usr/local/man, respectively. To install it in NEWS/bin and NEWS/man, try these configure options: --exec-prefix=NEWS --prefix=NEWS How do I make the newsx binary as small as possible? Instead of `make install', try: make install-strip Newsx seems to fetch news without complaint, but when I start up my newsreader I don't see anything? The articles are probably still residing in the `/var/spool/news/in.coming'. To move these files to the news spool proper, you must do in C News: newsrun And in INN: rnews -U But the incoming spool is also empty? The local server has probably rejected the articles, in which case you will find them in `/var/spool/news/in.coming/bad'. Have a look in the logs to find out why they were rejected. For C News, look at `/usr/lib/news/errlog'. When you have corrected the configuration, do: cd /var/spool/news/in.coming mv bad/* . Finally rerun newsrun/rnews as shown in the previous question to try again. My outgoing posts are not going out. Why? Outgoing posts are selected by the newsserver itself, placing a referrence to outgoing articles in the out.going spool for the external newsserver. It will appear as one text line per outgoing article. Newsx excepts the first item on any line to be the file name of the article. If a locally posted article does not appear in out.going, this is either because the newsfeeds (C news: sys) file is configured wrong, or because rnews -U (C news: newsrun) has not been run yet. How can I verify that history database lookups are indeed working? Find the message ID of any article in the newsspool, and feed it to newsx via the -q option. The response should be `yes'. Likewise, any non-existing message ID should respond with `no'. Why doesn't the history lookup function work? The reason is probably that the history database is in a non- standard location, see the question about history configuration. It could also be that the database format is wrong. The standard configuration assumes that the news server uses the `dbz V3.4' database manager, or the `dbz V6.1.1'. If your system has been configured otherwise, you have to manipulate the configuration variables DBZ_VERSION et al. What shall I do? The `ps' process status doesn't work, and/or crashes the system. Give the --without-ps-status option when running configure, and re-make. Also, please run config.guess and report what it says to egil@kvaleberg.no so that the configure script can be made to disable this feature by default for future releases for the system you are using. The dynamic process status probably has some portability problems some systems, but is simply too useful to take away. What on earth is this tiny.server this in included in the distribution? One of the special features of newsx is that it can fetch news using NNTP relayed over Telnet. The point of the tiny.server is to offer people who can log onto machines that does not offer any NNTP connections a way of fetching news. Since tiny.server is a Unix shell script, you don't even need access to a C compiler. Note that the shell script idea makes tiny.server quite inefficient. So tiny.server mostly exists for its hack.value. Also, only a subset of NNTP is offered. Does newsx figure out who to send outgoing news to in the same way that it downloaded news? By design of the news system, it is in fact INN (or C News) that decides on the outgoing news. newsx simply sends on whatever appears in the outgoing queue for the external news server. On the other hand, it is newsx that decides what to fetch. The reason why this appears to be a symmetric process is that newsx interprets the newsfeeds (or sys) file in the same manner as the rest of the news system. I got a `ctlinnd flush failed' message. What does that mean? When running with INN, newsx performs an INN `ctlinnd flush' operation. If this operation fails, the above message will appear. Any error message from ctlinnd should appear just before this message, e.g. `No such site'. Consult the INN documentation for more information: `man ctlinnd' might be a good start. When I fetch articles, I get error messages that says "newsx: ARTICLE 13225 in misc.test did not contain 'acme' in 'Path:'" Take a look at the next line, it should say something in the order of: newsx: Path line was 'news.acme.net!peer.zoomnews.net! newsfeed.somewhere.edu!news.force.no!not-for-mail' The significant part here is the first (or sometimes second) item. For INN to properly know where news should be forwarded, all remote server entries in your 'newsfeeds' file must be set up with the corresponding path 'exclude' name. In this case, probably something like: acme/news.acme.net:*,!junk*,!control*:Tf,Wf: For more information, try: man newsfeeds For C news, the same applies, except the setup file is called 'sys'. By setting the exclude string to `newsx', newsx will add the string `newsx' to the path of incoming news. For uncomplicated news sites, this is a simple manner of handling this problem. In the INN outgoing spool directory, I have found a file with a `.retry' suffix. What is that for? The file with the `.retry' suffix is used by newsx to handle retries of article posting. The exact procedure is: 1. Apply lock for other newsxes 2. Open outgoing spool 3. If a spool.togo exists: rename/append it to spool.retry 4. Rename the spool file to spool.togo 5. Do a ctlinnd flush spoolname 6. If a spool.retry exists: Append it to spool.togo and erase 7. When posting: Any articles with problems go to spool.retry 8. When finished: Get rid of spool.togo 9. Rename spool.retry to spool.togo if it exists I am trying to understand how newsx is compiled, but I find the Makefile and configure script very hard to understand. Why? That is because these files were never meant for human consumption or maintenance. They were produced by the GNU tools automake and autoconf, respectively. The real source files are Makefile.am and configure.ac, which should be a bit more readable. Especially if referring to the automake and autoconf documentation. I find the newsx trace log confusing. It is especially so since newsx by default runs with windowing (streaming) enabled. To get a more logical flow of NNTP requests, try running with windowing disabled: --window 0 What is the significance of the ME field? It depends on which local newsserver you are using. In Cnews, the ME field is the local site's general subscription list for incoming news. In INN, the content of ME is less intuitive, but perhaps more useful: Its content is simply prepended to all other entries, and bears no direct influence on incoming news that INN will accept. Does newsx open up several connections in parallell? No, it doesn't. The --window option controls the number of outstanding requests that may be sent to the other party. $Id: FAQ,v 1.22.2.2 2003/01/22 13:15:25 egil Exp $
