Changes in 2.5.1 * Fixed a segfault in imap_connection which could occur when SASL was used. * Fixed a segfault in the keyword generation code which was assuming that an article was nul-terminated. Fixed another segfault in the keyword generation code when an article already contained a Keywords: header. Thanks to Nix for the bug reports. * Owing to the US-CERT vulnerability note VU#238019, Cyrus SASL library has slightly changed. imap_connection and nnrpd now handle that change. Otherwise, some answers are too long to be properly computed during SASL exchanges. * Fixed a memory allocation problem which caused nnrpd to die when retrieving via HDR/XHDR/XPAT the contents of an extra overview field absent from the headers of an article. The NEWNEWS command was also affected on very rare cases. Thanks to Tim Woodall for the bug report. * HDR/XHDR/XPAT answers are now robust when the overview database is inconsistent. When the overview schema was modified without the overview database being rebuilt, wrong results could be returned for extra fields (especially a random portion of some other header). The desired header name is now explicitly searched for in the overview information. * Fixed the source which is logged to the news log file for local postings when the local server is not listed in incoming.conf. A wrong name was used, taken amongst known peers. The source is now logged as "localhost". * Fixed a bug in the timecaf storage method: only the first 65535 articles could be retrievable in a CAF, though everything was properly stored. (A Crunched Article File contains all the articles that arrive to the news server during 256 seconds.) The storage token now uses 4 bytes to store the article sequence number for timecaf, instead of only 2 bytes. Thanks to Kamil Jonca for the bug report and also the patch. * Fixed a bug in both timecaf and timehash which prevented them from working on systems where short ints were not 16-bit integers. * When there is not enough space to write an entire CAF header, the timecaf storage manager now uses a larger blocksize. On 32-bit systems, the CAF header is about 300 bytes, leaving about 200 bytes for the free bitmap index (the remaining of a 512-byte blocksize). On 64-bit systems, the size of the CAF header could exceed 512 bytes, thus leaving no room for the free bitmap index. A 1 KB blocksize is then used, or a larger size if need be. * A new CNFS version has been introduced by Miquel van Smoorenburg in the CNFS header. CNFSv4 uses 4 KB blocks instead of 512 bytes, which more particularly makes writes faster. CNFSv4 supports files/partitions up to 16 TB with a 4 KB blocksize. Existing CNFS buffers are kept unchanged; only new CNFS buffers are initialized with that new version. * grephistory -l now returns the contents of the expires history field as well as the hash of the message-ID. Besides, when the storage API token does not exist, grephistory -v now also returns the hash of the requested message-ID. * The check on cancel messages when *verifycancels* is set to true in inn.conf has been changed to verify that at least one newsgroup in the cancel message can be found in the article to be cancelled. This new feature is from Christopher Biedl. The previous behaviour was to check whether the cancel message is from the same person as the original post, which is extremely easy to spoof; besides, RFC 5537 (USEPRO) mentions that "cancel control messages are not required to contain From: and Sender: header fields matching the target message. This requirement only encouraged cancel issuers to conceal their identity and provided no security". * The way the "/remember/" line in expire.ctl works has changed. History retention for an article was done according to its original arrival time; it is now according to its original posting date. Otherwise, unnecessary data may be kept too long in the history file. To achieve that, the HISremember() function in history API now expects a fourth parameter: the article posting time. Note that article expiration has not changed and is still based on arrival time, unless the -p flag is passed to expire or expireover, in which case posting time is used. * The default value for "/remember/" has changed from 10 to 11 because it should be one more than the *artcutoff* parameter in inn.conf, so that articles posted one day into the future are properly retained in history. * auth_krb5 has been rewritten by Russ Allbery to use modern Kerberos APIs. Note that using ckpasswd with PAM support and a Kerberos PAM module instead of this authenticator is still recommended. * A new -L flag has been added by Jonathan Kamens to makehistory so as to specify a load average limit. If the system load average exceeds the specified limit, makehistory sleeps until it goes below the limit. * As UTF-8 is the default character set in RFC 3977, "ctlinnd pause", "ctlinnd readers", "ctlinnd reject", "ctlinnd reserve", "ctlinnd throttle" and "nnrpd -r" commands now require the given reason to be encoded in UTF-8, so that it can be properly sent to news readers. The creator's name given to "ctlinnd newgroup" is also expected to be encoded in UTF-8. * The output of consistency checks for article storage and the history file no longer appears by default when "cnfsstat -a" is used. A new -v flag has been added to cnfsstat so as to see it. * The default path for TLS certificates has changed from *pathnews*/lib to *pathetc*. It only affects new INN installations or generations of certificates with "make cert". Besides, a default value has been added to *tlscapath* because it is required by nnrpd when TLS is used. * gzip(1) is now the default UUCP batcher in send-uucp instead of compress(1) because gzip is more widely available than compress, due to old patent issues. Note that there is no impact on decompression as it is handled by rnews. * cnfsheadconf now uses the Perl core module "Math::BigInt" rather than the deprecated bigint.pl library. When used without specifying a CNFS buffer, it now properly displays the status of all CNFS buffers. Upgrading from 2.4 to 2.5 The following changes require your full attention because a manual intervention may be needed: * In order to process control messages, controlchan now needs the "MIME::Parser" module. Packages are available from most distributions, or you can install the module directly from CPAN ("MIME-tools" in modules/by-module/MIME/, for instance on ftp.perl.org). Perl 5.8.0 or later is recommended for INN. If you are using an earlier version, you will also need the "Encode" module for correct processing of control messages. (It is included with Perl itself in 5.8.0 and later.) * Checkgroups control messages are now differently handled by controlchan: all matching lines in control.ctl will be used for a given checkgroups and a doit action will really be executed (adding, removing and changing the status of newsgroups). You should make sure that your local configuration does not rely on the previous behaviour of only mailing changes, without actually performing them. * You should use the new control.ctl.local file shipped with INN in *pathetc* and, at the same time, update your control.ctl and moderators files. Also make sure that your active.times, distrib.pats and newsgroups files are properly encoded in UTF-8, as it is strongly recommended by RFC 3977. * The overview.fmt file is no longer used by INN. Two new parameters have been added to inn.conf: *extraoverviewadvertised* and *extraoverviewhidden*. Although innupgrade takes care of the change during "make update", you should make sure that your overview database is consistent with all the fields declared in overview.fmt because they will all be advertised, and "Xref:full" forced as the eighth overview field. See the inn.conf(5) man page for more information about these parameters. * The innreport configuration file has slightly changed. The new innreport.conf file shipped with INN should be used and your possible changes backported to this new version. * The $SPOOLBASE variable has been renamed to $SPOOLDIR in innshellvars in order to be more consistent. It impacts shell scripts only. If you import innshellvars and use that variable in your scripts, you will have to rename it. * gpgverify is no longer included in INN, pgpverify now has better support for GnuPG and should be used instead. * The auth_smb authenticator program to check passwords with an SMB authentication is no longer included in INN. It was a stripped-down version of pam_smbpass, wasn't maintained, and likely had security problems. To authenticate to an SMB server such as Samba, use PAM and ckpasswd's PAM support instead. The parameters used by nnrpd to provide TLS support are now *tlscafile*, *tlscapath*, *tlscertfile* and *tlskeyfile* in inn.conf. The sasl.conf file used for that in previous versions of INN is obsolete. innupgrade takes care of the change during "make update". The *nntpactsync* parameter has been renamed to *incominglogfrequency* in inn.conf; innupgrade handles this renaming during the update. In newsfeeds, innfeed should be run directly rather than through startinnfeed. innupgrade will attempt to take care of this modification during "make update". When starting innd by hand, innd can just be run directly rather than using inndstart. If you get error messages about resetting the file descriptor limits, you may need to increase the file descriptor limits. See the sample init script in contrib for an example of how to do this. If you are upgrading from a version prior to INN 2.4, see also "Upgrading from 2.3 to 2.4". Changes in 2.5.0 * Ken Murchison has contributed SASL authentication support for nnrpd, implementing the AUTHINFO SASL section of RFC 4643. If the --with-sasl option is given to "configure", nnrpd will be able to authenticate clients via secure SASL mechanisms. * Julien Elie has implemented in nnrpd the new version of the NNTP protocol described in RFC 3977, RFC 4642 and RFC 4643. Consequently, nnrpd now recognizes the CAPABILITIES command, the HDR and LIST HEADERS commands, the second optional argument to specify a range of articles to LISTGROUP, the OVER command, as well as the ":bytes" and ":lines" metadata items. * Heath Kehoe has added the ability to compress overview data before it is stored in ovdb. It significantly improves the performance of this storage method and reduces the time spent by expireover. See the new --with-zlib option to "configure" and the ovdb(5) man page. * Alexander Bartolich has greatly improved innreport and especially its XHTML output (a XSL transformation is also provided, if needed, in innreport-filter.xslt, in the contrib directory). * inndstart and startinnfeed are no longer part of INN and are no longer used. Instead, a separate setuid root helper program written by Russ Allbery is used to bind to the news ports (and does only that), and is run by innd and nnrpd when necessary. This means that INN may not be able to increase file descriptor limits for itself the way that it could before. If you get error messages about resetting the file descriptor limits, you may need to increase the file descriptor limits as root before running rc.news as the news user. See the sample init script in contrib for an example of how to do this. More information on file descriptor limits can be found in INSTALL. * INN's IPv6 support was largely rewritten by Russ Allbery. IPv4 and IPv6 are now handled through the same code wherever possible, the new IPv6-aware APIs are used everywhere possible, and replacement functions are provided for systems that don't have them yet. The network code is now much more centralized, eliminating lots of duplicate code and adding better IPv6 support to some utilities. * INN now uses autoconf 2.61 or later for configuration. As a result, some "configure" options have changed slightly and more of the standard --*dir options should be supported in lieu of the old INN-specific options. See "configure --help" for the available options. * Thanks to Kirill Berezin, the buffindexed overview method now supports buffers larger than 2 GB. It is not necessary to compile INN with large file support to use such large buffers with buffindexed. Buffindexed is now also more robust with mmaped files and uses more optimized data placement. * tinyleaf, a miniature IHAVE-only leaf server written by Russ Allbery, is now included. See the tinyleaf(8) man page for more information. * controlchan recognizes the new application/news-groupinfo entity described in USEPRO and can handle character set conversions of newsgroup descriptions. The "MIME::Parser" and "Encode" modules are used. Processing control messages has been greatly improved, especially checkgroups: the active and newsgroups files are now properly updated when they are processed, and all matching lines in control.ctl for a given checkgroups are honoured (which for instance allows to use both drop and doit actions for the same checkgroups message). A new control.ctl.local file has also been added in *pathetc*. Rules set in that file override rules in control.ctl, allowing administrators to specify local rules for some control messages without modifying the control.ctl configuration file that comes with INN. It also specifies encodings to use for the newsgroups file. By default, UTF-8 will be used for newsgroup descriptions, as strongly recommended by RFC 3977. * The Perl and Python *filter_mode* hooks are now called when innd is shutting down via either "ctlinnd shutdown" or "ctlinnd xexec" with a new mode value of "shutdown". This will allow the Perl hooks to save filter data across innd restarts without requiring that the news administrator throttle the server first. (Python already had a separate close hook that is also called.) * The legacy innshellvars.pl script has been replaced with a real INN Perl module "INN::Config" for Perl programs. The location of Perl modules can be set with the --with-libperl-dir option to "configure". All Perl scripts shipped with INN have been converted to use that module. You may want to consider using "INN::Config" in your Perl scripts, though innshellvars.pl is still provided with INN. * Support for embedded Tcl filters in innd has been removed. It hasn't worked for some time and causes innd crashes if compiled in (even if not used). If someone wants to step forward and maintain it, we recommend starting from scratch and emulating the Perl and Python filters. * If *strippath* is set in readers.conf, the whole user-supplied Path: header will now be stripped. Previously, the final component of the user-supplied Path: would still be retained. * news2mail can now set the envelope-from address of the mails it sends. A third optional part in news2mail.cf entries has been added by D. Stussy to achieve that. * The -g option to nnrpd is no longer supported. If you are verifying passwords against the system password database, see the ckpasswd(8) man page, and in particular the -s option. (A much better idea would be to just use PAM, which ckpasswd supports.) * Fixed a bug in "ctlinnd renumber" which was resetting the low and high water marks of empty newsgroups in the active file. This command now makes the low water mark one more than the real high water mark. The answers to LIST ACTIVE, GROUP and LISTGROUP have also been fixed to do that. * Support for bzip2-compressed batches (with bunbatch) has been added. * news.daily now processes innfeed dropped files during daily maintainance, running procbatch. * Support for *runasuser* and *runasgroup* parameters in inn.conf allows to set the news user and the news group under which the news server runs. Thanks to Ivan Shmakov for this feature. New other options have been added to configuration files: *ignore* in incoming.conf, *logstatus*, *nnrpdflags* and *verifygroups* in inn.conf, and *log-time-format* in innfeed.conf. The --with-http-dir option has also been added to "configure" to set *pathhttp* in inn.conf. The *nntpactsync* parameter has been renamed to *incominglogfrequency* in inn.conf. * The sasl.conf file has been removed in favour of new parameters in inn.conf to deal with TLS support: *tlscafile*, *tlscapath*, *tlscertfile* and *tlskeyfile*. * The overview.fmt file has been removed in favour of new parameters in inn.conf to deal with transition periods to accommodate overview reconfigurations. It is now possible to specify on the one hand the fields that should be advertised by nnrpd in response to LIST OVERVIEW.FMT and used for HDR, XHDR and XPAT requests (see the new *extraoverviewadvertised* parameter) and on the other hand the additional fields that should be silently generated (see the new *extraoverviewhidden* parameter). * Support for Berkeley DB versions prior to 4.3 has been dropped. You will have to use at least Berkeley DB 4.4; the recommended version is 4.7. * INN now builds entirely free of warnings from GCC with fairly aggressive warning options enabled. This involved lots of cleanup of const strings, signed versus unsigned type handling, correcting printf formats, and other changes that fixed obscure bugs and made INN's code more robust. Russ Allbery has also done considerable cleanup work on some of INN's internals, simplifying, refactoring, and removing duplicate code. * INN's test suite is now much more comprehensive and tests some high-level functions as well as more of the portability and utility function layer. * A lot of work has been done on documentation: improvements of existing documents, new documentation, and proof-reading. Sample configuration files are also more detailed. Changes in 2.4.6 * Fixed the segfault of the radius authenticator when none of the radius servers respond. Thanks to Matija Nalis for this patch. * Fixed a lost initialization in buffindexed, which resolves a potential segfault, thanks to a patch by Kirill Berezin. * INN now properly supports Perl 5.10.0 (and also 5.8.9); Perl filters were causing innd to segfault on a few systems like FreeBSD. * Fixed a long-standing bug which affected Perl hooks for innd: the variable containing the body of an article was not properly created, which caused regular expressions matching new lines to fail. It especially affected filters like Cleanfeed which sometimes failed to detect unwanted articles. To fix that issue, Julien Elie added the use of a shared string, available since Perl 5.7.2, with a fall back to a slower but reliable copy of such bodies in case the function is not available. Using a Perl version superior to 5.7.2 is therefore recommended. * Fixed two bugs which could prevent nnrpd from being run as a daemon in FreeBSD. Thanks to Johan van Selst for having identified the problem and to Kai Gallasch for having provided a testing FreeBSD server. The listening address was not initialized to "::0" or 0.0.0.0 when the -b flag was not used and an incorrect size was given when IPv6 was enabled and the binding done using IPv4. * Some annoying assertion failures occurring in innfeed have been fixed by Russ Allbery and Julien Elie. * Fixed a bug in mod-active for aliased newsgroups. Only "=" was written to the active file. Thanks to D. Stussy for this patch. * Fixed a bug which caused innd not to honour the Ad flag in newsfeeds. * Fixed a bug in the IP address displayed for "localhost" in innd's status file. It was not correctly initialized. * Fixed a permission issue: XHDR and XPAT were not checking the rights the user had to read articles when accessing them by their message-ID. * Fixed a bug in the replies of XHDR, XOVER and XPAT when the newsgroup is empty. Two initial replies were sent instead of one: the right 420 code followed by a wrong 224 code. * When no newsgroup is selected, LISTGROUP now returns the right 412 code (instead of 481). * inncheck now uses a range of permissions to see whether the file modes are correctly set. Therefore, different configurations depending on the security the user wants to enforce on his sytem are possible. * A new improved version of docheckgroups is shipped with INN. The -u flag permits to automatically update the newsgroups file (with a proper number of tabulations and an alphabetical sort), removing obsolete descriptions and adding new ones. A second argument on command-line permits to specify which newsgroups should not be checked, so as not to treat them. * An *email=* keyword has been added by James Ralston to news.daily in order to supply another mail address than the one set at configure time for Usenet daily reports. * An updated moderators file with information about the aioe.*, perl.* and si.* hierarchies is provided; control.ctl is also up to date. * INN supports Berkeley DB 4.7, which is the recommended version to use owing to various bugs affecting previous versions of Berkeley DB. * Other minor bugs have also been fixed. Changes in 2.4.5 * Fixed the "alarm signal" around "SSL_read" in nnrpd: it allows a proper disconnection of news clients which were previously hanging when posting an article through a SSL connection. Moreover, the *clienttimeout* parameter now works on SSL connections. Thanks to Matija Nalis for the patch. * SO_KEEPALIVE is now implemented for SSL TCP connections on systems which support it, allowing system detection and closing the dead TCP SSL connections automatically after system-specified time. Thanks to Matija Nalis for the patch. * Fixed a segmentation fault when an article of a size greater than remaining stack is retrieved via SSL. Thanks to Chris Caputo for this patch. * Fixed a few segfaults and bugs which affected both Python innd and nnrpd hooks. They no longer check the existence of methods not used by the hooked script. An issue with Python exception handling was also fixed, as well as a segfault fixed by Russ Allbery which happened whenever one closes and then reopens Python in the same process. Julien Elie also fixed a bug when reloading Python filters (they were not always correctly reloaded) and a segfault when generating access groups with embedded Python filters for nnrpd. Many thanks to David Hlacik for its bug reports. * The nnrpd.py stub file in order to test Python nnrpd hooks, as mentioned in their documentation, is now installed; only INN.py was previously installed in *pathfilter*. Also fixed a bug in INN.py and add missing methods to it. * Fixed a long-standing bug in innreport which prevented it from correctly reporting nnrpd and innfeed log messages. * Fixed a hang in Perl hooks on (at least) HP/PA since Perl 5.10. * Fixed a compilation problem on some platforms because of AF_INET6 which was not inside a HAVE_INET6 block in innfeed. * Fixed a bug in innfeed which contained thrice the same IPs for each peer; it unnecessarily slowed the peer IP rotation for innfeed. Thanks, D. Stussy, for having seen that. Miquel van Smoorenburg provided the patch. * A new *heavily* improved version of pullnews is shipped with this INN release. This new version is provided by Geraint Edwards. He added no more than 16 flags, fixed some bugs and integrated the backupfeed contrib script by Kai Henningsen, adding again 6 other flags. A long-standing but very minor bug in the -g option was especially fixed and items from the to-do list implemented. Many thanks again to Geraint Edwards. * New headers are accessible through Perl and Python innd filtering hooks. You will find the exact list in the INN Python Filtering and Authentication Hooks documentation (doc/hook-python) and in Python samples. Thanks to Matija Nalis for this addition of new useful headers. * New samples for Python nnrpd hooks are shipped with INN: nnrpd_access.py for access control and nnrpd_dynamic.py for dynamic access control. The nnrpd_auth.py script is now only used for authorization control. See the readers.conf man page for more information (especially the *python_auth*, *python_access* and *python_dynamic* parameters). The documention about INN Python Filtering and Authentication Hooks has also been improved by Julien Elie. Changes in 2.4.4 * Fixed incomplete checking of packet sizes in the ctlinnd interface in the no-Unix-domain-sockets case. This is a potential buffer overflow in dead code since basically all systems INN builds on support Unix domain sockets these days. Also track the buffer size more correctly in the client side of this interface for the Unix domain socket case. * Group blocks in incoming.conf are now correctly parsed and no longer cause segfaults when loading this file. * Fixed a problem with innfeed continuously segfaulting on amd64 hardware (and possibly on lots of 64-bit platforms). Many thanks to Ollivier Robert for his patch and also to Kai Gallasch for having reported the problem and provided the FreeBSD server to debug it. * scanlogs now rotates innfeed's log file, which prevents innfeed from silently dying when its log file reaches 2 GB. * Perl 5.10 support has been added to INN thanks to Jakub Bogusz. * Some news clients hang when posting an article through a SSL connection: it seems that nnrpd's SSL routines make it wrongly wait for data completion. In order to fix the problem, the select() wait is now just bypassed. However, the IDLE timer stat is currently not collected for such connections. Thanks to Kachun Lee for this workaround. * Fixed a bug in the display of the used compressor ("cunbatch" was used if arguments were passed to gzip or bzip2). * Fixed a bug in mailpost and pullnews which prevented useful error messages to be seen. Also add the -x flag to pullnews in order to insert Xref: headers in articles which lack one. * If compiling with Berkeley DB, use its ndbm compatibility layer for ckpasswd in preference to searching for a traditional dbm library. INN also supports Berkeley DB 4.4, 4.5 and 4.6 thanks to Marco d'Itri. * ovdb_init now properly closes stdin/out/err when it becomes a daemon. The issue was reported by Viktor Pilpenok and fixed by Marco d'Itri. * Added support for Diablo quickhash and hashfeed algorithms. It allows to distribute the messages among several peers (new Q flag for newsfeeds). Thanks to Miquel van Smoorenburg for this implementation in INN. * innd now listen on separate sockets for IPv4 and IPv6 connections if the IPV6_V6ONLY socket option is available. There might also be operating systems that still have separate IPv4 and IPv6 TCP implementations, and advanced features like TCP SACK might not be available on v6 sockets. Thanks to Miquel van Smoorenburg for this patch. * The two configuration options *bindaddress* and *bindaddress6* can now be set on a per-peer basis for innfeed. Setting *bindaddress6* to "none" tells innfeed to never attempt an IPv6 connection to that host. Thanks to Miquel van Smoorenburg for this patch. * Added a *nnrpdflags* parameter to inn.conf (modeled on the concept of *innflags*) to permit passing of command line arguments to instances of nnrpd spawned from innd. * A new inn.conf parameter called *pathcluster* has been added: it allows to append a common name to the Path: header on all incoming articles. *pathhost* and *pathalias* (if set) are still appended to the path as usual, but *pathcluster* is always appended as the last element (e.g. on the leftmost side of the Path: header). Thanks to Miquel van Smoorenburg for this feature. * simpleftp has been rewritten to use "Net::FTP". Indeed, ftp.pl is no longer shipped with Perl 5 and the script did not work. * perl-nocem will now check for a timeout and re-open the socket if required. Additionally, perl-nocem will switch to cancel_ctlinnd in case cancel_nntp fails after sending the Message-ID. Thanks to Christoph Biedl for the patch. A more detailed documentation has also been written for perl-nocem(8). * The RADIUS configuration is now wrapped in a "server {}" block in radius.conf. * Checkgroups when there is nothing to change no longer result in sending a blank mail to administrators. Besides, no mail is sent by controlchan for the creation of a newsgroup when the action is "no change". * Checkgroups are now properly propagated even though the news server does not carry the groups they are posted to. * controlchan and docheckgroups now handle wire format messages so that articles from the spool can be directly fed to them. * Newgroup control messages for existing groups now change their description. If a mail is sent to administrators, it reminds them to update their newsgroups file. It also warns when there are missing or obsolete descriptions. Furthermore, the newsgroups file is now written prettier (from one to three tabulations between the name of the group and its short description) and to.* groups cannot be created. * The sample control.ctl file has been extensively updated. * Fixed empty LISTGROUP replies which were not terminated. Thanks to David Canzi for the patch. * In response to a LIST [file] command, if the file does not exist, we assume it is not maintained and return 503 instead of 215 and an empty file. Moreover, capability to LIST ACTIVE.TIMES for a wildmat pattern as its third argument has been added in order to select wanted newsgroups. * inews now tries to authenticate if it does not receive a 200 return code after MODE READER. Indeed, it might be able to post even with a 201 return code and also with another codes like 440 or 480. * If creating a new history file, set the ownership and mode appropriately. inncheck also expects fewer things to be private to the news user. Most of the configuration files will never contain private information like passwords. * Other minor bug fixes and documentation improvements. Changes in 2.4.3 * Previous versions of INN had an optimization for handling XHDR Newsgroups that used the Xref: header from overview. While this does make the command much faster, it doesn't produce accurate results and breaks the NNTP protocol, so this optimization has been removed. * Fixed a bug in innd that allowed it to accept articles with duplicated headers if the header occurred an odd number of times. Modified the programs for rebuilding overview to use the last Xref: header if there are multiple ones to avoid problems with spools that contain such invalid articles. * Fixed yet another problem with verifying that a user has permissions to approve posts to a moderated group. Thanks, Jens Schlegel. * Increase the send and receive buffer on the Unix domain socket used by ctlinnd. This should allow longer replies (particularly for innstat) on platforms with very low default Unix domain socket buffer sizes. * rnews's handling of articles with nul characters, NNTP errors, header problems, and deferrals has been significantly improved. * Thomas Parmelan added support to send-uucp for specifying the funnel or exploder site to flush for feeds managed through one and fixed a problem with picking up old stranded work files. * Many other more minor bug fixes, optimization improvements, and documentation fixes. Changes in 2.4.2 * INN is now licensed under a less restrictive license (about as minimally restrictive as possible shy of public domain), and the clause similar to the old BSD advertising clause has been dropped. * "make install" and "make update" now always install the newly built binaries, rather than only installing them if the modification times are newer. This is the behavior that people expect. "make install" now also automatically builds a new (empty) history database if one doesn't already exist. * The embedded Tcl filter code has been disabled (and will be removed entirely in the next major release of INN). It hasn't worked for some time and causes innd crashes if compiled in (even if not used). If someone wants to step forward and maintain it, I recommend starting from scratch and emulating the Perl and Python filters. * ctlinnd should now successfully handle messages from INN up to the maximum allowable packet size in the protocol, fixing problems sites with many active peers were having with innstat output. * Overview generation has been fixed in both makehistory and innd to follow the rules in the latest NNTP draft rather than just replacing special characters with spaces. This means that the unfolding of folded header lines will not introduce additional, incorrect whitespace in the overview data. * nnrpd now uniformly responds with a 480 or 502 status code to attempts to read a newsgroup to which the user does not have access, depending on whether the user has authenticated. Previously, it returned a 411 status code, claiming the group didn't exist, which confuses the reactive authentication capability of news readers. * If a user is not authorized to approve articles (using the "A" *access* control in readers.conf), articles that include Approved: headers will be rejected even if posted to unmoderated groups. Some other site may consider that group to be moderated. * The configuration parser used for readers.conf and others now correctly handles "#" inside quoted strings and is more robust against unmatched double quotes. * Messages mailed to moderators had two spaces after the colons in the headers, rather than one. This bug has been fixed. * A bug that could cause heap corruption and random crashes in innd if INN were compiled with Python support has been fixed. * Some problems with innd's tracking of article size and enforcement of the configured maximum article size have been fixed. * pgpverify will now correctly verify signatures generated by GnuPG and better supports GnuPG as the PGP implementation. * INN's code should now be more 64-bit clean in its handling of size_t, pointer differences, and casting of pointers, correcting problems that showed up on 64-bit platforms like AMD64. * Improved the error reporting in the history database code, in inews, in controlchan, and in expire. * Many other, more minor bugs have also been fixed. Changes in 2.4.1 * SECURITY: Handle the special filing of control messages into per-type newsgroups more robustly. This closes a potentially exploitable buffer overflow. Thanks to Dan Riley for his excellent bug report. * Fixed article handling in innd so that articles without a Path: header (arising from peers sending malformatted articles or injecting malformatted articles through rnews) would not cause innd to crash. (This was not exploitable.) * Fixed a serious bug in XPAT handling, thanks to Tommy van Leeuwen. * "configure" now looks for sendmail only in /usr/sbin and /usr/lib, not on the user's path. This should reduce the need for --with-sendmail if your preferred sendmail is in a standard location. * The robustness of the tradindexed overview method has been further increased, handling more edge cases arising from corrupted databases and oddly-named newsgroups. * innd now never decreases the high water mark of a newsgroup when renumbering, which should help ameliorate overview and active file synchronization problems. * Do not close and reopen the history file on ctlinnd reload when the server is paused or throttled. This was breaking ctlinnd reload all during a server pause. * Various minor portability and compilation issues fixed. Substantial numbers of compiler warnings have been cleaned up, thanks largely to work by Ilya Kovalenko. * Multiple other more minor bugs have been fixed. * Documentation and man pages have been clarified and updated. Upgrading from 2.3 to 2.4 The inn.conf parser has changed between INN 2.3 and 2.4. Due to that change, options in inn.conf that contain whitespace or a few other special characters must be quoted with double quotes, and empty parameters (parameters with no value) are not allowed. INN 2.4 comes with a script, innupgrade, run automatically during "make update", that will attempt to fix any problems that it finds with your inn.conf file, saving the original as inn.conf.OLD. This change is the beginning of standardization of parsing and syntax across all of INN's configuration files. The history subsystem now has a standard API that allows other backends to be used. Because of this, you now need to specify the history method in inn.conf. Adding: hismethod: hisv6 will tell INN to use the same history backend as was used in previous versions. innupgrade should take care of this for you. ovdb is known to have some locking and timing issues related to how nnrpd shuts down (or fails to shut down) the overview databases. If you have stability problems with ovdb, try setting *readserver* to true in ovdb.conf. This will funnel all ovdb reads through a single process with a cleaner interface to the underlying Berkeley DB database. If you use Perl authentication for nnrpd (if *nnrpdperlauth* in inn.conf is true), there have been major changes. See "Changes to Perl Authentication Support for nnrpd" in doc/hook-perl for details. Similarly, if you use Python authentication for nnrpd (if *nnrpdpythonauth* in inn.conf is true), there have been major changes. See "Changes to Python Authentication and Access Control Support for nnrpd" in doc/hook-python for details. If you use send-uucp, it has been completely rewritten and now takes a configuration file to specify its behavior. See its man page for more information. If you use sendbatch, it is no longer included in INN since the new send-uucp can handle all of the same functionality. The wildmat API has been renamed (to uwildmat and friends; see uwildmat(3) for the interfaces) to distinguish it from Rich $alz's original version, since it now supports UTF-8. This may require changes in other software packages that link against INN's libraries. If you are upgrading from a version prior to INN 2.3, see "Upgrading from 2.2 to 2.3". Changes in 2.4.0 * IPv6 support has been added, disabled by default. If you have IPv6 connectivity, build with --enable-ipv6 to try it. There are no known bugs, but please report any problems you find (or even successes, if you use an unusual platform). There are a few changes of interest; further information is available in doc/IPv6-info. * The tradindexed overview method has been completely rewritten and should be considerably more robust in the face of system crashes. A new utility, tdx-util, is provided to examine the contents of the overview database, repair inconsistencies, and rebuild the overview for particular groups from a tradspool news spool. See tdx-util(8) for more details. * The Perl and Python authentication hooks for readers have been extensively overhauled and integrated better with readers.conf. See the Changes sections in doc/hook-perl and doc/hook-python for more details. * nnrpd now optionally supports article injection via IHAVE, see readers.conf(5). Any articles injected this way must have Date, From, Message-ID, Newsgroups, Path, and Subject headers. X-Trace and X-Complaints-To headers will be added if the appropriate options are set in readers.conf, but other headers will not be modified/inserted (e.g. NNTP-Posting-Host, NNTP-Posting-Date, Organization, Lines, Cc, Bcc, and To headers). * nnrpd now handles arbitrarily long lines in POST and IHAVE; administrators who want to limit the length of lines in locally posted articles will need to add this to their local filters instead. * nnrpd no longer handles the poorly-specified RFC 977 optional fourth argument to the NEWGROUPS command specifying the "distributions" that the command was supposed to apply to. Clients that use that argument will break. There are not believed to be any such clients, and it's easy enough to just filter the returned list of newsgroups (which is generally fairly short) to achieve the same results. * nnrpd no longer accepts UTC as a synonym for GMT for NEWGROUPS or NEWNEWS. This usage was never portable, and was rejected by the NNTP working group. It is being removed now in the hope that it will be caught before anyone starts to rely on it. * innfeed supports a new peer parameter, *backlog-feed-first*, that if set to true feeds any backlog to a peer before new articles, see innfeed.conf(5). When used in combination with *max-connections* set to 1, this can be used to enforce in-order delivery of messages to a peer that is doing Xref slaving, avoiding cases where a higher-numbered message is received before a lower-numbered message in the same group. * Several other, more minor protocol issues have been fixed: connections rejected due to the connection rate limiting in innd receive 400 replies instead of 504 or 505, and ARTICLE without an argument will always either retrieve the current article or return a 423 error, never advance the current article number to the next valid article. See doc/compliance-nntp for all of the known issues with INN's compliance with the current NNTP draft. * All accesses to the history file for all parts of INN now go through a generic API like the storage and overview subsystems do. This will eventually allow new history implementations to be dropped in without affecting the rest of INN, and will significantly improve the encapsulation of the history subsystem. See the libinnhist(3) man page for the details of the interface. * INN now uses a new parser for the inn.conf file. This means that parameters containing whitespace or other special characters must now be quoted; see inn.conf(5). It fixes the long-standing bug that certain values must be included in inn.conf even if using the defaults for the use of shell or Perl scripts, and it will serve as the basis for standardizing and cleaning up the configuration file parsing in other parts of INN. innupgrade is run during "make update" and should convert an existing inn.conf file for you. * send-uucp has been replaced by a completely rewritten version from Marco d'Itri, Edvard Tuinder, and Miquel van Smoorenburg, which uses a configuration file that specifies batch sizes, compression methods, and hours during which batches should be generated. The old sendbatch script has been retired, since send-uucp can now handle everything that it did. * Two "configure" options have changed names: --with-tmp-path is now --with-tmp-dir, and --with-largefiles is now --enable-largefiles, to improve consistency and better match the "autoconf" option guidelines. * Variables can now be used in the newsfeeds file to make it easier to specify many similar feeds or feed patterns. See the newsfeeds(5) man page for details. * Local connections to INN support a new special mode, MODE CANCEL, that allows efficient batch cancellation of messages. This is intended to be the preferred interface for external spam and abuse filters like NoCeM. See "CANCEL FEEDS" in innd(8) for details. * Two new options, *nfsreader* and *nfswriter*, have been added to inn.conf to aid in building NFS based shared reader/writer platforms. On the writer server configure *nfswriter* to true and on all of the readers configure *nfsreader* to true; these options add calls to force data out to the NFS server and force it to be read directly from the NFS server at the appropriate moments. Note that it has only been tested on Solaris 8, using CNFS as the storage mechanism and tradindexed as the overview method. * A new option, *tradindexedmmap*, has been added to inn.conf. If set to true (the default), then the tradindexed overview method will use mmap() to access its overview data (in 2.3 you couldn't control this; it always used mmap). * Thanks to code contributed by CMU, innfeed can now feed an IMAP server as well as other NNTP servers. See the man page for innfeed(8) for more information. * An authenticator, auth_smb, that checks a username and password against a remote Samba server is now included. See auth_smb(8) for details. * The wildmat functions in INN now support UTF-8, in a way that should allow them to still work with most simple 8-bit character sets in widespread use. As part of this change, some additional wildmat interfaces are now available and the names have changed (to uwildmat, where "u" is for Unicode). See uwildmat(3) for the details. * The interface between external authenticators and nnrpd is now properly documented, in doc/external-auth. A library implementing this interface in C is provided, which should make it easier to write additional authenticators resolvers. See libauth(3) for details, and any of the existing programs in authprogs/ for examples. * INN now checks to ensure that the configured temporary directory is not world-writeable. Additionally, most (if not all) of the temporary file creation in INN now uses functions that create temporary files properly and safely. * All of the applicable bug fixes from the INN 2.3 STABLE series are also included in INN 2.4. Changes in 2.3.5 * Clients using POST are no longer permitted to provide an Injector-Info: header. * Fixed a bug causing posts with Followup-To: set to a moderated group to be rejected if the posting user didn't have permission to approve postings. * Fixed bugs in inncheck with setuid rnews or setgid inews, in *innconfval* with inn.conf parameters containing shell metacharacters but no spaces, and in parsedate.y with some versions of yacc. Fixed a variety of size-related printf format warnings (e.g., %d vs. %ld) thanks to the work of Winfried Szukalski. Changes in 2.3.4 * LIST ACTIVE no longer returns data when given a single group argument if the client is not authorized to read that group. * XHDR and XPAT weren't correctly parsing article headers, resulting in searches for the header "newsgroup" matching the header "newsgroups". * Made CNFS more robust against crashes by actually syncing the cycbuff headers to disk as was originally intended. Fixed a memory leak in the tradspool code. * Two bugs in pgpverify when using GnuPG were fixed: it now correctly checks for gpgv (rather than pgp) when told to use GnuPG and expects the keyring to be pubring.gpg (not pubring.pgp). * Substantial updates to the sample provided control.ctl file. * Compilation fixes with Perl 5.8.0, Berkeley DB 4.x, current versions of Linux (including with large file support), and Tru64. inndf fixes for ReiserFS. * Various bugs in the header handling in nnrpd have been fixed, including hangs when using virtual domains and improper processing of folded headers under certain circumstances. * Other minor bug fixes and documentation improvements. Changes in 2.3.3 * pgpverify now supports using GnuPG to check signatures (rather than PGP) without the pgpgpg wrapper. GnuPG can check both old-style RSA signatures and new OpenPGP signatures and is recommended over PGP 2.6. If you have GnuPG installed, pgpverify will use it rather than PGP, which means that you may have to create a new key ring for GnuPG to use to verify signatures if you were previously using PGP. * Users can no longer post articles containing Approved: headers to moderated groups by default; they must be specifically given that permission with the *access* parameter in readers.conf. See the man page for more details. * Two bugs in repacking overview index files and a reliability bug with writing overview data were all fixed in the tradindexed overview method, hopefully making it somewhat more reliable, particularly for makehistory. * If rc.news.local exists in the INN binary directory, it will be run with the start or stop argument whenever rc.news is run. This is available as a hook for local startup and shutdown code. * The default history table hash sizes were increased because a too-small value can cause serious performance problems (whereas a too-large hash just wastes a bit of disk space). * The sample control.ctl file has been extensively updated. * Wildmat exclusions ("@" and "!") should now work properly in storage.conf newsgroup patterns. * The implementation of the -w flag for expireover was fixed; previously, the value given to -w to change expireover's notion of the current time was scaled by too much. * Various other more minor bug fixes, standards compliance fixes, and documentation improvements. Changes in 2.3.2 * innxmit can again handle regular filenames as input as well as storage API tokens (allowing it to be used to import an old traditional spool). * Several problems with tagged-hash history files have been fixed thanks to the debugging efforts of Andrew Gierth and Sang-yong Suh. * A very long-standing (since INN 1.0!) NNTP protocol bug in nnrpd was fixed. The response to an ARTICLE command retrieving a message by Message-ID should have the Message-ID as the third word of the response, not the fourth. Fixing this is reported to *possibly* cause problems with some Netscape browsers, but other news servers correctly follow the protocol. * Some serious performance problems with expiration of tradspool should now be at least somewhat alleviated. tradspool and timehash now know how to output file names for removal rather than tokens, and fastrm's ability to remove regular files has been restored. This should bring expiration times for tradspool back to within a factor of two of pre-storage-API expiration times. * Added a sample subscriptions file and documentation for it and innmail. * Various other bug fixes and documentation updates. Changes in 2.3.1 * inews no longer downloads the active file, no longer tries to send postings to moderated groups to the moderator directly, and in general duplicates less of the functionality of nnrpd, instead letting nnrpd handle it. This fixes the problem of inews not working properly for users other than news without being setgid. * Added a man page for ckpasswd. * A serious bug in the embedded Perl authentication hooks was fixed, thanks to Jan Rychter. * The annoying compilation problem with embedded Perl filtering on Linux systems without libgdbm installed should be fixed. * INN now complains loudly at configure time if the configured path for temporary files is world-writeable, since this configuration can be a security hole. * Many other varied bug fixes and documentation fixes of all sorts. Upgrading from 2.2 to 2.3 There may be additional things to watch out for not listed here; if you run across any, please let <inn-bugs@isc.org> know about them. Simply doing a "make update" is not sufficient to upgrade; the history and overview information will also have to be regenerated, since the formats of both files have changed between 2.2 and 2.3. Regardless of whether you were using the storage API or traditional spool under 2.2, you'll need to rebuild your overview and history files. You will also need to add a storage.conf file, if you weren't using the storage API under INN 2.2. A good default storage.conf file for 2.2 users would be: method tradspool { newsgroups: * class: 0 } Create this storage.conf file before rebuilding history or overview. If you want to allow readers, or if you want to expire based on newsgroup name, you need to tell INN to generate overview data and pick an overview method by setting *ovmethod* in inn.conf. See INSTALL and inn.conf(5) for more details. The code that generates the dbz index files has been split into a separate program, makedbz. makehistory still generates the base history file and the overview information, but some of its options have been changed. To rebuild the history and overview files, use something like: makehistory -b -f history.n -O -T /usr/local/news/tmp -l 600000 (change the /usr/local/news/tmp path to some directory that has plenty of temporary space, and leave off -O if you're running a transit-only server and don't intend to expire based on group name, and therefore don't need overview.) Or if your overview is buffindexed, use: makehistory -b -f history.n -O -F Both will generate a new history file as history.n and rebuild overview at the same time. If you want to preseve a record of expired Message-IDs in the history file, run: awk 'NF==2 { print; }' < history >> history.n to append them to the new history file you created above. Look over the new history file and make sure it looks right, then generate the new index files and move them into place: makedbz -s `wc -l < history.n` -f history.n mv history.n history mv history.n.dir history.dir mv history.n.hash history.hash mv history.n.index history.index (Rather than .hash and .index files, you may have a .pag file if you're using tagged hash.) For reader machines, nnrp.access has been replaced by readers.conf. There currently isn't a program to convert between the old format and the new format (if you'd like to contribute one, it would be welcomed gratefully). The new file is unfortunately considerably more complex as a result of its new capabilities; please carefully read the example readers.conf provided and the man page when setting up your initial configuration. The provided commented-out examples cover the most common installation (IP-based authentication for all machines on the local network). INN makes extensive use of mmap(2) for the new overview mechanisms, so at the present time NFS-mounting the spool and overview on multiple reader machines from one central server probably isn't feasible in this version. mmap tends to interact poorly with NFS (at the least, NFS clients won't see updates to the mapped files in situations where they should). (The preferred way to fix this would, rather than backing out the use of mmap or making it optional, to add support for Diablo-style header feeds and pull-on-demand of articles from a master server.) The flags for overchan have changed, plus you probably don't want to run overchan at all any more. Letting innd write overview data itself results in somewhat slower performance, but is more reliable and has a better failure mode under high loads. Writing overview data directly is the default, so in a normal upgrade from 2.2 to 2.3 you'll want to comment out or remove your overchan entry in newsfeeds and set *useoverchan* to false in inn.conf. crosspost is no longer installed, and no longer works (even with traditional spool). If you have an entry for crosspost in newsfeeds, remove it. If you're importing a traditional spool from a pre-storage API INN server, it's strongly recommended that you use NNTP to feed the articles to your new server rather than trying to build overview and history directly from the old spool. It's more reliable and ensures that everything gets put into the right place. The easiest way to do this is to generate, on your old server, a list of all of your existing article files and then feed that list to innxmit. Further details can be found in the FAQ at <http://www.eyrie.org/~eagle/faqs/inn.html>. If you are using a version of Cleanfeed that still has a line in it like: $lines = $hdr{'__BODY__'} =~ tr/\n/\n/; you will need to change this line to: $lines = $hdr{'__LINES__'}; to work with INN 2.3 or later. This is due to an internal optimization of the interface to embedded filters that's new in INN 2.3. Changes in 2.3.0 * New readers.conf file (replaces nnrp.access) which allows more flexible specification of access restrictions. Included in the sample implementations is a RADIUS-based authenticator. * Unified overview has been replaced with an overview API, and there are now three separate overview implementations to choose from. One (tradindexed) is very like traditional overview but uses an additional index file. The second (buffindexed) uses large buffers rather than separate files for each group and can handle a higher incoming article rate while still being fast for readers. The third (ovdb) uses Berkeley DB to store overview information (so you need to have Berkeley DB installed to use it). The *ovmethod* key in inn.conf chooses the overview method to use. Note that ovdb has not been as widely tested as the other overview mechanisms and should be considered experimental. * All article storage and retrieval is now done via the storage API. Traditional spool is now available as a storage type under the storage API. (Note that the current traditional spool implementation causes nightly expire to be extremely slow for a large number of articles, so it's not recommended that you use the tradspool storage method for the majority of a large spool.) * The timecaf storage method has been added, similar to timehash but storing multiple articles in a single file. See INSTALL for details on it. * INN now supports embedded Python filters as well as Perl and Tcl filters, and supports Python authentication hooks. * There is preliminary support for news reading over SSL, using OpenSSL. * To simplify anti-abuse filtering, and to be more compliant with news standards and proposed standards, INN now treats as control messages only articles containing a Control: header. A Subject: line beginning with "cmsg " is no longer sufficient for a message to be considered a control message, and the Also-Control: header is no longer supported. * The INN build system no longer uses subst. (This will be transparent to most users; it's an improvement and modernization of how INN is configured.) * The build and installation system has been substantially overhauled. "make update" now updates scripts as well as binaries and documentation, there is better support for parallel builds ("make -j"), there is less "make" recursion, and far more of the system-dependent configuration is handled directly by "autoconf". libtool build support (including shared library support) should be better than previous releases. * All of the applicable bug fixes from the INN 2.2 STABLE series are also included in INN 2.3. Changes in 2.2.3 * INN no longer installs inews setgid news or rnews setuid root by default. If you need the old behavior, --enable-uucp-rnews and/or --enable-setgid-inews must be given to "configure". See INSTALL for more information. * A security hole when *verifycancels* is turned on in inn.conf (not the default) was fixed. * Message-IDs are now limited to 250 octets to prevent interoperability problems with other servers. * Various other security paranoia fixes have been made. * Embedded Perl filters fixed to work with Perl 5.6.0. * Lots of bug fixes. Changes in 2.2.2 * Various minor bug fixes and a Y2K bug fix. The Y2K bug is in version version 2.2.1 only and will show up after Jan 1st, 2000 when a news reader issues a NEWNEWS command for a date prior to the year 2000. Changes in 2.2.1 * Various bug fixes, mostly notably fixes for potential buffer overflow security vulnerabilities. Changes in 2.2.0 * New storage.conf file (replaces storage.ctl). * New (optional) way of handling non-cancel control messages (controlchan) that serializes them and prevents server overload from control message storms. * Support for actsyncd to fetch active file with ftp; configured by default to use <ftp://ftp.isc.org/pub/usenet/CONFIG/active.Z> if you run actsyncd. Be sure to read the manual page for actsync to configure an actsync.ign file for your site, and test simpleftp if you do not "configure" with wget or ncftp. Also see <ftp://ftp.isc.org/pub/usenet/CONFIG/README>. * Some options to "configure" are now moved to inn.conf (*merge-to-groups* and *pgp-verify*, without the hyphen). * inndf, a portable version of df(1), is supplied. * New cnfsstat program to show stats of CNFS buffers. * news2mail and mailpost programs for gatewaying news to mail and mail to news are supplied. * pullnews program for doing a sucking feed is provided (not meant for large feeds). * The innshellvars.csh.in script is obsolete (and lives in the obsolete directory, for now). $Id: news.pod 8657 2009-10-10 07:58:53Z iulius $