MailNews

Revision as of 13:39, 24 November 2011

Nemo's notes on making mutt work for newsgroups, via some helper utilities that talk nntp, and scripts which bind them together.

Reading

slrnpull pulls messages into it's own spool. This spool is readable with mutt directly (it seems to treat it as an 'mh' format mailbox.

However, this means that the messages default to being marked as 'read', and unless you control your own spool, you can't even change that!

newsspool2Maildir

newsspool2Maildir is basically an extremely simple news client. It's not even an nntp client, since it only looks at the local news spool as created by slrnpull), and ALSO acts as a MDA to the appropriate Maildirs.

It operates by

1. discovering locally "subscribed" newsgroups. (for me, they are ~/Maildir/.nntp.$GROUPNAME Maildir folders)
2. for each group, find new messages from the equivalent slrnpull spool, and copying them over to the Maildir as new messages

Messages can now be read, flagged, deleted, etc, as normal in mutt (or within any other mail client of choice that can read the Maildir, either directly or via IMAP/etc)

It operates in a read-only fashion on the spool, and keeps state of where it read up to for each group via a state file within the group Maildir). Flushing the spool is outside the scope of this utility. (should be handled by slrnpull)

Finally, we check if slrnpull's conf file matches the groups subscribed to within the Maildir and alert to logs if they differ.

• Future functionality may write the slrnpull conf file to match also!

Writing

Write messages in mutt as normal, but massage them on the way out, so they go to the slrnpull outgoing queue.

I believe this can be done by setting the $sendmail variable within mutt - to a custom script.

Can this variable be set as a folder-hook so that it's only set when reading "Newsdir" maildir folders?

Editing in mutt

I wrote a test email in mutt, adding a newsgroup: header, and then saved a local copy of the message as seen in the mutt editor (vim) before saving a local copy, and also setting $sendmail to a script which would save that copy too!

vim copy:

From: Nemo Thorx <nemo@house>
To: xxx@example.com
Cc: 
Bcc: 
Subject: test
Reply-To: 
Organization: Thorx Enterprises
Newsgroups: alt.test

test
.test1
..test2
.
...test3

-- 
  ------------------------------------------ --------------------------
                                                    earth native

note that I had to fill in To: field ... mutt wont let it send without that filled in! (it will let it go with as minimal as 'xxx@' - but if I try just 'xxx', then mutt auto-fills the domain. The custom headers (including 'newsgroups' are not visible in the mail summary screen, but are not lost.

local saved copy:

Date: Thu, 27 Oct 2011 23:09:05 +1000
From: Nemo Thorx <nemo@house>
To: xxx@example.com
Subject: test
Message-ID: <20111027130905.GB24456@nemo.house.cx>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Organization: Thorx Enterprises
Newsgroups: alt.test
User-Agent: Mutt/1.5.20 (2009-06-14)
Status: RO

test
.test1
..test2
.
...test3

-- 
  ------------------------------------------ --------------------------
                                                    earth native

notice the empty headers removed and new headers added in

copy sent to "sendmail" was the same as the local copy, but without the 'Status' header.

So, to make the "sendmail" copy friendly for newsgroups - we need to delete the mail specific headers, pad the leading .lines, and save into the slrn out.going spool.

Easy huh?

thoughts: could the To: header be renamed on the fly to Newsgroups? (but what about the requirement in mutt that To: include email addresses? Two options?

1. the To: field is ignored - ie, any address is ignored
2. user-munged newsgroup names. eg

• • alt@fan.douglas-adams, alt@test,
  • ignorethis@alt.fan.douglas-adams
  • alt.fan.douglas-adams@nntp ...this is probably my favourite, since mutt will auto-append the domain to a the group name when entered into To:, and the domain can be easily thrown away again by sendmail2newspool :)

Note also that mutt WILL allow sending without a subject, whilst that will fail to nntp (which appears to require at minimum a "Subject: " header)

Posting a reply to author from mutt?

I'm not sure if this is possible without getting convoluted. (detect a non-group email address and pipe the message through to sendmail?)

Minimum post requirement for slrnpull

A post in the var/spool/slrnpull/out.going/ directory needs to be only named "X", and can get away with only From:, Newsgroups: and Subject: headers, along with a body. Upstream servers will fill out the remainder.

The file needs to be protocol encoded - that is, any lines with a leading "." needs that . to be doubled.

For eg, a file named 'X' with these contents

From: Nobody <devnull@home>
Newsgroups: alt.test
Subject: test 2

test
.testone
..testtwo
...testthree
.
test after a terminating .

returns from the same server, in the same slrnpull loop as when it was posted, as:

Path: news.astraweb.com!news-xref3.astraweb.com!not-for-mail
From: Nobody <devnull@home>
Newsgroups: alt.test
Subject: test 2
Date: 27 Oct 2011 12:17:36 GMT
Lines: 4
Message-ID: <4ea94be0$0$14069$c3e8da3$76491128@news.astraweb.com>
Organization: Unlimited download news at news.astraweb.com
NNTP-Posting-Host: a82854f6.news.astraweb.com
X-Trace: DXC=m2DM4:=`Ee2o^d=IeQT`3>L?0kYOcDh@:[^oANOZPWM0m8]Ha;Fb[[:l9B_G1R=S=7YoIj:2_Q5X<fQ:A^;C6QQ0\M[9B[:?8U:
Xref: news.astraweb.com alt.test:2319994

test
testone
.testtwo
..testthree

Note that the '.' on a line itself ended the post!

The headers provided by slrn in a post are:

Newsgroups: 
From:
Subject:
Reply-To: 
Followup-To: 
Keywords: 
Summary: 

whilst if posted with minimal additions (ie, Newsgroups:, From: and Subject: only are set), then the following headers are posted (content removed except for User Agent: and Message-ID:)

Newsgroups:
From:
Subject:
User-Agent: slrn/pre1.0.0-18 (Linux)
Message-ID: <slrnjckq9p.6gk.nemo@falcon.house.au>

Headers analysis

common headers that the user (or MUA) has control over...

mail-only headers (will need removing)

• To
• Cc
• Bcc

Common headers

• From
• Subject
• Organization
• Date - auto generated
• Message-ID - auto generated
• User-Agent - auto generated
• References - (mutt does not generate this one, though it reads it)
• In-Reply-To
• Reply-To

news-only headers (needs adding or converting)

• Newsgroups
• X-Newsreader

Headers that appear in every news message

(analysis from one group - 3000 messages over 3 years)

Date
From
Message-ID
Newsgroups
Path
Subject
Xref

sendmail2newspool design

My ideas so far

• Analyse To/Cc/Bcc for email addresses and if they exist, then send a copy of the message to sendmail (with newsgroups removed from these headers)
• Coalesce all newsgroups from the To/Cc/Bcc headers into a single Newsgroups header.
  • note: how to distinguish between newsgroup fake email addresses, and real email addresses??
• Should there be any newsgroups in the 'Reply-To' header, move those into 'Followup-To' header. Leave email addresses in this header intact (assuming this is RFC-OK)
• possibly munge the message-ID header for the sake of script vanity?
• prepend . to any .lines in the message
• save file to a valid unique name within slrnpull out.going directory

Handling bounces

Two options

1. sendmail2newspool triggers the slrnpull post (normally otherwise handled by cron?) and if the posting fails, grabs the message out of the reject folder, and moves it into a bounce Maildir of some kind for the user to see

• • pro: instant
  • con: this sort of thing is news2Maildir's job
  • also: dunno if this will work with a system spool

1. news2Maildir checks the reject folder along with it's other checks.

• • pro: this is exactly the point of news2Maildir
  • con: not exactly timely, since this is a cron thing.

Best of both worlds: sendmail2newspool triggers news2Maildir at the end of it's posting, and so we satisfy both efficiency of code, and promptness of seeing the post either immediately accepted or bounced :)

• to check: what happens when there are multiple slrnpull instances running simultaneously?

leafnode?

This could be used against leafnode without too much difficulty, but leafnode provides a local nntp server - implying the use of a local nntp client. And the whole point of this exercise is to not in fact need that!

However, details?

• newsspool2Maildir needs some directories changing, and that's about it, I think! The stored files (including magic .overview) otherwise work the same.
• mutt2newsspool - needs to write to leafnode's out.going directory, and set it chmod u+r (0400). Failed postings are in failed.postings.
  • for more info: http://leafnode.sourceforge.net/doc_en/fetchnews.8.html

Other Leafnodeisms:

• Needs allow_8bit_headers set lest some articles not be downloaded
• If an article has a header but no body, the header will be lost. (slrnpull keeps the header at least!)
• the leafnode method of knowing which groups are 'interesting' would be easier to sync than the slrnpull config file method

• See also Maildir on this wiki