MailNews

Revision as of 00:19, 28 October 2011

Nemo's notes on making mutt work for newsgroups, via some helper utilities that talk nntp, and scripts which bind them together.

Reading

slrnpull pulls messages into it's own spool. This spool is readable with mutt directly (it seems to treat it as an 'mh' format mailbox.

However, this means that the messages default to being marked as 'read', and unless you control your own spool, you can't even change that!

news2Maildir

news2Maildir is basically an extremely simple news client (not even an nntp client, since it only looks at the local news spool), and ALSO acts as a MDA to a very select number of Maildirs.

It will operate by copying each message from the spool and deliver it to a Maildir folder. Thus messages appear as new and unread to mutt, and can be read, flagged, deleted, as expected.

news2Maildir will need to keep track, per group, as to which message it saw last. Maildir folders should be named exactly as the group is named.

It should operate in a read-only fashion on the spool, though if you operate your own spool, it could have the option to clear it for you. (imho, better to let slrn do it's own job there)

news2Maildir should only fill the Maildir's that are in existance. However, if you operate slrnpull as a user, then perhaps it could update slrnpull.conf appropriately if newsgroup Maildir's are created/deleted.

I propose these newsgroup based Maildirs live within ~/Newsdir

Writing

Write messages in mutt as normal, but massage them on the way out, so they go to the slrnpull outgoing queue.

I believe this can be done by setting the $sendmail variable within mutt - to a custom script.

Can this variable be set as a folder-hook so that it's only set when reading "Newsdir" maildir folders?

Editing in mutt

I wrote a test email in mutt, adding a newsgroup: header, and then saved a local copy of the message as seen in the mutt editor (vim) before saving a local copy, and also setting $sendmail to a script which would save that copy too!

vim copy:

From: Nemo Thorx <nemo@house>
To: xxx@example.com
Cc: 
Bcc: 
Subject: test
Reply-To: 
Organization: Thorx Enterprises
Newsgroups: alt.test

test
.test1
..test2
.
...test3

-- 
  ------------------------------------------ --------------------------
                                                    earth native

note that I had to fill in To: field ... mutt wont let it send without that filled in! (it will let it go with as minimal as 'xxx@' - but if I try just 'xxx', then mutt auto-fills the domain. The custom headers (including 'newsgroups' are not visible in the mail summary screen, but are not lost.

local saved copy:

Date: Thu, 27 Oct 2011 23:09:05 +1000
From: Nemo Thorx <nemo@house>
To: xxx@example.com
Subject: test
Message-ID: <20111027130905.GB24456@nemo.house.cx>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Organization: Thorx Enterprises
Newsgroups: alt.test
User-Agent: Mutt/1.5.20 (2009-06-14)
Status: RO

test
.test1
..test2
.
...test3

-- 
  ------------------------------------------ --------------------------
                                                    earth native

notice the empty headers removed and new headers added in

copy sent to "sendmail" was the same as the local copy, but without the 'Status' header.

So, to make the "sendmail" copy friendly for newsgroups - we need to delete the mail specific headers, pad the leading .lines, and save into the slrn out.going spool.

Easy huh?

thoughts: could the To: header be renamed on the fly to Newsgroups?

Note also that mutt WILL allow sending without a subject, whilst that will fail to nntp (which appears to require at minimum a "Subject: " header)

Minimum post requirement for slrnpull

A post in the var/spool/slrnpull/out.going/ directory needs to be only named "X", and can get away with only From:, Newsgroups: and Subject: headers, along with a body. Upstream servers will fill out the remainder.

The file needs to be protocol encoded - that is, any lines with a leading "." needs that . to be doubled.

For eg, a file named 'X' with these contents

From: Nobody <devnull@home>
Newsgroups: alt.test
Subject: test 2

test
.testone
..testtwo
...testthree
.
test after a terminating .

returns from the same server, in the same slrnpull loop as when it was posted, as:

Path: news.astraweb.com!news-xref3.astraweb.com!not-for-mail
From: Nobody <devnull@home>
Newsgroups: alt.test
Subject: test 2
Date: 27 Oct 2011 12:17:36 GMT
Lines: 4
Message-ID: <4ea94be0$0$14069$c3e8da3$76491128@news.astraweb.com>
Organization: Unlimited download news at news.astraweb.com
NNTP-Posting-Host: a82854f6.news.astraweb.com
X-Trace: DXC=m2DM4:=`Ee2o^d=IeQT`3>L?0kYOcDh@:[^oANOZPWM0m8]Ha;Fb[[:l9B_G1R=S=7YoIj:2_Q5X<fQ:A^;C6QQ0\M[9B[:?8U:
Xref: news.astraweb.com alt.test:2319994

test
testone
.testtwo
..testthree

Note that the '.' on a line itself ended the post!

Headers analysis

common headers that the user (or MUA) has control over...

mail-only headers (will need removing)

• To
• Cc
• Bcc

Common headers

• From
• Subject
• Organization
• Date - auto generated
• Message-ID - auto generated
• User-Agent - auto generated
• References - auto generated? (does mutt use these at all, or only In-Reply-To  ?)
• In-Reply-To
• Reply-To

news-only headers (needs adding or converting)

• Newsgroups
• X-Newsreader

sendmail2newspool design

My ideas so far

• Analyse To/Cc/Bcc for email addresses and if they exist, then send a copy of the message to sendmail (with newsgroups removed from these headers)
• Coalesce all newsgroups from the To/Cc/Bcc headers into a single Newsgroups header.
• Should there be any newsgroups in the 'Reply-To' header, move those into 'Followup-To' header. Leave email addresses in this header intact
• possibly munge the message-ID header for the sake of script vanity?
• prepend . to any .lines in the message
• save file to a valid unique name within slrnpull out.going directory

Handling bounces

Two options

1. sendmail2newspool triggers the slrnpull post (normally otherwise handled by cron?) and if the posting fails, grabs the message out of the reject folder, and moves it into a bounce Maildir of some kind for the user to see

• • pro: instant
  • con: this sort of thing is news2Maildir's job
  • also: dunno if this will work with a system spool

1. news2Maildir checks the reject folder along with it's other checks.

• • pro: this is exactly the point of news2Maildir
  • con: not exactly timely, since this is a cron thing.

Best of both worlds: sendmail2newspool triggers news2Maildir at the end of it's posting, and so we satisfy both efficiency of code, and promptness of seeing the post either immediately accepted or bounced :)

• to check: what happens when there are multiple slrnpull instances running simultaneously?

• See also Maildir on this wiki