Discourse to NNTP gateway

OffByOne

Paging @codinghorror, re: https://meta.discourse.org/t/emails-are-not-threaded-in-outlook-2013/16501/31

Look, it's not difficult:

The List-* headers have nothing to do with threading.
The References header is what is used by mail clients to construct the conversation threads. It's in the RFC. It's what you can rely on to be sufficient for all mail clients.
You don't need anything more than a correct References header to have proper threading.
It doesn't matter whether References refers to the Message-ID of a mail that the recipient has not received.
Because of how that header is constructed, proper threading still works. Even when the top level mail is absent, mail clients can still thread related mails as children of a "virtual" mail.
The In-Reply-To header is not used for threading. See also 2 bullet points higher: only References is used for that.
When some software sends mails with broken headers, like Discourse did, mail clients may try to construct correct threading through other means. In-Reply-To may be used. The subject and Re: prefix may be used.
If a mail client does that, it's a courtesy, not something you can expect to work on every client, it's not even behaviour you can reasonably expect to be implemented in a client.
You can't even expect the threading to be completely correct (because the mail client has insufficient information for correct threading, it needs to make assumptions which may be wrong).
If you rely on client side workarounds for broken mail implementations to render mails you send correctly, your system is broken by design. Fix it.
Instead of mucking around with headers that you think might have something to do with the problem, read the relevant RFC. You can expect mail clients to implement that, so if you want to get specific behaviour out of a client, the RFC gives you exact guidelines on how to achieve that.
Just guessing and have other people be your testers wastes time, patience and makes you look unprofessional. Throwing stuff at the wall and seeing what sticks is not a viable development style, especially when you have a clear and unambiguous specification of how your software must behave to interoperate correctly with other software.
@rumpelsepp changed headers in a mail archive on his system (see post https://meta.discourse.org/t/emails-are-not-threaded-in-outlook-2013/16501/29). That's the kind of testing you should have done.

See also my post higher up in this topic: http://what.thedailywtf.com/t/discourse-to-nntp-gateway/4660/101?u=offbyone.
It talks about the exact same thing. The only difference is context: my post has NNTP as context, the meta.d discusion has mail as context. NNTP and mail both use the same message format and use exactly the same headers for threading.
You must have read my post I refer to, because you replied in this topic after I made that post.
Since, according to your own guidelines that you defend almost religiously, you're supposed to read all posts in a topic before replying, I can be certain you've read that post (I assume you follow the same guidelines you expect other people to follow; that you're not a "do as I say, don't do as I do" kind of guy).

Jaloopa

@OffByOne said:

Paging @codinghorror

@codinghorror doesn't do paging

OffByOne

@Jaloopa said:

@codinghorror doesn't do paging

Well, since he showed up in this topic earlier without being paged, I thought that summoning him explicitly just maybe could get him here again.

Even if he doesn't show up, writing that post helped me vent my frustrations.

I said uptopic that I don't get irritated easily and all that, but somehow this particular thing really got under my skin.
Maybe it's because that's the functionality I'm working on in DiscoNews, so I'm right in the middle of that subject matter.
Maybe I'm just not as resistant to this kind of... (have tried for 2 minutes to find an apt yet unoffensive word to describe this behaviour, but I can't) as I thought I was.

Jaloopa

whoosh?
s /paging/pagination/

Luhmann

@OffByOne said:

(have tried for 2 minutes to find an apt yet unoffensive word to describe this behaviour, but I can't)

You could use the forum-favorite: BELGIUM

JBert

I only see black blocks. You didn't post a password, did you?

Luhmann

@JBert said:

You didn't post a password, did you?

No my password is ********

Luhmann

@JBert said:

black blocks.

There is only one word on the blocked word list and that is B ELGIUM

JBert

@Luhmann said:

No my password is ********

Now I wonder how many people would use a password consisting of nothing but asterisks...
Then again, not too much filters of the "1 Uppercase, 1 lowercase, 1 number" kind would allow it.

OffByOne

@Jaloopa said:

whoosh?
s /paging/pagination/

That was a genuine woosh. I was too riled up to get your subtle wordplay.

Now that you explained the joke:
Also, have a like.

OffByOne

@Luhmann said:

You could use the forum-favorite: BELGIUM

Requirement failure: the most offensive word in the galaxy is not "unoffensive" ;)

Luhmann

@OffByOne said:

the most offensive word in the galaxy is not "unoffensive"

Better move to a different country then ...

end

You can reply on the meta topic based on the current version. Comparing GitHub headers and ours, the lack of message-Id header was in fact the issue.

PJH

@OffByOne said:

Well, since he showed up in this topic earlier without being paged, I thought that summoning him explicitly just maybe could get him here again.

Seriously? o_O

OffByOne

@PJH said:

Seriously? o_O

Being agitated and frustrated are barriers to Getting The Joke.

I'll wear my badge with appropriate shame, but a look at the badge page tells me I'm in good company ;)

OffByOne

@codinghorror said:

You can reply on the meta topic based on the current version. Comparing GitHub headers and ours, the lack of message-Id header was in fact the issue.

I was contemplating that, but I'm not convinced creating an account on meta.d just to repeat what I said here already is worth the hassle.
You can post a link to my posts on the meta.d topic though, if you think there is relevant information here that hasn't been said there.

I'm hesitant to create an account on meta.d, because some of TDWTF readers who have done so, have experienced this treatment (non-exhaustive list):

their on-topic to-the-point posts got deleted without explanation
when asking a staff member why their posts got deleted, they got no reply
some even got tempbanned (again without explanation)

Some have done a little experiment to find out why that is. They created a sockpuppet on meta.d that has no links to TDWTF whatsoever, whereas their main account has links to this forum/community.
The exact same report that got mistreated by the DiscoDevs when posted with their (TDWTF-linked) main account, got a reasonable response when posted with the sockpuppet account.
It was not a one-off random fluke, this little experiment was repeated a few times with very similar results.

So... Explain to me why I should create an account on meta.d, suffer through TL0 before I can make a decent post, only to be treated as a third class citizen because I'm TDWTF?
If you can give me a good reason and if you want to make an effort to treat TDWTF members on meta.d with respect, I'll consider it.
From my end, I promise to be civilized in my discourse, not use naughty words, I won't troll and I'll be reasonable. I expect the same from you though, or the deal is off.

end

@OffByOne said:

You don't need anything more than a correct References header to have proper threading.

This is patently untrue based on what people reported on meta.

Before:

Message-ID: <54752ba335286_633fee205cc454883a2@tiefighter6-virtual_host.mail>
In-Reply-To: <topic/91@talk.thumbtack.io>
References: <topic/91@talk.thumbtack.io>

After:

Message-ID: <topic/22572/86235@meta.discourse.org>
In-Reply-To: <topic/22572@meta.discourse.org>
References: <topic/22572@meta.discourse.org>

Which corresponds almost exactly to what GitHub was sending...

Message-ID: <discourse/discourse/pull/3006/c64666078@github.com>
In-Reply-To: <discourse/discourse/pull/3006@github.com>
References: <discourse/discourse/pull/3006@github.com>

the only difference is the message-id field was not specified, we were letting the mailserver set it.

OffByOne

@codinghorror said:

This is patently untrue based on what people reported on meta.

It is exactly true and is fully supported by the evidence you provide.

@codinghorror said:

Before:

Message-ID: &lt;54752ba335286_633fee205cc454883a2@tiefighter6-virtual_host.mail&gt;
In-Reply-To: &lt;topic/91@talk.thumbtack.io&gt;
References: &lt;topic/91@talk.thumbtack.io&gt;

After:

Message-ID: &lt;topic/22572/86235@meta.discourse.org&gt;
In-Reply-To: &lt;topic/22572@meta.discourse.org&gt;
References: &lt;topic/22572@meta.discourse.org&gt;

Which corresponds almost exactly to what GitHub was sending...

Message-ID: &lt;discourse/discourse/pull/3006/c64666078@github.com&gt;
In-Reply-To: &lt;discourse/discourse/pull/3006@github.com&gt;
References: &lt;discourse/discourse/pull/3006@github.com&gt;

the only difference is the message-id field was not specified, we were letting the mailserver set it.

The only difference is the References header in your before example refers to a non-existing Message-ID and in your after example it does refer to an existing Message-ID.

You fixed that ID mismatch by creating a Message-ID, so you have control over its value, and using that predictable Message-ID in subsequent References headers. That was the sane solution.

Your other option would have been to somehow retrieve the Message-ID generated by Mandrill of the message you want the References header to point to and use that retrieved value. That would have been a WTF solution.

end

That message ID (the first post in the topic) may or may not exist as an actual email though. Remember you might be notified for a string of responses way after the topic is created.

And a "reply" may have no reference to other posts if it is a @name mention.

OffByOne

@codinghorror said:

That message ID (the first post in the topic) may or may not exist as an actual email though. Remember you might be notified for a string of responses way after the topic is created.

If it exists, no problem.

If it doesn't exist:

@OffByOne said:

It doesn't matter whether References refers to the Message-ID of a mail that the recipient has not received.
Because of how that header is constructed, proper threading still works. Even when the top level mail is absent, mail clients can still thread related mails as children of a "virtual" mail.

This means that if multiple mails are received by a mail client, all with the same References header, referring to the message ID that doesn't correspond to any mail the mail client has previously received, those mails will still be displayed as children of a single, non-existent-in-reality (hence "virtual" in my quote) mail that is synthesised by the mail client especially for the occasion.
How that looks on the screen exactly, is mail client dependent of course.

Anyway, that's not something you can fix, unless you send out a dummy mail with the topic Message-ID, so the replies have a real mail to be attached to retroactively.
Even then you can't guarantee that the top level mail has arrived. It might be eaten by a spam filter for example. It might have been deleted by the user. It might have been moved to a different folder that the reply mail is in.
Those are things you can't fix and nobody expects you to. Your software has done its duty as good as it can.

@codinghorror said:

And a "reply" may have no reference to other posts if it is a @name mention.

If it doesn't reference anything, don't put in a References header. The absence of that header tells the mail client that this is a top level mail.

dkf

@OffByOne said:

If it doesn't reference anything, don't put in a References header. The absence of that header tells the mail client that this is a top level mail.

Unless you're dealing with Outlook, which has its own “special” way of doing things so far as I can tell. (Or at least did for the brief period long ago that I used it, and was one of the reasons I stopped very quickly.) I suggest you stick to following the RFCs.

OffByOne

@dkf said:

Unless you're dealing with Outlook, which has its own “special” way of doing things so far as I can tell. (Or at least did for the brief period long ago that I used it, and was one of the reasons I stopped very quickly.)

Can you elaborate on that "special" way? IIRC Outlook does threading based on subject (ignoring a possible Re: prefix).

In general, working around limitations, bugs or retarded behaviour in specific implementations is an admirable thing to do, but only after
@dkf said:

you stick to following the RFCs.

dkf

@OffByOne said:

IIRC Outlook does threading based on subject (ignoring a possible Re: prefix).

I think it completely ignored the References: header when the title changed. Maybe they've changed that… but maybe not. :(

OffByOne

@dkf said:

I think it completely ignored the References: header when the title changed. Maybe they've changed that… but maybe not. :(

That's... retarded.

The DiscoDevs could try to work around that after implementing the RFC, so well-behaved clients Do the Right Thing™.
My guess is Outlook compares subject lines to do the threading, which should already work with the mails Discourse sends out.

I guess (recent) Outlook looks at the References: header, otherwise the linked thread on meta.d wouldn't exist and having a correct message ID in the References: header wouldn't have fixed the problem for the OP over there.
He reported on the meta.d thread that his issue was solved after Jeff changed Discourse so it generates Message-ID itself.

accalia

@dkf said:

I think it completely ignored the References: header when the title changed. Maybe they've changed that… but maybe not.

IIRC default configuration has it prefer subject to references for threading, but it does handle changing the subject and keeps threading going. At least in outlook 2013 and for emails sent from outlook to outlook.

YMMV

dkf

@accalia said:

IIRC default configuration has it prefer subject to references for threading, but it does handle changing the subject and keeps threading going. At least in outlook 2013 and for emails sent from outlook to outlook.

So they did fix that. I wonder when? I've not used it for… well, over a decade I suppose, but I vaguely remember that only Outlook and Lotus Notes back then had real problems with maintaining a conversation thread properly (and they had different problems too; one didn't handle title changes, one thought references were irrelephant…)

accalia

well threading conversations still isn't default in outlook, but Windows Mail (built-in to windows 8+) does it beautifully so i looked up how to enable it for outlook.

it's a little weird in how it handles the threading but is so awesome that i can pull up old emails from an email chain without hunting through my inbox that i put up with it. I don't get discourse notification emails through outlook much so i don't know how they are handled.

loopback0

@accalia said:

I don't get discourse notification emails through outlook

I just don't get them - I only have them enabled for PMs and I've never had an email when I've received a PM even when offline. That's hardly a new thing, though.

Yamikuronue

In Outlook, when I ask it to find related messages, it seems to do a subject-line search.

loopback0

I always assumed conversations were linked by whatever headers existed, considering I can separately sort by Subject and Conversation in Outlook 2010.

EDIT- that should have been a reply to the topic, not necessarily that post.

FrostCat

@accalia said:

it's a little weird in how it handles the threading

The worst part about turning that on is discovering how many people use lame, generic subjects like "Error", thus polluting the conversation. Or they want to email you, but for some reason don't know how to hit the New Message button so they track down a six-month-old email from you on a completely different subject and hit Reply.

accalia

well there's that too, but the 6 month old email would still be threaded if the client handled the references header properly too.