Help test IMAP access to the IETF archives

[Robert Sparks]

Folks -

We have been testing IMAP access to the IETF archives for several days
now, with volunteers from the tools-discuss and ietf chairs lists, and
it's been going well. Thank you if you've been helping test already (and
please continue to do so)!

Now we need more testers. We'd like to get a simple measure of load, so
we'd like to have as many folks as are willing accessing the test
instance at the same time with various clients during the next week. It
would be particularly helpful to have several people using the system
next Thursday.

If you're willing to take the time to do this, please drop me a note
directly (I've set reply-to on this message) so I have a feel for how
many people are participating.

Instructions for accessing the test instance are below.

Please report any issues you find directly to me. Do not open tickets
with the secretariat. Be aware that there are known issues with the data
on the test instance (some lists have blank or truncated messages) that
have not yet been addressed - it would be good to report any more you
find to me, but they won't get fixed immediately.

Please use tools-***@ietf.org for general conversation. If you have
feature requests, that's a good place to send them. One request we've
had from a few testers so far is to provide an additional shared folder
with a mailbox for each list that has only the "recent" messages from
that list - perhaps the last 6 months. It would be good to hear from
folks, after they've tried what's there now, whether that would be
useful to spend development time on, or if that time would be better
spent elsewhere.

The test instance is getting mail with only a slight delay after it
comes through the lists - you should see current traffic.

Please resist the temptation to download the entire archive of every
list. I've done it - it's huge (~28G when MailMate does it) and it it's
a severe torture test for clients. Interrupt your client if it starts
trying to do that to you. Caching a copy of this test instance will not
be useful when we deploy the production instance.

Instead, what would help us the most is to configure your client to
access those lists you normally subscribe to, and to spend some time
next week browsing and searching those lists and exploring a few lists
that are new to you.

You can find the details for where the test instance is listening, and
rudimentary instructions for setting up a few clients, at
<http://trac.tools.ietf.org/group/tools/trac/wiki/ImapTesting>. Please
improve that page as you see the opportunity.

Thanks in advance for any time you can give this.

RjS

[Robert Sparks]

Testers: Thank you for your help!

Please move to the actual server now.
See <http://trac.tools.ietf.org/group/tools/trac/wiki/Imap>
(Be sure to delete and create a new account in your client - simply
changing the hostname will not work well).

RjS

On 8/27/15 3:37 PM, Robert Sparks wrote:
> Folks -
>
> We have been testing IMAP access to the IETF archives for several days
> now, with volunteers from the tools-discuss and ietf chairs lists, and
> it's been going well. Thank you if you've been helping test already
> (and please continue to do so)!
>
> Now we need more testers. We'd like to get a simple measure of load,
> so we'd like to have as many folks as are willing accessing the test
> instance at the same time with various clients during the next week.
> It would be particularly helpful to have several people using the
> system next Thursday.
>
> If you're willing to take the time to do this, please drop me a note
> directly (I've set reply-to on this message) so I have a feel for how
> many people are participating.
>
> Instructions for accessing the test instance are below.
>
> Please report any issues you find directly to me. Do not open tickets
> with the secretariat. Be aware that there are known issues with the
> data on the test instance (some lists have blank or truncated
> messages) that have not yet been addressed - it would be good to
> report any more you find to me, but they won't get fixed immediately.
>
> Please use tools-***@ietf.org for general conversation. If you
> have feature requests, that's a good place to send them. One request
> we've had from a few testers so far is to provide an additional shared
> folder with a mailbox for each list that has only the "recent"
> messages from that list - perhaps the last 6 months. It would be good
> to hear from folks, after they've tried what's there now, whether that
> would be useful to spend development time on, or if that time would be
> better spent elsewhere.
>
> The test instance is getting mail with only a slight delay after it
> comes through the lists - you should see current traffic.
>
> Please resist the temptation to download the entire archive of every
> list. I've done it - it's huge (~28G when MailMate does it) and it
> it's a severe torture test for clients. Interrupt your client if it
> starts trying to do that to you. Caching a copy of this test instance
> will not be useful when we deploy the production instance.
>
> Instead, what would help us the most is to configure your client to
> access those lists you normally subscribe to, and to spend some time
> next week browsing and searching those lists and exploring a few lists
> that are new to you.
>
> You can find the details for where the test instance is listening, and
> rudimentary instructions for setting up a few clients, at
> <http://trac.tools.ietf.org/group/tools/trac/wiki/ImapTesting>. Please
> improve that page as you see the opportunity.
>
> Thanks in advance for any time you can give this.
>
> RjS
>
>
>
>
>
>

[Robert Sparks]

So, what's showing is the "Inbox", which some clients assume always
exists, but doesn't at this server.

We've made some tweaks during the testing phase to not irritate those
clients - we'll look at some more while in Yokohama. If you'll be there,
please show me the result you're getting. (FWIW, I'm not getting any
error badges on the clients I've set up - it would be interesting to see
how we're configuring differently, and update the wiki).

It is safe to ignore. I understand it's annoying, and we will work to
help get rid of at least the warning badge = but you might have to talk
to your client vendor before the mailbox will go completely away.

RjS

On 10/21/15 1:03 PM, Tim Wicinski wrote:
>
> I've run into the same problem with Thunderbird (Earlybird actually as
> I'm running the latest builds) as Ralph. I assume it's not possible
> to remove regardless if I log in anonymously or with my account.
>
> tim
>
>
> On 10/21/15 5:27 PM, Ralph Droms wrote:
>> Robert - I followed the directions for Mail.app and I'm able to
>> access the archives from the lists I subscribed to.
>>
>> I have one question: I now have an "IETF Archive" subfolder in my
>> Inbox folder, which shows a warning icon because the mailbox name is
>> invalid. Is there any way to remove that subfolder?
>>
>> - Ralph
>>
>>> On Oct 21, 2015, at 10:55 AM 10/21/15, Robert Sparks
>>> <***@nostrum.com> wrote:
>>>
>>> Testers: Thank you for your help!
>>>
>>> Please move to the actual server now.
>>> See <http://trac.tools.ietf.org/group/tools/trac/wiki/Imap>
>>> (Be sure to delete and create a new account in your client - simply
>>> changing the hostname will not work well).
>>>
>>> RjS
>>>
>>> On 8/27/15 3:37 PM, Robert Sparks wrote:
>>>> Folks -
>>>>
>>>> We have been testing IMAP access to the IETF archives for several
>>>> days now, with volunteers from the tools-discuss and ietf chairs
>>>> lists, and it's been going well. Thank you if you've been helping
>>>> test already (and please continue to do so)!
>>>>
>>>> Now we need more testers. We'd like to get a simple measure of
>>>> load, so we'd like to have as many folks as are willing accessing
>>>> the test instance at the same time with various clients during the
>>>> next week. It would be particularly helpful to have several people
>>>> using the system next Thursday.
>>>>
>>>> If you're willing to take the time to do this, please drop me a
>>>> note directly (I've set reply-to on this message) so I have a feel
>>>> for how many people are participating.
>>>>
>>>> Instructions for accessing the test instance are below.
>>>>
>>>> Please report any issues you find directly to me. Do not open
>>>> tickets with the secretariat. Be aware that there are known issues
>>>> with the data on the test instance (some lists have blank or
>>>> truncated messages) that have not yet been addressed - it would be
>>>> good to report any more you find to me, but they won't get fixed
>>>> immediately.
>>>>
>>>> Please use tools-***@ietf.org for general conversation. If you
>>>> have feature requests, that's a good place to send them. One
>>>> request we've had from a few testers so far is to provide an
>>>> additional shared folder with a mailbox for each list that has only
>>>> the "recent" messages from that list - perhaps the last 6 months.
>>>> It would be good to hear from folks, after they've tried what's
>>>> there now, whether that would be useful to spend development time
>>>> on, or if that time would be better spent elsewhere.
>>>>
>>>> The test instance is getting mail with only a slight delay after it
>>>> comes through the lists - you should see current traffic.
>>>>
>>>> Please resist the temptation to download the entire archive of
>>>> every list. I've done it - it's huge (~28G when MailMate does it)
>>>> and it it's a severe torture test for clients. Interrupt your
>>>> client if it starts trying to do that to you. Caching a copy of
>>>> this test instance will not be useful when we deploy the production
>>>> instance.
>>>>
>>>> Instead, what would help us the most is to configure your client to
>>>> access those lists you normally subscribe to, and to spend some
>>>> time next week browsing and searching those lists and exploring a
>>>> few lists that are new to you.
>>>>
>>>> You can find the details for where the test instance is listening,
>>>> and rudimentary instructions for setting up a few clients, at
>>>> <http://trac.tools.ietf.org/group/tools/trac/wiki/ImapTesting>.
>>>> Please improve that page as you see the opportunity.
>>>>
>>>> Thanks in advance for any time you can give this.
>>>>
>>>> RjS
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>

[John Dickinson]

On 21 Oct 2015, at 15:55, Robert Sparks wrote:

> Testers: Thank you for your help!
>
> Please move to the actual server now.
> See <http://trac.tools.ietf.org/group/tools/trac/wiki/Imap>
> (Be sure to delete and create a new account in your client - simply changing the hostname will not work well).
>
> RjS

Thanks for this it works very well with MailMate. Maybe I am asking a stupid question but in order to post to a list do I need to stay subscribed to it in the traditional manner or does the imap subscription do something clever?

regards
John

[Robert Sparks]

On 10/22/15 6:17 AM, John Dickinson wrote:
> On 21 Oct 2015, at 15:55, Robert Sparks wrote:
>
>> Testers: Thank you for your help!
>>
>> Please move to the actual server now.
>> See <http://trac.tools.ietf.org/group/tools/trac/wiki/Imap>
>> (Be sure to delete and create a new account in your client - simply changing the hostname will not work well).
>>
>> RjS
> Thanks for this it works very well with MailMate. Maybe I am asking a stupid question but in order to post to a list do I need to stay subscribed to it in the traditional manner or does the imap subscription do something clever?
If the list has a subscribers-only posting filter, you will need to stay
subscribed to the list - nothing clever, as you suggest, is happening
when you subscribe to the mailbox in IMAP.

>
> regards
> John

[Rich Kulawiec]

1. I'm planning on testing this with fetchmail, which is a command line
POP/IMAP/etc. client. I'd like to minimize the load that I impose on
your side and on the bandwidth that I chew up, so could you point me
to a mailbox that's (relatively) small?

2. Mailman stores list traffic in standard Unix mbox format. I think
it would make sense to export those mboxes (via rsync) on a daily basis
to somewhere where they can be accessed via either https or rsync.
(Why? Because that way someone attempting to retrieve an entire archive
doesn't need to do so one message at a time. And anyone trying to
maintain their own local archive can just update it periodically by
re-rsync'ing.)

---rsk

[Dave Crocker]

On 10/21/2015 2:03 PM, Tim Wicinski wrote:
> I've run into the same problem with Thunderbird (Earlybird actually


It's been a continuing issue for regular' Thunderbird...

d/

--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

[Joseph Lorenzo Hall]

Totally the wrong thread, so I'll apologize profusely now.

GMail is set to turn on DMARC reject strict by sometime early next
year [1] being one of the last big web mail providers to do so; so,
Mailman "From:" rewriting (or turning on the ability for IETF list
subscribers to toggle this for their sends) might need to be done for
IETF lists.

best, Joe

[1]: https://threatpost.com/google-moving-gmail-to-strict-dmarc-implementation/115125/

On Thu, Aug 27, 2015 at 4:37 PM, Robert Sparks <***@nostrum.com> wrote:
> Folks -
>
> We have been testing IMAP access to the IETF archives for several days now,
> with volunteers from the tools-discuss and ietf chairs lists, and it's been
> going well. Thank you if you've been helping test already (and please
> continue to do so)!
>
> Now we need more testers. We'd like to get a simple measure of load, so we'd
> like to have as many folks as are willing accessing the test instance at the
> same time with various clients during the next week. It would be
> particularly helpful to have several people using the system next Thursday.
>
> If you're willing to take the time to do this, please drop me a note
> directly (I've set reply-to on this message) so I have a feel for how many
> people are participating.
>
> Instructions for accessing the test instance are below.
>
> Please report any issues you find directly to me. Do not open tickets with
> the secretariat. Be aware that there are known issues with the data on the
> test instance (some lists have blank or truncated messages) that have not
> yet been addressed - it would be good to report any more you find to me, but
> they won't get fixed immediately.
>
> Please use tools-***@ietf.org for general conversation. If you have
> feature requests, that's a good place to send them. One request we've had
> from a few testers so far is to provide an additional shared folder with a
> mailbox for each list that has only the "recent" messages from that list -
> perhaps the last 6 months. It would be good to hear from folks, after
> they've tried what's there now, whether that would be useful to spend
> development time on, or if that time would be better spent elsewhere.
>
> The test instance is getting mail with only a slight delay after it comes
> through the lists - you should see current traffic.
>
> Please resist the temptation to download the entire archive of every list.
> I've done it - it's huge (~28G when MailMate does it) and it it's a severe
> torture test for clients. Interrupt your client if it starts trying to do
> that to you. Caching a copy of this test instance will not be useful when we
> deploy the production instance.
>
> Instead, what would help us the most is to configure your client to access
> those lists you normally subscribe to, and to spend some time next week
> browsing and searching those lists and exploring a few lists that are new to
> you.
>
> You can find the details for where the test instance is listening, and
> rudimentary instructions for setting up a few clients, at
> <http://trac.tools.ietf.org/group/tools/trac/wiki/ImapTesting>. Please
> improve that page as you see the opportunity.
>
> Thanks in advance for any time you can give this.
>
> RjS
>
>
>
>
>
>



--
Joseph Lorenzo Hall
Chief Technologist
Center for Democracy & Technology
1634 I ST NW STE 1100
Washington DC 20006-4011
(p) 202-407-8825
(f) 202-637-0968
***@cdt.org
PGP: https://josephhall.org/gpg-key
fingerprint: 3CA2 8D7B 9F6D DBD3 4B10 1607 5F86 6987 40A9 A871

[Brian E Carpenter]

Joe,

It's easier to change the subject field than to apologise.

Experts,

Will the problem with Mailman distribution of DMARC-contaminated messages
be fixed in time for this,or must we gmail users start to look for a
more reasonable provider?

Regards
Brian

On 22/10/2015 07:57, Joseph Lorenzo Hall wrote:
> Totally the wrong thread, so I'll apologize profusely now.
>
> GMail is set to turn on DMARC reject strict by sometime early next
> year [1] being one of the last big web mail providers to do so; so,
> Mailman "From:" rewriting (or turning on the ability for IETF list
> subscribers to toggle this for their sends) might need to be done for
> IETF lists.
>
> best, Joe
>
> [1]: https://threatpost.com/google-moving-gmail-to-strict-dmarc-implementation/115125/
>
> On Thu, Aug 27, 2015 at 4:37 PM, Robert Sparks <***@nostrum.com> wrote:
>> Folks -
>>
>> We have been testing IMAP access to the IETF archives for several days now,
>> with volunteers from the tools-discuss and ietf chairs lists, and it's been
>> going well. Thank you if you've been helping test already (and please
>> continue to do so)!
>>
>> Now we need more testers. We'd like to get a simple measure of load, so we'd
>> like to have as many folks as are willing accessing the test instance at the
>> same time with various clients during the next week. It would be
>> particularly helpful to have several people using the system next Thursday.
>>
>> If you're willing to take the time to do this, please drop me a note
>> directly (I've set reply-to on this message) so I have a feel for how many
>> people are participating.
>>
>> Instructions for accessing the test instance are below.
>>
>> Please report any issues you find directly to me. Do not open tickets with
>> the secretariat. Be aware that there are known issues with the data on the
>> test instance (some lists have blank or truncated messages) that have not
>> yet been addressed - it would be good to report any more you find to me, but
>> they won't get fixed immediately.
>>
>> Please use tools-***@ietf.org for general conversation. If you have
>> feature requests, that's a good place to send them. One request we've had
>> from a few testers so far is to provide an additional shared folder with a
>> mailbox for each list that has only the "recent" messages from that list -
>> perhaps the last 6 months. It would be good to hear from folks, after
>> they've tried what's there now, whether that would be useful to spend
>> development time on, or if that time would be better spent elsewhere.
>>
>> The test instance is getting mail with only a slight delay after it comes
>> through the lists - you should see current traffic.
>>
>> Please resist the temptation to download the entire archive of every list.
>> I've done it - it's huge (~28G when MailMate does it) and it it's a severe
>> torture test for clients. Interrupt your client if it starts trying to do
>> that to you. Caching a copy of this test instance will not be useful when we
>> deploy the production instance.
>>
>> Instead, what would help us the most is to configure your client to access
>> those lists you normally subscribe to, and to spend some time next week
>> browsing and searching those lists and exploring a few lists that are new to
>> you.
>>
>> You can find the details for where the test instance is listening, and
>> rudimentary instructions for setting up a few clients, at
>> <http://trac.tools.ietf.org/group/tools/trac/wiki/ImapTesting>. Please
>> improve that page as you see the opportunity.
>>
>> Thanks in advance for any time you can give this.
>>
>> RjS
>>
>>
>>
>>
>>
>>
>
>
>

[Ross Finlayson]

> Will the problem with Mailman distribution of DMARC-contaminated messages
> be fixed in time for this,or must we gmail users start to look for a
> more reasonable provider?

Or perhaps people who subscribe to IETF mailing lists can start doing so using more professional-looking email addresses :-)

(Note that it’s possible, I think, to continue to use 3rd-party email services like ‘Gmail’ with email addresses that use your own domain name rather than “@gmail.com" (and, as a beneficial side effect, you won’t be subject to DMARC).)

[Brian E Carpenter]

On 22/10/2015 14:59, Ross Finlayson wrote:
>> Will the problem with Mailman distribution of DMARC-contaminated messages
>> be fixed in time for this,or must we gmail users start to look for a
>> more reasonable provider?
>
> Or perhaps people who subscribe to IETF mailing lists can start doing so using more professional-looking email addresses :-)

Um, why? We don't represent our employers here, and many people change
jobs or even countries but prefer their IETF email to continue uninterrupted.
I started using gmail precisely to avoid the problems and stupidities of corporate
email, and of access across travel and relocation, and I know I'm not alone.

> (Note that it’s possible, I think, to continue to use 3rd-party email services like ‘Gmail’ with email addresses that use your own domain name rather than “@gmail.com" (and, as a beneficial side effect, you won’t be subject to DMARC).)

Only, I think, if your corporate provider has made a suitable deal with G.

Brian

[Andrew G. Malis]

There is an easy answer to this, that we discussed on this list over a year
ago - just bring mailman up to a more recent revision. It can handle DMARC
re-writing just fine.

Cheers,
Andy


On Wed, Oct 21, 2015 at 10:21 PM, Brian E Carpenter <
***@gmail.com> wrote:

> On 22/10/2015 14:59, Ross Finlayson wrote:
> >> Will the problem with Mailman distribution of DMARC-contaminated
> messages
> >> be fixed in time for this,or must we gmail users start to look for a
> >> more reasonable provider?
> >
> > Or perhaps people who subscribe to IETF mailing lists can start doing so
> using more professional-looking email addresses :-)
>
> Um, why? We don't represent our employers here, and many people change
> jobs or even countries but prefer their IETF email to continue
> uninterrupted.
> I started using gmail precisely to avoid the problems and stupidities of
> corporate
> email, and of access across travel and relocation, and I know I'm not
> alone.
>
> > (Note that it’s possible, I think, to continue to use 3rd-party email
> services like ‘Gmail’ with email addresses that use your own domain name
> rather than “@gmail.com" (and, as a beneficial side effect, you won’t be
> subject to DMARC).)
>
> Only, I think, if your corporate provider has made a suitable deal with G.
>
> Brian
>
>

[Russ Housley]

> There is an easy answer to this, that we discussed on this list over a year ago - just bring mailman up to a more recent revision. It can handle DMARC re-writing just fine.


My understanding is that we are using Mailman 2.1.15. Mailman 2.1.20 and 3.0.0 are available. We have not asked the Secretariat to reviewed every change, but I am told that they are comfortable that Mailman 2.1.20 is fine. We have not looked at Mailman 3.0.0 at all.

Versions of Mailman beyond 2.1.15 have no capacity for sending out password reminders; it was removed. Brian, you have loudly argued for this capability to be retained.

It seems to me that DMARC re-writing is a more important feature for this community. I think we should drop support for the password messages and move to a newer version. I'd like the tools team to check this out, and then if the newer version will not introduce other surprises, move to the newer version.

Russ

[Andrew G. Malis]

Russ,

Thanks, I heartily agree.

Cheers,
Andy

On Thu, Oct 22, 2015 at 9:57 AM, Russ Housley <***@vigilsec.com> wrote:

>
> > There is an easy answer to this, that we discussed on this list over a
> year ago - just bring mailman up to a more recent revision. It can handle
> DMARC re-writing just fine.
>
>
> My understanding is that we are using Mailman 2.1.15. Mailman 2.1.20 and
> 3.0.0 are available. We have not asked the Secretariat to reviewed every
> change, but I am told that they are comfortable that Mailman 2.1.20 is
> fine. We have not looked at Mailman 3.0.0 at all.
>
> Versions of Mailman beyond 2.1.15 have no capacity for sending out
> password reminders; it was removed. Brian, you have loudly argued for this
> capability to be retained.
>
> It seems to me that DMARC re-writing is a more important feature for this
> community. I think we should drop support for the password messages and
> move to a newer version. I'd like the tools team to check this out, and
> then if the newer version will not introduce other surprises, move to the
> newer version.
>
> Russ
>
>

[Paul Wouters]

On Thu, 22 Oct 2015, Russ Housley wrote:

> Versions of Mailman beyond 2.1.15 have no capacity for sending out password reminders; it was removed. Brian, you have loudly argued for this capability to be retained.

I must have missed it between all the email reminders with passwords :)

> It seems to me that DMARC re-writing is a more important feature for this community. I think we should drop support for the password messages and move to a newer version. I'd like the tools team to check this out, and then if the newer version will not introduce other surprises, move to the newer version.

It is Oct 21 2015. While I am not insisting we have flying cars or
hooverboards, I think the IETF not sending plaintext passwords through
unencrypted email is highly overdue. It would be great if on Nov 1st
when my plane lands in Yokohama, I am not treated by a number of IETF
password emails.

Paul

[Phillip Hallam-Baker]

On Thu, Oct 22, 2015 at 10:38 AM, Paul Wouters <***@nohats.ca> wrote:
> On Thu, 22 Oct 2015, Russ Housley wrote:
>
>> Versions of Mailman beyond 2.1.15 have no capacity for sending out
>> password reminders; it was removed. Brian, you have loudly argued for this
>> capability to be retained.
>
>
> I must have missed it between all the email reminders with passwords :)
>
>> It seems to me that DMARC re-writing is a more important feature for this
>> community. I think we should drop support for the password messages and
>> move to a newer version. I'd like the tools team to check this out, and
>> then if the newer version will not introduce other surprises, move to the
>> newer version.
>
>
> It is Oct 21 2015. While I am not insisting we have flying cars or
> hooverboards, I think the IETF not sending plaintext passwords through
> unencrypted email is highly overdue. It would be great if on Nov 1st
> when my plane lands in Yokohama, I am not treated by a number of IETF
> password emails.

+1

The point of eating the dog food is to make better dog food, not get
used to the taste.

Right now we have a broken system of email of which the passwords are
a symptom. In general any time you have a protocol that involves a
password, it is a broken, obsolete system. Unfortunately we seem to
keep coming up with new ways to support passwords rather than an
approach that makes public key based auth as easy to use.


Mailing lists are a broken technology. Currently the IETF is running a
mail server that typically sends several hundred copies of each
message to gmail. Some of those can be batched, but not all of them.
Mine can't be batched because there is no way for mailman to know that
***@hallambaker.com is actually an alias forwarded to a gmail
account. I have something like 5 gig of mail in my gmail account that
is from various mailing lists I subscribe to.


The way the system should work is through a variation of IMAP. Instead
of mailing lists being a layered application that is poorly supported
by and constantly fighting the SMTP infrastructure, mailing list
subscriptions should be handled in the mail client and by the server.
Essentially a mailing list is an IMAP folder that has multiple people
reading it, each of which need to track what has been read separately.

We used to have a system of that type, it was called NNTP.

[Rich Kulawiec]

On Thu, Oct 22, 2015 at 11:02:00AM -0400, Phillip Hallam-Baker wrote:
> Mailing lists are a broken technology. [big snip]

I disagree in part and agree in part. Mailing lists are still massively
useful and *clearly* superior, by a huge margin, to things like web-based
message boards. I'll go so far as to say that no organization or project
can be taken seriously unless it supports at least one mailing list
for announcements and/or communication. On the other hand, mailing lists
have (recently) taken a beating thanks to the ill-advised deployment
of incompatible technology (like DMARC) by some major email providers.

Ironically, the very same email providers which have allegedly done
this to prevent email abuse are systemic and chronic *sources* of
email abuse AND are among the very worst about fielding complaints
and acting on them in a timely, efficient, comprehensive manner.

Thus while I agree in part with you, I would argue that it's not
mailing lists which are broken, per se, it's some very large email
operations that are broken, thanks to incompetence and negligence,
salted liberally with arrogance.

On the other hand, while mailing lists scale well -- to a point -- they
do not scale as well as Usenet news:

> We used to have a system of that type, it was called NNTP.

There are methods (supported by Mailman, incidentally) that allow
bidirectional gatewaying between mailing lists and newsgroups.
The Python language folks have been using that for years, and while
it's not without its occasional issues, it seems to work quite well.
(I've been monitoring both the SMTP- and NNTP-delivered versions
and comparing them.)

Is that kind of gatewaying something that might be appropriate
for the IETF's lists?

---rsk

[Miles Fidelman]

Russ Housley wrote:
>> There is an easy answer to this, that we discussed on this list over a year ago - just bring mailman up to a more recent revision. It can handle DMARC re-writing just fine.
>
> My understanding is that we are using Mailman 2.1.15. Mailman 2.1.20 and 3.0.0 are available. We have not asked the Secretariat to reviewed every change, but I am told that they are comfortable that Mailman 2.1.20 is fine. We have not looked at Mailman 3.0.0 at all.
>
> Versions of Mailman beyond 2.1.15 have no capacity for sending out password reminders; it was removed. Brian, you have loudly argued for this capability to be retained.
>
> It seems to me that DMARC re-writing is a more important feature for this community. I think we should drop support for the password messages and move to a newer version. I'd like the tools team to check this out, and then if the newer version will not introduce other surprises, move to the newer version.
>
> Russ

Or, one might consider a switch to a different list manager entirely.
Sympa is really excellent for handling large numbers of lists, handles
DMARC issues, doesn't send cleartext password reminders, is incredibly
well supported.

Just a thought,

Miles Fidelman

--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra

[Pierre-Elliott Bécue]

On jeu. 22 oct. 2015 à 12:23:42, Miles Fidelman wrote:
> Russ Housley wrote:
> >>There is an easy answer to this, that we discussed on this list over a year ago - just bring mailman up to a more recent revision. It can handle DMARC re-writing just fine.
> >
> >My understanding is that we are using Mailman 2.1.15. Mailman 2.1.20 and 3.0.0 are available. We have not asked the Secretariat to reviewed every change, but I am told that they are comfortable that Mailman 2.1.20 is fine. We have not looked at Mailman 3.0.0 at all.
> >
> >Versions of Mailman beyond 2.1.15 have no capacity for sending out password reminders; it was removed. Brian, you have loudly argued for this capability to be retained.
> >
> >It seems to me that DMARC re-writing is a more important feature for this community. I think we should drop support for the password messages and move to a newer version. I'd like the tools team to check this out, and then if the newer version will not introduce other surprises, move to the newer version.
> >
> >Russ
>
> Or, one might consider a switch to a different list manager entirely. Sympa
> is really excellent for handling large numbers of lists, handles DMARC
> issues, doesn't send cleartext password reminders, is incredibly well
> supported.
>
> Just a thought,

Actually, Mailman 3 got released during this year and seems to be really
good at fixing some great issues from previous versions.

But sure, the migration won't be painless.

--
PEB

[Rich Kulawiec]

On Thu, Oct 22, 2015 at 09:57:58AM -0400, Russ Housley wrote:
> My understanding is that we are using Mailman 2.1.15. Mailman 2.1.20
> and 3.0.0 are available. We have not asked the Secretariat to reviewed
> every change, but I am told that they are comfortable that Mailman 2.1.20
> is fine.

I have been doing extensive testing of 2.1.20 and my impression is that
a migration from 2.1.1X (including 2.1.15) should not present any major
issues. This may not be the best long-term fix (although I remain
convinced that Mailman is best-available mailing list manager software)
but in the short-term, I think it will solve some of the problems
with a low investment in effort.

---rsk

[Brian E Carpenter]

On 23/10/2015 02:57, Russ Housley wrote:
>
>> There is an easy answer to this, that we discussed on this list over a year ago - just bring mailman up to a more recent revision. It can handle DMARC re-writing just fine.
>
>
> My understanding is that we are using Mailman 2.1.15. Mailman 2.1.20 and 3.0.0 are available. We have not asked the Secretariat to reviewed every change, but I am told that they are comfortable that Mailman 2.1.20 is fine. We have not looked at Mailman 3.0.0 at all.
>
> Versions of Mailman beyond 2.1.15 have no capacity for sending out password reminders; it was removed. Brian, you have loudly argued for this capability to be retained.

Yes, but amazingly enough I lost that battle. It's a PITA, but one that can be lived with.

>
> It seems to me that DMARC re-writing is a more important feature for this community. I think we should drop support for the password messages and move to a newer version. I'd like the tools team to check this out, and then if the newer version will not introduce other surprises, move to the newer version.

The primitive rewriting of the From is a bug in itself, because it destroys
important information (who sent the message, even if they are a non-subscriber).
What John Levine describes is hopeful, but it would be nice to have some assurance
from Google that they will actually wait until it's available before changing
their DMARC policy.

Brian

[Christian Huitema]

On Thursday, October 22, 2015 12:41 PM, Brian E Carpenter wrote:
>
> On 23/10/2015 02:57, Russ Housley wrote:
> > ...
> > It seems to me that DMARC re-writing is a more important feature for this
> community. I think we should drop support for the password messages and
> move to a newer version. I'd like the tools team to check this out, and then
> if the newer version will not introduce other surprises, move to the newer
> version.
>
> The primitive rewriting of the From is a bug in itself, because it destroys
> important information (who sent the message, even if they are a non-
> subscriber).

+1.

Rewriting the "From:" header trains users to only look at the user friendly name, and to overlook the rewritten address. The potential for phishing is interesting.

> What John Levine describes is hopeful, but it would be nice to have some
> assurance from Google that they will actually wait until it's available before
> changing their DMARC policy.

Yes. But we may want to look a little bit at privacy issues. The privacy problems with disclosing the current IP address of the user in the Received field are flagged in RFC 7624. We need to make sure that the new ARC field does not amplify this issue.

-- Christi

[Dave Crocker]

On 10/22/2015 8:07 PM, Christian Huitema wrote:
> On Thursday, October 22, 2015 12:41 PM, Brian E Carpenter wrote:
>>
>> On 23/10/2015 02:57, Russ Housley wrote:
>>> ...
>>> It seems to me that DMARC re-writing is a more important feature for this
>> community. I think we should drop support for the password messages and
>> move to a newer version. I'd like the tools team to check this out, and then
>> if the newer version will not introduce other surprises, move to the newer
>> version.
>>
>> The primitive rewriting of the From is a bug in itself, because it destroys
>> important information (who sent the message, even if they are a non-
>> subscriber).
>
> +1.
>
> Rewriting the "From:" header trains users to only look at the user friendly name, and to overlook the rewritten address. The potential for phishing is interesting.

Christian,

I don't like the re-writing either, mostly because it causes email
software to think that one person is (at least) two, when doing sorting
and searching, and therefore causes it to have some new semantic failure
scenarios.

But your premise that users get trained by any of this mostly goes
against research and experience: Users mostly don't notice nuance in
the information in the message header and mostly don't notice anything
reliably and mostly can't be trained.

And no, that's not a slam at users, it's a reality of human factors
design and the body of interactive computer use research.


d/

--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

[Rich Kulawiec]

On Fri, Oct 23, 2015 at 04:33:57AM -0400, Dave Crocker wrote:
> But your premise that users get trained by any of this mostly goes
> against research and experience: Users mostly don't notice nuance in
> the information in the message header and mostly don't notice anything
> reliably and mostly can't be trained.

+1. I don't always agree with Dave about everything, but this is one
case where I'll echo his thoughts, with amplification. "Attempting
to train users and/or use them as part of security/anti-abuse efforts"
is a known-failed approach, with precious few isolated and rare exceptions.

It would be nice if it were otherwise. It's not. It's never going to be.
So when evaluating various bits of technology like this, "can users be
trained?" is not a question that needs to be on table. We already know
the answer. We've known for decades.

---rsk

[Joseph Lorenzo Hall]

On Thu, Oct 22, 2015 at 9:57 AM, Russ Housley <***@vigilsec.com> wrote:
>
> It seems to me that DMARC re-writing is a more important feature for this community. I think we should drop support for the password messages and move to a newer version. I'd like the tools team to check this out, and then if the newer version will not introduce other surprises, move to the newer version.

+1

Rewriting From: at the mailing list level seems to be a simple fix (so
not DMARC checks will fire or they'll fire against ietf.org's DNS
records and presumably go through fine) and writing a Reply-To: from
the original sender should avoid default reply-all nastiness.

This may be in fact what newer versions of Mailman do... best, Joe

[John Levine]

>Totally the wrong thread, so I'll apologize profusely now.

Subject line changed.

>GMail is set to turn on DMARC reject strict by sometime early next
>year [1] being one of the last big web mail providers to do so; so,
>Mailman "From:" rewriting (or turning on the ability for IETF list
>subscribers to toggle this for their sends) might need to be done for
>IETF lists.

Members of the DMARC group have submitted two drafts
draft-andersen-arc-00 and draft-jones-arc-usage-00 which describe a
mutation of DKIM intended to let mailing lists coexist with DMARC
without having to do ugly hacks like rewriting the From: line. I have
reason to believe that several large mail systems intend to implement
this reasonably soon.

The obvious place to discuss this is the DMARC WG, not here.

R's,
John

[JORDI PALET MARTINEZ]

There is a much bigger problem with the IETF mail servers, I’ve reported it several times to AMS and other folks from the IAOC have been informed as I understand, but nobody is providing any solution, and as a consequence, we are subjected to receive SPAM, as we are accepting emails from subscribers which are in spam black-lists.

This problem also means that if you own mail server (which many of us don’t control), is configured to reject messages from black-listed IPs, then when you reject emails from such folks, the IETF considers that your email address is bouncing, and unsubscribes you. I’ve this problem every other week, in the general area list and also in some others related to DNS (which probably means that the black-listed servers are from people contributing mainly to DNS exploders).

This is my view about the problem, which has been ignored up to now:

1) IETF is accepting messages from subscribers with MX which are Blacklisted.

2) IETF send those emails also with the original headers.
—> that’s why when AMS staff send the logs that I requested, it appears
like the IETF is black-listed, which is not the case, but a couple of
senders to IETF exploders.


3) Mail servers inspect the IETF and original headers looking for
black-listed servers.

4) If they are strict, bounce emails from IETF even if the IETF records
aren’t black-listed but the original sender MX is black-listed

So, we need a policy here. Either:

A) IETF doesn’t accept messages from black-listed servers.
——> if we don’t do so, we may be affecting ALL the subscribers, and even
worst, a SPAMER could subscribe to IETF and send SPAM to ALL our
subscribers. An even worst, once IETF mail es bounced because the
black-list, some antispam systems increase the IETF rating as “possible
spammer”, which is not good.

Or

B) IETF clean-up the original sender header (before sending to the mail
exploder) to avoid bounces.

Alternatively, not sure how this will work, we could just allow “bounces”
from subscribers (may be even a big number such as 40-50), instead of
disabling the membership.

We may need to retry more times sending emails to subscribers, increasing the time between retries, and after no bouncing for “x” many days, resetting counters for this subscriber.


When I got the logs from AMS (august 2015), the offenders where:

149.171.193.32 - Australia Sydney University Of New South Wales, black-listed by sorbs


And

198.57.247.250 - United States New York City Unified Layer, black-listed by spamhaus




Regards,
Jordi








-----Mensaje original-----
De: ietf <ietf-***@ietf.org> en nombre de John Levine <***@taugh.com>
Responder a: <***@taugh.com>
Fecha: jueves, 22 de octubre de 2015, 17:42
Para: <***@ietf.org>
Asunto: DMARC stuff

>>Totally the wrong thread, so I'll apologize profusely now.
>
>Subject line changed.
>
>>GMail is set to turn on DMARC reject strict by sometime early next
>>year [1] being one of the last big web mail providers to do so; so,
>>Mailman "From:" rewriting (or turning on the ability for IETF list
>>subscribers to toggle this for their sends) might need to be done for
>>IETF lists.
>
>Members of the DMARC group have submitted two drafts
>draft-andersen-arc-00 and draft-jones-arc-usage-00 which describe a
>mutation of DKIM intended to let mailing lists coexist with DMARC
>without having to do ugly hacks like rewriting the From: line. I have
>reason to believe that several large mail systems intend to implement
>this reasonably soon.
>
>The obvious place to discuss this is the DMARC WG, not here.
>
>R's,
>John
>
>

[Dave Crocker]

On 10/22/2015 12:41 PM, JORDI PALET MARTINEZ wrote:
> 1) IETF is accepting messages from subscribers with MX which are Blacklisted.


There are many blacklists. Some are well run. Some less so.

I'll suggest that any effort that goes down the path of providing
abuse-related directions to IETF ops staff needs to develop a detailed
policy specification -- including citing the abuse information lists to
be consulted -- that there can be review and consensus on exactly what
is to be done.

d/

--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

[David Morris]

On Thu, 22 Oct 2015, Dave Crocker wrote:

> On 10/22/2015 12:41 PM, JORDI PALET MARTINEZ wrote:
> > 1) IETF is accepting messages from subscribers with MX which are Blacklisted.
>
>
> There are many blacklists. Some are well run. Some less so.
>
> I'll suggest that any effort that goes down the path of providing
> abuse-related directions to IETF ops staff needs to develop a detailed
> policy specification -- including citing the abuse information lists to
> be consulted -- that there can be review and consensus on exactly what
> is to be done.


+1 on Dave's suggestion. I've had to fight through issues caused by
poorly run blacklists. I'd be very concerned about any proposal to
make binary reject decisions based on any blacklist not controlled
by the IETF.

Dave Morris

[JORDI PALET MARTINEZ]

Hi Dave,

I understand that, but because we can’t “set” the policy of the companies that employee our subscribers, we need to be able to work with as many email providers as possible, which means being VERY strict.

There are two ways to achieve that, I think:

1) Using a comprehensive set of DNS-BL
http://www.dnsbl.info/dnsbl-database-check.php

2) Not using any DNS-BL but then we need to clear the original headers before re-sending the email to the IETF exploders.

Regards,
Jordi









-----Mensaje original-----
De: Dave Crocker <***@dcrocker.net>
Organización: Brandenburg InternetWorking
Responder a: <***@bbiw.net>
Fecha: jueves, 22 de octubre de 2015, 21:19
Para: Jordi Palet Martinez <***@consulintel.es>
CC: <***@ietf.org>
Asunto: Re: DMARC stuff

>On 10/22/2015 12:41 PM, JORDI PALET MARTINEZ wrote:
>> 1) IETF is accepting messages from subscribers with MX which are Blacklisted.
>
>
>There are many blacklists. Some are well run. Some less so.
>
>I'll suggest that any effort that goes down the path of providing
>abuse-related directions to IETF ops staff needs to develop a detailed
>policy specification -- including citing the abuse information lists to
>be consulted -- that there can be review and consensus on exactly what
>is to be done.
>
>d/
>
>--
>Dave Crocker
>Brandenburg InternetWorking
>bbiw.net
>

[Rich Kulawiec]

First, the correct slang term for unsolicited bulk email is "spam",
never "SPAM". The latter is a product of the Hormel Corporation
and has nothing to do with email; the former is not an acronym and
thus should never be rendered in all-caps.

On Thu, Oct 22, 2015 at 06:41:54PM +0200, JORDI PALET MARTINEZ wrote:
> 1) IETF is accepting messages from subscribers with MX which are Blacklisted.

Second, blacklisted by *whom*? There are hundreds of public blacklists
and no doubt several orders of magnitude more private ones. Some of
these list enormous swaths of network space. I wouldn't be surprised
if *most* of the messages accepted by IETF mail servers originate from
hosts that are in some blacklist, somewhere.

(Also, I don't think you mean their MX's, necessarily. You mean
their outbound mail servers, which may or may not be the same as
the hosts listed in their MX records.)

> 3) Mail servers inspect the IETF and original headers looking for
> black-listed servers.
>
> 4) If they are strict, bounce emails from IETF even if the IETF records
> aren???t black-listed but the original sender MX is black-listed

Third, there's your problem. Don't do that.

If you subscribe to a mailing list, IETF or otherwise, you should accept
all traffic from that mailing list. It's up to the keepers of the mailing
list to keep it from of spam, which these days isn't all that hard modulo
the occasional compromised list-member account. And it's up to you to
accept all messages from the mailing list *because you subscribed to it*.

If you don't want to cede authority over list traffic decision-making
to the keepers-of-the-list, then don't subscribe to it.

---rsk

[John R Levine]

> > Members of the DMARC group have submitted two drafts
> > draft-andersen-arc-00 and draft-jones-arc-usage-00 which describe a
> > mutation of DKIM intended to let mailing lists coexist with DMARC
> > without having to do ugly hacks like rewriting the From: line. I have
> > reason to believe that several large mail systems intend to implement
> > this reasonably soon.
>
> Can you at least summarize where the drafts are in the process, and if they
> require changes to senders, mailing lists, receivers, or all of the above?

ARC adds some new trace headers that provide a signed chain of custody.
If you're already doing DKIM signing or verification, it should be
straightforward to use upgraded DKIM libraries to apply them. Receivers
look at the ARC headers to decide when to accept a message despite DMARC
failure. As I said, big mail systems plan to do this. There aren't any
mailing list changes other than adding ARC headers which would be
invisible to users.

> Are the proposals (which I guess are not yet adopted) complementary, or will
> a beauty contest be required?

The work is outside the IETF, no beauty contest.

> I think that ***@ietf.org needs to know if we'll have a solution in time,
> or if the secretariat needs to be asked to deploy mailman 3 sooner.

I'd be pretty surprised if Gmail started publishing a DMARC policy before
lists were able to deploy ARC.

Regards,
John Levine, ***@taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

[Dave Crocker]

On 10/22/2015 12:11 PM, Michael Richardson wrote:
> Can you at least summarize where the drafts are in the process, and if they
> require changes to senders, mailing lists, receivers, or all of the above?


the effort is nascent. the specifications have only just been released.

while promising, there is still quite a bit more hope than hard
knowledge about its efficacy.

also, while john's 'soon' is not unreasonable, we need to be careful
about making near-term operational decisions based on brand new
specifications that have no implementations, nevermind no operational
experience.

d/

--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

[John R Levine]

>> Can you at least summarize where the drafts are in the process, and if they
>> require changes to senders, mailing lists, receivers, or all of the above?
>
> the effort is nascent. the specifications have only just been released.

That's certainly true, but it's pretty clear that the DMARC crowd wants to
make it work so we mailing list users stop hating on them.

Regards,
John Levine, ***@taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

[Dave Crocker]

On 10/22/2015 8:27 PM, John R Levine wrote:
>>
>> the effort is nascent. the specifications have only just been released.
>
> That's certainly true, but it's pretty clear that the DMARC crowd wants
> to make it work so we mailing list users stop hating on them.


That is nice, but does not affect my recommendation, which is predicated
on a simple rule:

When seeking to solve immediate issues, use existing, tested tools.
/Do not rely on anything that is in the pipeline because you have
cannot have any reliable knowledge of when it will be operationally viable./


d/

--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

[John Levine]

In article <74C1DAFA-87CC-469A-8AA9-***@vigilsec.com> you write:
>
>> There is an easy answer to this, ...

It's not an answer, it's an ugly hack, and turns out not to be needed,

Please see the two ARC drafts that I mentioned in a recent message.

R's,
John

[Martin Rex]

Christian Huitema wrote:
> Brian E Carpenter wrote:
>>
>> On 23/10/2015 02:57, Russ Housley wrote:
>>> ...
>>> It seems to me that DMARC re-writing is a more important feature for this
>> community. I think we should drop support for the password messages and
>> move to a newer version. I'd like the tools team to check this out, and then
>> if the newer version will not introduce other surprises, move to the newer
>> version.
>>
>> The primitive rewriting of the From is a bug in itself, because it destroys
>> important information (who sent the message, even if they are a non-
>> subscriber).
>
> +1.
>
> Rewriting the "From:" header trains users to only look at the user
> friendly name, and to overlook the rewritten address.
> The potential for phishing is interesting.

I do not see any increased potential for phishing
Rather the opposite -- DMARC could be abused to give users a false
sense of security and fall to the flawed assumption that it would
authenticate the EMail author (which it doesn't).


Interestingly, there essentially is a legal requirement for rewriting
the From: header field for addresses that publish DMARC records when
delivering EMail to places where DMARC processing isn't outright illegal
(DMARC processing is clearly illegal in the EU), because handing an
EMail to an MTA that might be processing DMARC will be illegal in the EU
just as processing DMARC oneself. So one of the alternatives to
unconditionally bouncing a DMARC-impaired EMail, will be rewriting the From:.


-Martin

[Rich Kulawiec]

On Fri, Oct 23, 2015 at 08:36:31PM +0200, Martin Rex wrote:
> I do not see any increased potential for phishing
> Rather the opposite -- DMARC could be abused to give users a false
> sense of security and fall to the flawed assumption that it would
> authenticate the EMail author (which it doesn't).

Precisely. My spamtraps observe messages all day, every day that pass
whatever validation happens to be in play -- but are clearly forgeries.
And it's a VERY rare end user who is capable of making that same
determination. Thus the warm fuzzies provided by mail clients that mark
messages as "validated" or "authenticated" or whatever term is used are
going to make these problems worse, not better.

Until the underlying security issues are fixed -- and I see absolutely
no signs that any of the 500-pound gorillas even *intend* to address
those at scale, let alone are actively engaged in doing so -- this (DMARC
and related) just wallpapers over the problem.

---rsk

[Brian E Carpenter]

On 25/10/2015 02:33, Rich Kulawiec wrote:
> On Fri, Oct 23, 2015 at 08:36:31PM +0200, Martin Rex wrote:
>> I do not see any increased potential for phishing
>> Rather the opposite -- DMARC could be abused to give users a false
>> sense of security and fall to the flawed assumption that it would
>> authenticate the EMail author (which it doesn't).

Just for fun, I looked at a small sample of spam: the most recent 24
messages that gmail itself tagged as junk.

No false positives.
4 tagged as DMARC pass.
5 tagged as DMARC fail (gmail does not currently obey p=discard)
15 with no DMARC status.

Which suggests that DMARC status is pretty much orthogonal to spam detection,
on this small sample.

Brian

>
> Precisely. My spamtraps observe messages all day, every day that pass
> whatever validation happens to be in play -- but are clearly forgeries.
> And it's a VERY rare end user who is capable of making that same
> determination. Thus the warm fuzzies provided by mail clients that mark
> messages as "validated" or "authenticated" or whatever term is used are
> going to make these problems worse, not better.
>
> Until the underlying security issues are fixed -- and I see absolutely
> no signs that any of the 500-pound gorillas even *intend* to address
> those at scale, let alone are actively engaged in doing so -- this (DMARC
> and related) just wallpapers over the problem.
>
> ---rsk
>
> .
>

[Randy Bush]

< rant >

it really makes little difference. face it, email standards are now
made by google, yahoo, aol, ... and shoved down our throats. have you
seen the hoops one has to jump through to smtp over v6 to google? they
outsource their alleged pain to the rest of the internet.

my paranoid brother wonders if this is intentionally pushing small smtp
sites to give up and use the mail cartel. but i think that is just
collateral damage.

randy

[Ted Lemon]

On Oct 24, 2015, at 7:40 PM, Randy Bush <***@psg.com> wrote:
> have you
> seen the hoops one has to jump through to smtp over v6 to google? they
> outsource their alleged pain to the rest of the internet.

No, actually I haven’t. It Just Works for me. I was surprised, but pleased.

[Harald Alvestrand]

Den 25. okt. 2015 02:58, skrev Ted Lemon:
> On Oct 24, 2015, at 7:40 PM, Randy Bush <***@psg.com
> <mailto:***@psg.com>> wrote:
>> have you
>> seen the hoops one has to jump through to smtp over v6 to google? they
>> outsource their alleged pain to the rest of the internet.
>
> No, actually I haven’t. It Just Works for me. I was surprised, but
> pleased.
>

The requirement in v6 that didn't exist in v4 is that you have to have a
PTR record for your MTA's IPv6 address before Google wants to let you
send SMTP over IPv6.

In IPv4, the absence of a PTR record was taken as a spam signal,
together with all the other spam signals; in IPv6, they decided to just
say "no".

[Randy Bush]

> The requirement in v6 that didn't exist in v4 is that you have to have a
> PTR record for your MTA's IPv6 address before Google wants to let you
> send SMTP over IPv6.

and spf

randy

[Harald Alvestrand]

Den 25. okt. 2015 04:02, skrev Randy Bush:
>> The requirement in v6 that didn't exist in v4 is that you have to have a
>> PTR record for your MTA's IPv6 address before Google wants to let you
>> send SMTP over IPv6.
>
> and spf

I don't have it. They let me (mork.alvestrand.no) in with only the PTR.

>
> randy
>

[Ted Lemon]

On Oct 24, 2015, at 11:02 PM, Randy Bush <***@psg.com> wrote:
>> The requirement in v6 that didn't exist in v4 is that you have to have a
>> PTR record for your MTA's IPv6 address before Google wants to let you
>> send SMTP over IPv6.
>
> and spf
>
> randy

If you don’t have spf your mail is going to get scored way down my my mailer too. SPF is an IETF standard, so you really can’t complain that Google are scary renegades for using it.

[Paul Wouters]

>
> In IPv4, the absence of a PTR record was taken as a spam signal,
> together with all the other spam signals; in IPv6, they decided to just
> say "no".

Which is stupid because many ISPs didn't run proper reverse for v6 and couldn't delegate to their customers.

Which broke many small email servers

Upon which the ISPs started auto generating PTR records.

Upon which the anti spammers rejected auto generated PTR as valid and rebroke the small email servers.

Upon which those mail systems just stopped using IPv6.

Upon which we all lost

[Phillip Hallam-Baker]

On Sun, Oct 25, 2015 at 7:33 AM, Paul Wouters <***@nohats.ca> wrote:
>
>
>>
>> In IPv4, the absence of a PTR record was taken as a spam signal,
>> together with all the other spam signals; in IPv6, they decided to just
>> say "no".
>
> Which is stupid because many ISPs didn't run proper reverse for v6 and couldn't delegate to their customers.
>
> Which broke many small email servers
>
> Upon which the ISPs started auto generating PTR records.
>
> Upon which the anti spammers rejected auto generated PTR as valid and rebroke the small email servers.
>
> Upon which those mail systems just stopped using IPv6.
>
> Upon which we all lost

What you are demonstrating is the need for a coordination body that:

1) Listens to all the stakeholders
2) Takes account of all the stakeholder concerns
3) Arrives at a consensus based on those concerns
4) Publishes them

The problem with the approach here that a lot of people take here is
that they are not interested in step 1 or 2. They treat the IETF
process as a means to get the outcome THEY desire and tough luck to
anyone else who isn't there.

If you want to arrive at an approach that sticks, you have to look at
the interests of all the stakeholders, even the ones that don't bother
to show up.

[Ted Lemon]

If we decide that the long-established semantics are the right ones, then I think our email standards deserve to die, because they don't currently work. I share your concern about email turning into closed bulletin boards, but the way to fix that is to accept that email as it is is badly broken, and try to fix it, not to get mad at Google et al. for refusing to continue to suffer the brokenness.

[Ted Lemon]

On Oct 26, 2015, at 2:45 PM, John C Klensin <john-***@jck.com> wrote:
>> If we decide that the long-established semantics are the right
>> ones, then I think our email standards deserve to die, because
>> they don't currently work.
>
> Ted, I think millions of users, passing around tens or hundreds
> of millions of messages around a day, would probably disagree
> with "don't currently work" or at least dismiss it as rather
> extreme hyperbole.

I will admit to using an extreme form of the term "work," which is "work without massive difficulties." Most of the massive difficulties happen behind the scenes, so we fortunates do not have to deal with them, but the lengths mail providers like Google have to go to to keep spam out of our inboxes cast their shadow over the experience of every email user. Web registration systems now tell users to whitelist mail from their domains, which would be cool if it could be done automatically rather than through manual intervention, and users are now accustomed to searching for important mail that hasn’t arrived in their junk folders. I suspect the annual number of person-hours spent on these two tasks would humble us.

My experience of "badly broken": pre-filter, for my email address (***@fugue.com <mailto:***@fugue.com>) alone, spam is currently arriving at a rate of about three messages per minute. Why? Because the design assumptions for email did not account for the fact that in the wild, email is an ecosystem, not a cooperative venture, and there is money to be made with scattershot spam, and money to be made filtering it.

I’ve attempted to implement a whitelist on my server to make sure that mail from people I already know will get through. What I’ve discovered is that aside from the big providers, nobody sets their SPF up right, so I can’t rely on it to validate whitelisted senders: I either have to just hope for the best, and accept the occasional joe job attack, or else I whitelist IP addresses, or come up with heuristics like "if the MX for a domain points to google, pretend that the domain owner set up SPF according to Google’s docs."

The amount of brainpower that’s required to keep this rickety train on the rails is astonishing. It is no longer the case that someone like you or I with the resources of an individual can have a reasonably painless experience of operating an SMTP server. To my mind, this means that SMTP does not "work." There is no dependable method by which I can ask the question "did this email message come from the source that it claims to have come from" and get an answer.

That’s what I mean by "works." And that’s why I have every sympathy with mail providers who are throwing up their hands in disgust and saying I’m going to be draconian, even though the specs don’t technically allow it. If we want something different to happen, we have to figure out a way to allow it to happen, and not just say there’s no problem and Google ought to follow the specs.

> Perhaps I haven't been looking in the right places, but I
> haven't heard Google claim that email is "badly broken", much
> less "doesn't work". What I have heard is some claims about
> blocking of some messages originating from bogus or unauthorized
> senders. That is a sender authentication problem, not a "broken
> email protocol" one.

It’s true that if we had a reliable way of validating senders, SMTP could continue to operate. And it might even be that if we had this mechanism in place, spammers would stop dumping crap on my mail server, so I wouldn’t have to pay for gigabytes per day of useless traffic to my server. But in practice, it would be much better to use a protocol that didn’t even trigger a data transfer until the sender had been validated.

> Equally important, if Google really
> cares about either sender authentication or verification that a
> sender who uses a particular backward-pointing address today is
> the same entity who used it yesterday, we know of a large
> variety of ways to approximate at least the latter. The
> observation that Google isn't doing any of those things, even
> the ones they could support with a very large fraction of their
> users and without protocol changes, suggests that isn't the
> issue.

Yes, we do know ways to do this. Most of them are too expensive to be practical.

> So, if you are going to claim that our existing standards don't
> work, I think it would be good to have a clear explanation of
> what you mean and what, precisely, doesn't work. Of course, I
> can only hope that, contrary to your apparent claim, this
> message will reach you in spite of non-working protocols and you
> will be able to reply.

I have all the IETF mailing lists whitelisted, and IETF has a correct SPF setup (and, actually, I get my IETF mail through nominum’s servers, and Nominum pays Google a lot of money to filter spam, and they use their big data analytics to do a good job of that, so in that sense it does indeed "work," but I shudder to think what the carbon footprint of each valid delivered message is if you amortize the cost over the total number of messages that had to be transmitted and examined.

[Dave Crocker]

> Web registration systems now tell users to whitelist mail from their
> domains, which would be cool if it could be done automatically rather
> than through manual intervention, and users are now accustomed to
> searching for important mail that hasn’t arrived in their junk folders.


Consider that example as a touchstone to the larger set of issues, here:

Anti-abuse mechanisms (and many other 'fixes') for email tend to be
added piecemeal and locally, rather than considered as end-to-end
systems issues.

On the average, the focus of concern is local operations, rather than
user interaction and user experience. It's simpler and quicker to work
that way, but it winds up turning email into a technical and functional
Tower of Babel. This approach to dealing with email problems has been an
issue for at least 40 years. Recipients complained to local operators
who put some local hack in place to deal with the symptoms. Often the
originator of the problem never hears about it.

Email is by far the largest and most complex distributed system on
the Internet. Having complex objects travel multiple hops along many
independent administrations is a pain. So let's minimize hops and
administrations and eliminate the cases that need to greater flexibility
email gave with the more highly distributed administrative and
functional design.

We need to properly distinguish identities from identifiers and we
need to properly distinguish among the many independent actors in the
handling sequence and we need to distinguish trust from mistrust. Each
of these distinctions can produce very different approaches for
proposals. For example, trying to look for bad actors is fundamentally
different from looking for good actors. So is evaluating an operator
versus an author.

What most folk have moved to is a model in which a user's identity,
identifier and operational support are all tightly integrated and
controlled by a single administration. For simple scenarios, that works
well enough to cause people to treat more varied scenarios as 'problems'
rather than 'userful requirements'.

The current evolution of systems changes for anti-abuse are largely
reactive -- the abusers set the agenda -- and largely turn email into a
Procrustean bed: Only usage scenarios that are supported and protected
cleanly are the easiest and most restrictive on users. Since that
satisfies the vast majority of current use, the disenfranchisement of
the much smaller minority of other, legitimate scenarios can be ignored,
independent of the problems caused to those who are disenfranchised.

Producing general, end-to-end changes that afford necessary protections
and support the widest possible range of functionality is much more work
than doing localized, incremental hacks. (Not that the localized,
incremental hacks are cheap or quick either.)

We need to stop thinking we can eliminate abuse. We need to start
thinking in terms of end-to-end trust models that scale and support
varied usage scenarios. We need to focus on being able to use trust of
operators, more than on trust of authors.

And mostly, we need to stop creating one hack to fill in the gaps for
another hack. Or rather -- since the hacks often are the result of an
emergency -- we need to stop thinking the hacks actually 'solve'
problems rather than merely creating new ones.

d/

--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

[Ted Lemon]

John, Dave, this is exactly the conversation I want to have. Is this conversation happening somewhere in the IETF? I confess that I haven’t put a ton of effort into figuring that out—I’ve been studying the current documents and have done some test implementations to eliminate some of my ignorance on this topic, but I haven’t quite gotten to the point where I know more of what to say than random ideas like "how do we close the loop between the browser and the MUA so that wanted notifications can be securely marked for reliable delivery" and that sort of thing.

> On Oct 27, 2015, at 1:47 PM, John C Klensin <john-***@jck.com> wrote:
>
> For whatever it is worth (and the rarity in email protocol and
> architecture discussions is, IMO, worth noting), I completely
> agree with Dave's comments and analysis below.
>
> Let me add one observation about scope: to a considerable
> extent, "solving" abuse by mechanisms close to the target
> recipient also misses a lot what should be the point because the
> damage -- measured in added hacks or additional processing and
> bandwidth requirements-- is done almost as soon as the problem
> message is sent. By ignoring it and applying patches and
> workarounds at the recipient end (whether those are to
> accommodate badly-formatted messages or to mitigate deliberate
> abuse) reduces or eliminates pressure on originating systems to
> get things right, prevent the abusive behavior, or create legal
> deterrents to that behavior that are enforced sufficiently to
> act as a deterrent. As Dave says, it has been that way for
> 40-plus years. However, if we want fixes that keep the system
> working, rather than more and more hacks that drive us toward a
> small and homogeneous set of "acceptable" mail systems, we'd
> better start looking at the whole system, figuring out what we
> want, and making it work.
>
> john
>
>
> --On Tuesday, October 27, 2015 07:56 -0700 Dave Crocker
> <***@dcrocker.net> wrote:
>
>>
>>> Web registration systems now tell users to whitelist mail
>>> from their domains, which would be cool if it could be done
>>> automatically rather than through manual intervention, and
>>> users are now accustomed to searching for important mail that
>>> hasn't arrived in their junk folders.
>>
>>
>> Consider that example as a touchstone to the larger set of
>> issues, here:
>>
>> Anti-abuse mechanisms (and many other 'fixes') for email
>> tend to be added piecemeal and locally, rather than considered
>> as end-to-end systems issues.
>>
>> On the average, the focus of concern is local operations,
>> rather than user interaction and user experience. It's
>> simpler and quicker to work that way, but it winds up turning
>> email into a technical and functional Tower of Babel. This
>> approach to dealing with email problems has been an issue for
>> at least 40 years. Recipients complained to local operators
>> who put some local hack in place to deal with the symptoms.
>> Often the originator of the problem never hears about it.
>>
>> Email is by far the largest and most complex distributed
>> system on the Internet. Having complex objects travel
>> multiple hops along many independent administrations is a
>> pain. So let's minimize hops and administrations and
>> eliminate the cases that need to greater flexibility email
>> gave with the more highly distributed administrative and
>> functional design.
>>
>> We need to properly distinguish identities from identifiers
>> and we need to properly distinguish among the many independent
>> actors in the handling sequence and we need to distinguish
>> trust from mistrust. Each of these distinctions can produce
>> very different approaches for proposals. For example, trying
>> to look for bad actors is fundamentally different from looking
>> for good actors. So is evaluating an operator versus an
>> author.
>>
>> What most folk have moved to is a model in which a user's
>> identity, identifier and operational support are all tightly
>> integrated and controlled by a single administration. For
>> simple scenarios, that works well enough to cause people to
>> treat more varied scenarios as 'problems' rather than 'userful
>> requirements'.
>>
>> The current evolution of systems changes for anti-abuse are
>> largely reactive -- the abusers set the agenda -- and largely
>> turn email into a Procrustean bed: Only usage scenarios that
>> are supported and protected cleanly are the easiest and most
>> restrictive on users. Since that satisfies the vast majority
>> of current use, the disenfranchisement of the much smaller
>> minority of other, legitimate scenarios can be ignored,
>> independent of the problems caused to those who are
>> disenfranchised.
>>
>> Producing general, end-to-end changes that afford necessary
>> protections and support the widest possible range of
>> functionality is much more work than doing localized,
>> incremental hacks. (Not that the localized, incremental hacks
>> are cheap or quick either.)
>>
>> We need to stop thinking we can eliminate abuse. We need to
>> start thinking in terms of end-to-end trust models that scale
>> and support varied usage scenarios. We need to focus on being
>> able to use trust of operators, more than on trust of authors.
>>
>> And mostly, we need to stop creating one hack to fill in the
>> gaps for another hack. Or rather -- since the hacks often are
>> the result of an emergency -- we need to stop thinking the
>> hacks actually 'solve' problems rather than merely creating
>> new ones.
>>
>> d/
>
>
>

[Dave Crocker]

On 10/26/2015 1:01 PM, Ted Lemon wrote:
> My experience of "badly broken": pre-filter, for my email address
> (***@fugue.com <mailto:***@fugue.com>) alone, spam is currently
> arriving at a rate of about three messages per minute. Why? Because
> the design assumptions for email did not account for the fact that in
> the wild, email is an ecosystem, not a cooperative venture, and there is
> money to be made with scattershot spam, and money to be made filtering it.

For 25-30 years, the email system we use now wasn't "in the wild". The
service was developed, operated and used in a more constrained
environment. Failure to anticipate massive and fundamental changes in a
computing and communication environment, decades hence, is not exactly a
design failure. A problem, of course. But that's different from a failure.

On the other hand, assessing it as a failure fits into an increasingly
common template of changing the email operating environment's rules and
then suddenly reclassifying usage which worked fine in the past as now
being a 'violation'.


> The amount of brainpower that’s required to keep this rickety train on
> the rails is astonishing. It is no longer the case that someone like
> you or I with the resources of an individual can have a reasonably
> painless experience of operating an SMTP server. To my mind, this
> means that SMTP does not "work."

If I'm reading your text correctly, you are saying that the fact that a
service that was once simple to operate but that now requires extensive
staffing and expertise subjects the service to an assessment of 'does
not work'?

Yet I'm pretty sure that that kind of transition from simplicity to
complexity that requires staffing and expertise is the hallmark of many
(all?) infrastructure services in all technologically developed cultures.

Few consumers operate their own telephone exchange or their own air
traffic control center or their own water purification center or...

d/
--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

[Niels Dettenbach]

Am Mittwoch, 28. Oktober 2015, 17:11:22 schrieb Viktor Dukhovni:
> So, from where I site, the real problem is that mailboxes are cost
> free, and the large providers have it uneconomical to run a mail
> service that is accountable for abuse by its users, and yet are
> "too large to block".

Setting up an an email service is typically easy, but running an reliable
email service is a much harder job. This is where many MTA admins or companies
on the road of implementing their own MTA / internet email service fail over
the time. Around 2-5 years ago there was an hype of having your own MTA /
internet mailer system even as the smallest company - leading to many "crap"
within the net, because of many different standard violations, bugs and
security holes in the setups, made it very difficult to fight against abusive
smtp usage.

Anyhow: These times seems to be a bit over now... ß)

There are professional operating MTA / Relay and/or mail providers which offer
email services on any level and for any expectations of user audiences -
including very effective anti spam tech and active anti abuse management.

Ii someeone is happy with some kind of "cost free" service at all, who usually
has to save bucks on some points or generate revenue over other vectors (i.e.
selling user profiles, sell ads, sell reputation of their users traffic
etc.pp.), this is OK, but this is his decision.

I know that there are companies relying on email and gelegated this to any
lowest entry service - but users who really do expect a really (!) cost free
high reliable / high volume, high quality email service are just out of
reality. This is like on any service on the market.

(just my two cents)


best regards,


Niels.



--
---
Niels Dettenbach
Syndicat IT & Internet
http://www.syndicat.com
PGP: https://syndicat.com/pub_key.asc
---

[Ted Lemon]

Neils, the question is, why is it so hard to set up your own mail server and have it basically do what you need? Is it because it's really that hard, or is it because the right tools don't exist, or the problem hasn't been properly reduced to practice? I think it's more the latter than the former. It sounds like Postfix has improved since I last did a Postfix configuration, but it's still got too many knobs, because it addresses too many disjoint use cases. And it looks like mailpile is starting to wrestle with the identity problem in a useful way, and there is other interesting work going on in the usability/identity/security areas, but I'm not hearing people here talk about how that's being worked on in the IETF. I would hate to see a bunch of ad-hoc protocols with no spec and no interop.

[Hector Santos]

On 10/29/2015 2:08 PM, ned+***@mauve.mrochek.com wrote:
>> Neils, the question is, why is it so hard to set up your own mail server and
>> have it basically do what you need?
>
> Ted, you keep saying things like this as if they are true. Simply put, they're
> not. The obvious counterexample is MS Exchange, which, my personal dislike for
> it notwithstanding, is hardly rocket science to install and configure. Lots of
> people with very modest technical skills do it routinely. (The place where they
> get into trouble is when they outgrow what they originally did, have to upgrade
> the hardware, etc. This is where things can get to be a bit of a PITA with MS
> Exchange.)
>
> In fact it's the existence of MS Exchange that keeps a lot of vendors from
> bothering with this space. (It's absolutely why we started moving away from it
> ~15 years ago.) Even so, there are other offerings, such as the server IETF
> participant Hector Santos works on. Perhaps he can comment on how
> straightforward his product is to set up.
>
> ...
>
> Since I categorically reject your thesis that initial setup of a simple mail
> server is the Hurculean task you claim it is, I have no idea what work the IETF
> could undertake to solve this nonexistent problem.
>

Not sure if this helps... and I have not read the entire thread, I
agree, SMTP is relatively simple. I think a good job was done in this
regard for the IETF "SMTP" default design and operation.

Every package will differ. We are 100% commercial, shrink wrapped,
boxed, CD, etc. So I have always strived for "Customer comes first,"
"Getting it right the first time," high quality, lower TCO,
simplicity, shorter lead time in installation to full operation so my
IETF inputs are generally with that engineering mindset. SMTP is just
the networking part of the integrated application hosting platform (a
BBS for the old school folks), so everything needs to be
installed/setup relatively easy for the non-expert, lower experienced"
administrator, sysop, operator, even "help desk" tech support guy.

Overall, we have to program the IETF output for our customers and for
the most part, I used the sensible "IETF defaults" for "out of the
box" operations. I don't want additional support cost/problems so I'm
very conservative and only support technically logical new things or
changes that can work without trouble and doesn't open additional can
of worms. I also tend to avoid anything that isn't cooperatively
competitive, i.e, I won't support a vendor idea that benefits them or
few only and exerts more trouble for others. For any exploration, I
will add the necessary flexibility, revert switches, etc, if it makes
sense and always, backward compatibility is first.

Server-side p-code compiled scripting allows for programmable hooks
into the hosting platform, including the SMTP process. Threaded slave
calls are made at each SMTP state. SPF is processed at RCPT state to
begin processing the collected client IP, EHLO, MAIL FROM and RCPT TO
SMTP level session parameters. A YES/NO (250/550) response at RCPT.
Greylisting, DKIM, ADSP, ATPS and now DMARC is processed at DATA
state. A 250/45x/55x) response at DATA. SPF FAIL Rejects are done at
SMTP before DATA. All this extra processing is important because it
points to the increased SMTP overhead that we experience that can
affect scaling needs. What is the industry new average increase
session time? For my site where I have everything enabled, its about
1-5 seconds or more.

Finally, in regards to the DMARC parts of this thread. I see more DKIM
related changes that has a high potential for more overhead, more
processing, more waste and work, i.e, double DKIM signature
processing. I am not a fan of the 5322 rewriting idea. It has the
potential to conflict 5322.From stable platform designs. But I will
explore the "controls" that can be added or that may already exist.

--
HLS

[Mark Rousell]

On 23/10/2015 19:36, Martin Rex wrote:
> Interestingly, there essentially is a legal requirement for rewriting
> the From: header field for addresses that publish DMARC records when
> delivering EMail to places where DMARC processing isn't outright
> illegal (DMARC processing is clearly illegal in the EU), because
> handing an EMail to an MTA that might be processing DMARC will be
> illegal in the EU just as processing DMARC oneself. So one of the
> alternatives to unconditionally bouncing a DMARC-impaired EMail, will
> be rewriting the From:. -Martin

I've been following the whole DMARC saga with some interest but I've not
previously noticed any suggestion that processing DMARC might be illegal
in the EU. Can you explain why you take the view that processing DMARC
should be illegal in the EU? For the avoidance of doubt, when you refer
to "DMARC processing" to what exactly are you referring?

--
Mark Rousell

[John Levine]

>Or, one might consider a switch to a different list manager entirely.
>Sympa is really excellent for handling large numbers of lists, handles
>DMARC issues, doesn't send cleartext password reminders, is incredibly
>well supported.

I think Sympa is just dandy, and am in the process of switching to it
from mj2, but its DMARC hackery isn't much different from Mailman's.
It does have remarkably good DKIM and S/MIME handling.

I have a back end shim that rewrites mail addresses as required to
avoid DMARC while remaining deliverable, e.g., if a message is from
***@yahoo.com, it rewrites it to ***@yahoo.com.dmarc.fail and
sets up a temporary local forward.

It's grody but it works remarkably well. Anyone wants it, just ask.
It's written in perl and the full forwarding groditude uses a litttle
daemon (also written in perl) that will need some rewriting if your
MTA isn't qmail.

R's,
John

[John Levine]

>Which suggests that DMARC status is pretty much orthogonal to spam detection,
>on this small sample.

DMARC has very little to do with spam detection. Its original purpose
is to deter phishing of famous brands, of which paypal is the poster
child. It works pretty well for that, since those organizations tend
to send all of their mail from a few places they control and (other
than Linkedin which is no great loss) the staff members who are on
mailing lists use addresses in other domains.

Last year AOL and Yahoo repurposed it after they each separately
allowed crooks to steal their users' address books, so AOL and Yahoo
users were getting spam from the AOL and Yahoo addresses of people
they knew, sent from outside AOL and Yahoo. So AOL and then Yahoo
turned DMARC on, which was quite effective at stopping that particular
flavor of spam, but in the process forcing the costs of their security
failures on everyone else.

R's,
John

PS: To the obvious question of why don't crooks phish paypal from
lookalike domains, they do, but a remarkable number of them still use
the exact domain. Partly that's because it's easier, partly because
if the exact address gets through, it can match entries in the
recipients' address books and get displayed in ways that makes it look
more credible.

[John Levine]

>No idea how flexible mailman is, but how hard would it be to write some
>code that ...

Instead of guessing, five seconds of googlage found this informative page:

http://wiki.list.org/DEV/DMARC

Re your suggestion, how many subscribers to IETF lists currently use
gmail? What would happen if they were all summarily ejected?

R's,
John

[Brian E Carpenter]

On 26/10/2015 09:41, Philip Homburg wrote:
> In your letter dated 25 Oct 2015 20:24:55 -0000 you wrote:
>>> No idea how flexible mailman is, but how hard would it be to write some
>>> code that ...
>>
>> Instead of guessing, five seconds of googlage found this informative page:
>>
>> http://wiki.list.org/DEV/DMARC
>
> I guess this is a complex way of saying, no mailman doesn't do that?
>
>> Re your suggestion, how many subscribers to IETF lists currently use
>> gmail?

Only a kind person at the Secretariat could answer that definitively.
Based on a couple of lists I happen to administer, I estimate that
it's 15 to 20%.

Based on recent editions of the Narten posting summary, a similar
fraction of the traffic on this list is from gmail addresses.

>> What would happen if they were all summarily ejected?

In fact, today's problem (from my observations) is the small proportion
of IETF folk who post from yahoo.com addresses. Their traffic is simply
lost as far as any dmarc-obeying receiver is concerned.

Gmail won't be a problem until June 2016, as indicated in the story that
prompted this thread.

> My proposal was not to eject them but to mark them as second class citizens.

Indeed. And if it came to that, speaking for myself, I would switch to an address
not contaminated by dmarc p=reject, unless a good solution for mailing
lists was found first.

Brian

> People who use unresponsive big e-mail providers off load a lot of problems
> to the community.
>
> Using a constant from should be a good incentive to look for alternatives
> but does not provent people from contributing.
>
>

[Jelte Jansen]

On 10/25/2015 09:41 PM, Philip Homburg wrote:
> In your letter dated 25 Oct 2015 20:24:55 -0000 you wrote:
>>> No idea how flexible mailman is, but how hard would it be to write some
>>> code that ...
>>
>> Instead of guessing, five seconds of googlage found this informative page:
>>
>> http://wiki.list.org/DEV/DMARC
>
> I guess this is a complex way of saying, no mailman doesn't do that?
>

>From what I read on that page, that is essentially what mailman does (at
least as one of the options), but in a slightly less confrontational way
(i.e. not From: 'bad mail config at sender', but simply From: 'replaced
by list'). It even notes that it breaks reply-to and search, though it
certainly doesn't make an exhaustive list of UI/client processing problems.

Jelte

[John R Levine]

> For better or worse, we've got a perfectly good, standard, way
> of doing that when the list is to be treated as the sender.
> That would be to ask developers of list software to use
> Resent-From.

ARC adds a cryptographic chain of signatures adopted from DKIM. I suppose
you could use an extended version of Resent-From but it'd amount to the
same thing.

Regards,
John Levine, ***@taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

[John R Levine]

> That doesn't mean that an ARC signature chain is a bad idea. I
> have doubts about its effectiveness in practice, but, if it
> help, that is great. But changing the intuitive (and
> business-letter-based) semantics of the "From:" field strikes me
> as a bad idea ...

It strikes pretty much everyone as a bad idea other than a handful who've
drunk too much DMARC koolaid, a group that does *not* include the ARC
authors.

The whole point of ARC is that lists do whatever they do now, and add ARC
headers for automated parocessing like they added DKIM headers a few years
back. None of the headers visible to users are supposed to change.

Regards,
John Levine, ***@taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

[Phillip Hallam-Baker]

On Mon, Oct 26, 2015 at 12:26 PM, John C Klensin <john-***@jck.com> wrote:
>
>
> --On Monday, October 26, 2015 16:54 +0100 Jelte Jansen
> <***@sidn.nl> wrote:
>
>> From what I read on that page, that is essentially what
>>> mailman does (at
>> least as one of the options), but in a slightly less
>> confrontational way (i.e. not From: 'bad mail config at
>> sender', but simply From: 'replaced by list'). It even notes
>> that it breaks reply-to and search, though it certainly
>> doesn't make an exhaustive list of UI/client processing
>> problems.
>
> For better or worse, we've got a perfectly good, standard, way
> of doing that when the list is to be treated as the sender.
> That would be to ask developers of list software to use
> Resent-From. It has its own set of issues and the various bits
> of software that try to tie keys and DNS records to mail
> originators would need to be upgraded to understand the
> semantics of "Resent-*" (and, btw, Sender) fields, but it would
> conform to the standards, not loose information, and not closely
> resemble a cheap hack.

Clearly some folk are going to have to change their behavior.

What is the more practical approach, to ask a small group of
technically literate people to modify their behavior in a modest
fashion or to demand updates to every mail client and server on the
Internet?

Of course the response might be different depending on whether the
objective is to produce a document or arrive at a deployed solution.

[Martin Rex]

Mark Rousell wrote:
>Martin Rex wrote:
>
>> Interestingly, there essentially is a legal requirement for rewriting
>> the From: header field for addresses that publish DMARC records when
>> delivering EMail to places where DMARC processing isn't outright
>> illegal (DMARC processing is clearly illegal in the EU), because
>> handing an EMail to an MTA that might be processing DMARC will be
>> illegal in the EU just as processing DMARC oneself. So one of the
>> alternatives to unconditionally bouncing a DMARC-impaired EMail, will
>> be rewriting the From:. -Martin
>
> I've been following the whole DMARC saga with some interest but I've not
> previously noticed any suggestion that processing DMARC might be illegal
> in the EU. Can you explain why you take the view that processing DMARC
> should be illegal in the EU? For the avoidance of doubt, when you refer
> to "DMARC processing" to what exactly are you referring?

I have been elaborating on this issue here before

http://www.ietf.org/mail-archive/web/ietf/current/msg87704.html

The issue is that an MTA operated by an ISP is a telecommunications
provider and subject to the telecommunication secrecy requirements
based on EU directive 2002/58/EC.

The sender of the telecommunication is the peer that connects the
MTA and sends the MAIL FROM ... of the SMTP transaction.

The receiver of the telecommunication is the identity (or multiple identities)
specified in the RCPT TO of the SMTP transaction.

The publisher of a DNS DMARC policy record that happens to match
the domain part from the rfc822-From: from contents of an rfc5322
message that is conveyed within the SMTP transaction is a legal
third party to the telecommunication.

It is a criminal offence for the telecommunications provider to
tell legal third parties about telecommunications between senders
and receivers (such as what a DMARC policy p=report tries to achieve).

It is a criminal offence for the telecommunications provider to
take notice of the *Contents* of the telecommunication except for an
extremely narrow set of probable cause circumstances. rfc822-From:
is part of the communication contents, not part of the "metadata".
(only the SMTP envelope is the communication metadata).

It is a criminal offence for the telecommunications provider to
suppress telecommunication for non-technical reasons of the
communication link from sender to receipient (such as processing
a DMARC p=reject policy).

Since these are all legal prohibitions, it is impossible to circumvent
these prohibitions through contract clauses. Any contract clause that
tries to circumvent a legal prohibition is Null and Void from the start
(certainly here in Germany, Para. 134 BGB), and probably in several
other EU member states as well.


-Martin

[John Levine]

>The issue is that an MTA operated by an ISP is a telecommunications
>provider and subject to the telecommunication secrecy requirements
>based on EU directive 2002/58/EC.

By that theory, spam filtering would be illegal, too, yet all the
German ISPs I know manage to do it.

R's,
John

[Niels Dettenbach (Syndicat.com)]

Am 28. Oktober 2015 01:02:46 MEZ, schrieb John Levine <***@taugh.com>:
>>The issue is that an MTA operated by an ISP is a telecommunications
>>provider and subject to the telecommunication secrecy requirements
>>based on EU directive 2002/58/EC.
>
>By that theory, spam filtering would be illegal, too, yet all the
>German ISPs I know manage to do it.

If i understand correctly, it *is* illegal, except if the ISP / mail service customer allows it (i.e. by contract). Where spam fighting is getting "intrusive" be the meaning of that law (i.e. using Headers, subject, body etc.), is a bit unclear afaik.

So the most Herman ISPs (should) have a corresponding part in their contracts...


best regards,

Niels.
- ---
Niels Dettenbach
Syndicat IT & Internet
http://www.syndicat.com

[Dave Crocker]

On 10/28/2015 12:48 AM, Niels Dettenbach (Syndicat.com) wrote:
>> By that theory, spam filtering would be illegal, too, yet all the
>> >German ISPs I know manage to do it.
> If i understand correctly, it *is* illegal, except if the ISP / mail service customer allows it (i.e. by contract). Where spam fighting is getting "intrusive" be the meaning of that law (i.e. using Headers, subject, body etc.), is a bit unclear afaik.
>
> So the most Herman ISPs (should) have a corresponding part in their contracts...


On the average over the years, substantive discussion of legal details,
relative to work in the IETF, has proved unhelpful to that work.

Sometimes the entertainment value has been significant, however, but not
any intended engineering guidance.

d/

--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

[Martin Rex]

John Levine wrote:
>
>>The issue is that an MTA operated by an ISP is a telecommunications
>>provider and subject to the telecommunication secrecy requirements
>>based on EU directive 2002/58/EC.
>
> By that theory, spam filtering would be illegal, too, yet all the
> German ISPs I know manage to do it.

I agree with Dave Crocker, that discussing legal issues on IETF mailing
lists usually doesn't work well at all. I would really prefer to stop
this particular sub-discussion here.


The problem with your "by that theory" is that this legal theory doesn't
work the way you think it does.



An anti-spam filtering scheme that happens before right before an EMail
is delivered into a personal mailbox, and which happens on a purely
opt-in basis by each individual EMail receipient is compatible with
the EU laws.

The EMail recipient as specified by the EMail sender is a rightful
participant of the communication, and the processing happens
"at the communication endpoint", and not "in transit".
The receipient is allowed to decide which data to "receive" and
which data to ignore or delete, and the receipient is also allowed
to voluntarily(!!) outsource spam filtering or pre-sorting to
his Mail provider.


Maybe a brief description of a decision from the German Federal Labour court
from 2009 helps understanding (1 AZR 515/08 from 20-Jan-2009).


A labour union sent (unsolicited) EMail advertisements to employees
at their work Email accounts, and the employer sued them to stop
sending such EMails to work email accounts of employees.

The German Federal Court (the highest appelate court on that matter)
ruled, that the employer must tolerate such EMails. Only employees
themselves, individually and voluntarily, can send "opt-out" requests
for such EMails to unions, which the union will then have to respect.
Since labour unions are privileged by the german constitution, they
even enjoy a "cold call" privilege that is an exception to the cold
call prohibition for other businesses.

If an employer would try to filter out such union Emails by technical
means, it will be a criminal offense, and no amount of contracts and
contract clauses can make such action non-criminal.


-Martin