I just started getting some weirdness with some email sent to my mail servers. Greylisting "come back later" (451) messages are being interpreted (it appears) by remote MTAs as a 5xx level error, meaning they immediately abort send attempts and bounce the email! I have never seen this before. I've seen it from two separate remote MTAs so far, but from reports I'm getting in, it's happening on more. Here's what I get from umce3cip03.ad.umanitoba.ca: Remote Server returned '554 5.0.0 <[50.71.247.87] #5.0.0 smtp; 5.1.0 - Unknown address error 550-'<foobar@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later' (delivery attempts: 0)>' On my server sendmail log I see: Aug 16 08:43:15 pog sendmail[14379]: v7GDhFav014379: Milter: to=<foobar@tecnopolis.ca>, reject=451 4.7.1 Greylisting in action, please come back later So umce3cip03.ad.umanitoba.ca appears to be taking my 451 and turning it into a 500/554/510 permanent error. I also see this from yahoomail, but that's a different situation because yahoo is always considered "broken mta" in milter-greylist and we must whitelist all of their servers... but, the usual/previous symptom was yahoo would keep retrying as normal, just with a different IP each time, thus never passing greylisting. And now yahoo is doing the same thing as umce3cip03.ad.umanitoba.ca above (when I haven't yet whitelisted the particular IP that day, grrr): Sorry, we were unable to deliver your message to the following address. <foo@tecnopolis.ca>: 550: <foo@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later So at least 2 MTAs, probably more, are changing 451 greylist into 5xx. Is there some new massive change out there to basically take greylist MTAs as broken? Is there a way to find out what MTA (or outsourced service provider) umce3cip03.ad.umanitoba.ca is using? Perhaps there is just one brand that unilaterally decided on this action? I can find no google hits on any of this. :-( There is an easy workaround/gotcha: if people wait the greylist timeout (usually 2 to 20 minutes) and then resend the same to-from-ip tuple email, it will go through as their server will have been whitelisted in the interim! But that makes it harder to troubleshoot this problem because it then becomes transient. If any MUUGers have problems with MUUG mailing lists bouncing, please resend your email after 1 hr if it hasn't showed up yet (you can check the website mail list archives section) and/or please email me directly with your bounce email message (twice, wait 20 mins!) so I can solve this for both MUUG servers and my own. If this problem is a) permanent/deliberate, and b) widespread, I think that spells the death of greylisting (grrrrr...) and the nearing of "spf strict" enabling.
On 17/08/2017 3:22 AM, Trevor Cordes wrote:
I just started getting some weirdness with some email sent to my mail servers. Greylisting "come back later" (451) messages are being interpreted (it appears) by remote MTAs as a 5xx level error, meaning they immediately abort send attempts and bounce the email! I have never seen this before.
I've seen it from two separate remote MTAs so far, but from reports I'm getting in, it's happening on more.
Here's what I get from umce3cip03.ad.umanitoba.ca:
Remote Server returned '554 5.0.0 <[50.71.247.87] #5.0.0 smtp; 5.1.0 - Unknown address error 550-'<foobar@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later' (delivery attempts: 0)>'
On my server sendmail log I see:
Aug 16 08:43:15 pog sendmail[14379]: v7GDhFav014379: Milter: to=<foobar@tecnopolis.ca>, reject=451 4.7.1 Greylisting in action, please come back later
So umce3cip03.ad.umanitoba.ca appears to be taking my 451 and turning it into a 500/554/510 permanent error.
That's very weird... I hadn't seen that one before. I'll have to check my own server logs, since I use greylisting too.
I also see this from yahoomail, but that's a different situation because yahoo is always considered "broken mta" in milter-greylist and we must whitelist all of their servers... but, the usual/previous symptom was yahoo would keep retrying as normal, just with a different IP each time, thus never passing greylisting. And now yahoo is doing the same thing as umce3cip03.ad.umanitoba.ca above (when I haven't yet whitelisted the particular IP that day, grrr):
Sorry, we were unable to deliver your message to the following address. <foo@tecnopolis.ca>: 550: <foo@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later
So at least 2 MTAs, probably more, are changing 451 greylist into 5xx. Is there some new massive change out there to basically take greylist MTAs as broken? Is there a way to find out what MTA (or outsourced service provider) umce3cip03.ad.umanitoba.ca is using? Perhaps there is just one brand that unilaterally decided on this action?
The UofM is using Cisco Ironport servers as the front-end SMTP servers. Maybe that will help in your searches...
I can find no google hits on any of this. :-(
There is an easy workaround/gotcha: if people wait the greylist timeout (usually 2 to 20 minutes) and then resend the same to-from-ip tuple email, it will go through as their server will have been whitelisted in the interim! But that makes it harder to troubleshoot this problem because it then becomes transient.
If any MUUGers have problems with MUUG mailing lists bouncing, please resend your email after 1 hr if it hasn't showed up yet (you can check the website mail list archives section) and/or please email me directly with your bounce email message (twice, wait 20 mins!) so I can solve this for both MUUG servers and my own.
If this problem is a) permanent/deliberate, and b) widespread, I think that spells the death of greylisting (grrrrr...) and the nearing of "spf strict" enabling.
As far as the MUUG servers go, I don't think greylisting has been terribly effective, so it's not a big deal if we have to disable it. OTOH, on my workplace e-mail server, greylisting is sometimes effective (some days, it's in 2nd or 3rd place in the summary of rejected e-mail causes). The effectiveness of greylisting does seem to be decreasing over time, though. Many of the more "dedicated" spammers have adjusted to this, and now retry. (You can see that on the days where we're really hit hard, and the greylist rejection count actually seems to be lower.) -- Gilbert E. Detillieux E-mail: <gedetil@cs.umanitoba.ca> Dept. of Computer Science Web: http://www.cs.umanitoba.ca/~gedetil/ University of Manitoba Phone: (204)474-8161 Winnipeg MB CANADA R3T 2N2 Fax: (204)474-7609
The ironports were upgraded last night so this may be transitory. in any case i will forward this thread to the interested parties. On Thu, Aug 17, 2017 at 9:58 AM Gilbert E. Detillieux < gedetil@cs.umanitoba.ca> wrote:
On 17/08/2017 3:22 AM, Trevor Cordes wrote:
I just started getting some weirdness with some email sent to my mail servers. Greylisting "come back later" (451) messages are being interpreted (it appears) by remote MTAs as a 5xx level error, meaning they immediately abort send attempts and bounce the email! I have never seen this before.
I've seen it from two separate remote MTAs so far, but from reports I'm getting in, it's happening on more.
Here's what I get from umce3cip03.ad.umanitoba.ca:
Remote Server returned '554 5.0.0 <[50.71.247.87] #5.0.0 smtp; 5.1.0 - Unknown address error 550-'<foobar@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later' (delivery attempts: 0)>'
On my server sendmail log I see:
Aug 16 08:43:15 pog sendmail[14379]: v7GDhFav014379: Milter: to=<foobar@tecnopolis.ca>, reject=451 4.7.1 Greylisting in action, please come back later
So umce3cip03.ad.umanitoba.ca appears to be taking my 451 and turning it into a 500/554/510 permanent error.
That's very weird... I hadn't seen that one before. I'll have to check my own server logs, since I use greylisting too.
I also see this from yahoomail, but that's a different situation because yahoo is always considered "broken mta" in milter-greylist and we must whitelist all of their servers... but, the usual/previous symptom was yahoo would keep retrying as normal, just with a different IP each time, thus never passing greylisting. And now yahoo is doing the same thing as umce3cip03.ad.umanitoba.ca above (when I haven't yet whitelisted the particular IP that day, grrr):
Sorry, we were unable to deliver your message to the following address. <foo@tecnopolis.ca>: 550: <foo@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later
So at least 2 MTAs, probably more, are changing 451 greylist into 5xx. Is there some new massive change out there to basically take greylist MTAs as broken? Is there a way to find out what MTA (or outsourced service provider) umce3cip03.ad.umanitoba.ca is using? Perhaps there is just one brand that unilaterally decided on this action?
The UofM is using Cisco Ironport servers as the front-end SMTP servers. Maybe that will help in your searches...
I can find no google hits on any of this. :-(
There is an easy workaround/gotcha: if people wait the greylist timeout (usually 2 to 20 minutes) and then resend the same to-from-ip tuple email, it will go through as their server will have been whitelisted in the interim! But that makes it harder to troubleshoot this problem because it then becomes transient.
If any MUUGers have problems with MUUG mailing lists bouncing, please resend your email after 1 hr if it hasn't showed up yet (you can check the website mail list archives section) and/or please email me directly with your bounce email message (twice, wait 20 mins!) so I can solve this for both MUUG servers and my own.
If this problem is a) permanent/deliberate, and b) widespread, I think that spells the death of greylisting (grrrrr...) and the nearing of "spf strict" enabling.
As far as the MUUG servers go, I don't think greylisting has been terribly effective, so it's not a big deal if we have to disable it. OTOH, on my workplace e-mail server, greylisting is sometimes effective (some days, it's in 2nd or 3rd place in the summary of rejected e-mail causes).
The effectiveness of greylisting does seem to be decreasing over time, though. Many of the more "dedicated" spammers have adjusted to this, and now retry. (You can see that on the days where we're really hit hard, and the greylist rejection count actually seems to be lower.)
-- Gilbert E. Detillieux E-mail: <gedetil@cs.umanitoba.ca> Dept. of Computer Science Web: http://www.cs.umanitoba.ca/~gedetil/ University of Manitoba Phone: (204)474-8161 Winnipeg MB CANADA R3T 2N2 Fax: (204)474-7609 _______________________________________________ Roundtable mailing list Roundtable@muug.ca https://muug.ca/mailman/listinfo/roundtable
-- Micah Garlich-Miller <micahgarlichmiller@gmail.com>
Trevor, The problem may be at your end. (Have you changed anything there recently?) My reply to your address bounced when sent from my sendmail server, which hasn't changed in a long time. Here's what I got in the bounce message... ----- The following addresses had permanent fatal errors ----- <trevor@tecnopolis.ca> (reason: 550 <trevor@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later) ----- Transcript of session follows ----- ... while talking to tecnopolis.ca.:
DATA <<< 550 <trevor@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later 550 5.1.1 <trevor@tecnopolis.ca>... User unknown <<< 503 5.0.0 Need RCPT (recipient)
--v7HEwiKb114371.1502981924/copper.cs.umanitoba.ca Content-Type: message/delivery-status Reporting-MTA: dns; copper.cs.umanitoba.ca Received-From-MTA: DNS; lockwood3.cs.umanitoba.ca Arrival-Date: Thu, 17 Aug 2017 09:58:37 -0500 Final-Recipient: RFC822; trevor@tecnopolis.ca Action: failed Status: 5.1.1 Remote-MTA: DNS; tecnopolis.ca Diagnostic-Code: SMTP; 550 <trevor@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later Last-Attempt-Date: Thu, 17 Aug 2017 09:58:44 -0500 On 17/08/2017 3:22 AM, Trevor Cordes wrote:
I just started getting some weirdness with some email sent to my mail servers. Greylisting "come back later" (451) messages are being interpreted (it appears) by remote MTAs as a 5xx level error, meaning they immediately abort send attempts and bounce the email! I have never seen this before.
I've seen it from two separate remote MTAs so far, but from reports I'm getting in, it's happening on more.
Here's what I get from umce3cip03.ad.umanitoba.ca:
Remote Server returned '554 5.0.0 <[50.71.247.87] #5.0.0 smtp; 5.1.0 - Unknown address error 550-'<foobar@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later' (delivery attempts: 0)>'
On my server sendmail log I see:
Aug 16 08:43:15 pog sendmail[14379]: v7GDhFav014379: Milter: to=<foobar@tecnopolis.ca>, reject=451 4.7.1 Greylisting in action, please come back later
So umce3cip03.ad.umanitoba.ca appears to be taking my 451 and turning it into a 500/554/510 permanent error.
I also see this from yahoomail, but that's a different situation because yahoo is always considered "broken mta" in milter-greylist and we must whitelist all of their servers... but, the usual/previous symptom was yahoo would keep retrying as normal, just with a different IP each time, thus never passing greylisting. And now yahoo is doing the same thing as umce3cip03.ad.umanitoba.ca above (when I haven't yet whitelisted the particular IP that day, grrr):
Sorry, we were unable to deliver your message to the following address. <foo@tecnopolis.ca>: 550: <foo@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later ...
-- Manitoba UNIX User Group E-mail: <gedetil@muug.ca> c/o Gilbert E. Detillieux Web: http://muug.ca/ University of Manitoba Phone: (204)474-8161 Winnipeg MB CANADA R3T 2N2 Fax: (204)474-7609
I forgot to mention: the timestamps on your problem emails happen before they started work on the ironports. So I don't think it's related. On Thu, Aug 17, 2017 at 10:21 AM Gilbert E. Detillieux < gedetil@cs.umanitoba.ca> wrote:
Trevor,
The problem may be at your end. (Have you changed anything there recently?) My reply to your address bounced when sent from my sendmail server, which hasn't changed in a long time. Here's what I got in the bounce message...
----- The following addresses had permanent fatal errors ----- <trevor@tecnopolis.ca> (reason: 550 <trevor@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later)
----- Transcript of session follows ----- ... while talking to tecnopolis.ca.:
DATA <<< 550 <trevor@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later 550 5.1.1 <trevor@tecnopolis.ca>... User unknown <<< 503 5.0.0 Need RCPT (recipient)
--v7HEwiKb114371.1502981924/copper.cs.umanitoba.ca Content-Type: message/delivery-status
Reporting-MTA: dns; copper.cs.umanitoba.ca Received-From-MTA: DNS; lockwood3.cs.umanitoba.ca Arrival-Date: Thu, 17 Aug 2017 09:58:37 -0500
Final-Recipient: RFC822; trevor@tecnopolis.ca Action: failed Status: 5.1.1 Remote-MTA: DNS; tecnopolis.ca Diagnostic-Code: SMTP; 550 <trevor@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later Last-Attempt-Date: Thu, 17 Aug 2017 09:58:44 -0500
On 17/08/2017 3:22 AM, Trevor Cordes wrote:
I just started getting some weirdness with some email sent to my mail servers. Greylisting "come back later" (451) messages are being interpreted (it appears) by remote MTAs as a 5xx level error, meaning they immediately abort send attempts and bounce the email! I have never seen this before.
I've seen it from two separate remote MTAs so far, but from reports I'm getting in, it's happening on more.
Here's what I get from umce3cip03.ad.umanitoba.ca:
Remote Server returned '554 5.0.0 <[50.71.247.87] #5.0.0 smtp; 5.1.0 - Unknown address error 550-'<foobar@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later' (delivery attempts: 0)>'
On my server sendmail log I see:
Aug 16 08:43:15 pog sendmail[14379]: v7GDhFav014379: Milter: to=<foobar@tecnopolis.ca>, reject=451 4.7.1 Greylisting in action, please come back later
So umce3cip03.ad.umanitoba.ca appears to be taking my 451 and turning it into a 500/554/510 permanent error.
I also see this from yahoomail, but that's a different situation because yahoo is always considered "broken mta" in milter-greylist and we must whitelist all of their servers... but, the usual/previous symptom was yahoo would keep retrying as normal, just with a different IP each time, thus never passing greylisting. And now yahoo is doing the same thing as umce3cip03.ad.umanitoba.ca above (when I haven't yet whitelisted the particular IP that day, grrr):
Sorry, we were unable to deliver your message to the following address. <foo@tecnopolis.ca>: 550: <foo@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later ...
-- Manitoba UNIX User Group E-mail: <gedetil@muug.ca> c/o Gilbert E. Detillieux Web: http://muug.ca/ University of Manitoba Phone: (204)474-8161 Winnipeg MB CANADA R3T 2N2 Fax: (204)474-7609 _______________________________________________ Roundtable mailing list Roundtable@muug.ca https://muug.ca/mailman/listinfo/roundtable
-- Micah Garlich-Miller <micahgarlichmiller@gmail.com>
On 2017-08-17 Micah Garlich-Miller wrote:
I forgot to mention: the timestamps on your problem emails happen before they started work on the ironports. So I don't think it's related.
DATA <<< 550 <trevor@tecnopolis.ca>... 451 4.7.1 Greylisting in action, please come back later
Thanks guys, Gilbert was right, the problem was on my side. I upgraded to Fedora 26 2 weeks ago, and that's the only time sendmail and milter-greylist changed, but I thought, nah, this couldn't have been broken for a whole 2 weeks and I'm just hearing the complaints now! Dropping email for 2 weeks... don't even want to think about that... But, a quick telnet port 25 and some commands later, yup, it's me. Looks like a bug that hit in 2015 and was supposedly fixed but is now regressed. It could be a Fedora thing, as it looks like it's a missed patch because Fedora is still using sendmail 8.15 instead of 8.16 (why?) which has the official fix. https://bugzilla.redhat.com/show_bug.cgi?id=1482808 The quick fix: dnf downgrade sendmail Now gives me correct: 451 4.7.1 Greylisting in action, please come back later instead of buggy: 550 <validuser@problemsendmailhost.com>... 451 4.7.1 Greylisting in action, please And yes, it looks like I'm once again the first one in the world using Fedora 26 + sendmail + greylist. Bah. Always the guinea pig. Oh well, it is my pleasure to serve, so all you CentOS/RHEL people can have your bug-free major releases ;-)
From my bz, as to why this wasn't immediately noticeable: It's really insidious because email doesn't just stop: there are still 2 cases where emails get through: the tuple is in gm's hard or auto-whitelist, or the end user resends the email after the gm timeout period. So many emails will get through and mail admins may not know there is a problem until they start getting some phone calls about the sporadic missing emails...
Thanks to Adam for testing too.
participants (3)
-
Gilbert E. Detillieux -
Micah Garlich-Miller -
Trevor Cordes