BATV stands for Bounce Address Tag Validation and is a method to prevent backscatter from spam runs. It works by modifying (danger! technical content ahead!) the Envelope From address in an SMTP session from email@example.com to firstname.lastname@example.org. If this email is undeliverable, it will be send back to email@example.com instead of to firstname.lastname@example.org and your mail host knows that this is a valid undeliverable message.
So what has Postgrey to do with this? Postgrey is a greylisting server. It is (danger! technical content ahead!) forcing email deliveries from addresses and hosts which are not yet known to be retried later. Why? Earlier this century, emails sent by viruses and spam-hosts weren't smart enough to understand this and the email with the malicious payload was not accepted by your mailhost.
Yes, but what has greylisting to do with it? Greylisting delays every email from / email to / sending host combination it hasn't seen before. So if BATV changes the email from address every day, the first email from that user will be delayed every day. Every day! So Postgrey needs to be taught what the real email address is. Luckely BATV keeps this information in the from address: email@example.com. Small patch, and it works.
And now the tricky stuff: Not every read the documentation properly, and the two following formats have been seen:
Brilliant! They swapped it around! So my four line patch becomes an eight line firstname.lastname@example.org email@example.com
Anyway, the patch is available and submitted to the Postgrey author.
Note: Please note that I've made a little change to the patch to pick the second field (as the standard suggests) instead of the wrong standard. Not that it ever should come to there, but it's a "just in case" thing.
About two years ago we re-implemented greylisting, sender verification and RBL blacklisting on our MTAs. As a result, we had a very steep drop in unsolicited email, but also very interesting mailbounces from applications which didn't set a proper envelope-from address. It taught us that we needed two things: a whitelist service and a blocked-email reporting service.
The blocked-email reporting service was easy, just parse the Postfix log files once a day and check all the NOQUEUE entries. Then match the temporary failed deliveries against the successful deliveries and the leftover is a failed delivery.
The whitelist service is not too difficult, thanks to the check_policy_service feature which gives the black/whitelist daemon the sending MTA, and the envelope-from and envelope-to email addresses. A lookup against the black/whitelist database and all goes fine.
It is not really one lookup, it is a set of seven lookups:
Recently we were getting strange errors in the blocked-email reporting service, talking about Internal server configuration errors which were happening on our primary MTA. Not that the email got blocked by it, it just was delivered to our backup MTA. But still it was something which didn't make sense. We found out very soon that it was caused by a slow black/whitelist daemon. Not really a slow black/whitelist daemon, more a slow database. And not really a slow database, just a busy database.
A busy database? All it is doing is euhmmm... let's see... 70 queries per second?!?!?! A quick look at the Cacti graphs showed me that over the last three weeks mail servers went from an average of two messages per second to an average of six messages per second (message delivery attempts that is), and since every message has seven database queries it kind of shot through the roof and the black/whitelisting daemon we had couldn't handle it fast enough.
Time for a redesign. We have about 1500 entries in the black/whitelist database, and a hit-rate of about zero (most of the rejecting of email is done via sender verification and via DNS-RBLs). With such a low hit-rate, it shouldn't be a bad thing to locally cache a copy of it and refresh it every 15 minutes. That will save about 5400 database queries during that time period.
Three hours later, some rewriting of the black/whitelist daemon and the database server is happy again with the two queries per 15 minutes now. And the mail? It kept flowing...
Todays idiot mail server of the week award goes to webcontrolcenter.com!
All seems to go fine, until you try to send email as postmaster:
[~] edwin@k7>telnet mail22.webcontrolcenter.com 25 Trying 188.8.131.52... Connected to mail22.webcontrolcenter.com. Escape character is '^]'. 220 mail22.webcontrolcenter.com ehlo edwin.adsl.barnet.com.au 250-MAIL22 Hello [184.108.40.206] 250-SIZE 31457280 250-AUTH LOGIN CRAM-MD5 250 OK mail from:<> 250 OK <> Sender ok rset 250 OK mail from:<firstname.lastname@example.org> 250 OK <email@example.com> Sender ok rset 250 OK mail from:<firstname.lastname@example.org> 550 Sender is not allowed. Connection closed by foreign host.
And they rudely close the TCP session for you too!
I've just enabled sender verification on our mailservers.
What does that mean you might wonder, and how is that different from grey-listing.
With sender verification, postfix checks if the address in the MAIL FROM command gets accepted by the MX servers of that address. It tries to do it realtime, but if it takes too long (>6 seconds) it will temporary fail the SMTP session and waits until the sending MTA retries.
Grey-listing on the other just temporary fails the first delivery attempt and waits until the sending MTA retries.
Keep in mind that two different things are checked here:
One day I'll be brave enough to do also SPF checking (Allowed delivery of email for that domain by that MTA) and everything is as open as it will be, or as closed as it will be.
If you think this is bad: (mavetju.org isn't served by 220.127.116.11)
Received: from mavetju.org ([18.104.22.168]) by imta02sl.mx.bigpond.com with ESMTP id <20050524232049.GTMA2733.email@example.com>; Tue, 24 May 2005 23:20:49 +0000 message-id: <UHUh4a7dWj6_CpI3ZmfY@mavetju.org>
Wait until you see this:
Return-Path: firstname.lastname@example.org Received: from APlessis-Bouchard-152-1-59-216.w82-121.abo.wanadoo.fr (APlessis-Bouchard-152-1-59-216.w82-121.abo.wanadoo.fr [22.214.171.124]) by mx1.midcoast.com.au (8.13.1/8.13.1) with SMTP id j4N6sWvS003077 for <email@example.com>; Mon, 23 May 2005 16:54:47 +1000 Received: from mail3.barnet.com.au by APlessis-Bouchard-152-1-59-216.w82-121.abo.wanadoo.fr (8.9.3/8.9.3) with ESMTP id PCEIP7onXFNw for <firstname.lastname@example.org>; Mon, 23 May 2005 14:41:04 -0700 Received: from (root@localhost) by mail3.barnet.com.au (8.12.8/8.12.8/Submit) id 1GaCy2wErDj5Ks for <email@example.com>; Mon, 23 May 2005 14:41:04 -0700 Date: Mon, 23 May 2005 14:41:04 -0700 From: Edwin Groothuis <firstname.lastname@example.org> Reply-To: Edwin Groothuis <email@example.com> Message-ID: <firstname.lastname@example.org>
What do the headers says?
Why is this worsening? It is because the email actually looks, for the untrained eye and a lot of automatic header-parser programs, like it was coming from mail3.barnet.com.au:
In the first example, everybody who knows a little bit about SMTP headers first checks if 126.96.36.199 is somewhat related to 16wardell.com.au.
In the second example, you have two more lines to parse. I admit that the syntax of the second-last line isn't proper (it is missing the hostname/ip address between brackets in the from field), but for the rest looks pretty good.
What is still wrong with it?
Could this have been prevented if mx1.midcoast.com.au would have done SPF checks? Yes. The SPF tests would have failed on every received line with a hostname.
Some mailers are dumb (see previous article), but some people who configure them are dumb too! After a mass mailing, I got the following errors:
If it is a temporarily error, give me a 4xx error, not a 5xx error!<email@example.com>: host mailhub.totel.ru[188.8.131.52] said: 550 sorry, user is temporarily unavailable (#5.1.1 - chkusr) (in reply to RCPT TO command).
Yes. But I'm in Australia. Wrong hemisphere mate!<firstname.lastname@example.org>: host chaos.fxp.org[184.108.40.206] said: 554 <mailout2.barnet.com.au[220.127.116.11]>: Client host rejected: No China Spam (in reply to RCPT TO command)
Can you please check my SPF records? Thanks!<kouichi@MysticWALL.COM> host mailmaster.MysticWALL.COM[18.104.22.168] said: 550 <mailout2.barnet.com.au[22.214.171.124]>: Client host rejected: We don't accept mail from spammers (in reply to RCPT TO command)
At least they checked the content, but it would be nice if they could get me the score from the spam-scanner so I could see if I was maxed out or that it was just with my toes over the line.<email@example.com>: host mx.argosnet.com[126.96.36.199] said: 554 5.7.1 The message was categorized as SPAM. (in reply to end of DATA command)
Recently we implemented so called greylisting on our mail servers. This means that all incoming SMTP sessions with the following new combinations of sending mail server IP address, sender email address and recipient email address gets temporary rejected (SMTP error code 450, meaning: try again later).
From RFC2821: 4yz: Transient Negative Completion reply
The command was not accepted, and the requested action did not occur. However, the error condition is temporary and the action may be requested again. The sender should return to the beginning of the command sequence (if any). It is difficult to assign a meaning to "transient" when two different sites (receiver- and sender-SMTP agents) must agree on the interpretation. Each reply in this category might have a different time value, but the SMTP client is encouraged to try again. A rule of thumb to determine whether a reply fits into the 4yz or the 5yz category (see below) is that replies are 4yz if they can be successful if repeated without any change in command form or in properties of the sender or receiver (that is, the command is repeated identically and the receiver does not put up a new implementation.)
This saves us about 99% of the incoming spam and viruses and is a relief for our mailboxes and the email virusscanners.
Now the bad news, there are some very brain-dead SMTP servers on the internet...
And guess what? They all run on MS Windows. Who had expected that?
Here is the list of them:
MailMax from SmartMax Software Inc.. When receiving an 450, they bounce the mail back to the sender. And this is the error message they are getting:
The 'To' address firstname.lastname@example.org was rejected by the remote server.Which mailserver was it talking to? What was the full error message? What was the error code? And why do you say it's a permanent error while it was a transient error? BRAIN DEAD!
This is permanent error, and the message will not be retried any further.
And they claim on their website:
IMPORTANT: Codes that start with 4 and 5 are the ones that tell you that your message won't be sent until you find and fix the problem.You! Yes you! You should fix the problem, and not the other side, or the MailMax mail server.
Update: With their latest version *version 5.5), at least the error messages are better:
Sorry, your message from <email@example.com> to <firstname.lastname@example.org> could not be delivered. The specific error is: 450 <email@example.com>: Recipient address rejected: BarNet Engineered Transit Delay -- 39 seconds This is permanent error, and the message will not be retried any further.
Still it's a 'permanent' error, but at least it's visible for the person the email was returned to that they interpreted it wrongly.
Sorry, your message from <firstname.lastname@example.org> to <email@example.com> could not be delivered. The specific error is: 450 <firstname.lastname@example.org>: Recipient address rejected: BarNet Engineered Transit Delay -- 45 seconds 2 attempts will be made to re-send your e-mail. Each attempt will be 3 hours apart.
That's much better! Everybody upgrade to the latest version!
CapeSoft Mailer by CapeSoft. It also immediately bounces the email without retrying. BRAINDEAD!
Bigpond.com by Telstra
This is all the attempt of Telstra (Australian ISP) to handle SMTP sessions with a 450 status:
Oct 29 16:17:56 mag postfix/smtpd: NOQUEUE: reject: RCPT from gizmo06ps.bigpond.com[188.8.131.52]: 450 <email@example.com>: Recipient address rejected: BarNet Engineered Transit Delay -- 45 seconds; from=<firstname.lastname@example.org>, to=<email@example.com> proto=SMTP helo=<gizmo06ps.bigpond.com>
That's all: one attempt. And the sender doesn't get an "Your email has failed" message. BRAINDEAD!
InterMail from Openwave Systems Inc..
It doesn't retry at all. (experienced with ozemail.com.au)
Remember the old rule of the thumb regarding email and viruses? "As long as you don't start the attachment, you are safe."
The computer world has evolved since then... Read on!
Up to the release of MS-Word 6, the line between safe and unsafe attachments in your email was simple: if the file was executable (i.e. did it end with .exe, .com or .bat), then it was not safe. Was it is a text file, a document, a spreadsheet, then it was safe to open.
MS-Word 6 introduced a new feature: A scripting language in the word-processor, and it was installed with scripting enabled by default. There were a couple of issues with it:
The thin border between safe and unsafe attachments was breached: documents could suddenly have executable payload.
The first Word virus was the Concept Word Macro virus. It didn't do much besides displaying a dialogbox with a "1" on the screen when an infected document was loaded and infecting other documents with itself. It didn't do any damage further, it was just annoying.
The next feature was the capability of the Word scripting language to interact with the MS Outlook mail reader. Where a Word viruses up to that point was only able to propagate slowly via shared Word documents, it suddenly had access to a faster path: It could send itself via Outlook to everybody in the address book on the infected computer, and with a little bit of luck the recipients would open the Word document and the infection would spread.
Beside the technical capabilities to get this virus spreading, the virus needed to convince the receiver that it was safe to open the attachment. Welcome to the social engineering department. The first step is to make sure the receiver trusts the source. Since the virus gets the receivers' address from the address book of the user whose Word document is infected, that means there is some kind of trust relationship between the sender and receiver, and thus most likely also between the receiver and the sender. The second step is to use a catchy text in the body while referring to the attachment, for example by referring to the document as a shared secret between the sender and receiver.
The first virus which successfully combined these two issues was the Melissa virus. When a user opens an infected document, the virus is activated and it infects the NORMAL.DOT file so all newly created documents will contain the virus; sends itself in an email with the following text to the first 50 people in the adddress book of the infected user:
Subject: Important Message From username Here is that document you asked for ... don't show anyone else ;-)'
It is hard to resist emails with this text, even if you are made suspicious by the fact that you got five of them already, all from different sources...
When displaying emails, most of the modern browsers only show the text and/or the HTML parts of the email. The last line of defense with regard to email based viruses was the fact that users needed to open the attachment before the virus became active.
If you could execute code in the HTML part of a document, you would be able to infect the computer without having to open an attachment.
So if you were able to make an HTML message with ActiveX components which bypassed the ActiveX security, you would be able to create a file on disk from just viewing the email message.
The BubbleBoy virus was the first virus which contained ActiveX code which bypassed the security and wrote a file to disk to the startup directory of Windows. That was all the email did. But when the computer was restarted, the file was started and the the virus started to propagate to everybody in the address book of the infected user.
After these viruses, the users knew they had to keep an eye open for attachments which were executables, and Word and other MS-Office related documents. Luckily, images and audio files (GIFs and MP3s for example) are still safe. So a quick visual inspection of the name of the attachment should tell if it is safe or not.
MS-Outlook, for example, doesn't by default display the full filename. So an attachment with the name test.doc would be displayed as a test with a Word document icon above it. And an attachment with the name test.gif.exe would be displayed as test.gif with an executable icon above it. If the user only checks the filename displayed, he would see the image filename and assume it was safe to open. Of course he would know immediately know he had ran a program instead of having opened an image, but the damage is done.
The payload of the Badtrans virus appeared under the filenames of README.TXT.exe and s3msong.MP3.pif. When the extension wasn't shown, the filenames looked innocent. When opening the attachment the executable was run. To soothe the user into thinking that the application hadn't ran, it showed a dialog box saying "File data corrupt: probably due to bad data transmission or bad disk access".
By sending infected emails through the mail application on the infected computer, MS-Outlook creates some side effects:
To overcome these problems, viruses started to be designed with their own SMTP engines. This way the ISP would be circumvented and would it be harder for the sender to find out where the infected emails were coming from.
Because the sender address could be faked now, more social engineering tricks can be performed. For example, email can be faked to come from the MAILER-DAEMON, which is normally computer generated email coming from SMTP gateways informing you that the email sent couldn't be delivered, and often attaching the full original email to the bounced message. Opening the attachment is one of the ways to find out which email wasn't able to be delivered.
To counter these kind of viruses, ISPs started to block all outgoing SMTP sessions except the ones coming from and to their own SMTP gateways.
The Frethem virus was the first virus with its own SMTP engine on board.
The MyDoom virus was one of the first viruses with its own SMTP engine on board and which also sends emails to common names (bill, john, mike etc) of target domains. The faked sender addresses will get the undeliverable messages.
The virus scanner on both the SMTP gateway of the ISP and the computer which retrieves the email should be able to open the attachments to check for viruses. But what if the attachment is an encrypted ZIP file and the key to open it is given in the email? Unfortunately, there is no clear method to prevent problems with these kind of email viruses.
The Mimail-M virus had an encrypted ZIP archive attached:
For unzip archiver download WinZip: http://download.winzip.com/winzip81.exe Password for archive is "kiss". Attached file: wendy.zip