When [SPAM] and [SureSPAM] slide through

Lately we have been fielding a lot of questions about why [SPAM] and [SURESPAM] messages keep on sliding through to the end users. We have also seen a lot of activity with users complaining about SPAM making it to them uninterrupted when it comes from an email address within their domain. Here is the problem:

In nearly all cases that we investigated, the user actually whitelisted their own domain or their own domains email address.

Why would this happen? Well, users tend to scan messages and look for familiar names and subjects. When they encounter something they recognize, like an email address from their colleague or from themselves, they trust the sender. When they trust the spoofed address, all future mail comes through, causing frustration for everyone involved.

Advise your users not to trust their own email address space when it shows up in ExchangeDefender SPAM reports. ExchangeDefender only intercepts messages going in and out of the organization, it does not filter internal messages. Any mail with the domains address space caught by ExchangeDefender is highly likely to be spoofed.

Of course, usage and configuration of ExchangeDefender is up to you, we make the product flexible enough to allow you to set your own policies. But blindly trusting entire domains and mirrored trust sets (from exchangedefender.com to exchangedefender.com for example) will only let dangerous items through. Consider tightening up the ship if you are seeing ExchangeDefender starting to slip, our metrics show that our detection rate keeps on going up as both volume and percentage.

As always, thanks for letting us clean your mail.

New ExchangeDefender outbound servers

With ExchangeDefender 4.x infrastructure already in place on the inbound servers, we are moving our focus to the implementation of ExchangeDefender 4.x on the outbound servers. The new systems will go into production this weekend, April 5-6, and will feature new IP address relay servers:

outbound1.exchangedefender.com 65.99.255.236
outbound2.exchangedefender.com 65.99.255.232

This is the same IP range that is currently in use so you should not have to make any modifications or changes to your systems. Everything will just work transparently.

If you are currently using SPF, which we do not recommend, you will have to adjust it to include the new IP addresses. For illustration purposes, here is a look at the possible SPF record:

domain.com = “v=spf1 ip4:65.99.255.236 ip4:65.99.255.232 ip4:65.99.192.8 ip4:65.99.192.91 a:outbound.exchangedefender.com -all”

This record will restrict relaying for domain.com to ExchangeDefender outbound servers only (naturally you should include your own IP as well for non-smarthosted deliveries) but the above should get you started.

We don’t expect any issues as the new system has been in beta testing for a few months with no significant problems. Performance, logging, reporting and enforcement functionality that it delivers are far beyond comparison with the current service so we are really looking forward to it!

The New Soul of ExchangeDefender

At times in software development it becomes easier to make little optimizations to the process so long as the processing power becomes more affordable and available. Then one day you look up from what you’ve built on your monitor and see a wall of lights, indicators, pie charts and trends that use pretty pictures and colors to paint the portrait of the complexity that manage all the “little” issues you didn’t fix along the way. We are happy to report that after working around the clock for weeks we have regained a lot of control and a wall!

The story of ExchangeDefender is pretty simple. In the 90’s, we could not keep our Exchange servers up, fighting with relaying, viruses, hackers and platform instability became a daily frustration. So we figured, “Let’s put a box in front of it to just scan the junk and pass on valid stuff to Exchange” and it worked. The little AMD K5-100 workstation with an IDE hard drive and McAfee virus scanner did enough to keep our Exchange infrastructure running. Fast forward 12 years, the rise of SPAM, the rise of Malware, the pdf spam, the composite jpg spam, the rise of botnets and we have the ExchangeDefender of today, a 2,800 server network spread over 26 data centers and a full shift dedicated to just racking the new hardware and provisioning new nodes as our company and our load grow.

With data center space starting to shrink, power demands growing, power cost growing and the trend of SPAM increases turning vertical, we had a choice: change the way we manage SPAM flow or start breeding hamsters to generate power. Since nobody volunteered to clean the cage, we changed the way content is passed through and into our network.

The Old Way

Messages were accepted and scanned in the order they arrived with no bias shown for RBLs, origin or content. We took the message, checked it, and if it was clean we delivered it. With the SPAM loads constantly over the 95% range, legitimate messages had a 95% chance of being scanned with less urgency than their SPAM counterparts. To further complicate the issue, we scanned all messages equally: SPAM checks, heuristics, viruses, RBL presence, message integrity, the works. This created an immense demand for processing power, storage and network capacity.

Simply put, imagine a window of one second during which 10 messages were received and scanned sequentially from first to last. If the clean message was message #1, no worries. But if it was #10 in the scanning order it would have to wait for us to check for every minor chance that the nine messages before it had content anywhere from a pharmaceutical ad to banking fraud.

The New Way

We first changed the way we rely on our extensive knowledge of SPAM sources and senders and we created a system in which a bias is placed on messages coming from reputable senders. RBLs and statistical models are not perfect, that is why we will never bounce or delete an incoming messages without 100% certainty. Legitimate organizations can end up on RBLs from time to time, etc. But if we received over a million SPAM messages from a host in China with no reverse DNS, why should we treat the message #1,000,0001 from the same IP address with the same priority as the messages from hosts that constantly deliver legitimate mail that our users want to read? The good news is, we no longer do because messages are prioritized well before they hit the beef of ExchangeDefender.

Over the past month we have implemented a new SPAM filtering traffic shaping process that takes messages from known SPAM sources and puts them into a low priority queue to be scanned with far less of our arsenal than the messages from sources that are not as proven of SPAM heavens. So if you’ve mismanaged your infrastructure enough to end up in SpamCop, SpamHaus, SORBS, have an open relay, no reverse DNS the only thing we need to know is if the recipient trusts you. If they don’t, store/drop/delete, if they do, we scan for viruses, complex checks, custom filters and more.

This system allows us to enhance our scrubbing of unknown mail and keep you safe from more innovative threats while keeping SPAM stored just in case. Messages will still be accessible, still scanned. This new process shifts the burden of mail server management on the sender to keep their infrastructure in check while improving the security of our recipient customer. The era of trust on the Internet is long gone, mail servers that are mismanaged to the extent that they spew millions of SPAM messages, viruses and malware cannot be trusted not to have their logs scanned and malware routes replicated through the social means of replaying legitimate mail traffic to bypass filtering systems.

The Story So Far

So far, we have been able to improve nearly all tracked metrics of ExchangeDefender.

The level of SPAM detected is way up. The level of false positives (messages ExchangeDefender thought were SPAM but were released from quarantines by end users) is way, way down. The latency in delivery, scanning and processing are way down. Nearly everything we do has significantly improved over the past few weeks and while it took an enormous amount of hard work we feel it leaves the Internet a safer place for our clients and a far less profitable or even viable business for spammers.

Results

We have done our way to make sure the messages you receive get to you faster, safer while reducing the time needed to manage whitelists, review junk folders or worry about non-receipts. Our support has reflected this as well, despite February being the strongest sales month on record for ExchangeDefender, ever, we have had less support requests, ever!

Now, after a little deserved break, we turn to ExchangeDefender 4.x and think of ways we can use the new savings to make our service work for you even better.

As always, thank you for your business and thank you for trusting us to keep your inbox clean.

Update on ExchangeDefender Maintenance

Generally we keep ExchangeDefender maintenance on the NOC site but due to the extent of the maintenance done we felt it was important to place it here as well. Over the past 12 hours we have conducted urgent maintenance on the following systems:

1. ExchangeDefender reporting servers (ongoing, 8-10 hours remaining)
2. ExchangeDefender antispam engine (ongoing, up to 99.2% efficiency)
3. ExchangeDefender outbound network (complete)
4. ExchangeDefender global load balancing (complete)
5. ExchangeDefender cluster upgrades (complete)

It will take another 2-3 hours for the latency across the network to go down. The email reports will resume later today. The new outbound network hardware is online and active. The new reporting servers will go online this weekend along with the global network upgrade to Microsoft Exchange 2007 Service Pack 1.

We apologize for the inconvenience the additional mail latency has caused you and your clients, this will be the last major network upgrade for quite some time and the last major maintenance before the start of the ExchangeDefender 4.0. Most of the engine underneath of ExchangeDefender is running 4.x code already.

For updates on the ExchangeDefender maintenance please keep an eye on our NOC site at www.ownwebnow.com/noc

Enhanced SPAM filtering engine now online

Over the weekend we deployed a few enhancements to the antispam engine and hopefully starting tomorrow you should see far less junk in your Inbox. We have been testing this engine in parallel and have not seen a significant increase in false positives so we are positive this will be an improvement in overall performance and filtering rates.

As usual, if anything slips please forward it to spam@ownwebnow.com

Note on the forwards:

In the event that an obvious SPAM message gets through ExchangeDefender and into your mailbox, you can forward it to us at spam@ownwebnow.com for further inspection. There are times that the SPAM you received was from a source that is not known for spamming or that the message just did not contain enough obvious junk content to be filtered through. We also ask that you forward inline SMTP headers (in Outlook right click on the message in the listing, select Message Options and copy Internet headers) to us so we can verify that the message indeed passed through our network and that it was not released by one of your coworkers that it was cc’ed to (this happens all the time, someone releases the message from a familiar looking sender/subject and its a SPAM that was either cc’ed or bcc’ed to everyone in the organization). For 4.0, we are working on an Outlook addin.

If you are one of our partners and would like to hide the ownwebnow.com domain name, you can have messages sent to spam@exchangedefender.com. If you would rather give your customers your own address just create a mail enabled contact in your organization and forward it to either alias. Be advised that we may sometimes reply from that address in which case they will see our company name and our contact information – however, this is very infrequent! (mostly when people forward a few dozen identical items to which they have obviously subscribed to, such as popular newsletters, shopping sites, etc)

Change of policy in ExchangeDefender non-deliveries

As mentioned previously, we are starting to tighten up the network backend before the UI overhaul of ExchangeDefender 4.0 takes place. In preparing for it we are trying to adjust our network to help reduce load as well as improve the quality of service and transparency.

First the problem: Message sent to user XYZ was not received, can you please check?

There are three possible outcomes to the question above:

1. Almost always the message was delivered properly but the remote users server either discarded it as SPAM or the user outright deleted it and just did not want to fess up to it. We have invested thousands of man hours with our partners where the message from customer A went to customer B and can be backed up with transaction logs of the sending server, ExchangeDefender and recipients mail server. It made it to the mailbox. Then it disappeared. That we can’t do much about but with LiveArchive we have made it possible for our own customers to retrieve those mysteriously undeliverable items.

2. The message was delivered but discarded at some point in the process (ExchangeDefender, third party mailer, broken DNS, etc)

3. Something that we aim to fix: Message was delivered to ExchangeDefender but did not get delivered to the remote server because the remote server uses greylisting, callbacks, long delays, internal content blacklists, guy in the basement writing SpamAssassin rules.

The Solution: Less time in the queue, faster alerts to the user that there is a problem.

Our current settings will retry to deliver the outgoing message for up to five days. Error reports are sent within 24 hours. That is clearly too long as the users complain of issues within hours of sending the email. We will be generating errors sooner than later to help the IT staff and IT Solution Provider explain to the customer that the issue is on the remote recipients server, not on their own or ExchangeDefender which will help with the endless troubleshooting requests over the proven infrastructure just to find out that the recipients server has a poorly configured mail server.

This change in policy will only affect outbound mail, inbound mail will remain the same.

Going forward we will be issuing non-delivery alerts within 3 hours. We will return the message to the sender if the message is not delivered within 24 hours. Based upon the advice of our IT Solution partners and our internal testing, we have concluded that this is an adequate amount of time to attempt to deliver the message and if the delivery fails this gives the sender an option of contacting the recipient through other means until their mail problems have been solved.

ExchangeDefender 4.0 will feature full access to the ExchangeDefender transaction logs both inbound and outbound but in the meantime we felt it was important to introduce this change due to the growing number of hosts that are without protection and just unable to properly handle the amount of inbound mail sent to them. Each undeliverable and unreported issue with the remote server unfortunately increases the cost of support and reduces the satisfaction with the mail delivery because users tend to start blaming their own IT staff first. We hope these errors help the users see where the issue is sooner than later so they can continue to communicate with the remote party through other means.

New policy on ExchangeDefender activations and billing

We are implementing two new policies and processes for ExchangeDefender activations and billing. Some of these have been with the service for a while but since we didn’t enforce them we have opened ourselves and our partners to abuse.

One of our long standing policies is to have a 1-to-1 mapping of all valid e-mail addresses. Every e-mail address on your server that receives email must be on ExchangeDefender as well. This allows us to lock down your server to valid recipients only and thereby reduce a lot of traffic that really shouldn’t exist in the first place.

Going forward, all new ExchangeDefender accounts will be locked down to valid recipients only. Messages sent to non-existent accounts will be rejected with the following error:  “550 ExchangeDefender does not protect this email address (directory harvesting attack rejected)”. If a valid recipient notices this error, make sure you add their account to ExchangeDefender.

Every 20th of the month a new accounts summary will be sent to you so you can adjust your billing for the new accounts that have been created that month. This will allow us all to be in sync with the accounting.

New domains will have their activations provisioned and created within four hours (previously 24). If you choose to do your activations yourself please keep in mind that your ExchangeDefender deployment will be locked down immediately, so do not point the MX record until you have created your users accounts.

In ExchangeDefender 4.0 slated for 2nd Q of 2008, the activations will be immediate and we are also working on a server agent that will keep the server in sync with the ExchangeDefender network eliminating double data entry.

ExchangeDefender 4.0: Scalable LiveArchive

To say that LiveArchive has been a minor part of ExchangeDefender would be a tremendous understatement.

So with ExchangeDefender 4.0, we are taking our business continuity one step further.

Currently, LiveArchive is hardcoded to keep only last seven days of your email on a continuous basis. Unfortunately, business disasters do not come in nice friendly intervals, they can be extended, unpredictable, painful.

With ExchangeDefender 4.0 (scheduled for April/May timeframe) we will be providing LiveArchive that can scale up to 30 days, free of charge! While the default will still be 7 days, you can bump that time frame up and down depending on the needs of your client.

Note: The feature set of ExchangeDefender 4.0 is still forming. If you can think of something that would make ExchangeDefender more valuable to your organization I hope you take a moment to mention it to us. ExchangeDefender is built on user feedback.

Do spammers work business hours only?

One of the frequent comments we get from our users is the apparent sudden increase in SPAM between 9 AM and 7 PM. Overnight, hardly anything comes through but during the daylight hours the spammers seem to get going along with the rest of us!

Yes, yes they do. But not in the way you would imagine.

Most SPAM today originates from workstations, office computers, home computers, etc. There are many studies on the Internet that put the level of systems compromised by a worm or a virus at 25%. That means that one in four computers in use is being abused to send junk mail.

When the workers get to the office and power on their systems, they also power on the SPAM amplifiers that hackers have turned those computers into. With more companies going green and mandating computer shutdowns outside of regular business hours, we see a bigger trend in the SPAM activity start and end times.

This also contributes to delays and deferrals during the business hours. Because networks lay practically dormant overnight, as the millions of computers (“spam zombies”) come online, the mail servers are hit with a huge load that is amplified further by all the “opt in, confirm your email” systems and so on. These tend to overload the mail servers and cause huge delays and disconnects all over the place.

This in part is why we only troubleshoot issues during business hours. After hours everything appears to run correctly because there is significantly less load placed on the network. However, those tasks during business hours can quickly identify a host that is overloaded and not taking any mail.