The New Soul of ExchangeDefender
At times in software development it becomes easier to make little optimizations to the process so long as the processing power becomes more affordable and available. Then one day you look up from what you’ve built on your monitor and see a wall of lights, indicators, pie charts and trends that use pretty pictures and colors to paint the portrait of the complexity that manage all the “little” issues you didn’t fix along the way. We are happy to report that after working around the clock for weeks we have regained a lot of control and a wall!
The story of ExchangeDefender is pretty simple. In the 90’s, we could not keep our Exchange servers up, fighting with relaying, viruses, hackers and platform instability became a daily frustration. So we figured, “Let’s put a box in front of it to just scan the junk and pass on valid stuff to Exchange” and it worked. The little AMD K5-100 workstation with an IDE hard drive and McAfee virus scanner did enough to keep our Exchange infrastructure running. Fast forward 12 years, the rise of SPAM, the rise of Malware, the pdf spam, the composite jpg spam, the rise of botnets and we have the ExchangeDefender of today, a 2,800 server network spread over 26 data centers and a full shift dedicated to just racking the new hardware and provisioning new nodes as our company and our load grow.
With data center space starting to shrink, power demands growing, power cost growing and the trend of SPAM increases turning vertical, we had a choice: change the way we manage SPAM flow or start breeding hamsters to generate power. Since nobody volunteered to clean the cage, we changed the way content is passed through and into our network.
The Old Way
Messages were accepted and scanned in the order they arrived with no bias shown for RBLs, origin or content. We took the message, checked it, and if it was clean we delivered it. With the SPAM loads constantly over the 95% range, legitimate messages had a 95% chance of being scanned with less urgency than their SPAM counterparts. To further complicate the issue, we scanned all messages equally: SPAM checks, heuristics, viruses, RBL presence, message integrity, the works. This created an immense demand for processing power, storage and network capacity.
Simply put, imagine a window of one second during which 10 messages were received and scanned sequentially from first to last. If the clean message was message #1, no worries. But if it was #10 in the scanning order it would have to wait for us to check for every minor chance that the nine messages before it had content anywhere from a pharmaceutical ad to banking fraud.
The New Way
We first changed the way we rely on our extensive knowledge of SPAM sources and senders and we created a system in which a bias is placed on messages coming from reputable senders. RBLs and statistical models are not perfect, that is why we will never bounce or delete an incoming messages without 100% certainty. Legitimate organizations can end up on RBLs from time to time, etc. But if we received over a million SPAM messages from a host in China with no reverse DNS, why should we treat the message #1,000,0001 from the same IP address with the same priority as the messages from hosts that constantly deliver legitimate mail that our users want to read? The good news is, we no longer do because messages are prioritized well before they hit the beef of ExchangeDefender.
Over the past month we have implemented a new SPAM filtering traffic shaping process that takes messages from known SPAM sources and puts them into a low priority queue to be scanned with far less of our arsenal than the messages from sources that are not as proven of SPAM heavens. So if you’ve mismanaged your infrastructure enough to end up in SpamCop, SpamHaus, SORBS, have an open relay, no reverse DNS the only thing we need to know is if the recipient trusts you. If they don’t, store/drop/delete, if they do, we scan for viruses, complex checks, custom filters and more.
This system allows us to enhance our scrubbing of unknown mail and keep you safe from more innovative threats while keeping SPAM stored just in case. Messages will still be accessible, still scanned. This new process shifts the burden of mail server management on the sender to keep their infrastructure in check while improving the security of our recipient customer. The era of trust on the Internet is long gone, mail servers that are mismanaged to the extent that they spew millions of SPAM messages, viruses and malware cannot be trusted not to have their logs scanned and malware routes replicated through the social means of replaying legitimate mail traffic to bypass filtering systems.
The Story So Far
So far, we have been able to improve nearly all tracked metrics of ExchangeDefender.
The level of SPAM detected is way up. The level of false positives (messages ExchangeDefender thought were SPAM but were released from quarantines by end users) is way, way down. The latency in delivery, scanning and processing are way down. Nearly everything we do has significantly improved over the past few weeks and while it took an enormous amount of hard work we feel it leaves the Internet a safer place for our clients and a far less profitable or even viable business for spammers.
Results
We have done our way to make sure the messages you receive get to you faster, safer while reducing the time needed to manage whitelists, review junk folders or worry about non-receipts. Our support has reflected this as well, despite February being the strongest sales month on record for ExchangeDefender, ever, we have had less support requests, ever!
Now, after a little deserved break, we turn to ExchangeDefender 4.x and think of ways we can use the new savings to make our service work for you even better.
As always, thank you for your business and thank you for trusting us to keep your inbox clean.