January 2007

We have been fielding numerous reports from customers complaining that their email has simply disappeared over the last few days. Either mail sent directly to them, or quarantine reports or anything suspicious.

Over the past two days we have been able to identify two causes of this on target Windows 2003 servers running Trend Micro or IMF. As you may be aware, Exchange team had a brief pause in IMF updates (few weeks) and the most recent update that was pushed out is far more aggressive than the previous ones. It is likely that if you do have IMF installed it is either archiving or deleting messages. In Trend’s case, messages are simply disappearing.

To identify if this issue is affecting you please turn on message tracking and try to search for messages you know you should have received. You will likely see “Message sent to categorizer” at which point there is no further information on the message. With IMF you will see a report by IMF indicating that the message was either deleted or archived by IMF.

These issues are totally unrelated to ExchangeDefender but they will impact the performance of clients that use both for protection. As always, please uninstall/remove/disable mail hygene products to troubleshoot the core of the problem. We have facilities to help you find out if ExchangeDefender is at fault, if you suspect it please call (877) 546–0316

Dear ExchangeDefender User, Partner, Friend:

Today marks the completion of all the changes and improvements we have been making to the core infrastructure of ExchangeDefender. Thank you so much for putting up with us over the past three months as we’ve virtually rewritten what ExchangeDefender is and how it works. Over the next 30 days we will be enabling all these features will become manageable – we need your feedback. Your patience and your loyalty will not go without notice, if you scroll down you’ll see we’re giving a very nice present to all of our customers and partners.

The purpose of this email is to inform you of the changes to ExchangeDefender and how they will be implemented over the next 30 days. These improvements fit into four categories and were made 100% based on user  and partner feedback: user experience, administrative functionality, centralized (MSP) control and core infrastructure.

 

User Experience

The user experience of ExchangeDefender will be changing only slightly because we do recognize how sensitive non-IT users are to interface changes. The behavior, location and use of ExchangeDefender will not change to the point that they will require re-training.

– Whitelists, policies and trusts are now implemented in realtime via SQL

– Ability to reassign quarantine management to another user

– Ability to request a resend of daily report

– Ability to search the quarantine

– Ability to mark reviewed messages so only unread counts show

– Ability to receive multiple daily reports

– Ability to adjust the time zone so the times in quarantines and reports reflect users local region

– Ability to mass-release multiple messages from the quarantine

– Ability to view messages over the past 7 days in the Live Quarantine even if the Exchange server is down

– Paging support to limit the number of messages shown on the screen

 

Administrative Functionality

Administrative functionality will allow the domain administrator to designate a “SPAM Czar” who can be in charge of reviewing the organizations quarantines. Administrator will also have access to make all the changes necessary in realtime. Additionally, the administrator will be:

– Able to add users from the ExchangeDefender control panel

– Resend welcome/activation messages

– Reset passwords, policies, settings for a particular user

– Access all quarantines, messages, archives for a particular user

– Test delivery with internal, external and remote facilities to help troubleshoot mail delivery times

– Ability to establish file level attachment policy settings (still no .exe’s sorry; but you can rename them)

– Statistical breakdown of daily attack profiles and SPAM counts

– Ability to audit outgoing mail and enforce outbound policies

 

Centralized (MSP) Control

Perhaps the most demanded feature set by a long shot. ExchangeDefender was not originally designed to be reseller friendly, it was designed to protect users from exploits and time consuming SPAM on our own servers. As the product has grown so has its reseller base and we’re providing the management interface you’ve asked us for:

– Ability to self-provision accounts by providing domain, ip, policies and user dumps

– Ability to consolidate entire ExchangeDefender network management under one screen

– Ability to receive realtime billing, user count and statistical breakdowns

– Ability to access any information available for every other screen (user & administrator)

– Ability to provide executive reports

– Ability to retrieve data in realtime for internal reporting systems (web service for Kaseya, LPI, Connectwise integration)

– Branding support for the portal, daily report emails and activation messages.

 

Core Infrastructure & Training

The most significant changes to the core infrastructure is the way we manage, deploy and support the network. In the past ExchangeDefender nodes had local configuration databases that were generated on-the-fly and loaded at preset intervals. The new ExchangeDefender v3 setup is directly driven by a SQL database and all policies, settings and services are driven by a distributed database.

The most significant improvement is the addition of Live Archive. The live archive gives your users the ability to access their email through a web based interface hosted at Own Web Now Corp – even while the target mail server may be down. This radically transforms ExchangeDefender from a protection system to a business continuity tool. The messages are spooled both on the ExchangeDefender node and a Live Archive system. This way if the server goes down your users will be able to access mail through a secure (and highly auditable) webmail client and the original message will still sit in the queue waiting for delivery to the target mail server. When that mail server does come up the messages will be delivered but your users would have had the option of continuing to write, read and respond to email while you were down preserving company identity and communication channel.

The best part of this is that even though the pricing for ExchangeDefender will be changing, our current active partners will get this feature free of charge and will be grandfathered at the current pricing model even for the future clients that are added to the service.

 

Changes

All of these changes are influenced by our partners and end users. As such, we want to make sure that we have taken your feedback in account properly and that the changes are implemented in a manner that is most beneficial to our users. As such, we will be deploying one feature at a time, allowing you to use it for a few days, taking feedback and adjusting the behavior and look of the system for everyone’s benefit. Later tonight you will receive an invitation to participate in the design of ExchangeDefender v3, we welcome your feedback. We expect the entire deployment to be complete in less than 30 days as 99% of the code is already written, we just want to make sure it does what you need it to do.

Additionally, we will be investing in the training resources for ExchangeDefender. While half of the security can easily be done by ExchangeDefender alone, the other half consists of properly training the user and the administrator. We will be releasing series of white papers and web video lessons that will help people get started with ExchangeDefender. This will help free you time and make ExchangeDefender deployments less stressful. 

In closing, ExchangeDefender v3 is a drastically different product from v2. It is perhaps the most ambitious effort to provide SMB with a reliable and distributed communication system that leaves the end user in control while providing the security blanket of a large network. I thank you for all your patience with us, thank you for all your money, and hope we continue to deliver the services you’ve come to expect from us.

Sincerely,

Vlad Mazek, MCSE

Yesterday (~10PM GMT) we rolled out our new whitelist and blacklist functionality. The new functions allow ExchangeDefender to apply whitelist rules in realtime and minimize the chances of a false positive.

While the entire ExchangeDefender network is SQL driven much of the functionality was not transaction based – we simply regenerated node configuration via SQL which allowed for quick provisioning and central management. Unfortunately, with as large and as diverse of a network as we operate, that also introduced latency in rule updates and gave us a hard time troubleshooting.

So what’s changed? Well, every time you send an email somewhere it is temporarily stored in a SQL database as a reverse whitelist entry. What this means is that we automatically whitelist the people you send mail to. When they respond, they will get through without being put through the extensive SPAM filtering process ExchangeDefender uses. And because the process is SQL driven that recipient is “trusted” in near realtime.

I hope you like this feature.

Note: It took a better part of overnight to switch over to the new process. As result, some previously trusted messages (ie: Yahoo Groups) that are on several blacklists would have ended up in your SPAM quarantine. Our global trusts for our partners (Yahoo, AOL, MSN, Hotmail) were programmed in overnight so all should be well now. If its not, please email support@ownwebnow.com and we’ll address it immediately.

Over the past week we have been working very hard at expanding our storage network capacity. The demand for our new off-site backup system has been far greater than we ever imagined so we had to grow our network significantly to accomodate all of our partners that wanted to deploy this service for their customers. Who can blame them, we use the same system to backup all of the servers we manage as well.

So what’s new? Additional network capacity and additional storage. Newer version of AhSay. Improved replication system. The amount of work that went into putting this together was significant, we have dedicated over 5 days to it.

How does it work?

Each client (workstation or server) installs a small branded agent on the PC that is being backed up. The agent runs on Windows, Linux, Mac OS X, Solaris and Novell. The agent backs data up to the server on the byte level, this way only changes within the file are propagated instead of the entire file each time there is a change. The client keeps the information on the backup schedule and retention policies (how long a file should be available for restore) and is capable of backing up Microsoft SQL Server, Microsoft Exchange (both on store level and optionally mailbox brick level down to the message or folder), Volume Shadow Copy, Oracle, MySQL, etc.

The infrastructure, frankly, is impressive. The main AhSay OBS server runs on the perimeter of our Dallas/InfoMart network and receives/sends data from the client agent. The entire server is backed up daily to a local SAN for quick disaster recovery purposes (in case of RAID failure). The entire server is also replicated in realtime to a geographically redundant data center in Chicago/CBOT using AhSay replication server. We know how important your data is so we do our best to protect it as best as we can, 3–way replication with 2 of them in sync at all times.

We’d like to thank our customers for their patience during this process. If you were on trial and experienced any issues please contact us and we will extend your backup trial by additional 30 days.

The partner response to our new data protection service has been simply incredible. As a result the amount of bandwidth and storage we are in process of allocating to it is equally enterprising.

We are expanding SAN and network allocation to our public AhSay servers. As a result you will likely experience some performance issues until the new storage systems are in place and the additional bandwidth is brought online. This will not affect your daily backup processes dramatically as the AhSay agent is set to automatically retry/resume as availability changes.

We expect the work on our storage networks to be complete by early Friday AM, EST; 9/26/2007. We’re sorry about any inconvenience this may cause you.

ExchangeDefender has enjoyed a very good 2006. As a result of such phenomenal growth, we’re pleased to announce that we are adding two more data centers to our network. We are adding two network locations in Los Angeles, CA and Newark, NJ. These facilities are expected to nearly double the size of the current network by the end of 2007.

If you are currently restricting network access on port 25 to Own Web Now Corp IP address ranges you will have to add two more blocks to the allow list:

64.182.164.0/24 (NJ)
64.182.133.0/24 (CA)

These network ranges are Class C address spaces so you can expect a connection from any host, 1-255. We will start relaying from these locations on Monday, January 15th, 2007.

Earlier today we have made some significant changes to the tar-pitting mechanism under ExchangeDefender. The new mechanism is designed to reject messages from hosts that do not follow the proper RFC SMTP dialog and attempt to smash tar-pitting. More on the basic concept of tar-pitting is described here.

The Problem

While tar-pitting is great for throttling remote mail servers and reducing their ability to efficiently deliver a lot of messages, the concept only applies against botnet servers that are attempting to deliver mail in bulk. Anotherwords, tar-pitting is only effective against servers that are concerned about getting the message out as fast as possible. By delaying the SMTP greeting banner, in theory, the remote mail server would have to wait a pre-determined amount of time before starting to send mail. Many open connections at once would overload a single node.

However, spammers no longer exclusively use single nodes in a full force attack. They use the botnet concept by load balancing their broadcasts through multiple servers. As such, those servers connect every few minutes and only relay a single message. By doing so its hard to blacklist them immediately because their overall reputation does not have enough data to be determined. These botnets are designed to bypass tar-pitting by opening a connection and sending data as soon as the connection is opened.

The conversation looks somewhat like this:

Trying xx.xx.xx.xx.exchangedefender.com.

Escape character is ‘^]’.
ehlo spamming-idiot.org
mail from: spammer@spammer.org
rcpt to: vlad@ownwebnow.com
data
Subject: Get a college diploma.
Ohio State University may be a loser but they’ll give you a Ph.D in nuclear physics based on your life experience.
.

Now the (target) tar-pitting mail server has accepted the connection but it has never sent the SMTP greeting. However, it will process the message as soon as its tar-pitting interval passes, thus in part bypassing the tar-pitting and delivering the message. Not good.

Notice that the client above did not wait for the 200 greeting banner, did not wait for the 250 Hello, did not wait for the server to acknowledge the recipient and the sender. They just wrote to the socket and waited. Now even though this does diminish the spammers performance a little (by taking 5 seconds to deliver the message) the message still gets delivered. That’s a problem.

The Solution

The solution is fairly simple: Drop connections with mail servers that are not adhering to RFC. The second the mail server issues a command before the 2.2.0 hostname greeting banner it will get dropped, logged and its tar-pitting interval extended.

Instead of a tar-pitting process that delays the connection a few seconds, this process allows for a connection immediately but delays the SMTP greeting banner a few seconds. As such, it can eliminate server load caused by spammers that think they have found a way around tar-pitting.

We ran this in testing on our production systems and have found 0 false positives over the course of one week. All hosts that were rejected were also on multiple RBLs. The implementation is transparent to the user and administrator and introduces a random (less than 5) second pause on all connections that do not have a reputation rating with ExchangeDefender. Less spam, less stuff to review, less bandwidth and less stress for you.