ExchangeDefender SQL Balancer Maintenance Window

This Friday, July 19th, 2013 we will be conducting an upgrade, stress testing and DDoS simulation against ExchangeDefender SQL backend, load balancer backend and the overall infrastructure fault tolerance starting at 3PM and ending at 6PM EST.

During the maintenance window mail flow will not be interrupted and mail delivery will not be delayed. Access to email, LiveArchive, mobile services and so on will not be interrupted either. Access to https://admin.exchangedefender.com site, including the ability to make configuration changes or release SPAM, will be interrupted periodically. Again, no email will be lost or delayed.

Following the upgrade our capacity across the core ExchangeDefender network will be increased which will enable us to launch additional services that will be discussed in the upcoming webinar. We apologize for the inconvenience this causes your clients as they may be presented with the message “We are currently conducting routine maintenance, please try your request again in a few minutes”

Scope of the effort

Isn’t this something you could do outside of the peak hours?

Yes, and we routinely make changes and upgrades during the standard weekend maintenance intervals.

For the purposes of this upgrade and an assessment of the new failover systems brought to the new version of ExchangeDefender, we wanted to run a maintenance interval during the peak usage and ascertain both our ability to handle traffic with the brains of the system offline (ExchangeDefender SQL) as well as test geographic redundancy during peak hours.

We felt it was better to have this done at the same time on a scheduled and controlled circumstances than under the unplanned service outage or DDoS.

As I will discuss next week, we have spent a fair amount of money and time upgrading the network not just to be able to sustain excessive outages but also distributed attacks, hacks and so on. We expect to be able to operate the network in a distributed and autonomous environment should we experience massive changes in Internet routing and availability of global networks. If you’ve been paying attention to the news, international government response to NSA espionage, rise in sophisticated hacking and network exploit activity, the reasons and the manner in which we are executing this maintenance window should be self explanatory.

As always, our staff will be available in higher than usual counts so please count on us.

We will also be providing a play-by-play updates as usual via NOC and our @xdnoc twitter.

Sincerely,
Vlad Mazek
CEO, ExchangeDefender
877-546-0316 x500

How our SPAM detection systems work

ExchangeDefender SPAM protection is a layered process that includes many components which together determine if a message is SPAM or not. While there are too many layers to list and the combination of what we use changes on an hourly basis, we wanted to share some detail of how our process works so you can understand it and explain it if necessary.

RBL – We use several free, commercial and our proprietary realtime blacklists that contain IP addresses of known spammers. These IP addresses typically belong to dialup, cable and DSL/fiber client addresses that shouldn’t be running a mail server, compromised/infected mail servers, unused IP ranges, compromised workstations and devices. These IP addresses account for a bulk of all SPAM mail and most of them will never even make it inside of the ExchangeDefender network.

Reputation blacklist – We use commercial and our proprietary reputation blacklists that indicate how trustworthy the IP address is – have they sent us a lot of SPAM in the past? Are they suddenly sending us hundreds of thousands of messages if they only accounted for 10 in the past month? Do they have legitimate DNS, reverse DNS.

URL blacklist – We use several commercial and our proprietary URL blacklists that identify web site addresses used in previously confirmed SPAM. If you’ve ever received a SPAM message you know that it had an external image or a link to a web site: we look at all web site links in an email and compare it to a list of known SPAM site targets.

Distributed checksums – We use several commercial and proprietary statistics models (warehouses) to determine the likelyhood of bulk mail. Because the only way spammers can distribute mail efficiently is through massive blasts, nearly all message bodies are identical. Each message has a unique signature (an MD5 checksum) that can be compared with other messages and when identical checksums are found it’s more likely that bulk mail is being sent.

Proprietary header checks – We use proprietary header and message checks to determine if the message is a part of an existing conversation between a third party and our own client. We also check if messages have been spoofed or if they have made their way through several gateways, if the language in the message does not match the language of the machine it was sent from, etc.

SPAM keyword and heuristic checks – We use a wide array of SPAM characteristic checks that take into the account the size of the message, subjects, fonts and images. For example, legitimate email typically doesn’t come without a subject or a subject that contains a lot of special characters, it doesn’t come without a persons name in the From line or other “weird stuff”. The “weird stuff” category is so wide and so contextual that it takes most of the time to process.

Now that you are familiar with some of our processes you’re probably getting the idea that SPAM filtering is very similar to the way virus scanning works – we use patterns from known SPAM messages and existing spammers to build a statistical model that tells us if a message is SPAM or not. It also explains why certain messages are delivered instantly while others may take up to a minute or longer to process (delayed header checks, suspicious web site in the email, conversation thread checks, temporary DNS failures or large attachments that require scanning)

We build most of our proprietary infrastructure based on your feedback – we look at the pattern of messages that you release which builds a model that adjusts scores along the way specifically for you – if you release a TON of messages with impotency drugs you are more likely to receive a Viagra message than another user that only releases financial newsletters. This is why your feedback through ExchangeDefender Outlook Addin is so valuable to us. When you hit “Report SPAM” that message is dispatched to us and reviewed by a live human being that generates a scan rule to eliminate that specific message in the future.

ExchangeDefender SPAM engine uses third party scanning engines with realtime data feeds and our own proprietary engine is updated hourly.

That may seem excessive but keep in mind that spammers adapt their message content for each batch – adding a different subject, different web site, different spacing and subjects every few thousand messages. In order for us to keep up the need for both the realtime update and constant reengineering of the SPAM engine itself is cruical to eliminating as much of the annoyance as possible.

How come SPAM messages slip through?

There is no such thing as a “slip through” when it comes to SPAM, all mail that is not sent by a sender on an RBL that passes our virus scanning is considered legitimate until proven otherwise. Tens of thousands of checks later (within a split second) calculate a score that identifies if message is SPAM (90% confidence) or SureSPAM (99.9%). There is no person on the gateway reading each message, the score is assigned by the computer based on the statistical model – so even if you looked at the message and could clearly tell it’s SPAM, the artificial intelligence is not quite there yet.

How come legitimate messages end up in SPAM quarantines?

Bad scanning methodology. For example, say you’re receiving a lot of “lotto” SPAM. Someone at ExchangeDefender may adjust your model to get really aggressive when it encounters the word “lotto” in the subject or headers. It works flawlessly, until you are sent a message by your business partner and the message is also copied to Lisa Lotto. This is clearly an oversimplification but it happens.

How come updates happen every hour and why does it take so long sometimes to make changes?

Most of the implied delay is due to the size and the scope of the ExchangeDefender network – to protect millions of people we have a very large network and very sophisticated layered infrastructure. Smaller updates are done very frequently (within an hour) but are staggered (happening at the different minute of the hour) because during engine reloads individual nodes cannot process inbound mail. If we restarted the engine on every node at the exact same time we’d basically shut the network down for a few seconds every hour.

Some large-scale changes (when we add a plugin or change our model analysis or whitelists or blacklists or new detection procedures) can take on the order of weeks or months. Typically we don’t deploy new software across the whole network – it is done in stages to eliminate anomalies in the deployment. Working with large scale distributed systems is different than managing individual servers, clusters or even networks: Changes to a globally distributed network with linked load balancers (while great for redundancy and service availability) is a challenge that requires careful monitoring and rollout procedures to minimize destabilization across the whole network.

We get hack attempts and DDoS attacks thousands of times an hour – the systems have an automated process for dealing with that activity – so when we roll out new software that makes the network overload quickly (and make it think it’s under attack) we have a big mess on our hands. This is why the rollouts are staggered and have a procedure that is at times excruciatingly long (especially if you’re sitting in the NOC watching the queues and your blood pressure goes up with every uptick in the percent utilization)

How come whitelists don’t always work and I have to whitelist the whole domain?

This is an inconsistent behavior that is reported from time to time and it has to do with the senders email address and client software. ExchangeDefender looks at the From line (the same one you see in Outlook) and whitelists only the senders actual address. This works 99.9999% of the time.

Some senders use impersonalization or sending on behalf of a different email address or use specific taglines in the From address that end up being randomly generated for each email. While Outlook may see message from Vlad Mazek <vlad@ownwebnow.com> the message itself may have come From: vlad+2381ekr259@ownwebnow.com with a fingerprint used to track read receipts.

Long story short, if you have a person that is using these systems or CRM to alter the message for tracking/sales/marketing purposes, you can’t trust their From line for a whitelist, you will have to whitelist their whole domain. Good news is, this is typical for smaller domains and you’ll never be whitelisting all of @aol.com or @gmail.com

How come there are delivery delays?

Our support team can help you with that.

Half the time the issue is related to the DNS and the other half is due to the temporary network or Exchange issues on the client side. Almost all of the tests we have done for our clients fall into these two categories and almost all of the things you can do to minimize delays are outlined in our deployment guide.

In the event that the issue is our fault due to network congestion, filter failure, virus scanner malfunction, DDoS, routing issue, misconfiguration, etc – the issue is noted at http://www.exchangedefender.com/noc – please subscribe to the feed or @xdnoc or post your cell phone in our portal – we text, tweet, blog and (everything except email, for obvious reasons) notify you of any ongoing network issue that might impact service.

So much for tech..

Technology is only a piece of the whole puzzle.

This is going to sound cliché but I absolutely mean it 100%: At ExchangeDefender killing SPAM is our passion. Every piece of junk that passes through our network is unwelcome and we have people here around the clock working on eliminating it from making it’s way to your inbox. It’s a human process and it’s a technology process and the very practical implementation of artificial intelligence that learns and adapts in realtime to fight with SPAM. But just like training a dog takes time, training a computer is a challenge that requires consistency and strict implementation of the process, rules, management and monitoring. With millions and millions of messages passing through our gateways, even a slight insignificant modification can impact SPAM statistics models. It’s not something that’s “broken” that could be “fixed”, it’s just a process of continuous training that we take very seriously and enjoy very much.

Thank you for your business.

Sincerly,
Vlad Mazek, MCSE
CEO, Own Web Now Corp
vlad@ownwebnow.com

Setting Up Your ExchangeDefender Admin Portal

Recently I have been receiving calls from new and existing partners in regards to the process of setting up their ExchangeDefender Admin portals, specifically ExchangeDefender Essentials. It has been a long time since most of you have had to create your Service Provider portal, but with the addition of ExchangeDefender Essentials there has been an influx in the creation of Service Provider portals. So consider this a refresher course!

As many of you know, ExchangeDefender is a very robust product; therefore we provide an entirely separate portal to manage anything ExchangeDefender related. When you sign up to become a partner with ExchangeDefender you are immediately sent credentials to our support portal. The support portal is where you order all of your services, including the initiation of your Service Provider portal.

There are two separate admin portals for ExchangeDefender, one for the full version and one for the Essentials version. If you have not already created each portal, I encourage you to go ahead and create the remaining one or both; the process is simple and takes just a few minutes.

To create the Essentials portal

· Login to your Support Portal

· Click on the Service Manager Tab

· At the very bottom of the page there is an option for ExchangeDefender Essentials, click subscribe.

· Then click the plus sign.

· Next, the Second page of the order form will ask you for a MSP ID; this will be your username, so we suggest using your company name. You will also be asked for a product name, you may choose anything you’d like, some examples include ownwebnow filtering service, ownwebnow email security service, and ownwebnow cloud security solution.

· The next screen will finalize your order and the credentials for https://admin.exchangedefender.com will be sent to you.

· If you haven’t already created a portal for the full version of ExchangeDefender, go back to the support portal, click on the service manager, and on the right hand side there is a box with a list of services, click on New ExchangeDefender SP. You will be asked for the same information, but your MSP IP will have to be a variation of your previous MSP ID. So if you used your company name, you will need to add something additional to it, if you are incorporated I suggest using ownwebnowinc, or ownwebnowcorp, whatever works best for you.

If you have not created your admin portals and you’re looking to sign up for Hosted Exchange, there is no need to create an admin portal, the portal will automatically provision for you once you have ordered your Hosted Exchange accounts.

I hope this clears up any ambiguity, if you have any questions, please feel free to contact me directly at: Anastasia@ownwebnow.com or by phone , 877-546-0316 x739.

Anastasia Wiggins
Partner Communications Manager, ExchangeDefender
(877) 546-0316 x739
Anastasia@ownwebnow.com