Learning from Experience…
The past two weeks have been rather stressful and unpleasant for our partners and your end users on DEWEY due to the recent outage that left 17% of the users without full access to their entire mailbox for two weeks. Not only has this been our longest outage with any service, but it also has been one of the most “bumpy road” recoveries we’ve ever experienced. Our partners and their end users could compare almost every step of the recovery to pulling teeth; every step of the way, we let them down.
We’re extremely sorry…this was (and still is) extremely unfortunate, but we have learned a lot from this experience.
Here’s a quick break down of the experience:
10/31/11 – Users on two databases both lost connection to Exchange.
11/01/11 – Users are switched to dial tone recovery mode. Users with mobile phones lose contacts and calendar events on next sync.
11/06/11 – Users on DB2 regain access to old mail, but lose access to any new mail, contacts, or calendar entries since the outage began.
11/07/11 – Mailbox data for users on DB2 is fully restored; new and old data merged.
11/17/11 – Users on DB3 regain access to old mail, but lose access to any new mail, contacts, or calendar entries since the outage began.
11/19/11 – Mailbox data for users on DB3 is fully restored; new and old data merged.
In speaking to partners along every step of the way we heard every issue experienced by end users with the biggest issue being the dial tone recovery. During dial tone recovery any users with ActiveSync based connections will lose all Exchange contacts and calendar items on next sync after dial tone is activated.
When you break down our responsibilities and duties to our clients, at the very minimum, we need to provide a live running service as quickly as possible. Technology and software unfortunately have issues and can break, but as long as we can minimalize the direct impact faced by end users we can generally get through issues without upsetting a lot of partners. By erasing all cached access by mobile phones with the temporary mailbox, we put a lot of stress on our partners.
We are considering a disaster recovery policy control for partners..
The policy controls will give partners control of how we treat each mailbox during an outage. In giving partners the ability to control how we respond, this will greatly improve the overall experience during an outage and will allow the partner to provide direct expectations to their clients.
For instance, say partner ABC123 Computers has a client, Big Electric Company, with 10 mailboxes.
The partner wants to mark 3 users as “Do not activate dial tone” as these users are mobile and depend on their contacts and calendars and they cannot afford to lose them once dial tone is activated.
The partner then marks the CEO and CFO as “Do not reimport data after outage” as the partner plans to reimport cached data because the CFO and CEO cannot work on dial tone mail alone.
By allowing the partner to directly control our recovery process for mailboxes, the partner will then be able to set direct SLA expectations for their end users during the outage.
Plans for the future…
In the event that we experience another catastrophic failure (anticipated 6+ hours of downtime), we will wait at least one hour before activating dial tone recovery for mailboxes that are not opted out.
During the first hour, we will reach out to all partners via telephone whose mailboxes are not opted out from dial tone recovery to make them aware of the expected experience for end users. If the partner wishes to not activate dial tone recovery we will activate the mailbox option in service manager to opt out of dial tone recovery. Additionally, partners could ask (via support ticket) that the dial tone activation to be postponed for a few hours later if the partner wants to advise the end user to disable ActiveSync.
Once the original (or backup) database is back online we will once again reach out to all partners who have not opted out of mailbox data restore to let them know of the expected experience for end users. If the partner wishes to opt the mailbox out of data restore then the mailbox will remain on the dial tone database until the recovery of data for all users on the affected database is completed.
If you think the outage policy control would be a beneficial add on to our Hosted Exchange service, please let me know travis@exchangedefender.com.
Travis Sheldon
VP Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com
Happy Thanksgiving!
From all of us at ExchangeDefender, we wish you a very Happy Thanksgiving!
Sincerely,
The ExchangeDefender Team 
Black Friday!
Black Friday is one of the, if not THE, biggest shopping day of the year! Take advantage of this and shop for yourself! We are running a one-day promotion for Compliance Archiving for only $1.00/user. This is a promotion is solely for partners only, and not for resale.
Are you curious as to what Compliance Archiving is all about? What does it include? How do I use it? What are the benefits?
Check out the Compliance Archiving page of our website, http://exchangedefender.com/features_compliance_archiving.php.
This is a great offering that you will want to take advantage of! Simply enter the coupon code “XDBlackFriday” when ordering Compliance Archive and you will receive the great $1.00 price point for all of the accounts that you order, and they will stay grandfathered in at the rate for the duration of the account! This promotion is ONLY valid on Black Friday, 11/25/2011, so don’t hesitate!
If you have any questions please feel free to contact me!
Stephanie Hasenour
VP Marketing, ExchangeDefender
stephanie@ownwebnow.com
(877) 546-0316 x735
Trusting Senders and Whitelisting
In the coming weeks I’m going to be taking a, “Did you know?” approach to my blog as I keep running across items that are offered in our feature set that folks find surprising that we offer that would have saved them time had they known. These items are part of ExchangeDefender that folks don’t come across often but they are there. The first topic we’ll touch are White listing and trusting senders.
One of the more common inquiries that arise out of the Trusted Sender is why does it seem to not work consistently. Here’s the logic behind the Trust Sender function in ExchangeDefender. Trust Sender will in essence white list that specific sender address to that specific recipient address. The times when folks over look these details and it turns into a problem are as follows:
Domain administrators often “Trust Sender” from SPAM Czar under the assumption that the sender is being trusted for the domain, the feature set has not changed it will still white list that sender to that specific recipient.
Senders that use disposable email addresses like carlos+is+cool+3425223@exchangedefender.com work around the mechanism, since the from address is always different, so this type of sender is better suited for a domain level whitelist.
That takes us to the options for whitelist formatting. ExchangeDefender will access the following types of entries:
user@domain.com Basic per user whitelist
domain.com Domain level whitelist
a.domain.com Third level domain whitelist
123.123.123.123 IP address whitelist
123.123.123 /24 whitelist
All this information can actually be found in our Knowledge Base inside of our support portal.
Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737
The Power of an All-In Solution
Hosted email can be very simple – pay one flat price per month per mailbox for a set of great services that meets all the client’s needs. Yet some providers choose to make it very complicated – just determining what the per-mailbox cost will be can seem like solving an algebra problem. Hidden add-ons and fees abound, and add up very quickly to costs much higher than anticipated. When anything changes, like storage needs or mailboxes or additional services, account changes can be complicated, may affect the price dramatically (which wreaks havoc on your profitability), and can be difficult to manage.
At ExchangeDefender, our Hosted Exchange 2010 + SharePoint 2010 offering makes it very simple. For one flat, all-inclusive rate per mailbox per month, you can offer your clients a hosted messaging solution that is flexible enough for their varying needs today and tomorrow, and robust enough to enable you to turn a profit on every mailbox every month.
“Under promise and over deliver”
Let’s say your client, like so many small businesses, indicates in the sales conversations that they “just need a basic email solution” that will work and keep them safe and secure. Well what if you could offer that to them, at a great price, with all the features that they could ever want to “add on” later built right in and ready to go when that time comes?
ExchangeDefender Hosted Exchange 2010 + SharePoint 2010 solution comes with full ExchangeDefender protection, an Outlook desktop license, Live Archive service, web filtering, web file sharing, and even more great features built right in. If your clients don’t need or want any of them, you can simply disable them. You can deliver that fully-customized functionality with just the click of a button in your management portal. There’s no migration, no additional provisioning or payments, no updates. It’s just that simple, for you and your clients.
We offer ExchangeDefender Hosted Exchange 2010 + SharePoint 2010 to our partners for $10 per mailbox per month, with additional discounts available based on volume. You get to mark this up and charge your client at your discretion. We typically see the service being resold for $15-20 per mailbox per month, providing a nice healthy margin to the MSP partner. And the price includes everything, with no fine print or hidden catches. $15 a month for a full-scale, customizable Microsoft Exchange email service sounds very appealing.
It’s easy to manage, so you’ll spend a little less time managing each client’s changing email and messaging needs, which adds up to big impact on the bottom line. Remember that when the fee is fixed, the goal is to minimize your internal costs for managing and delivering each service to each account, to be most profitable.
Do you find this information useful?
If you’d like a lot more in-depth discussion about the cloud and how it affects you and your clients, visit Looks Cloudy http://www.lookscloudy.com where I blog daily about the adoption of the cloud in SMB, conduct live webcasts and podcasts with industry leaders and more.
Kate Hunt
VP Community Development, ExchangeDefender
kate@ownwebnow.com
(877) 546-0316 x777
Shockey Monkey Reloaded
This week the development team has been focused on cleaning up Shockey Monkey. This process is mainly composed of three focused categories (UI, Performance, and Bug/Warning Fixes). I would say this has been the largest update to Shockey Monkey since it was originally created over seven years ago. This update should also help lay the framework for any future additions or tweaks.
User Interface
– We’ve applied a consistent look and feel across the entire portal.
– Implemented a themed backend that allows quick and easy branding.
– Improved cross browser support
Performance
– Improved loading performance across the portal.
– Minified most of the JavaScript files to reduce fetch & loading times.
– Implemented more session variables to reduce the amount of queries.
Bug Fixes
– Decreased Disk IO access by resolving more than 1000+ warnings that were constantly being written to a local log file.
– We’ve fixed a handful of bug over the past few weeks and have been continuing to resolve these issues as we enhance the portal.
– If you have a bug, submit it here: https://support.ownwebnow.com/create.asp?item=development
Performance is being improved in several ways across the Shockey Monkey portals. The first thing we did was minified thousands and thousands of lines of JavaScript. The compression ratio on average across all of our scripts was about 65%, which is very good. We’ve also been fixing trivial “warning” messages as we progress through the code. By resolving these warning messages that are generated on the back end, it reducing the amount of disk IO that is required to store this information. Typically this is not a big deal, but multiply that by 1000’s of warning and you get some nasty seek times.
If we go a little bit into the cleanup of the User Interface, it was a project that was mainly focused on consistency. We are also making sure that the look and feel between browser types is very similar. There are also a few special features of the new UI that will really help bring the User experience to a new level. One of these features is the ability to view any page in “Full Screen” mode. What this does is remove the top and side menus from the portal. This allows you to manage any page without the annoyance of having to resize or scroll to achieve simple tasks.
We have a lot more information and a ton more features that we are saving for the unveiling. If you would like to be a part of the Shockey Monkey Reloaded webinar on Thursday, December 1, 2011 12:00 PM – 1:00 PM EST, visit the following link: https://www1.gotomeeting.com/register/812869640.
Hank Newman
VP Development, ExchangeDefender
hank@ownwebnow.com
Exchange 2007: Leaving Old Technology in the Past
A few weeks ago (10/31/11) our DEWEY cluster suffered a major outage at the result of old conflicting RAID policies that led to the dirty dismount of 50% of the DEWEY databases. Immediately, our team started the recovery process, which unfortunately took longer than ever expected (More details @ https://www.exchangedefender.com/noc) and prompted our side to start developing a new solution for DEWEY users to restore and maintain the expected service availability and reliability.
Lets be forthright…DEWEY has been somewhat of a nuisance not only to our staff, but also to our partners and your users.
Overall, Exchange 2007 was a drastic leap and improvement from Exchange 2003, but still lacked many features for service providers or any rapidly expanding company such as the ability to properly scale the solution and implement additional failovers after initial deployment. In comparison, I think of Exchange 2007 as Windows Vista – it brought a lot of new features and easier management from 2003, but after a year of management it feels like a bad tease.
Over a year ago, we decided to immediately start offering Exchange 2010 as soon as it RTM because we felt we hit the ceiling in terms of scalability with Exchange 2007. Sure, we could add more CAS servers and MBOX servers, but that would eventually end up with partners having multiple configurations to keep track of for clients in the same company. Taking everything into account Exchange 2007 was no longer the road we traveled down for new deployments.
Going back to DEWEY, we started experiencing more frequent issues in terms of reliability over the past six months. Between services failing to respond in a timely manner or even worse, not responding at all we knew we had to provide a solution before our partners clients threw up their hands to say “I’m done!”.
Upgrade, Upgrade, Upgrade
In order to provide a solution we must keep a few essential components in mind:
1. The upgrade must be seamless with no mail loss and no major service interruption.
2. The upgrade must be automated. We can’t expect partners to import/export data or reconfigure Outlook.
3. The upgrade must be scheduled so partners are kept in the loop on the migration progress.
4. The upgrade must be flexible for partners who want to slowly migrate or for partners who want to immediately move.
Unfortunately, Exchange 2007 does not provide the ability to seamlessly move users directly to any remote Exchange 2010 cluster (i.e. Rockerduck), but we can upgrade to a local 2010 cluster, then move the client to ROCKERDUCK.
The Solutions is Here…
In the past few weeks we have deployed and successfully tested a 40 user migration from 2007 DEWEY to 2010 ROCKERDUCK with very minimal effort on the partner’s end. Below is a rough overview of the process which was designed for a 2 week migration for clients:
1. Partner adjusts autodiscover CNAME record to NEWDEWEY (2010).
2. Move requests get created from 2007 DEWEY to 2010 DEWEY (Week 1, Monday).
3. Users on DEWEY will remain on 2007 throughout the work week.
4. Move requests will be finalized on Friday evening. Users will now be serviced on 2010 DEWEY (Week 1, Friday).
a. Users with BES devices will remain on 2007 until the final move to ROCKERDUCK.
5. Users come in on Monday, restart Outlook.
6. Move requests to ROCKERDUCK are created (Week 2, Wednesday).
7. Partner adjusts autodiscover record to point to ROCKERDUCK (Week 2, Friday at close of business).
8. Move requests will be finalized Saturday evening (Week 2, Saturday).
9. User comes in on Monday, selects “Repair account” on ‘Email Accounts’ list. User is now serviced on ROCKERDUCK.
10. Users re-setup ActiveSync and any Blackberry Enterprise Server accounts.
With the above outlined process users should not see any service interruption nor will users have to recreate their Outlook profiles.
Travis Sheldon
VP Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com
ExchangeDefender University
Whether you are a current partner of ExchangeDefender or just interested in working with us, ExchangeDefender University is a great resource for you to use in order to have a successful partnership with ExchangeDefender.
During the rebranding and rebuilding of the new ExchangeDefender site, we have added all kinds of new information to ExchangeDefender University. We have a Training section where we have all of our training videos and deployment guides available to assist you with the set-up and implementation of our services. There is a section titled Start-up Guides which includes deployment guides and corresponding instructional videos for some of our most popular services. There is also a Partner Resource section that includes everything that you will need to be profitable with ExchangeDefender Included in Partner resources are links to the following:
· Marketing Collateral
· Current Promotions and Special Offers
· Profitability Best Practices
· Frequently Asked Questions
· Upcoming Events & Webinars
· Partner Testimonies
There are also links to all of our other websites, like the Support Portal, our Corporate ExchangeDefender Blog, the Network Operations (NOC) Blog, and our Knowledge Base.
In addition to all of that, there is a step-by-step walk through process of how to successfully work with ExchangeDefender. It runs through a Portal Tour, Account Management, Placing an Order, Managing Services, and how to go about Getting Help. Just click on the big blue arrow to get started. When you are finished there is a link in the last step that will bring you back to the ExchangeDefender University homepage.
Visit http://www.exchangedefender.com/XDUniversity.php!
ExchangeDefender University is a great tool to utilize – and a great site to bookmark and refer to! Please take advantage of it – it is there to help you every step of the way! And of course, if you have questions…don’t hesitate to Contact Us!
Stephanie Hasenour
VP Marketing, ExchangeDefender
stephanie@ownwebnow.com
(877) 546-0316 x735
Shockey Monkey Media & Partner Briefing
Shockey Monkey will be receiving a massive refresh in both features and the business model in early December. Throughout 2011 we have been adding new features, new integrations and have entered several partnerships that just begin to showcase what is possible with Shockey Monkey. With that, we are about to revise the pricing, positioning, marketing and access to Shockey Monkey and want our partners to be aware of the changes that are coming.
Shockey Monkey Media & Partner Briefing
Tuesday, November 22nd. Noon EST.
Contact us for an invitation to the webinar.
Due to the sensitive nature of the information we will be sharing, this event is open by invitation only to vendors in the MSP space and media sites/blogs that cover them. We will be discussing:
– New pricing models
– New support procedures
– Sponsorship options
– Breakdown of user metrics
– Integration overview
– Usage patterns and value
If you’re interested in attending let us know, we are on pace to launch 2.0 in the first week of December. At that time we will be holding a user-facing webinar to announce all the cool new stuff that we hope is a real game changer for our users as well as for our partners.
Sincerely,
Vlad Mazek, MCSE
CEO, Own Web Now Corp
vlad@ownwebnow.com
(877) 546-0316 x500
(407) 536-VLAD
How to WIN During an Outage
Don’t get me wrong, by win, I mean limit the bleeding to as minimal as possible. Let’s face it, nobody wins during an outage, everybody gets yelled at by someone else for the entire duration and thereafter. Knowledge of technology goes out the window and gives way to frustration and anger at other folks. I’m going to keep my opinions out of the post and just recap the some of the “best” solutions we saw come from partners. And I’ll additional preface this by saying, this isn’t meant to imply what could have been done, because it’s not my place to say that, period. But these were some things that I saw executed that worked for partners and their clients and I figured I would share.
This list isn’t in any sort of order…
Plan A – LiveArchive – Contact Export Combo
This was pretty impressive to me, out of the gate. I had a partner that rolled their clients to LiveArchive and they actually had links on their desktop to do this, like we recommend. The first person to pull this off, had it the easiest for sure. Most smartphones support Contact exports from ActiveSync to the phone SD Card. So he was able to have them have their email via LiveArchive and still not have their contacts disappears once we went to the Dial tone migration route.
Once we enabled that, we had another partner that had the same idea but a day too late, so he had to go fetch the contacts from the old OST to put them into the Temp Mailbox to facilitate the process above. Then he had to move the contacts under a different contacts folder to avoid duplication once we merged the data back.
Plan B – Migrate to Rockerduck, like yesterday!
We had a few of those that basically saw a pattern of hardware issues and limitations on the Exchange 2007 software and decided they’d had enough and they moved to Rockerduck on day 1. Yes, they took a hit on the man hours required to export then upload PSTs, but this group of folks were done with the outage as soon as those PSTs finished uploading.
Plan C – Find an ExchangeDefender employee and hire some people to beat them with bats on sight.
I was hoping they would find Vlad, but that didn’t happen. Obviously, there are great variables that prohibit or make it more difficult for folks to try do items A & B, and I understand there is a cost in labor that was assumed at the time of deployment and it is hard to take that hit again. We understand and we have a plan in place to migrate all users that do not opt out to 2010 via online moves. This is going to take some voodoo work to make it seamless but our Exchange team has a plan designed to make it happen. Unfortunately, some of the prep work caused some extra authentication windows to pop up for already irritated end users, but this is one of those “one step backwards to go to two steps forward” scenarios.
Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737