Schedule and SLA during Hurricane Irma
As many of you know, ExchangeDefender HQ is in Orlando, FL and as of noon today that office is closed until further notice. As of the latest hurricane projection, Orlando is likely facing the worst case scenario of being in the NE quadrant very close to the eye of the storm that will roll over the county next to us. Best case scenario, we will be returning to our Orlando offices on Tuesday, Septembrer 12th. Our capacity at that point is obviously questionable because roads, infrastructure and staff may be impacted.
ExchangeDefender is a global business and we have made several contingency plans and do not expect any interruption to our services. Although the majority of our staff is in Florida, we don’t have a single piece of computer/network infrastructure here so the show will go on as usual. The SLA for normal support cases will still be within 4 hours, we will still answer support tickets and phone calls. You should not be able to tell the difference for ordinary issues.
Where we expect to have the most significant challenge is in the areas of projects and special “courtesy” assistance that we typically go well above and beyond the call of duty – things like exporting mail, adding accounts by hand, reaching out directly to clients on your behalf, sales engineering and proposal writing, legal advisory services, compliance officer training, marketing collateral design and event support services are highly dependant on manpower which is something we may be lacking. We will still not say no but we do hope you’ll extend us some extra patience as we go on.
I feel it is my responsibility to be honest and up front about our priority process as we go through next week. This is the typical disaster priority schedule our entire staff is briefed and drilled on but with obvious bias towards Florida and the hurricane.
1. Most important priority will be given to the US and Florida government organizations that are coordinating efforts in the disaster zone. Staff is directed to drop literally anything else they may be doing to assist them.
2. Urgent tickets, system outages and performance issues will be treated as a secondary priority.
3. Ordinary support tickets will still be resolved within the SLA 4 hour window.
We understand that any issue impacting our service is a critical issue to those unable to go about their work because of something in our realm of responsibility. We have made numerous contingencies, failover and backup plans and expect to execute the plan accordingly. We do not employ warm bodies and people that do not care – everyone that is a part of a team understads service delivery is a team effort and wants to see everything continue to work for you as flawlessly and transparently as possible. Hopefully this blog post gives you additional insight as to how we will be handling that with regards to Irma. Personally, I understand that many of our partners in Texas are reeling, that many of our partners and clients in Florida/Alabama/Georgia/Carolinas are scared and we have been working unprecedented hours to make sure everyone has their contingency and failover plans implemented, tested and documented. To everyone in the way, we’re in this together. There is no hurricane party or a 5 day weekend for us, it’s all hands on deck with that deck now spanning 4 states. We’ll get through this. We may move a bit slower than usual, but we will get through this.
Sincerely,
Vlad Mazek, CEO.
Some tips on getting ready for a disaster with ExchangeDefender
I held a quick webinar to get all of our partners and clients in the way of Hurricane Irma up to speed and all the contingencies you need to make in advance of the storm. In Florida and Texas we unfortunately get plenty of time to practice and live roll with disasters but if you have new staff, this is something you need to be doing with your clients continuously. If you don’t feel like watching a 12 minute video, here is a brief writeup.
First of all: We’ll be here. 24/7. If you’re in the path of the storm you will likely face severe outages, downtime, lack of reception, eventual lack of battery power and network congestion in a disaster area. Here are things to keep in mind and get on top of as far as we are concerned.
In the event our support portal ever goes offline, the secondary infrastructure in Los Angeles will pick up the load at:
https://support2.ownwebnow.com
In the event of an outage, we probably aren’t the only vendor you need to stay on top of. So if you do social media, you can track our activity and advisories here:
www.fb.com/ExchangeDefender
www.twitter.com/xdnoc
In the event that the net goes down, and it usually does, make sure you can get in touch with us. Our numbers are 877-546-0316 or 407-465-6800
In the event that you lose network connectivity or your LTE/4G suddenly becomes Edge/GPRS, you will likely be calling us for support and network changes. We’ll need your PIN to make any account or configuration changes to make sure you and your techs have that ready now. If you haven’t seen your pin in a while or if you’d like to reset it, it’s at support.ownwebnow.com under your contact preferences.
In the event that your power goes down, Exchange/ActiveSync will chew through your battery quickly. Make sure you setup your ExchangeDefender LiveArchive IMAP/SMTP now. Make sure you test the IMAP-SSL/SMTP-SSL part in the event that your mobile carrier blocks 587. We have a few LiveArchive frontends so if you’ve only ever heard of https://livearchive.exchangedefender.com you’ll want to login into the admin portal and find out which LiveArchive client access server you should be using. Do yourself a favor and set your LiveArchive-To-Go account, it’s free and it can sit on your phone dormant, just flip it on when you need it. Not only will it use far less bandwidth than Exchange/EAS, it will make your battery last a lot longer.
Finally, print out the client auth dump. This will give you all the credentials you’ll need to provide to your clients to get them back to work if their servers or networks go offline. Trust me, you want to do this now. I have fielded countless panicked calls from IT people charging their cell phone in the trunk of their car trying to get credentials reset and configured for their clients – plan this now.
Some other ideas and suggestions:
1. Any important data that you imagine you’ll need on the go: Encrypt it and take it with you on physical media. For really important stuff like insurance, routing, VPN, etc make sure it’s printed out and you have all the relevant phone numbers.
2. Take pictures of all the gear you’re leaving behind and do a video walkthrough of the setup. Trust me, you’ll forget to take a picture of something.
3. I’d like to repeat #1: Print a hard copy of contact points, client info, anything you will need to obtain assuming both computers and networks are unavailable or batteries drained.
4. Agree on a cloud based chat system to keep in touch with your staff. This is essential.
5. Publish and announce an offside advisory site. As you start restoring services, you need to keep your clients in the loop – nothing worse than recovering from a disaster and simultaneously having to listen to someone whine or nag for an ETA that is just slowing you down. Make sure you have a Twitter or Facebook account to point them to for an update. You probably don’t want to call a thousand people back. Expect their phones to be off too because they probably have power issues as well.
6. I cannot stress this enough: Reconsider your name server infrastructure. If your DNS is provided by the same company, or if all your name servers are on the same network, subnet, etc and it goes down you are dead.
7. Setup a cloud failover crash site. If you own your own mail server, setup mailboxes just in case – if you’re going to be down for days you’ll probably want to use email in the meantime. If you’re wrong worst case scenario is you’re out a month worth of email hosting.
8. Fly someone far away. I heard Utah is beautiful and relatively hurricane-proof.
9. This one is personal: setup a remote VPN server. If all your routes, trusted networks and servers are offline you’ll have a hard time getting to the infrastructure you’re expected to manage – your clients may come back online before you do. Stand up an openvpn cloud instance with a static IP address for the time being and trust it across your network.
10. This is a bummer, and almost illegal for me to say as a Floridian, but as this is a category 5 storm it’s a really good idea to stay sober. You may be needed at a moments notice, if things get really bad you’ll really want to be aware of your surroundings and have full capacity to handle multiple outages and issues at once. Many of you are IT solution providers and someone is paying you to think about #1 – #9 I mentioned here… I’ve seen things explode well before the hurricane comes in and days after the storm passes.
Stay safe. Plan and prepare accordingly but rest assured that we’re here for you. And some of us are in Utah too. 🙂
P.S. Yes, our logo is a hurricane. Yes, we noticed 🙂