Rockerduck: What to Expect During an Outage
Over my next two blogs I will be overviewing the fail over procedures for Rockerduck and what clients should expect should a fail over occur. This blog post will go over the actual back end process and what factors influence whether we activate our fail over procedure. The next blog post will review the client experience once an outage occurs, the fail over and the recovery.
First, let’s qualify the differences between an “issue” and an “outage”. Issues are typically minor inconveniences or temporary “unavailability” such as a router reboot, temporary power outage, or network ‘blip’. Outages/failures can occur outright or can manifest from a minor issues. By rule of thumb, if the service is expected to be impacted for more than an hour, we consider the situation to require a fail over. Our fail over procedure is not ‘automated’ as we’ve elected to run the Database Availability Group for Rockerduck in DAC (Datacenter Activation Coordination) mode. When DAG’s run in DAC mode the secondary data center must be manually activated to mitigate an outage. This is done to prevent ‘Split brain syndrome’ where both data centers concurrently activate the same mailbox database.
There is a very specific reason we do not activate our fail over procedure for minor ‘issues’.
The fail over procedure by nature is risky and can lead to longer ‘down time’ if the issue is resolved before the fail over procedure completes or if an unforeseen event occurs during fail over.
For instance, imagine that our Dallas data center has a network issue and goes completely offline from the internet. Before receiving complete details on the outage from our data center, we decide to activate our Los Angeles data center. During the process of activating the LA copy, we switch DNS records to point away from Dallas to Los Angeles. Shortly after modifying DNS, imagine that our Dallas data center comes online and tries to take back control of the DAG (as communication was only lost to the internet). Dallas would then control the DAG databases while our entry point records would point to Los Angeles. This would yield poor results for clients as they would be proxying requests through LA to Dallas.
So what really goes on during a fail over?
After qualifying that any issue requires activation of our fail over procedure, immediately we will notify partners about the fail over activation. Before any changes get made, we review the health of our Los Angeles network and servers to ensure stability of the fail over. Once all services receive approval, we perform the following steps:
Step 1 – Modify cas.rockerduck.exchangedefender.com to point to the IP for cas.la.rockerduck.exchangedefender.com (TTL 5 Minutes)
Step 2 – Stop services on all Dallas mailbox servers
Step 3 – Restore DAG quorum in California
Step 4 – Mount databases in California
Step 5 – Modify inbound.rockerduck.exchangedefender.com to point to the multihomed MX record for Rockerduck LA (TTL 5 Minutes).
By keeping the TTL record for cas.rockerduck.exchangedefender.com at 5 minutes clients should automatically connect to the California data center to resume service without any modifications. In the same token, mail flow should automatically queue up in ExchangeDefender and upon update of the DNS records queued mail and new mail should deliver to Rockerduck LA.
Travis Sheldon
VP Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com
Exchange 2010 Essentials from ExchangeDefender: Costs & Benefits
Purchase decisions are all about a cost/benefit comparison, and always will be. What clients seek when making cost/benefit comparisons is very simple: good enough. Most want neither the bottom of the barrel, nor the top of the line, but a solution somewhere in the middle that addresses their core needs at a reasonable price. As your client’s VAR or MSP Trusted Technology Advisors, you get to give that “Goldilocks advice”—what’s too little, what’s too much, and what’s just right.
Often email and messaging are among the most important services to a business, tools they depend on every day. Just the same, in many cases these business clients of ours are willing to trade off a few features or accept a lower service level for the benefit of a lower price, but only to a point. The question becomes: how important is email to the client’s business? How much downtime can they afford? An hour? A day? Can that client justify the cost of guaranteeing that their email will always be available?
Some SMB clients don’t want or need a full-featured messaging and collaboration solution and want to minimize their costs of maintaining basic functionality. While Google Apps for Business offers an attractive price point with “pared down” messaging features, its unfamiliar interface may turn off folks who are comfortable with the Microsoft user experience. Exchange 2010 Essentials delivers the best of both worlds – minimal complexity using Outlook Web Access protected by ExchangeDefender Essentials along with the ever-popular Public Folders and Distribution Groups at a competitive price point, just $6 per month per mailbox. Clients can even integrate with their existing copies of Outlook to get that fat client experience they know and love.
Help to protect your valuable client relationships by offering them the right ExchangeDefender solution to meet their real needs.
Do you find this information useful?
If you’d like a lot more in-depth discussion about the cloud and how it affects you and your clients, visit Looks Cloudy http://www.lookscloudy.com where I blog daily about the adoption of the cloud in SMB, conduct live webcasts and podcasts with industry leaders, and more.
Kate Hunt
VP Community Development, ExchangeDefender
kate@ownwebnow.com
(877) 546-0316 x777
References…References…References…
We’ve been calling partners a lot recently to get a pulse directly from our client base on what things we can do to improve our service and one that came up, that’s pretty easy to address, is our documentation is hard to find and navigate.
We’ll go in order of relevance.
ExchangeDefender University – ExchangeDefender University is a very basic how to guide that includes some documentation links. This link is meant for a new partner that wants to know how to order or our services and deploy them.
http://exchangedefender.com/XDUniversity.php
ExchangeDefender Documentation – The documentation resources is a set of instructions is a bit more advanced as it goes beyond the standard deployments. However, these guides are scoped down to specific features so they’re more detailed and they’re more geared towards the guy that likes to print a doc and go do it to 20 machines, phones, etc.
http://exchangedefender.com/documentation.php
Support Knowledge Base – The Knowledge Base articles are for advanced users mainly. The details are provided for various custom deployments, repeat issues, advanced configurations. Now in the past while holding a lot of information, this option hasn’t been as appealing because it was not searchable. We recently made the Knowledge Base searchable which should improve its usefulness. The search is the same search box on the top right, it will now yield matching KB articles within the search results.
This link does require authentication, please use your partner portal credentials.
I hope you found this information useful!
Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737
If 2 is good, 50 is better, (Or is it)?
Over the weekend (12/09/11 – 12/10/11) we performed critical, preemptive upgrades for Rockerduck. During our upgrade cycle we were able to increase memory resources for Mailbox servers, rebalance resource distribution on Client Access servers and add additional Mailbox servers for quorum retention and additional high availability.
Mailbox, mailbox, mailbox…
By utilizing the current mailbox server layout, we were able to increase memory in Rockerduck mailbox servers in a staggering pattern without disrupting service to clients on Rockerduck. As each mailbox server was prepared for the upgrade, we moved all active mailboxes from the server to any passive mailbox node and then blocked the mailbox server from activating any database copy. After the memory upgrades were completed we then stress tested each server for 8 hours with a memory stress test for consistency. Once the upgrades were completed on the nodes, we were being the node back into the DAG and back up to availability.
Labs vs. Real World Results
Mailbox servers were not the only servers in Rockerduck to be upgrades. Over the past two weeks we’ve been monitoring the response statistics on CAS servers with a new memory / processor configuration.
Originally when we performed initial testing / scaling Rockerduck we seen the overall lowest latency and response time for RPC and Web Services from having a fewer CAS servers with higher RAM and processor. Over time, we’ve noticed the real world utilization result of overall latency on RPC was significantly outside the scope of our original Lab results causing us to reevaluate our delivery of CAS services.
All CAS servers for Rockerduck sit behind a hardware based load balancer. Each client that connects to the load balancer gets assigned to a specific CAS node for up to 5 hours on certain services (RPC, EWS) based off of the client WAN IP. Original design for the CAS nodes was 3 nodes with 8GB of RAM and 4 Processor cores available.
Unfortunately, this “least connected” model had the potential (and sometimes did) tie larger groups of users together from different IP addresses, essentially choking the server with queued requests.
The new setup for the CAS nodes is a balance of 6GB of RAM with 3 Processor cores available. This new configuration allowed us to introduce two new CAS servers to more efficiently process requests across multiple nodes without any additional “upgrades” to the CAS roles.
During our statistical collection phase, the new configuration nodes had a 40% reduction in response time on RPC requests and Address Book requests:
Originally: 22 ms
Now: 13.2 ms
Travis Sheldon
VP Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com
Business Slowing Down For the Holiday Season? How to Make It Productive and Profitable…
It’s the holidays, and things seem to have slowed down. It’s not that there isn’t anything to write about (there’s plenty), it’s that a bunch of folks are “checked out” mentally (or physically or both) for the season and there’s not as much going on as usual.
Many of our clients are probably in “vacation mode” now, too. Some of their employees and customers are Out, too. The phones are quieter. The service board seems empty. It can be tempting to slow or halt our own operations in response to this, but these quiet periods do not constitute an excuse for us to check out as well. In fact, choosing to follow suit on this one could be quite costly.
Think about it this way: when clients’ businesses are slower, they’ve got that much more time and bandwidth to sit down and have meaningful conversations with their Trusted Technology Advisor. Conversations which, when properly leveraged, can translate into real business gains for them and profitable work for us.
When are we most likely to get the time and attention of a busy CEO? Assuming that person is not out already, this is it. That busy prospect decision maker just might be able to carve out a few extra minutes during this time when they have less of the normal “day-to-day” to deal with.
The smart MSPs are kicking sales and marketing efforts into high gear while many others seem to be resting. The team probably has more downtime than is normal, as well.
Get on the phones, schedule those meetings, write up the reports, then get out there and Sell, Sell, Sell.
Do you find this information useful?
If you’d like a lot more in-depth discussion about the cloud and how it affects you and your clients, visit Looks Cloudy http://www.lookscloudy.com where I blog daily about the adoption of the cloud in SMB, conduct live webcasts and podcasts with industry leaders and more.
Kate Hunt
VP Community Development, ExchangeDefender
kate@ownwebnow.com
(877) 546-0316 x777
Where is My Mail Flow?
We’re going to take it back to the basics, because as we grow, we get new batches of partners which in turn comes with a new batch of tech support staff. And let’s face it, their knowledge of mail flow can vary from the guy that can write his own sendmail/postfix custom scripts, to the guy that just had great upbeat personality that most of tech folks lack. So… here we go.
There you are minding your business, possibly catching up on your company allowed tech blogs then your phone lines explode and you hear the dreaded words “We have not received any mail in the past “x” hours!! FIX THIS NOW”! Here at ExchangeDefender, this is easy to handle and you don’t need to call anyone, you don’t need to open and wait for a ticket for 99% of the scenarios.
Step 1
Check the mx record.
From windows:
nslookup -type=mx domain.comFrom linux:
dig mx domain.comFrom the web:
http://www.mxtoolbox.com/The one and only result you should get back is:
inbound30.exchangedefender.com
If you have anything else in there, and you’ve been getting complaints about random mail not arriving, the odds are great that they arrived at the alternative location and got bounced because of IP restrictions. But let’s not get sidetracked.
If there was no mx record in place, that’s your problem. Contact your client’s DNS provider and get inbound30.exchangedefender.com added ASAP. If inbound30 is the one and only record in place, go to step 2.
Step 2
Send a test message from a non ExchangeDefender domain. Here’s why, we route all outbound mail directly to the mail for processing speed so it won’t show up on your client’s mail log, but yours. Once you’ve sent your test message you want to note the from and to address and go to Step 3.
Step 3
Log into https://admin.exchangedefender.com with your Service Provider ID (Hint: it’s not a domain or an email address). Once you’ve logged in, click on Mail Log. Now you can search the Mail Log for your client for the very message you tested with and your answer will be at your finger tips.
You will see a line by line log of the SMTP transaction the most important one being the last one to your delivery IP. Generally, it will tell you exactly what the problem is, ranging from your ISP blocking the port, or a new tech blocking the port from us, or the email server running low on resources it’s all right there.
Blue – The recipient servers and the status returned, if it starts with a 2 it was accepted, if it starts with a 4 it was deferred, if it starts with a 5 it was permanently rejected.
Green – If it was accepted and your end user says they don’t see it. You can search the mail logs on that server for the message ID. (odds are there’s a rule in place if you get this far.)
Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737
Shockey Monkey, Becoming Self-Aware
This week was our Shockey Monkey v.2 Bug week, where we worked very closely with you (our partners) and quickly addressed the majority of the bugs that were reported. Overall, the volume of bugs reported was relatively small and typically very easy fixes!
***A special thanks to everyone who tested the SM Agent and provided us with feedback. We’ve identified several issues with the Agent and are now planning our revision process, which should take about a month to complete.
Let’s talk a little about the SM Agent and the direction we are looking to take upon its actual release. The SM Agent is essentially a component that you would install on your clients machines across an entire organization. Once this component is installed, you can now monitor the machine in several ways. One way is through data collection of things like (drivers, hard drives, optical devices, installed applications, event logs and more). The second way is through direct interaction, AKA Remote Desktop or Screen Sharing. The last core piece of this software will be scheduled and on-demand script execution. This would allow you to configure monthly, daily or hourly events that would trigger your custom scripts on a per machine basis.
One of the biggest requests we have had for implementing software of this nature has been for network installation. We are currently exploring several methods to help complete this goal and make network discovery and installation across and entire infrastructure possible. Obviously the goal here is to be able to deploy the agent across hundreds of machines with very little leg work and time spent on site.
The next huge request has been for an automatic ticket alerting and sending notifications when something breaks or reaches a certain threshold. With the introduction of the SM Agent, we believe we finally have a way to create such a system. The idea is simple, we allow you to basically script in the desired action dynamically based on your collection of devices.
For Example:
Logical Interpretation
IF [MACHINE]->[OBJECT]->[Attribute]->[VALUE] <= [VALUE] THEN [ALERT]->[OBJECT]Human Script Builder ( Drop Down Boxes )
IF “Desktop_XYZ”->“Hard Drive”->“Free Space”->”Size” <= “2GB” THEN “Open Ticket”.
What we hope to accomplish over the first few months of 2012 is implementing these advanced management features and then directly linking them back into the Dashboard and sections of Shockey Monkey.
Hank Newman
VP Development, ExchangeDefender
hank@ownwebnow.com
The Value of “Value”
Value is a word that carries many meanings among many people. According to dictionary.com value is defined as:
“Monetary worth, importance, the worth of something in terms of the amount of other things for which it can be exchanged or in terms of some medium of exchange.”
However, “value” in a marketing sense is expressing the benefit in your offering and you should strive for value to be the number one goal for your business to get across to your potential clients. This quick marketing tip can help you produce amazing results on an ongoing basis.
At ExchangeDefender, we are adamant about the importance of pumping as much value into your products or services as possible. The marketplace is fed up with always seeing the normal or average products and services. Thus, when a product or service comes along that exudes great value; they LOVE it and feel as if they got to HAVE it.
This also goes along with a statement that is important to remember: “There is no business on the earth that will buy a solution for a problem that they don’t have.” Yet, most marketers and businesses jump right into features without taking the time to fully understand if the people that they are trying to market to believe that they even have a problem to begin with. Thus, the marketing challenge at hand turns into a whole new task: “sell the problem”.
This creates an interesting oxymoron. Most people aren’t willing to acknowledge that they even have a problem unless they also believe that there is a solution to the problem. So a huge part of selling a problem is hinting that there’s a solution that others are using. The key part about that is letting them know that the solution they will ultimately need or be looking for is YOU! This is accomplished by marketing the VALUE and benefits of your solution and presenting it in a manner that allows them to see the solutions to the problem at hand.
How to Add Value to Your Offering?
One way you can increase the value of your offering, is to lower your prices or fees associated with your product of service. While this method can be effective, it can also drag you down into a price war. Another reason to stay away from this option is that many people that are attracted solely to low prices, are the most likely to leave and change providers as soon as another company beats your price; and that is exactly what you are trying to stay away from. In my opinion, it’s far better to find ways to pack a ton of value into your product and sell your offering for a great “value for the money” price, rather than to offer an average product and sell it cheap.
How ExchangeDefender Offers YOU Value
At ExchangeDefender we help our partners to offer VALUE in the most simplistic ways. For example:
· All of our products and services are all-inclusive. This packaging structure makes it easy for you to offer your clients services that contain a lot of VALUE to them, but yet you are not being nickel and dimed for every feature. This way you can focus on what is most important to your clients and be able to offer it to them, thus becoming the solution to their problem without the additional cost associated with it.
· There are no minimums with our services, no contracts, no additional fees. Whether you have 1 or 100 users, you can use our solutions. This allows us to cater to businesses of all shapes and sizes to offer them solutions that can help to make them profitable.
· We offer marketing collateral and proven best practices. We believe and know for a fact that you are busy! As are we, so we make marketing collateral available for partners to use for their clients so that they can focus on the business rather that how to sell it. This creates value because it is one less thing that you have to do. Also, we have a wealth of knowledge that has been documented from our partners success with offering ExchangeDefender’s services. These profitability best practices are great ways to get ideas on how to market services to your clients and maintain profitable relationships with them with the help of us here at ExchangeDefender.
· We don’t compete with you. We work with you to make a sale by offering you a solution set that will be of value to your clients and provide support of those services.
· We are available 24x7x365! We have support via the phone and through are ticketing system to be here to support you and make our partnership successful.
There are many more examples of the value that we offer our partners and help them to embrace the cloud.
Ultimately, the marketplace is attracted to VALUE – The bigger the value, the more attracted a customer will become. By making your product offering as valuable as possible, you will never have to rely solely on selling based on price again!
If there is anything that we can help you with, please feel free to let me know!
Stephanie Hasenour
VP Marketing, ExchangeDefender
stephanie@ownwebnow.com
The Database Availability Group is Supposed to be Completely Fault Tolerant…
Earlier this week we created a NOC entry/notification for partners about a maintenance interval we scheduled for ROCKERDUCK. The entry outlined an issue we faced where on DB (DB7) was running on the logs drive instead of the DB drive and our proposed outline of the work to be completed. Unfortunately, because the issue affects all database copies, correcting the issue would involve reducing DB7 to a single mailbox server, moving the database, which would take DB7 offline, and then re-seeding the copies to all passive nodes.
Shortly after posting the NOC entry I received an email from a partner demanding that I explain to them why the Database Availability Group (DAG) could not prevent service interruption for users on DB7.
So why does the DAG not protect from every single event possible?
Simply said; all servers in a DAG must be identical in terms of storage location for databases and logs across all servers. In a DAG, only one mailbox can act as the “active” mailbox database and all other copies on other nodes are purely “database copies” that can be switched to the active/primary database.
In the case of moving the database path, we cannot switch the current active database over to a passive node, move the DB, then switch it back to the original primary as this would break the DAG and we would then have split copies of ‘active’ data. We cannot use passive copies to keep service active while we physically modify the database properties/layout of the ‘active’ copy.
If this was a case where a database experienced a failure on the active copy or there was a network communication issue, the DAG would mount the passive copy of the database and continue providing service to users.
So all this jibber-jabber means what?
In short, we would remove all copies of DB7 across all nodes except the primary node. After all copies are removed, we would start the move of DB7 to the proper location and then remount the database. By calculation of the DB size, service would be interrupted for about 10-15 minutes. Finally, after the move completes we would re-add the database copy across each node and then bring service back into full redundancy. A fifteen minute outage is unfortunately a necessary evil to provide an overall more redundant solution to our partners and their clients.
Travis Sheldon
VP Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com
Three Common Business Challenges…& How Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender Solves Them
Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender can address a lot of different kinds of problems for a lot of different kinds of businesses. It can help many of our clients in a multitude of ways. Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender minimizes the upfront costs for a business to take advantage of stable, secure web communication and collaboration. Because it’s hosted in our cloud, it’s accessible from anywhere, 24 hours a day 7 days a week, on any device. We include everything that a business needs to leverage the power of Microsoft Exchange for their growing business for a low monthly fee.
Here are three common business challenges and how Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender solves them:
Losing time, wasting time, or not getting enough done as a group
For growing businesses, profitability is all about efficiency – doing more with fewer resources, less money, less time. It is mission-critical to make sure that employees are actually working, and can be really important to be able to see who is free to take on a job at the last minute, at any time.
Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender allows your clients and all of their employees to share calendars and tasks easily across the web. They’ll be able to tell who’s busy and who’s free right inside Outlook, schedule appointments on other users’ calendars, and save time by avoiding all the complexity of phone calls and emails for scheduling and coordination.
Lost or delayed orders from email server crashes
Many businesses today use email to track orders, orders from their suppliers and vendors, and also orders from their clients. When email is down, it can take a severe and expensive toll on the business.
With Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender, data is stored on our globally-redundant infrastructure and accessible via Outlook Web Apps, Outlook desktop client, and mobile devices. Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender is protected by the world-class ExchangeDefender security suite, so in addition to being reliable, messages will be free of malicious code and safe for consumption anytime, anywhere, and also comes with the LiveArchive service built right in. LiveArchive is the ultimate redundancy feature that enables real-time access to one year’s worth of messages via Outlook Web Access anytime, anywhere in case of any kind of hardware failure or outage.
Losing money by being chained to the office park
So many businesses today are beginning to recognize the opportunity. There are many efficiencies and benefits to deploying a mobile workforce. Smartphones, tablets, telecommuting, and the like are becoming more accessible, less expensive and generally more viable options for all businesses, especially growing ones.
Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender supports automatic wireless synchronization with your Android, iOS, Windows Phone 7 device. BlackBerry support is available for a small additional fee, as well. Whichever mobile platform the client is on, contacts, calendars, and messages are kept up to date at all times.
…and another thing:
Speaking of losing money, perhaps the number one limitation faced by growing businesses is being tight on capital. Hosted Exchange is available for a flat monthly fee per user without the big up-front cost of servers and software licenses that come with traditional applications. It’s fully compatible with existing supported Microsoft Office desktop applications, and fully supported 24/7.
Do you find this information useful?
If you’d like a lot more in-depth discussion about the cloud and how it affects you and your clients, visit Looks Cloudy http://www.lookscloudy.com where I blog daily about the adoption of the cloud in SMB, conduct live webcasts and podcasts with industry leaders and more.
Kate Hunt
VP Community Development, ExchangeDefender
kate@ownwebnow.com
(877) 546-0316 x777