If 2 is good, 50 is better, (Or is it)?
Over the weekend (12/09/11 – 12/10/11) we performed critical, preemptive upgrades for Rockerduck. During our upgrade cycle we were able to increase memory resources for Mailbox servers, rebalance resource distribution on Client Access servers and add additional Mailbox servers for quorum retention and additional high availability.
Mailbox, mailbox, mailbox…
By utilizing the current mailbox server layout, we were able to increase memory in Rockerduck mailbox servers in a staggering pattern without disrupting service to clients on Rockerduck. As each mailbox server was prepared for the upgrade, we moved all active mailboxes from the server to any passive mailbox node and then blocked the mailbox server from activating any database copy. After the memory upgrades were completed we then stress tested each server for 8 hours with a memory stress test for consistency. Once the upgrades were completed on the nodes, we were being the node back into the DAG and back up to availability.
Labs vs. Real World Results
Mailbox servers were not the only servers in Rockerduck to be upgrades. Over the past two weeks we’ve been monitoring the response statistics on CAS servers with a new memory / processor configuration.
Originally when we performed initial testing / scaling Rockerduck we seen the overall lowest latency and response time for RPC and Web Services from having a fewer CAS servers with higher RAM and processor. Over time, we’ve noticed the real world utilization result of overall latency on RPC was significantly outside the scope of our original Lab results causing us to reevaluate our delivery of CAS services.
All CAS servers for Rockerduck sit behind a hardware based load balancer. Each client that connects to the load balancer gets assigned to a specific CAS node for up to 5 hours on certain services (RPC, EWS) based off of the client WAN IP. Original design for the CAS nodes was 3 nodes with 8GB of RAM and 4 Processor cores available.
Unfortunately, this “least connected” model had the potential (and sometimes did) tie larger groups of users together from different IP addresses, essentially choking the server with queued requests.
The new setup for the CAS nodes is a balance of 6GB of RAM with 3 Processor cores available. This new configuration allowed us to introduce two new CAS servers to more efficiently process requests across multiple nodes without any additional “upgrades” to the CAS roles.
During our statistical collection phase, the new configuration nodes had a 40% reduction in response time on RPC requests and Address Book requests:
Originally: 22 ms
Now: 13.2 ms
Travis Sheldon
VP Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com
The Value of “Value”
Value is a word that carries many meanings among many people. According to dictionary.com value is defined as:
“Monetary worth, importance, the worth of something in terms of the amount of other things for which it can be exchanged or in terms of some medium of exchange.”
However, “value” in a marketing sense is expressing the benefit in your offering and you should strive for value to be the number one goal for your business to get across to your potential clients. This quick marketing tip can help you produce amazing results on an ongoing basis.
At ExchangeDefender, we are adamant about the importance of pumping as much value into your products or services as possible. The marketplace is fed up with always seeing the normal or average products and services. Thus, when a product or service comes along that exudes great value; they LOVE it and feel as if they got to HAVE it.
This also goes along with a statement that is important to remember: “There is no business on the earth that will buy a solution for a problem that they don’t have.” Yet, most marketers and businesses jump right into features without taking the time to fully understand if the people that they are trying to market to believe that they even have a problem to begin with. Thus, the marketing challenge at hand turns into a whole new task: “sell the problem”.
This creates an interesting oxymoron. Most people aren’t willing to acknowledge that they even have a problem unless they also believe that there is a solution to the problem. So a huge part of selling a problem is hinting that there’s a solution that others are using. The key part about that is letting them know that the solution they will ultimately need or be looking for is YOU! This is accomplished by marketing the VALUE and benefits of your solution and presenting it in a manner that allows them to see the solutions to the problem at hand.
How to Add Value to Your Offering?
One way you can increase the value of your offering, is to lower your prices or fees associated with your product of service. While this method can be effective, it can also drag you down into a price war. Another reason to stay away from this option is that many people that are attracted solely to low prices, are the most likely to leave and change providers as soon as another company beats your price; and that is exactly what you are trying to stay away from. In my opinion, it’s far better to find ways to pack a ton of value into your product and sell your offering for a great “value for the money” price, rather than to offer an average product and sell it cheap.
How ExchangeDefender Offers YOU Value
At ExchangeDefender we help our partners to offer VALUE in the most simplistic ways. For example:
· All of our products and services are all-inclusive. This packaging structure makes it easy for you to offer your clients services that contain a lot of VALUE to them, but yet you are not being nickel and dimed for every feature. This way you can focus on what is most important to your clients and be able to offer it to them, thus becoming the solution to their problem without the additional cost associated with it.
· There are no minimums with our services, no contracts, no additional fees. Whether you have 1 or 100 users, you can use our solutions. This allows us to cater to businesses of all shapes and sizes to offer them solutions that can help to make them profitable.
· We offer marketing collateral and proven best practices. We believe and know for a fact that you are busy! As are we, so we make marketing collateral available for partners to use for their clients so that they can focus on the business rather that how to sell it. This creates value because it is one less thing that you have to do. Also, we have a wealth of knowledge that has been documented from our partners success with offering ExchangeDefender’s services. These profitability best practices are great ways to get ideas on how to market services to your clients and maintain profitable relationships with them with the help of us here at ExchangeDefender.
· We don’t compete with you. We work with you to make a sale by offering you a solution set that will be of value to your clients and provide support of those services.
· We are available 24x7x365! We have support via the phone and through are ticketing system to be here to support you and make our partnership successful.
There are many more examples of the value that we offer our partners and help them to embrace the cloud.
Ultimately, the marketplace is attracted to VALUE – The bigger the value, the more attracted a customer will become. By making your product offering as valuable as possible, you will never have to rely solely on selling based on price again!
If there is anything that we can help you with, please feel free to let me know!
Stephanie Hasenour
VP Marketing, ExchangeDefender
stephanie@ownwebnow.com
The Database Availability Group is Supposed to be Completely Fault Tolerant…
Earlier this week we created a NOC entry/notification for partners about a maintenance interval we scheduled for ROCKERDUCK. The entry outlined an issue we faced where on DB (DB7) was running on the logs drive instead of the DB drive and our proposed outline of the work to be completed. Unfortunately, because the issue affects all database copies, correcting the issue would involve reducing DB7 to a single mailbox server, moving the database, which would take DB7 offline, and then re-seeding the copies to all passive nodes.
Shortly after posting the NOC entry I received an email from a partner demanding that I explain to them why the Database Availability Group (DAG) could not prevent service interruption for users on DB7.
So why does the DAG not protect from every single event possible?
Simply said; all servers in a DAG must be identical in terms of storage location for databases and logs across all servers. In a DAG, only one mailbox can act as the “active” mailbox database and all other copies on other nodes are purely “database copies” that can be switched to the active/primary database.
In the case of moving the database path, we cannot switch the current active database over to a passive node, move the DB, then switch it back to the original primary as this would break the DAG and we would then have split copies of ‘active’ data. We cannot use passive copies to keep service active while we physically modify the database properties/layout of the ‘active’ copy.
If this was a case where a database experienced a failure on the active copy or there was a network communication issue, the DAG would mount the passive copy of the database and continue providing service to users.
So all this jibber-jabber means what?
In short, we would remove all copies of DB7 across all nodes except the primary node. After all copies are removed, we would start the move of DB7 to the proper location and then remount the database. By calculation of the DB size, service would be interrupted for about 10-15 minutes. Finally, after the move completes we would re-add the database copy across each node and then bring service back into full redundancy. A fifteen minute outage is unfortunately a necessary evil to provide an overall more redundant solution to our partners and their clients.
Travis Sheldon
VP Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com
LiveArchive: Why? Where? How?
I’m going to address an age old question from folks that do not like to read our feature pages, in hopes that you read this blog. As part of the DR (Disaster Recovery) we have two primary items that can help during and after an outage. This post will help educate your teams on the expectation of how things work, so your expectations as well as your clients are managed to the correct level.
During an outage
During an outage the best place to have surefire access is to type https://livearchive.exchangedefender.com into your browser. This is the sure fire way to ensure that regardless of which cluster is live (Dallas or Los Angeles) your clients can get to it. A best practice is having a shortcut ready for your clients on their Desktop or Start Menu. If I had a penny for each time that someone’s server catches fire and it’s that juncture that a tech asks “How do I get to LiveArchive?”. You are already putting yourself in front of the barrel. If you don’t have a solution in hand and you have to “call someone else”, it’s that point that your client’s confidence starts eroding.
Where is LiveArchive?
LiveArchive is located at https://livearchive.exchangedefender.com
What are my LiveArchive credentials?
Your LiveArchive credentials are the same as your ExchangeDefender credentials; which are your email address and your ExchangeDefender password. Remember if you forgot this password and your email is down your best bet in an outage scenario is to open a ticket for your client in our portal and request their passwords. Sadly, folks often try their email passwords and assume that something is wrong (see above: more erosion). The key to all of this is to get the right answer on the first try.
So let’s move forward, now you either knew everything above upfront and only have to deal with your end users once or you had to go back and forth a few times to get it hammered out. Regardless, your clients have access to all of their internet mail now, now your hard job starts. Get the defibrillator and resurrect their Exchange server, obviously this can range from a simple reboot to a week long pain staking process. One thing you have in the back of your mind is, thank goodness ExchangeDefender is holding all of my mail. The most important thing to remember while you and your team are doing your best to perform thoracic surgery to the server is make sure the server is offline!!
Here’s why, by RFC rules we can only hold mail that is being deferred by your server. If your server is online and “REJECTING” mail due to bad configuration or your troubleshooting, all that mail is purged because your client’s server is telling our software this is permanent rejection. This is the biggest key in the process, luckily this doesn’t happen often but there are teams that will have the server permanently rejecting mail for a week and then ask for their mail. And even though this is digging yourself a grave, we MAY still be able to help you.
First off our Mail “Spooling” or “Bagging” service is in place for up to 7 days. The way it works is, after the initial real-time attempt to deliver your mail, your mail is moved to a retry queue. This queue in an effort to not hammer client servers reattempts to deliver from each node every 20 minutes or so, staggered. This process is fully automated and constantly running, you don’t have to call us or open a ticket saying, “Our server is up release our mail”. If your server really is up and accepting mail from our servers your mail will start to flow on its own, but it can take up to a couple of hours for all of your mail to deliver depending on your queued volume. Again, we don’t want to pound your client’s server into submission and cause it to trigger the Exchange backpressure mechanism.
Now, if you made the unfortunate mistake to bring back a server online after rebuild without the process IP restrictions and anonymous delivery settings and all of your spool was lost there is still one possibility. If the mail is in LiveArchive, due to our hub and transport design you can actually forward all that mail to your individual client’s mailboxes one by one. This is a fully manual process that can is pretty time consuming but when faced with the choice of telling a client you lost all their mail for the past x number of days or telling them you need a couple more hours to make them whole, the choice becomes easy.
Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737
Shockey Monkey 2
With the release of Shockey Monkey 2 coming early next week, we have been working like craaaaaaaaaaaazy these last few weeks to finish the major revisions and clean up all the banana peels! This will be the LARGEST update to the Shockey Monkey core that has happened since it was created several years ago. The first thing you will notice is the new UI and dashboard layout on the main screen. With this redesign we tried to encompass the majority of all the vital information to one central and eye catching location. It will give you and your staff the ability to see a complete ticket overview, recent tickets, announcements, your agenda and tasks at a moment’s glance. The best part… it’s compatible in virtually every browser!
Out of the gate, the new UI will come with 5 preloaded themes that you can switch between inside of the portal. It won’t make it in time for the launch next week, but the goal is to implement a dynamic theme configuration tool inside of the portal. This would allow you to make the solution really and truly represent the look and feel that encompasses your company. We’ve also made several improvements to other aspects of branding that will give you an ease of comfort when navigating your settings.
We have also added several new features that really help Shockey Monkey stand out from the crowd! You now have the ability to go completely full screen and remove the menu bars while working inside of our solution. They are can be controlled by a toggle anytime from the bottom left toolbar and your setting will remain persistent for the duration of your session. This allows you to view more information and stay focused on the task at hand without any compromise to performance. We’ve also implemented a chat and notification system to allow interoffice communication and to give your customers the ability to chat directly with your support or sales team. We’ve improved the logging system and enhanced the search mechanism that will allow you to navigate through the activity inside of your portal.
On the backend we have been continuing to reduce the code overhead and processing load across all pages. This isn’t a difficult process, it’s just a very tedious and mind numbing experience. It may not make a lot of sense right now, but essentially this process is laying the foundation for very exciting future releases. Simply put, it’s all about being dynamic and allowing the solution to be put together to meet your individual needs. Overall the redesign has amazingly well and we will continue to improve dozen of areas even after the Shockey Monkey 2 launch.
Hank Newman
VP Development, ExchangeDefender
hank@ownwebnow.com
Shockey Monkey Reloaded
This week the development team has been focused on cleaning up Shockey Monkey. This process is mainly composed of three focused categories (UI, Performance, and Bug/Warning Fixes). I would say this has been the largest update to Shockey Monkey since it was originally created over seven years ago. This update should also help lay the framework for any future additions or tweaks.
User Interface
– We’ve applied a consistent look and feel across the entire portal.
– Implemented a themed backend that allows quick and easy branding.
– Improved cross browser support
Performance
– Improved loading performance across the portal.
– Minified most of the JavaScript files to reduce fetch & loading times.
– Implemented more session variables to reduce the amount of queries.
Bug Fixes
– Decreased Disk IO access by resolving more than 1000+ warnings that were constantly being written to a local log file.
– We’ve fixed a handful of bug over the past few weeks and have been continuing to resolve these issues as we enhance the portal.
– If you have a bug, submit it here: https://support.ownwebnow.com/create.asp?item=development
Performance is being improved in several ways across the Shockey Monkey portals. The first thing we did was minified thousands and thousands of lines of JavaScript. The compression ratio on average across all of our scripts was about 65%, which is very good. We’ve also been fixing trivial “warning” messages as we progress through the code. By resolving these warning messages that are generated on the back end, it reducing the amount of disk IO that is required to store this information. Typically this is not a big deal, but multiply that by 1000’s of warning and you get some nasty seek times.
If we go a little bit into the cleanup of the User Interface, it was a project that was mainly focused on consistency. We are also making sure that the look and feel between browser types is very similar. There are also a few special features of the new UI that will really help bring the User experience to a new level. One of these features is the ability to view any page in “Full Screen” mode. What this does is remove the top and side menus from the portal. This allows you to manage any page without the annoyance of having to resize or scroll to achieve simple tasks.
We have a lot more information and a ton more features that we are saving for the unveiling. If you would like to be a part of the Shockey Monkey Reloaded webinar on Thursday, December 1, 2011 12:00 PM – 1:00 PM EST, visit the following link: https://www1.gotomeeting.com/register/812869640.
Hank Newman
VP Development, ExchangeDefender
hank@ownwebnow.com
Dual Decryption Core Technology
Last week we have made a lot of improvements across several of our products. Monday & Tuesday were bug squashing days, we went through the bugs that have been submitted via https://support.ownwebnow.com/create.asp?item=development and resolved most of the outstanding issues. Remember, if you have an important bug make sure you submit it to our development queue. The rest of the week was spent implementing our new message decryption engine across several of our platforms.
One of the most common and repeating issues was in regards to rendering issues with messages viewed at https://encryption.exchangedefender.com and SPAM previews through ExchangeDefender. While most messages would render without any problems, there were still those handful of clients who had issues with some messages. Typically this was caused by the use of different 3rd party e-mail clients that would use an unexpected RFC type or submit the data using a weird/foreign content-encoding type. This is even more prevalent in SPAM previews, because most of the time those messages are garbled or gibberish! So that makes the decryption process even more challenging!
So these last few months, we’ve been examining a new core technology to decrypt these messages and help bring more legible content to our users and their clients. With this new decryption core it handles most MIME types flawlessly and adheres to most RFC standards for messages. We ran it through several test cases where previously: “the messages was unpleasant to read” and it converted them to “a perfect rendered instant of the original message”. However, there are still the few instances where our old engine is better than the new decryption core.
That’s why we’ve implemented dual decryption core technology. This is a custom implementation that will programmatically decide which engine is best to use on a per message basis. First, it examines the message and tries to load it with the new decryption engine, while it decodes the message it makes note of any issues that have occurred upon the decryption process. If there was an issue, it will run the message through the second decryption engine and compare the results, then render the best instance of that message.
You’re going to see a lot of rapid development over the following months here at ExchangeDefender. Like I discussed in previous posts, we’ve starting to implement SOAP across several platforms and it’s really taking hold. The Compliance Archive enhancements, Decryption Engine enhancement and even planned Shockey Monkey Dynamic Callback Mechanism have all been made possible by the use of SOAP that we’ve implemented here at ExchangeDefender over the past several months.
Hank Newman
VP Development, ExchangeDefender
hank@ownwebnow.com
ExchangeDefender LoadBalancer
On a random basis, one of the most pressing issues faced by our customers was mail delivery delays. These random delays happen to only about 12% of the nodes, however, due to the sheer volume of mail processed by our inbound network that 12% would inevitably cause our staff and customers quite an inconvenience.
As more and more companies begin to depend on email for the main source of communication for their business, mail delivery time becomes a major factor when partnering with a hosting vendor. Because of the critical need for instant delivery, we had to quickly overcome our growth and produce an immediate solution.
The Unforeseen Weakness in Round Robin
After the ATS power outage earlier this quarter we were forced to reevaluate or solutions across the board and make drastic improvements. One of the areas that desperately needed a redesign was our inbound mail load balancer.
In the early days of ExchangeDefender we utilized a round robin based load balancer. In short how it works is the MX records for ExchangeDefender clients are pointed to both our Dallas datacenter and our Los Angeles data center. After the SMTP connection hit either data center, the connection was then forwarded off to any random inbound node in the virtual server list.
Until earlier this year, the round robin design worked quite well, however, as the number of messages being processed grew, so did the delivery delays. We started to notice that the load balancer that was able to previously balance the connections was no longer balancing at all. Day after day, we saw some inbound servers having upwards of 200 concurrent connections at a time. More than half of the other inbound nodes in the respective data center had no connections at all.
The biggest issue preventing the round robin configuration from working was the randomized assignment of which data center would be used and which inbound server that would receive the connection. To begin to tackle the issue, we had to re-evaluate the entire load balancing solution because you can never properly balance a round robin based connection. We switched our load balancers to use a weighted least connection based routing scheme. Upon activation it seemed to balance connections a bit better than the round robin connection. Nearly an hour after activation however, we saw a large queue size being placed on a few inbound nodes.
Brand New Logic
To completely resolve the issue, we had to introduce additional logic to the load balancer. The recurring issue we faced was the basic nature of SMTP. An SMTP connection for one “message” being transmitted could equal four concurrent open connections. Therefore, naturally, the connection count cannot be relied upon for load balancing. We then decided to leverage our queue reporting service which reports the number of queued messages and open conversations with unique IPs. Finally, we created a PHP script that runs on the load balancer and splits itself to check the connection counts across all nodes every 10 seconds. We used a very simplistic formula for load balancing:
If(($numActiveServersInSite >= ($numServersInSite / 2)) && ($nodeConnections >= ($nodeAverageConnectionCount + 50)) && ($nodeConnections <= $highThreshold)){Shutoff new connections}
In non-code terms, we now calculate the number of servers in each site (DC). If there are more than half of the servers offline in a site, the load balancer will no longer shut off new node connections. If the current node connection count is either greater than the high threshold or has 50 active connections above the average connection count of servers in the site then it will shut off the node.
The End Result
After implementing the new load balancer algorithm we saw drastic improvements of balanced connections and delivery times for inbound messages. The most notable improvement took place on September 10th where we processed 2.8 million messages and we saw minuscule delays across the board and received zero complaints about deliver delays.
Travis Sheldon
VP Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com
ExchangeDefender Enhances Dynamic HTML Content Filtering and more…
This week I’ll be recapping some issues that may have affected multiple clients in the past week within the ExchangeDefender realm of our portfolio. Before I get to those items, I’d like to take this time to thank some of our partners who were extremely helpful and truly honor up to the title of partner. A couple of these issues required client side testing that we would not be able to emulate as they involved third party transactions and notifications.
The first multi-client issue we resolved last week was a content filtering issue. We received a couple of reports that ticket updates from one of the major PSA vendors were getting pretty mangled by our filtering, we had a couple of partners step up with excellent samples in .msg format, that allowed me to dive into the message content and header information to track down what portion of our filter was causing the issue.
Basically, as technology continues to grow and migrate more towards email as the primary method of communication, email content itself is becoming more rich and interactive. In the past, emails with form tags and rich html were often phishing scams, but in this case it was a significant amount of mail flow from a reputable source to probably a large portion of our client base. In order, to resolve the issue we basically turned the filtering dynamic from a flat (almost boolean) logic to a very dynamic, multi-level, per domain/ip policies. Thankfully, one of our partners was able to provide documentation from their PSA vendor that provided whitelisting guidelines. Once we received that, we were able to move forward, and make a portion of filtering that has been pretty rigid in the past into something more dynamic and flexible to reflect current email content. Please see the basic layout of the new web tag workflow:
The second issue we tackled involved our Encryption service’s feature that allows folks to reply to an encrypted message directly from the portal in an encrypted manner. Currently, on our primary smart host we use a very strict anti-spoofing measure that does not allow folks to relay as “from” a wide variety of free mail services. We had to put this measure in place to improve overall mail reputation and we’ve seen marvelous results so those changes will always remain in place. However, we hit a road block when someone from one of these free mail services attempted to relay an encrypted message because our anti-spoofing measures would not accept the messages in their current format.
The solution lied in having PHP use an alternate route to deliver messages generated by this script only, without altering the SMTP configuration. So after digging through pages and pages of configuration notes for the mail() function in php, I kept finding that you can pass “other parameters” to your SMTP process within the command. However, I only found folks had successfully used it in the past to rewrite the “from:” address, which we were already doing successfully.
Apparently what folks failed to mention is that what the “additional parameter” is doing is just appending your arguments at the end of the SMTP command defined in your php.ini file. Once you reach that point, you basically can run a different SMTP configuration for any script on demand instead of just blindly relying on whatever smarthost the server is running. To make the rest of this story short, folks using free mail services can now reply to Encryption messages within the portal without any issues.
Here are some tips:
1. Before submitting and subsequently waiting on a ticket, please take a look at the Knowledge Base inside of the portal. We’ve recently consolidated a large amount of information from our previous help blogs to make sure everything we’ve made available, can be viewed in one location. We’ll be taking a look at outdated items over the next week to make this a very valuable resource.
2. Rely on the portal. https://support.ownwebnow.com is where we are available 24/7. Remember there’s no such thing as too much information, but there is such a thing as not enough information. To avoid back and forth interactions, always error on the side of too much information it will always save you time.
Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737
ExchangeDefender Redundancy: Technical Implementation Details
In August our core data center in Dallas Texas suffered its worst outage in the past 8 years after an ATS power failure. Since the power outage was due to the ATS, backup power was not able to be routed to the redundant feeds to all our servers and as a result critical services were knocked offline. ExchangeDefender’s core focus is on-time clean mail delivery, and inbound mail delivery speed was impacted but was still available and processing mail for all customers. One of the most beneficial features of ExchangeDefender is LiveArchive which acts as a continuity solution for ExchangeDefender clients by keeping an additional copy of all client inbound and outbound mail. During the outage, LiveArchive was knocked completely offline for the first time since release. The full scale outage of LiveArchive prevented clients on Hosted Exchange and ExchangeDefender from being able to reach their continuity solution, creating lots of issues for partners and their clients. The outage caused us to reevaluate the redundancy in our solutions across the board starting with additional ExchangeDefender nodes and implementation of LiveArchive in our Los Angeles Data Center.
Today, I would like to update you on what we’ve done and announce the immediate availability of the new solution that will work even if one of our data centers is offline completely for an extended period of time.
Complexity & Size Problems
The biggest complexity of the additional LiveArchive network was accepting mail between multiple datacenters and routing it to the nearest network, and finally synchronizing the mail between with remote networks. Due to amount of mail processed by ExchangeDefender on a daily basis, we were prevented from using the built in redundancy in Exchange 2010 like Database Availability Groups and Microsoft software load balancers for OWA and SMTP.
On average our Dallas LiveArchive network processes and stores over 115 million messages a day. By adding redundant LiveArchive networks we will automatically double our message processing count on our Exchange transport servers due to synchronization between networks. The additional overhead would also create immense load on our network if we utilized Database Availability Groups as our Dallas network is already storing over 382 TB of data. The notion of having a desynchronized network or increasing LiveArchive mail delivery time was not a viable solution as LiveArchive would then fail as a continuity solution.
The Solution
To overcome the obstacles and limitations we faced, we started to develop custom solutions to tie in with our LiveArchive networks and redesigned our network layout and made minor changes to the LiveArchive storage length in Los Angeles to one month.
The first issue to tackle was the Active Directory layout as we could not tie Active Directory into our already established and quite frankly rock solid Dallas Active Directory network. In the event of another catastrophic failure in Dallas the inability to contact the primary Active Directory network would render the Los Angeles copy useless unless we create them as unique sites. Although this was possible, we felt with the power of our automation between ExchangeDefender and LiveArchive was better suited for the job as the above solution would cause too many changes to occur during a failure, including many DNS changes. We decided to have unique Active Directory domains between sites and we hooked user account creations for Los Angeles into our ExchangeDefender provisioning.
The second issue was mail delivery design between sites. The original design of LiveArchive mail routing would cause the LiveArchive copy of inbound mail to always be routed to Dallas for delivery to LiveArchive. The Achilles heel of the design was always in mail that was delivered to Data Centers outside of Dallas as it would cause the original mail delivery to be delayed a few seconds as a copy was dispatched to Dallas. Up until a few months ago our original design was acceptable and was only cause minor delays about twice a year. As the amount of mail processed increased, the delays became more noticeable. Mail delivered to networks outside of Dallas began to see increased processing time and eventually took twice as long compared to mail that was accepted by Dallas. By adding an additional LiveArchive network in Los Angeles we were able to route mail that arrived in Los Angeles to our Los Angeles LiveArchive network.
The final issue was mail synchronization between sites. We took advantage of the powerful extensions into Exchange by creating a custom Transport Agent that would copy submitted messages from each ‘local’ LiveArchive message to the remote network for processing. By utilizing custom routing and Edge servers we were able to successfully copy mail and provide real time processing and delivery between both LiveArchive networks.
In the event of a failure in either site, ExchangeDefender mail will automatically fail over to an additional DC and all LiveArchive mail will route to the network with available heartbeats. Upon creating an outage notification, our team will modify the DNS records for https://livearchive.exchangedefender.com to route to Los Angeles. To access the Los Angeles LiveArchive network directly users can login at https://la.livearchive.exchangedefender.com
Bookmarks:
LiveArchive: https://livearchive.exchangedefender.com
Los Angeles LiveArchive Cluster: https://la.livearchive.exchangedefender.com
Please make sure your clients are
Sincerely,
Travis Sheldon
VP Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com