Huge Maintenance Cycle Announcement
Three times a year Own Web Now Corp conducts a global network maintenance cycle. These maintenance cycles are meant to double-check the equipment, swap out aging infrastructure, improve cable management as execute a disaster recovery procedures. In plain terms, we take the network down at an announced time and work on it during off-peak hours so it doesn’t crash unannounced in the middle of the day.
Our global network maintenance cycle is scheduled for this Sunday, September 23, 2 AM – 5 AM Central (GMT -5).
All networks, all services, all customers will be affected. We will literally be shutting the NOC down and restarting it from scratch.
We will also have a minor ExchangeDefender Policy Engine upgrade during Saturday afternoon, the services should not be affected beyond perhaps a few minutes without control panels while we swap out the switches and nodes.
Scaling Up and potential latency issues
As we prepare for the massive upgrades coming this weekend we are obviously testing systems and making intermediate changes to the network. As a result, over the next 48 hours you are likely to see some latency in DNS query results which virtually impact all other services such as backups, ExchangeDefender, virtual servers and everything else thats being brought online.
While you are unlikely to notice any of these changes directly, if you do see slight performance issues they are probably related to the maintenance work being done on our end.
DNS and Time Infrastructure Overhaul
As our network grows even the most optimized of services need to scale. While its unlikely that you may have noticed an issue with DNS services, we have decided to both increase its capacity and reduce the scope of that service. We have also added the ability for you to sync with reliable internal time servers. Both modifications are nearing completion but you can take advantage of them right now as they prepare us for future growth.
DNS Modifications
Going forward our DNS servers will only answer authorative requests for the external network (ie, Internet) and full answers including caching will be provided to internal servers (ie, hosted networks, ExchangeDefender, colocation customers, infrastructure partners). More specifically, we will not provide “recursive lookups” for external users and will only answer authorative requests from the Internet.
Background: DNS servers resolve friendly hostnames such as www.ownwebnow.com into IP addresses such as 65.99.192.50. The DNS server, in our case ns1.ownwebnow.com is said to be authorative for a zone (in our case ownwebnow.com) if it is the official provider of the information that matches the hostname to the IP address. When you use a DNS registry such as Network Solutions to register your domain, you enter a set of name servers (ns1.ownwebnow.com and ns2.ownwebnow.com) which will provide resolution, or be authorative, for that domain. Clients, including remote networks, computers, servers and more use their own DNS servers to resolve hostnames into IP addresses so computers can locate one another over the Internet. When a remote server requests a lookup from their local server the local server checks if its authorative for the domain (ownwebnow.com) and if it is not authorative it starts the recursion process – it first looks at its root hints to find the top level domain (.com) and eventually receives an answer from the authorative server (ns1.ownwebnow.com) which it sends back to the client. By disabling recursion on our name servers we stand to reduce the load and increase performance on our network because we will only be providing the DNS service to our customers, not everyone on the Internet.
Time Server Modifications
As of late there have been many issues with the public pool of NTP servers that help computers and networks around the world syncronize their clocks. To make matters worse, there are many issues with virtual machines and the horrible drift (difference between real time and time in the virtual machine) in time thats introduced with new technologies.
If you are Internal to the Own Web Now network you can use time.ownwebnow.com as your time server. It should (and so far statistically it has) answer the time syncronization requests 100% of the time. Our previous time.ownwebnow.com was a round-robbin implementation that simply aliased time.ownwebnow.com to the various military and research organizations that had public time servers. Over time, that infrastructure has become less and less reliable so we’re providing the time sync for you if you’re on our network. Just use time.ownwebnow.com and you’re all set.
That is all for now, we expect all time and DNS related work to be complete by July 15th but you are welcome to use them now to improve your performance. This will be a very seamless and transparent implementation for our entire user base but we wanted you to be aware of what we’re doing to keep up. As always, thank you for your business.
Offsite Backup Troubleshooting
As blogged here previously, our offiste backup maintenance is complete. Not only do we have the storage to sustain the growth for the forseeable future but things are running near flawlessly.
If you are still experiencing issues please open up a trouble ticket and we will help you get to the bottom of it. One problem some customers have reported is that backups do not seem to work automatically, namely, you receive an email saying “Reminder: Scheduled backup missed > Username > Backup set name”
If you receive this error please try following these steps (click) as they are the first thing we will try to do once the ticket is opened.
If you receive any other error, please open up a trouble ticket in our support portal.
We have seen a remarkable improvement in performance for backups and restores since the maintenance interval, and will shortly be announcing some additional services related to the offline backup service, mostly around the management, support and DR crisis management.
Backup Server Maintenance
The partner response to our new data protection service has been simply incredible. As a result the amount of bandwidth and storage we are in process of allocating to it is equally enterprising.
We are expanding SAN and network allocation to our public AhSay servers. As a result you will likely experience some performance issues until the new storage systems are in place and the additional bandwidth is brought online. This will not affect your daily backup processes dramatically as the AhSay agent is set to automatically retry/resume as availability changes.
We expect the work on our storage networks to be complete by early Friday AM, EST; 9/26/2007. We’re sorry about any inconvenience this may cause you.