Upgrade Notice: Login Service Enhancements and Monitoring Improvements

Over the weekend, we designed, tested, and implemented new architectural solutions to address recent issues with the central login service for ExchangeDefender products. Additionally, we identified and began resolving a critical alerting issue that had prevented our NOC from receiving timely notifications about service outages.

To expedite improvements, we deployed a web cluster originally planned for a later release. This new cluster introduces advanced high-availability features, including self-healing capabilities and integration with modern, distributed monitoring solutions to ensure consistent global accessibility.

Given the scope of this upgrade, we opted for a phased rollout using A/B testing to ensure service reliability. Over the past three days, we’ve gradually increased traffic to the new cluster, starting at 12%, while monitoring server and load balancer performance metrics. Currently, 20% of traffic is routed through the new cluster, with the remaining 80% handled by the legacy system. In the event of a failure in either cluster, the load balancer will dynamically shift all traffic to the active system, even if a customer was initially pinned to the affected cluster.

Performance Improvements

The initial results have been highly encouraging, with noticeable performance gains. We’ve observed a 5x improvement in P95 latency and a 3x improvement in P99 latency compared to the previous setup.

Next Steps

Next weekend, we plan to implement the final phase of this upgrade, introducing automated transitions between data centers to address any performance or reliability issues proactively.

Addressing Notification System Failures

During our investigation, we identified a failure point in our notification system. Alerts were being throttled or discarded by our SMS gateway, particularly during cascading outages triggered by login server downtime. We’ve since refreshed our monitoring solution with modern analytics tools and implemented multiple alerting pipelines to prevent future disruptions. While we continue to work with our SMS gateway provider to resolve filtering issues, these changes significantly improve our ability to detect and respond to service issues.

Thank You for Your Patience

We sincerely appreciate your understanding as we worked to diagnose and resolve these challenges. We recognize how frustrating the repeated service interruptions have been and want to assure you that we’ve been actively addressing these issues with a focus on long-term reliability and minimal disruption.

Thank you for your continued trust in ExchangeDefender.