Upgrade Notice: Login Service Enhancements and Monitoring Improvements
Over the weekend, we designed, tested, and implemented new architectural solutions to address recent issues with the central login service for ExchangeDefender products. Additionally, we identified and began resolving a critical alerting issue that had prevented our NOC from receiving timely notifications about service outages.
To expedite improvements, we deployed a web cluster originally planned for a later release. This new cluster introduces advanced high-availability features, including self-healing capabilities and integration with modern, distributed monitoring solutions to ensure consistent global accessibility.
Given the scope of this upgrade, we opted for a phased rollout using A/B testing to ensure service reliability. Over the past three days, we’ve gradually increased traffic to the new cluster, starting at 12%, while monitoring server and load balancer performance metrics. Currently, 20% of traffic is routed through the new cluster, with the remaining 80% handled by the legacy system. In the event of a failure in either cluster, the load balancer will dynamically shift all traffic to the active system, even if a customer was initially pinned to the affected cluster.
Performance Improvements
The initial results have been highly encouraging, with noticeable performance gains. We’ve observed a 5x improvement in P95 latency and a 3x improvement in P99 latency compared to the previous setup.
Next Steps
Next weekend, we plan to implement the final phase of this upgrade, introducing automated transitions between data centers to address any performance or reliability issues proactively.
Addressing Notification System Failures
During our investigation, we identified a failure point in our notification system. Alerts were being throttled or discarded by our SMS gateway, particularly during cascading outages triggered by login server downtime. We’ve since refreshed our monitoring solution with modern analytics tools and implemented multiple alerting pipelines to prevent future disruptions. While we continue to work with our SMS gateway provider to resolve filtering issues, these changes significantly improve our ability to detect and respond to service issues.
Thank You for Your Patience
We sincerely appreciate your understanding as we worked to diagnose and resolve these challenges. We recognize how frustrating the repeated service interruptions have been and want to assure you that we’ve been actively addressing these issues with a focus on long-term reliability and minimal disruption.
Thank you for your continued trust in ExchangeDefender.
Unlock the Power of Free Email Backups: This Summer with ExchangeDefender LiveArchive (Take Survey!)
The new ExchangeDefender LiveArchive has been very popular with our clients as a live email backup service. With backups typically being the last thing organizations want to spend their IT budget on, it’s no surprise that a free backup service that you own and can archive indefinitely would be compelling.
We’re trying to help our partners and clients roll out LiveArchive this summer and we would like to hear from you. We’re organizing our summer webinar series and this 3-minute survey would help us focus the schedule and content to serve you better:
To help as many of you get started as fast as possible, ExchangeDefender is considering several short-form or small-group interactive webinars several times a week during July and August. During these webinars, we’ll cover the basics of the service and help you actually complete the deployment of LiveArchive (it only takes a few minutes with the public cloud)!
What is Live Archive Email Backup?
ExchangeDefender LiveArchive offers secure backup, long-term archiving, and eDiscovery services for Office 365, Gmail, and cloud mail.
Sales Collateral? Yes! We have a Data Sheet, Live Archive Email Backup Advantage sheet, plus case studies!
ExchangeDefender Supernet
Last month we announced a major upgrade and expansion of our network to better serve our clients in a more challenging cybersecurity world. I’m sure you’ve seen many stories in the news about cyber attacks and how some groups and nations are expected to attack our critical infrastructure.
We can assure you that those threats are real and are ongoing in a very focused fashion. In order to prepare for a more massive attack, we’ve had to rely on some BGP routing magic to make ExchangeDefender far more resilient.
Make sure you allow inbound SMTP traffic from ExchangeDefender’s 65.99.255.0/24 (255.255.255.0) range.
This range has been in use by ExchangeDefender since 2003 so if you’ve followed our deployment guide correctly you should be all set. If you’ve chosen to deploy ExchangeDefender differently and have other scanning/security active on that range, you might see email delivery delays and failures. Easy fix, just add the whole class C.
What is happening under the hood is that all of our different data centers are routing traffic via the same 65.99.255.x range. Even if half of our data centers disappear due to a telecom or power event, we will be able to continue email delivery.
As you’ve seen over the past year, we’ve focused on Inbox, LiveArchive, and upcoming Replay features to improve security and reliability. Like you, we wake up every day to another Exchange/Gmail event/issue/policy/fubar and nobody likes losing email or the ability to communicate. This is why having ExchangeDefender around your email infrastructure is critical if email is critical to your organization. The new supernet has been routing messages for over a month with no issue and on Wednesday, May 15th we will make it available for everyone.
ExchangeDefender Network Upgrade: Supercharged Security and Disaster Recovery for Your M365 Clients
At ExchangeDefender, we’re obsessed with keeping your clients’ emails safe and secure. That’s why we’ve just completed a massive network upgrade, designed to empower you, the IT Managed Service Provider, to offer unparalleled disaster recovery and enhanced security for your M365 clients.
Why the Upgrade? The Cloud Needs Saving Too!
Remember the good ol’ days of on-prem email outages? Well, guess what? The cloud isn’t immune!
- M365 Outages Happen: We’ve all been there – clients waking up to a down Office 365 tenant. Scary stuff.
- Data Loss Lurks: Lost emails due to strange glitches or language barriers (seriously, Croatian?) are a real threat.
Building on LiveArchive, we’re massively expanding our disaster recovery businesses due to increased demand for client protection on the Microsoft M365 network.
Our upcoming live mail caching service is here to save the day! We’ll cache copies of your clients’ emails for 24 hours, ensuring no email gets lost due to infrastructure issues. Need a message from the past day? Simply request a redelivery – problem solved!
Network Upgrade? More Like a Network Revolution!
To make Live Mail Caching a reality, here’s how we did it:
- Open-sourced LiveArchive: This empowers you to deploy massive email archiving solutions for your clients on their own cloud.
- Infrastructure Overhaul: We’ve upgraded nearly every switch, firewall, core router, and most of our backend to handle the massive data demands of live caching.
- Network Failover on Autopilot: BGP magic ensures seamless service delivery across multiple data centers, automatically routing around network issues and regional outages. No configuration changes on your end are needed!
In the coming quarter, we’re excited to unveil all these new services.
While email remains a prime target for cyberattacks (not a good thing!), our core focus continues to be protecting traffic and identifying threats. However, our M365 clients are facing a surge of issues and are turning to us for solutions.
For instance, when Microsoft experiences delivery delays or mail latency problems, our ExchangeDefender Inbox service becomes the go-to solution for accessing critical OTP/MFA tokens.
Many clients have also adopted Inbox as their preferred webmail client due to its speed and lightweight design.
As Microsoft prepares to sunset Basic Authentication, many legacy SMTP services and mail-enabled applications will become inoperable. To address this growing demand for reliable and secure email routing and delivery, we’ve significantly upgraded our network and services this past quarter.
Email delivery problems happen.
With ExchangeDefender, however, these problems won’t impact your operations. That’s our unwavering commitment, and we’re thrilled to announce the expanded ExchangeDefender network is here to serve you and ensure the continued security of your email.
P.S. Sounds expensive and boy was it!! We’re in an era where any excuse is used to raise prices, so this might come as a bit of a surprise: There will be no change in pricing as a result of all these upgrades. Thank you for keeping us in business in our 4th decade and thank you for trusting us with your email.
Enhanced MFA Enforcement Policy: Strengthening ExchangeDefender Security Measures
For years, ExchangeDefender users have enjoyed enhanced login security via multi-factor authentication security (aka MFA, 2FA, OTP). In our March feature update, we hope to improve your security and enhance MFA enrollment to keep you and your data safe. Allow us to introduce to you the new MFA Enforcement Policy!
Note from the boss: Before we get to any discussion of policies, our official recommendation is to enforce MFA on every service we provide and to rotate passwords at least once a quarter. I know, nobody likes the second validation prompt but this is a standard in the industry and I can’t think of a bank or a vendor that doesn’t require it. Maybe I’m jaded because we’re a cybersecurity company.. and with all the layers of MFA/VPN/auth we have in place I spend an insane amount of time trying to find my key or wait for the new pin to display in my authentication app. Because when I interact with a vendor that holds my information and they have no login security.. all I can wonder is what else they’re not doing to keep my data safe? Is this a real business or some WordPress plugin?
At the same time, I understand we have a ton of customers in SMB space where sometimes (obnoxious) tech can be slow and difficult to implement. But you pay us to keep you safe – and to keep your backups safe, and to keep your business continuity safe, and to do that we absolutely must require MFA. But we also can’t expect staff who have clicked on a Release/Trust link for over a decade to be cool with suddenly being forced into MFA enforcement workflow on Monday. So we designed a compromise. I hope it fits your organization and I hope you adopt it as fast as possible.
Sincerely,
–Vlad Mazek, CEO Own Web Now Corp.
In March 2024 you will have the ability to enforce or require MFA enrollment at the domain level. What this means is that you’ll be able to require MFA enrollment with an authenticator app for everyone with just one click at https://admin.exchangedefender.com:
With this policy, you can secure your users’ login with multi-factor authentication without making them enroll a device.
How does it work?
When your users go to admin.exchangedefender.com and attempt to access Inbox, WFS, or LiveArchive we will check their MFA enrollment and if none is found we’ll just tell them we have to verify their identity:
When they check their mailbox they’ll see an email from ExchangeDefender and just provide the code back.
This way your ExchangeDefender login is technically secured at ExchangeDefender with MFA even if you haven’t enrolled a device in MFA yet. There is no way to get into the secure areas of your account without enrolling into MFA. But what if someone guessed the password, they can just enroll a device and hijack the account, right? No.
When you set MFA Enforcement to Required/Forced, all your users are automatically set into an MFA mode that relies on their email address for secondary verification. When they visit admin.exchangedefender.com for the first time and provide their username and password the system will check their MFA enrollment and if email MFA is detected the system will send them an email with a verification code to proceed. This way we’re using the email MFA as a way to verify their identity and then we enroll their authenticator app as usual by scanning the QR code.
ExchangeDefender Quarantine Reports behavior will not change at all as a result of the change in the MFA Enforcement policy. ExchangeDefender Quarantine Reports do not rely on the login system at all as the user is never prompted to authenticate to trigger the release of the message. Instead, quarantine operations are tokenized and one “release” request does not automatically grant any other release or trust requests with the same token. It also doesn’t automatically log them into ExchangeDefender so there is no getting by the MFA!
Secure your logins now
It’s 2024 and the world has changed when it comes to authentication and identity when dealing with services online. Experiencing a cybersecurity threat or compromise is an incredibly expensive and frustrating experience that can easily be avoided by requiring an ID check before granting access to sensitive information. ExchangeDefender MFA Enforcement Policy is your way to protecting your organization from unauthorized and unverified access.
Improved SPAM Release Security: Quick Release
ExchangeDefender is giving users more power to lock down their valuable data. Now that ExchangeDefender handles business continuity and backups for M365/Gmail tenants there is even more information in ExchangeDefender that demands more flexibility with security policies. First, let’s talk about the upcoming feature that allows you to lock down your ExchangeDefender SPAM Quarantine Report activity.
Bit of background: ExchangeDefender Quarantine Reports are an immensely popular ExchangeDefender feature (coming up for an upgrade this spring btw!) that sends users a list of quarantined messages with a set schedule. Users tend to rarely look in Junk Items or review SPAM unless they are waiting for something so this is a cool feature that our users just love. Scroll down the list of quarantined messages and release or trust just by tapping the link. Super convenient, but does it meet your security requirements?
ExchangeDefender Quick Release feature now enables you to choose between convenience and a more secure release process. For many organizations, having the message released or get added to trusted senders with just a click is a huge time saver and user convenience. But if your Microsoft M365 / Gmail account gets compromised (which happens ALL the time) or you deploy a new security/business/CRM (mostly AI stuff) that scans links then this “convenience” can turn into an Inbox packed with SPAM messages that some hacker/service inadvertently released.
If this happens to you, know that ExchangeDefender can help with the “Secure Release” setting. By enforcing Secure Release, when the user clicks to release or trust a message they will be prompted to authenticate before they can release/trust the message. This way if you get hacked or install link scanning/crawling software in your tenant will not be able to access the messages without a password.
Now please, go setup your favorite authenticator app with ExchangeDefender MFA (next up, you can set it to be mandatory/required)
How To Get LiveArchive Web UI
By now we hope everyone who needs a backup has started it via ExchangeDefender LiveArchive and as previously mentioned we’ve got a huge new feature to show off this Wednesday:
During this event, we will discuss the ExchangeDefender LiveArchive Web UI, the open-source eDiscovery platform for your LiveArchive email backups. This stack can be hosted anywhere, even on your PC as in the example below. With database services holding metadata and the location of the email message on S3 storage, the LiveArchive user interface is very similar to webmail platforms. We even have themes that are inspired by GMail and M365 platforms so users can access their backups in the interface that feels similar to the one they are familiar with.
We know a lot of our partners and clients like to tinker, so if you like to get ahead here is a sample docker-compose.yml you can use to launch LiveArchive Web UI right now (production branch):
services:
app:
image: public.ecr.aws/y9g1h8n8/exchangedefender/livearchive:latest
extra_hosts:
- 'host.docker.internal:host-gateway'
ports:
- '${APP_PORT:-80}:80'
volumes:
- livearchive-config:/app/storage/app/settings
- livearchive-loadbalancer:/data
environment:
LIVEARCHIVE_PERSISTENCE: browser
networks:
- livearchive
volumes:
livearchive-config:
livearchive-loadbalancer:
networks:
livearchive:
driver: bridge
Then simply point your browser to the container or if you’re running it on your PC http://localhost
That should give you everything you need to access LiveArchive backups with a modern and responsive eDiscovery frontend you own – we should discuss more details about deployment, different cloud hosting options, best practices, and different ways you can use LiveArchive Web UI. This should be plenty to get you started and get your questions ready for the live webinar this Wednesday, February 7th. and we look forward to seeing you there.
P.S. If you’re looking for old LiveArchive business continuity features, they are now a part of ExchangeDefender Inbox! Dive into the new world of ExchangeDefender Inbox, our smartest, most advanced email outage protection. Unlock insights on top features and highlights with our new brochure!
Getting Ready for LiveArchive UI – What You Need to Know!
ExchangeDefender will be launching the LiveArchive Web UI during our webinar next week and we are looking forward to showing you how to launch it with a single command! In the meantime, this blog post is intended to give you a heads-up about the requirements and functionality so you can make design decisions.
Docker & Design
In a recent post, we discussed why MariaDB/RDS is required for the LiveArchive Web UI: it’s where we store the message metadata such as sender/recipient/subject/etc. These elements drive the UI and enable users to locate messages, search, and complete eDiscovery and email recovery tasks.
ExchangeDefender has organized the entire LiveArchive Web UI into a single container that can run on your own docker on a workstation or NAS in your office, across a wide variety of virtualization products and services, as well as public cloud like Amazon Web Services and Microsoft Azure. This makes ExchangeDefender Web UI easy to update, easy to manage, and easy to tweak to your requirements.
There are two ways to configure and start the container: preconfigured with environment variables or on-demand browser configuration. If you start the container without the environment defined you will be presented with a web configuration wizard that will prompt for S3 and RDS credentials. If this is the first time you’re deploying LiveArchive Web UI or just want to test it, this is the best way. Once you close your browser all the configuration vanishes and nobody else has access to your mail.
After you’ve configured everything to your liking and are looking to put the service into production, place the appropriate information in the container environment variables, and the container will always launch in production mode and bypass the wizard configuration.
Authentication
ExchangeDefender Web UI was designed to facilitate your email backup and eDiscovery needs. Our experience in compliance archiving and long-term email archiving has allowed us to work with countless organizations and one thing they all have in common is that they all have their own unique access and control needs.
ExchangeDefender Web UI by default presents all the available mailboxes and each email address has its own path. Using this predictable data storage process your Web Application Firewall can easily be configured to include or exclude data by path alone.
We designed the solution so it can be launched quickly, accessed, and managed without a lot of technical skill, and so it can be quickly modified/optimized for production. LiveArchive offers a lot of solutions to modern email problems and the flexibility means you can run different LiveArchive Web UI for different personnel or different tasks.
Resources & Customization
ExchangeDefender Web UI is completely free and open source. This means you can download it, modify it, and use it freely.
It also means that the solution will live even after ExchangeDefender as an organization is gone. You will not find any references or callbacks to our network and all the protocols are fully documented. This enables you to truly craft a failover email solution that can be completely disconnected from the Internet and placed into cold storage / safe.
Resource-wise the container is a little more than a web server and you can run hundreds of users with even the minimal 1 cpu / 1gb ram. This is possible because the SQL workload and data storage are handled by other services.
We hope you’re as excited about the launch as we are. Please join us for the webinar to see how it’s done and we’ll even help you set yours up right after the event! Just think of a good subdomain to point to your new LiveArchive backup platform.
Announcing LiveArchive.Next
We’re excited to introduce you to our next version of LiveArchive: a long-term email archiving and backup solution that you can take with you. With the new LiveArchive we had to solve some unique challenges our clients face in 2023 and beyond:
1. We don’t have any room in our IT budget (and need to save $)
2. If we get compromised our backups will get crypto locked too
3. We have to protect and backup our email for years
We heard you loud and clear and the LiveArchive.next was designed from the ground up to give you the peace of mind that you’ll have access to your critical data even if Microsoft wipes your account, even if you get crypto locked out of your network, even if we go out of business, and yes it can also do it for you nearly free of charge.
Choice of Storage
LiveArchive.next uses object-based storage backend that is widely used and available for free with full source code. You don’t have to worry about the product getting discontinued or the company going out of business and most modern IT workers will have experience/certification for it. ExchangeDefender provides documentation on how to set a service up with Amazon Web Services using the S3 object storage if you want to keep your backups in the cloud. We also provide docker documentation for those who want to keep the backups on their own Synology NAS or as affordable as a Raspberry Pi.
There is no “one size fits all” when it comes to how you protect your backups and the point is to have flexibility that will fit your needs. LiveArchive.next will accommodate you no matter where you point it. Some organizations may rely on LiveArchive in the cloud to reduce complexity while others will point LiveArchive to their office NAS where they can save years of mailboxes and terabytes of data on an appliance/drives that can be taken offsite like old-timey tape backups you heard about in in IT history class.
Choice of Database Options
LiveArchive.next can optionally store your message metadata in an SQL database. While this is an optional part that isn’t required for backups to work, it does make LiveArchive very user-friendly. Without the database, all your messages will still be backed up and you will still be able to retrieve them directly from the S3 object storage bucket through a friendly web user interface or S3 client.
With Amazon RDS or MySQL configured, ExchangeDefender will also send message metadata to your database: sender email address, recipient email address, subject, date, etc. This information is used by our LiveArchive Web UI service to show users the friendly webmail interface you’ve become accustomed to.
This feature is optional but highly recommended by ExchangeDefender and you’ll appreciate it immensely the first time you’re sent on a quest to retrieve an email from 4 years ago.
Choose to be Live
Most backups are still performed as a scheduled task / job and only run periodically. When there is an issue the alert may or may not be sent to your IT staff and responsible IT providers have staff that frequently manually checks and verifies backups.
ExchangeDefender LiveArchive is live, meaning that a message that was received in your Outlook will be received in your LiveArchive archive in seconds instead of hours or days. There are no jobs or backup tasks to monitor or review – and we’ll notify you the second we identify a problem.
When you enable database services your users will be able to access their LiveArchive in realtime as well, so for those of you trying to save $ on IT this will help you to diversify some of the monitoring and risk away.
Additionally, everything being live means that our service will backup and archive email the moment it arrives so there is no job to set up, no new mailboxes or services to enroll. If the mail is hitting a mailbox in your organization you can expect it to automatically show up in LiveArchive in seconds.
To find out more about the next LiveArchive please join us for an informative webinar on November 8th, 2023. Take the moment to sign up for the ExchangeDefender LiveArchive Migration Service if you’d like us to port your existing LiveArchive data to your next LiveArchive. Check out all the documentation we’re putting out there and get in touch with us early because this service is in heavy demand and December quickly books up with EOY projects.
We look forward to helping you expand and improve your email protection in 2024 and beyond!
Sunsetting LiveArchive
On December 31st, our current version of LiveArchive will be decommissioned. Inbox, a business continuity solution we launched last year, has already taken the workload of LiveArchive and it does the job better, faster, with fewer clicks.
LiveArchive served our client base well for over a decade and we’re thankful for all the disasters it’s saved us and our clients from. Now that we’re looking at 2024 and beyond, LiveArchive must solve new problems. For starters, most email is no longer hosted on low-grade hardware in SMB offices managed by part-time hobbyist IT: It’s now professionally managed in high-end data centers. The primary concern is no longer “What if my T1 Internet connection goes down?”; “BACKUPS ARE OUR RESPONSIBILITY” and keeping all your eggs in one basket is never a good idea.
COVID and the work-from-home era have only exacerbated the problem of how quickly (if at all) you’ll get your email back when the disaster occurs. Cloud operators are vague in their data protection statements and there is no way to audit it. Backup tools and services similarly offer few guarantees and the supply chain attacks have only gotten more prominent.
New LiveArchive Migration Service
New LiveArchive is designed to help solve the 3 problems clients have with protecting cloud email:
1. We don’t have any room in our IT budget (and need to save $)
2. If we get compromised our backups will get cryptolocked too
3. We have to protect and backup our email
ExchangeDefender LiveArchive.next webinar on November 8th, 2023 covered exactly how the next version of LiveArchive is going to help you solve all of these problems.
Furthermore, we announced a LiveArchive Migration Service for our clients who wish to have the LiveArchive data ported to the new LiveArchive. Because LiveArchive is IMAP based we can pull existing LiveArchive data into the new version. We can use the same IMAP process to bring over mailboxes hosted on any other IMAP accessible (M365, Office365, Gmail, Exchange, and virtually every legacy email service).
In order to get your data migrated all you have to do is configure your new LiveArchive service and put your ticket request in by December 1st, 2023. We take care of everything else and to reward our loyal clients over the years the service will be provided free of charge (est $499 value).