Summary

On Saturday, October 11th, we ran into unexpected system issues while performing a scheduled upgrade to one of our Domain Controllers (DCs), a core part of our network that helps systems find and communicate with each other.

Our goal was to improve reliability, but the upgrade caused an outage that made Nexus, Member Services, and Scorekeeper unavailable for everyone starting around 10:10 AM Central.

We initially thought the maintenance wouldn’t cause any downtime because we had carefully pointed all of our static servers to a backup DC before starting the upgrade. However, we later discovered that some of our systems, specifically the servers created automatically by AWS to scale our applications up and down, were still using the DC that was taken offline. When that DC went down, those systems couldn’t locate what they needed to operate, which triggered the outage.

At first, our staff focused on finishing the Domain Controller rebuild as quickly as possible so it could be brought back online and put an end to the outage. Unfortunately, in the rush to complete that work, several errors occurred, and the rebuild process itself failed multiple times.

At the same time we were looking for workarounds and ended up implementing one by manually reconfiguring those servers and network settings to use the working DC. By 2:30 PM, services were restored, and systems were stable again. We believed everything was back to normal and officially closed the issue around 7 PM once the new DC had been rebuilt properly, or so we thought.

Unfortunately, around 9:55 PM, when we reintroduced the new DC into service, the problem returned because it hadn’t been properly synchronized with the rest of the network. Once we realized that, we reverted the change and had systems stable again by 10:45 PM.

The newly upgraded DC has since been repaired and validated. We’ll be bringing it back online carefully later this week, during early-morning hours when few users are active.

We know this outage was frustrating, especially since it followed a couple of recent disruptions. We expect more of ourselves and you should too. We’re taking concrete steps to address the gaps we’ve uncovered. Each incident has taught us something valuable, and we’re using those lessons to harden our systems and processes moving forward.

Technical Details

Timeline

• 10:10 AM: Outage begins after maintenance starts on one of the DCs. Nexus, Member Services, and Scorekeeper go offline.
• 10:10 AM – 2:30 PM: DC upgrade fails extending outage. Investigation identifies DNS failures as the root issue. AWS Elastic Beanstalk servers (which auto-scale application instances) were still referencing the DC being upgraded.
• 2:30 PM: DNS settings updated to point only to the healthy DC. Services restored.
• ~7:00 PM: Issue considered resolved since we believed the DC upgrade was complete and it was fully operational again.
• 9:55 PM: DHCP Option Set reverted to its original configuration to reintroduce the newly upgraded DC as primary. Outage recurs.
• 10:45 PM: Change reverted again; services restored.
• Following business day: Investigation reveals the rebuilt DC still contained incomplete replication data, preventing proper DNS resolution.
• Later in the week: DC repaired and validated; recommissioning planned for early-morning hours later this week.

Root Cause

The incident was caused by a DNS configuration oversight and insufficient domain knowledge regarding how DHCP Option Sets and AWS Elastic Beanstalk instances resolve Domain Controllers.

While static servers had been correctly updated to point to the healthy DC, dynamic instances launched by AWS retained old DNS settings. When the primary DC was taken offline, these instances lost their ability to resolve network names, causing a cascading service failure.

The secondary outage that night was due to corrupted replication data on the rebuilt DC, which rendered it unable to handle DNS requests correctly once it was made primary again.

Resolution

• Elastic Beanstalk servers were manually updated to reference only the healthy DC.
• The VPC’s DHCP Option Set was changed to exclude the faulty DC.
• The bad DC was rebuilt, revalidated, and tested successfully in isolation.
• Reintroduction is planned for early-morning hours once full stability is confirmed.

Lessons Learned & Preventive Actions

1. DNS and DHCP awarenessWe learned that DHCP Option Sets are pivotal to our DC maintenance. This knowledge is now built into our maintenance planning process.
2. Monitoring improvementsWe’re implementing automated Domain Controller health checks to alert us if a controller stops properly syncing or responding to DNS requests. This would have surfaced the issue earlier and prevented the second outage.
3. Maintenance checklistsAll future infrastructure changes will follow a mandatory checklist that includes all step required for a successful update, verifying dependencies, defining a clear backout plan, and confirming that all dynamic and static systems are covered.

Zero-downtime disciplineOur long-term goal remains zero-downtime maintenance. These recent incidents have reinforced how essential that standard is and we’re building the operational discipline to achieve it consistently.