Timeline

On Monday, December 8, 2025 at 7:19 PM CST, our primary database began failing to complete requests. This created a cascading impact across Nexus, Scorekeeper, and Member Services. Scorekeeper users experienced a brief delay in symptoms (up to five minutes) due to cached data still being valid. By 7:24 PM, all users were affected.

Alerts fired immediately, and staff escalation began. The Tech Team was fully engaged by 7:35 PM. Interestingly, the database resumed responding around 7:33 PM, shortly before full staff involvement. This automatic recovery behavior is expected; however, performance remained suboptimal for approximately 20 minutes as we continued the diagnosis. We identified a stuck database process, restarted it, and system performance returned to normal by 7:51 PM.

• Total user-visible downtime: approximately 9–14 minutes
• Total degraded performance window: approximately 32 minutes
• Users could use the system during the degraded phase

Findings

After a detailed investigation over the following days, we isolated the root cause. The issue originated from the Tournament Management System (TMS) setup workflow in Nexus. During the TNT Setup Wizard, users can choose to sync a Tournament to Nexus or export it to TMS. A sync to Nexus operation for a mid-sized Tournament triggered a performance issue that led to prolonged locking of user-related records. These locks prevented Team Roster lookups from completing, which in turn caused failures in Scorekeeper and Member Services.

Once the sync process ended — it ultimately failed — the database was able to serve Team Roster queries again, and the system began to recover.

However, by that point, the API layer had entered its own recovery sequence. Because the APIs could no longer reach the database, they assumed they were unhealthy and began replacing themselves. Improvements made after an incident in September significantly reduced the time required for this replacement cycle, enabling a quicker restoration of stability. Overall, core functionality was restored within about 15 minutes.

The remaining period of degraded performance was traced to an internal email-alert database process that had entered an unhealthy state. Restarting that process completed the recovery.

Corrective Actions

• The Tournament sync process has been temporarily disabled for general use. Only APA staff can execute it while fixes are in progress. TMS export remains available for all users.
• Performance issues within the sync routine are being addressed.
• Once validated and re-tested, the Tournament sync capability will be re-enabled for all users.