Scorekeeper service disruption on September 24

On Thursday, September 24, Scorekeeper and other APA websites and tools were slow or unresponsive from 7:03 PM to 7:14 PM Central Time. Service was back to normal by 7:14 PM.

What you noticed

The Scorekeeper app was worthless. It would just stare at you and refuse with errors as its excuse. If you were lucky, it turned yellow and told you it couldn't do anything. Actually, all apps stopped working during this time.

What happened

An antivirus 🤦🏻 scan began on the database server at 7:00 PM. Shortly afterward, it used most of the server’s processing power for about eight minutes. Activity log updates fell behind, delaying scoring and other requests. The extra load ended at about 7:12 PM. The backlog cleared, and service returned to normal by 7:14 PM.

When a score action is taken, the system also writes to an activity log on the same database server as league and scoring data. The activity log receives more updates than any other part of that database. There's a veritable firehose of information being fed into it as it tracks every scoring action the scorekeeper takes. This activity appears to have been the greatest victim of the antivirus activity and essentially started blocking all other Scorekeeper traffic.

What we’re doing

We will be modifying the antivirus schedule to run during off-peak hours. We think this was an oversight caused by the server's time being UTC instead of Central, so we likely thought it was scheduled for midnight Central time, which is well past our peak utilization time. But, midnight UTC is 6 or 7 PM Central. Not good. We’re also adding more exclusions to avoid scanning all database-related files, rather than just some.

We had already identified the activity log as a likely source of slowdowns on busy league nights and planned to move it to its own database server. That change will prevent activity log traffic from competing with scoring for processing power on the same server. But now that we've discovered our antivirus scan is the primary issue, the activity log now looks like a noisy victim. We still plan to move that storage to another server.