Around 9:05 PM CST support staff alerted tech staff that issues with 9-ball score had been reported. A few members of the Tech Team engaged within minutes. During the investigation we found a database request that was performing poorly and had been since around 8:30 PM CST. In all fairness it had been performing poorly all along, but when the volume of Scorekeeper activity reached it’s peak the database server started buckling under the pressure of this poorly performing request. This database request was taking too long to return and resulted in users getting errors like 500: Internal Server Error, Gateway timeout, and Cannot return null for ....
We applied a fix that resolved the poor performance around 9:11 PM CST and those errors stopped. Performance returned to normal.
While this type of anomaly is uncommon, it can happen from time to time. That said, we are looking at ways to catch this type of issue before it can impact our users.