Earlier today we again had reports of slow response times. In some instances the application was completely unresponsive. As yesterday, we performed routine corrective actions and were able to resolve the immediate issue. When the issue began to recur later in the day we were able to quickly resolve it. But, of course, we weren’t satisfied with these temporary fixes.
Deeper analysis of our log files revealed a very large spike in network bandwidth utilization during the incidents, which we traced to an unusually larger volume of API usage by a customer. We are working with this customer to ensure that their integration solutions do not risk system performance for other customers. To prevent this issue from re-occuring in the future we are working on measures to automatically throttle-back API calls by any customer in the event their usage is causing a system-wide problem.
Until this is in place, we will be actively monitoring the situation on a 24-hours basis looking for any indication of a problem so that we can take immediate corrective action before customers are affected. To ensure that we are immediately notified of other issues in the future, we have now implemented automated monitoring and alerting from the following global locations. And we have broadened the scope of this monitoring so it covers all functional areas within the application.
New York, New York
We once again apologize for the problem and any inconvenience you may have suffered. We deeply appreciate those customers who provided us additional information to help isolate the problem.
CTO and Co-Founder