Issue Report – 06/09/2011

Earlier today we again had reports of slow response times. In some instances the application was completely unresponsive. As yesterday, we performed routine corrective actions and were able to resolve the immediate issue. When the issue began to recur later in the day we were able to quickly resolve it. But, of course, we weren’t satisfied with these temporary fixes.

Deeper analysis of our log files revealed a very large spike in network bandwidth utilization during the incidents, which we traced to an unusually larger volume of API usage by a customer. We are working with this customer to ensure that their integration solutions do not risk system performance for other customers. To prevent this issue from re-occuring in the future we are working on measures to automatically throttle-back API calls by any customer in the event their usage is causing a system-wide problem.

Until this is in place, we will be actively monitoring the situation on a 24-hours basis looking for any indication of a problem so that we can take immediate corrective action before customers are affected. To ensure that we are immediately notified of other issues in the future, we have now implemented automated monitoring and alerting from the following global locations. And we have broadened the scope of this monitoring so it covers all functional areas within the application.

New York, New York
London, England
Mumbai, India
Sydney, Australia

We once again apologize for the problem and any inconvenience you may have suffered. We deeply appreciate those customers who provided us additional information to help isolate the problem.

Stephen Franklin
CTO and Co-Founder

Chris Hefley

Chris Hefley is a co-founder of LeanKit. After years of coping with “broken” project management systems in software development, Chris helped build LeanKit as a way for teams to become more effective. He believes in building software and systems that make people’s lives better and transform their relationship with work. In 2011, he was nominated for the Lean Systems Society’s Brickell Key Award. Follow Chris on Twitter @indomitablehef.