Levi9 had the first go-live for a core product developed for a customer. The go-live was after business hours on a Thursday and the customer expected everything to work on Friday morning. “After the deploy, we found a long queue of unprocessed tasks in the system causing the software not to work.” 

To help meet the deadline and find the solution, the whole team agreed to stay and even called the architect for support during his vacation. At first bug was suspected, but the problem could not be duplicated, and no bug was found in the test environment.

After 5 or 6 hours, the team finally discovered the issue was with the interval of time set between tasks. “We had thousands of tasks waiting to be processed and the timeout was pausing for one full minute with every task. The only solution was to shorten the timeout interval and wait for the tasks to be completed.”

Everyone who worked late received overtime pay, but that’s not all. “Levi9 also ordered dinner for us, arranged taxis to take everyone home, and we were given the next day off.”

Before the next implementation, they knew better and set the timeout to 5 seconds.