Ah this reminds me of a perl app I was in charge of. It was this beautiful multi-threaded application. Ran clustered across 7 machines. You really have never seen a perl program written this beautifully. I know even the idea is an oxymoron for some...
After a while we started noticing that around the same time of night the queuing system would go completely out of control. It would suddenly decide that all the tests we put in to keep it from requesting more jobs then it could handle were ignored and it was just sucking in as much work as it could get.
We tracked this down to a daemon loop like this:
while($running) {
if(isWorkToDo()) {
doWork();
}
sleep(5000);
}
Turns out that perl's sleep function call doesn't actually sleep under load and so we had a beautifully hidden race condition. We had logs that had log4perl timestamps and I can't tell you how long we stared at the timestamps in the logs, not believing that the program was actually asking for 10 jobs in the same second. For those who might suspect their own programs the solution is:
use Time::HiRes qw(sleep);