The Server Lab



  • My employer is an accounting company, where I work in the security services industry group. We provide generalist and technical security consulting, and I happen to be a manager for the technical testing. Like many other security groups we have a lab, and it's segmented from internal IT for all of those important reasons. In fact the firm's IT group is scared to death of us so much that they ask to be told when someone is considering leaving the company so that they can be treated as a hostile employee.

    So our lab happens to be in a 10x6 storage closet. For quite a while we didn't have dedicated cooling. In fact, during the winter we have dedicated HEATING of the lab. Most of our "server" equipment consists of commodity PCs that have been handed down by firm acquisitions. They're usually 3rd hand by the time they get to us. But we essentially have all sorts of desktops running our security tools, websites, databases, etc. About two years ago we started getting decent budgets from IT for new purchases, and started buying blade servers, and practically filled up a rack. Of course we still run the other systems, because the lab manager hasn't had enough time or energy to decommission the systems. Additionally the "lab" is a complete pig sty with empty boxes all over the place, equipment taken out of storage containers and strewn across our lab benches, stacks of unused workstations and laptops all over the place. I won't even mention the hideous cabling messes.

    About a year ago we started having serious heating issues with the added equipment. Since the lab was connected to the normal building air conditioning they AC would get shut off after hours and our lab would heat up out of control. One weekend had to go into the office to collect some equipment for a wireless security test the following Monday and was greeted with door handles that were practically too hot to touch. After entering the lab I could barely breathe, let alone touch anything that was metal. Once I grabbed the equipment I went to my office and typed an urgent email about the heating situation, and how something needed to be done immediately.

    The response from the principals in my group were not very positive. In particular, one principal got angry enough at my email to publicly yell at me for the unnecessary "unprofessional" urgency of the situation, because at home he just shuts off his computers and we should be doing that too. I tried to explain to him (since he basically runs our industry group) that we have scans running from these systems and other operations that we just can't have downtime on. That the cost of adding AC to the room would far outweigh the cost to our lost productivity if the servers die, let alone the $100k+ price tag to replace damaged equipment. However I was branded as a moron who "just doesn't get it" and we added a thermometer to the room to record the temp so it could be "proven" that there wasn't a problem. During business hours we were to leave the door propped open as well. This would allow anybody at the firm, including anybody who snuck in from the street to have access to all of our equipment, including forensically acquired images of hard drives that effectively invalidates the chain of custody.

    Well the principals learned that the lab was reaching into the 120's and that we already lost several of our legacy equipment from hard drive death. That was written off with a "well it was going to die eventually". None of our newer equipment was capable of powering itself down after a certain temp was reached, and the fans would run at full speed for weeks on end. Finally (after over a month) the powers that be decided that they should talk to building maintenance to get some cooling because powering down our systems to keep the temp below 95 was actually causing problems with running the business. It took another month before cooling was added, and several more weeks before it worked correctly. All in all we only lost 5 of our most critical systems and none were the newer hardware. Once everything was said and done the principals had a talk with me to explain to me that the death of those systems could have been prevented if I would only have acted quicker and with more urgency. Thanks guys.



  •  I hope you brought a copy of the email you sent earlier to that talk!  At least that email means job security.  Print it out, date it, sign it and lock it up!



  • The sad thing that I've learned about this company is that they don't believe in job security. They've fired people in the past when their managers screw up at a client because the need a fall guy. Luckily I fell into the whipping boy position in the past year as a result of backing off on my workload from 80hr weeks to 60hr weeks. :)



  • @octal said:

    The sad thing that I've learned about this company is that they don't believe in job security. They've fired people in the past when their managers screw up at a client because the need a fall guy. Luckily I fell into the whipping boy position in the past year as a result of backing off on my workload from 80hr weeks to 60hr weeks. :)

    Luckily?

    I've said it time and again: managers should do the right thing, but when they inevitably don't, make sure you have a politely but strongly worded email to/from them indicating the problem, your concerns, and instructions to do <whatever> their way, in order to CYA!

     



  • That the cost of adding AC to the room would far outweigh the cost to our lost productivity if the servers die, let alone the $100k+ price tag to replace damaged equipment.

    Well, there's your problem.  The cost of adding AC to the room would be far less than the cost of the lost productivity and hardware replacement cost.  You told them that A/C would cost more than downtime and new equipment.


Log in to reply