Lunch or Servers, Take your Pick



  • Based on reading @Frank's experience living in a server sauna room, I thought I'd share some WTFery about my project's server room(s).

    Our original office suite was in the "nicer" set of buildings, ya know, glass and steel building instead of concrete. Well, at least the outside of the building looked nice. The suite itself was probably half of the floor, so a standard L-ish configuration. Two kitchenettes with full size fridges, sinks and microwaves, a major conference room and a back door "draft" conference room, bull-pen area, not too bad at all. Sort of.

    The office had two breakers, one for the front half; one for the back half. The back half was our two testing labs, server room and developer offices. So 3 full size racks of servers, about 12 user(dev/tester/whoever) laptops and about 20 test laptops once we ramped up. Nothing that should stress power too much. Technically the servers should have been on their own breaker but whatever, we didn't have any trouble.

    Once we started to ramp up the project however... A few weeks in we would regularly have the breakers flip during the day. So hearing 'God-damn-son-of-a-bitch' in the middle of the afternoon got to be pretty common place because it is hard to access a SAN/VMs/Network-resources when they goes down once a day.

    Turns out we'd trip the breaker around lunch when we did lose power. Remember how I mentioned the two kitchenettes? Well it was the microwave in the back half that would push the breaker overboard. Yeah I know, TRWTF is the servers weren't on their own breaker. By time we figured that out management had decided we were going to move because you can bill more if your server room is bigger or something.

    So we moved into a "better" suite(ie across the street but in the same complex and shitter. It does have a dedicated breaker for the server room). Problem with the newer space? The original HVAC was apparently powered by hamsters. Since we came to our new space the HVAC has died at least 6 times, and at least 4 of them the lab team didn't find out until developers sent email complaining systems were offline. Server room had gotten to 94F+ (34.4C) in those cases. We stopped a few meltdowns because the dev test-lab was on the other side of the wall to the server room. So when we started to feel toasty we knew shit was going south.

    As a result of one of those meltdowns we (only) lost(sort-of) a week of work. But as a result we did get a new HVAC and a brand new SAN. The only WTF with the SAN was we had to have an HP rep on site so the lab team could put it together, but since the HVAC has been fixed it's been mostly smooth sailing.



  • Server rooms and broom closets are synonymous in Management's mind.

    I have a server room that is cooled by a single split-system unit. And a laundry basket underneath it to catch the occasional drip.


  • BINNED

    Ah, yes, that kind of a server room...

    At one time I was forced to work in one that had a single shitty consumer grade A/C which was actually off most of the time. To add to that, the whole electrical system was so crappy that it managed to take half of the servers down when a starter on one of the flourescent lights went and decided to explode. No, I have no idea how they managed to screw up that badly.

    You know what's also fun? There was so much static in the air that merely standing in the room for about a minute left you with enough charge to zap yourself or someone else with a visible spark.

    Bonus WTF: me actually having to sit there on my laptop instead of just connecting to the servers from anywhere else. Nobody either knew or wanted to configure their firewall to allow me access to the server without opening access to everyone else as well. So, for security reasons, I had to sit in the server room. Unsupervised. Because that's fine. But giving me the access from outside the room was a no-no. $DEITY forbid one of the office drones who barely can handle Word would have a way to open an SSH session to the server with no sensitive data on it (at the time).



  • Used to do a lot of operational work, some things I found:

    • server room/broom closet under the stairs. Everything just sitting on the floor. Including the monitor, mouse and keyboard. Keyboard/mouse cord was terribly short. The box used an old PS/2 keyboard connector and didn't accept my USB replacement that I carried around for just that kind of thing. Had practically lay on the ground on my belly to be able to type and look at the screen. The old CRT was showing all colors except for the ones it should have been showing.
    • Work boots: working on an old beige box standing against the wall. The wall behind it has some telco shit and an ac bolted to it. The top of the beige tower features one distinct food print of a boot. I suspect one of the guys of the national telco company didn't bother getting a ladder from his van and use the pc as a step.
    • Extra cold: nice server room, very organized and all. AC keeping it cool. Obliged to work through a remote desktop from a management pc in the server room for security reasons. It wasn't even allowed to remote to the server from the IT office down the hall. The cold airflow was directed right at the freaking desk that had the management pc on it. I went to heat up several times in the warm isle behind the servers while installing patches/rebooting. IT guys kept a dirty old winter coat around. Went to work on hot summer days with a hoodie.
    • Extra hot: new server pc kept rebooting/shutting down on hot days. Customer claimed that it couldn't be the room temperature because the older hardware worked just fine. On the on site visit on a rainy day I had to vent the room first before so the temperature dropped so I could enter that oven.
    • Moving out: very enterprisy customer. Moving out of an old server room. The building was from the early 1900s and had always been a telco room. There where rows and rows of very olds skool phone patches. And a soldering iron to maintain it. All was beeing moved out. Or servers where the last ones standing exept for the patching boards. Raised floor was half removed, including around the lonely server rack. Had a hard time standing in a pit reaching up to the server who was top of the rack.
    • No power: customer makes a middle of the night emergency call to complain that our server is down. Only after I tell him that I can't remote into the server he tells me that the power is down in half the building due to an electrical problem. Told him to call me back the next day when power in the server room was restored.
    • Breaker: server rack stuffed in a conference room next to the kitchen area. Found out that they are both on the same breaker after the coffee machine kept blowing the fuse. Told the customer that I wasn't coming over unless there was coffee.


  • @Luhmann said:

    Extra cold: nice server room, very organized and all. AC keeping it cool. Obliged to work through a remote desktop from a management pc in the server room for security reasons. It wasn't even allowed to remote to the server from the IT office down the hall. The cold airflow was directed right at the freaking desk that had the management pc on it. I went to heat up several times in the warm isle behind the servers while installing patches/rebooting. IT guys kept a dirty old winter coat around. Went to work on hot summer days with a hoodie

    I work in Brisbane and had to spend a day in a client's server room a few summers ago. The thermostat wasn't working and the thermometer was hovering around 14c (57f). It was lovely at first after the work to their office but after about half an hour I was shivering. Like you I had to take regular breaks to warm up but it was a long day in there.


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.