Electrify that server!



  • We just found out that as part of modernizing our remote server room, they were replacing the usual plastic-y raised floor tile with the much more homey-looking carpeted raised floor tile.

    Yes, it looks nicer. But it also creates a fuck-load of STATIC ELECTRICITY as you walk across it.

    While our servers are fairly resilient to the odd power-surge and lightning bolt, it turns out that if you zap them enough times with static electricity, they will schitz out.

    While the data center manager was giving a tour to show off our brand new carpeted server room to the brass, a bunch of them must have been shuffling their feet, and touching every server (oooh, pretty blinkenlights) as they walked by, because several of the servers died just after they walked by.

    SA's came running. Who the hell are you people and what are you doing to these servers?!

    Unfortunately, it was too late. Boards were fried.  Disks were spinning down. Alarms were shrieking.

    I can't wait to see who senior management blames for this one.

     



  • I thought carpet in the kitchen or the bathroom was weird. Carpet in the server room? Seriously, what was the reasoning there?

    Since it's carpeted, the safest thing to do (static discharge-wise) is to walk barefoot. And besides, you should touch the rack before you touch the thingies in the rack.



  • @smxlong said:

    I thought carpet in the kitchen or the bathroom was weird. Carpet in the server room? Seriously, what was the reasoning there?

    Since it's carpeted, the safest thing to do (static discharge-wise) is to walk barefoot. And besides, you should touch the rack before you touch the thingies in the rack.

    So...: Excuse me, Mr. Executive, please remove your shoes and socks while walking in here....



  • Snoofle, you da masta of da WTF. If you make up all this stuff, it's still great, but it has a terrifying ring of veracity in it.



  • From one crisis directly to another...



  • Wow, just wow. What is your employer's purpose? Manufacturing new and interesting layers of WTF?

    It makes me think that every server room construction or refurb should include budget for constructing a small fake server room for people to tour. Blinkenlights, machines that go ping, maybe even some old tape drives whirring backwards and forwards. The people that have business being in a server room don't need tours, and the people that don't have business being in a server room won't know the difference and can't break anything.

    (I freely admit to being in the latter category.)




  • @Paddles said:

    Wow, just wow. What is your employer's purpose? Manufacturing new and interesting layers of WTF?

    This calls for a nickname for snoofle's company. The WTFactory? ProblemFactory?


  • Considered Harmful

    @Paddles said:

    It makes me think that every server room construction or refurb should include budget for constructing a small fake server room for people to tour.



  • Static electricity is a possible problem, but I'd say dust is the bigger one. Carpets create and capture dust and dust in fans, cooling etc, will cause another problem.

    Get them to replace the carpet with tiles coated with broken glass. Guarantees no more future tours.



  • @Quango said:

    Static electricity is a possible problem, but I'd say dust is the bigger one. Carpets create and capture dust and dust in fans, cooling etc, will cause another problem. Get them to replace the carpet with tiles coated with broken glass. Guarantees no more future tours.

    Especially if you don't revoke the "remove your shoes" requirement.



  • @snoofle said:

    While the data center manager was giving a tour to show off our brand new carpeted server room to the brass,

    ... take your time...

     



  • @Paddles said:

    ... a small fake server room for people to tour. Blinkenlights, machines that go ping, maybe even some old tape drives whirring backwards and forwards....
     

    That's... an EXCELLENT idea!

    @Quango said:

    but I'd say dust is the bigger one.

    Knowing snoofle's company, they've addressed this problem by hiring a cleaner who vacuums the carpet weekly.

     



  •  @Cassidy said:

    Knowing snoofle's company, they've addressed this problem by hiring a cleaner who vacuums the carpet weekly.

    Yes and them being not stupid they will provide a dedicated power outlet. On the UPS...

     



  • I know of a popular web analytics company that keeps a fake NOC for exactly that purpose. I feel sorry for the guys who have to work in it-- I assume they're the bottom-of-the-rung NOC interns.



  • @blakeyrat said:

    bottom-of-the-rung NOC interns.
     

    Clearly a promotion up from top-gun DBAs at snoofle.org...



  •  Tiled server rooms floors are dangerous, as they get slippery when wet.



  • Been a lurker for a while, and a big fan of snoofle, but I cry shenanigans on this one.

    There are so many people involved in changing the flooring in a server hall that I can't believe that putting carpet tiles in would not have been rejected by someone during the process.

    Even allowing for the dumkpoferry exhibited at snoofle's place, it would just have taken one person (even the carpet tile supplier) to raise sufficient concern for this plan to be rejected.
    So I suspect a wee bit of embellishment on this one...



  • @skotl said:

    Been a lurker for a while, and a big fan of snoofle, but I cry shenanigans on this one.

    There are so many people involved in changing the flooring in a server hall that I can't believe that putting carpet tiles in would not have been rejected by someone during the process.

    Even allowing for the dumkpoferry exhibited at snoofle's place, it would just have taken one person (even the carpet tile supplier) to raise sufficient concern for this plan to be rejected.
    So I suspect a wee bit of embellishment on this one...

     

    There used to be a time when I would agree with you. Having seen though how decisions are made in corporations by the people "least" qualified to make them but with the most political swing,I can say I totally believe this.

    Reminds me of how my company installed a big backup generator for our server room outside the building. The idea being that if we lost power it would cut in and keep the business critical operations going. The problem with that being the most likely cause of a power outage for us is a hurricane (We are in florida), and the building is in a flood zone...and as the generator is too big for the roof of our building they installed it two feet off the ground. All it will take is one huricane and the generator will be submerged and our servers will be down.

    Of course management with zero technical skills (or common sense) who came up with this solution can't see the problem with this.



  • @codefanatic said:

    the most likely cause of a power outage for us is a hurricane (We are in florida), and the building is in a flood zone...and as the generator is too big for the roof of our building they installed it two feet off the ground. All it will take is one huricane and the generator will be submerged and our servers will be down.

    Of course management with zero technical skills (or common sense) who came up with this solution can't see the problem with this.

    Did your management perchance have a TEPCO gig a while back?



  • @skotl said:

    (even the carpet tile supplier) to raise sufficient concern for this plan to be rejected.
     

    They'll lose out on business for warning the customer, so I don't think that's a desirable choice from their POV. Given that the centre manager was showing people around, it looked like the choice of carpet was made by an authorised decision-maker that should have known better.

    @codefanatic said:

    Of course management with zero ... common sense.. can't see the problem with this.

    That.

    Managers reach a certain level when they no longer manage products but are instead managing people who manage those products, so lose product knowledge.  Their skills should be one of stakeholder analysis then information-gathering to provide a business case for the decision, not ploughing with what their limited knowledge deems sufficient.



  • @snoofle said:

    We just found out that as part of modernizing our remote server room, they were replacing the usual plastic-y raised floor tile with the much more homey-looking carpeted raised floor tile.

    I have no experience with it .. but there is such an item as "anti-static" carpet that is designed for server rooms.



  • @snoofle said:

    they were replacing the usual plastic-y raised floor tile with the much more homey-looking carpeted raised floor tile.

    Yes, it looks nicer. But it also creates a fuck-load of STATIC ELECTRICITY as you walk across it.

    I've never seen a carpet that generates as much static as rolling a wheelchair across a linoleum floor.  A memory I will carry to my dying day was pulling a three-inch-long blue spark off my infirm mother's ear every ten yards.

     



  • Snoofle, every time I read a post from you I laugh my ass off. To where can I Fedex you beer?



  • ASHRAE recommends 60% relative humidity to minimize the static electrical build up in just such a case- And your racks and cases would of course be properly grounded to minimize the effects of any static discharge...

    Whose running your server room, Geek Squad?


  • Discourse touched me in a no-no place

    @codefanatic said:

    Reminds me of how my company installed a big backup generator for our server room outside the building. The idea being that if we lost power it would cut in and keep the business critical operations going.
    Did they remember to put the network switches and the AC on the backup power too?

    We have a setup rather like yours (but without the risk of hurricane problems) and it used to be used to run a nice big supercomputer. I forget what it's highest ranking in the top-500 was, but that was all rather many years ago. However, none of the network switches were on the backup power, and nor were the AC units. When we lost grid power to the building one day to a backhoe event, the supercomputer continued just fine. It had, however, just become completely unable to communicate with its SAN or the outside world, so any results it produced would be lost; it was doing useless work. What was more, this useless work was rapidly heating up the now-totally-uncooled room; cue panic to shut things down before the heat caused real problems.

    All this was compounded by the fact that all the emergency lighting totally failed, and the server room had a lot of its floor tiles up for maintenance. Headless chickens, running around in the dark, with pitfalls and a supercomputer belching the best part of 5MW of power into a room not designed to take it. Fun times, fun times.

    I don't remember if that was the time when restoring the power started a fire...



  • @Paddles said:

    Wow, just wow. What is your employer's purpose? Manufacturing new and interesting layers of WTF?

    It makes me think that every server room construction or refurb should include budget for constructing a small fake server room for people to tour. Blinkenlights, machines that go ping, maybe even some old tape drives whirring backwards and forwards. The people that have business being in a server room don't need tours, and the people that don't have business being in a server room won't know the difference and can't break anything.

    (I freely admit to being in the latter category.)


    My company seems to be too cheap for the fake NOC but we did slap together a "video tour" of our brand new facility that has video clips from Google's data center, Facebook's data center, and a handful of YouTube clips of completely different data centers slapped together with a voice over about our "new generation data center" to show exec and visiting customers.

    I mean, or DC does look like crap and someone tried to make something that looked good... but blatant IP theft is a bit over the top...



  • @dkf said:

    ..............

    supercomputer belching the best part of 5MW of power into a room ..........

     

     

    A 5MW supercomputer...... Nawwwwww

     



  • @Helix said:

    @dkf said:

    ..............

    supercomputer belching the best part of 5MW of power into a room ..........

     

    A 5MW supercomputer...... Nawwwwww 

     

     

    http://top500.org/list/2012/11/

    No sortying by power, but only top 3 are doing more than 5MW :)

    Several other doing 3-4MW.

     



  • @Shinhan7 said:

    @Helix said:

    @dkf said:

    ..............

    supercomputer belching the best part of 5MW of power into a room ..........

     

    A 5MW supercomputer...... Nawwwwww 

     

     

    http://top500.org/list/2012/11/

    No sortying by power, but only top 3 are doing more than 5MW :)

    Several other doing 3-4MW.

     

     

    Exactly - none of them reside in Manchester, UK

     


  • Discourse touched me in a no-no place

    @Helix said:

    Exactly - none of them reside in Manchester, UK
    I did say it was some time back. We left the supercomputer game about 5 years ago (for various reasons that I don't care to rehash) but the CSAR service definitely did make the Top 500 back when it was new. It also drew a lot of power; newer systems have a lot better MJ/MIPS rating so they get far better bang for the buck. I hated coding for that machine: the IRIX C compiler was horrible and very picky, and the rest of the toolchain was really rather exotic. We also had gcc, but that would only build 32-bit binaries, and the weird 64-bit toolchain associated with the IRIX compiler had the feature of being something that libtool just could not cope with. (Libtool would do the digital equivalent of a shrugged "christ, what an imagination I've got" and auto-roll-back any attempt to fix it. No idea why. I despise libtool! It's the linker wrapper of Codethulhu!)

    We still have that backup generator, and it is an awesome piece of kit. You can't see it at street level due to some fencing, but my office has a good view.


  • Discourse touched me in a no-no place

    @Helix said:

    A 5MW supercomputer...... Nawwwwww
    I might have the power rating wrong. It was a decade or so ago, and I wasn't on the operations team. (I wrote services that built on top of it.)

    The curious thing is that it had a different architecture to modern supercomputers: it had a special memory backplane which allowed you to use OpenMP on thousands of nodes at once (i.e., large scale parallel memory coherence at speed). These days, you have to use MPI for the most part because that doesn't require such enormously complex hardware to support. The loss of coherence lets you get a lot more speed provided you can use algorithms that don't require a consistent memory view. That's non-trivial in general.



  • @dkf said:

    @Helix said:
    Exactly - none of them reside in Manchester, UK
    I did say it was some time back. We left the supercomputer game about 5 years ago (for various reasons that I don't care to rehash) but the CSAR service definitely did make the Top 500 back when it was new. It also drew a lot of power; newer systems have a lot better MJ/MIPS rating so they get far better bang for the buck. I hated coding for that machine: the IRIX C compiler was horrible and very picky, and the rest of the toolchain was really rather exotic. We also had gcc, but that would only build 32-bit binaries, and the weird 64-bit toolchain associated with the IRIX compiler had the feature of being something that libtool just could not cope with. (Libtool would do the digital equivalent of a shrugged "christ, what an imagination I've got" and auto-roll-back any attempt to fix it. No idea why. I despise libtool! It's the linker wrapper of Codethulhu!)

    We still have that backup generator, and it is an awesome piece of kit. You can't see it at street level due to some fencing, but my office has a good view.

     

    Pic of said 5MW generator set:

    http://www.generator-turbine-powerplant.com/products-page/generators-power-plants-60-hz/diesel-generators-power-plants-60-hz/5-mw-man-9l2838-hfo-power-plant/

     

    Talk about ass-about-face units, if you used the MJ/MIPS units in my workplace i would get you fired.

     



  • @dkf said:

    the IRIX C compiler
     

    *shudders*



  • @skotl said:

    Been a lurker for a while, and a big fan of snoofle, but I cry shenanigans on this one.

    There are so many people involved in changing the flooring in a server hall that I can't believe that putting carpet tiles in would not have been rejected by someone during the process.

    Even allowing for the dumkpoferry exhibited at snoofle's place, it would just have taken one person (even the carpet tile supplier) to raise sufficient concern for this plan to be rejected.
    So I suspect a wee bit of embellishment on this one...

     

     

    of all the people involved there will only be 3 or so that will think of countering management in this; all the rest will be too busy brownnosing to care



  • A 5mW supercomputer......  Now that would be something to see!!!!!



  • @Helix said:

    Talk about ass-about-face units, if you used the MJ/MIPS units in my workplace i would get you fired.

    Agreed, the proper form is Js/I.



  • TRWTF is that a necroed thread appears on the RSS feed.



  • @Zecc said:

    TRWTF is that a necroed thread appears on the RSS feed.

    Maybe you should try a better RSS client, like Excel?



  • @Ronald said:

    @Zecc said:
    TRWTF is that a necroed thread appears on the RSS feed.

    Maybe you should try a better RSS client, like Excel?

    The awesomeness of Excel never ends.

     


  • Discourse touched me in a no-no place

    @El_Heffe said:

    The awesomeness of Excel never ends.
    Excel's awesomeness is in an infinite loop?



  • @dkf said:

    @El_Heffe said:
    The awesomeness of Excel never ends.
    Excel's awesomeness is in an infinite loop?
    This just in -- Microsoft Word is also awesome.  It was used to design iOS 7. You can even download the Word file.



  • @dkf said:

    @El_Heffe said:
    The awesomeness of Excel never ends.
    Excel's awesomeness is in an infinite loop?
    Excel is so awesome it got stuck in an infinite loop and survived.


  • ♿ (Parody)

    @mott555 said:

    @dkf said:

    @El_Heffe said:
    The awesomeness of Excel never ends.
    Excel's awesomeness is in an infinite loop?
    Excel is so awesome it got stuck in an infinite loop and survived.

    It just clears the undo stack and escapes.


  • Considered Harmful

    @mott555 said:

    @dkf said:

    @El_Heffe said:
    The awesomeness of Excel never ends.
    Excel's awesomeness is in an infinite loop?
    Excel is so awesome it got stuck in an infinite loop and survived.


    Excel can divide by zero.


  • BINNED

    @joe.edwards said:

    Excel can divide by zero.
    Excel is the new Chuck Norris.



  • Hey joe!, your tags are leaking into other people's posts again.



  • @Zecc said:

    Hey joe!, your tags are leaking into other people's posts again.
    It was my tag first!

    Though I should point out that you only see my tags if you have signatures disabled.



  • @joe.edwards said:

    Excel can divide by zero.
    Excel uses Chuck Norris

     



  • @El_Heffe said:

    @joe.edwards said:
    Excel can divide by zero.
    Excel uses Chuck Norris

     

    Excel is kicking the ass of free office suites.


  • ♿ (Parody)

    @Ronald said:

    Excel is kicking the ass of free office suites.

    That's not surprising, but I'll bet the Open/LibreOffice is more common than IT decision makers know.


Log in to reply