The Multi-Outcast



  • I used to work in a building used as a call center, though the group I was in was not, thankfully, on the phones. We were doing 2nd-level support, taking on cases that stumped the call centre folks, in a particular product group. In one case, a colleauge of mine was sent a Ghost image of a disk on a customer's server, and set about writing that image to a disk on a server in our computer lab. We had little experience of Symantec (formerly Norton) Ghost, since our employer made a competing product.

    Shortly after he started the image extraction, the building network crashed: phone agents could not log calls in the logging system. Then calls stopped coming in to the building. About 800 call center agents were soon idle, while the IT department ran around in headless chicken mode, scrambling to find the cause of the outage.

    After about twenty minutes, they pounced on my colleage's server or, I should say, its port on the switch: it had single-handedly brought the building down. The reason, as we soon learned, was that Ghost was multicasting the image on the building network. The Ethernet switches were misconfigured, passing gigabytes of multicast packets on all ports, swamping the building network. When the packets hit the computer controlling the switchboard, it choked, preventing phone calls from being answered until it was rebooted. It's hard to say how many phone calls went unanswered, but it may have been in the tens of thousands, as customers tried and tried again.

    I suppose the WTF here is that a fairly standard application, doing its job, can cause so much trouble due to a little misconfiguration. At least, that was what we were told: my colleague didn't admit to anything untoward, but he was PNG* in the IT department for quite some time afterwards...

    * Persona Non Grata



  • @bnt said:

    The reason, as we soon learned, was that Ghost was multicasting the image on the building network.
     

    That's almost a happy end. I was totally expecting to read that your colleague had wiped out a database that was, for some reason, running on a test machine rather than on a production server. That would've been a bit harder to recover from :)



  • @bnt said:

    When the packets hit the computer controlling the switchboard, it choked, preventing phone calls from being answered until it was rebooted. It's hard to say how many phone calls went unanswered, but it may have been in the tens of thousands, as customers tried and tried again.
     

    Well, that's all right then. Considering how much a support call costs, your colleague probably saved your company millions.



  • @bnt said:

    a little misconfiguration
    I would disagree.  This is pretty severe.

    Now, why the ghost image would multicast, I have no idea.



  • @bnt said:

    We had little experience of Symantec (formerly Norton) Ghost, since our employer made a competing product.

    I find this to be a WTF in itself.  If you make a competing product you should know all of your competitor's products inside and out. 



  • @belgariontheking said:

    Now, why the ghost image would multicast, I have no idea.

    Norton Ghost comes with a multicast server to send an image to several client PCs simultaneously.  It sounds like the multicast server was turned on which was certainly a mistake by the OP's co-worker. 



  • @morbiuswilters said:

    @bnt said:

    We had little experience of Symantec (formerly Norton) Ghost, since our employer made a competing product.

    I find this to be a WTF in itself.  If you make a competing product you should know all of your competitor's products inside and out. 

     

    probably not the support people.  The people making the software and designing it should though. 



  • I see someone beat me to what I was going to say. We had nothing to do with development of our company's disk imaging product - different division, different continent. We used it, just as e.g. a Microsoft support person would use IE in their job without being an expert on Firefox. Keeping track of everything made by all our competitors would be impossible - it was a massive multinational company.



  • @morbiuswilters said:

    @bnt said:

    We had little experience of Symantec (formerly Norton) Ghost, since our employer made a competing product.

    I find this to be a WTF in itself.  If you make a competing product you should know all of your competitor's products inside and out. 

    TRWTF is that they were not using said competing product internally (unless it was only in the early stages of development).



  • @Physics Phil said:

    TRWTF is that they were not using said competing product internally (unless it was only in the early stages of development).
    They received a Norton ghost image, logic suggests (since it is most likely some sort of proprietary format) that they use Norton ghost to image the drive. Although if the product the produced could read and write Norton images fine then yes, it would be a WTF to use Norton in that case.



  • @bnt said:

    After about twenty minutes, they pounced on my colleage's server or, I should say, its port on the switch: it had single-handedly brought the building down.
     

    This is a WTF on both my part and the ISP, but I was once employed by a media company that worked closely with an ISP.  We were working with a product from a third company (the "technology provider" in this case, but that's a WTF in itself), which used multicast.  Probably the biggest WTF on the ISPs part was to wire us directly into their NOC network, and to have a routing loop between their PA and OH hubs, causing the multicast to multiply, much like in the old days of broadcast ICMP.  Probably the biggest on ours was not setting TTL to 1.  Combined, we wound up eating up all bandwith in all pops in a national ISP, causing a short-lived, but spectacular nation-wide outage.



Log in to reply