Network switches are bad mojo



  • Another great story about our System Administrator.  If you missed the other sidebar about him check it out:


    Earlier I refered to him as a Network Engineer, actually his title is Senior Systems Administrator (which makes the earlier post more of a wtf).

    I used to work in this room with only two network jacks.  I shared the room with one other person.  There was also a network printer that everyone used so it obviously took one of the jacks.  My office mate and myself had to fight for the other one.  Obviously that didn't work, so I asked for a little 4 port switch.  Not understanding the difference between a router and a switch, the Senior System Administrator told me no, and that they would get another jack installed.  Well, as you guessed another jack was never installed.  Finally I brought a switch from home, plugged it in and hid it in a filing cabinet.  This worked just fine for several months until one day the network printer stopped working.  The Senior System Administrator came to investigate (while I was out), and noticed a cable going from the other jack into a filing cabinet.  When I returned I found a note on the cabinet saying "See me to retrieve your router".  After unsuccessfully trying to explain to him that it was not a router, he gave me a lecture about putting personal equipment on the network.  I brought up how several months ago I was told a new jack would be installed and it never was.  About a week later another jack was installed.



  • He's actually somewhat right on this one.  The router/switch confusion is idiocy, but it is far better to run a few jacks than it is to let people install their own hubs/switches all over the place.  Those little POS home switches can do bad things, and end up flooding your network.

    We keep having this weird issue at work between two home-grade ($20) switches, in which, in some cases, they start firing the same packets back and forth to eachother in a loop, making both unusable.



  • We had a problem here recently where some of our computers were falling off the network (network drives died, no internet, no ping, no email...) - turned out that someone had read an article somewhere on the internet about how a firewall will prevent viruses blah blah blah. They brought their home firewall into thier office, set it up, and plugged in to the other jack in thier office, without turning DHCP off.  So instead of our company 10.0.0.1## standard, it was serving out 192.168.43.### addresses to some of our computers whose IP addresses had expired.  We figured it out pretty quick, but it took us hours to find the rogue router.



  • @Albatross said:

    We had a problem here recently where some of
    our computers were falling off the network (network drives died, no
    internet, no ping, no email...) - turned out that someone had read an
    article somewhere on the internet about how a firewall will prevent
    viruses blah blah blah. They brought their home firewall into thier
    office, set it up, and plugged in to the other jack in thier office, without turning DHCP off
    So instead of our company 10.0.0.1## standard, it was serving out
    192.168.43.### addresses to some of our computers whose IP addresses
    had expired.  We figured it out pretty quick, but it took us
    hours to find the rogue router.
    Seeing the ip addresses used you
    could not have had more than 100 computers. Go to the wiring closet,
    floodping the router and one by one disconnect the switchports with the
    most trafic :P Once the pings timeout you either have the cable to (the
    next step towards) the router, or the cable to your own box. You do
    know where in the building each patchpanelport ends up, right? If not,
    that's the True WTF(tm) of your story ;)

    If you're worried you might overload some uplink between switches by
    the floodping (but with a SOHO-router that is unlikely) you can do a
    normal ping and disconnect everything, but that would take slightly
    longer.



    Preferably do this on a quiet moment, so you won't unplug that million
    dollar company-critical SQL-server at the peak of its job...



    Or, if you have managable switches, look at the MAC-address table to
    see which switchport you need :P (Get a rogue IP from the router, 
    ipconfig /all to find the routers ip-address, ping it (once is enough,
    if the DHCP-procedure wasn't enough already), look in the ARP-table for
    the MAC-address of the router, etc)



    So I wonder, what part took all the hours? Waiting for  the SQL-server to finish its job?



  • I'm not gonna take sides on the "we'll install a switch" versus "we'll install another port" argument, but I'd definitely agree that it's good practice not to allow people to bring their own random equipment in.

    Home networking equipment has been known to do some very bad things.

    That said, personally, I'm a Java Wizard. (Honestly, it jokingly says so on my door.) So I'll bring in whatever weird computers and networking gear I please, and I'll install anything I want on my company issued computer. [bravado] I know what I'm doing, and nothing could possibly go wrong. :^b [/bravado]



  • Going down the switch routecan make sense if it's a stopgap measure, or
    if getting another cable run is impractical/not worth it. That said,
    there's a strict policy that anyone doing anything bad to MY network
    gets smacked in the head with the sharp end of an axe.



    But a Network Engineer/"Senior Systems Administrator"that doesn't know
    a router from a switch? Ugh. That's like a typist that doesn't know
    what a keyboard is.



  • @pnieuwkamp said:

    @Albatross said:
    We had a problem here recently where some of our computers were falling off the network (network drives died, no internet, no ping, no email...) - turned out that someone had read an article somewhere on the internet about how a firewall will prevent viruses blah blah blah. They brought their home firewall into thier office, set it up, and plugged in to the other jack in thier office, without turning DHCP off.  So instead of our company 10.0.0.1## standard, it was serving out 192.168.43.### addresses to some of our computers whose IP addresses had expired.  We figured it out pretty quick, but it took us hours to find the rogue router.
    Seeing the ip addresses used you could not have had more than 100 computers. Go to the wiring closet, floodping the router and one by one disconnect the switchports with the most trafic :P Once the pings timeout you either have the cable to (the next step towards) the router, or the cable to your own box. You do know where in the building each patchpanelport ends up, right? If not, that's the True WTF(tm) of your story ;)
    If you're worried you might overload some uplink between switches by the floodping (but with a SOHO-router that is unlikely) you can do a normal ping and disconnect everything, but that would take slightly longer.

    Preferably do this on a quiet moment, so you won't unplug that million dollar company-critical SQL-server at the peak of its job...

    Or, if you have managable switches, look at the MAC-address table to see which switchport you need :P (Get a rogue IP from the router,  ipconfig /all to find the routers ip-address, ping it (once is enough, if the DHCP-procedure wasn't enough already), look in the ARP-table for the MAC-address of the router, etc)

    So I wonder, what part took all the hours? Waiting for  the SQL-server to finish its job?

    Well, when the rogue router shows up at 10:00 in the morning of a very busy day, it generally isn't a good idea to start unplugging people's network cables :)



  • @Albatross said:

    Well, when the rogue router shows up at 10:00
    in the morning of a very busy day, it generally isn't a good idea to
    start unplugging people's network cables :)
    Maybe the Spanning
    Tree Protocol will spoil it all, but otherwise people would probably
    hardly notice the temporary disconnection, you'd see the stop in the
    floodping in under half a second, and people were already being
    randomly (when their lease expired) being cast of the net, so I'd say
    disconnecting a dozen people, one at a time, for half a second each,
    would far outweigh letting the problem persist for a couple of hours.



    Just out of curiosity, how did you find it? Physically checking every
    outlet? Managable switches' MAC-tables? Tarrot cards? Executive
    meetings at Starbucks?


Log in to reply