What was your absolute worst day in IT



  • As the title states and I'll start.

    We all like to think we are bad-asses amazingly great at what we do but sometimes the worst moment are the most notable so I have decided to share one.

    I've had shit days and if there is one that to this day scratches at the back of my mind the most was when I was doing Y2K conversions for Northop-Grumman. I was escorted into the Stealth Bomber IT facility to swap a hard drive.

    I walked into a holding area where guy behind a glass window got my personal info my signature gave me a special badge then hit a switch. A series of red alarm lights (no sound) let everyone in the office know there was a gimp IT person in the facilities.

    The guy lead my to the computer that I was to be working on. These were circa 1996 IBM desktop machines when all black systems started to hit the market. These machines had standard internals and I knew how to get to everything. They were essentially all clones.

    So I'm going through my check list. This hard drive was preloaded with the operating system and tools the people needed. Just swap and go.

    I get to where I am removing the case and looking through the chasis. There were two hard drives were there should have been only one (policy and obviously an exception to this policy).

    So I was looking at how the drives were set up on the hard drive cable to see if I could determine the slave drive when there was a short, some smoke, the smell of tin and panic from me.

    I decided to play it cool and get the hard drive in place and get the computer up and running which I did.

    I tried to get to the two other hard drives figured out through the bios but this was when you had to configure settings and move jumpers to get things to talk. I couldn't get either to talk and I was essentially fucked.

    I contacted the tech lead and explained what happened. He wanted me to talk to the user to see if there was anything critical on the second drive.

    Guess what? There was very important shit on it. So important that my contract ended that day. Had a shitty exit interview, had to go to another building to have my badges inactivated then had to sit there and wait till my contract company sent a rep to escort me off the premises.

    I replay that day sometimes in my head and when I do I think to myself: out of that list of 6 work orders on my tiny 15 inch CRT monitor...why on earth did I picked that one.


  • BINNED

    @Frank said:

    So I was looking at how the drives were set up on the hard drive cable to see if I could determine the slave drive when there was a short, some smoke, the small of tin and panic from me.

    I've had that happen to me once when I accidentally pulled on an IDE cable while the machine was on. It wouldn't boot and I was looking at the POST code: the board didn't beep, it had two 7-seg LED displays for codes. Basically, look at where it got stuck, take out the instructions, find the code and, usually, work your way backwards to first code that could cause a failure (most of the time it would get stuck in some mid-state AFTER the failed one).

    I think the cable was in the way, and as I tried to move it I probably yanked on it too hard and it unplugged partially. It was the cable that burned in the end, the drives were fine. So who knows, maybe you could've gotten away with it if it was the same problem.



  • Not very IT related, more tech.

    We were making a laser harp for Embedded Systems at my uni. We did woodwork, all the cabling, hooked up photoresistors, hooked up rangefinders, soldered together two huge ass PCBs, did the programming, and above all, a lot of money was spent.

    So, about two days before the due date, I decided to do the final test. I hooked it up and tried to play for a few minutes. Instead, I watched the lasers (which were pretty much cemented into the woodwork) slowly shrivel and ultimately die, because the potentiometers were set wrong and it only now has occured to me that we've never tried running the thing for more than 10 seconds.



  • Yea that was a hardcore WTF moment which segwayed into damage control and then segwayed into fear for my life.



  • Something about your post reminded me of this terrible and deadly WTF moment with the Therac-25 chemotherapy machine.

    Many here might be privy but I will post the link anyways.

    Filed under: Race condition



  • Not an IT, but I did manage to burn my HDD a few months ago.

    At the time, I found myself with an extra SATA drive and no free SATA power cables in my box. However, I did have a bunch of old IDE plugs and an IDE-to-SATA converter. Years ago, I would have checked the jumpers, the lead wire and all that crap. But these new dummy-proof sockets made me lazy, so I just kind of showed one thing into another until it fit.

    So yeah, I plugged the IDE part the wrong way around (cheap ass plastic converter didn't help). There was a spark and a smell of burned electronics. The coating from the melting cable was literally dripping all over my casing by the time I managed to shut the power off. Had to release the charge from the PC, but other than the fried electronics on the HDD, everything was OK. It pays to have a good PSU.

    On the bright side, that solved my problem with an extra HDD.



  • This wasn't so much a bad day as much as it was a horrible WTF feeling. At my last job (was there for 3.5 years), everything was done on the cheap. It was a 30-person company, the owner was a bit out of touch with reality, and the only reason why I even worked there is that the people that I worked closely with were pretty good. Software developed long before I started there was really bad, so I ended up rewriting a good number of applications. The worst commits were from one developer that I'll just name NR (not his real initials), and I don't think that I've ever seen code as bad as his.

    Fast-forward three years, my manager had left for a new company and I was reporting to the owner. So, in other words, I was busy looking for a new job. There was a pet project that the owner had me working on with no real specs or aim, which he thought was running behind for all the reasons not related to him, when the exact opposite was true. So one Monday, I come in, and the owner has a meeting set for 9am (my start time). He probably wasn't happy when I showed up at 9:15, but that's his own fault. He had a habit for sending me meeting invites during my lunch break, for meetings that started immediately after lunch.

    Anyways, I come into the meeting, and there's a new guy at the table. It turns out that the owner of the company had hired the NR developer as a contractor to get the project done sooner. Not only would it take longer since this guy had to have his environment set up for him, he duplicated code, didn't understand code, and did so little right that I rewrote most of his stuff (again). But this is a classic case of throwing more programmers (an incompetent one) at a project that doesn't need more code, it needs better direction.

    So that meeting pretty much brought out my two week notice, and made me feel sorry for my friends that still work there. Most have moved on, but one of the DBAs is still there, and I sincerely hope that he leaves soon.



  • Haven't had too many awful days, but there was one that is pretty funny in hindsight.

    I was still in high school and worked IT for the school (SMALL school, like 300 students total for K-12), mostly during the summers but also sometimes after the school day. This was during the summer, and my supervisor was away on vacation for a week. He wanted me to clean up the main wiring cabinet that had all the LAN feeds from across the campus plugged into the main switches.

    The cabinet was just some DIY plywood thing and was not well-lit at all. Cables went every which way, wrapped around each other, and even tied in knots before plugging into a switch. I got some wire cutters out and started clipping all the zip ties holding stuff together.

    Pretty soon, I started getting phone calls from other staff saying the Internet was out. Long story short, one of the zip ties wasn't. The main T1 line was wrapped around a bundle of cords in the far back of the cabinet, and in the dim light and from the front while standing on a stepladder I'd thought it was a zip tie.

    No biggie, I thought. I'll strip some insulation off and twist the wires back together and it'll all be good. Except then I found out the hard way that a T1 line has significant line voltage! I actually jumped off the ladder in surprise.

    Ten years later, my friends still tease me about the time I cut the school's T1 line and shocked myself while trying to fix it.

    That wiring closet was NEAT when I was done though. I trimmed cords to length, got them all tied off to one side, and even labelled most of them with the label printer.



  • What a great example! I laughed from a good healthy place on this. You can't beat real fuckups and the willingness of people to share!

    Bonus points because you are the guy who DoS'd his own thread and broke what.thedailywtf.com for short time...that was seriously funny too.



  • Me and stupid DoS attacks go way back.

    Several years before my last story, I figured out how to arbitrarily access anyone's network drive on the school network. They were hidden shares, but you could still navigate to them if you knew the scheme which was really simple. And there were no ACL's set up. Eventually it was fixed, and IT was glad I found the flaw and not some other student who'd probably delete all the teachers' files and stuff.

    So I went in to one of my best friends' network drive and put a folder called "White Rabbit". Inside that, another folder called "White Rabbit". I recursed until it wouldn't let me anymore, then put a shortcut back to the top-level "White Rabbit". This was just a couple years after The Matrix was released so that's the reference. My friend found it, thought it was funny, but we'd get an error whenever we tried to delete it.

    Fast-forward 2 - 3 years when I started getting involved with the school district's IT. The network's main file server had been failing its tape backup every single night for YEARS now and no one could figure out why. And it was a hard failure, too. The first thing anyone did when they arrived in the morning was punch the reset button on the front of the server to resuscitate it.

    The network was based on Novell Netware, so when they hired some guy from Novell to come do a bunch of routine maintenance they wanted him to look into the backups too. I heard that the school paid Novell $600 an hour from the time the guy left Novell's office to the time he returned. He was with us for a good week.

    Not sure how long it took him, but eventually he figured out that the backup failed because of some deeply-recursed "White Rabbit" folder structure in a student's network storage. Somehow it was corrupted and couldn't be deleted, and the backup software was failing due to infinite recursion as well. Apparently it was dumb enough to follow folder shortcuts.

    To fix the issue, he had to manually copy ALL network storage EXCEPT for my friend's over to temporary storage, re-format the old partition, then copy everything back. I didn't have the heart to tell anyone it was all my fault.



  • @mott555 said:

    I didn't have the heart to tell anyone it was all my fault.

    How things change, haha.


  • :belt_onion:

    @mott555 said:

    even labelled most of them with the label printer.

    Did you label the T1 when you went back?


  • :belt_onion:

    @mott555 said:

    backup failed because of some deeply-recursed "White Rabbit" folder structure in a student's network storage.

    I've seen recursive whiterabbit folders break shittily implemented utilities before.
    At a place I worked, someone accidentally dragged one of their mapped server drives onto itself in windows explorer and windows happily created the shortcut. It continually killed some process that happened to scan that folder for files to pick up and ended up scanning the whole server's drive recursively until it ran out of space for copying the copies of copies of copies of copies of copies of...



  • #The bad ideas thread

    Mounting any directory inside itself.


  • BINNED

    I really have to test that myself. In a VM. Because with my luck I'll wreck my entire system if I try on an actual install.



  • I'm using Linux exclusively in VM. Install some reasonable desktop like XFCE and it works like a charm. Been doing professional work like this for years.


  • BINNED

    I'm using LinuxWindows exclusively in VM. Install some reasonable desktop like
    XFCE and it works like a charm.
    I manage with it's window manager although I miss some stuff in it. Been doing professional work like this for years.

    There, updated for myself :P


  • Discourse touched me in a no-no place

    @darkmatter said:

    I've seen recursive whiterabbit folders break shittily implemented utilities before.

    Oh yes. I remember when we managed to confuse an NFS server thoroughly by persuading it to put a directory inside a subdirectory of itself (and yes, it was also normally reachable). Everything was fine until someone did a simple find on one of the NFS client systems, and the NFS server — which was also the university's central Unix service — died horribly. Reportedly, the CPU load got to over 400 but nobody's sure because load monitors weren't very reliable (funny that!) and because this was all driven by kernel processes, none of the admins could work out why it happened.

    Good times.


  • 🚽 Regular

    @mott555 said:

    Except then I found out the hard way that a T1 line has significant line voltage! I actually jumped off the ladder in surprise.

    I've done similar. Some idiot managed to jank my office phone line out of a small coupling box on the wall. No need to wait the three days for the phone services company, I'm an electronics engineer!

    Got up the ladder and the cable was not just pulled out it was snapped. Cut the end off and started stripping and twisting the ends, it all went fine until someone rang my extension. I think it was more surprise more than anything but luckily no-one saw and I only acquired bruises. That ring voltage hurts.


  • Discourse touched me in a no-no place

    @Cursorkeys said:

    Got up the ladder and the cable was not just pulled out it was snapped. Cut the end off and started stripping and twisting the ends, it all went fine until someone rang my extension. I think it was more surprise more than anything but luckily no-one saw and I only acquired bruises. That ring voltage <b>hurts</b>.

    That reminds me of an experiment I did with electrostatics at school (trying to measure the strength of the electric force). I was working with about a 5kV source, but with plenty of resistance and very low capacitances, so not very much risk from the experiment. Except that the source — kept under a lab bench with perspex guards around it to make things a bit safer to others — had a loose connection inside it and occasionally made its case go live. Yowch!

    The bruise on the back of my head from where I hit the bench took a couple of days to go down. No lasting damage though. <twitch> Be careful with high voltage kit; it can surprise you…



  • I don't want to go into too many details but...

    Let's just say there was a young, inexperienced, naive me; a brusque, busy manager who fired out half assed, somewhat incorrect instructions and then asked not to be disturbed; and a pile of production code that suddenly got a whole lot more broken when I started trying to follow said instructions! 😃

    The blame was put on me and I was too new and shy to argue or even really understand it wasn't my fault. If that happened today I wouldn't feel at all bad, if someone else's instructions caused the trouble, but back then I was close to tears...


  • 🚽 Regular

    I work at small companies and in addition to being an electronics engineer I'm usually IT dogsbody as well.

    My worst day in IT was when I managed to take down all power to a server room while attempting to show off some fancy new gear.

    We had an electrician in to do something that required all the building power to be off for about 20mins. I had also recently got management to buy a lovely second-hand behemoth of a UPS with modular and redundant everything.
    So it was solely providing A+B feeds to everything instead of the two very overtaxed tower UPS we had before.

    The electrician asked how long I needed to get everything shut down and I proudly told him we didn't need to do that as the fancy UPS had reserves for an hour. I then reached into the rear power panel to open the breaker for the mains incomer and got...the EPO (Emergency Power Off) breaker instead.

    The noise of a room full of servers and lots of networking gear spinning down from hurricane to silence is possibly the worst noise in the universe.


    I told the electrician I had to go but that he could start work and then did the adult thing of telling the staff that unfortunately a power cable had been found to be defective while doing essential maintenance but that I would have it fixed very soon.

    With incredible luck all the gear came back up cleanly with no data loss. Not my finest hour though 😦



  • @Onyx said:

    I'm using LinuxWindows exclusively in VM. Install some reasonable desktop like XFCE and it works like a charm.I manage with it's window manager although I miss some stuff in it. Been doing professional work like this for years.

    There, updated for myself

    But the games, man. Will someone think of the games!?


  • BINNED

    Steam. Wine. Most of the new AAA stuff is mostly uninteresting to me. And a very large portion of interesting ones has a Linux port these days.

    I still haven't completed a third of my smallish Steam library anyway.


  • BINNED

    Ground wire? What's that?

    Seriously, why the hell wasn't that thing grounded?


    Filed under: Reading posts in reverse order, Doing it Wrongâ„¢


  • ♿ (Parody)

    Solitaire has been ported to most platforms. And 2048 is a simple web page.


  • Discourse touched me in a no-no place

    @Onyx said:

    Seriously, why the hell wasn't that thing grounded?

    Because the grounding wasn't working. If it had been, I wouldn't have been shocked. (Hey, it was a long time ago, and this was at school.)


  • Discourse touched me in a no-no place

    Argh! That game! (I've yet to get higher than 65536; I'm not even sure if it is possible…)


  • BINNED

    @dkf said:

    Because the grounding wasn't working. If it had been, I wouldn't have been shocked. (Hey, it was a long time ago, and this was at school.)

    Oh, so it was there but faulty. That's something, I guess.


  • ♿ (Parody)

    @dkf said:

    Argh! That game! (I've yet to get higher than 65536; I'm not even sure if it is possible…)

    It is:

    Google tells me that others have gotten much higher, but that's my best. I haven't let myself play in a while.


  • I survived the hour long Uno hand

    When I worked at an ATM company, I was given one of our brand new testing rigs to play with. It contained a real ATM monitor, the back-screen on a hinge so I could swing it around to view both at once, and a series of bare circuit boards that simulated ATM hardware (as close to real as would work naked).

    Inside of a week I'd fried the bill dispenser board. Twice. Why did the stupid power plug even have the CAPABILITY of being plugged in backwards? Not to mention carpet and circuitboards don't mix. I am just not a hardware person >.>


  • Discourse touched me in a no-no place

    @boomzilla said:

    It is:
    [...]

    Sorry, I meant as an individual square score. My total score for my current game is 1671992, but I'm trying to cut down the time I spend on it…



  • GREAT STORY!!! I have had moments like this.


  • ♿ (Parody)

    Oh, yeah, I stopped myself waaay before I got that far.



  • Oh...this is perfect...Great story!



  • I am thinking of staying positive each day. Once you are brimming with positivity, things can just not go bad. Also there is no one hindu god of mischief.



  • @Onyx said:

    Steam. Wine. Most of the new AAA stuff is mostly uninteresting to me. And a very large portion of interesting ones has a Linux port these days.

    I still haven't completed a third of my smallish Steam library anyway.

    Yay!

    Seriously, Linux in VirtualBox, you can do everything, literally everything you need for work. Then you go back to Windows and enjoy some nice leisure time, almost washing away the slight tinge of Microsoft's dick in your mouth.

    With Linux host, it's always something missing. You can forget about using any exotic-ish hardware. Have an old PCI TV card or a custom device you need to interface with? Tough luck. The best PC games ever made are all for Windows. New games, it's 1 in 5 chance there will be Linux version. They MIGHT work like dog in WINE or using the buggy VirtualBox 3D acceleration support.

    There's just too much legacy, man. The only way Windows is going away is if PC goes away. Don't see it happening.



  • @cartman82 said:

    Seriously, Linux in VirtualBox, you can do everything, literally everything you need for work. Then you go back to Windows and enjoy some nice leisure time, almost washing away the slight tinge of Microsoft's dick in your mouth.

    That's exactly what I do for development work on my home computer. I have VirtualBox set up to use the second monitor in full screen, so I can have Windows on one screen and Linux on the other.



  • I install virtual box from Oracle, but not able to get any other OS to work. More study is required, but I am having only so much time when I am on phone with onshore call with clients.



  • @dkf said:

    Argh! That game! (I've yet to get higher than 65536; I'm not even sure if it is possible…)

    Surely there's a theoretical maximum.


  • I survived the hour long Uno hand

    Let's see... if 2048 is the largest tile (two 2048s don't combine to make a 4096 tile, which is implementation-dependant anyway

    Then I scrolled up and realized that they do. So let's call the max tile N.

    ), and there's 16 squares, the theoretical max is 16N.
    But at the move before you had that, you'd have had to have two N-1tiles to combine, leaving one empty square when you do so. Which makes the max 15
    N. But to get to N-1 you had to have two N-2 tiles to combine....

    Then I realized the scoring isn't based on the number showing on the tiles and I gave up.



  • I just played for a bit, and since stuff disappears, I suppose it's open ended.

    @Yamikuronue said:

    Then I realized the scoring isn't based on the number showing on the tiles and I gave up

    You get the score of the combined square, looks like.


  • Discourse touched me in a no-no place

    It's made more complicated by the fact that tiles that appear are either 2 or (with lower probability) 4. Also, how far you can get depends a lot on what mode you're playing in.

    Getting to have a 2048 tile without undo is fairly tricky.
    Getting to have a 65536 tile even with undo is damn hard.

    I've no idea if a 131072 tile is possibly attainable at all. If it is, it requires both perfect and lucky play. (The issue is that you need a 65536, a 32768, a 16384, a 8192, … down to a 4, and they all need to be perfectly placed next to each other too.)

    #WARNING
    This game is a massive time magnet for the somewhat OCD-affected!



  • @dkf said:

    Getting to have a 2048 tile without undo is fairly tricky.

    Oh, I got one playing for the first time ever.

    Got stuck rather quickly after that though.


  • ♿ (Parody)

    @dkf said:

    Getting to have a 65536 tile even with undo is damn hard.

    Ah, maybe this explains a lot. I've never played with an undo.



  • I think I may have been full of shit.

    256 was the highest box I got.

    oh memory


  • ♿ (Parody)

    That sounds reasonable for a first try. It took me a couple of games to really understand what was going on.



  • Damnit, almost got a second 256.



  • work?

    what?



  • this game didn't have stay power for me. Played it for 3 days a few times a day went back to bejeweled.

    (which is my old standard)


Log in to reply