Oh Crap.



  • Does anyone want to share an "Oh Crap." experience?

    Basically, it goes like this. You think you're the man. You think you're the best coder on the planet. You decide to show off your skills in front of other (less intelligent than you) programmers, only to realize that you made a complete fool of yourself...in front of them all.

    Usually the comments afterwards are along the lines of "Nice work Einstein." or "Way to go genius." or "Slick one!"

    And that's usually followed with an embarrassing slap on the back which says "Nice job. Now find a way to fix the issue that lost the company x amount of dollars."

    Has anyone been in a similar situation?
     



  • We just did one of these a couple months ago:  http://forums.worsethanfailure.com/forums/thread/114558.aspx



  • On the project I'm working on now, I tried to show my boss an, imo, pretty damn cool feature I made.  Esentially, the feature gives a bunch of customization to the program such that, theoretically, I don't have to make major changes to program.  Need the program to act differently?  A couple lines of XML will change the behavior under different conditions.  Really useful for the testers, which is who the program is aimed at, since it's good for testing boundry conditions and such.  So I sit down with him, all proud of myself, and go to run the program with a customized option.  Program is running and gets to the point where the customized portion should take over... it doesn't.  OK.  I figure I defined it wrong, double check some stuff and run it again.  Still doesn't work.  It had JUST been working.  And I'm annoyed, manager thinks it's funny and isn't all that impressed, and walks away.  A half hour later, I realize my mistake.  I had commented out the wrong line of code, commenting out the call to the method that makes the customized stuff run. 

     D'oh.  
     



  • True story.

    At the last job I was at we had a developer that was in control of our source control server. We were in the process of migrating from an old box to a much newer, faster one. During this whole process, the decision was made to not back up the server's contents (don't get me started).

    Anyways, it came down to the day where the new server was set up and we even knew the backups would work on it. All the developer had to do was copy over the appropriate files. Well, this developer wasn't too swift, and he did a shift-delete on the entire directory when he meant to simply copy and paste.

    All of our development C# stuff was gone in an instant. Weeks of work down the drain. Some of us had local copies of certain stuff, but other projects were simply lost...

    His excuse was that he had just been shift-deleting through some emails and got into a routine. That was a stupid answer so he was fired that afternoon.



  • @CPound said:

    All of our development C# stuff was gone in an instant. Weeks of work down the drain. Some of us had local copies of certain stuff, but other projects were simply lost...

    At least it was only weeks worth of work, not years.  Reminds me of a similar story.  At a previous company (all these stories seem to be "previous company") we had a very strange backup procedure.  I don't remember the exact details but we would only backup files less than a certain size, maybe 1MB, but that seems large actually.  Well, one day the server HDs up and died, so we go back to the backups.  Hrm... where'd our entire revision control system (PRCS) data go?  Oh, the RCS files were too big to be "worthy of backup".  Luckily most of us had recent checkouts of the code so we got most things back.  There were a few code branches that we never got back, one of them being a very recent release of software that we still had to maintain.  Those were the days.  Of course after that day they decided that the extra costs for a proper backup plan was worth it.



  • @CPound said:

    His excuse was that he had just been shift-deleting through some emails and got into a routine. That was a stupid answer so he was fired that afternoon.

    He may have been stupid, and had a stupid answer, but the real villain here is the process that enabled a shift-delete to remove weeks of development.

    I'm sure I am preaching to the choir here, but I just had to say it. 



  • @R.Flowers said:

    @CPound said:

    His excuse was that he had just been shift-deleting through some emails and got into a routine. That was a stupid answer so he was fired that afternoon.

    He may have been stupid, and had a stupid answer, but the real villain here is the process that enabled a shift-delete to remove weeks of development.

    I'm sure I am preaching to the choir here, but I just had to say it. 

    I sort of agree. Where are the daily tape backups that could have kept losses down to 1 day?

     

    I don't remember any Oh Crap situations from myself, but I doubt I'm that perfect and so I conclude I have repressed them.

    A couple of coworkers of mine erroneously changed permissions on a non-local, hosted box via RDP.

    And locked themselves out.



  • @dhromed said:

    A couple of coworkers of mine erroneously changed permissions on a non-local, hosted box via RDP.

    And locked themselves out.

    I did something similar with my WiFi router a few months ago, I turned on some encryption protocol without checking first to see if my wifi card supported it. Since you configure the router using via its WiFi network and one of the security features was that the chosen security protocol wouldn't reset when you physically reset it, I nearly turned the whole box into an expensive, rectangular Christmas tree decoration. I had to get a friend to bring his laptop over so he could log in and fix it >_<



  • you didn't have a spare Ethernet cable around?



  • @tster said:

    you didn't have a spare Ethernet cable around?

     ... Because the ethernet wouldn't be encrypted...

     ... Oh yeah ...

    I feel a bit stupid now :p
     



  • Mine was while demo-ing a change to an Access system.  I needed to empty out a test copy of the database for us to feed some test data through etc..   Guess what, I had the wrong database open when I emptied the 4 data tables.  We only noticed when the new features didn't work.  Not a problem, the user has a tape drive on this PC and uses it.  Not for the last 4 days!  That was a bad oh crap feeling.  Thankfully he had taken a copy of the data to do some reporting that morning so we did get it all back, but I was thinking I was heading for the sack for a while.

    A co-worker of mine once missed the tape eject button on a server.  Bear in mind that this is THE MOST critical application that we support.  He hit the power button, before the days of hold for 4 seconds, the box just turned off.  Can't remember if the system in question was Ingres or Oracle at this point, but it didn't like being turned off and took hours to get going again.


Log in to reply