Lost Source Code
-
About 10 years ago, I worked at a rather large company, that had all sorts of policies on source code repositories, code reviews, etc. Over time, numerous systems were built that tended to daisy chain data from one system to the next. One such system had an applet (running on a user's PC) access a servlet (running on a Unix server) which accessed an EJB-session bean (on another Unix box) which did an RPC to a CORBA (C++) server (on yet another Unix box) that made a COM call which accessed a DB on an AIX box. Then the whole thing repeated, in reverse sequence, in order to return data to the applet.
One day, my boss's boss pulls me into his office and tells me that we need to upgrade the COM service, and the following conversation ensued:
Me: No problem, where's the source? Boss: It's been lost Me: How can it be lost? All the repositories are backed up! Boss: The manager of the system didn't follow procedures, the code was stored on someone's local C: drive, and the PC has since been trashed Me: OoooooKaaaayyy - where are the spec's? We can just rewrite it Boss: It was a one-off project - no specs Me: Where are the original people? Programmer? Manager? Someone who might remember what the thing was supposed to do? Boss: All have left the company Me: What systems access it? We can look at that code and see what it's trying to do, and determine the specs like that Boss: Unknown Me: We can put a packet-sniffer on the box and see where all inbound packets are coming from, then see what applications are running on those machines and work it backward from there Boss: Already tried - the security folks won't go for it - corporate policy Me: Turn off the COM server and see what breaks, then work it backwards from there Boss: Can't do that: risk to business is unknown Me: (laughing hysterically) - short of getting Jeannie to cross her arms and blink...Two months later, we get approval to shut it down. Within two hours, we got about two dozen system errors, and tracked back the requests. We were able to reverse engineer the requirements in two days, and replace the system in about 10 days.
God, I love my job, if only for the entertainment value :)
-
Ha! Awesome story - sounds a lot like where I work.
Only where I work, it's because of me.
-
@Jeremy D. Pavleck said:
Ha! Awesome story - sounds a lot like where I work.
Only where I work, it's because of me.Care to elaborate?
-
@codeman said:
@Jeremy D. Pavleck said:
Ha! Awesome story - sounds a lot like where I work.
Only where I work, it's because of me.Care to elaborate?
Oh, I suppose so hehe.
I work in Systems Management - the only guy in my company that does so. In fact, I've been doing this 10 years and I've always been the only guy that does it. Annoying, in hind sight.
Anyway, I deal with graphing server & application metrics, monitoring core processes and alerting the proper parties when things are failing or going to fail.
It's a mixed shop (Windows and Sun) so I have MS Operations Manager and Nagios + a ton of little tools here and there, like scripts to monitor certain email folders to look for problems. And a lot of little programs and scripts that, all together, create a pretty impressive looking monitoring portal I designed.
All the code resides on A. my laptop and B. one of my management servers.
I'm not allowed access to the developer things, even though technically I'm a developer. All the config files, tricks to install X, etc are all on one or two servers as well.
I've been here 3 years, and just recently starting writing documentation (When things got to jumbled together to figure it out on their own).
My portal, for example, relies on 1 mySql DB, 3 MSSQL databases, and the outputs of 4 other programs, all spread out on different servers and feeding into the main one.
It's this huge spaghetti mess of code but, it's the best I can do considering I'm swamped and they won't hire anyone to help me. You'd probably be in the same situation if you were the solution Architect, evaluated solutions, designed, tested, deployed, then did day to day maint whilst developing brand new solutions to new problems.
All in all, it's nerve-wracking, but I *love* it. I'm essentially allowed to do what I want; work on what I want to work on, implement what I want to implement, code when I feel like it, etc etc. Change Authorization docs? Pfffft, I'll just do my changes live, in production.
I'm one of those guys in the company that plays a crucial role to the well-being of the systems, but you don't know that - you don't know what he does, or why he's here, but he's here. My boss is envious of my position, and views it as what everyone in lvl 4 (Networks, Design, Infrastructure & Architect) should be doing.
If I wasn't in Systems Management, I probably wouldn't be in IT still. To many politics and all.
This is probably way to long, but hey - the Adderall kicked in.
-
@Jeremy D. Pavleck said:
@codeman said:
@Jeremy D. Pavleck said:
Ha! Awesome story - sounds a lot like where I work.
Only where I work, it's because of me.Care to elaborate?
Oh, I suppose so hehe.
I work in Systems Management - the only guy in my company that does so. In fact, I've been doing this 10 years and I've always been the only guy that does it. Annoying, in hind sight.
snip...Sounds like fun <jealousy>
-
@codeman said:
Me: No problem, where's the source?
Boss: It's been lost
Me: How can it be lost? All the repositories are backed up!
Boss: The manager of the system didn't follow procedures, the code was
stored on someone's local C: drive, and the PC has since been trashed
This obviously doesn't apply exactly the same everywhere, but here's something we do in my IT shop--at least in my group anyway. We write WebSphere Portal applications and deploy them as WAR files. (For those that don't know: JAR = Zip with particular layout and specs to store a Java application module; WAR = JAR with more particular layout and specs, for a "Portal" application, IIRC.) We include all the source code in the deployed WAR file. Thus, if the application is up and running, we are guaranteed to have at least one copy of the source code in an obvious place where it can be found. I did the same thing with the special HTTP proxy server that I concocted to help our search engine make sense of the portal.
It's just like duct taping hardware driver CDs to the inside of a desktop PC's case. Keep copies of the ingredients close to the product itself.
