It pays to have a spine
-
I was assigned to automate a particularly convoluted process that would fool around with production DB tables in the middle of the day. One of the requirements was that it could not fail, no matter what, because the business depended upon it. This was intended to be used by developers as intervention in production when a rare circumstance occurred and management declared that this "procedure" be performed.
I went through and coded the basic steps of performing the process. Since I didn't want to have to do any support for this potential nightmare, I started in on checking for every conceivable error condition that I could imagine. For each error, I not only displayed what it was attempting to do at the time (and with what commands), but what likely went wrong, a likely reason as to why, and a list of specific steps to correct the situation so that the process could be restarted at the next step.
It was a serial sequence of scripted tasks; the amount of error checking dwarfed the actual processing.
When I handed it off to the QA team, they found that testing the error handling required waaaaaay more time than management would allow them, so they (correctly IMHO) refused to sign off on it. Management decided that since they couldn't spare more time for testing, that the best solution was for me to remove the more convoluted error checking.
QA and myself pointed out that given the can-not-ever-fail constraint that this was not wise. Still, they persisted.
I refused to do it because it would make me liable (in this business, it doesn't matter if users or managers assume the risk; if something goes wrong, YOU should have forced the issue, and as such, YOU are to blame). Since $millions were at stake, I stood my ground. This did not win me any friends in management.
They had someone else remove (comment out) the error checking.
The first few times it was used, nothing bad happened.
The next time out, something failed silently (without throwing an exception), so the script could not detect the error, and all sorts of badness happened, requiring a full db restore (3 hours) plus lots of redoing of work since the last checkpoint, all costing lots of $$$. After lots of managers shouting at everyone in development and QA, I pointed out that one of the commented-out error checks would have caught the error. Then I had to prove it, so we recreated the failure in QA and sure enough, the problem was caught.
Of course, the commented error checks were uncommented and QA was ordered to take the time to do the necessary testing.
Why oh why does management always need to get burned before they listen to us?
-
I have nothing to add other than awesome story. Thanks for sharing!
-
At what point will Alex finally take notice of how awesome snoofle is and make him a front-page writer?
-
@Welbog said:
At what point will Alex finally take notice of how awesome snoofle is and make him a front-page writer?
I agree in full. snoofle always has the best stories (poor snoofle).
-
Sadly, these "stories" are what I live in real life on a daily basis.
Take a job on Wall Street working for any one of the brokerages or big banks and you too can get paid to watch this stuff unfold in real time...
-
@snoofle said:
Sadly, these "stories" are what I live in real life on a daily basis.
Take a job on Wall Street working for any one of the brokerages or big banks and you too can get paid to watch this stuff unfold in real time...
Do I get a 6+ figure bonus?
-
@blakeyrat said:
@snoofle said:
Sadly, these "stories" are what I live in real life on a daily basis.
Take a job on Wall Street working for any one of the brokerages or big banks and you too can get paid to watch this stuff unfold in real time...
Do I get a 6+ figure bonus?
About 12 years ago, I did get (low) 6 figure bonuses for a couple of years, but that's long over with for tech folks.
The WTF'ery is astounding, but you can get paid very well if you're good at what you do.
-
Please tell me u don't work for nyfix.
-
@astonerbum said:
I don't. I almost did though.Please tell me u don't work for nyfix.
-
Great, tragic, and dramatic story. It has the hallmarks of a classic story, although usually in classic stories, the disaster would happend because the mortals wouldn't listen to the gods, instead of self-appointed gods not listening to the mortals. I guess in those days, the hoi polloi simply had to swallow their frustrations.
With respect to your last question: I guess you already know the answer, don't you?
-
@Welbog said:
At what point will Alex finally take notice of how awesome snoofle is and make him a front-page writer?
Don't be daft, snoofle's posts don't have nearly enough grammatical errors to qualify for front page material
-
Great story. I'm sure we've all seen many many similar management WTFs of this nature over the years.
Why is it that as soon as someone puts on a suit and tie they lose the innate human ability known as Pattern Matching?
e.g.
You: "We shouldn't do this."
PHB: "Why?"
You: "Because of [insert logic here]. Remember all the previous projects that failed in this way?"
PFB: "Ah, but *this* time we'll use [latest fad / dumb suggestion]."
You: WTF.
-
Nice story. Good thing you stood your ground.
