It's Whose Resposibility?

snoofle

This just happened today.

Although our system is technically a whole separate system, politically, it's owned by another team. Naturally, even though we have our own app-level DBAs, our database also falls under the purvue of the db-server DBAs of the other team.

Of course, they can make/veto application-level changes to our db, but any changes we want to make to our stuff that references their tables must go through them (their stuff is all read-only to us).

My boss told me to run several scripts written by another developer. These scripts changed our stuff in the db of the other team, so I sent the request to their dba. He punted back to me saying I could do it (in dev). I replied I was not given the passwords and the rule was that we are not allowed to touch their side of things. Five emails later, he asks me to have another DBA do it. The other DBA replies to all that he is not allowed to change their DB because he doesn't have the password. The first DBA emails the password back. The receiving DBA then complains to auditing that a cleartext password was just mailed to him.

Hilarity, and an email storm ensues.

200+ emails later, we get back to the original request, only this time, half the firm is on the cc list, and everyone is replying to all.

50 emails (plus thousands of CCs) later, they decide that the guy with the password (the original DBA at the start of the email chain) must do it. He declines because it did not come from his boss (who has been on the cc list for about 100 emails at this point).

More hilarity, and another email storm ensues, complete with at least 25 diffferent off topic threads, all cc'd to everyone.

At this point, the email is approaching 100K and growing rapidly. And the space available to the Exchange mail server was shrinking just as rapidly.

Finally, the server runs out of space, and the email storm stops.

But the script has still not been run because they can't decide who owns the fucking database.

On the other hand, I'm being paid to sit here waiting for someone to run these scripts (probably 30 seconds worth of work), and I get to go home soon.

hoodaticus

Sometimes I cause these bureaucratic system-wide spinlocks on purpose because I don't feel like working. But shutting down the exchange server... that takes it to a whole new level.

Sutherlands

Well, you have the password now, can't you just run it? :)

locallunatic

Isn't this also a place with way too many progress meetings? You'll get to report "I did squat cause someone won't run the dumb script" (hopefully it will be able to reignite the email arguing).

snoofle

@Sutherlands said:

Well, you have the password now, can't you just run it? :)

I had the password all along, but the rule (from the other team) is that we are not ALLOWED to run scripts in their db (there have been formal meetings about not touching their db); we are supposed to ask them.

Tehnically, I *could* do it, but who wants the headaches?

snoofle

@locallunatic said:

Isn't this also a place with way too many progress meetings? You'll get to report "I did squat cause someone won't run the dumb script" (hopefully it will be able to reignite the email arguing).

I hadn't even considered that. +1

Medezark

If you did run it, since apparently everyone now has the password, how would they know it was you?

hoodaticus

@snoofle said:

Tehnically, I *could* do it, but who wants the headaches?

Maybe you could get one of their backup tapes, make your changes to it, return the tape, and then sneak in and massage the drive array with a rubber mallet?

Just a thought.

hoodaticus

@Medezark said:

If you did run it, since apparently everyone now has the password, how would they know it was you?

Management tends to go after the last person to express interest.

@hoodaticus said:

Maybe you could get one of their backup tapes, make your changes to it, return the tape, and then sneak in and massage the drive array with a rubber mallet?
Just a thought.

Quoted for awesome deviousness.

blakeyrat

@snoofle said:

200+ emails later, we get back to the original request, only this time, half the firm is on the cc list, and everyone is replying to all.
50 emails (plus thousands of CCs) later, they decide that the guy with the password (the original DBA at the start of the email chain) must do it. He declines because it did not come from his boss (who has been on the cc list for about 100 emails at this point).
More hilarity, and another email storm ensues, complete with at least 25 diffferent off topic threads, all cc'd to everyone.

Did anybody email "UNSUBSCRIBE" to the thread yet? That's always hilarious.

da_Doctah

@blakeyrat said:

Did anybody email "UNSUBSCRIBE" to the thread yet? That's always hilarious.

*Insert standard Gern Blanston message here*

hoodaticus

@blakeyrat said:

Did anybody email "UNSUBSCRIBE" to the thread yet? That's always hilarious.

I do that all the time. I'm not sure if anyone gets the reference though - to that dark day at Microsoft, when even Exchange couldn't keep up with the traffic.

lolwtf

@blakeyrat said:

UNSUBSCRIBE

me too

tgape

@hoodaticus said:

@blakeyrat said:
Did anybody email "UNSUBSCRIBE" to the thread yet? That's always hilarious.
I do that all the time. I'm not sure if anyone gets the reference though - to that dark day at <every big company>, when even <any mail server> couldn't keep up with the traffic.

FTFY. Btw, Exchange may be awesome (or it might not be - I'm not weighing in on that discussion), but its performance generally doesn't exceed that of stripped-down MTAs such as sendmail, qmail, and postfix. Sure, I've seen Exchange servers that could handle more email than a particular qmail server, for example, but only because the Exchange server in question had many fast CPUs, one or more huge RAID 1+0 (or 0+1) arrays, gobs of memory, and the qmail server had a single, ancient, slow CPU, one hard drive, and less RAM than the Exchange server's OS's minimum RAM to boot. Not that these are the fastest MTAs. I can't remember what the fastest MTA I've heard of was, but I remember that it was one that was *actually* stripped down, rather than just not having ever been technically built up as much as Exchange or Notes.

Consider yourself lucky, though - at one of the companies at which I worked, the message was to the 'all technical employees' list. The CEO considered himself a technical employee. He chose to forward all of the unsubscribe requests to HR: these people don't want to be technical employees anymore. Of course, this did happen at a mostly competent company, so there were only a few dozen people that had a very rude surprise the next day... Of course, the real problem there hadn't been the 'UNSUBSCRIBE' requests, but the people responding to the few who tried unsubscribing (but, of course, replying all), explaining part of why they were unwise and should just hold tight.

cdietrich

To me the single funniest event ever is the UNSUBSCRIBE email to all 100+ people. Then, the subsequent, "I ALSO WISH TO BE REMOVED FROM THIS LIST"
Edit: this is in reply to blakeyrat's message Edit 2: I remember being able to format the style on TDWTF but it's late and I've given up

dhromed

@snoofle said:

Finally, the server runs out of space, and the email storm stops.

:D

Straight out of The IT Crowd.

@snoofle said:

I had the password all along.

BONUS.

Gern_Blaanston

@da Doctah said:

@blakeyrat said:
Did anybody email "UNSUBSCRIBE" to the thread yet? That's always hilarious.

*Insert standard Gern Blanston message here*

What?

SteamBoat

@lolwtf said:

@blakeyrat said:
UNSUBSCRIBE
me too

me too.

I was there when that happened. I have a fantasy that it was me that started it. At the very least I "me too'd" too.