Make noise when something fails
-
A few days ago, I told you all about a newly born Developmestuction environment. It crashed and burned after one day.
However, not to be discouraged, I suggested to this same boss that when something dies, just writing a log entry and continuing on with a null pointer might not provide the best reliability (from the customer's perspective). I suggested that we simply have the code, at the point of failure, call some common routine to send an email to the support team (ok, it's just the developers). He decided that people might not check their email frequently enough, and that we needed something "better". Thus was born the concept of the noise-maker.
This is a process that uses a heartbeat-like ping to all monitored processes (ok, this part is reasonable). Upon the absence of 3 heartbeats, the monitor process will play an audio file on a pc located in the room that contains the sysadmin's desk, telco switches, servers, 2 air conditioners and six 20 inch fans. This way, he will audibly HEAR that there is a problem.
I queried what would happen if the sysadmin (the only one who sits in there) were not there (eg: bathroom, lunch, gabbing with others, doing actual support at someone else's desk) to hear the audio file being played, or further, if the fans and what-not were simply overwhelming the music?
He thought for a moment and decided that it should play very loudly to be heard (over the fans, etc) by folks in the [i]other[/i] rooms. It took about an hour to implement. The developer even went so far as to allow programmable music for each environment, so you could tell by the song that was blasting just what had happened, and in which environment.
For example, we have "Don't Fear the Reaper" when a reaper process (that cleans up left-over resources) croaks.
Now, every time someone bounces a server, the thing blares the warning-song, at fog-horn decible levels - throughout the entire office. After just 4 hours, folks are intentionally bouncing dev servers just to hear the music.
I am resigning tomorrow.
-
why resign? how about becoming the office DJ and arranging beautiful mixes? Or, instead of songs, you can set up words and phrases. Then, start wreaking havoc on the servers in order to create conversations.
Okay, in the time it took to write that sentence, it went from funny to just plain sad.
Run.
-
What happens if you hire a deaf sysadmin? Better add strobe lights, too.
-
I think you should change the music to "Take this job and shove it!"
-
@djgerend said:
I think you should change the music to "Take this job and shove it!"
Funniest thing I heard all day!
-
I was thinking more along the lines of a big 'flush', but your idea is better. Unfortunately, they've decided that the door to the server room (sysadmin's office) should be kept locked - for security, so now we just hear muffled music when stuff happens.
-
What's the server bouncing theme song? Can I recommend "Can I get a?" by Jay Z?
-
At my old job we had a server that would ping our various external-facing applications. When one of them went down, it would play a soundfile as well. The soundfile was, basically, an air-raid siren (wooooOOOOOOOOOOOOOOOoooooo!!!!). So when a server went down, we had no idea whether it was a bad server or if we were under attack by the Soviets :)
-
I wonder what happens when "Super Important Amazing Master God Database" server dies, but then Joe Schmo's "test development" server dies half a second later.
-
What does the sys admin think of blasting music in his room/office thing? And this is the funniest topic I've read this whole week
-
Hmmm... so do you have to pay the RIAA whenever a server crashes?
-
@batasrki said:
What does the sys admin think of blasting music in his room/office thing? And this is the funniest topic I've read this whole week
We just had a floor-wide power outage - apparently there is a single 300 amp fuse that runs the whole place, and we have not one single ups in the place; The building engineer had to run to a couple of hardware stores to find one (yes, he only bought one).
Anyway, I just asked the admin what he thinks of the loud music; he doesn't notice so much as "I'm shell-shocked from all the fans, air conditioners and servers - a little more noise doesn't matter". He did, however, request that we choose music more to his liking (he's into classical, so I can't imagine what selections HE would choose).
With apologies to all those inflicted with wtf after wtf after wtf, but this week at this place has been the mother-of-all-wtf's, and I am *SO* glad that tomorrow is my last day here.
-
@Foosball Girl In My Dreams said:
... and I am SO glad that tomorrow is my last day here.
I am not! (well, maybe for you)
I have enjoyed your stories.
-
@dande said:
@Foosball Girl In My Dreams said:
... and I am SO glad that tomorrow is my last day here.
I am not! (well, maybe for you)
I have enjoyed your stories.
Well, this is my last day at THIS company. I'm sure the new company (big place) will have some worthy wtf's to tell, and I shall continue to post.
BTW, IIRC, my posts this week have been 100% true. I find it sadly interesting that truth is stranger than fiction.
-
@Foosball Girl In My Dreams said:
I am resigning tomorrow.
Indeed, definitely time to start looking for the nearest hills
-
Reminds me of this story:
[QUOTE user="Ping Website"]
The best ping story I've ever heard was told to me at a USENIX conference, where a network administrator with an intermittent Ethernet had linked the ping program to his vocoder program, in essence writing:
ping goodhost | sed -e 's/.*/ping/' | vocoder
He wired the vocoder's output into his office stereo and turned up
the volume as loud as he could stand. The computer sat there shouting
"Ping, ping, ping..." once a second,
and he wandered through the building wiggling Ethernet
connectors until the sound stopped. And that's how he found the
intermittent failure.[/QUOTE]
-
@savar said:
What's the server bouncing theme song? Can I recommend "Can I get a?" by Jay Z?
I suggest "Drop it like it's hot".
Crazy stuff, this.
-
That must be the funniest wtf I've heard in weeks. All you need now is to change the music file to one of a woman screaming and run a pool on how long it'll take for the cops to arrive.
-
One final update before I run out of this place...
Today we moved to a new office in the same building, and had to drag the equipment ourselves. When the move started, the admin forgot to turn down the volume on the pc in his office. Because it was near 100 degrees outside and the A/C was struggling to keep up, he left the door to his office open. Naturally, the servers were off line as they were being moved. Of course, the "alarms" kept blasting though the office. The CEO, whose office is about 30 feet from the admin's office got royally PO'd at the stupidity of it all, raised hell, and demanded to know what idiot (his word) decided to blast music throughout the office. In a selfless display of revenge, I stepped forward and pointed at our boss. He is still in the CEO's office getting ripped a new one as I write this.
Parting is such sweet sorrow...
-
@cconroy said:
What happens if you hire a deaf sysadmin? Better add strobe lights, too.
And a disco ball.
And a floor with random flashing lights.
-
@jgayhart said:
@cconroy said:
What happens if you hire a deaf sysadmin? Better add strobe lights, too.
And a disco ball.
And a floor with random flashing lights.
Emergency! Paging Dr. Boot! EMERGENCY!
*runs*
-
I was looking at a new unit testing framework for some embedded sensors that I'm working with. Their presentation had a couple of (obviously tongue-in-cheek) suggestions for ways to alert developers when they break the build or cause regressions.
My favorite suggestion was triggering a smoke machine installed inside the server. I'm sure it's incredibly awful for the electronics inside; but I just love the way it implies that your code is so awful that it actually made the build machine melt down.
-
@SomeGuy said:
I was looking at a new unit testing framework for some embedded sensors that I'm working with. Their presentation had a couple of (obviously tongue-in-cheek) suggestions for ways to alert developers when they break the build or cause regressions.
My favorite suggestion was triggering a smoke machine installed inside the server. I'm sure it's incredibly awful for the electronics inside; but I just love the way it implies that your code is so awful that it actually made the build machine melt down.
Make it a gas release + spark mechanism, or some apparatus that flings bits of server shrapnel into the office.
-
@dhromed said:
Make it a gas release + spark mechanism, or some apparatus that flings bits of server shrapnel into the office.
heh heh...
"Uh oh... Exchange001's down. HIT THE DECK!!"
*BOOM* *Sounds of low flying, high velocity pieces of server* *THWAP*
"Aagh! My eye!"
"Dude, you should really put a bandaid on that."
ah... the wonders of an overactive imagination...
</offtopic>
(And I've just noticed that, my comment seems to go really well with my avatar.)
