On-Call at the Muse Concert



  • So, I was reminded of something that happened last year, and remains one of the bigger WTFs from my working life.
    It takes place late last year, when I was still officially part of the support team for my work, and thus part of the on-call roster. Our company works with various emergency service agencies, so any out of hours calls are extremely urgent (literally, life and death), but also extremely rare (as they have layers of support staff, us being at the top of the pyramid, and the only time when things goes tits up is when someone is making changes to the system).

    By now, if you've read the topic title, you know where this is going. Muse is playing, and I've got tickets for my gf and I. I'm on-call, but not thinking much of it, seeing as there's no known work happening with our clients, and the chances of something randomly falling over is next to 0. Found our seat and the supporting act is over. Lights starting to dim and the opening song is starting... and my phone gets a text message from our on-call paging service. Fuck! Seriously, right now? Alright, let's duck out into the foyer and see what they want.
    System has ground to a halt, none of the users can do anything. Oh, and this all started when they started some routine maintenance that they hadn't informed us they were doing. I fucking run back my car, grab the laptop, run back (talking my way through security) and set up and dial in in the foyer. All the time, I can hear the muffled sounds of the Muse concert. Eventually, I've gotten the client back at a point where the system is usable, but needless to say the maintenance isn't continuing tonight. Leave the laptop with the administration desk, head back in to my seat...just catch the end of a song before they have some downtime before the encore. At least I'll still catch some of it, right?

    Nope. Just as the encore starts, my phone rings again. Mother. Fucker. I go back out to the foyer to take the call, which is just basically a debriefing and discussion on what we're going to do tomorrow to work out why it ground to a halt (spoiler: they forgot to turn off a setting that stops the system reading from the server they were taking down to do maintenance on. The system being unusable was because every action would have to wait for a connection attempt to the offline server to time out). By the time this call ends, I've missed the encore too.

    In the aftermath, the client was ripped into a bit for having not notified us of the planned maintenance for that night (had we known, I obviously would have swapped on-call to someone else to avoid exactly what happened). The only compensation I got from the whole ordeal was a sympathy bottle of wine from the owner of the company. And a couple months later, I was promoted to software development, so I no longer have to be on-call.

    Oh, and I'm obviously still pissed off about the whole debacle.

    tl;dr - client doesn't notify us of planned maintenance the night I'm going to a Muse concert. A night I happen to be on-call for. Murphy's Law happens and I miss the whole fucking concert.



  • I've always wondered if it's ever worth it to be on-call.

    I'm not too interested in finding out the answer, though.



  • I'd say, only if you have nothing else planned, and if you get either time in lieu or overtime pay for it. iirc, I took the following morning off after the above incident.


  • Discourse touched me in a no-no place

    In my experience, being on call just means you're working without pay.

    TRWTF is Spencer going to a concert without trading on call with someone else, although I'm not going to go so far as to call you stupid or anything. Even if I expected no chance of being called, I would never go to something like that while I had the pager (or whatever.)



  • @FrostCat said:

    TRWTF is Spencer going to a concert without trading on call with someone else

    And in hindsight, I will be too. Although the chances of me doing on-call again are slimmer than the one-call-maybe-every-three-months-or-longer out of hours calls that lulled me into expecting I wouldn't be called.


  • Discourse touched me in a no-no place

    To be fair, when I was running on call, it was more than twice a month (for <6 months), but I got called several times the first few times I was it, so I was primed to treat it as "expect to be interrupted".


  • Discourse touched me in a no-no place

    We get stand-by pay for being on call, and then over time for actually being called out. Minimum of 2 hours over time, even if the call out lasts 5 minutes.
    Stand-by pay isn't amazing but it's alright. I'd draw the line at going to a gig, or on a proper night out, but I don't let it otherwise stop me doing things, I just make sure I'm within 20ish minutes of being able to connect (laptop is generally in the car).



  • Reminds me of a time I was doing on-call... Like you, call outs were pretty rare but certainly not life and death.

    I had a chance to go to a sporting event in another city and for this I'd normally swap on-call with one of the other guys. As fate would have it they were both on holiday but, given the fact we hardly ever got called out I decided to chance it. I had my phone and laptop with me at all times... in the pub on the afternoon... at the event itself in the evening... in the pub afterwards. As expected, no call out.

    7am the next morning and I was lying in bed in the hotel, just dozing and feeling a bit rough when the phone rang. One of the main systems we support had filled its transaction logs and no-one could do anything. No problem, I'll log in. Fire up the laptop, switch on the wi-fi, nothing. (I should mention that at the time my laptop had this weird issue where it would take literally 10 minutes to unlock. You'd hit CTRL-ALT-DEL and nothing would happen for 10 minutes, then you'd enter your password and off you go.)

    I tried turning it on and off; nothing. I tried tethering my phone; nothing. The wi-fi was dead.

    At this stage I was getting panicky so I called reception and asked if they had an ethernet cable but they did not. I'd had a few phone calls by now from my boss and the customer and about an hour had passed. I fobbed them off saying I was having trouble connecting and with laptop in hand (screen open so it didn't lock) I set about finding a wired internet connection.

    I phoned home, not for assistance but just because I needed to talk to someone and my wife suggested the Hilton as they'd have a business centre. I jumped in a cab and piled out at 10 minutes later at the Hilton. No I don't have any bags, I just need the business centre. When I get to the business centre I find that you need a room key to get in but there's an executive lounge on the same floor. I walk in, spot a printer with an ethernet cable, unplug it and now I have a connection but I need a password to get out.

    I call reception (I don't mention I'm not a guest and they don't ask) and ask for tech support. They send a guy up and I explain my lack of wi-fi, borrowed ethernet, please help me. Two minutes later I'm in and five minutes after that the issue is resolved.


  • Grade A Premium Asshole

    @loopback0 said:

    We get stand-by pay for being on call, and then over time for actually being called out. Minimum of 2 hours over time, even if the call out lasts 5 minutes.

    I do the same for our employees. If you are expected to be available, then you should expect to be paid. If you have to work, then pay is on a sliding scale of when. Straight OT for anything outside of 8-5, but holidays and such are a LOT more. I once had a client call needing someone to come to their place on xmas eve. He was all shitty when I quoted him an exorbitant price (I think it was $300/hour, a lot for someone to look at a QuickBooks issue). I explained to him that we do not work on xmas eve and if I am going to ask one of my employees to take time away from their family then I am going to offer them more money than they can refuse or bitch about in order to do so.

    He was not a client much longer after that. It bugged me that he expected his underpaid employees to work on xmas eve.



  • Good on you for cutting him off.

    As a freelancer I get to be pretty picky about who I take as clients. It keeps the blood vessels from popping, but I can understand how if you've got employees you need to file payroll.



  • @Spencer said:

    Muse

    Muse is far inferior to {{all_the_music_I_have_a_quasi-religious_devotion_to}}.

    When asked, I will reel off some additional generic bands that make me sound like a douchebag who can memorise three strings.



  • I did get paid to be on-call, not much but a plus is always good. Also, it made me force my team to be more proactive on fixing shit and not releasing crap that would bring a server down in the middle of the night. We're talking here continuous integration, unit tests, Hadoop for logs. I guess we spent more money and time than for 30 years of oncall service, but at the time I departed from that job I haven't received a call in months.


  • Discourse touched me in a no-no place

    We get extra for Christmas Day etc - especially if we're called out as there's an extra payment. It's rare that our team actually gets called out but it's not unheard of for a Christmas call out of a couple of hours to result in a couple of days worth of pay.
    We get a day extra annual leave per day of bank holiday we're on call for too.



  • Muse are my favourite band. I'd have ignored the pager for the two hours of the concert :)



  • Did on-call back in the day and was called out on Christmas Day, by top-priority client. We only had a pager that showed a call to the office had been made; had to go to the office to actually get to work on the problem. (Seriously - these were times of 56k modems, and we could only reach the client via specific telephone lines) Of course the problem was a paper jam in a printer - nothing to do with our application. And the droid reporting it had simply worked his way down all the support numbers, irrespective of how relevant they were, until he found someone who answered...



  • Though not my all-time favourite, I do love them.
    The next time they're touring down this way, I think I'll be getting tickets for a couple of cities, just to make up for what happened.



  • @Spencer said:

    Though not my all-time favourite, I do love them.

    Same here. Though their last two albums are... meh. They have their strong points, but overall they got pretty bland compared to, say, Origin of Symmetry.

    And I'm still wondering if "Survival" is a legit song, or an elaborate joke.



  • I actually really enjoyed their last two. Not quite as top tier as you'd like but still really great I thought. I love when bands go all over the place with their style and always try new things.



  • @Shoreline said:

    Muse is far inferior to {{all_the_music_I_have_a_quasi-religious_devotion_to}}.

    I was going to say it sounds like the on-call issue saved him from suffering through a concert by a mediocre band! :trollface:



  • I spent 2 years on call (i was the only DBA) for my company covering 200 SQL Servers. Those servers range in size from half a terabyte to 10TB+.

    Once we hired another we started switching off on call duties. After I took a 2 month break that is. We also now get paid a flat fee per week we are on call regardless of whether we are actually called.

    I also got some pretty hefty raises every few months so I can't complain.



  • @Shoreline said:

    When asked, I will reel off some additional generic bands that make me sound like a douchebag who can memorise three strings.

    Do you mean "three chords"?



  • I got a call once when I was in the middle of a dog show in the middle of nowhere in Kansas -- miles from the hotel, and right in the middle of the competition when my wife was showing the dog. Some bug that was in the code I was working on; and of course I was the only one who could fix it (not really, but my manager thought that) and of course it was holding up the release. So I had to abandon my wife, drive back to the hotel and fix the problem; I missed most of the dog show which was the entire reason we'd taken this vacation.

    No mention of this when I got back to the office. No compensation, no recognition, or anything.



  • @Bort said:

    Do you mean "three chords"?

    I'm pretty sure he means strings. Here's a tip, remember 'E', it's a twofer!


  • Discourse touched me in a no-no place

    @Julia said:

    And the droid reporting it had simply worked his way down all the support numbers, irrespective of how relevant they were, until he found someone who answered...

    That's the kind of thing where there ought to be a "frivolous use" penalty.



  • I'm on call one week in four or five, and I get nearly double pay that week. Monthly that means +20% on my take-home pay after taxes and social security and such. And the number of hours actually worked plus two hours per week is deductible off normal working hours, to be taken if you're too tired next morning or later when service permits. I just took a week off using that, and I have four other weeks waiting.

    The only problem is that even at +20%, I'm grossly underpaid 😦 and very seriously contemplating better-paying jobs with worse work-life balance.



  • @Bort said:

    Do you mean "three chords"?

    That too.



  • @loopback0 said:

    We get stand-by pay for being on call, and then over time for actually being called out. Minimum of 2 hours over time, even if the call out lasts 5 minutes.

    We get a (small) stand by bonus, but fuckall for actual time worked ($employer: "You're classified exempt. STFU and get back in your 6'x8' cloth lined box").

    @Eldelshell said:

    Also, it made me force my team to be more proactive on fixing shit and not releasing crap that would bring a server down in the middle of the night.

    We're horribly overbooked on project work, so we have almost no time for proactive maintenance. Naturally, this leads to a level of fragility that makes pages in the middle of the night a near certainty.


  • Grade A Premium Asshole

    @loopback0 said:

    We get a day extra annual leave per day of bank holiday we're on call for too.

    That's a good idea. I might look into incorporating that in to our compensation. If I am going to ask them to give up part of a major holiday, then it only seems fair. It is very rare that we work on holidays though. I do not like doing it, so I am very picky and very fair about having anyone who works with me do it.


  • Discourse touched me in a no-no place

    We have bill runs scheduled every 4 days so while we don't actively plan things to happen on bank holidays, it often happens through unfortunate timing.

    Christmas and New Year get evenly spread across the 4 of us on call for our team so we only do 1 of each every 4 years.



  • @KillaCoder said:

    I actually really enjoyed their last two. Not quite as top tier as you'd like but still really great I thought. I love when bands go all over the place with their style and always try new things.

    I don't mind them going outside their comfort zone ("Unsustainable" was, I think, the only decent use of dubstep I've ever heard). But their lyrics get silly in a bad way too often now, ("Animals" is one of the stupidest pieces of writing ever, barring only "Survival", which I still think is a joke), and a lot of songs get too radio-friendly, too orchestrated and just... bland and forgettable.

    They're really good at sheer power, but instead they opt for electronica and symphonies, which they can't quite pull off.



  • This post is deleted!


  • Sit down @sneakydave. Now open your mouth and lets have a look at that brain.


Log in to reply