SysAdmins are not flawless... a.k.a. my biggest screw-up ever
-
I might not be a real programmer, but at least I am a better-than-average scripter. Well usually.
A customer hates to invest in IT, so he still has an Exchange 2003 running. Obviously he also doesn't want to invest in a backup solution. Who needs backups?
Well, we did this week after a power outage: one of the storage devices did not survive. Fortunately we did have backups, as I wrote a pretty decent script that takes care of backups. For about 500 VMs, every night, it does simplified the following:
- Check which VMs belong together (i.e. share a Netapp storage)
- Connect to the appropriate ESXes
- Take a snapshot of the VMs
- Take a NetApp snapshot
- Remove VM snapshots
- Disconnect from ESXes
The reason my script takes an ESX snapshot first instead of simply a Netapp snapshot is consistency. Surely databases - and Exchange is a database - don't like inconsistency.
This script worked perfectly when I made it in October. According to my logging, it still did. Until we restored the Exchange server and we got some strange behavior. Turns out none of my backups have a snapshot and thus none of my backups are consistent. A quick check learned that this is the case since... December.
I found the problem quite fast. I don't have my real source here at home, but in my improv-code you will see immediately what happened.
God kill me. I screwed up a lot of things by at one point wanting to test my Catch block and commenting out my command that takes snapshots (because I didn't want to execute that during the test). Every time the script runs, it happily goes into the Try block and doesn't encounter any problems there of course. On a closer look, it turned out that I did generate errors during the removal of the snapshots - and I catched those as well. But due to another logical fail (a Catch at that point no longer interrupts the script), my monitoring e-mails still happily made a note 'Huge success'.
Now goodbye while I go and kill myself.
-
@YellowOnline said in SysAdmins are not flawless... a.k.a. my biggest screw-up ever:
'Huge success'
It was a triumph.
-
@Tsaukpaetra It's hard to overstate his satisfaction
-
@YellowOnline Your story motivated me to go and check my backup crons. Turns out they were erroring out for weeks.
So... good job, I guess?
-
@RaceProUK But there's no sense crying over every mistake.
-
Yes yes we all know the damned song. Repeating someone else's joke is not a joke.
-
@YellowOnline Protip:
try { throw new Exception(@"Test!"); ...Normal code } catch(Exception ex) { .... }
Throw instead of comment out if you want to test your catch blocks.
-
@Matches I will remember that. In Powershell I can just do Throw "Test!". Porting C# to PoSh is usually very easy (because we're all .NET family...).
-
@YellowOnline And the fact that powershell supports actual c# scripts...
-
@blakeyrat said in SysAdmins are not flawless... a.k.a. my biggest screw-up ever:
Yes yes we all know the damned song. Repeating someone else's joke is not a joke.
You obviously need some cake.
-
@Rhywden But you wouldn't know, would you?
-
@blakeyrat said in SysAdmins are not flawless... a.k.a. my biggest screw-up ever:
Repeating someone else's joke is not a joke.
Saying something obviously false (and with no hint of irony whatsoever) then laughing really hard also isn't a joke, but no one ever told the guy who sits next to me.
-
@YellowOnline said in SysAdmins are not flawless... a.k.a. my biggest screw-up ever:
@Matches I will remember that. In Powershell I can just do Throw "Test!". Porting C# to PoSh is usually very easy (because we're all .NET family...).
Well, you could just do
Write-Warning
orWrite-Error
.Throw
is a terminating error; you don't always want that.
-
@YellowOnline said in SysAdmins are not flawless... a.k.a. my biggest screw-up ever:
Obviously he also doesn't want to invest in a backup solution. Who needs backups?
Let them burn. They don't deserve your saving
-
@Jaime said in SysAdmins are not flawless... a.k.a. my biggest screw-up ever:
@blakeyrat said in SysAdmins are not flawless... a.k.a. my biggest screw-up ever:
Repeating someone else's joke is not a joke.
Saying something obviously false (and with no hint of irony whatsoever) then laughing really hard also isn't a joke, but no one ever told the guy who sits next to me.
Phil Ken Sebben "Ha Ha Ha" supercut – 03:58
— OAFE
-
@JazzyJosh said in SysAdmins are not flawless... a.k.a. my biggest screw-up ever:
@Rhywden But you wouldn't know, would you?
Funny story, I forgot how much of a big, fat mess you are.