A Backup Story
-
I'm working at a small company which wants to be Web 2.0 but noone actually knows what that would mean. The IT department consists of me, J. and an apprentice. I'm writing the software that we use to provide services to our customers. J. adapts it to the specific needs of our clients. He is also the sysadmin of our web and email server.
Just to get you in the mood, J. is the kind of guy who writes functions like this:
def foo(bar):
if bar == True:
return False
But I'm not going down this road. That would be too harmful to my blood pressure. This is the story of a backup.
For some time we had severe problems with the reliability of our web and email server or more generally with the systems that J. takes care of.
The story begins with a system storing data that where collected over the past years. It was actively used by one of our clients who processed the data and saved the processing state on this server, too. When that system crashed, the software could be restored from a local copy. But all the data was lost. J. dealt with this by inserting new data into the system that was supposed to be similar to the old data. But the original data and the processing information of the client were lost forever. I still can't understand why this client didn't leave us on the spot.
The next chapter is about our mail server. The system had some strange behavior that resulted in it refusing to send mail to some domains. Usually that manifested itself in the following way: Someone from marketing sent an email to a potential client. About a week later they phoned the potential client and asked about the email. The potential client told them that they didn't get an email. The marketing guy went to J. and complained about the mail system not sending his mail. J. inserts the domain of the email address into some file on the mail server so that the server will no longer ignore mail to this potential client. Repeat this for maybe two potential clients per week and too many weeks.
Since it was obvious that there were some problems with our servers it was decided to do something about it. The solution was not to read the manual and configure the software properly, but to change to a different hosting company. And since it was realized that we need some kind of backup we got not only a new web and mail server but also an additional backup server.
After some time our brand new server developed a bad habit of crashing now and then, usually once per week. J. worked some time on the problem, but it persisted. It was decided that the easiest fix was that the first one to notice that the server was down, just had to go to the website of the hosting company and click on a link to initiate a hardware reset. This kind of solution worked for some months until the big crash happened.
In the meantime our backup server showed a first weakness. Our hosting company cut it from the net because of excessive traffic from it. As it turned out the server fell victim to an automated attack and was used to scan other servers for security vulnerabilities. In the following emergency meeting J. stated that it wasn't his fault because he never took care for this server. I interpret that as "I'm the sysadmin and I didn't do my job."
When the big crash happened the server did not come up again. After booting the rescue system and figuring out how to call fsck, J. tried to repair the filesystem. But after the repair everything was gone. But luckily we had a backup server, right? We just need to populate the database from a backup and switch the ip address on the DNS server and we are up again, right?
Well, it turned out that the backup server was only used to store backups and since the backup server was obviously insecure the backups had to be encrypted. Unfortunately the only place the encryption key was stored, was the crashed server. After frantically trying for seven days to bring up the company's web and email service, J. gave up. Our boss ordered me to have a look at the backup. It turned out that there where older backups in various places: /var/backups, /home/backups and /root/backup. Obviously several backup system had been tested and the test backups were still intact. In one of them I actually found the encryption key for the currently used backup software. Then I had a look at the backup. It consisted of one full backup that included /dev and /var/lib/mysql (I'm pretty sure without first shutting down the database server) and then three months and over 4000 files of differential backups.
But the saddest fact and biggest WTF is that J. is still in charge of all the important system and still applies the same principles: no knowledge, no documentation and "I don't care".
-
Sometimes, reading TDWTF gives me a laugh, brightens my day.
Other days, like today, it just makes me sad.
I hope they're paying you a lot to put up with this.
-
@whyohwhy said:
Just to get you in the mood, J. is the kind of guy who writes functions like this:
def foo(bar):
if bar == True:
return FalseLet me guess... when you explain to him why this is wrong he just doesn't care/understand.
@whyohwhy said:
it turned out that the backup server was only used to store backups and since the backup server was obviously insecure the backups had to be encrypted. Unfortunately the only place the encryption key was stored, was the crashed server.
Beautiful. Just beautiful. Reminds me a little of the time I had stored a copy of winrar in a safe place in case I needed to do a complete reinstall of windows. When the time came to reinstall it I realised I had zipped winrar.
-
@DOA said:
@whyohwhy said:
Just to get you in the mood, J. is the kind of guy who writes functions like this:
def foo(bar):
if bar == True:
return FalseLet me guess... when you explain to him why this is wrong he just doesn't care/understand.
@whyohwhy said:
it turned out that the backup server was only used to store backups and since the backup server was obviously insecure the backups had to be encrypted. Unfortunately the only place the encryption key was stored, was the crashed server.
Beautiful. Just beautiful. Reminds me a little of the time I had stored a copy of winrar in a safe place in case I needed to do a complete reinstall of windows. When the time came to reinstall it I realised I had zipped winrar.
Imagine Rar-ing everything you wanted to save (including a licensed copy of winrar) before a destructive restore, doing that, and then realising you had rar-ed WinRar & the WinRar installer so you were no longer able to open files because the internet connection to the comp was gone do to a malicious virus.
-
@Steeldragon said:
Imagine Rar-ing everything you wanted to save (including a licensed copy of winrar) before a destructive restore, doing that, and then realising you had rar-ed WinRar & the WinRar installer so you were no longer able to open files because the internet connection to the comp was gone do to a malicious virus.
I've had that happen before (not the virus part, just re-install). But, having a spare connection is one of the benefits of having multiple computers. Also, burning off the installs of all major software downloads helps out tremendously on a re-install. My external drive has hundreds of installers for various utilities and other applications that I use, too. I can cart it around between computers. Right now I have it hooked up at work since I keep all of my ripped MP3s on it.
-
It sounds all too similar to my last job... Sorry to hear it.
-
@AbbydonKrafts said:
I've had that happen before (not the virus part, just re-install). But, having a spare connection is one of the benefits of having multiple computers. Also, burning off the installs of all major software downloads helps out tremendously on a re-install. My external drive has hundreds of installers for various utilities and other applications that I use, too. I can cart it around between computers. Right now I have it hooked up at work since I keep all of my ripped MP3s on it.
I do more or less the same thing. Whenever I want to do a clean install, I make a list (on paper) of all the programs I have currently installed that I wish to keep. Then I check off the ones I already have install CDs for - Visual Studio, Office, etc. Then I download new installers for the rest (avast antivirus, OpenOffice, mIRC, FileZilla, etc), burn them all to a CD, and throw out my old "installers" CD. This way I always have a relatively recent "installers" CD in case something scary happens and I have to do an emergency clean install. I keep my e-mails on an IMAP server somewhere out there in the ether, so I just have to backup my savegames, ripped mp3s, and documents...
Used to be that all of my music and documents and stuff fit on one CD. The last iteration used two full DVDs (not including the installers CD).
-
I don't keep any documents in C:, that's strictly for OS and programs.
The documents are all on D:, a 1TB partition, that is regularly backed-up. A folder in there holds approx. 60GB of app installers, including the stuff I also have on CDs. It's MUCH faster to install software from a SATA RAID drive than from a CD/DVD...
So if I want to reinstall, I only have to do a quick 10min check that I don't have stuff laying on C: by mistake, then format it, and resinstall everything from the installers on D:. Usually I'm done in less than 3 hours, with all the programs I need/want.
I also experienced the RAR'd winrar once when I had my first PC. Seeing the number of people having done it, I guess it's part of the learning curve...
-
@Heron said:
I do more or less the same thing. Whenever I want to do a clean install, I make a list (on paper) of all the programs I have currently installed that I wish to keep. Then I check off the ones I already have install CDs for - Visual Studio, Office, etc. Then I download new installers for the rest (avast antivirus, OpenOffice, mIRC, FileZilla, etc), burn them all to a CD, and throw out my old "installers" CD. This way I always have a relatively recent "installers" CD in case something scary happens and I have to do an emergency clean install. I keep my e-mails on an IMAP server somewhere out there in the ether, so I just have to backup my savegames, ripped mp3s, and documents...
That's the most sane way I've ever heard of to backup a personal PC. <sarcasm>The only thing missing is merging all your text into one giant text file</sarcasm>
@Heron said:
Used to be that all of my music and documents and stuff fit on one CD. The last iteration used two full DVDs (not including the installers CD).
I think I'm up to 26 single-layer DVDs just for music. I went on a downloading rampage a few years ago.
-
@belgariontheking said:
26 DVDs? At what point do you think "hmm a cheap external HD will be far faster and more efficient, yet still in the same ballpark cost/GB-wise"?I think I'm up to 26 single-layer DVDs just for music. I went on a downloading rampage a few years ago.
(Please say you're joking...)
-
@RayS said:
@belgariontheking said:
26 DVDs? At what point do you think "hmm a cheap external HD will be far faster and more efficient, yet still in the same ballpark cost/GB-wise"?I think I'm up to 26 single-layer DVDs just for music. I went on a downloading rampage a few years ago.
(Please say you're joking...)
It actually sounds an awful lot like SpectateSwamp....
-
@RayS said:
@belgariontheking said:
26 DVDs? At what point do you think "hmm a cheap external HD will be far faster and more efficient, yet still in the same ballpark cost/GB-wise"?I think I'm up to 26 single-layer DVDs just for music. I went on a downloading rampage a few years ago.
(Please say you're joking...)
I guess he isn't, but that will come soon enough. I did that step last year, having a pile of mixed archive CDs/DVDs (as the first ones were about 10 years old), totalling maybe 30 for program installers, 25 for music, 45 for videos, another 30 for my own content,...
I bought a 400GB drive just for that, along with a new DVD drive as I knew some discs would be problematic and a new / second drive would be a good help, and the copying / sorting started. And took me a mere 3 weeks full time.
So yeah, you DO know it's better to go with an HD, but you also fear the work you'll have to go through to reorganise your stuff. For sure there's no need to keep 10 year-old programs...
Now everything is on the PC's internal drive, so no more juggling either with discs or the external HD(s), and the latter only holds a backup. Soo... data integrity is also much better guaranteed. Optical media sucks big time for that, and it's always what you want the most that ends being on the disc that's unreadable...
And I've recently done the same with another set of about 100 CDs, everything thrown onto a 320GB external (stuff I VERY rarely use, and don't even really care if I lose, so no backup needed). Makes quite a lotta room on the shelves too.
-
@Kilrah said:
For sure there's no need to keep 10 year-old programs...
I do it for nostalgia. I've got ones going back to DOS. I only wish I could've extracted stuff off of the floppies I have before they got corrupted. Now they are just my personal museum pieces.
-
Oh don't worry, I do have some I keep for that purpose too. Just only those that make some sense keeping, at least to me. Gotta love shoving an MP3 onto Winamp 0.2a once in a while. Preferably one of the first 3 MP3s I was given somewhere in 1998.
-
@Heron said:
Used to be that all of my music and documents and stuff fit on one CD. The last iteration used two full DVDs (not including the installers CD).
Amazing. You have but 8 gigs worth of stuff? I always thought my ~30 was on the small side. I know someone with 120G of music alone.
@Kilrah said:You also fear the work you'll have to go through to reorganise your stuff.
Heh. Organize while you work, is my motto. Most of my stuff is copy-ready as it is. Though I'd have to look around for TB email data and savegames and settings files for programs (I'm a hell of a customizer and'd really hate to set things up again if I ever need to reinstall).Not sure I understand the joy in having Winamp 0.2a. Old shit >> /dev/null :)
-
@RayS said:
26 DVDs? At what point do you think "hmm a cheap external HD will be far faster and more efficient, yet still in the same ballpark cost/GB-wise"?
(Please say you're joking...)
Eh, I really only have them for major disaster recovery. All my multimedia (and savegames) are on a separate internal hard drive. So I can format the C:\ drive and be fine. The problem I'm having right now is that I have to hunt down all the stuff on the C:\ drive and back IT up before formatting.
I've only created the 26 DVDs once, and I only plan to create them once.
-
I've found at least one solution to dealing with these kinds of people. Don't. Get a new job. I spent 2.3ish years in a job where I was fourth on the totem pole and the only other competent people were directly above and below me. The top two went to a GIS guy and someone with a CS degree from the Cracker Jack University of Bogota. Imagine losing 3+ months of data because someone decided they didn't feel like backing up that folder any more... Or explaining to your boss why running VMware on the crusty old hardware you're trying to replace with VMware is a bad idea.
I'm now conracting at a much larger company, and I'm so much happier.
-
@dhromed said:
@Heron said:
Used to be that all of my music and documents and stuff fit on one CD. The last iteration used two full DVDs (not including the installers CD).
Amazing. You have but 8 gigs worth of stuff? I always thought my ~30 was on the small side. I know someone with 120G of music alone.I haven't measured my stuff in bytes for years - the numbers keep changing because of different formats and storage mediums and the like.
I measure my stuff in feet of shelf space. Currently at around three.
