Pre-WTF: Help (dear god, help!)


  • Trolleybus Mechanic

     After a couple client requests, and a near-miss or two on our own servers, the place I work have decided we should "do backups". We had a meeting, figured out what needs to be backed up, and talked about backup strategies. Everything seemed to be good. We nailed down a plan for incremental backups, storing data offsite, having a restore plan, etc. All good... until today.

     The boss has this idea, you see. Why should we bother with things like Windows Backup, or having removable drives, when there's a MUCH better way. You see, you can set a network folder to "Make available offline", you see. And then it copies it it to your local computer. So it's backed up!  So every developer will chose a folder, and have it offline on their workstation. And now everytime a developer logs in, the folder is automatically backed up!

     I've never been steamrolled by such a massively-- umm-- bad idea. I was literally (literally) speechless. There were just so many screaminly awful things about the plan that they just jammed up in my brain. It would technically work, in the same way that you can technically warm an apartment by putting a flaming bag of nail-polish soakd poo in your refigerator.  All I managed was "I have-- concerns".

    So he asked me to "do some research" and figure out if this would work or not. I've come up with a, let's say, gentle list of cons. But am I totally offbase on this one? Is it a cromulant backup strategy that I'm just overlooking?

    Or, if you prefer, how would you point out the myriad of ways this is wrong?

    I'm being given a chance to prevent a wtf. Help!



  • a) this doesn't do offsite

    b) fred goes out of town for a week and nothing's backing up his designated folder because he's not logging in

    c) joe gets hit by a bus... can anyone get onto his system to get the backed up data?

    d) this doesn't do incremental

    e) a user has to be able to read a folder to use "make available offline". Be careful who's backing up payroll data.

    f) this is not a system that is going to satisfy client requests if they know what's actually going on

     



  • @Lorne Kates said:

    Or, if you prefer, how would you point out the myriad of ways this is wrong?

    Jesus fuck.

    Just expense a copy of Mozy and put it on the server behind his back. It's $7 + $.50/GB a month. And it does Exchange and SQL databases. You can't possibly do backups cheaper, or easier, than that on your own.

    BTW, is the server just files? What about your database? How does he "plan" to back those up?



  • (adding to Suutar's answer)

    1. it might inadvertently back up data in the middle of update producing inconsistent and therefore useless backup (while normal backup tool can temporarily stop updates)
    2. it only keeps the latest backup, so if that already contains the damaged data, the good data no longer exist
    3. need for restoring backup will usually be found during working hours when everybody is logged in, but by that time the damaged data were already backed up (and thus overwrote any good ones)

    As a result the chance of successfully restoring a working version from this is almost zero, so it's useless.



  • j. Staff turnover will be a problem; do you reallocate backup folders whenever someone leaves or joins? Which leads to:
    k. Sooner or later, someone is going to leave and some of the folders they were backing up are going to be missed in the redistribution.
    l. How were you planning to verify the backup?
    m. Are any of the network folders too large to be backed up on the developers' machines?



  • @Lorne Kates said:

    Or, if you prefer, how would you point out the myriad of ways this is wrong?

    I'm being given a chance to prevent a wtf. Help!

     

    Well, the main reason is, that's not a backup, it is, at best, a mirror. And not even really that.

    Someone accidently deletes a file - can you restore it? maybe, it depends if it's also been deleted from the offline files or not yet.

    Someone spent a week redesigning something for a client, only for the client to say they actually preferred the previous version after all. Can you restore from 'backup' that file as it was a week ago? Unlikely.

    Zad



  • Try to equate what everyone else has mentioned to monetary loss. If your boss ignores that, his superiors may step in.  For example, if back ups are in the contract with your clients and you don't implement a "Standard" strategy, then your clients may sue.  Or after they find out you may have a mass exodus of clients.  Either way it will financially hurt your company in the short and long term.



  • The most straightforward reason why this is a useless "backup" strategy is because every time a change is made to the real data, that change will be mirrored on the "backup". Makingit utterly useless except in the rare case where the developer assigned to mirror the folder that got screwed up happened to be on vacation during the screwup. 

    And here's what would happen in that case:

    "Oh shit, we deleted IMPORTANT_DATA!"

     "Don't worry, that data is backed up on Paula Bean's computer!"

     "Oh shit, we'd better call Paula Bean in from her vacation!"

     *Paula is rushed in, where she promptly logs in to her computer to save the day*

     *IMPORTANT_DATA is deleted from Paula Bean's computer to match the network drive*

     

    So essentially this plan is completely useless.



  • I can also add (from many, many experiences) that the windows CSC/Offline Files component shits itself in a variety of fun ways regularly. The most 'hilarious' failure was when it corrupted the local cache so it thought it was empty, and then deleted everything off the network share to match...

     Thankfully we have proper backups so I restored everything from tape after having to use csccmd.exe to manually un-bugger things on the client.

     I do like a program called 'SecondCopy' as a folder sync tool through, shame its not free but it does do what Offline Files is supposed to do.

     

    edit: also if you replace the server hosting the network files or move the shares it will never ever work properly ever again, ever. Trying to coax it to migrate with GPOs, csccmd.exe and finally manually editing the registry haunt my dreams... 



  • @Cursorkeys said:

     I do like a program called 'SecondCopy' as a folder sync tool through, shame its not free but it does do what Offline Files is supposed to do.
    In addition to this and blakey's suggestion, what's wrong with Windows' native backup tool (assuming the servers are running Windows)?  Doesn't get any free-er than that.

    Additionally, there's SyncBack along with probably a slew of other tools, but I use SyncBack to back up my HTPC to a second internal hard drive weekly.  It's painless and gives me a report of what it backed up.  Not that I read it, of course (30 some pages of file lists a week), but if needed, I could find out what was backed up when.  However, unlike Windows Backup, it doesn't create a backup file that you need their backup software to navigate and restore from.  It just straight up copies the files over, so you don't get a historical backup.

    Point out that their solution has all these problems and that other solutions are simpler and less prone to problems, and equally free.

     



  • @belgariontheking said:

    @Cursorkeys said:
     I do like a program called 'SecondCopy' as a folder sync tool through, shame its not free but it does do what Offline Files is supposed to do.
    In addition to this and blakey's suggestion, what's wrong with Windows' native backup tool (assuming the servers are running Windows)?  Doesn't get any free-er than that.

    I wager they're too cheap to buy disks. Which is an incredible false economy, but then again, we're talking about a company that didn't do backups at all. They also seem to be unaware of this whole "cloud storage" thing, meaning they don't need to buy disks to get good backups. So there's a high WTF factor at work here.

    @belgariontheking said:

    Additionally, there's SyncBack along with probably a slew of other tools, but I use SyncBack to back up my HTPC to a second internal hard drive weekly.

    SyncBack is good mojo, and also cheap. We use it for mirroring the content of a webserver (files, no databases) so that we can use a QA server based in Seattle, instead of a slow and laggy one based in Atlanta. Works like a champ, I never have to even think about it.



  • @Cursorkeys said:

    I can also add (from many, many experiences) that the windows CSC/Offline Files component shits itself in a variety of fun ways regularly. The most 'hilarious' failure was when it corrupted the local cache so it thought it was empty, and then deleted everything off the network share to match...
    This. I've seen it happen more than once.



  • @blakeyrat said:

    Just expense a copy of Mozy and put it on the server behind his back. It's $7 + $.50/GB a month. And it does Exchange and SQL databases. You can't possibly do backups cheaper, or easier, than that on your own.
     

    I've no idea how large Lorne's company is, but that would be about $500/month for a 1TB data store. Probably a reasoneable price for a backup service, but if they can afford that from the coffee fund, I'd like to apply for a job please, backup or no backup.



  • @PSWorx said:

    @blakeyrat said:
    Just expense a copy of Mozy and put it on the server behind his back. It's $7 + $.50/GB a month. And it does Exchange and SQL databases. You can't possibly do backups cheaper, or easier, than that on your own.
    I've no idea how large Lorne's company is, but that would be about $500/month for a 1TB data store. Probably a reasoneable price for a backup service, but if they can afford that from the coffee fund, I'd like to apply for a job please, backup or no backup.

    That's less than a single employee's benefits for a month. (Or if not, damned close.) Your company would have to have pretty fucked-up priorities to consider $500/month for backups some kind of crazy-high expense. For any company big enough to have enough employees to "divvy up" backups on their workstations, it's completely trivial.

    And it's still probably an order of magnitude less than doing it in-house, if you add in labor costs. (Mozy is like a single day worth of setup, max. Then maybe a few hours a month testing a restore.)

    Edit: one trend I've noticed on this and a lot of other tech forums, IT workers frequently have no clue how much labor actually costs. I find that weird.



  • @blakeyrat said:

    @PSWorx said:
    @blakeyrat said:
    Just expense a copy of Mozy and put it on the server behind his back. It's $7 + $.50/GB a month. And it does Exchange and SQL databases. You can't possibly do backups cheaper, or easier, than that on your own.
    I've no idea how large Lorne's company is, but that would be about $500/month for a 1TB data store. Probably a reasoneable price for a backup service, but if they can afford that from the coffee fund, I'd like to apply for a job please, backup or no backup.
    That's less than a single employee's benefits for a month. (Or if not, damned close.) Your company would have to have pretty fucked-up priorities to consider $500/month for backups some kind of crazy-high expense. For any company big enough to have enough employees to "divvy up" backups on their workstations, it's completely trivial.

    And it's still probably an order of magnitude less than doing it in-house, if you add in labor costs. (Mozy is like a single day worth of setup, max. Then maybe a few hours a month testing a restore.)

    Edit: one trend I've noticed on this and a lot of other tech forums, IT workers frequently have no clue how much labor actually costs. I find that weird.

    Hmm, labor is cheap in some parts of the world (read about outsourcing)

    Also common sense does not factor into this, if it did then this site wouldn't exist



  • @blakeyrat said:

    @Lorne Kates said:
    Or, if you prefer, how would you point out the myriad of ways this is wrong?

    Jesus fuck.

    I find your language offensive. Take your infantilism elsewhere.



  • @blakeyrat said:

    That's less than a single employee's benefits for a month. (Or if not, damned close.) Your company would have to have pretty fucked-up priorities to consider $500/month for backups some kind of crazy-high expense. For any company big enough to have enough employees to "divvy up" backups on their workstations, it's completely trivial.
     

    Obviously it's peanuts for company budget. But it's probably a bit expensive for employees to pay on their own if they want to put it on the server behind the boss' back. :)

     



  • @PSWorx said:

    @blakeyrat said:
    That's less than a single employee's benefits for a month. (Or if not, damned close.) Your company would have to have pretty fucked-up priorities to consider $500/month for backups some kind of crazy-high expense. For any company big enough to have enough employees to "divvy up" backups on their workstations, it's completely trivial.
    Obviously it's peanuts for company budget. But it's probably a bit expensive for employees to pay on their own if they want to put it on the server behind the boss' back. :)

    That's why you expense it. Like I said in the original post.



  • @SilentRunner said:

    @blakeyrat said:
    @Lorne Kates said:
    Or, if you prefer, how would you point out the myriad of ways this is wrong?

    Jesus fuck.

    I find your language offensive. Take your infantilism elsewhere.

    I find the stick up your ass amusing. Please continue to bitch about stupid shit here.



  • @toth said:

    @SilentRunner said:
    @blakeyrat said:
    @Lorne Kates said:
    Or, if you prefer, how would you point out the myriad of ways this is wrong?

    Jesus fuck.

    I find your language offensive. Take your infantilism elsewhere.

    I find the stick up your ass amusing. Please continue to bitch about stupid shit here.

    His posting history is awesome. It's mostly being an insufferable Grammar Nazi, with some hints of being an actual Nazi. "You pluralized a corporation's name in a way that is technically correct but I don't personally like! Damned Democraps!"


  • Trolleybus Mechanic

    Excellent points. I didn't even think about Paula scenerio.

    I think I am going to pitch an offsite, automated solution, like Blakey suggested.  (And for the record, "Jesus Fuck" is a perfectly acceptable response).  Based on the pricing, it's cheaper than having a dev try to figure it out.

    I don't know why we don't just go with Windows Backup + a few $80, 2TB removable drives (one for the backup, one for the backup of the backup, and one for the backup that goes home with someone as to be offsite). When we were first talking about a backup strategy, that was the plan.

    Thankfully, this was meant to be an internal solution only. Backing up the data and websites of any client on an unmanaged server is whole other story. Thankfully, not a front-page worthy story (yet).



  • @belgariontheking said:

    @Cursorkeys said:
     I do like a program called 'SecondCopy' as a folder sync tool through, shame its not free but it does do what Offline Files is supposed to do.
    In addition to this and blakey's suggestion, what's wrong with Windows' native backup tool (assuming the servers are running Windows)?  Doesn't get any free-er than that.

    Oh boy, just get rsync and forget about it. For a monthly dollar I can give you ssh access to a dreamhost account, I bet my insurance it is going to be better than whatever your management will force you to implement.



  • @Lorne Kates said:

    Or, if you prefer, how would you point out the myriad of ways this is wrong?

    It would be easier to point out how EVERY OTHER BACKUP SOLUTION EVER INVENTED is better than your boss's "idea". (In this case, "EVERY OTHER BACKUP SOLUTION EVER INVENTED" includes Sony's backup plan of "storing customer records on paper behind the company dumpster", a la VGcats.)



  • @blakeyrat said:

    "You pluralized a corporation's name in a way that is technically correct but I don't personally like! Damned Democraps!"

    You mean “the name of a corporation”, not “a corporation's name”. Non-personal entities may not use the contractive possessive.

     



  • @Mr. DOS said:

    Non-personal entities may not use the contractive possessive.

    Why not? Racist!



  • @blakeyrat said:

    Just expense a copy of Mozy and put it on the server behind his back. It's $7 + $.50/GB a month. And it does Exchange and SQL databases. You can't possibly do backups cheaper, or easier, than that on your own.

    Well, I have one server that is currently holding 10 TB of data, so that is $500/mo, or $6000/yr...not to mention the fact that this is fairly volatile data, and even my 25 MBPS (Upload) network connection would not keep up



  • @blakeyrat said:

    Just expense a copy of Mozy and put it on the server behind his back. It's $7 + $.50/GB a month. And it does Exchange and SQL databases. You can't possibly do backups cheaper, or easier, than that on your own.

    Well, I have one server that is currently holding 10 TB of data, so that is $500/mo, or $6000/yr...not to mention the fact that this is fairly volatile data, and even my 25 MBPS (Upload) network connection would not keep up



  • All of the above. CSC is neither a backup nor a strategy. CACHE! CACHE! CACHE!

    The fun thing with WSB, if you have a single server to back up, is that it can do bare-metal recovery. Just boot from the Win2008 DVD with the backup disk attached and after a few clicks it will do a full system recovery for you (to similar hardware).

    Not familiar with Mozy, but I would have a method of bootstrapping a replacement server ready, just in case. Also, the amount of data may become prohibitive

    I recommend a minimum of three drives to keep off-site backups and rotation.

    Keep pushing.



  • Remember that you don't test a backup strategy. You test a restore strategy. If all else fails, you should plant a dummy file, delete it, and then run a "omg deleted file!" drill and see how poorly it fares.



  • @TheCPUWizard said:

    @blakeyrat said:

    Just expense a copy of Mozy and put it on the server behind his back. It's $7 + $.50/GB a month. And it does Exchange and SQL databases. You can't possibly do backups cheaper, or easier, than that on your own.

    Well, I have one server that is currently holding 10 TB of data, so that is $500/mo, or $6000/yr...not to mention the fact that this is fairly volatile data, and even my 25 MBPS (Upload) network connection would not keep up

    Lets see... A 16 tape rotation (5 dailies, 4 weeklies, 6 monthlies, and 1 end of year) times 6 tapes per backup (assuming LTO 5 and poor compression) is about $7500 in tapes. A decent size library with two drives runs about $15000. A copy of NetBackup runs about $10000 with a library license. In the ideal case where the cost of the library and software are spread over many systems and the tapes last three years, you are looking at about $5000/yr before labor to back up 10TB yourself. You can do disk-to-disk cheaper, but then you don't have an off site backup (or you are back to your bandwidth problem again).



    Your numbers may vary from the sample presented here, but the point is that $6000 isn't all that expensive for 10TB of backup. On the other hand, if you ever need to do a disaster recovery of 10TB from Mozy, you are totally screwed. 10TB takes an entire day to transmit over a gigabit connection.



  • @Jaime said:

    On the other hand, if you ever need to do a disaster recovery of 10TB from Mozy, you are totally screwed. 10TB takes an entire day to transmit over a gigabit connection.

    Mozy will FedEx disks, but you bet your ass they charge for it.

    Still, you'd be "SAVIOR OF THE BUSINESS!!!" so keep that in mind.


Log in to reply